Examining the Use of Real‐World Evidence in the Regulatory Process

Abstract

The 21st Century Cures Act passed by the United States Congress mandates the US Food and Drug Administration to develop guidance to evaluate the use of real‐world evidence (RWE) to support the regulatory process. RWE has generated important medical discoveries, especially in areas where traditional clinical trials would be unethical or infeasible. However, RWE suffers from several issues that hinder its ability to provide proof of treatment efficacy at a level comparable to randomized controlled trials. In this review article, we summarized the advantages and limitations of RWE, identified the key opportunities for RWE, and pointed the way forward to maximize the potential of RWE for regulatory purposes.

Real‐world data (RWD) and real‐world evidence (RWE) have received substantial attention from medical researchers and regulators in recent years.1, 2 The US Food and Drug Administration (FDA) defines data relating to patient health status and the delivery of healthcare (such as electronic health records (EHRs), claims and billing activities, product and disease registries, and patient‐generated data) as real‐world data (RWD), and the analysis of these data regarding usage and effectiveness are termed real‐world evidence (RWE).3 The European Medicines Agency (EMA) similarly defines RWD as defined as “routinely collected data relating to a patient's health status or the delivery of health care from a variety of sources other than traditional clinical trials” and expressed interest in using RWD for regulatory decision making.4 RWE presents great potential to accelerate therapy development and to monitor the successes and failures or both newly approved and existing therapies.5 It is critical that stakeholders, including researchers, both academic and industry, providers, regulators, administrators, and patients understand the limitations of RWE. RWE is not generated with a particular study question in mind but are generated primarily for clinical care and billing purposes. As such, appropriate use of RWE must be driven by well‐designed guidelines and regulations to ensure accurate, unbiased findings. If used correctly, RWE could supplement traditional clinical research to aid therapeutic development, clinical decision making efficiency gains in healthcare, and improved access to therapeutics for underserved populations. Examples of promise in each of these areas can be seen in the results of the EMA's adaptive pathways pilot.6 If used incorrectly, RWE could lead to spurious approvals, financial waste, and most importantly cause harm to patients.

To date, RWE has been used primarily to perform postmarketing surveillance to monitor drug safety and detect adverse events. RWE has also been particularly effective when the outcome of interest is rare, in cases where a very long follow‐up period is required to assess the health outcomes, or when it is difficult to perform randomized controlled trials (RCTs), such as in pediatric or pregnant populations. An early example was the discovery of a link between the ingestion of diethylstilbestrol during pregnancy and vaginal adenocarcinoma of the offsprings using observational data.7 More recent studies linked the use of angiotensin‐converting enzyme inhibitors while pregnant to congenital malformations8 and the exposure of selective serotonin reuptake inhibitors to persistent pulmonary hypertension in newborns.9, 10 Most recently, RWE has shown postmarketing evidence that it may have an important role to play in understanding drug effectiveness and adverse events based on differences of metabolism in various racial and genetic groups.11, 12, 13, 14

There is a growing interest in the usage of RWE by regulatory agencies to evaluate the safety and efficacy of medical treatments.2, 5, 15, 16, 17 In particular, Congress has mandated that the FDA increase focus on RWE for regulatory decision making both for new approvals and evaluating additional indications for approved therapies18 and the FDA has testified on progress toward implementing this focus.19 The EMA recently accepted an RWE‐based control arm during their analysis of Alecensa effectiveness compared with the standard of care.20, 21, 22 In addition, the FDA has established partnerships with private companies whose goal is to use RWD in regulatory decision making, including using synthetic control arms.20

Although some have been enthusiastic about the ability for observational RWD to substitute for RCTs, others have expressed caution. Booth et al.23 contend that RWD should not be used as a replacement for clinical trials due to the inability to compare outcomes of nonrandomized groups. A recent comprehensive empirical analysis of treatments in oncology confirms this finding. The results on replicating clinical trials in observational data are highly mixed, Concato et al.24 concluded that well‐designed observational trials closely estimated the effects of treatment when compared with RCTs on the same subject. On the other hand, Soni et al.25 found a poor correlation between the hazard ratio seen in observational studies vs. randomized trials on the same topic. There is limited evidence that some RCTs may be difficult to replicate.26, 27, 28 This could be due to population and effect sizes, population demographics, or other factors.

RWE may provide valuable insight into the effectiveness and generalizability of interventions in practice, even as RCTs are unlikely to be supplanted as the gold standard for measuring intervention efficacy. It is important to consider the higher level of evidence RCTs provide than RWE when making a regulatory decision.29 We should aim to use the highest standard possible while acknowledging it is not feasible for RCTs to answer all clinical questions related to drug effectiveness. The reasons where RCT might not be feasible include (i) prohibitive cost,30, 31 (ii) when the standard of care is effective and/or administering a placebo is unethical,32 and (iii) in rare diseases where patient recruitment is challenging.33, 34, 35 In addition, RCTs are typically performed in a relatively homogenous cohort that is less diverse than the real‐world population in terms of age, race, socioeconomic status, geography, clinical setting, disease severity, patient history, and patient willingness to seek treatment.36, 37, 38, 39, 40 Finally, over time, indications for therapeutics often expand to indications and population groups they were not originally tested in. Pragmatic and other modern trial designs may mitigate some of these challenges, for example, by improving generalizability, but fundamental issues of cost, time, and difficult recruitment remain.41 Pharmaceuticals have been approved without RCTs but should be limited to cases where the potential burden of an incorrect treatment estimation is outweighed by the burden of conducting an RCT.42 It is critical to utilize alternatives to supplement but not supplant RCTs both in the form of pragmatic trials and RWE‐based analyses.

The FDA has taken action in an attempt to reduce the burden in both time and cost of bringing a new therapy to market, through the accelerated regulatory decision regulations initially put in place in 1992 and expanded in 2012.43 These regulations allow for surrogate and intermediate end points when a therapy addressed a serious condition without existing options. In the 5 years following final guidance from the FDA in May 2014, 71 therapies have been approved through the accelerated pathway.44 This is in comparison to 25 in the 5 years prior to this guidance. In addition, in 2012, the FDA established the “breakthrough therapy designation” for therapies intended to treat serious or life‐threatening conditions where the therapy may demonstrate substantial improvement.45 From April 2015 to March 2019, 24 drugs have received breakthrough therapy designation. Due to the fact that clinical end points (i.e., outcomes that show direct clinical benefits, such as increased overall survival) may take a long time to develop, many drug trials with breakthrough therapy designation use surrogate end points, which are measurements or signs predictive of clinical outcomes but do not directly measure clinical benefits.46, 47 Examples of surrogate end points include blood pressure for hypertension drugs and serum low‐density lipoprotein cholesterol for hypercholesterolemia treatments.47 Shorter trials and the use of surrogate end points present a strong need for postapproval surveillance for both safety and effectiveness, especially in the context of traditional clinical end points. The use of shorter trials and surrogate end points to accelerate regulatory decisions may suggest that the traditional process is unnecessarily slow and wasteful. However, it is also possible that the extensive use of these shortcuts will lead to suboptimal decisions because there is no guarantee improvement as measured by surrogate end points will translate to traditional end points.

In this light, we ask the question: “What is the role of RWE in the regulatory process?” In this review, we first lay out some of the primary reasons RWE is not suited to replace RCTs. We then examine some areas RWE is well‐suited to supplement and enhance the regulatory process as well as providing postapproval guidance.

Limitations for RWE for Regulatory Decision Making

Here, we examine the limitations of commonly used methods for applying RWD to inform regulatory decision making. We focus on the limitations of RWE to be used in the comparative effectiveness analyses conducted in phase II and phase III RCTs. We, therefore, narrowly examine three types of studies that have been proposed as ways to perform these comparative effectiveness analyses of therapies from RWD: (i) Virtual Comparative Effectiveness Studies ascertain outcomes in both an intervention and control group from an RWD source.48, 49 (ii) Studies using Historical Control Arms compare retrospective RWD‐derived controls against an uncontrolled treatment arm.50 (iii) Studies using Synthetic (Real‐World) Control Arms pair an uncontrolled treatment arm with concurrent RWD controls.22

Each of the three study designs of RWE has significant issues that prevent the ability to achieve a level of evidence on par with RCTs. Because of challenges in drawing causal conclusions of treatment efficacy from RWE they are not suited to replace RCTs. Table 1 shows how the specific limitations discussed apply to each form of study design. Most of these mechanisms to use RWE are affected by multiple limitations. Below, we describe these concerns and then link them to each of these three forms of evidence.

Table 1.

Examination of the issues pertinent to each purpose of RWE

	Virtual comparative effectiveness studies	Historical control arms	Synthetic control arms
A. Unobserved confounders	X	X	X
B. Medicine changes over time		X
C. Trials may change participant and provider behavior	X	X	X
D. Closer monitoring of adverse effects in trials	X	X	X
E. Lack of pretrial registration and the potential for multiple testing errors	X
F. Weaknesses of propensity score matching	X	X	X
G. Inability to compare RWD preapproval	X
H. Opportunities for conflicts of interest to affect results	X	X	X
I. Patterns of completeness of data and loss of follow‐up differ	X	X	X
J. Measurement error in identifying patient status from RWD	X	X	X

RWD, real‐world data; RWE, real‐world evidence.

A. Unobserved confounders

Of the sources of RWD, the EHR is generally considered to provide the most granular view of patient care; although insurance claims may provide a higher level of completeness of care. Neither of these data sources is designed for secondary analysis: EHR is primarily intended for patient care, whereas claims data is designed for financial billing and reimbursement. As a result, there are potentially unobserved factors influencing a physician's decision to pursue a particular course of treatment in a systematic manner, preventing the direct comparison of outcomes between treatment arms or direct comparisons to RCT findings. Traditionally, these factors are addressed using randomization because random treatment assignment would not allow for systematic differences between exposed/nonexposed arms. This is not possible using observational data. Several examples of these factors include:

1. Physician opinion. A physician may have just read a paper or attended a seminar recommending a particular treatment, may have seen the treatment work well for a patient that they deem similar, or they may simply have a gut feeling that a treatment is right for a patient. Pessimistically, pharmaceutical companies may exert influence regarding the choice of treatment.51, 52 This influence can often be associated with patient characteristics.

2. Patient request. A patient, through his or her own research or through advertisement, may request a specific course of treatment.53, 54 This bias is particularly evident in cases when the patient is attempting to optimize a different outcome from the trial (e.g., shared decision making).55

3. Knowledge of a trial. Patients who choose to enroll in a trial are likely to be distributed differently than the general disease population. Similarly, clinicians participating in a trial or prescribing an off‐label drug are likely to make different choices regarding patient care. In particular, there is some evidence that physicians are more likely to attempt experimental treatment37, 40 in those who appear healthier. For this reason, unobserved confounders can affect all three considered uses of RWE.

4. Differential access to treatment. The availability or ease of offering a particular treatment may be dependent on administrative, logistical, or insurance coverage‐based barriers.56 These differences are magnified by the integration of multisite and geographically diverse data into observational studies. Consistent annotation or quantification of these factors is not a central theme within RWD datasets, nor are the magnitudes and directions of these effects on physician behavior completely understood.

B. Medicine changes over time

Historical control arms are by nature historical and as new treatments and technologies, new guidelines, and environmental or socioeconomic changes are introduced medicine changes. This was demonstrated by Sacks et al.50 in 1982 when they showed that 80% (44 of 56) historical control trials found the treatment of interest better than the control, but only 20% of RCTs agreed. For six different clinical areas, they found the results of the trials were more dependent on the method of control groups than on the therapy being considered. In a similar vein, Zia et al.57 identified 43 phase III clinical trials that used identical therapeutic regimens to their corresponding phase II study. Only 28% of the phase III studies were “positive” and 81% had lower effect sizes than their corresponding phase II study. The effect of time trends in medicine is evident even over the course of a single outcome‐adaptive trial.58, 59 Outcome‐adaptive trials work by adjusting treatment assignment probabilities based on which treatment arm is doing better in order to subject as many participants as possible to the most promising treatment. When a treatment arm seems promising at the beginning of a trial, patients are disproportionately enrolled in the promising arm. This means that the average date of enrollment can be much later in some arms than others.

C. Trials may change participant and provider behavior—the “Hawthorne Effect”

Clinical trial protocols may result in different behavior than typical clinical practice. Although difficult to measure, there is weak evidence of a protocol or “Hawthorne effect” leading to participants of clinical trials having better outcomes than typical clinical practice.60 McCarney et al.61 performed a placebo‐based randomized trial in dementia, which found that participants receiving more frequent follow‐up visits achieved better cognitive and carer‐rated quality‐of‐life outcomes. Similarly, behavioral changes, such as the more frequent follow‐up of clinical trial participants, may result in better medication adherence than real‐world settings.62 In traditional RCTs, both arms of the trial may experience the Hawthorne effect. When using historic or synthetic controls only the traditional intervention arm would experience the Hawthorne effect.

D. Closer monitoring of adverse effects in trials

The combination of more frequent follow‐up and specific attention paid to adverse drug events may lead to lower rates of adverse drug events in routine clinical practice compared with clinical trials. In addition, relatively minor adverse effects may not be billed against or recorded in the context of more serious diagnoses (e.g., nausea on a chemotherapy protocol). Several studies have shown that harm in oncology RCTs is underreported and may not even follow protocols.63 RWE may underestimate important patient safety concerns.

E. Lack of pretrial registration and multiple testing

It is critical that clinical trials be registered prior to any analysis of results that occur to prevent unreported multiple testing.64, 65 Many statistical methods have been proposed to minimize false discovery in studies involving testing multiple hypotheses simultaneously, and the need for correction is well recognized in biomedical research. However, some forms of multiple testing are more difficult to identify. As an illustration, there are several widely distributed datasets (e.g., Truven MarketScan, Optum Claims data, etc.), and it is likely that multiple investigators will ask similar questions using these datasets. The multiple testing nature of large and uncoordinated efforts may not be apparent to individual investigators involved nor to the scientific community, and the likelihood of false positives would remain uncorrected.66 To highlight this issue, Silberzahn et al.67 distributed the same dataset to 29 teams with a total of 61 data analysts and asked the question, “Are soccer referees more likely to give red cards to dark‐skin‐toned players than light skin‐toned‐players?” Twenty of the 29 teams found a positive effect, the other 9 found no significant relationship and the estimated effect sizes ranged from 0.89–2.93. This study makes explicit a phenomenon that is largely hidden from view, multiple chances and analytical approaches to explore an observational hypothesis may result in different point estimates. Given the financial stakes in regulatory outcomes, there are strong incentives for reporting of positive results. Because explorational analyses may be performed prior to registration or in the absence of registration, multiple hypothesis testing may plague RWE efforts. Preregistration of methods in RCT studies prevents this type of manipulation, both deliberate and inadvertent, and provides a measure of methodological transparency.

F. Weaknesses of propensity score methods

Propensity score matching and adjustment are popular approaches to account for high‐dimensional confounders in observational studies.68 In propensity score matching, researchers construct a matched population of treated and nontreated individuals based on their probabilities of receiving the treatment. By pairing every treated patient with one or more nontreated patients that were roughly equally likely to have received the treatment, propensity score matching seeks to balance the underlying factors associated with treatment assignment. In propensity score adjustment, the high‐dimensional confounders are summarized by a propensity score, which can then be adjusted in downstream analyses.68, 69

In practice, propensity score methods are difficult to properly execute and evaluate.70 For instance, it is very difficult to determine if a given propensity score model has been correctly specified. The traditional method for evaluating predictive performance fails to properly evaluate the quality of a propensity score model because unmeasured confounding and the randomness in treatment assignment both contribute to the deviation of the model from the observed data. As such, it can be difficult for a reader to determine whether the model has sufficiently adjusted for confounding.70

The deployment of the propensity score is also not straightforward. Because propensity scores from two patients are rarely exactly equal, defining “close enough” propensities for two patients to count as a match involves a delicate balance between excessively loose cutoffs (which risks undercutting the notion of matching itself) and highly stringent cutoffs (which may exclude too many patients from the analysis). The process of propensity matching almost inevitably changes the population being studied in ways difficult to interpret by systematically excluding some subset of patients that fail to achieve a proper match in the other “arm” of the analysis. Ultimately, this means that propensity score matching may provide causal estimates about the effect of the intervention on a different population than the study originally sought to investigate.

G. Inability to compare RWD preapproval of the experimental treatment

The efficacy of a treatment cannot be evaluated from RWD until it is used widely enough for there to be a substantial body of data. In general, this means that treatment needs to already be approved for the indication of interest. In rare cases, there may be significant off‐label usage, but the reasoning behind this off‐label usage should be carefully considered in terms of its impacts on patient selection.71, 72 Significant differences between the patients receiving the off‐label treatment vs. the existing standard of care could exist, resulting in issues of generalizability. The inability to compare observations preapproval to observations in the general population is especially pertinent to the experimental arm, but there may also be subtle changes in the standard of care that can be difficult to identify from RWD (e.g., dosing, timing, and adherence).73, 74 Finally, off‐label prescription with the hope of generating RWD is an inefficient mechanism of hypothesis testing, which is often optimized by formal trials.

H. Opportunity for conflict of interest

Therapy approval decisions are binary outcomes with the potential for profound impact on patients, employees, shareholders, and other stakeholders. Because of this, there represents an outsized incentive for a variety of parties to cheat the system, regardless of the study design. It was recently revealed that there was “data manipulation” in the data provided to the FDA during the approval process for Zolgensma.75, 76 In RWE, this poses a potentially bigger issue given the retrospective nature of data. The bar for exploitation is lower than prospective studies and exploitation becomes less black and white. It is not possible to ensure that trial organizers have not already analyzed the data to ensure that the control arm is penalized. This could be done through manipulation of the inclusion and exclusion criteria, the method of propensity matching between the trial subjects and the subjects derived from RWD or other intentional selections. For example, Sacks et al.50 found that historical control groups generally did significantly worse than RCT control groups across 50 reported clinical trials. The retrospective population selection task exhibits the opportunity for exploitation that is difficult to uncover, especially when considering potential financial implications. Although there is the opportunity for bad actors to cheat the system regardless of study type, RWE‐based studies can be performed by a smaller set of investigators where there are less exposure and transparency to the protocol.

I. Completeness of data and loss of follow‐up

Of the three most commonly used sources of RWD, EHR and hospital‐based administration both suffer from the fact that they do not record any care received outside of a particular hospital system and oftentimes contain incomplete regard with respect to a particular EHR (e.g., inpatient vs. ambulatory). This leads to challenges in ensuring completeness of care in RWD‐based studies using EHR data. In addition, over 50% of the United States receives healthcare insurance through their employment.77 The average tenure of employment is just 4.3 years for men and 4 years for women.78 This short tenure means even insurance claims datasets present challenges when considering the completeness of care and follow‐up coverage. Record incompleteness, defined as instances when fewer than 50% of enrollees have at least one claim in a given year, can be caused by administrative phenomena, such as company or record mergers, as well as subcontracting of service.79

J. Measurement error in identifying patient status from RWD

EHRs and administrative data were not initially intended for specific studies, thus, these records may not be sufficiently granular to ascertain the phenotypic status of the patients. Researchers need to make additional assumptions or resort to proxy measures to infer the disease status of the patients under study. Additionally, previous studies showed that different hospitals have dissimilar approaches of disease and procedure coding, even when using the same standard lexicon of diagnostic and procedure codes. Data harmonization efforts and calibration studies are necessary to enhance the accuracy of inferring patient statuses using the limited, and sometimes inconsistent, descriptors in the RWD.

The Way Forward: How Can We Capture Value From RWE to Effectively Improve Patient Treatment?

Although RWD analyses are susceptible to the biases and issues summarized above, they hold promise in complementing trial analyses and enable the evaluation of numerous biological hypotheses at a minimal cost. Below, we discuss the approaches that could maximize the value and potential of RWE with particular attention to the approval and postapproval surveillance processes (Table 2).

Table 2.

Key needs and opportunities for RWE

Key areas	Examples and approaches
1. Measuring postapproval safety and effectiveness	Integrate multiple RWD study designs and leverage modern statistical approaches to better estimate the effects of treatments
	Compare differences between effectiveness and efficacy
	Establish best practices, guidelines, and reporting standards
	Follow accelerated approval and surrogate end‐point trials to understand the long‐term effects on traditional end points
2. Development of future therapies	Identify populations underserved by current therapies and clinical trials
	Discover disease subtypes or potential patient subpopulation that might benefit from novel treatment modalities
	Facilitate trial recruitment at diverse clinical sites and the inclusion of diverse populations in future studies
3. Measuring healthcare value and quality	Determine the value‐based reimbursement of drugs
	Evaluate how closely clinical guidelines are followed and whether guidelines lead to positive outcomes

RWD, real‐world data; RWE, real‐world evidence.

Integrate results from multiple designs of observational studies to triangulate effect estimates

As discussed in the previous section, RWE generated from different study designs suffers from different sources of biases, and causal inference based on RWD relies on strong assumptions rarely met in practice. Nonetheless, by combining the results generated by different study designs, we can better estimate the risks and benefits of treatment strategies.80 For example, cohort studies using RWD suffer from unmeasured confounding, whereas the use of instrumental variables relies on instrumental assumptions.81 Because these two study designs require different sets of assumptions, we can estimate the extent of assumption violation and its impact on the risk estimates. Similarly, we can further incorporate the results from natural experiments82 and negative control analyses83 to gauge the effects of treatments and unmeasured confounders. By comparing the results from different approaches, we can determine the possible range of the true estimates with a greater level of confidence. Nonetheless, given the fact that different study designs and analytical methods possess distinct pros and cons, researchers need to be vigilant about the interpretation of their combined results.80 Effect triangulation approaches have successfully estimated causal effects in settings with strong confounding, such as the effects of lowering systolic blood pressure on the risk of coronary heart disease.80, 84

Leverage new statistical approaches for causal analyses

Philosophers have attempted to understand causality since the age of enlightenment,85 and epidemiologists proposed several criteria to evaluate the linkage between causes and effects.86 Recent developments of statistical methods allow causal inference from observational data while minimizing biases insurmountable by conventional approaches.70 As an illustration, in the presence of time‐varying confounders, traditional variable adjustment approaches will inevitably result in biases, due to the fact that subsequent measurements after the baseline are likely affected by the treatments. The g‐methods, a group of statistical approaches, can account for the time‐varying confounders and treatment‐confounder feedback that commonly reside in observational data.87, 88, 89 In addition, recently developed multiple robust statistical approaches can reduce the risk of model misspecification by relaxing the assumptions needed to achieve unbiased estimates.90, 91 For example, doubly robust methods can consistently estimate the effects of treatments if either the confounder‐treatment relation or the confounder/treatment‐outcome relation is correctly modeled,90 which would be helpful in settings where model misspecification raises significant concern. It is worth noting that the causal identification conditions (i.e., exchangeability, positivity, and consistency) still need to hold in order to get accurate effect estimates. In high‐dimensional settings, machine‐learning approaches can model the high‐level interactions among the variables and facilitate dimension reduction.92 These methods can accommodate the large number of variables extracted from EHRs and other RWDs.

Establish a robust infrastructure for randomized registry trials

Integration of EHRs into a large‐scale data registry would allow for real‐time matching against clinical trials and for physicians to be immediately notified if a patient was a potential fit.93 After patients were enrolled in the trial, the treatment could be randomized and follow‐up could occur at their normal point of care.94 This would have the potential to massively reduce enrollment and follow‐up costs while increasing the diversity of populations included in clinical trials. This cost reduction could enable randomized trials to answer a wider scope of questions. In particular, it may allow randomized registry trials to be performed postapproval for comparative effectiveness analysis and additional trials sponsored by government and nonprofit organizations.

Perform postapproval surveillance and validate that efficacy translates to effectiveness

Although clinical trials measure drug efficacy, it is important to perform postapproval surveillance to determine whether efficacy is generalizable to a broader population. Postapproval surveillance would allow drug pricing to take into account the real‐world value delivered. In addition, due to size restrictions, clinical trials may not capture rare adverse effects95 or drug–drug interactions96 that could be discovered through RWD analysis.

Use RWD to measure how closely trial populations resemble real‐world populations

This would enable trial organizers to plan representative trials and regulators to determine where additional trials may be necessary. In addition, if there is truly a heterogeneity of effects, it is unlikely to be detected in a homogenous trial population. It is, therefore, important to monitor both effectiveness and safety in the larger heterogeneous population that the therapy is given to.

Preregistration of RWD analyses

Establishing a preregistration requirement for RWD analyses is an effective approach to reduce the risk of multiple hypotheses testing and p‐hacking.97, 98 However, many observational datasets, such as Medicare,99 Medicaid,100 and Marketscan,101 are available and fully accessible to the researchers before the conception of specific RWD studies, which makes adequate preregistration a significant challenge. Combining a preregistration mechanism with a requirement to validate the identified effects on prospective patient cohorts could mitigate the risk of false discovery due to p‐hacking.97

Complement RCTs and pragmatic trials with RWD

Although RCTs and pragmatic trials generate high‐quality evidence for establishing causality24, 102 and inform real‐world practice,41, 103 respectively, it is infeasible to answer all clinically important research questions by setting up a series of trials. In addition to using multiple sources of RWE to arrive at better risk estimates (“Integrate Results From Multiple Designs of Observational Studies to Triangulate Effect Estimates” section), researchers could further leverage the hypotheses generated by RWD to inform trial design, or when the subgroup analyses of trials are underpowered, conduct RWD studies to further identify the participants who would likely benefit from the treatments under study.104

Establish reporting guidelines of RWE

Many reporting guidelines have been established for observational studies, but the specific requirements for using RWE for regulatory approval remains unclear. Widely accepted guidelines for academic publication include the REporting of studies Conducted using Observational Routinely collected health Data (RECORD)105 and STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) guidelines.106 However, researchers have called for more stringent and specific approaches for subfields of epidemiology, such as the RECORD‐PE statement for pharmacoepidemiological research.107 Similar efforts are needed to enhance the validity and reproducibility of RWE for regulatory purposes. In particular, regulatory agencies need to set specific requirements for the study participants, variables, measurement methods, data curation procedures, analytical approaches, and accessibility of the data and codes used in demonstrating the effectiveness of treatments using RWD.

Establish a structure for the postapproval evaluation of clinical practice guidelines and predictive algorithms

Historically, postapproval marketing of drugs involved a wide network of pharmaceutical sales representatives attempting to visit physicians to provide samples and detailing.51, 108 In the 21st century, there has been an increased interest in establishing standardized clinical practice guidelines, for example, Choosing Wisely,109 and in providing predictive drug recommendations as a part of personalized,110 and precision medicine,111 Standardizing care is a core component of improving the process with which care is delivered in the structure, process, and outcome framework for evaluating healthcare quality.112 Personalized drug recommendations offer the promise to provide patients with the drugs they are most likely to benefit from. This transition from individualized decision making presents a great potential to improve care and to deliver evidence‐based medicine. However, it also presents the potential for postapproval marketing to shift from one‐on‐one encounters to influence at the system level through guidelines and algorithms. It is critical for RWE‐based guidelines and recommendation systems to be thoroughly evaluated by independent third parties in the form of regulatory agencies and physician societies.

Determine value‐based reimbursement of drugs

Value‐based drug pricing has been discussed for over a decade.113, 114, 115 RWD analyses can reveal the actual effectiveness of the drugs in real‐world use cases, and, hence, inform the value created for the patients.116 Tracking the longitudinal health outcomes of patients receiving the treatments under the usual circumstances of healthcare practice is crucial for determining the true value of therapies. If it is to succeed, how value is attributed must be carefully regulated and monitored.

Conclusions

RWE presents a unique opportunity to accelerate the development of new therapies and to evaluate both the efficacy and the effectiveness of these treatments. However, researchers and regulators should take heed of the limitations of RWE and the potential biases lurking in RWD. Although there is significant value in utilizing RWE preregulatory approval (e.g., identifying subpopulations of need) and postregulatory approval (e.g., safety and surveillance), there are significant barriers to reliably using observational data as a key component of the regulatory process. Conflicting studies on attempts to replicate clinical trials using RWE show the potential risks and brittleness of RWE‐based comparative effectiveness.24, 25 This view is further supported by the discrepancies between trial results and those from RWD,117, 118, 119, 120 including a recent failed attempt at Facebook121 to replicate RCTs with large‐scale RWD analyses in a nonmedical context.

The appropriate use of RWE must be driven by forward‐thinking best practices, guidelines, and regulations to avoid spurious or biased findings. Increased data availability presents many opportunities but also brings with it the potential for biases in a system with outsized financial incentives. Traditional approaches, including preregistration, may not be sufficient when it is not possible to know which analyses have already been performed. It is critical to acknowledge the history and strengths of traditional RCTs especially in regard to initial approvals. RCTs should not be replaced for approval but can be supplemented to better understand treatment effectiveness in the real world. Prospective follow‐up studies driven by unconflicted parties will be critical to decision making around postapproval therapy surveillance and reimbursement. The appropriate use of RWE offers promise to accelerate the development of therapies while making their delivery safer, more targeted, and more efficient in real‐world settings.

Funding

This study was funded by the National Library of Medicine (NLM) T15LM007092 (B.K.B.‐J.), Harvard Data Science Fellowship (K.‐H.Y.), and National Institute of General Medical Sciences (NIGMS) T32GM007753 (S.G.F.).

Conflict of Interest

All other authors declared no competing interests for this work.

Disclosures

Dr. Prasad reports receiving royalties from his book Ending Medical Reversal; that his work is funded by the Laura and John Arnold Foundation; that he has received honoraria for grand rounds/lectures from several universities, medical centers, and professional societies, and payments for contributions to Medscape; and that he is not compensated for his work at the Veterans Affairs Medical Center in Portland, Oregon, or the Health Technology Assessment Subcommittee of the Oregon Health Authority. Dr. Beaulieu‐Jones reports owning equity in Progknowse Inc. outside of the submitted work. Progknowse is a company working with academic and community‐based health systems to integrate clinical data and enhance data science capabilities.