Abstract
Randomized clinical trials (RCTs) are no longer the sole source of data to inform guidelines, regulatory, and policy decisions. Real-world data (RWD), collected from registries, electronic health records, insurance claims, pharmacy records, social media, and sensor outputs from devices form real-world evidence (RWE), which can supplement evidence from RCTs. Benefits of using RWE include less time and cost to produce meaningful data; the ability to capture additional information, including social determinants of health that can impact health outcomes; detection of uncommon adverse events; and the potential to apply machine learning and artificial intelligence to the delivery of health care. Overall, combining data from RCTs and RWE would allow regulators to make ongoing and more evidence-based decisions in approving and monitoring products for diabetes.
Keywords: real-world evidence, randomized clinical trials, diabetes, regulatory approval
Clinical care, regulatory, and policy decisions are based on a continuum of clinical research from intensively monitored randomized clinical trials (RCTs) to real-world observational trial evidence, from tightly controlled, homogeneous populations to broader ones seen in usual clinical practice.1 In making decisions to approve new pharmaceutical and medical device products, regulatory bodies traditionally rely only on the evidence generated by RCTs.2 However, RCTs provide data from a select population and do not necessarily reflect a product’s performance in a broader population. The term “real-world evidence” (RWE) is used by those who develop medical products or who study, deliver, or pay for health care.3-5 Regulatory bodies have already used RWE to make postmarket decisions, such as labeling changes or product removals, but they have been reluctant in using RWE in premarket reviews. In this article, we suggest that RWE collected from broad, diverse populations can help supplement evidence derived from RCTs and therefore better inform regulatory decisions.6
Limitations of RCTS
The main differences between data from RCTs and RWE relate to their distinct aims. RCTs are used to assess efficacy (ie, how a drug or a device performs under well-defined and controlled conditions), notably with a priori defined inclusion and exclusion criteria for the research participants. It is noteworthy that the majority of clinical guidelines are based on evidence from RCTs. With RWE, however, data from the background population provide information on treatment efficacy in a population with fewer predefined restrictions. Some of the differences between how data from RCTs and RWE are utilized are summarized in Table 1.
Table 1.
Characteristic | RCTs | RWE |
---|---|---|
Standard of evidence | Gold standard | Complementary to RCTs |
Cost | Costly to develop and conduct | Less costly |
Patient population | Well-defined within constraints of specific inclusion criteria Results reflect outcomes in limited population |
Broader and promotes evaluation of patient populations less often studied in clinical trials Patient data derived from other sources, including insurance claims |
Sample size | Limited Requires sample size calculation to be performed in advance |
Orders of magnitude larger |
Efficacy | Randomized and blinding lead to minimized risk of data bias and confounding | Randomization and blinding may not be feasible Risk of unrecognized data bias and confounding greater |
Adverse events | Only more frequently occurring adverse events revealed | Can reveal adverse events with much lower frequency and those requiring longer exposure to occur |
Approval or clearance of new medical products | Considered the gold standard necessary for new drug approval, and when feasible for new device approval | Not generally accepted for approving new drugs but can complement RCT findings, accepted for new device indications |
Role in diabetes | Define efficacy and provide a preliminary safety profile in a well-defined and controlled population | Allows estimation of more realistic treatment effects of a wide range of diabetes interventions and evaluation of interactions such as social determinants of health and comorbidities |
Other issues | May be less useful when strong signals are available from RWE or early-phase trials | Facilitates postmarketing surveillance of adverse events and assessment of the product effectiveness Results may be less credible due when a control group is not included |
Source: Adapted from Gyawali et al.7
An additional consideration is that persistence with a therapy or device in the real world might be less than in a RCT (given the intense frequency of follow up of participants in most trials). Consequently, RWE may reveal that a therapeutic product may have less effectiveness than suggested by conventional RCTs.8
Traditional RCTs may also not be adequate to help understand clinically meaningful pharmacogenomic and pharmacokinetic differences between individuals, which may impact the effectiveness of new products. For example, there is evidence that one-fifth of therapies approved in recent years have differences in drug metabolism or response by race or ethnicity,9 yet in the United States, certain racial and ethnic groups that already bear a disproportionate burden of diabetes have been underrepresented in clinical trials of drugs and new technologies.10,11 Ultimately, if more comprehensive data are obtained for purposes of guiding drug or device selection, this will both benefit regulators and improve opportunities for precision medicine.
Real-World Evidence
In FDA’s 2018 strategic framework announcement,12 RWE is defined as the clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of real-world data (RWD). The guidance describes RWD as “data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources. Examples of RWD include data derived from electronic health records (EHRs), claims and billing data, data from product and disease registries, patient-generated data including in home-use settings, and data gathered from other sources that can inform on health status, such as mobile devices. RWD sources (eg, registries, collections of EHRs, and administrative and health care claims databases) can be used as data collection and analysis infrastructure to support many types of trial designs, including, but not limited to, randomized trials, such as large simple trials, pragmatic clinical trials, and observational studies (prospective and/or retrospective).”
RWE and RWD are often thought to refer to evidence produced in nonrandomized, observational trials, but FDA’s framework indicates that these terms can also describe randomized trials in the clinical trial setting. These trials are typically very large RCTs and are referred to as large simple trials (LST),13 pragmatic trials, or more recently “Efficacy-to-Effectiveness (E2E) Clinical Trials.”14-16 One of the first examples of the LST was the Salk Vaccine Trial, which led not only to regulatory approval of the vaccine but to a massive nationwide response to a public health crisis.17 LSTs have been used to support regulatory approval18 and are not uncommonly used to evaluate diabetes therapies.19,20
Although RWE, especially from non-RTC studies, has advantages over evidence from conventional RCTs, including lower costs, less time to obtain data, and fewer restrictions in population inclusion, some consider RWE to be less reliable.21 RWE studies are subject to design flaws, including unrecognized bias (due to unequal distribution of confounding factors), incomplete datasets, classification errors, and record linkage errors.22 A nonrandomized RWE study is more likely than a RCT to contain unrecognized bias, which can lead to an inappropriate approval or inappropriate new indication of a product if the RWE study is used without RCTs. Due to these potential flaws, a consensus on standard methods for RWE studies is needed to improve the quality and confidence of RWE studies.23,24
Overall, robust nonrandomized RWE studies, whether conducted in broad populations or in specific intended-target populations (ie, for postmarketing studies evaluating safety, effectiveness, or economic analyses), can complement the evidence generated by RCTs.25 RWE can provide insights into safety, effectiveness, and resource utilization for both a specific intervention and a comparison of multiple alternate interventions. This information can be important to regulators, public health planners, and payers to estimate the effect of a product or intervention on a large population.
The 21st Century Cures Act
The 21st Century Cures Act of 2016 (Cures Act), signed into law on December 13, 2016, is designed to accelerate the discovery, development, and delivery of new cures and treatments for disease. As part of the act, there was authorization to include $500 million over 9 years to help the FDA cover the cost of implementing the changes in the law.26 Among its goals, the act intends to “modernize” clinical trials, including the means by which safety and efficacy data are accumulated and analyzed.27 This would be achieved by the specification of new policies created by the FDA for regulating drugs and biologics.28
On December 6, 2018, FDA launched a new program, the Real-World Evidence Program, to promote the use of RWE as part of its regulatory decision making processes for drugs and biologics.12 In the document, an introductory scope of the program, which was established by the Cures Act, mentions specifically that this program is intended to support the approval of new indications for an already approved drug or biologic or to help support postapproval studies. The framework document contains: (1) definitions of both RWE and RWD, which is the information that is analyzed to become RWE; (2) current uses of RWD for evidence generation in safety and effectiveness regulatory studies and nonregulatory studies; (3) examples of trials using RWE for nonregulatory purposes to assess comparative effectiveness of treatment regimens; and (4) plans for data standards. The framework document explicitly excludes devices but refers to its corresponding device guidance.29
FDA Regulation of Drugs and Devices Using Real-World Evidence
Drugs
The use of RWE in general is gaining momentum at the Center for Drug Evaluation and Research (CDER). Director Janet Woodcock stated in her 2017 congressional testimony “that real-world evidence is applicable across all phases of drug development. Real-world evidence can help support new indications for existing drugs, because those drugs then would already be on the market, and real-world evidence can also help demonstrate how a drug works in populations that weren’t studied in the trial or relative to another drug not included in the study.”30
Multiple ongoing randomized and observational studies are producing RWE in studying the prevention and care of diabetes-related outcomes. An example is a retrospective, observational study by Sanofi (2018) that compared clinical outcomes in insulin-naïve adults with type 2 diabetes initiating insulin glargine or Insulin Degludec in a real-world clinical setting. Improvements in glycemic control, hypoglycemia outcomes, and discontinuation rates were similar to those of the first completed RCT of insulin glargine versus insulin degludec.31 This kind of RWE can therefore serve as a complement to evidence from RCTs for evaluating drug products under conditions that reflect the use of treatments in actual clinical practice and in more diverse patient populations.
As another example, the use of RWE from randomized pragmatic trials has been and will continue to be used to evaluate major adverse cardiac events (MACE) to address both safety and benefit of diabetes therapies. MACE can be objectively assessed even in nonblinded circumstances.32
While RWE trials cannot completely replace RCTs in assessing the safety and efficacy of new and approved drugs, RWE can certainly augment and expedite efforts in clinical research and approving drugs and devices for diabetes. It is not foreseeable that nonrandomized RWE will serve as well-controlled clinical trials required for drug approval purposes,33 but large, conventional RTCs that have some degree of pragmatism are already being used for important regulatory purposes.34,35
Device regulation has developed under a different set of rules. For new or high-risk devices, the FDA has to ensure that a device is safe and effective before approving it to enter the market. For moderate risk devices, such as glucose meters or insulin pumps, the FDA has to ensure that the device is as safe and effective as those already on the market.
Devices
The evidentiary standard set by regulation for devices is more flexible and allows the FDA to use the gold standard of RCTs, as well as partially controlled studies, studies and objective trials without matched controls, and reports of significant experience with a marketed device (ie, real-world evidence). Specifically, the FDA states that “valid scientific evidence is evidence from well-controlled investigations, partially controlled studies, studies and objective trials without matched controls, well-documented case histories conducted by qualified experts, and reports of significant human experience with a marketed device, from which it can fairly and responsibly be concluded by qualified experts that there is a reasonable assurance of the safety and effectiveness of a device under its conditions of use.”36
Given issues previously discussed with RWE and not well controlled trials, the FDA will typically default to requesting RTCs when feasible. On July 27, 2016, the FDA issued a draft guidance on the use of RWE to support regulatory decision making, which was finalized on August 31, 2017.29 This guidance clarified how manufacturers could use RWE to expand of the indications for use of devices that are already on the market. In fact, at about the time that the draft guidance was issued, the FDA also held an advisory panel meeting to determine whether there was enough evidence to change the intended use of the Dexcom G5 continuous glucose sensors from an adjunctive device that complemented fingerstick measurement of blood glucose levels to a device that replaced fingersticks.37 At that time, a majority of the panel recommended that there was reasonable assurance of safety and effectiveness of the device as a replacement for fingersticks based on clinical point and trend accuracy data and reports of significant human experience. As result, the new intended use was approved on December 20, 2016.38
Although this is a good example of the FDA’s willingness to use RWE, it also highlights some of the issues associated with the use of RWE. The first continuous glucose monitoring system (CGM) as adjunctive devices were first approved by the FDA in 2000. It was reported to the FDA advisory panel that most people with diabetes using CGM between 2000 and 2016 had at some point used the sensors as replacements for fingerstick measurements, but because that use was off-label and not recorded, there were no formal data that experts or the FDA could use to analyze. Therefore, the FDA panel and reviewers were left only with anecdotal accounts.
FDA Software Precertification Program
FDA has also announced plans to create a Software Precertification Program. This program is envisioned as a voluntary pathway to assess the safety and effectiveness of software technologies. The program’s goal is to provide more streamlined and efficient regulatory oversight of software-based medical devices from manufacturers who have demonstrated a robust culture of quality and organizational excellence and are committed to monitoring real-world performance.39 Details of how real-world evidence will be selected, collected, and analyzed are pending.40 This program will represent another instance in which FDA is using RWE to support regulatory decisions.
Conclusion
Growing appreciation of the value of RWE to complement evidence from RCTs presents opportunities for facilitating the regulatory approval processes for new drug and device therapies and diagnostics. In the United States, the value of RWE is recognized increasingly by the FDA. In addition, it is likely that real-world evidence can also be of value to public health planners and payers to estimate the effect of a product or intervention on a large population. Key challenges for advancing the use of RWE include improving data collection and data quality, and improving the means of analyzing relevant data to mitigate possible biases. These efforts will help to experts and regulatory bodies to rely on RWE to enhance timely decision making for the benefit of all stakeholders involved in diabetes care.
Acknowledgments
The authors would like to thank Annamarie Sucher for her expert editorial assistance.
Footnotes
Abbreviations: CDER, Center for Drug Evaluation and Research; CGM, continuous glucose monitoring; E2E, Efficacy-to-Effectiveness; EHRs, electronic health records; LST, large simple trials; MACE, major adverse cardiac events; RCTs, randomized clinical trials; RWD, real-world data; RWE, real-world evidence.
Declaration of Conflicting Interests: The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: DCK is a consultant for Ascensia, EOFlow, Lifecare, Merck, Novo, Roche Diagnostics, and Voluntis. AG is Partner at NDA Partners and advices multiple device companies in regulatory issues. AF is Executive Chairman of Kinexum, which advises multiple health-product companies in the fields of metabolism, cardiovascular, oncology, and dermatology. DK is a Medical Advisor for Glooko and Vicentra and a Consultant for NovoNordisk and Sanofi.
Funding: The author(s) received no financial support for the research, authorship, and/or publication of this article.
References
- 1. Roche N, Reddel H, Martin R, et al. Quality standards for real-world research. Focus on observational database studies of comparative effectiveness. Ann Am Thorac Soc. 2014;11 (suppl 2):S99-S104. [DOI] [PubMed] [Google Scholar]
- 2. Krause JH, Saver RS. Real-world evidence in the real world: beyond the FDA. Am J Law Med. 2018;44:161-179. [DOI] [PubMed] [Google Scholar]
- 3. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence—what is it and what can it tell us? N Engl J Med. 2016;375:2293-2297. [DOI] [PubMed] [Google Scholar]
- 4. Network for Excellence in Health Innovation. Real world evidence: a new era for health care innovation. 2015. Available at: http://www.nehi.net/writable/publication_files/file/rwe_issue_brief_final.pdf. Accessed February 21, 2019.
- 5. US Food and Drug Administration. Use of real-world evidence to support regulatory decision-making for medical devices: draft guidance for industry and Food and Drug Administration staff. 2016. Available at: https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm513027.pdf. Accessed February 21, 2019.
- 6. Faraoni D, Schaefer ST. Randomized controlled trials vs. observational studies: why not just live together? BMC Anesthesiol. 2016;16:102. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Gyawali B, Parsad S, Feinberg BA, Nabhan C. Real-world evidence and randomized studies in the precision oncology era: the right balance. JCO Precis Oncol 2017;1:1-5. [DOI] [PubMed] [Google Scholar]
- 8. Pantalone KM, Misra-Hebert AD, Hobbs TM, et al. Clinical inertia in type 2 diabetes management: evidence from a large, real-world data set. Diabetes Care. 2018;41:e113-e114. [DOI] [PubMed] [Google Scholar]
- 9. Bonham VL, Callier SL, Royal CD. Will precision medicine move us beyond Race? N Engl J Med. 2016;374:2003-2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Hoppe C, Kerr D. Minority underrepresentation in cardiovascular outcome trials for type 2 diabetes. Lancet Diabetes Endocrinol. 2017;5:13. [DOI] [PubMed] [Google Scholar]
- 11. Huyett L, Dassau E, Pinsker JE, Doyle FJ, III, Kerr D. Minority groups and the artificial pancreas: who is (not) in line? Lancet Diabetes Endocrinol. 2016;4:880-881. [DOI] [PubMed] [Google Scholar]
- 12. US Food and Drug Administration. Framework for FDA’s real world evidence program. 2018. Available at: https://www.fda.gov/downloads/ScienceResearch/SpecialTopics/RealWorldEvidence/UCM627769.pdf. Accessed February 21, 2019.
- 13. Yusuf S, Collins R, Peto R. Why do we need some large, simple randomized trials? Stat Med. 1984;3:409-420. [DOI] [PubMed] [Google Scholar]
- 14. Selker HP, Oye KA, Eichler HG, et al. A proposal for integrated efficacy-to-effectiveness (E2E) clinical trials. Clin Pharmacol Ther. 2014;95:147-153. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Selker HP, Gorman S, Kaitin KI. Efficacy-to-effectiveness clinical trials. Trans Am Clin Climatol Assoc. 2018;129:279-300. [PMC free article] [PubMed] [Google Scholar]
- 16. Selker HP, Eichler HG, Stockbridge NL, et al. Efficacy and effectiveness too trials: clinical trial designs to generate evidence on efficacy and on effectiveness in wide practice. Clin Pharmacol Ther. 2019;105:857-866. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Meldrum M. “A calculated risk”: the Salk polio vaccine field trials of 1954. BMJ. 1998;317:1233-1236. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Group I-SISoISC. Randomised trial of intravenous streptokinase, oral aspirin, both, or neither among 17,187 cases of suspected acute myocardial infarction: ISIS-2. ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Lancet. 1988;2:349-360. [PubMed] [Google Scholar]
- 19. ClinicalTrails.gov. Exubera large simple trial to evaluate long-term pulmonary and cardiovascular safety (VOLUME). 2010. Available at: https://clinicaltrials.gov/ct2/show/results/NCT00359801?view=results. Accessed February 21, 2019.
- 20. Wiviott SD, Raz I, Bonaca MP, et al. Dapagliflozin and cardiovascular outcomes in type 2 diabetes. N Engl J Med. 2019;380:347-357. [DOI] [PubMed] [Google Scholar]
- 21. Lieberman T. With media watchdogs on the sidelines, pharma-funded advocacy groups pushed Cures Act to the finish line. 2016. Available at: https://www.healthnewsreview.org/2016/12/with-media-watchdogs-sidelined-pharma-funded-advocacy-groups-pushed-cures-act-to-the-finish-line/. Accessed February 25, 2019.
- 22. Schwartz JL. Real-world evidence, public participation, and the FDA. Hastings Cent Rep. 2017;47:7-8. [DOI] [PubMed] [Google Scholar]
- 23. Berger ML, Sox H, Willke RJ, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE special task force on real-world evidence in health care decision making. Value Health. 2017;20:1003-1008. [DOI] [PubMed] [Google Scholar]
- 24. Corrao G. Building reliable evidence from realworld data: methods, cautiousness and recommendations. Epidemiol Biostat Public Health. 2013;10:e8981-1-e-40. [Google Scholar]
- 25. Maissenhaelter BE, Woolmore AL, Schlag PM. Real-world evidence research based on big data: motivation-challenges-success factors. Onkologe (Berl). 2018;24(suppl 2):91-98. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. US Food and Drug Administration. 21st Century Cures Act. 2018. Available at: https://www.fda.gov/regulatoryinformation/lawsenforcedbyfda/significantamendmentstothefdcact/21stcenturycuresact/default.htm. Accessed February 21, 2019.
- 27. US House of Representatives Committee on Energy and Commerce. The 21st Century Cures Act: an innovation game-changer, a once-in-a-generation, transformational opportunity to change the way we treat disease. 2016. Available at: https://archives-energycommerce.house.gov/sites/republicans.energycommerce.house.gov/files/documents/114/analysis/20161128%20Cures%20Fact%20Sheet.pdf. Accessed February 21, 2019.
- 28. Office USGP. Public Law 114-255—December 13, 2016. 2016. Available at: https://www.congress.gov/114/plaws/publ255/PLAW-114publ255.pdf. Accessed February 21, 2019.
- 29. US Department of Health and Human Services, Food and Drug Administration. Use of real-world evidence to support regulatory decision-making for medical devices: Guidance for Industry and Food and Drug Administration Staff. 2017. Available at: https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm513027.pdf. Accessed February 22, 2019.
- 30. Woodcock J. Real world evidence transcript. US Food and Drug Administration. 2017. Available at: https://www.fda.gov/Drugs/ScienceResearch/ucm583448.htm. Accessed February 22, 2019.
- 31. Sullivan SD, Bailey TS, Rousel R, et al. Clinical outcomes in real-world patients with type 2 diabetes switching from first- to second-generation basal insulin analogues: Comparative effectiveness of insulin glargine 300 units/mL and insulin degludec in the DELIVER D+ cohort study. Diabetes Obes Metab 2018;20:2148-2158. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Dreyer NA. Advancing a framework for regulatory use of real-world evidence: when real is reliable. Ther Innov Regul Sci. 2018;52:362-368. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. US Department of Health and Human Services, Food and Drug Administration. Guidance for industry: providing clinical evidence of effectiveness for human drug and biological products. 1998. Available at: https://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm072008.pdf. Accessed February 22, 2019.
- 34. Zinman B, Wanner C, Lachin JM, et al. Empagliflozin, cardiovascular outcomes, and mortality in type 2 diabetes. N Engl J Med. 2015;373:2117-2128. [DOI] [PubMed] [Google Scholar]
- 35. US Food and Drug Administration. FDA approves Jardiance to reduce cardiovascular death in adults with type 2 diabetes. 2016. Available at: https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm531517.htm. Accessed February 22, 2019.
- 36. Regulation 21 Code of Federal Regulations 860.7(c)(2), April 1, 2018. [Google Scholar]
- 37. US Food and Drug Administration. FDA executive summary—Dexcom G5 Mobile Continuous Glucose Monitoring System. 2016. Available at: https://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMaterials/MedicalDevices/MedicalDevicesAdvisoryCommittee/ClinicalChemistryandClinicalToxicologyDevicesPanel/UCM511810.pdf. Accessed February 22, 2019.
- 38. US Food and Drug Administration. PMA P120005/S041: FDA summary of safety and effectiveness data. 2016. Available at: https://www.accessdata.fda.gov/cdrh_docs/pdf12/P120005S041B.pdf. Accessed February 22, 2019.
- 39. US Food and Drug Administration. Digital health software precertification (Pre-Cert) program. 2019. Available at: https://www.fda.gov/medicaldevices/digitalhealth/digitalhealthprecertprogram/default.htm. Accessed February 22, 2019.
- 40. King F, Klonoff DC, Ahn D, et al. Diabetes technology society report on the FDA digital health software precertification program meeting. J Diabetes Sci Technol. 2019;13:128-139. [DOI] [PMC free article] [PubMed] [Google Scholar]