The US Food and Drug Administration Sentinel System: a national resource for a learning health system

Abstract

The US Food and Drug Administration (FDA) created the Sentinel System in response to a requirement in the FDA Amendments Act of 2007 that the agency establish a system for monitoring risks associated with drug and biologic products using data from disparate sources. The Sentinel System has completed hundreds of analyses, including many that have directly informed regulatory decisions. The Sentinel System also was designed to support a national infrastructure for a learning health system. Sentinel governance and guiding principles were designed to facilitate Sentinel’s role as a national resource. The Sentinel System infrastructure now supports multiple non-FDA projects for stakeholders ranging from regulated industry to other federal agencies, international regulators, and academics. The Sentinel System is a working example of a learning health system that is expanding with the potential to create a global learning health system that can support medical product safety assessments and other research.

FDA SENTINEL SYSTEM AND A COMMON INFORMATICS INFRASTRUCTURE

The US Food and Drug Administration (FDA) created the Sentinel System in response to a requirement in the FDA Amendments Act of 2007 (FDAAA 2007) that FDA “link and analyze safety data from multiple sources” to monitor risks associated with drug and biologic products.¹ The system’s initial instantiation, the Mini-Sentinel pilot (2009–2015), developed and tested a distributed data network and its associated analytic infrastructure as well as the governance needed to support FDA’s mission.^2,3 Since 2016, the FDA has used the Sentinel System to meet its regulatory requirements under FDAAA, assess the characteristics of real-world data (RWD) sources, better use RWD in distributed data environments,^4,5 and better understand how to generate real-world evidence (RWE) to support regulatory decision-making.^6–8

The Sentinel System uses a secure distributed data network, common data model (Sentinel Common Data Model),⁹ curated data, and distributed analytic tools to conduct analyses in more than a dozen data partners. Data partners maintain physical and operational control of their data, which are collected as part of routine clinical practice or operations. They regularly transform their source data into the version-controlled Sentinel Common Data Model and have to pass a robust data quality assurance process before the transformed data can be used for any queries. Health insurance claims are the primary data source, augmented by electronic health record (EHR) data for about 5% of available health plan members.¹⁰ Sentinel also has access to several EHR-only data sources.^11,12 Work is directed by FDA and coordinated by the Sentinel Operations Center.¹³ The Sentinel Operations Center has created a suite of publicly available analytic tools compatible with the Sentinel Common Data Model to standardize and expedite the execution of queries within the distributed data network. Figure 1 shows the workflow of a typical Sentinel query. The Sentinel Common Data Model and analytic tools enable rapid query creation, execution, and reporting ranging from simple descriptive queries to complex comparative analyses. Over the years, the System’s capabilites have been expanded to include new data domains (eg, laboratory results, prescriptions, inpatient medication administrations, mother-infant linkage, patient-reported measures),⁹ updated data characterization and curation,¹⁴ enhanced analytic tools (including tools for assessing medication use in pregnancy and maternal and infant outcomes),^15–23 and new data sources.

Figure 1.

Architecture of the US Food and Drug Administration Sentinel System. FDA: Food and Drug Administration; FISMA: Federal Information Security Management Act; SOC: Sentinel Operations Center.

Sentinel governing principles were jointly developed by the Sentinel Operations Center, FDA, and scientific and data partners to facilitate Sentinel’s role as a national resource. These principles include:

• Contracting language that allows data partners to use their curated Sentinel Common Data Model-formatted database for any purpose while maintaining robust conflict of interest policies to ensure independence of evaluations^24,25;

• A data modeling approach focused on the most granular data elements possible to maximize analytic flexibility and extensibility¹³;

• Public posting of Sentinel reports, tools, data curation and other processes, and lessons learned^9,26–28;

• Public training and resources to help others use the Sentinel Common Data Model and tools^29,30; and

• Support for improving the use of RWD, including approaches for extracting and using EHR data³¹ and collecting data directly from patients.⁷

USES OF SENTINEL’S COMMON INFORMATICS INFRASTRUCTURE TO SUPPORT A LEARNING HEALTH SYSTEM

The use of RWD collected by a diverse group of health plans and delivery systems for evidence generation, along with the guiding principles described above, allows the Sentinel System to support FDA’s obligations under FDAAA and regulatory decision-making, and also more broadly serve as a national infrastructure for a learning health system, in which evidence is generated and applied as part of real-world clinical care (Figure 2).^3,32,33 The Sentinel System has completed hundreds of analyses and published approximately 200 papers in the peer-reviewed literature,³⁴ with results used to inform regulatory decisions³⁵ and FDA Advisory Committee meetings.³⁶ Going beyond medical product safety, FDA has used the Sentinel infrastructure to gather information about the performance of its regulated medical products, including interactions and interventions with patients. The FDA built a mobile application, the MyStudies app, to obtain patient reported data not available in the Sentinel Common Data Model. The FDA MyStudies app lets patients provide data that can be linked to traditional clinical trials, real-world pragmatic trials, observational studies, and registries and obtain electronic consent.^7,37 FDA also sponsored the IMPACT Afib (IMplementation of a randomized controlled trial to imProve treatment with oral AntiCoagulanTs in patients with Atrial Fibrillation) trial, a proof-of-concept randomized trial conducted using the Sentinel infrastructure to inform future interventional studies that are designed to utilize existing healthcare data.^6,38 These projects both inform FDA’s understanding of RWE generation and provide examples and tools for the larger research community. FDA also used Sentinel to describe the natural history and epidemiology of COVID-19.¹¹

Figure 2.

Uses of Sentinel’s common informatics infrastructure to support a learning health system.

Further, approximately 40 publicly reported large-scale research projects are known to have used or are using the Sentinel informatics infrastructure; many other projects and systems use Sentinel informatics infrastructure for internal decision-making. These “leveraged projects” use Sentinel curated data, Sentinel Common Data Model, analytic tools, and secure distributed querying architecture on behalf of investigators at, or funded by, the US National Institutes of Health, the US Centers for Disease Control and Prevention (CDC), the Patient-Centered Outcomes Research Institute, the Reagan-Udall Foundation’s Innovation in Medical Evidence Development and Surveillance, the Academy of Managed Care Pharmacy Biologics and Biosimilars Collective Intelligence Consortium, and individual pharmaceutical companies.^39,40 Details of these projects are described in Table 1. Like Sentinel, participation by data partners in these leveraged projects is completely voluntary. Many of these projects use Sentinel for activities similar to FDA’s use, namely medical product safety surveillance, including studies of medication safety, vaccine safety, and medication and vaccine use during pregnancy. In the coming years, many of the ongoing studies will be published and several will be submitted to FDA and European Medicines Agency as part of regulatory commitments. Importantly, several projects use the infrastructure and curated data for studies unrelated to medical product safety surveillance, such as patterns of cancer screening, diffusion of medical products, and natural history and epidemiology of disease.⁴¹

Table 1.

Leveraged projects that have used the US FDA Sentinel resources, by funding source

Treatment patterns among patients with epilepsy

Funding source		Leveraged project	Reference
Organization	Brief information
Biologics and Biosimilars Collective Intelligence Consortium (BBCIC)	A nonprofit initiative established by the Academy of Managed Care Pharmacy to provide post-marketing evidence on the safety and effectiveness of biologics and biosimilars Participants include Sentinel data partners, product manufacturers, managed care organizations, and other nonprofit organizations; the collaboration is funded primarily by fees contributed by industry partners Collaborates with 5 Sentinel data partners who have information on over 95 million patient-lives available73 Research projects are selected and overseen by the BBCIC Planning Board and Science Committee Has completed multiple research studies since 2016 on the use of biologics and biosimilars, including a series of descriptive analyses of biosimilars and related products and studies intended to inform the design of future comparative effectiveness studies	Completed
		Assessment of the utilization and patient characteristics related to the use of biologics and biosimilars
		Characterization of the use of originator and follow-on insulin products and users of these products	74 , 75
		Utilization patterns and patient characteristics for anti-inflammatory originator biologics and their biosimilars	76
		Utilization patterns and patient characteristics for short-acting granulocyte-colony stimulating factor (G-CSF) products	77
		Utilization patterns and patient characteristics for the originator and follow-on insulin glargine drugs	78
		Utilization patterns and patient characteristics for trastuzumab originator and biosimilar products	79
		Utilization patterns and patient characteristics for erythropoietin stimulating agents in hemodialysis patients	White paper in progress to be published on the BBCIC website (www.bbcic.org)
		Examination of clinical outcomes associated with biologics use
		Medical-attended severe hypoglycemia, modified major adverse cardiac events, and hemoglobin A1c level related to long- and intermediate-acting insulin	80
		Rates of serious infections among autoimmune disease patients using anti-inflammatory biologics	81
		G-CSF use in patients with breast or lung cancer receiving and risk of chemotherapy-induced febrile neutropenia	82
		Methodologic studies/thought leadership
		Approaches for comparative analyses for product switching	83
		Use of National Drug Codes of biologics and biosimilars in physician-office claims	84
		International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) to ICD-10-CM codes mapping for conditions of interest	85
		Barriers and facilitators to conduct high-quality, large-scale safety and comparative effectiveness research: the Biologics and Biosimilars Collective Intelligence Consortium experience	86
		Ongoing
		Inferential study to examine the comparative effectiveness and safety study of G-CSFs	No publications yet
		Utilization patterns and patient characteristics for bevacizumab originator and biosimilar products	No publications yet
Innovation in Medical Evidence Development and Surveillance (IMEDS)	A public-private partnership of the Reagan-Udall Foundation for the Food and Drug Administration (FDA) that provides an avenue for regulated industry and others to leverage the Sentinel distributed data and analytic tools Collaborates with 9 Sentinel data partners that have approximately 111 million person-lives of data available for research Sponsors fund individual research projects directly with IMEDS, who then subcontracts with research partners and others to implement the work Nine projects funded by 5 industry sponsors have been initiated	Completed
		Evaluation of venous thromboembolism (VTE) rates in new users of second and fourth generation oral contraceptives	87
		Assessment of the impact of the FDA’s 2010 proton pump inhibitor (PPI) class label change on the PPI dispensing patterns, incident fractures, and osteoporosis screening or interventions	88
		Evaluation of the risks of VTE among rheumatoid arthritis patients treated with biologic and nonbiologic disease-modifying antirheumatic drugs (DMARDs)	89
		Assessment of pain medication treatment patterns and incidence of total join replacement in persons with osteoarthritis	90
		Ongoing
		Multiple studies for FDA and European Medicines Agency (EMA) commitments, including a study examining the risk of angioedema with a drug for heart failure, a study of diabetic ketoacidosis risk associated with a diabetes medication, and an investigation of pregnancy exposures and outcomes among women treated with a biologic product for psoriasis	91 , 92
National Institutes of Health (NIH) Collaboratory Distributed Research Network (DRN)	Includes 14 Sentinel partners that enable investigators funded by NIH and other not-for-profit sponsors to use the Sentinel resources, in addition to the partners’ ability to engage both their providers and members and to link to external registries to support pragmatic clinical trials embedded in clinical care settings, as well as observational studies39 Has conducted 12 studies across topics as diverse as cancer screening, dementia, opioid use, prescribing cascades, and stain use in older individuals	Completed
		Cancer screening rates and follow-up screenings for breast, colorectal, and cervical cancers. Used Sentinel analytic tools and data to assess screening rates among health plan members, the rate of abnormal findings, and the rate and timing of follow-up screenings after abnormal findings	41
		Assessment of the prevalence of Alzheimer’s disease and dementia in Medicare Advantage populations	93
		A multisite study of chemotherapy-induced peripheral neuropathy among over 400 000 patients receiving neurotoxic and nonneurotoxic chemotherapies; results were used to support an NIH grant application	94
		Statin use in older patients with and without cardiovascular disease and type 2 diabetes mellitus; feasibility assessment for a pragmatic clinical trial	95
		Antibiotic dispensing following pediatric ambulatory and emergency department encounters	46
		Considerations for using distributed research networks to conduct aspects of randomized trials	96
		Antidopaminergic-Antiparkinsonian medication prescribing cascade in persons with Alzheimer’s disease	97
		Practical challenges in the conduct of pragmatic trials embedded in health plans: lessons of IMPACT-AFib, an FDA-Catalyst trial	98
		Prescribing cascades in persons with Alzheimer’s disease: engaging patients, caregivers, and providers in a qualitative evaluation of print educational materials	99
Regulated industry and others	Include medical product manufacturers Have directly funded multiple studies using the Sentinel System’s curated data, Sentinel Common Data Model, and analytic tools	Completed
		Feasibility assessment for an observational study on the effect of long-acting beta agonists with inhaled corticosteroid therapy on asthma mortality	100
		Treatment patterns among patients with epilepsy	No publications yet
		Vaccine use during pregnancy	101
		Comparative effectiveness of novel asthma therapies	102
		Replication of an FDA Sentinel System report103 on the risks of acute myocardial infarction and stroke among mirabegron users	60
		Comparison of the Observational Medical Outcomes Partnership (OMOP) and Mini-Sentinel Common Data Models, focusing on data model applicability to signal detection and early signal refinement	104
		Ongoing
		Safety and effectiveness of an adult vaccine (multiple studies in different patient populations); 2 of these studies are supporting FDA regulatory commitments, and one is supporting an EMA requirement	105
		Evaluation of the safety and effectiveness of Covid-19 vaccines in the real-world environment; in collaboration with multiple industry sponsors	No publications yet
		Descriptive analysis of exposure to monoclonal antibodies (asthma, systemic lupus erythematous) during pregnancy to support ongoing registries for these products	No publications yet

SELECT EXAMPLES OF HOW SENTINEL CONTRIBUTES TO A LEARNING HEALTH SYSTEM

Use of the Sentinel System by FDA has led to multiple labeling changes or decisions that no labeling changes were needed. These regulatory decisions, which were informed by data generated from clinical care, help guide clinical practice. For example, a Sentinel analysis found an elevated risk of nonmelanoma skin cancer associated with long-term use of hydrochlorothiazide, a commonly used antihypertensive medication.⁴² The results were consistent with findings from studies conducted in Europe. FDA used the Sentinel results to make a label change to all hydrochlorothiazide-containing products to include information about the risk of nonmelanoma skin cancer.⁴³ FDA has also used the Sentinel System to examine systemic corticosteroid use for COVID-19 in outpatient settings.⁴⁴ The study found increasing use of systemic corticosteroids to treat outpatient COVID-19 cases. Findings from the analysis were used in a health advisory by the CDC to advise against using systemic corticosteroid in patients with mild to moderate COVID-19.⁴⁵

Beyond FDA, the Sentinel informatics infrastructure has been used by the National Institutes of Health to inform clinical trial designs to evaluate patterns in cancer screening and care, and to assess antibiotic prescribing patterns. A study found that potentially inappropriate antibiotic prescribing patterns decreased unevenly and that antibiotic stewardship initiatives in the outpatient setting were warranted.⁴⁶ The Sentinel informatics infrastructure is also being used to evaluate the safety and effectiveness of COVID-19 vaccines in collaboration with multiple industry sponsors. These studies are examples of how Sentinel is being used as a national resource to support a learning health system.

SENTINEL HELPING EXPAND COLLABORATIONS IN THE UNITED STATES AND ACROSS THE GLOBE

Sentinel was built on the experiences of other research networks⁴⁷ (eg, the CDC-funded Vaccine Safety Datalink,⁴⁸ the Health Care Systems Research Network,⁴⁹ the National Institutes of Health-funded Cancer Research Network⁵⁰) and now other networks have adapted the Sentinel Common Data Model and tools. The Patient-Centered Clinical Research Network (PCORnet)⁵¹ common data model and analytic toolkit were based on the Sentinel Common Data Model and tools, and PCORnet adopted the same secure distributed networking software (PopMedNet) as Sentinel.²⁸ This alignment has facilitated collaboration between PCORnet and FDA, with several PCORnet partners joining Sentinel as collaborating institutions and several Sentinel partners joining PCORnet. FDA collaboration with PCORnet includes COVID-19 projects,^12,52 cofunding of the PCORnet RELIANCE (RofLumilast or Azithromycin to preveNt COPD Exacerbations) trial that will link trial data with the Medicare fee-for-service data to provide additional information on the primary and select secondary outcomes; the project will also test distributed regression methods with vertically partitioned data.⁵³

FDA is leveraging the Sentinel infrastructure and lessons learned to help establish international collaborations with other regulatory authorities and data networks. Recent examples include an investigation of the impact of the recall of angiotensin receptor blockers due to nitrosamine contamination on the use of these medications⁵⁴ and COVID-19 related activities.⁵⁵ Health Canada and researchers at the Canadian Network for Observational Drug Effect Studies (CNODES) have adopted the Sentinel Common Data Model and analytic approaches,^56,57 enabling the exact same analysis to be run in CNODES as among Sentinel data partners.⁵⁸ Through a partnership with the University of Southern Denmark and the Danish Medicines Agency, cohorts extracted from the Danish National Healthcare Databases have been formatted in the Sentinel Common Data Model for joint analyses. The Taiwan Ministry of Science and Technology has funded transformation of the National Health Insurance Research Database, which includes longitudinal claims data with information on over 23 million individuals, into the Sentinel Common Data Model.⁵⁹ The use of the Sentinel Common Data Model by numerous countries is efficient and convenient for conducting studies but more importantly, it ensures that studies are conducted in the exact same way, resulting in true study replication in different data sources and patient populations. In response to frequent requests for information from regulatory authorities around the world, FDA and the Sentinel Operations Center have provided technical advice on how to operate a secure distributed data network, Sentinel Common Data Model, data quality review, and analytic tools for RWE generation to regulators and researchers around the world, including Singapore, Canada, South Korea, Japan, Taiwan, European Union, and China, and to promote worldwide regulatory collaborations.

Other uses of the Sentinel System infrastructure include analyses conducted by pharmaceutical companies and contract research organizations⁶⁰ that have formatted their internal data resources into the Sentinel Common Data Model to enable use of the Sentinel tools and to help the companies better understand how FDA uses the Sentinel System. These companies often have data in multiple data models (eg, Observational Medical Outcomes Partnership [OMOP], Sentinel Common Data Model) to make use of the analytic tools best suited to their needs. In addition, health data and analytics companies are using Sentinel analytic tools such as its implementation of TreeScan™⁶¹—a data mining method to identify unexpected potential adverse events follow medical product use—to support commercial offerings.

AREAS FOR FURTHER INVESTMENT

FDA continues to invest in promoting the expanded use of the Sentinel infrastructure to a broader set of stakeholders. Over the past few years, FDA has conducted multiple public trainings targeting industry, academic, and regulatory stakeholders.²⁹ The trainings regularly reach capacity and are attended by a range of stakeholders. The publicly available Sentinel reports are also becoming a source to guide research. A recent review article described Sentinel analyses focused on nephrology and outlined pathways for researchers to leverage Sentinel infrastructure for generating RWE in renal care.⁶² Another study used Sentinel reports describing the results of mapping International Classification of Diseases, ninth revision, Clinical Modification (ICD-9-CM) to ICD-10-CM codes, including variation in incidence and prevalence.⁶³

The recently created Sentinel Innovation Center⁶⁴ is working to improve Sentinel’s capabilities through the development of new approaches for extracting structured and unstructured EHR data for incorporation into Sentinel Common Data Model and use in distributed querying, new ways to identify health outcomes of interest in RWD,⁶⁵ and investigation of new causal inference methods.⁶⁶ In addition, the Sentinel Community Building and Outreach Center (CBOC) is focused on expanding the community of users through projects designed to increase awareness of the Sentinel Initiative to a more diverse scientific community, improve usability of the Sentinel tools and infrastructure, and enable stakeholders to more effectively contribute to advancing the scientific foundation of the Sentinel System.⁶⁷

SUMMARY

The Sentinel System has grown from a pilot program into an important part of FDA’s postmarketing drug safety system. Consistent with FDA’s initial vision, the Sentinel System infrastructure is now supporting multiple non-FDA projects for stakeholders ranging from regulated industry, to other federal agencies, international regulators, and academics.⁴⁰ Demand for information on how to make use of Sentinel System data resources and tools is growing, and resources to support this growth are expanding. The Sentinel System is a working example of a learning health system that is expanding with the potential to create a global learning health system that can support medical product safety assessments and other research.

Experiences from the Sentinel System’s creation, growth, and use beyond FDA can help inform the creation and expansion of other learning health systems across the globe. Establishing a strong set of governing principles for the collaboration was critical in fostering Sentinel’s initial success by helping all partners focus on meeting FDA’s needs while keeping an eye towards future uses within and beyond FDA. The core principles that fostered collaboration within Sentinel and the expansion of capabilities for FDA while providing governance and technical resources to the community enabled FDA to meet the dual goals of having a robust postmarketing active safety surveillance system and creating a national resource to support evidence generation well-beyond FDA. The Sentinel governance and culture encouraged the data partners to use their Sentinel Common Data Model-formatted data and the Sentinel analytic tools for other purposes and encouraged the Sentinel Operations Center to support leveraged projects; those other uses of Sentinel infrastructure were celebrated as successes and not viewed as competing interests or a distraction. Beyond the core Sentinel governing principles, consistent engagement with stakeholders through public meetings, seminars, trainings, and scientific dissemination were critical to the acceptance of the Sentinel informatics infrastructure as a viable option as a national data resource. It also was critical that the public communications and engagements were scientifically rigorous and transparent. Although this paper focuses on Sentinel, we note many other large-scale collaborations using RWD to generate RWE around the world.^68–72 Our hope is that these efforts, taken together and through a spirit of collaboration, can help expand the depth and breadth of research and public health surveillance capabilities across the world.

FUNDING

This work was supported in part by the U.S. Food and Drug Administration (FDA) through the Department of Health and Human Services (HHS) contract number 75F40119F19001.

AUTHOR CONTRIBUTIONS

JSB conceived of the idea for this work. JSB, ABM, YHN, and ST prepared the first draft of the manuscript. All authors revised the manuscript for important intellectual content and approved the final manuscript to be submitted for publication. All authors agree to be accountable for all aspects of the work.

ETHICS APPROVAL

The authors state that no ethical approval was needed.

Data Availability

No new data were generated or analyzed in support of this article.