Using and improving distributed data networks to generate actionable evidence: the case of real-world outcomes in the Food and Drug Administration’s Sentinel system

Abstract

The US Food and Drug Administration (FDA) Sentinel System uses a distributed data network, a common data model, curated real-world data, and distributed analytic tools to generate evidence for FDA decision-making. Sentinel system needs include analytic flexibility, transparency, and reproducibility while protecting patient privacy. Based on over a decade of experience, a critical system limitation is the inability to identify enough medical conditions of interest in observational data to a satisfactory level of accuracy. Improving the system’s ability to use computable phenotypes will require an “all of the above” approach that improves use of electronic health data while incorporating the growing array of complementary electronic health record data sources. FDA recently funded a Sentinel System Innovation Center and a Community Building and Outreach Center that will provide a platform for collaboration across disciplines to promote better use of real-world data for decision-making.

INTRODUCTION: FDA SENTINEL SYSTEM

The US Food and Drug Administration (FDA) Sentinel Initiative began over a decade ago to create a national system for monitoring the safety and performance of FDA-regulated medical products. The Sentinel Initiative is the FDA’s response to the FDA Amendments Act (FDAAA) of 2007 requirement that the FDA use electronic healthcare data to rapidly and robustly assess the safety of approved medical products. Since 2016, the FDA has used the Sentinel System to meet its FDAAA requirements.^1–4

The Sentinel System uses a distributed data network, a common data model (CDM), curated real-world data (RWD), and distributed analytic tools to generate real-world evidence (RWE) for FDA decision-making.^5–7 Source data are largely based on health insurance plan claims data that have complete capture of medical encounters and outpatient pharmacy dispensing during health plan enrollment periods—a critical need for medical product safety assessments. Since its launch, the Sentinel System has executed thousands of distributed queries and findings have been used to inform regulatory decisions and FDA Advisory Committee meetings.^8,9 The Sentinel System has allowed the FDA to design, conduct, and analyze several postmarketing studies that FDA previously would have required of medical product sponsors as a postmarketing requirement, thereby meeting a critical goal of the system.

From the start, the Sentinel System was envisioned as part of a national infrastructure to support a learning health system, and several initiatives in the US and abroad are based on the Sentinel common data model, curated data, tools, and networking capabilities.^10–16 While the first 10 years’ work focused on meeting the FDAAA’s legislative requirements, the 21st Century Cures Act of 2016 requires the FDA to develop a framework to evaluate the potential use of RWE to support new indications for an already approved drug or to satisfy postapproval study requirements.¹⁷ To help the FDA understand how best to meet this new mandate, while continuing to improve the Sentinel System’s ability to evaluate medical product safety, new RWE generation approaches need to be explored.

The Sentinel System experience provides a use case to help identify common goals and shared solutions to support a learning health system able to generate evidence for decision-making. In this article we review the FDA’s experience assessing whether RWD is “good enough” to answer FDA postmarketing safety questions, identify areas of data and common data model insufficiency, and discuss options for improvement, especially within the context of increasing access to electronic health record (EHR) data, the growth of distributed data networks and common data models, and emerging technologies, such as natural language processing and deep learning tools.

WHEN IS REAL WORLD DATA GOOD ENOUGH?

FDAAA 2007 required the FDA to create a system for Active Risk Identification and Analysis (ARIA) to address safety issues.¹⁸ Under FDAAA, the FDA cannot require a medical product manufacturer to conduct a postmarketing study unless FDA has determined that both the FDA Adverse Event Reporting System and the ARIA system are insufficient to address the safety issue; this is referred to as the “sufficiency” assessment. Within Sentinel, ARIA is defined as the curated data available in the Sentinel common data model and the validated analytic tools able to query those data; analyses that require additional data elements, medical chart review, or extensive de-novo analytic programing are, for regulatory purposes, outside of the ARIA framework.³ The ARIA regulatory framework provides a lens for considering when RWD data are “good enough” to meet the FDA’s postmarketing safety decision-making needs and when not, to identify areas for improvement to guide future investments in making RWD more useful for the FDA and others.

The ARIA sufficiency experience

Since the 2016 launch of Sentinel System, the FDA found the ARIA system sufficient to answer about half of the safety issues evaluated. A common reason for insufficiency was the inability to identify health outcomes of interest (HOIs) with a computable phenotype that had an acceptable positive predictive value as determined by the FDA review team, for regulatory decision-making. Other reasons for insufficiency included inability to identify the exposure, cohort, or covariates of interest with the data available in the common data model and analytic tool limitations (eg, inability to monitor infant outcomes resulting from in utero exposures).

The ARIA experience established that the Sentinel System is well suited to identify clinically unambiguous, acute, and serious outcomes, such as hospitalized stroke, acute myocardial infarction, intracranial hemorrhage, venous thromboembolism, patients hospitalized with neutropenia and fractures.¹⁹ Conversely, outcomes with complex and/or delayed clinical presentation (eg, neurodevelopmental delays, severe acute liver injury, device implant complications, Hepatitis B reactivation, malignancies), outcomes that may not lead to a medical encounter (eg, migraine, diarrhea), outcomes that may not have standardized data elements available within the CDM (eg, opportunistic and serious infections, immune-mediated disorders), and out-of-hospital deaths are all examples of HOIs that have led to or would likely lead to an insufficiency finding due to the lack of a valid and robust computable phenotype. Expanding the number of HOIs that can be robustly and efficiently identified within the Sentinel System is a critical need for advancing the FDA’s mission.

Computable phenotypes: What’s the best approach?

Improving the Sentinel System’s ability to apply computable phenotypes quickly and efficiently will require an “all-of-the-above” approach that makes better use of health plan claims data while incorporating other data sources, such as EHRs, public health and medical product and disease registries, patient-reported information, and wearables. Beyond computable phenotypes, system improvement also will require new approaches for extracting and storing data elements from EHRs, new analytic methods, use of machine learning and other novel tools, and development of better technologies to enable privacy-protecting distributed analytics and linkages.²⁰

Sentinel system requirements

The Sentinel System has several requirements that influence how to address system limitations (Table 1). A unique characteristic of the Sentinel System use case is that it requires readily available curated data to enable rapid implementation of highly customized and targeted analyses. This contrasts with systems that require high throughput analyses (eg, millions of analyses at a time), hypothesis-free “all-by-all” investigations, just-in-time data extraction and curation, or cross-national comparisons that may have different design and computational needs. The Sentinel System requirements dictate data model and computable phenotype implementation decisions and the associated solutions for HOI identification and phenotype implementation.

Table 1.

Summary sentinel system requirements for ARIA

Requirement	Description
Secure distributed network	Sentinel must use a distributed network approach that allows partners to maintain physical and operational control of their data and approve all data uses. Queries and results must be securely transferred. All locally installed software must be institutionally approved by each data partner.
Curated and quality-checked data	Data must be formatted to the Sentinel CDM and approved by the Sentinel Operations Center before use in ARIA analyses.21 The Sentinel CDM guiding principles include minimization of data transformations or mapping and a reliance on local expertise to support data curation and analysis. Extraction of project-specific data elements, although possible, is outside the ARIA framework.
Analytic flexibility and standardized tools	ARIA analyses must use approved data in the Sentinel CDM and validated analytic tools in the Sentinel Routine Querying System. Extensive customized distributed programming is not included within ARIA but is possible outside the ARIA context.
Longitudinal capture of medical events	Sentinel data sources must have complete capture of medically attended events during a defined period to enable creation of periods of defined person-time. This is primarily achieved through use of health plan insurance data that can leverage enrollment windows as periods during which all medically attended care should be captured.
Transparency and reproducibility	ARIA analyses must be posted online with sufficient detail to be transparent and readily reproducible. Use of source data values, the CDM, and validated tools support this requirement. All computable phenotypes and associated codes must be available.
Access to full text medical records	All partners must have access to full text medical records to support potential follow-up investigations of ARIA findings.
Protection of patient privacy and partner proprietary information	ARIA analyses cannot require transfer of protected health information, personally identifiable information, or confidential information. The CDM does not include direct patient identifiers.
Large populations	Sentinel analyses often require very large populations to identify rare outcomes or study uncommon exposures or cohorts. Therefore, phenotypes should use the largest possible source population which, for Sentinel, is health plan insurance claims data.
Phenotype requirements	Binary phenotype definitions (yes/no) without an onset date are rarely acceptable as HOIs for FDA’s most common use case that requires HOIs to be new events and have an exact onset date that can be related to (typically) an exposure index date; FDA needs to know the temporal sequence of events, not just if an event occurred.

Data models design

The pharmacoepidemiology and informatics communities continue to work on how to best implement common data models in distributed data networks and develop and use computable phenotypes within those networks.^22–28 The FDA’s requirements led the Sentinel System to focus on use of the most granular data available (ie, the source data values) to define phenotypes, avoiding embedded terminology mapping (eg, ICD-10-CM to SNOMED) to maximize transparency and analytic flexibility. This “organizing data model” approach makes use of standard terminologies available at all the Sentinel data partners, meaning there is no need to map across terminologies to effectively use the available data.²² A potential disadvantage of this approach is that it requires “on the fly” phenotype computation. As computable phenotypes increasingly use data from EHRs (eg, excerpts of narrative text) and become more complex, this approach will be challenged and demand new solutions that match the expected use with the best available data. Other use cases and common data models (eg, the Observational Health Data Sciences and Informatics program’s use of the Observational Medical Outcome Partnership data model) espouse mapping standard terminologies found in the source data to clinical concepts and using those concepts to define HOIs.^25,28,29 This mapping approach—a “pre-configured rules system” data model—has efficiency benefits and can help simplify cross-national analyses but introduces risk of information loss, lack of transparency, mapping ambiguity (eg, when codes map to multiple concepts or to no concepts), misleading granularity (eg, when codes are mapped to more granular terminologies), maintenance costs, and inflexibility.^22,25,29,30

In addition, because computable phenotypes typically require different levels of sensitivity and specificity based on the specific context, it is important to allow phenotype definitions maximum flexibility through use of source data values rather mapped values. A condition like Type 2 diabetes mellitus used as an exclusion criterion will have a different computable phenotype than that same condition used as an outcome or as a covariate. This means that the same computable phenotype can be sufficient in one context and insufficient in another. So, although some high-throughput analytic use cases require the analytic efficiency enabled by terminology mapping, the added uncertainty introduced by terminology mapping creates analytic inflexibility. Such gains in analytic efficiency at the expense of analytic inflexibility is considered highly undesirable for the uses that the FDA makes of the Sentinel System.

SENTINEL’S APPROACH TO PHENOTYPES

Given the complexity of developing robust phenotypes that are fit for purpose, Sentinel investigators use a multipronged phenotype development approach that includes review of phenotypes used in prior Sentinel analyses, literature review, code searches across multiple terminologies, forward-backward mapping between ICD-9-CM and ICD-10-CM (when validated ICD-9 definitions are available), clinical expertise, local data system knowledge, and data science expertise, all framed within the context of the specific question and intended use. This phenotype assessment approach addresses issues of missing data, misclassification, local data variability, and potential biases that could be introduced in phenotype definitions.^29,31,32 For the reasons noted above, defining a phenotype often takes weeks or months. When necessary, Sentinel undertakes validation studies with medical chart review to develop phenotypes; Sentinel is currently developing phenotypes using medical chart review for serious infection, stillbirth, anaphylaxis, acute pancreatitis, and lymphoma.³³ This approach, while critical to demonstrating the validity and reproducibility of outcome algorithms, is slow and expensive. To improve this process, Sentinel has launched a project to standardize use of natural language processing and machine learning for complex HOIs using anaphylaxis as a starting point. How best to incorporate the data extracted from such analyses into an “organizing data model” approach is an open challenge.

Using unmapped source data values meets the FDA’s needs for analytic flexibility and transparency, but system improvement (ie, reducing the number of insufficiency findings) is dependent on access to additional standardized data elements, richer clinical data (eg, laboratory result values, vital signs, cancer stage), linked data, distributed querying technologies, and new data resources from which to develop phenotypes. As a starting point, Sentinel is expanding the CDM to capture the case status for patients included in Sentinel chart validation studies, thereby allowing use of the case status to, for example, develop improved algorithms in the future. The case status data will point to the original source data elements that led to the case status determination. This is an initial attempt to maintain the granularity of the data needed to meet the FDA’s requirements and provide an option for recomputing case status if a better algorithm is developed while improving the efficiency of computation. Other system improvements require advances in natural language processing, data provenance and metadata standards, and statistical methods—especially as they relate to missing data and variation in data availability across sites within a network. The good news is that the next steps for improving RWD to enable robust phenotype implementation are identifiable, but progress will require a clear vision with strong use cases and effective collaboration across stakeholders.

Call for collaboration

Our Sentinel System experience suggests that improving the ability to define computable phenotypes in distributed networks is a critical next step that has multiple immediate downstream consequences that would benefit from a collaborative effort across disciplines. FDA’s use case prioritizes identification of incident outcomes and onset dates rather than general assessments of whether a patient should be considered to have a specific condition or not. To that end, a critical step will be to develop best practices for populating common data models with information from EHRs that allows temporally sequenced analyses or information with specific dates of onset. While focused on HOIs, collaborative efforts (Table 2) will have broad value for other purposes and support the FDA’s general mission to use RWD to inform decision-making.^39,40 The FDA recently funded an Innovations Center and a Community Building and Outreach Center that will provide a nexus for collaboration across disciplines and communities with a goal of improving Sentinel System functionality specifically, and real-world data capabilities generally.⁴¹

Table 2.

Areas for collaboration

Data infrastructure and analytic tools
Data models	Encourage data model developers to maintain access to source data values to maximize analytic flexibility and support transparency.
Analytic tools	Encourage analytic tool developers to enable use of source data values.
Rapid data linkage	Given the complex technical, operational, and governance barriers to creating all-purpose standing linked datasets, work is needed to extend prior work to develop approaches to enable just-in-time linkages between data sources (eg, enterprise master patient index).34,35
Rapid data quality assessment	Data quality assessment introduces time and effort burdens that are sometimes inconsistent with intended use case; data do not have to be perfect to be useful. Rapid data quality reviews of only the relevant data elements can facilitate just-in-time data extraction that can together support multiple use cases.
Standardized data quality metrics	As use of distributed health data networks grows there is a need for ways to compare and contrast data sources to assess fitness for use. A set of data model agnostic data quality metrics would help investigators assess data source fitness for use.
Data extraction
Focus on data value granularity	System improvement requires standardized extraction of the most granular clinical data elements to enable building phenotypes with those data elements. The community should focus on identification of data elements in the context of clinical concepts rather than clinical concepts alone. For example, better to extract ejection fraction and the date of service than to identify a patient with heart failure.
Improved natural language processing (NLP) for unstructured text	Improve the accuracy NLP functionality to help identify clinical values from clinical notes, focusing on extraction and standardization of atomic data elements rather than computed concepts alone.36
Computable phenotype automation and characterization
Phenotype transparency	Expand upon ongoing efforts (eg, Phenotype KnowledgeBase [PheKB]) to standardize phenotype definitions with goal of sharing phenotypes, transparency, and reproducibility.37
Learning Lab	Develop a network of learning laboratories that can quickly develop and test new computable phenotypes, including the ability to rapidly extract required information and test phenotypes across data sources, and investigation of probabilistic phenotyping.38

FUNDING

This work was supported in part by US Food and Drug Administration contract # HHSF223201400030I.

AUTHOR CONTRIBUTIONS

JSB and RB collaborated on the original draft; JCM and MN provided critical comments and revision for important intellectual content. All authors approved the final version and agree to be accountable.