Development and application of two semi-automated tools for targeted medical product surveillance in a distributed data network

Abstract

Purpose of Review

An important component of the Food and Drug Administration’s Sentinel Initiative is the active post-market risk identification and analysis (ARIA) system, which utilizes semi-automated, parameterized computer programs to implement propensity-score adjusted and self-controlled risk interval designs to conduct targeted surveillance of medical products in the Sentinel Distributed Database. In this manuscript, we review literature relevant to the development of these programs and describe their application within the Sentinel Initiative.

Recent Findings

These quality-checked and publicly available tools have been successfully used to conduct rapid, replicable, and targeted safety analyses of several medical products. In addition to speed and reproducibility, use of semi-automated tools allows investigators to focus on decisions regarding key methodological parameters. We also identified challenges associated with the use of these methods in distributed and prospective datasets like the Sentinel Distributed Database, namely uncertainty regarding the optimal approach to estimating propensity scores in dynamic data among data partners of heterogeneous size.

Summary

Future research should focus on the methodological challenges raised by these applications as well as developing new modular programs for targeted surveillance of medical products.

Introduction

Targeted surveillance using multiple large databases is an increasingly important part of post-market medical product safety in the United States [1, 2]. Such active, semi-automated surveillance is characterized by evaluation of specific exposure-outcome relations of interest in longitudinal, routinely-collected healthcare data, and provides safety information that complements passive surveillance or spontaneous reporting systems. Spontaneous reporting systems -- the Vaccine Adverse Event Reporting System (VAERS) and the Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) for vaccines and drugs, respectively -- have traditionally been the backbone of medical product safety monitoring, but they are limited by selective reporting [3]. Further, and most importantly, they lack denominator data to allow for accurate quantification of the magnitude of associations of interest.

Longitudinal databases that capture routinely-collected healthcare utilization information have long been available to evaluate pharmacoepidemiologic questions of interest. However, single databases often lack the needed sample size to identify rare safety issues that also elude detection in relatively small clinical trial populations, necessitating multi-database studies in the post-market setting. Traditionally, multi-database or multi-site studies required data use agreements to be signed among all participating databases to allow their data to be centralized for analysis.

A newer model has emerged – the distributed data network [4, 5] – in which multi-site, routinely-collected, longitudinal observational data are used without requiring databases to share patient-level data with each other. One of the first distributed surveillance systems for medical products in the United States was the Vaccine Safety Datalink (VSD), initiated in 1990 [6–9]. The Centers for Disease Control and Prevention (CDC) partnered with four health maintenance organizations across the country who agreed to provide de-identified information on vaccine usage, basic demographic characteristics, and medical records to the CDC, but did not share their data with other databases. That model would go on to be replicated in other national networks.

In 2008, the FDA launched the Sentinel Initiative to create a national, actively-monitored, and distributed data system to evaluate the safety of regulated medical products in the United States [10•, 11]. The Sentinel Infrastructure, a component of the Sentinel Initiative (see Figure 1), uses a common data model to facilitate distributed analyses across multiple data partners who might otherwise format data differently [12]. The distributed data network includes 17 observational databases that cover 223 million lives with 43 million patients currently accruing new data [10, 13, 14]. Data in the Sentinel Infrastructure can also support analyses performed with customized programming according to fully developed protocols, including large observational analyses [15, 16] and cluster randomized trials [17].

Figure 1. Schematic of the Food and Drug Administration’s Sentinel Initiative^a.

This figure outlines the hierarchy of the Sentinel Initiative, which contains a Sentinel Infrastructure comprised of the Sentinel System (the focus of this manuscript) as well as the FDA-Catalyst.

^aReprinted from https://www.sentinelinitiative.org/sentinel/about

Contained within the Sentinel Initiative is the congressionally-mandated active post-market risk identification and analysis (ARIA) system [18]. While the same underlying electronic data model supports both the ARIA system and the greater Sentinel Infrastructure, the distinguishing feature of ARIA is that it is limited to pre-programmed, reusable, quality-controlled and parameterized computer programs or queries. Thus, ARIA can rapidly support public health safety surveillance, as no de novo programming is required.

In brief, these queries or computer programs work by executing on each participating data partner's data that have been formatted into the Sentinel Common Data Model behind their firewalls, and then returning non-identifiable aggregate information to a central analysis center for final synthesis [19••]. In addition to leveraging the interoperability of electronic health information in a multi-site setting, such a public-private partnership allows data partners to maintain control of their data and protect patient privacy [4, 5].

Here, we focus on two programs used within ARIA for assessing associations between medical products and health outcomes of interest: the propensity-score (PS) adjusted active comparator analysis and the self-controlled risk interval design. We first discuss prior work leading to their development as pre-programmed analysis tools before moving on to describing their applications within the Sentinel System, highlighting unique challenges associated with their use in conducting safety surveillance across distributed databases.

Propensity Score Adjusted Analyses

In this section, we describe literature relevant to the development of the PS adjusted active comparator analysis tool in ARIA. Summary scores have a long history as a confounding adjustment technique in observational designs because they summarize patient characteristics into a single number [20–24], enabling efficient adjustment for a large number of potential confounders while preserving patient privacy and limiting transfer of patient-level data. Two types of summary scores are currently used: PSs, [25], which model the relationship between potential confounders and the exposure, and disease risk scores (DRSs) [26], which model the relationship between potential confounders and the outcome.

Both scores can be advantageous in the setting of newly marketed medications [27, 28]. A DRS-based approach may be preferable in the very early post-market period when there are too few exposed patients or too few outcomes to support PS modeling or traditional outcome regression models [22]. In particular, a DRS can be developed in an historical cohort of comparator patients and the resultant coefficients can be used to adjust for confounding in the analysis cohort without the need for fitting a statistical model in this group. When PS models can be fit in the analysis cohort, they tend to perform at least as well as DRS-based approaches [29]. PS approaches have the additional benefit that covariate balance after adjustment is easily displayed in a traditional baseline covariate characteristics table. For these reasons, PS based methods were implemented as a parameterized ARIA targeted surveillance tool.

Adjusting for a large number of covariates in a PS potentially allows an investigator to partially control for important confounders that are unmeasured or incompletely captured in electronic healthcare databases, such as frailty, through proxy adjustment. For example, variables that might not be direct risk factors of an outcome, such as oxygen tank usage, may in fact be correlated with underlying frailty and worth including in a PS model. Such proxy adjustment is central to the high-dimensional propensity score (hd-PS) approach, which is an algorithm that empirically identifies large numbers of potential confounding variables for inclusion in a PS model by ranking them based on their potential to cause bias [30].

Propensity scores in small samples

In the early post-market period, it is in the interest of patients and regulators to detect safety issues as soon as possible. However, in this period there will typically be few outcomes and few users of the new medical product relative to a comparator. Few new users of the target medical product can be problematic for PS methods, and especially for hd-PS, which typically includes hundreds of potential confounders in the PS model. To address this concern, the performance of hd-PS in settings where there are few exposed patients or outcome events was evaluated by re-sampling from four new user cohorts identified in administrative claims data to create increasingly smaller cohorts [31]. In these down-sampled cohorts, the authors examined the performance of hd-PS and sought to define the optimal number of empirically identified covariates to include. When there were greater than 50 exposed patients with an event, the hd-PS algorithm that ranked potential confounders by their relationships with both exposure and outcome, termed the bias-based algorithm, performed well in terms of confounding adjustment. When there were fewer than 50 exposed events, ranking covariates for inclusion based only on their association with exposure yielded better performance.

Propensity scores in distributed data settings

Compared with analyses in single databases, performing analyses in multiple databases improves statistical power, which can enable more rapid identification of potential medical product safety issues. A challenge of the distributed data setting is attaining confounding control while simultaneously minimizing or eliminating sharing of patient-level data. Three recent studies compared various methods for achieving this goal, which included comparing full patient-level data sharing, cell-aggregated sharing, distributed regression analysis, meta-analysis, and PS-based methods [27, 32, 33]. These evaluations concluded that PS methods provided adequate flexibility, confidentiality, and analytic integrity in distributed data settings. There are several ways of employing PS-based methods in distributed environments. In its simplest form, individual data partners estimate a PS within their own data and share only the patient-level PS, along with a randomly generated patient identifier, exposure and outcome status, and relevant subgroup indicators with a central analysis hub like the Sentinel Operations Center. The feasibility of this patient-level PS-sharing approach was demonstrated in a cohort of new users of clopidogrel and proton pump inhibitors versus new users of clopidogrel alone monitored for risk of myocardial infarction. Data were shared among four data partners without compromising patient privacy by using PSs [32]. Since the development of the ARIA targeted surveillance program for PS adjustment, further adaptations of the program have employed a risk set-based [34] approach that enables a PS-adjusted analysis in a distributed setting without sharing patient-level PSs [35]. The risk-set based approach has been shown to provide statistically equivalent results to a pooled analysis and has been recently applied to actual safety assessments in the Sentinel Distributed Database [15, 16].

Propensity scores in prospective analyses

While PS methods have demonstrated utility in distributed data, prospective analyses, where data periodically update over time, provide a unique set of challenges. To assess PS performance in prospective data, a recent study developed an approach to prospective, targeted safety monitoring and evaluated it using three drug-outcome pairs with known or suspected safety issues: paroxetine and severe upper gastrointestinal bleed, lisinopril and angioedema, and ciprofloxacin and Achilles tendon rupture [36]. The authors divided a single dataset into sequential monitoring periods to mimic the prospective accumulation of data, and 1:1 matched new users of the study drugs within each monitoring period on a PS estimated only among those new users in each period. Two alerting algorithms were compared: a maximized sequential probability ratio test [37] and a method that generated an alert if the effect estimate exceeded a predefined threshold for more than 3 consecutive monitoring periods. Alerts were generated by at least one of the alerting algorithms in each example over the follow-up period, demonstrating the feasibility of such a PS-matched prospective monitoring system to quickly identify adverse events. It is important to note that while this assessment emulated prospective analyses, the dataset was static and therefore did not account for issues related to changing data over time that can occur when using dynamic datasets in the Sentinel Distributed Database.

Propensity scores in both distributed settings and prospective analyses

Given the success of PS methods in both distributed data and prospective analyses separately, the next step was to assess PS methods in distributed, prospective data simultaneously. One of the concerns regarding automated safety surveillance systems in multiple databases is the opportunity for many false positives or false negatives [38]. As a first step in evaluating these concerns, a PS-based semi-automated safety monitoring approach was utilized to assess the safety of three medications across three claims databases [39]. Data from each database were divided into three-month intervals to mimic prospective accumulation of data, and new users of the three study drugs were 1:1 PS matched to new users of comparator products within each interval. The three drug-outcome relationships were rosuvastatin and rhabdomyolysis, rosuvastatin and diabetes mellitus, and telithromycin and hepatotoxicity. As expected based on prior research, none of these three examples generated a safety alert, providing some reassurance about the system's robustness against false positives. By sharing only PSs with a central data analysis center, confounding was mitigated without compromising patient privacy.

This semi-automated program served as the precursor program to the PS-adjusted active comparator analysis tool utilized in ARIA, and a subsequent paper described the program in more detail and discussed the steps required to conduct prospective semi-automated surveillance using PS matching in distributed databases [19••]. The PS adjustment tool has several functions and includes each of the features noted above. When tested on retrospective data for nine drug-outcome pairs, which included positive and negative controls, the tool generated results consistent in direction and magnitude with expectations for each. Alerts were generated for all positive controls when power was adequate, which included the three known positive associations described above. These findings further demonstrated the ability of the system to avoid generating false positives and negatives.

The PS adjustment precursor program was also applied to a true prospective safety monitoring scenario in which data accumulated in real time to evaluate the risk of hemorrhagic and ischemic events in patients taking prasugrel versus clopidogrel during the first two years of prasugrel's market availability in the US [40•]. Using the prototype, patients were matched on hd-PS and followed for outcomes using dynamic data that were updated on a bi-monthly basis. In general, results were consistent with those from randomized trials. While the first monitoring period had to be lengthened to accumulate enough users to validly estimate the hd-PS, the tool performed well and demonstrated the feasibility of this type of analysis in a prospective, distributed database setting like Sentinel. In light of these successful applications, the semi-automated PS adjusted active comparator design tool was officially implemented into the Sentinel Initiative as part of ARIA [41, 42].

Applications within the Sentinel Distributed Database

The PS adjustment tool in use by ARIA in the Sentinel Initiative can perform propensity score matching (both 1:1 and 1:n) and propensity score stratification [43]. It has been used to perform both retrospective safety assessments as well as prospective safety assessments within the Sentinel Distributed Database. Retrospective safety assessments are one-time analyses in which the analysis is performed using all existing data. Retrospective safety assessments included the following drug-outcome pairs: angiotensin converting enzyme inhibitors (ACEIs) vs. beta blockers on angioedema [44•], clindamycin vs. penicillin on risk of C. difficile infection, glyburide vs. glipizide on hypoglycemia [45], warfarin vs. statins on bleeding, dabigatran vs. warfarin on bleeding/acute myocardial infarction, apixaban vs. warfarin on gastrointestinal hemorrhage and stroke, niacin vs. fenofibrate on bleeding and stroke, and levetiracetam vs. topiramate or lamotrigine on agranulocytosis. The ACEI, clindamycin, glyburide, and warfarin vs. statins assessments were all test cases in which elevated outcome risk associated with the primary exposure was expected. These assessments were performed as empirical test cases as opposed to novel regulatory questions or considerations.

The PS adjustment tool has also been used for one prospective safety assessment, in which data regularly accumulate in real time and are re-analyzed after each update. The prospective queries targeted rivaroxaban vs. warfarin on gastrointestinal hemorrhage and stroke risk. A full description of each retrospective and prospective query has been provided elsewhere [46••]. A high-level summary of each query is available in Table 1.

Table 1.

Tabular Summary of Queries Using the Propensity-score Adjustment Tool within Sentinel

Query	Outcomes	#DPs Returned/ Sent	Query Period1
Test cases of known associations
ACEI vs. beta-blockers	Angioedema	13/17	01/01/2001–09/30/2013
Clindamycin vs. oral penicillins	C.Difficile infection	13/14	01/01/2006–12/31/2013
Glyburide vs. Glipizide	Hypoglycemia	7/13	01/01/2008–12/31/2014
Warfarin vs. statins	Bleed	14/14	01/01/2012 – 09/30/2015
One-time drug safety assessments
Dabigatran vs. warfarin	ICH	4/4	11/01/2010–12/31/2013
	GI Bleed
	Ischemic Stroke
	AMI
Apixaban vs. warfarin	GI Bleed	4/4	02/01/2013–05/31/2015
	ICH
	Stroke
Niacin vs. fenofibrate	GI Bleed	4/4	01/01/2007–05/31/2013
	ICH
	Stroke
Levetiracetam vs. lamotrigine/topiramate	Agranulocytosis	10/17	01/01/2000–10/31/2013
Prospective drug safety assessments
Rivaroxaban vs. warfarin	Ischemic Stroke	4/4	11/01/2011–4/30/2015
	ICH
	GI Bleed

ACEI – Angiotensin-Converting Enzyme Inhibitor; AMI - Acute Myocardial Infarction; DP – Data Partner; GI - Gastrointestinal; ICH - Intracranial hemorrhage

¹Query period represents available data across all Data Partners included in that query

Self-Controlled Risk Interval Design

The self-controlled risk interval design is a specific application of the self-controlled case series design in which eligible patients are those exposed to a medical product of interest and are indexed on exposure. This contrasts with a case-crossover study, in which eligible patients are indexed upon occurrence of the outcome. However, both types of self-controlled designs control for all time-invariant confounding so most complicating factors in self-controlled designs involve adding additional model terms to account for time-varying confounding [47]. While the self-controlled case series design has been used extensively in epidemiological research [48–51], the self-controlled risk interval design was first used in the VSD and the Post-licensure Immunization Safety Monitoring system [52] for vaccine safety studies [53–60]. In this design, an exposed person acts as his or her own control and the comparison occurs between a period when the person is thought to be at heightened risk for experiencing a health outcome of interest (i.e., the risk window) and a period without heightened risk (i.e., the comparison window). Post-exposure comparison windows (as compared to pre-exposure comparison windows) are typically used because experiencing the outcome is often presumed to affect the person’s likelihood of receiving a subsequent vaccination (i.e., the healthy vaccinee effect) [61, 62]. This design was also used in the Sentinel System before it was standardized into a pre-parameterized usable program for use in ARIA [63•–65]. Of these prior uses, two were retrospective analyses and one was prospective sequential surveillance. Like the PS adjusted tool described above, a prototype of the pre-parameterized program was tested in a dynamic data setting to ensure its performance with a known association between an exposure-outcome pair [66].

Since the conversion of the self-controlled risk interval design into a semi-automated program, it has been planned to prospectively monitor the safety of nine-valent human papilloma virus vaccine [67] and has been used retrospectively to assess seizures following ranolazine exposure [68].

Discussion

We described applications of semi-automated, targeted evaluations of particular exposure-outcome pairs of interest in the Sentinel Distributed Databases using one of two quality-checked, parameterized computer programs: the PS adjustment tool and the self-controlled risk interval design tool. Both of these epidemiologic designs were well-studied in single database settings and were adapted for multi-site distributed database uses when minimizing transfer of patient-level data is of paramount importance. When these programs have been piloted on known medical product-outcome associations, they have correctly generated safety alerts and avoided false positives while maintaining patient privacy. The successful performance of the program allowed investigators to confidently apply it to true safety analyses to support post-market medical product regulatory decisions.

The successful development and application of pre-written, semi-automated surveillance programs for both drugs and vaccines has highlighted several of their strengths. The primary advantage of converting these commonly used study designs into semi-automated, pre-written programs is that safety analyses can be performed more quickly than if these designs had to be repeatedly programmed de novo. In the immediate post-approval setting, where important adverse effects may be unknown due to their rarity or differences in patient populations from the pre-market trials, rapidly identifying adverse events is critical to patient safety. Another strength of pre-programmed tools is that by automating programming, investigators have more time to spend considering key clinical and epidemiological decision points such as the relevant confounders and statistical analysis plan. Pre-programmed tools also ensure results are consistent and reproducible by preventing coding errors that can occur from ad hoc programming.

The applications described above have also identified certain challenges when conducting targeted surveillance in distributed databases. First, the constituent databases of the Sentinel Distributed Database are of heterogeneous size. When performing PS estimation, the model may fail to converge at a subset of smaller databases as a result of small sample size. The best approach for estimating PSs in small data partners within distributed data networks is still unknown, and such problems may be an inherent limitation of distributed databases like Sentinel, if small data partners are included. An alternative in such situations could be to instead use a DRS for confounding adjustment, especially if the outcome of interest is more common than new use of the target therapy. In settings where outcomes are also rare, use of historical data from the time period immediately prior to the introduction of the target therapy may be preferable for DRS development [22]. Databases of heterogeneous size are not as challenging to deal with when using a self-controlled risk interval design, however only databases with recorded outcomes are informative to the analysis.

Second, sequential analyses of prospectively accumulating data present a distinct set of methodological challenges. The Sentinel Distributed Database is a dynamic database and data are continually refreshed over time. Claims adjustments and late-arriving data can result in changes to a patient’s exposure, outcome, or covariate status over time. For example, a patient who experiences the outcome of interest in the first monitoring period may not be classified as such until the second monitoring period. However, sequential statistical analyses, first used in the clinical trial setting, depend (or are anchored on) data in prior monitoring periods remaining stable and constant [69]. Historically, in the VSD, follow-up time for prospectively monitored patients was limited to a pre-defined and short observational window. Because of this, data were allowed to stabilize following the completion of the observational window before they were ready for analysis. A notable exception to this stabilization occurs with influenza vaccine safety surveillance when partially-accrued data must be analyzed in order to have actionable information during the ongoing influenza season [54, 60, 65, 69].

In the pre-parameterized self-controlled risk interval design tool in ARIA, patients are ascertained in the analytic dataset only after they have had the opportunity to complete their full observation window. Thus, a patient’s relevant data are contained within a single snapshot of the dynamic dataset. However, outcomes that have longer time-to-onset (on the order of months) and that further require an investigator to follow the same patient over multiple data refresh periods are particularly challenging. In PS adjusted analyses, follow-up time is often “as treated,” i.e., as long as the patient is actively exposed to the medical product. Therefore, it is not feasible to prospectively monitor a patient’s entire relevant follow-up in a single snapshot of the dynamic dataset. Adapting sequential statistical analyses to cope with dynamic data over multiple data refreshes in an observational database is the subject of ongoing methods work.

While PS-based methods have been successful, alternative methods for analyzing distributed data networks while maintaining patient privacy are available. The PS adjustment tool utilizes matching and stratification, but in theory could be extended to allow for inverse probability of treatment weighting. The tool could also be extended to alternative summary scores such as the DRS, which would offer advantages in situations where outcomes are more common than new initiators of the medical product of interest as discussed above [22].

A distinct method which does not rely on covariate summary scores is distributed regression analysis, in which data partners run regression models locally and then share only intermediate summary statistics with a central analysis hub. Distributed regression has been proven to provide identical results as a fully pooled analysis [27, 70, 71]. Like the other programs developed in Sentinel, distributed regression requires a common data model and shares only aggregated information that preserves patient privacy. Efforts to automate this methodology in the Sentinel System are underway.

Conclusions

In conclusion, the development of semi-automated, parameterized computer programs that perform inferential analyses in multi-site distributed databases continues to evolve, allowing for timely and reproducible evaluations of particular exposure-outcome pairs. These programs are quality-checked and available to the public. They are also currently in routine in use as part of the FDA’s ARIA system. Future research should focus on the methodological challenges raised by these applications, as well as on developing new modular programs for targeted surveillance of medical products.

Key Points.

• Semi-automated, pre-programmed tools for implementing propensity score-adjusted active comparator and self-controlled risk interval designs have been successfully applied to multiple prospective safety analyses in the Sentinel Distributed Database

• Public availability of these tools enables rapid and consistent analyses across multiple data sites

• Future research should focus on addressing challenges to implementing these designs in distributed, prospective databases like Sentinel