Successful Comparison of US Food and Drug Administration Sentinel Analysis Tools to Traditional Approaches in Quantifying a Known Drug-Adverse Event Association

Abstract

The US Food and Drug Administration’s Sentinel system has developed the capability to conduct active safety surveillance of marketed medical products in a large network of electronic healthcare databases. We assessed the extent to which the newly developed, semi-automated Sentinel Propensity Score Matching (PSM) tool could produce the same results as a customized protocol-driven assessment, which found an adjusted hazard ratio (HR) of 3.04 (95% confidence interval [CI], 2.81 to 3.27) comparing angioedema in patients initiating angiotensin-converting enzyme (ACE) inhibitors versus beta-blockers. Using data from 13 Data Partners between January 1, 2008 and September 30, 2013, the PSM tool identified 2,211,215 eligible ACE inhibitor and 1,673,682 eligible beta-blocker initiators. The tool produced a HR of 3.14 (95% CI, 2.86 to 3.44). This comparison provides initial evidence that Sentinel analytic tools can produce findings similar to those produced by a highly customized protocol-driven assessment.

INTRODUCTION

In 2009, the US Food and Drug Administration (FDA) established the Mini-Sentinel pilot project to develop a national system for monitoring the safety of medical products as mandated by the 2007 FDA Amendments Act.¹ During the five-year pilot phase, the Mini-Sentinel program developed the data infrastructure, governance, and analytic capacity to perform active safety monitoring of medical products.^2–4 The pilot has transitioned to Sentinel, which is now a functioning medical product safety surveillance system. The Sentinel program uses a distributed data system that allows data to be stored locally under the control of the 18 participating Data Partners that contribute, and regularly update, administrative claims and clinical information in a common data model.^1,5–7

A key component of the existing Sentinel analytic framework is a set of customizable modular programs that are compatible with the Sentinel common data model and enable FDA to perform analyses to evaluate associations between medical products and pre-specified health outcomes of interest. These programs include tools to perform both self-controlled and cohort-type analyses using complex design and analysis strategies, including self-controlled risk interval analyses⁸ and new user cohort analyses with confounding adjustment via propensity score matching.^9,10 The modular programs were designed to enable retrospective assessments to examine the safety of more established products as well as prospective, sequential assessments of new medical products as data accrue over time in the distributed databases.

An effort is underway to assess the performance of the modular program tools in the Sentinel distributed database environment. This report describes the results of the first such assessment, focusing on the well-known association between angiotensin-converting enzyme (ACE) inhibitors and angioedema.^11–13 In particular, the objective of this investigation was to assess the extent to which the Sentinel Propensity Score Matching (PSM) tool could produce the same results as a more traditional customized protocol-driven assessment, also conducted within the Sentinel distributed data network, that involved a comparison of angioedema risk between ACE inhibitors and beta-blockers.¹⁴

RESULTS

Using data from January 1, 2008 to September 30, 2013 from 13 Data Partners (see Acknowledgment), the PSM tool identified 2,211,215 eligible initiators of ACE inhibitors and 1,673,682 eligible initiators of beta-blockers (Table 1). The average age of ACE inhibitor initiators was 55 years and the average age of beta-blocker initiators was 54 years. A slight majority of ACE inhibitor initiators were male (55%) whereas a slight majority of beta-blocker initiators were female (57%). ACE inhibitor initiators were more likely to have a recorded prior diagnosis of diabetes (21% vs. 10%) and less likely to have a recorded prior diagnosis of ischemic heart disease (5% vs. 13%). Beta-blocker initiators tended to have higher health service utilization in the baseline period, including higher average number of filled prescriptions (8.9 vs. 7.5) and higher average number of ambulatory encounters (6.9 vs. 4.8).

Table 1.

Cohort of new Initiators of angiotensin-converting enzyme inhibitors and beta-blockers (unmatched)

	Primary Analysis				Covariate Balance
Characteristic	Angiotensin-converting enzyme Inhibitors		Beta-blockers
	N	%	N	%	Absolute Difference	Standardized Difference
Patients	2,211,215	100%	1,673,682	100%
Events while on therapy	5,158	0.2%	1,292	0.1%
Person-time at risk (days)	186.9	266.6	149.2	235.1
Patient Characteristics
Gender (F)	997,962	45.10%	946,344	56.50%	−11.4	−0.2
Mean age (standard deviation)	54.6	12.7	53.7	15.6	0.9	0.1
Recorded History of:
Allergic reactions	207,344	9.4%	190,387	11.4%	−2.0	−0.1
Diabetes	471,661	21.3%	173,083	10.3%	11.0	0.3
Heart failure	41,060	1.9%	74,897	4.5%	−2.6	−0.1
Ischemic heart diseases	109,948	5.0%	224,681	13.4%	−8.4	−0.3
NSAID use	318,298	14.4%	250,697	15.0%	−0.6	0.0
Health Service Utilization Intensity:	Mean	Standard deviation	Mean	Standard deviation
Number of generics	3.4	3.5	4.1	4.0	−0.7	−0.2
Number of filled prescriptions	7.5	9.6	8.9	10.8	−1.4	−0.1
Number of inpatient hospital encounters	0.1	0.4	0.2	0.6	−0.1	−0.3
Number of non-acute institutional encounters	0.0	0.6	0.1	0.9	−0.1	−0.1
Number of emergency room encounters	0.2	0.7	0.4	1.0	−0.2	−0.2
Number of ambulatory encounters	4.8	6.3	6.9	8.4	−2.1	−0.3
Number of other ambulatory encounters	1.1	2.6	1.5	3.6	−0.4	−0.1

During 1.1 million person-years of follow-up in the ACE inhibitor cohort and 0.7 million person-years of follow-up in the beta-blocker cohort, we observed 5,158 and 1,292 angioedema events, respectively, corresponding to crude incidence rates of 4.6 and 1.9 per 1,000 person-years. The crude HR was 2.55 (95% CI, 2.40 to 2.71).

A total of 1,309,104 matched pairs were formed, comprising 78% of the beta-blocker cohort (the smaller of the two cohorts; Table 2). ACE inhibitor and beta-blocker initiators in the matched cohort were well balanced on all baseline confounders. The mean age of the matched cohort was 54 years and 12% had a recorded prior diagnosis of diabetes. Appendix 1 illustrates the propensity score distributions at one Data Partner before and after matching. Incidence rates in the propensity score-matched cohorts were similar to those in the overall cohorts (5.0 per 1,000 person-years for ACE inhibitors and 1.8 per 1,000 person-years for beta-blockers; Table 3). The HR in the propensity score-matched cohort was 3.14 (95% CI, 2.86 to 3.44).

Table 2.

Cohort of new Initiators of angiotensin-converting enzyme inhibitors and beta-blockers (matched)

	Primary Analysis				Covariate Balance
Characteristic	Angiotensin-converting enzyme Inhibitors		Beta-blockers
	N	%	N	%	Absolute Difference	Standardized Difference
Patients	1,309,104	59.2%	1,309,104	78.2%
Events while on therapy	3,311	0.3%	988	0.1%
Person-time at risk (days)	183.8	263.7	151.8	238.9
Patient Characteristics
Gender (F)	723,955	55.3%	689,617	52.7%	2.6	0.1
Mean age (std dev)	54.1	13.1	54.4	14.9	−0.3	0.0
Recorded History of:
Allergic reactions	137,920	10.5%	134,933	10.3%	0.2	0.0
Diabetes	150,036	11.5%	150,551	11.5%	0.0	0.0
Heart failure	35,302	2.7%	38,966	3.0%	−0.3	0.0
Ischemic heart diseases	102,200	7.8%	106,786	8.2%	−0.4	0.0
NSAID use	191,798	14.7%	189,612	14.5%	0.2	0.0
Health Service Utilization Intensity:	Mean	Standard deviation	Mean	Standard deviation
Number of generics	3.7	3.7%	3.6	3.6%	0.0	0.0
Number of filled prescriptions	8.1	10.2%	8.0	9.9%	0.1	0.0
Number of inpatient hospital encounters	0.1	0.5%	0.1	0.5%	0.0	0.0
Number of non-acute institutional encounters	0.1	0.7%	0.1	0.7%	0.0	0.0
Number of emergency room encounters	0.3	0.8%	0.3	0.8%	0.0	0.0
Number of ambulatory encounters	5.6	7.3%	5.6	6.6%	0.0	0.0
Number of other ambulatory encounters	1.2	2.9%	1.3	3.0%	0.0	0.0

Table 3:

Results by analysis type

Exposure	New Users	Person-Years at Risk	Average Person-Years at Risk	Number of Events	Incidence Rate per 1,000 Person- Years	Risk per 1000 New Users	Incidence Rate Difference per 1,000 Person- Years	Difference in Risk per 1000 New Users	Hazard Ratio (95% CI)	Wald P-Value
Unmatched Analysis (Data Partner-adjusted only)
ACE inhibitors	2,211,215	1,131,526	0.51	5,158	4.56	2.33	2.67	1.56	2.55 (2.40, 2.71)	<0.0001
Beta-blockers	1,673,682	683,614	0.41	1,292	1.89	0.77
1:1 Matched Analysis, stratified on matched pair; Caliper=0.025
ACE inhibitors	1,309,104	248,697	0.19	1,819	7.31	1.39	4.98	0.95	3.14 (2.86, 3.44)	<0.0001
Beta-blockers	1,309,104	248,697	0.19	580	2.33	0.44

ACE, angiotensin-converting enzyme

DISCUSSION

The Sentinel PSM tool was able to produce an effect estimate (HR, 3.14; 95% CI, 2.86 to 3.44) that was very consistent with that produced by a highly customized protocol-driven assessment (HR, 3.04; 95% CI, 2.81 to 3.27). While this is only one example, it demonstrates that the use of a semi-automated analytic tool in the Sentinel distribute database can produce results similar to those of safety assessments that involve de novo programming. Results of both the PSM tool analysis and the prior protocol-based assessment are also consistent with previous studies performed in other administrative data sources and using other protocol-driven approaches.^15,16

To the extent possible, the analysis plan and inputs used in the PSM tool matched those that were used in the protocol-driven analysis. However, some differences are worth noting. The protocol-driven analysis included data from 17 Data Partners from January 1, 2001 to December 31, 2010. The inclusion of more data explains the slightly narrower confidence intervals observed in the protocol-driven analysis. The PSM tool analysis included all of the pre-defined covariates that were used in the protocol-driven analysis, plus additional measures of health service utilization (e.g., number of unique molecular entities dispensed, number of emergency room visits, number of hospitalizations, etc.) that were automatically included in the propensity score model in the beta version of the PSM tool. Whereas the PSM tool analysis used matching on the propensity score to control confounding by measured factors, the protocol-driven approach used stratification on quintiles of the propensity score. It is possible that the inclusion of more potential confounding factors in the PSM analysis or residual confounding within propensity score quintiles could explain the small numerical difference between the main results. Despite these differences, the results of the two approaches were very similar.

The analytic capabilities of the current PSM tool are more limited than what is possible with a highly-customized protocol-based assessment. However, the Sentinel analytic framework has been developed in a modular fashion, with design and analytic modules that can perform other operations. For example, a separate module can perform analyses using a self-controlled risk interval design, which is particularly useful for vaccine safety surveillance; this module has not yet been tested in the same way as the current assessment of the PSM tool. As the Sentinel system grows, the goal is that the analytic capabilities of the modular tools will also expand to answer more complex medical product safety questions. A key advantage of such pre-programmed analytic tools is that use of validated code reduces opportunities for errors as compared to writing new code for each assessment. The use of clearly defined and specified input parameters also increases the transparency of Mini-Sentinel assessments.

The Sentinel system is intended to complement existing surveillance systems, such as the FDA Adverse Event Reporting System and the FDA Vaccine Adverse Event Reporting System. One of the goals for the transition from the Mini-Sentinel pilot to the Sentinel system is for FDA to incorporate the semi-automated modular analysis tools into the routine work processes for monitoring the safety of marketed medical products. Thus, assessments like the current one are vital to ensure the performance of these modules. Protocol-driven analyses will still be utilized to answer specific questions that go beyond the capabilities of the modular analysis tools.

We applied the PSM tool to a single drug-outcome pair. Focusing on the well-known association between ACE inhibitors and angioedema is an important first step in evaluating the performance of the PSM tool, which helps ensure that the tool is able to identify large and well-characterized associations. However, additional evaluation is needed to ensure that the tool has acceptable performance characteristics across a wider range of known associations. Additional empirical evaluations of known associations (i.e., glyburide and hypoglycemia, clindamycin and risk of clostridium difficile infection, and warfarin and bleeding) are underway within Sentinel and results are forthcoming. Additional ongoing or planned validation activities involve simulated evaluations to more fully understand the strengths and weakness of the PSM tool. Moreover, the PSM tool is designed to perform both one-time analyses, as was the case in the current assessment, as well as repeated, sequential assessments to evaluate the safety of new medical products as experience with those products accrues prospectively in the Sentinel database. More work is needed to evaluate the performance of the tool in the latter setting.

While this evaluation found comparable results from a PSM tool analysis and a highly-customized protocol-based assessment, limitations of the Sentinel distributed database can affect the performance of both approaches in general, and specifically for quantifying the association between ACE inhibitors and angioedema. Most of the data that populate the Sentinel distributed database are administrative claims, which usually lack information on certain potentially important confounders, such as body mass index, smoking, and other lifestyle factors.¹⁷ Race information is not uniformly available across all participating Sentinel Data Partners and rates of angioedema with ACE inhibitors have been found to be higher in certain racial groups.¹⁶ The Sentinel distributed database also does not include genetic information. Specific genetic variants have been found to be associated with ACE inhibitor-induced-angioedema.¹⁸ We expect that these limitations would equally affect analyses using the PSM tool and protocol-based assessments.

METHODS

Data

As of August 2015, the Sentinel distributed database comprised data on approximately 193 million covered lives from 18 Data Partners from 2000 to 2015, including 39 million individuals who are currently enrolled and contributing data. All Data Partners contribute administrative medical and pharmacy insurance claims and demographic data. Some Data Partners also contribute additional clinical data elements, such as laboratory values, body mass index, and vital sign measurements. Data reside behind each Data Partner’s firewall and only the minimum amount of data necessary to respond to a specific query is shared with the centralized operations center. The majority of individuals in the distributed database are privately insured, with approximately two-thirds of individuals between ages 18 and 65 years.

For this study, we used administrative claims data from health care services rendered between January 1, 2008 and September 30, 2013 from 13 Data Partners.

Sentinel tools

The Sentinel Cohort Identification and Descriptive Analysis (CIDA) and PSM tools enable parallel group, active-comparator, propensity score-matched new user cohort studies and can be used for both one-time and sequential analyses in the Sentinel distributed database. Focusing on new users and requiring an active comparator group helps to ensure the appropriate assessment of temporality among exposures, outcomes, and other study variables; ensures that outcomes that occur shortly after initiation and lead to drug discontinuation are captured; and can reduce confounding to the extent that the outcome risk factors similarly determine exposure to the product of interest and the comparator. Propensity score methods are used to further minimize confounding by balancing a potentially large number of possible confounders between the treatment groups.

The program identifies new users of the product of interest and new users of a specified comparator product, based on pre-specified codes (e.g., National Drug Codes). New use is defined by no prior use of the product (or potentially of other specified products such as other products with the same indication) during a specified period preceding each individual’s product initiation (i.e., index) date. Outcomes are identified over a specified risk window following product initiation. Potential confounders are identified in a baseline period of specified length preceding each individual’s index date. A separate propensity score model is fit at each Data Partner and individuals are matched by the propensity score within each Data Partner. Primary measures of association are calculated using a Cox proportional hazards model stratified on Data Partner. This assessment used version 1 of the PSM tool, which has since undergone additional development and assimilation into the Sentinel analytic framework. Enhancements to subsequent versions of the tool increase the options available and improve the utility of the outputs, but do not affect the core analytic function, which was the focus of this study.

Design

The application of the PSM tool was designed to replicate, to the extent possible based on the capabilities of the PSM tool, the customized analysis performed by Toh and colleagues in the Sentinel distributed database. Among several analyses, Toh and colleagues used a propensity score-adjusted design to compare the rate of angioedema among initiators of ACE inhibitors and beta-blockers, finding a hazard ratio (HR) of 3.04 (95% confidence interval [CI], 2.81 to 3.27), which is consistent with previous studies.^11–14 Table 4 summarizes the similarities and differences between this analysis and the prior protocol-based analysis by Toh and colleagues.

Table 4.

Comparison of protocol-based and propensity score matching tool analyses

	Protocol-based analysis	Propensity score matching tool analysis
Number of Data Partners	17	13
Data date range	January 1, 2001 to December 31, 2010	January 1, 2008 to September 30, 2013
Design	New user cohort design	New user cohort design
Exposure of interest	ACE inhibitors	ACE inhibitors
Comparator	Beta-blockers	Beta-blockers
New-use wash-out period	183 days	183 days
Outcome of interest	Angioedema	Angioedema
Outcome definition	ICD-9-CM code 995.1 identified in any care setting	ICD-9-CM code 995.1 identified in any care setting
Confounders (assessed during 183-day baseline period)	Age; sex; recorded history of allergic reaction, diabetes, heart failure, ischemic heart disease, non-steroidal anti-inflammatory drug use	Age; sex; recorded history of allergic reaction, diabetes, heart failure, ischemic heart disease, non-steroidal anti-inflammatory drug use; number of unique generic drugs dispensed; number of prescriptions filled; number of inpatient hospital encounters; number of non-acute institutional encounters; number of emergency room encounters; number of ambulatory encounters
Confounding adjustment method	Propensity score stratification (quintiles)	Propensity score matching
Start of follow-up	Day after drug initiation date	Day after drug initiation date
End of follow-up	First of end of continuous exposure, occurrence of angioedema, or prescription for a drug in the other class, angiotensin II receptor blocker, or aliskiren	First of end of continuous exposure, occurrence of angioedema, or prescription for a drug in the other class, angiotensin II receptor blocker, or aliskiren
Outcome model	Cox proportional hazards model	Cox proportional hazards model

We identified individuals aged 18 years or older who initiated an ACE inhibitor (i.e., benazepril, captopril, enalapril, fosinopril, lisinopril, moexipril, perindopril, quinapril, ramipril, and trandolapril; including combination products involving these agents) or an oral beta-blocker (i.e., acebutolol, atenolol, bisoprolol, carvedilol, labetalol, metoprolol, nebivolol, pindolol, propranolol, timolol; including combination products involving these agents). The index date was the date of first prescription dispensation of one of these drugs following a period of at least 183 days with continuous medical and drug coverage in the database (excluding gaps of <45 days) and with no prior prescription dispensation for an ACE inhibitor, beta-blocker, angiotensin II receptor blocker, or aliskiren. We excluded patients with a diagnosis of angioedema in any care setting during the baseline period.

Follow-up and outcomes

We followed patients for as long as they were continuously exposed to their index medication. We allowed up to 14-day gaps between prescriptions in defining continuous exposure. We censored follow-up upon prescription dispensation for a drug in the other class, angiotensin II receptor blocker, or aliskiren. Consistent with the analysis by Toh et al, we defined angioedema as an inpatient, outpatient, or emergency department claim, including International Classification of Diseases, 9^th Revision, Clinical Modification (ICD-9-CM) code 995.1, indicating angioedema, in any diagnostic position. Prior validation studies have found that this claims-based algorithm has a high positive predictive value, ranging from 90% to 95%.^15,16,19

Statistical analysis

We used a logistic regression model to estimate patients’ propensities to initiate an ACE inhibitor versus a beta-blocker. We used the 183-day baseline period to assess potential confounders. Consistent with the prior analysis by Toh et al, we measured demographic characteristics (i.e., age at index date and sex) and determined whether patients had prior claims for allergic reactions, diabetes, heart failure, ischemic heart disease, and prior prescription nonsteroidal anti-inflammatory drug use. We also assessed several measures of health service utilization intensity,²⁰ including number of filled prescriptions, number of inpatient hospital encounters, number of ambulatory encounters in the baseline period, and a comorbidity score that combines conditions from the Charlson Index and the Elixhauser comorbidity system.²¹

We used the resulting predicted probabilities to match ACE inhibitor and beta-blocker initiators, without replacement, in a 1:1 ratio using a nearest-neighbor matching algorithm with a maximum matching caliper of 0.025 on the propensity score scale.

The propensity score estimation and matching was performed separately within each Data Partner. Data Partners returned de-identified data files containing propensity score values, an indicator of treatment group, a binary outcome indicator, and number of days of follow-up between index date and outcome or censoring date. We used a Cox proportional hazards model to estimate a crude HR and 95% CIs in the unmatched population and a separate model to estimate an adjusted HR and 95% CI in the 1:1 propensity score-matched cohort. Both models were stratified by Data Partner and the latter model was further stratified on matched pair.

The Sentinel system is a public health activity under the auspices of the FDA and is not under the purview of institutional review boards.^2,22

STUDY HIGHLIGHTS.

• What is the current knowledge on the topic?

The US Food and Drug Administration’s Sentinel system has developed the capability to conduct active safety surveillance of marketed medical products in a large network of electronic healthcare databases, but the performance of the tools has not been evaluated.

• What question did this study address?

This study assessed the extent to which the newly developed, semi-automated Sentinel Propensity Score Matching (PSM) tool could produce the same results as a more traditional customized protocol-driven assessment.

• What this study adds to our knowledge

The Sentinel PSM tool was able to produce an estimate that was very consistent with that produced by a highly customized protocol-driven assessment.

• How this might change clinical pharmacology or translational science

This comparison provides initial evidence that Sentinel program tools can produce findings similar to those produced by a highly customized protocol-driven assessment.