Active safety monitoring of newly marketed medications in a distributed data network: application of a semi-automated monitoring system

Abstract

We developed a semi-automated active monitoring system that uses sequential matched-cohort analyses to assess drug safety across a distributed network of longitudinal electronic healthcare data. In a retrospective analysis, we showed that the system would have identified cerivastatin-induced rhabdomyolysis. In this study, we evaluated whether the system would generate alerts for three drug-outcome pairs: rosuvastatin and rhabdomyolysis (known null association), rosuvastatin and diabetes mellitus, and telithromycin and hepatotoxicity (two examples for which alerting would be questionable). During >5 years of monitoring, rate differences (RDs) comparing rosuvastatin to atorvastatin were -0.1 cases of rhabdomyolysis per 1,000 person-years (95% CI, -0.4, 0.1) and -2.2 diabetes cases per 1,000 person-years (95% CI, -6.0, 1.6). The RD for hepatotoxicity comparing telithromycin to azithromycin was 0.3 cases per 1,000 person-years (95% CI, -0.5, 1.0). In a setting in which false positivity is a major concern, the system did not generate alerts for three drug-outcome pairs.

INTRODUCTION

With the ongoing development of the US Food and Drug Administration’s Sentinel Initiative (1), and similar systems around the world, near-real-time active medical product safety monitoring may soon be a reality. These systems will enable regulators and other stakeholders to monitor outcomes of medical product use in distributed data networks comprising healthcare utilization data for many millions of patients (2). The amount of data contained in such systems raises concerns that the systems will generate intractable numbers of false positive alerts for purely statistical reasons (3). The non-randomized data that will feed these systems are collected in routine care and may therefore signal relations that arise from medical perceptions or administrative constraints rather than from biology, raising the additional specter of systematic false positive findings (4,5).

The challenges inherent in a broad-based monitoring system are similar to those confronted in every pharmacoepidemiologic study, and a system that incorporates sound design and analysis may reduce false positive signals. We have developed a semi-automated, sequential, propensity-score matched cohort approach (6) to drug safety monitoring built on validated methods for drug safety research and that can be easily deployed in distributed data networks (7,8). The approach involves: (1) identifying new users of a medical product; (2) matching them by propensity score to new users of a comparator product; (3) tabulating results across the distributed databases; and (4) applying an automated alerting algorithm selected from an earlier simulation study (9).

Had this system been in place at the time, it would have identified cerivastatin-induced rhabdomyolysis in longitudinal electronic healthcare data as early as a year before the drug was withdrawn from the US market (9). Given the concerns about statistical and systematic false positivity, we expanded the application of the approach to three additional examples including one in which we did not expect an alert (rosuvastatin and rhabdomyolysis), and two examples for which we were not certain whether alerting would be expected – telithromycin and hepatotoxicity, and rosuvastatin and diabetes mellitus. We describe what would have occurred had this system been in place at the time of US market approval of each drug.

RESULTS

Rosuvastatin and the risks of rhabdomyolysis and diabetes mellitus

Over more than five years of monitoring, we observed 8 cases of rhabdomyolysis among 57,998 propensity score-matched rosuvastatin and atorvastatin pairs who contributed a total of 45,571 person-years of follow-up (Figure 1). Two of the 8 rhabdomyolysis cases occurred among rosuvastatin-treated patients, resulting in a difference at the end of monitoring of -0.1 (95% CI, -0.4, 0.1) events per 1,000 person-years and a corresponding rate ratio of 0.4 (95% CI, 0.1, 1.9). None of the three selected algorithms generated alert in this example.

Figure 1.

Active monitoring for rhabdomyolysis among new users of rosuvastatin compared to new users of atorvastatin

Among similar follow-up times for the diabetes outcome, we observed 859 incident diabetes diagnoses among rosuvastatin-treated patients as compared to 1,055 among atorvastatin-treated patients, resulting in a rate difference at the end of monitoring of -2.2 (95% CI, -6.0, 1.6) events per 1,000 person-years (Figure 2) and a corresponding rate ratio of 0.9 (95% CI, 0.9, 1.0). None of the selected algorithms generated alerts for this outcome either.

Figure 2.

Active monitoring for diabetes mellitus among new users of rosuvastatin compared to new users of atorvastatin

Telithromycin and risk of hepatotoxicity

We identified and matched 106,658 initiators of telithromycin to the same number of initiators of azithromycin over more than 5 years of monitoring following telithromycin marketing authorization. The 106,658 matched pairs contributed a total of 17,720 and 17,416 person-years of follow-up to the telithromycin and azithromycin groups, respectively (Figure 3). We observed 41 cases of hepatotoxicity during follow-up, of which 23 (56%) occurred among telithromycin-treated patients. None of the three selected algorithms generated an alert. The rate difference at the end of monitoring was 0.3 (95% CI, -0.4, 1.0) events per 1,000 person-years and the rate ratio was 1.3 (95% CI, 0.7, 2.3).

Figure 3.

Active monitoring for hepatotoxicity among new users of telithromycin compared to new users of azithromycin

DISCUSSION

The proposed approach to active drug safety monitoring in electronic healthcare data, which combines semi-automated procedures and some expert inputs, previously detected cerivastatin-induced rhabdomyolysis, a known rare safety issue, as much as a year before the drug was withdrawn from the market. That the system detected this known drug safety issue provides some reassurance that it performs according to expectation. In the present application, the system did not generate alerts for rosuvastatin and rhabdomyolysis, a presumed true negative, or for two examples for which existing evidence is equivocal. False alerting is a major concern in emerging active medical product surveillance systems that will include data on many millions of patients. False positives, as well as false negatives, can arise from many sources including chance but, more importantly in observational data, from biases such as confounding by indication. Our approach to active monitoring is designed to mitigate such biases, by relying on new users, active comparators, and PS-matching, in order to minimize the number of false positive and false negative signals from the outset, before they can have adverse public health consequences.

Around the time that FDA authorized rosuvastatin for marketing in the US, concern had been raised regarding its associated with an elevated risk of rhabdomyolysis as compared to other available statins. In particular, development of the highest dose rosuvastatin tablets was discontinued because of an unacceptable increase in risk of rhabdomyolysis (10). However, the potential association between rosuvastatin and rhabdomyolysis has been evaluated in many post-marketing studies, none of which have found an elevated risk above and beyond that conferred by other available statins (11–15). Given the concern about false positivity in medical product safety monitoring systems that will monitor myriad products and outcomes among many millions of patients, our finding of a true negative is again reassuring.

Recently, concern has been raised about whether statins increase patients’ risk of developing type 2 diabetes mellitus. A meta-analysis of 13 randomized trials comprising 91,140 participants found that, over a mean follow-up of 4 years, statins were associated with a 9% increase in odds of diabetes as compared to placebo or no treatment (16). Another meta-analysis reported similar findings and that any effect of statins on diabetes is likely a class effect (17). Because we compared new users of rosuvastatin to new users of atorvastatin, our safety monitoring question is one of comparative, rather than absolute, safety. Consistent with the meta-analyses, the system did not identify a higher risk of diabetes among patients treated with rosuvastatin as compared to patients treated with atorvastatin. However, we did not assess whether the system would have identified the small diabetes risk associated with statins versus no treatment. Subsequent monitoring activities could explore the relative safety of statins on diabetes as compared to other drugs to treat dyslipidemia, such as ezetimibe.

Evidence is mixed regarding whether telithromycin increases the risk of hepatotoxicity relative to similar antibiotic agents. A 2001 FDA advisory committee voted against approval of telithromycin primarily because of concern about hepatotoxicity. Following the completion of a controversial trial, FDA approved telithromycin in 2004 stating that the frequency and severity of hepatotoxicity with telithromycin was similar to that of other macrolides (18). Through April 2006, FDA received 42 cases of telithromycin-associated liver injury to its Adverse Event Reporting System (19). Post hoc analyses of the spontaneous reports detected an association between telithromycin and reports of liver injury, including mild disorders up to fulminant hepatotoxicity (20,21). However, neither of two epidemiological studies that each included more than 100,000 initiators of telithromycin, found evidence to suggest that telithromycin increases risk of hepatotoxicity as compared to other macrolides (22). Telithromycin remains on the market in the US, carrying only a labeled warning about hepatotoxicity, but not a black box warning (23).

Our system did not generate alerts for telithromycin. Again, we used an active comparator, which reformulates the monitoring question to one of comparative safety. These findings from our system are compatible with those of the other two large post-marketing database studies. While the system did not generate alerts, the results also cannot rule out a small increased incidence of hepatotoxicity associated with telithromycin as compared to azithromycin. At the end of the monitoring timeframe, the incidence rate of hepatotoxicity was 1.3 events per 1,000 person-years in the telithromycin group and 1.0 events per 1,000 person-years in the azithromycin group, corresponding to a rate difference of 0.3 (95% CI, -0.5 to 1.0) events per 1,000 person-years and an incidence rate ratio of 1.3 (95% CI, 0.7 to 2.3). Although it is unclear whether the system’s lack of alerting in this example represents a true or false negative, this example highlights an important benefit of active monitoring systems, such as Sentinel. Even in the absence of an alert, active monitoring systems will provide useful, continuous decision support at low cost since secondary data are captured routinely and in near-real-time and analyses can be performed sequentially in an automated fashion.

Our sequential PS-matched cohort framework for active drug safety monitoring offers several advantages when focusing on pre-specified outcomes that may be related to medical products. As illustrated by the rosuvastatin examples, PS-matched cohorts easily enable monitoring of multiple outcomes per product. In addition, new-user cohort construction, PS estimation, and matching procedures can be largely automated (24). Balancing observed pre-treatment confounders via PSs separately in each database also simplifies data aggregation across multiple data sources while obviating privacy concerns (7,8). In a drug’s early market phase, there is usually an abundance of users of an active comparator relative to users of the monitoring drug, offering sufficient opportunity to successfully match almost all patients exposed to the newly marketed drug. Resulting estimates of association then pertain to the effect observed among those exposed to the monitoring product as compared to the effect that would have been observed had those exposed to the monitoring product actually been exposed to the comparator agent. In addition, PS-matching enables monitoring of simple marginal event rates between matched groups as the groups have been balanced on a large number of potential confounders (25). Event rates can be expressed as observed (i.e. rate in the new drug group) and expected (i.e. rate in the comparator group), which allows for direct estimation of both absolute and relative measures of association and simplifies the application of a broad range of alerting algorithms without the need for further regression adjustment.

We encountered several practical limitations in implementing our approach in the four examples. Because we began monitoring upon each drug’s market entry, we identified few new users of the new drug during the first few month after the product was launched, which precluded us from fitting large PS models in the first quarters. As a simple solution, we combined the first two quarters to create the first monitoring period. While this enabled us to fit PS models in data from the first two quarters, it necessarily delayed the time to earliest possible alerting. However, it is unlikely that among few exposed patients there would be a sufficient number of events to generate an alert. In addition, while many aspects of an active monitoring system can be automated, human inputs remain critical elements to any public health surveillance activity. As in any pharmacoepidemiology study, our system requires stakeholders to choose a comparison group, eligibility and exclusion criteria, and exposure, follow-up, and outcome definitions. The assumptions made in each of these decisions will have bearing on the system’s performance. For example, choosing a truly unexposed comparison group, rather than a group exposed to an active comparator, allows stakeholders to assess the absolute safety of a medical product, but also increases the opportunity for false positive and false negative alerts due to increased confounding by indication (26).

Known limitations of electronic healthcare data pose challenges for broad application of Sentinel-like systems. Data on many types of adverse drug events, such as rashes, allergic reactions, and headaches, may not be fully captured in these data and the validity of codes used to identify other events, such as seizures that do not result in emergency care, may be questionable. Further, bias due to residual confounding is difficult to rule out since electronic healthcare data often lack information on potential confounding variables, including smoking status, body mass index, functional status, frailty, and over-the-counter drug use. If, for example, clinicians preferentially prescribed telithromycin to sicker patients, the resulting population imbalances might not be fully adjusted using health insurance claims data, leading to residual confounding and a false positive alert. That the system did not generate an alert in this example provides some reassurance that confounding has been adequately addressed in this case. Finally, many facets of prospective medical product monitoring require further exploration, methods development, and testing. However, as our four examples to date suggest, active drug safety monitoring grounded in sound epidemiologic theory and application and appropriate clinical rationale can produce valid results despite conjecture to the contrary (27).

In summary, using examples we demonstrated the viability of a sequential PS-matched cohort approach to active monitoring that integrates clinical, epidemiological, and other expert stakeholder input with several semi-automated processes. In broad-based safety surveillance systems, false positivity is a major concern. Our system previously generated timely alerts about a known association, did not generate alerts for a known null association, and did not produce alerts in the other two examples for which alerting would be questionable.

METHODS

Data sources

We used data from three different sources to mimic monitoring across Sentinel’s distributed data network: (1) the HealthCore Integrated Research Database (HIRD); (2) New Jersey Medicare Parts A and B data linked to the Pharmacy Assistance for the Aged and Disabled (PAAD) program; and (3) Pennsylvania Medicare data linked to the Pharmaceutical Assistance Contract for the Elderly (PACE) program. The HIRD contains longitudinal claims data comprising all filled prescriptions and clinical encounters from 14 Blue Cross and/or Blue Shield licensed health plans in the northeastern, southeastern, mid-Atlantic, mid-western, and western regions of the US. Starting in the third quarter of 2004, the amount of HIRD data increased substantially as data from more plans became available. Both PACE and PAAD provide medications at minimal expense to patients aged 65 and older with low income but who do not meet the Medicaid annual income threshold. The Medicare Parts A and B data include hospital discharge information and all fee-for-service charges, with vital status information from the Social Security Administration’s Death Master File. We included PACE- and PAAD-linked Medicare data only through the end of 2005.

The Brigham and Women's Hospital Institutional Review Board approved this study.

Sequential matched cohort monitoring framework

We replicated prospective monitoring as if new data became available on a quarterly basis. We divided each database into sequential data sets defined by claims occurring in each calendar quarter and queried each data set to identify all new users of each of the drugs of interest and of their active comparators. We applied eligibility and exclusion criteria for each example as described below (28). Within each data set, we constructed separate propensity score (PS) models and used the PS to match new users of the monitoring drugs to new users of the active comparator drugs in a 1:1 ratio. We used the 180 days preceding each patient’s date of drug initiation (index date) to identify variables for the PS models, which included a set of pre-defined potential confounders for each example (listed below) plus potential confounders identified using the high-dimensional PS (hdPS) algorithm (29) with its small sample option (v2, hdpharmacoepi.org; 30). In each model, we considered up to 100 empirically-identified potential confounders from each of three domains – procedure codes, diagnosis codes, and drugs used. Patients exposed to the monitoring drugs and the comparators could only be matched if their index dates occurred in the same quarterly data set.

Data pooling and follow-up

In each sequential data set, we identified outcomes for PS-matched patients who remained under follow-up, as defined below for each example. We combined the exposure, outcome, and follow-up time information for each calendar quarter across the three databases. This approach to pooling across a distributed data network enables multivariable-adjusted analyses while maintaining data privacy (7,8). These data elements served as inputs into the alerting algorithms. For example, inputs from the first period included the number of matched pairs with index dates in the first calendar quarter, the number of outcomes in each exposure group, and the total follow-up time in each group through the end of the first period. The algorithm inputs from the second period included the same corresponding data for new users with index dates in the second calendar quarter plus additional follow-up and outcome data for patients with index dates in the first period whose follow-up continued into the second period (Appendix Figure 1). Because relatively few patients use a new drug when it first enters the market, we combined the first two calendar quarters in which prescriptions for the new drug began to appear in the databases to create the first monitoring period in each example.

Algorithm selection and application

Alerting algorithms refer to sequential statistical monitoring approaches, such as group sequential monitoring methods and sequential probability ratio tests, that could be used for medical product safety monitoring. In a statistical simulation study, we found that the relative performance of algorithms, with respect to the accuracy and timeliness of alerting, in the setting of prospective safety monitoring varied substantially depending on the characteristics of specific scenarios (9). In the Appendix we provide the list of algorithms (Appendix Table) that we tested in the simulation study along with their relative performances under varying parameter constellations (Appendix Figure 2). The relative performance of the algorithms depended on event frequency and on user preference for identifying potential safety issues with high sensitivity versus high specificity. We used the results of the simulation study to select three algorithms for each example, based on three values for preference for sensitivity versus specificity that regulators may define (31). To select the algorithms, we estimated the expected number of events based on the number of exposed patients observed in the first two calendar quarters (assuming that utilization of the drug would increase in each period) and on literature estimates of the incidence of each outcome. The selected algorithms and their operating characteristics from the simulation study are presented in Table 1. Additional operating characteristics of the 93 algorithms tested in 600,000 simulated scenarios are available from the authors.

Table 1.

Selected algorithms and their operating characteristics from based on results of a prior simulation study^a

Example	wb	Algorithm description	Overall sensitivity (95% CI)	Overall specificity (95% CI)	Event-based sensitivity (95% CI)c	Event-based specificity (95% CI)d
Rosuvastatin & rhabdomyolysis	0.05	Pocock-like boundary (α = 0.10)	0.1974 (0.1876, 0.2073)	0.9987 (0.9975, 0.9998)	0.2419 (0.2389, 0.2448)	0.9994 (0.9983, 1.0000)
	0.10	Nominal Type 1 error (α = 0.10)	0.3626 (0.3507, 0.3746)	0.9772 (0.9724, 0.9819)	0.3844 (0.3810, 0.3878)	0.9857 (0.9799, 0.9914)
	0.15	Nominal Type 1 error (α = 0.10)	0.3626 (0.3507, 0.3746)	0.9772 (0.9724, 0.9819)	0.3844 (0.3810, 0.3878)	0.9857 (0.9799, 0.9914)
Rosuvastatin & diabetes mellitus	0.05	Exact p-value for period-specific estimate < 0.000032	0.7839 (0.7721, 0.7958)	0.9987 (0.9977, 0.9997)	0.8091 (0.8089, 0.8093)	0.9981 (0.9980, 0.9983)
	0.10	Pocock-like boundary (α = 0.000001)	0.9534 (0.9473, 0.9594)	0.9862 (0.9830, 0.9893)	0.8881 (0.8879, 0.8882)	0.9905 (0.9902, 0.908)
	0.15	Pocock-like boundary (α = 0.000001)	0.9534 (0.9473, 0.9594)	0.9862 (0.9830, 0.9893)	0.8881 (0.8879, 0.8882)	0.9905 (0.9902, 0.908)
Telithromycin & hepatotoxicity	0.05	Nominal Type 1 error (α = 0.01)	0.4360 (0.4236, 0.4483)	0.9984 (0.9972, 0.9997)	0.4826 (0.4809, 0.4844)	0.9991 (0.9983, 0.9998)
	0.10	O’Brien-Flemming-like boundary (α = 0.20)	0.5532 (0.5408, 0.5656)	0.9903 (0.9871, 0.9934)	0.5196 (0.5178, 0.5213)	0.9968 (0.9954, 0.9982)
	0.15	Nominal Type 1 error (α = 0.05)	0.6321 (0.6201, 0.6441)	0.9605 (0.9543, 0.9667)	0.5825 (0.5808, 0.5842)	0.9887 (0.9861, 0.9913)

^aListed are the algorithms that achieved highest event-based performance at three values of w (defined below) in a simulation study.⁶ For each example, the results were restricted to scenarios that resembled monitoring for the particular example.

^bw is a user-defined weight that reflects trade-offs between the costs of false negatives and false positives; smaller weights reflect higher relative costs of false positives

^cEvent-based sensitivity is the proportion of observed exposed events in alert-worthy scenarios (i.e. scenarios in which a safety issue of interest exists; where the true underlying risk ratio ≥ the alerting threshold) that occurred after the given algorithm generated an alert

^dEvent-based specificity is the proportion of observed exposed events in non-alert-worthy scenarios (i.e. scenarios in which no safety issue of interest exists; where the true underlying risk ratio < the alerting threshold) that occurred before or in the absence of an alert by the given algorithm

Specifics of each example

Rosuvastatin and rhabdomyolysis and diabetes mellitus

We selected atorvastatin as an active comparator because it was available on the market before the introduction of cerivastatin and it has low risk of rhabdomyolysis (32). We defined new users of rosuvastatin or atorvastatin as no use of any statin in the 180 days preceding the index date. We excluded patients with evidence of diabetes, myopathy, renal dysfunction, or liver disease prior to the index date. We used the same sequential PS-matched cohorts for both rhabdomyolysis and diabetes mellitus outcomes. In addition to age and sex, we included in the PS models risk factors for rhabdomyolysis including hypothyroidism and use of drugs that either cause or interact with statins to cause rhabdomyolysis (33) plus the following diabetes risk factors: coronary artery disease, congestive heart failure, hemorrhagic stroke, hypertension, dyslipidemia, peripheral vascular disease, and use of angiotensin converting enzyme inhibitors, beta-blockers, non-statin cholesterol-lowering drugs, ticlopidine, clopidogrel, and nitrates.

We followed patients for rhabdomyolysis from their index dates until they experienced the event, discontinued their index treatment (as defined by a gap in treatment of greater than 14 days), switched to a different statin, died, or disenrolled. We defined rhabdomyolysis using the algorithm for claims data validated by Andrade et al, which had a positive predictive value of 74% in a network of managed care organization databases (34). For diabetes monitoring, we followed patients until they received a diagnosis of diabetes, discontinued their index treatment (plus a 60-day grace period added to the end of the days supply of the last prescription), switched to a different statin, died, or disenrolled. We defined diabetes using a validated algorithm for claims data, which has been found to have a positive predictive value of 97% (35).

Telithromycin and hepatotoxicity

We defined new use of telithromycin or azithromycin – an active comparator that shares similar indications as telithromycin, was available at the time of telithromycin approval, and was commonly used – as no prior use of either drug in the 180 days preceding the index date. We excluded patients with evidence of hepatic injury or impairment, including those with diagnoses for alcoholism. In addition to age and sex, we included codes for the following potential risk factors for liver injury as pre-defined covariates in the PS models (36–38): diabetes, illicit drug use, renal dysfunction, and drugs with potential for liver damage, defined as those considered category three hepatotoxic (i.e. clear literature evidence of life-threatening hepatotoxicity or death) in at least one out of five drug compendia as reported by Guo et al (39). We followed patients until they experienced acute hepatotoxicity, died, or for a maximum of 60 days. We defined hepatotoxicity as the occurrence of a diagnostic or procedural code indicating acute liver injury. These codes have been validated in the context of acetaminophen-induced hepatotoxicity, with a positive predictive value of 78% (40).