Near real-time adverse drug reaction surveillance within population-based health networks: methodology considerations for data accrual

Abstract

Purpose

This study describes practical considerations for implementation of near real-time medical product safety surveillance in a distributed health data network.

Methods

We conducted pilot active safety surveillance comparing generic divalproex sodium to historical branded product at 4 health plans from April – October 2009. Outcomes reported are all-cause emergency room (ER) visits and fractures. One retrospective data extract was completed (1/2002–6/2008), followed by seven prospective monthly extracts (1/2008–11/2009). To evaluate delays in claims processing, we used three analytic approaches: near real-time sequential analysis, sequential analysis with 1.5 month delay, and nonsequential (using final retrospective data). Sequential analyses used the maximized sequential probability ratio test. Procedural and logistical barriers to active surveillance were documented.

Results

We identified 6,586 new users of generic divalproex sodium and 43,960 new users of the branded product. Quality control methods identified 16 extract errors, which were corrected. Near real-time extracts captured 87.5% of ER visits and 50.0% of fractures, which improved to 98.3% and 68.7% respectively with 1.5 month delay. We did not identify signals for either outcome regardless of extract timeframe; slight differences in the test statistic and relative risk estimates were found.

Conclusions

Near real-time sequential safety surveillance is feasible, but several barriers warrant attention. Data quality review of each data extract was necessary. Although signal detection was not affected by delay in analysis, when using a historical control group differential accrual between exposure and outcomes may theoretically bias near real-time risk estimates towards the null, causing failure to detect a signal.

BACKGROUND

Recent high profile medical product safety withdrawals have heightened attention to post-market safety surveillance systems.^1–3 Such surveillance has traditionally relied on passive surveillance reporting systems such as the Adverse Event Report System (AERS) in the United States.^{4, 5} While voluntary reporting systems can contribute vital safety information, there are limitations to passive surveillance including lack of control populations, reporting bias, and unknown population size at risk.^{6, 7}

The 2007 Food and Drug Administration (FDA) Amendments Act mandated the establishment of active medical product safety surveillance. The subsequent launch of the FDA's Sentinel Initiative demonstrates the importance of developing active safety surveillance methodology to complement current systems.^{8, 9} Active safety surveillance calls for prospective monitoring of medical care utilization from health system participants such as health insurers and delivery systems. This approach relies on frequent extracts of medical utilization and medical product exposure data, with emphasis on using the most current data possible, i.e. near real-time. However, real-time data collection is susceptible to delayed, incomplete, and erroneous data capture caused by omissions during documentation at the time of service delivery or delays in the processing of claims within health insurers related to the adjudication process.^10–12 The impact of data capture issues on prospective safety surveillance systems is not fully understood. Further, many active surveillance systems combine data across institutions – a distributed data approach – that further complicates surveillance by introducing potential differences across data partners and the need to coordinate across partners.^13,14 Active prospective surveillance in a distributed environment requires best practices for coordinating data extracts across data partners, assessing quality of data extracts, and addressing delays in data availability.

Prospective surveillance of vaccine safety is routinely conducted by the Vaccine Safety DataLink, but prospective surveillance of drug safety is not yet routine.^10,15–17 Although prior studies have explored the feasibility of prospective drug surveillance by using retrospective data to mimic prospective surveillance, none have attempted near real-time prospective drug safety surveillance.^16–18 The primary purpose of this study was to implement near real-time prospective drug surveillance in a distributed data environment in order to assess barriers to implementation and examine the impact of delays in the processing of healthcare data.

METHODS

Overview

We conducted a prospective pilot study using sequential analysis techniques to assess the safety of generic divalproex sodium compared to the branded product. We conducted 7 monthly data extracts across 4 data partners to assess several potential adverse events; results for two of the 6 outcomes evaluated are presented in full as worked examples. Sequential analyses used the maximized sequential probability ratio test (MaxSPRT) to compare observed and expected counts of adverse events among new users of the generic and branded divalproex sodium products.

Selection of Generic Divalproex Sodium

Divalproex sodium was selected for this pilot due to high volume of use and concerns about the safety of generic anticonvulsant medications that might result in poor efficacy and a higher rate of adverse events.^19–22

Study Population

The study population consisted of all members from four health plans within the HMO Research Network's (HMORN's) Center for Education and Research on Therapeutics: Group Health, Seattle, WA; Harvard Pilgrim Health Care, Boston, MA; Kaiser Permanente Northern California, Oakland, CA; and Kaiser Permanente Southern California, Downey, CA. These plans provide coverage for about 8 million people and maintain electronic demographic, enrollment, medical care utilization, and pharmacy dispensing data.

Incident Exposure

Exposure to branded and generic divalproex sodium was assessed using National Drug Codes (NDCs) reported on outpatient pharmacy dispensing records. The exposed generic cohort included new users of generic divalproex sodium from the generic approval date of July 31, 2008 to the final extract in November 2009. These incident generic users had no exposure to any divalproex product for 6-months before the index dispensing date. Therefore, patients who switched from brand to generic exposure were excluded.

The historical control group consisted of new users of branded divalproex from January 2002 through June 2008. Incident branded users were defined as users of branded divalproex sodium who had no exposure to any divalproex product for 6-months before the index date.

Contributed Time

Only exposed time during health plan enrollment was included. Enrollment gaps of 45 days or less were bridged to create continuous enrollment periods. Treatment episodes were created using the date of dispensing and the number of days supplied; exposure gaps of 62 days or less between end of days supplied and subsequent dispensing were bridged to create continuous exposure periods. Each outcome was analyzed independently, therefore contributed and exposed time was calculated for each outcome separately. Contributed time ended at the completion of exposure, termination of enrollment, occurrence of an outcome, or 12 months of exposure, whichever came first. Surveillance was limited to 12 months to focus on shorter term adverse effects.

Incident Outcome Definition

We selected two outcomes for presentation, all-cause emergency room (ER) visits and fractures observed in the inpatient or emergency department setting as a proxy for falls related to seizure or dizziness. Fractures were identified using International Classification of Diseases (ICD-9) codes (see Appendix A). ER visits and fractures are presented because they were the most common overall and clinical outcomes, respectively. The analysis was limited to incident outcomes; and therefore members with a fracture or ER visit within 6 months before the index date were excluded from analysis for that outcome. Members could contribute to both outcomes independently.

Additional outcomes included in the pilot, but not described here, were inpatient hospitalization, pancreatitis, acute liver condition, and thrombocytopenia. Results for three outcomes were too sparse to yield meaningful results with respect to the data accrual and delay analysis; details can be viewed in online Appendices B and C.

Data Source and Extraction

Each data extract was obtained from the partner's most current internal data source and was transformed locally to a common data model – the HMORN Virtual Data Warehouse (VDW).^23–25 The internal data source at each site is populated by various internal and external data streams, including ambulatory electronic medical records, outpatient pharmacy dispensings, and claims for medical care reimbursement. These data streams are subject to administrative quality assurance and adjudication processes.

Each health plan completed one retrospective extract covering January 2002 to June 2008, followed by 7 monthly extracts from May to November 2009. The median day of extract was the 15^th of the month (Inter quartile range 9–18). Each monthly extract pulled data beginning from January 2008 through the date of extract and included health plan enrollment (start and stop dates), demographics (date of birth, sex), outpatient pharmacy dispensing (NDC, dispense date, days supplied), and medical care utilization (diagnoses, procedures, and visits). These datasets were created and maintained by each data partner. Following a distributed data model approach, analytic programs were created by the study coordinating center and distributed to each partner for local execution. Summary files containing stratified count data were returned for analysis.^{13, 14}

Each data extract included instructions for the creation and naming of data files, including population definition, extract timeframe, data dictionaries, and a list of NDCs for divalproex sodium. A new NDC list was generated for each extract to account for newly-introduced versions of the generic product. The work plans also included a quality control program.

The quality control reports examined monthly trends in patients, dispensings, diagnoses, age, sex, dates of service, and enrollment dates. Data were compared across sites and between extracts for each site. The coordinating center created summary quality control reports; data anomalies were reviewed by the site investigator and analyst for explanation or correction.

Calculating Expected Outcomes

Monthly expected outcomes were derived from indirect standardization by applying the rate of outcomes in the historical control group to the number of generic exposed days for each month. For covariate adjustment, rates were calculated for specific strata: sex (male and female), age group (10–14, 15–24, 25–44,45–54, ≥55 years), health plan (4 sites), and the number of months of drug exposure grouped into 3 categories (1–3, 4–6, 7–12 months). After these strata specific rates were applied to their corresponding strata for the generic exposed days, the total expected count was summed to obtain the final monthly expected count. The relative risk was calculated as the cumulative observed outcomes over cumulative expected for each month.

Statistical Analysis

The Poisson maximized MaxSPRT^{16, 26, 27} was used to test for a signal of excess risk by comparing outcomes among new users of the generic and branded products. MaxSPRT uses a log-likelihood ratio (LLR) test statistic with a critical value adjusted for multiple testing. The critical value is specified by the alpha level and by an upper limit on the length of surveillance, defined in terms of the expected count under the null. For this study alpha was set at 0.05 and the upper limit was set separately for each outcome based on the expected number of observed events; the upper limit was set to 700 expected cases under the null hypothesis for ER visits and 20 for fracture.

Assessment of Data Accrual Delay and Barriers to Near Real-time Data Extraction

To evaluate the impact of data accrual delays in prospective surveillance the analysis was limited to active prospective data collection conducted during the analysis period (April 2009 – October 2009). The sequential analysis was conducted using two different data extraction timeframes: 1) near real-time, with data extracted about 2 weeks after the end of the month, and 2) with a 1.5 month delay. As an example, for the April 2009 data, near real-time analysis represents data extracted in May, while the 1.5 month delay shows April data extracted in June. The sequential cumulative number of exposure days and outcomes are compared to the observed values from the final extract from November, which represented the most complete data available. All analyses used SAS (version 9.2) and Microsoft Excel 2007. This study was approved by Institutional Review Boards at all participating sites.

RESULTS

There were 43,960 new branded users identified from January 2002 to October 2009 and 6,586 new generic users identified from August 2008 through October 2009. The transition in dispensings from branded to generic product occurred within a few months (Figure 1). The two cohorts had similar mean age (41.6 years for brand vs. 42.4 for generic) and sex distributions (52.2% female brand vs. 56.3% for generic). Branded users had a longer exposure time (mean (range): 111.0 (1–367) vs. 68.0 (1–316) days), at least partly due to the shorter surveillance period for the generic product.

Figure 1.

Number of people with brand or generic divalproex sodium dispensing per month from January 2002 to October 2009.

Exposure and Outcome Accrual

Data accrual delay was observed for exposures and outcomes (Tables 1a & 1b). Allowing additional time between the observed month and the extract resulted in higher capture rates: for example of the 96 ER visits that occurred in June 2009 according to the final extract in November, 80.2% were observed in the July extract, increasing to 89.6% in August and 100.0% in September. Of note, the August extract of July data had particularly low capture (40.5% of exposed days and 33.7% of ER events.) This difference may be explained by the data extract dates for August, between the 4^th and 10^th, earlier than the median extract date of the 15^th. The number of observed events or exposed days in any specific month can exceed the final values due to health plan data adjudication.

Table 1a.

Number of days exposed to generic divalproex sodium for emergency room visit analysis and the percentage of the total observed days according to the final extract.

	Month Data Extracted
	MAY 2009		JUNE		JULY		AUG		SEP		OCT		Final (NOV)
Observed Month	n	%	n	%	n	%	n	%	n	%	n	%	n
AUG 2008	4,989	100.0	4,989	100.0	4,989	100.0	4,989	100.0	4,989	100.0	4,989	100.0	4,989
SEP	13,927	100.0	13,927	100.0	13,927	100.0	13,927	100.0	13,927	100.0	13,927	100.0	13,927
OCT	23,925	100.0	23,925	100.0	23,925	100.0	23,925	100.0	23,925	100.0	23,925	100.0	23,925
NOV	29,751	100.0	29,751	100.0	29,751	100.0	29,751	100.0	29,751	100.0	29,751	100.0	29,751
DEC	35,484	100.0	35,484	100.0	35,484	100.0	35,499	100.0	35,499	100.0	35,499	100.0	35,499
JAN 2009	37,122	100.1	37,074	100.0	37,074	100.0	37,094	100.0	37,083	100.0	37,077	100.0	37,083
FEB	36,777	100.2	36,714	100.1	36,714	100.1	36,710	100.1	36,687	100.0	36,670	100.0	36,687
MAR	43,513	100.4	43,397	100.1	43,397	100.1	43,366	100.0	43,313	99.9	43,349	100.0	43,349
APR	41,910	92.0	45,671	100.2	45,696	100.3	45,636	100.2	45,588	100.0	45,566	100.0	45,566
MAY			51,210	100.9	51,023	100.6	50,926	100.4	50,785	100.1	50,706	99.9	50,734
JUN					45,638	91.5	45,351	90.9	49,688	99.6	49,626	99.5	49,882
JUL							21,467	40.5*	52,892	99.8	52,965	99.9	53,019
AUG									53,849	100.1	53,743	99.9	53,804
SEP											51,295	99.9	51,372
OCT													53,821

^*Note: The lower capture rate in August may be due to early extract dates at the sites which occurred between the 4^th and 10^th rather than around the median extract date of the 15^th.

Table 1b.

Number of emergency room visits for people exposed to divalproex sodium and the percentage of the total observed events according to the final extract.

	Month Data Extracted
	MAY 2009		JUNE		JULY		AUG		SEP		OCT		Final (NOV)
Observed Month	n	%	n	%	n	%	n	%	n	%	n	%	n
AUG 2008	31	100.0	31	100.0	31	100.0	31	100.0	31	100.0	31	100.0	31
SEP	68	100.0	68	100.0	68	100.0	68	100.0	68	100.0	68	100.0	68
OCT	86	100.0	86	100.0	86	100.0	86	100.0	86	100.0	86	100.0	86
NOV	91	100.0	91	100.0	91	100.0	91	100.0	91	100.0	91	100.0	91
DEC	83	98.8	84	100.0	84	100.0	84	100.0	84	100.0	84	100.0	84
JAN 2009	93	97.9	93	97.9	93	97.9	94	98.9	95	100.0	95	100.0	95
FEB	86	97.7	86	97.7	86	97.7	87	98.9	88	100.0	88	100.0	88
MAR	89	95.7	94	101.1	94	101.1	94	101.1	94	101.1	93	100.0	93
APR	77	74.8	101	98.1	101	98.1	102	99.0	102	99.0	103	100.0	103
MAY			87	80.6	97	89.8	98	90.7	108	100.0	108	100.0	108
JUN					77	80.2	86	89.6	96	100.0	96	100.0	96
JUL							33	33.7*	98	100.0	97	99.0	98
AUG									90	91.8	94	95.9	98
SEP											97	94.2	103
OCT													102

^*Note: The lower capture rate in August may be due to early extract dates at the sites which occurred between 4^th and 10^th rather than around the median extract date of other months of 15^th.

For ER visits, from August 2008 to October 2009 there was an average of 89.6 events per month. Near real-time data extraction produced a mean capture rate of 87.5% for exposures and 75.9% for outcomes. A 1.5 month delay increased mean capture rates to 98.3% for exposed days and 94.7% for ER visits. There were about 1.7 fractures observed per month. As seen in ER visits, the 1.5 month delay resulted in improvement of mean capture rates compared to the near real-time extracts, from 50.0% to 68.8% for fractures, excluding one month with greater than 100% capture. A 2.5 month delay brought the mean capture rate of fractures to 93.8%.

Outcomes

Evaluation of the final extract (November 2009) did not identify any signals of excess risk. A total of 708 ER visits were observed over 358,198 exposed days resulting in a relative risk of 1.05 and a log-likelihood of 0.6963. In the fracture analysis, 17 outcomes were detected over 412,820 exposed days with a relative risk of 0.88 and a log-likelihood of 0.

To assess the impact of differential data capture over time, we conducted sequential MaxSPRT analyses for each timeframe of data extraction: near real-time and 1.5 month delay (Tables 2a & 2b). Due to the variability in exposure and outcome counts caused by data accrual lag, there were slight differences in the relative risk and log-likelihood ratio in the two analyses. Despite these differences, there was no impact on signal detection as the log-likelihood never approached the critical value. No signals of excess risk were observed for the outcomes not reported (see appendices).

Table 2a.

Emergency room visit sequential analysis by time of extract using maximized sequential probability ratio test (MaxSPRT)

	Near Real-Time Data Extraction					1.5 Month Delay Data Extraction					Final Pull (Nov 2009)α
Observed	Cumulative					Cumulative
Month	Exposed Days	Outcomes	Expected	RR	LLRβ	Exposed Days	Outcomes	Expected	RR	LLRβ	RR (95% confidence interval)	LLRβ
APR 2009	41,910	77	85.1	0.91	0	45,671	101	90.4	1.12	0.599
MAY	96,881	188	191.5	0.98	0	96,719	198	191.0	1.04	0.126
JUN	142,357	275	279.9	0.98	0	141,913	286	279.1	1.02	0.084
JUL	163,380	319	323.5	0.99	0	198,953	404	383.5	1.05	0.536
AUG	252,802	494	484.2	1.02	0.098	252,606	498	483.4	1.03	0.217
SEP	303,901	595	577.3	1.03	0.267	304,377	606	578.7	1.05	0.634
OCT	358,198	708	677.1	1.05	0.696	-	-	-	-	-	1.05 (0.97–1.12)	0.696

Note: RR - Relative Risk; LLR - Log-likelihood Ratio

^αNonsequential analysis using final pull.

^βSignal criteria for .05 alpha level of significance is a log-likelihood ratio of 4.3.

Table 2b.

Fracture sequential analysis by time of extract using maximized sequential probability ratio test (MaxSPRT)

	Near Real-Time Data Extraction					1.5 Month Delay Data Extraction					Final Pull (Nov 2009)α
Observed	Cumulative					Cumulative
Month	Exposed Days	Outcomes	Expected	RR	LLRβ	Exposed Days	Outcomes	Expected	RR	LLRβ	RR (95% confidence interval)	LLRβ
APR 2009	48,635	0	2.1	0	0	52,516	1	2.4	0.41	0
MAY	111,096	3	5.1	0.59	0	111,147	4	5.1	0.78	0
JUN	164,205	4	7.3	0.54	0	163,941	4	7.4	0.54	0
JUL	189,766	4	8.5	0.47	0	229,523	11	10.7	1.03	0.005
AUG	291,278	14	13.6	1.03	0.005	291,158	14	13.6	1.03	0.005
SEP	350,298	15	16.3	0.92	0	350,882	15	16.4	0.92	0
OCT	412,820	17	19.2	0.88	0	-	-	-	-	-	0.88 (0.46–1.31)	0

Note: RR - Relative Risk; LLR - Log-likelihood Ratio

^αNonsequential analysis using final pull.

^βSignal criteria for .05 alpha level of significance is a log-likelihood ratio of 3.7.

Sequential Data Extracts and Quality Control

Of the 28 planned monthly extracts, four were not executed on time and resulted in a skipped extract. Reasons for skipped extracts, which occurred at two sites, included lack of analyst availability, staff turnover, and routine data management upgrades.

Our monthly quality control analyses identified 16 data extraction errors that required corrections. Half occurred in the first historical extract, with the other half distributed amongst the 7 monthly extracts. These corrections were completed on the data as initially pulled and therefore did not impact the timing of the data extraction; however the corrections did delay analysis. Errors in the historical extract included: use of the wrong calendar window, ER visits coded as outpatient visits, and individuals in the procedure file who were not in the demographic file. Errors in the monthly extractions included: different numbers of individuals in demographic and pharmacy data, dramatic increases in number of divalproex users (more than double) caused by extracting data from wrong timeframe (January 2000–June 2009 vs. January 2008–June 2009), and a drop in the average number of enrollment intervals caused by a missed update to system-wide files.

In addition, several cross-site system-level data capture differences were identified. Two sites had fewer observed medical procedures per patient than the other sites, and at one site there was a substantial drop in observed procedures caused by a change in electronic medical record workflow. In each case the local analyst was able to quickly investigate and determine whether the anomaly was an extraction error or a true reflection of sites' data with differences attributable to variation in clinical workflow or coding practices.

DISCUSSION

We successfully implemented an active medical product safety surveillance demonstration study with 4 data partners. Using the most recent data for active surveillance proved challenging due to the data checking needed for each data extract and variable data accrual. Near real-time data capture had sufficiently stabilized data to conduct prospective sequential analyses, although waiting an additional month achieved nearly complete data capture for ER visits, but only about 70% capture for fracture. These findings suggest that planning the timeframe for data extraction and analysis may need to be customized to the outcome under review.

Overall, our assessment showed that analyses conducted in near real-time or with a 1.5 month delay can generate different rates and test statistics, but in this case did not impact signal detection. Incomplete data accrual and differential data capture between exposure and outcomes could bias results. The direction of bias is likely dependent on whether the control group is concurrent or historical. Using a historical control group means that all observable events and exposures have been captured. Since our data show that prospective data collection for the intervention group has a lower capture rate of outcomes compared to exposure, the use of historical controls will likely bias towards the null hypothesis. This type of bias will decrease with increasing length of surveillance since the proportion of unreported outcomes will decrease as the earlier data settles. Concurrent control groups could theoretically introduce bias towards or away from the null hypothesis, depending on whether data capture rates were differential between the two groups. Also, we note that the study of a generic drug is a special case that lends itself to a specific comparator (i.e.;, the branded product). Surveillance of newly approved drugs can make use of historical or concurrent active comparators, such as a drug in the same class with the same indication, as selected by the investigator. A self-control design could also be used for certain drug-outcome pairs.

Study limitations include those common to observational data, such as the use of diagnosis codes to identify adverse events and identification of exposure through dispensing records. In addition, s study outcomes were not validated with chart review. Also, our data partners have substantial experience implementing this type of distributed research; including a broader group of less experienced partners may result in more data quality issues and implementation challenges. Another limitation relates to the use of historical controls. It is possible that the characteristics of divalproex sodium users changed over time, due to changing clinical practices or new treatments entering the market. This may have resulted in differences in the clinical characteristics and disease severity between the study and control groups, and those differences may have changed over time. Finally, the exclusion of several extracts due to extraction errors may have overestimated the data accrual lag for certain months. However, the excluded extracts provided valuable information about the challenges of implementing monthly surveillance and accounted for less than 20% of all data.

The primary purpose of prospective drug safety surveillance is to rapidly identify potential safety issues that warrant further review. Research teams developing surveillance systems must grapple with the tension between the need for timely identification of signals versus the delay of analysis to decrease potential bias. For analyses with historical control group, there are several approaches to avoiding data accrual bias. One simple approach is to delay analysis to allow data to stabilize. We found a delay of 1.5 to 3.5 months would result in data capture of at least 90%. Alternatively, investigators could use sensitivity analyses to assess the potential impact of delays. Simple sensitivity analyses do not require new analytic approaches or assumptions and answer questions often posed by policy makers and stakeholders. Finally, some suggest formal modeling of the data accrual delay and adjusting the observed and expected counts accordingly.^{10, 15} This method assumes that the model, which is based on prior data accrual experience in a specific data system(s), can be applied in other scenarios. The approach taken to address data accrual delays will be based on the needs and characteristics of the specific study.

In conclusion, near real-time medical product safety surveillance in a distributed environment is possible. Implementation barriers include data accrual delays, the need for quality control reports for each data extract, and the potential for local data system changes to impact extracts. We found that the use of quality control reports for each extract was vital to the discovery and documentation of both human error and system changes. As the FDA progresses toward active prospective safety surveillance through the Sentinel Initiative,⁸ this study provides timely evidence that monthly surveillance is feasible and offers practical suggestions for the improvement of surveillance methodology.

Key Points

• Near real-time medical product safety surveillance can be accomplished

• Data accrual delay should be addressed to account for differential data capture

• Procedural and pragmatic barriers to frequent extracts require considerable attention and effort