Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2012 May 1.
Published in final edited form as: Contemp Clin Trials. 2011 Feb 26;32(3):309–317. doi: 10.1016/j.cct.2011.02.001

An Evaluation of a Distributed Medical Device Safety Surveillance System: The DELTA Network Study

Venkatesan D Vidi a, Michael E Matheny b,c,d, Sharon Donnelly a, Frederic S Resnic a
PMCID: PMC3070041  NIHMSID: NIHMS279670  PMID: 21356331

Abstract

Background

Post-market medical product safety surveillance is a complex task requiring standardized data collection, prompt adverse event reporting mechanisms and appropriate methodologies to identify low frequency safety threats and risk communication.

Purpose

To review the design of the DELTA (Data Extraction and Longitudinal Trend Analysis) network study of the medical device safety surveillance.

Methods

This is a multicenter prospective observational study designed to evaluate the safety of new cardiovascular devices used during percutaneous coronary intervention (PCI) performed through continuous analysis of the routinely collected American College of Cardiology- National Cardiovascular Data Registry (ACC-NCDR) data elements. The primary endpoint of the study is detection of adverse event rates specific to several classes of new medical devices, including drug eluting coronary stents, embolic protection devices, and vascular closure devices in patients undergoing PCI. Secondary endpoints include the time-savings between the DELTA network detection of a true device safety alert and the time taken to detect the same outcome using conventional retrospective data analysis, overall sensitivity, specificity, positive predictive value and negative predictive value of the DELTA network surveillance system.

Results

The details of the study are described including system design, eligibility criteria, methods and components of data collection, data security and statistical methods. In addition, the methods of adjudication and verification following an adverse event alert, overall study outcomes, end points, limitations and potential advantages are discussed.

Conclusion

This report describes the first multicenter prospective study of a computerized safety surveillance system to monitor and evaluate the safety of new cardiovascular devices.

Keywords: Post-market surveillance, Safety surveillance, Network study, Interventional cardiology

INTRODUCTION

Post-market safety surveillance of medical devices is a complex task compounded by rapid dissemination of new medical technology, lack of standards in data collection, current lack of unique device identifier information and inadequate adverse event reporting mechanisms. Current sources of nationwide data for post-market surveillance include the medical device reporting (MDR) database, the manufacturer and user facility device experience database (MAUDE) and other device databases/registries[14], which use data collected retrospectively after the occurrence of an adverse event. Each such source has strengths and weaknesses for assessing different aspects of real-world device use, performance, and safety. Most existing safety surveillance systems rely on passive reporting of events in a retrospective manner[5], limiting the interpretability and timeliness of safety analyses. Building an effective prospective medical device safety surveillance system is complicated by limitations of temporal availability of clinical datasets and lack of consensus regarding the most appropriate methodologies to be used to identify low frequency safety threats.

We have developed a unique computerized automated safety surveillance tool, DELTA (Data Extraction and Longitudinal Trend Analysis system), that prospectively monitors the adverse event rates of new medical products through the continuous surveillance of existing clinical databases using a variety of statistical monitoring strategies. [6]. We have tested and validated DELTA on a large clinical database[7] as well as on randomized clinical trial data in which significant safety issues were identified. [8, 9] The purpose of this study is to find out the adverse event rates specific to several classes of new invasive cardiovascular devices (drug-eluting stents, vascular devices and embolic protection devices) using the DELTA safety monitoring system in a multi-center environment.

MATERIAL AND METHODS

Objective

The primary objective of the study is to detect the adverse event rates specific to several classes of new medical devices, including drug eluting coronary stents, embolic protection devices, and vascular closure devices (Table 1) in patients undergoing percutaneous coronary interventions (PCI). The secondary objective of the study is to find out the time-savings between the DELTA network detection of a true device safety alert and the time taken to detect the same outcome using conventional retrospective safety monitoring methods used by the Massachusetts Department of Public Health (DPH), sensitivity, specificity, positive predictive value and negative predictive value of the DELTA network.

Table 1.

Clinical Outcomes for adverse event detection (Primary Endpoints)

Type of Device Clinical Outcomes of Interest
1 Drug eluting stent In-hospital death
Peri-procedural myocardial infarction (in-hospital)
Major adverse cardiac events: Composite of in- hospital death, peri-procedural myocardial infarction, stroke or unplanned coronary revascularization.
2 Vascular closure device Major vascular complications (including retroperitoneal hemorrhage, limb ischemia and any surgical/interventional repair)
Minor vascular complications (including groin bleeding, hematoma, pseudo aneurysm and arteriovenous fistula)
All vascular complications (in-hospital)
3 Embolic Protection Device In-hospital death
Peri-procedural myocardial infarction (in-hospital)
Major adverse cardiac events: Composite of in- hospital death, peri-procedural myocardial infarction, stroke or unplanned coronary revascularization.

Non Device related Continuous Outcomes of Interest

1 Quantitative change in serum creatinine (mg/dl) in high-risk patient subgroups including diabetics, patients with pre-existing renal dysfunction (estimated glomerular filtration rate < 50ml/min), procedures with contrast volume >300cc, and patients undergoing multivessel procedures in which >3 stents are deployed.
2 Quantitative change in post-procedure creatinine kinase MB fraction (CK-MB, IU/L) in patients at increased risk for peri-procedural myocardial infarction. Patient subsets will include those with stenting at a side branch, treatment of unstable angina, treatment of saphenous vein grafts, and use of adjunctive atherectomy or thrombectomy devices.

Application

DELTA was developed to provide prospective outcomes monitoring for any clinical data source that could be imported in regular intervals. The tool uses a web-based graphical user interface developed in Microsoft.NET (Microsoft, Redmond, VA), and stores data and algorithms in a SQL 2005 server. (Microsoft, Redmond, VA). DELTA allows the user to specify a desired confidence interval to generate an alerting threshold, and to select the time interval for analysis. When the application detects an elevated outcome rate for a given exposure, alerts are generated and emailed to the designated researcher. Full details of the specifications and design of the application are available elsewhere. [6]

System Design

The DELTA multi-center study will rely upon a distributed network of identical secure computerized safety surveillance systems at five participating centers in Massachusetts. The participating centers will include a range of healthcare environments from community hospitals to tertiary care teaching hospitals. The DELTA network system will be implemented as a two-tiered network of secure database servers (Figure 1), with remote DELTA “agents” (agent) deployed in five independent interventional cardiac catheterization laboratories and a secure central DELTA server (the “server”) located in a Partners Healthcare Systems secure research server cluster. The agents are responsible for de-identifying and packaging the clinical data from the local source outcomes database for use by the server for live prospective safety analyses and for receiving communication regarding the safety alerts and other messages from the central server. (See “Data Security and De-identification”)

Figure 1.

Figure 1

Figure representing the multi-center DELTA network implementation. (Only 3 of the 5 participating centers are illustrated for clarity)

DB = Database, DELTA = Data Extraction and Longitudinal Trend Analysis, SMTP = Simple Mail Transfer Protocol, SSL = Secure Socket Layer.

The fully de-identified data will be transmitted and stored in the DELTA server. Data in the DELTA server will then be available for comprehensive analysis and prospective safety monitoring. The local data for each specific institution is made available to users from that participating hospital for site-based DELTA analysis. In addition, the fully de-identified data aggregated from all the hospitals will be also be available to each institution for the purpose of local quality assurance and benchmarking. Although each client system will have access to the pooled cohort data on the central server to allow site-specific summary, the central server will not send the summaries to the sites. Data transmission will comply with all relevant provisions for secure information transfer under the Health Insurance Portability and Accountability Act (HIPAA). [10]

In addition to full de-identification of the patient, the central analysis server will be blinded to the origination site (participating center), and performing physician from all levels of user to preserve anonymity of the submitting center. The DELTA system will be integrated with a simple mail transfer protocol server to allow for automated e-mail notification of the appropriate analyst (including participating site) when the DELTA server issues a safety alert (Figure 1).

The study design relies upon the secondary use of coded clinical data routinely collected through the modified American College of Cardiology-National Cardiovascular Database Registry (ACC-NCDR) data collection tool at each participating site, which is used for mandatory submission to the DPH. [11, 12] Each hospital currently generates quarterly data submissions in a standard format for DPH quality assurance purposes. The current study will use the same format of this data submission, with data updated and generated on a monthly basis to minimize any need for customization of the local database interface at the participating sites.

Eligibility Criteria

In order to participate, hospitals must perform PCI during the study period and must participate in the state mandated reporting of the ACC-NCDR dataset to the Massachusetts Data Analysis Center (Mass-DAC) and to the ACC. Participating sites must agree to conduct structured clinical chart audits according to the DELTA chart auditing protocol (See under Data and Clinical Outcomes Audit). In addition, the participating sites must have ACC-NCDR submission software capable of submitting PCI dataset files, in the Mass-DAC format, electronically on a monthly basis. All patients (aged 18 or above) undergoing PCI during the study period will be eligible for inclusion in the study.

Data Collection for Prospective Monitoring

Candidate subjects include all patients (aged 18 or above) undergoing percutaneous coronary intervention (PCI) from January 2010 through December 2011. Retrospective data from the same sources will be collected for PCI’s performed between January 2008 and December 2009. Retrospective data which has not been previously available for studying device specific safety signals will be pooled with the prospectively collected data to provide increased specific device exposure sample size for performing propensity matched concurrent control analyses. Data elements will include all of those contained within the ACC-NCDR data collection tool. Each site is required to obtain local institutional review board (IRB) approval including a waiver for consent of patients on the basis of research use of clinical data, with low risk to individual patients through the secondary use of their clinical data. Data will be submitted to the DELTA server at least monthly and will include all new cases as well as any data updates to cases previously submitted. More frequent submission schedules to the DELTA server are permitted. The server will be connected via secure (https) internet connection to each local DELTA agent using SSL 128-bit encryption. The agents will request a unique identifying number for the patient and the case from the DELTA server, and the DELTA agent will maintain the map between the DELTA identifiers and the actual data. The fully de-identified data will then be transmitted and stored in the DELTA server such that it becomes available for comprehensive pooled multicenter analysis for study participants.

Traditional Methods of Monitoring by the DPH

The Massachusetts DPH utilizes an iterative approach to chart review when analyzing the statewide PCI data. Participating hospitals collect patient specific risk factor and outcome data using the ACC-NCDR data collection tool. Data is submitted by the hospitals to the Mass-DAC on a quarterly basis. Once received, Mass-DAC analyzes the data for inconsistencies and missing information. Data quality reports are generated and sent back to the hospitals who must resubmit their data until their data quality report is clean. Hospitals are allowed to resubmit data as many times as needed until the close of the data year which occurs annually in April. After the close of the year, Mass-DAC convenes a panel of interventional cardiologists and data managers who conduct chart reviews on all patients who have been reported as experiencing a complication, a death, or who are considered high risk for instance, those with cardiogenic shock or who meet compassionate use criteria. The committee reviews charts for fairness and appropriateness in the application of the ACC-NCDR data elements. The committee is given the authority to either adjudicate or make changes to the reported data if they feel that an error in documentation or application of a data element has been made. Mass-DAC also uses secondary sources of information to verify reported data, including the state mortality database, the national death index and outpatient, inpatient and emergency room case mix data obtained from the Massachusetts Inpatient Discharge Registry. In general, the time from the performance of an index PCI procedure to the release of a public report on hospital specific 30 day all cause risk standardized mortality rates ranges between 19–28 months. [13]

Data Security and De-Identification

The security structure will be modeled after the Microsoft Exchange model with each data source and analysis within the data source allowing access of specific users and user levels. Five user levels will be implemented at each site: analysis owner, editor, analyst, reviewer and administrator. Each level of user access will have specific entitlements and restrictions to view de-identified source data, view or modify analysis parameters, and view results and reports. User accounts and passwords will be maintained as encrypted data within the SQL (Structured Query Language) server database. Best practices for securing Structured Query Language server data will be used. [14]

Data submitted from all DELTA agents to the remote server will be fully de-identified relative to the patient, performing physician and participating center. No hospital record number, patient names, social security numbers, dates of birth, zip code information, physician name or any registration number will be allowed to pass to the DELTA server.

While anonymized data will be forwarded to the central server, there is no identification of the performing center and the central server does not permit any user to analyze data at the center level. The specific center information is not stored in the database, except as an anonymized code. Moreover, the analyses will use hierarchical methods to control for between-center variation in outcomes. We will not explore between-center differences in outcomes as a primary or secondary analysis of the study. A site-specific problem could be identified through pre-specified site-specific secondary analysis using the anonymized code of the site, however we will not explore this as an objective of the study.

Clinical Outcomes for Adverse Event Detection

The study will monitor the safety of six medical devices used routinely in interventional cardiology practice. The devices will include two recently introduced drug eluting coronary stent systems (the Abbott Vascular Xience stent and the Medtronic Endeavor stent), two vascular closure devices (the St. Jude Medical AngioSeal and the Abbott Vascular StarClose), and two embolic protection device systems (the Boston Scientific FilterWire and the EV3 Spider). Each medical device will be monitored for risk-adjusted adverse events specific to the device class (drug eluting stent, closure device, and embolic protection device) as described in Table 1. In addition, two non device-related analyses of continuous outcome variables will be performed as described in Table 1. Appendix A lists the covariates included in the propensity match analysis of each class of device. Clinical outcomes in certain candidate subgroups (Age>70, women and diabetics) will also be analyzed as such groups are relatively underrepresented in prospective clinical trials. Based on our prior analysis from the statewide PCI registry on 74, 427 PCI procedures during 4½ year period, these subgroups were noted to represent 1/3rd of the cohort (Age>70, 38.1%; women, 30.8%; diabetics, 30.4%) and had higher event rates of death (Age>70, 2.7%; women, 2.1%; diabetics 2%), peri-procedural MI (Age>70, 2.9%; women, 2.7%; diabetics, 2.4%) than their counterparts without such risk factor.

Based on historical trends, it is expected that the participating centers will perform approximately 22,000 PCI procedures during the course of the study. The anticipated exposure to particular medical devices will range from approximately 550 cases in which the EV3 spider device will likely be used, to over 10,200 cases in which the Xience drug eluting stent will likely be used. Assuming adverse event rates for the control populations approximate historical rates from MA statewide statistics (2003–2007), the proposed analyses will have one-sided power to detect a 50% increase in composite adverse events (corrected for type 1 error inflation using O’Brien-Fleming alpha-spending method), using a 10% alpha error, ranges from 72% for EV3 Spider embolic protection device to 99% for the Xience drug eluting stent, with 10 of the 18 proposed device-specific analyses having a power of greater than 85%.

Event Rate Expectations

For each device-outcome analysis, a propensity score matched concurrent control population will be developed based on previously published risk factors for the outcome of interest, as well as factors considered by domain experts to potentially influence the selection of one device versus another in its class. [9] The complete list of variables that were chosen for each device class is summarized in appendix A. Propensity scores were developed from a non-parsimonious hierarchical logistic regression analysis developed with the device of interest (exposure) used as the dependent variable, adjusting for baseline covariates listed in the appendix. This method removes measured confounding among the included covariates for patients in an observational cohort. [15] Large numbers of covariates may be used for this purpose, and the method can outperform traditional logistic regression adjustment for rare outcomes. [16, 17]

A statistical module was developed in DELTA in order to allow the system to generate propensity scores and perform matching between samples with and without the exposure of interest. Propensity score model development will be performed by SAS (version 9.1, Cary, NC) through the DELTA module. Propensity score matching will be enforced between the groups within a time interval. Matched observations without the exposure will be placed in the ‘control’ group, and matched observations with the exposure will be placed in ‘exposed case’ group. The cumulative number of events and observations per specified time period will be used to calculate a difference of proportions by the Wilson method in the ‘exposed case’ and ‘control’ groups. [18] Point estimates of the difference of proportions with confidence intervals will be generated by these calculations, and if the confidence intervals of an estimate do not cross 0, a statistically significant difference will be detected between the groups for that time period. Appendix A lists the covariates included in the propensity match analysis of each class of device. Matching cases will be randomly selected when candidate control cases were performed within 6 months of exposed case and had a propensity score match within 0.05 of the exposed case. A caliper width of 0.05 of the propensity score was chosen to reduce bias and optimize matching proportion. [19]

DELTA Safety Alerts

Figure 2 shows a flow diagram that summarizes the various steps involved in the DELTA network surveillance. The central DELTA server will monitor each of the target medical devices for the adverse events noted above as well as the continuous outcomes analyses, for a total of 20 simultaneous prospective monitoring analyses (18 device specific dichotomous outcomes, 2 continuous outcomes; see table 1). Event rates will be calculated monthly, with safety alerts triggered if the cumulative event risk exceeded the upper confidence limit of the propensity-matched control group using a 90% confidence interval by greater than 20% over baseline (control) risk (Proportional difference test) [18, 20], corrected for type 1 error inflation using the O’Brien-Fleming alpha-spending method. [21] O’Brien-Fleming alpha-spending method was chosen over other alpha-spending methods because of its strength to detect safety signals during the later stages of data collection. The use of a 20% increase in risk was chosen to increase the specificity for clinically meaningful differences in event rates. While use of a 90% confidence interval represents liberal alerting requirements, it has been selected to increase the sensitivity of detecting a safety issue and to minimize Type II error. If there is no device specific alert generated, we plan on exploring the time advantage to detecting a high likelihood signal regarding acute renal injury following PCI procedures in the subgroup of patients with pre-existing mild renal insufficiency. From our prior analysis from statewide PCI registry data, this subgroup represented 6% of the population.

Figure 2.

Figure 2

Flow diagram illustrating various steps involved in the DELTA network surveillance (For the purpose of clarity, Xience DES is chosen as an example)

ACC-NCDR = American College of Cardiology-National Cardiovascular Data Registry, BUDT = Bayesian updated distribution test, DELTA = Data Extraction and Longitudinal Trend Analysis, DES = Drug-eluting stent, DPH = Department of public health, LR = Logistic regression, MI = Myocardial infarction, PCI = Percutaneous coronary intervention, PM = Propensity matching, SPRT = Sequential probability ratio test.

An analysis that generates two or more consecutive safety alerts during cumulative monitoring will require case review, adjudication, and result verification (see Data and Outcomes Clinical Audits section). In addition, any such “significant” alert, defined as two or more significant alerts within a device-outcome analysis, will prompt detailed sensitivity analyses in order to confirm the adverse safety signal identified in the propensity matched analysis and to investigate the specific patient populations affected. These sensitivity analyses will include: periodic signal evaluation (to explore consistency of elevated rates and temporal trends in outcomes), assessment using alternative event expectations through the use of logistic regression adjustment (based on historical non-exposed cases), and exploration of potential imbalance of specific risk factors between exposed and matched control populations. The alternative risk modeling strategy for assessing the risk of adverse events was implemented within DELTA using a multivariate logistic regression prediction model for the risk of each adverse outcome of interest. The model will be developed and calibrated using control cases in the time period immediately preceding the study period for any particular device, and applied prospectively to the entire cohort of patients exposed to the device of interest. [6] This method allows for use of the entire exposed cohort, rather than the subset with adequate match to a concurrent control population.

Finally, each “significant alert” will prompt secondary exploratory signal detection methods including the Bayesian updated distribution test and the sequential probability ratio test (SPRT). The Bayesian updated distribution test relies upon the generation of a new posterior probability distribution with accumulating experience and evaluates for any significant change in probability by making quantitative comparison with the prior distribution through area of overlap analysis. [22] The DELTA system will issue an alert if 90% of the posterior distribution area exceeds the threshold value; determined to be the 120% of mean of the original prior distribution. SPRT will also be prospectively evaluated for each outcome by encoding the continuous risk adjustment models (or simplified risk scores) into the SPRT framework. [23] The SPRT method also offers an advantage of correcting for type I error inflation. [24, 25]The sensitivity, specificity, positive and negative predictive values will be calculated for each of these exploratory methods, by comparing their alerting behavior to the adjudicated results of the primary propensity matched analysis.

Data and Clinical Outcomes Audits

All clinical records connected with an adverse event will be audited at the local institution, by a local clinical team for confirmation of all adverse outcomes. Randomly selected “risk matched” case records without adverse events for each of the inpatient related outcomes will be audited using a 3:1 matching ratio to reduce the likelihood that adverse events have been systemically missed at any participating site. Records will be selected for review by calculating the predicted risk for each patient for each given outcome and then sampling patients from the highest risk quartile for each of the outcomes.

Each participating hospital will be responsible for providing a clinical record audit team which will include at least one physician and one nurse or data manager, with prior experience with the application of ACC-NCDR data elements and definitions. The clinical team will review and adjudicate the alerts and complete the audits of local clinical records for the identified high-risk patients within 2 weeks of generation of an adverse event alert.

Overall Study Outcomes and Endpoints

The primary endpoint of the study is to find out the adverse event rates of several classes of new cardiovascular devices (drug-eluting stents, vascular closure devices and embolic protection devices). We will compare the DELTA generated alerts with the adjudicated outcome results generated by manual chart review audits described above. Using the audited results as the “gold standard” for outcomes, one of the secondary endpoint of the study will be the time-savings (if any) between the DELTA network detection of a true device safety alert and the time the same outcome would have been detected using conventional retrospective data analysis methods. The latter will be determined from the publicly reported data from the DPH. Other secondary endpoints include the overall sensitivity, specificity, positive predictive value, and negative predictive value of the DELTA network surveillance system. In general, public reporting of Massachusetts’s statewide PCI outcomes data is done by the DPH 19–28 months after the index PCI. [26] Based on our prior validation study of DELTA on statewide coronary artery bypass surgery outcomes in Massachusetts, DELTA system was able to generate positive alerts approximately 13 months earlier than public reporting by DPH.

DISCUSSION

The DELTA system has been tested and validated in a single center clinical database evaluating in-patient mortality associated with Sirolimus drug-eluting coronary stent implantation [6] and was efficient in identifying low frequency events using statistical techniques such as statistical process control, logistic regression and Bayesian updating statistics. The DELTA system has also been validated using multi-center randomized trial data of oral anticoagulant therapy in which significant safety issues were identified during execution of the trial [8], using similar statistical techniques. Recent well-publicized examples of implantable cardiovascular devices found to have significant safety risks after initial clinical approval, supports the investigation of prospective surveillance systems as one mechanism for achieving time-efficient safety signal identification. [2730] Recently, we identified safety signals in a real world cardiovascular device use using the DELTA system in Massachusetts and have shown that Taxus Express 2 drug-eluting stent had 28% higher postprocedural MI rate and 21% higher MACE rate than alternative drug-eluting stent. Also, we showed that Angio-Seal STS vascular closure device had 51% increased risk of major vascular complications compared with alternate vascular closure devices. [31] Also, the DELTA system is currently being implemented in the prospective safety monitoring of hip implants and ICD-leads in other institutions where longitudinal follow-up data is available. Therefore, extending DELTA to prospective multi-center surveillance may offer significant public health advantages as outlined below.

Advantages

This study is novel in its approach towards prospective automated post-market surveillance through utilization of a network of multiple centers continuously sharing de-identified clinical data. The approach studied here may offer early warning of potential safety concerns to allow appropriate actions to be taken by public health and regulatory agencies to reduce risks to future patients. The DELTA system is designed to be resource efficient, by prompting review of only those devices that generate sustained safety signals; thereby avoiding unnecessary analyst (human) time spent monitoring low-likelihood medical device safety concerns. The current clinical trial design could form a template to perform prospective studies in the future to analyze the safety of many medical products using a similar network approach based on established clinical registry participation or electronic medical record clinical data extraction. With the rapid growth of electronic medical records, DELTA system could be applied to any clinical outcomes data repositories for prospective safety monitoring of selected devices and medications.

Challenges

While monthly data collection and analysis is a significant improvement over current schedules for collection and review of data (typically 6–12 months), there is an inherent delay between time of device exposure and ultimate data availability. Therefore, temporal availability of data will depend upon adherence to timely submissions of complete data sets from all the participating centers. In addition, the time frame for event adjudication at the participating institutions must be brief and efficient in order to validate any alerts issued by the system. Finally, for any alert generated, it will be critical to explore for potential confounders of the outcome observed, through rigorous sensitivity analyses as proposed above.

Limitations

The DELTA network system will rely upon data submitted to Massachusetts-DPH through the modified ACC-NCDR data submission tool and is therefore limited by the scope of the available clinical data. The DELTA network study is a non-randomized prospective observational study subject to biases inherent to observational studies, such as an imbalance of risk factors and exposures. The DELTA network study is a pilot study for evaluating the feasibility of distributed prospective medical device safety surveillance, and may have insufficient power to detect a safety signal at the thresholds for alerting selected.

CONCLUSION

The DELTA network study is a unique distributed automated multicenter prospective surveillance study based on analysis of ACC-NCDR data elements at a network of independent medical centers in Eastern Massachusetts. Such an approach to post-market prospective safety surveillance may offer significant advantages to conventional retrospective safety analyses and may complement the voluntary reporting of medical device hazards used by regulatory agencies today [32]. Successful completion of the DELTA network study will also evaluate whether prospective continuous monitoring of device outcomes offer time savings in the identification of significant safety signals so as to inform public health stakeholders and ultimately reduce patient exposures to potentially harmful medical devices.

Acknowledgments

This study was funded, in part, by grants from National Library of Medicine (NIH R01-LM008142) and the Food and Drug Administration (HHSF 223200830058C) as well as by the Veteran’s Administration Health Services Research and Development Service (CDA 2-2008-020).

Abbreviations

DELTA

Data Extraction and Longitudinal Trend Analysis

DPH

Department of Health

ACC-NCDR

American College of Cardiology- National Cardiovascular Data Registry

MASS-DAC

Massachusetts Data Analysis Center

PCI

Percutaneous Coronary Intervention

SPRT

Sequential probability ratio test

Appendix A: Variables used in the propensity match

A.1 Stent Device Propensity Match

Patient age

Patient height

Patient weight

History of diabetes

History of peripheral vascular disease

Pre-procedure renal insufficiency

History of chronic renal dialysis

History of chronic obstructive lung disease

Transfer from referring hospital

Performing center (anonymized)

ST-segment elevation myocardial infarction on presentation

NSTEMI on presentation

Left main disease (>50% severity)

Proximal LAD disease (>70% severity)

Multi-vessel disease

Pre-PCI TIMI flow rate

High-risk lesion (per NCDR)

Ejection fraction ≤ 30%

Minimal target vessel diameter

Lesion with restenosis

Total number of lesions treated during index PCI

Total number of stents used

Number of coronary vessels treated

Total number of PCI during admission

Left main PCI

Emergent procedure status

Salvage procedure status

Compassionate use procedure (per Mass-DAC definition)

Fluoro-time (surrogate for procedure duration)

A.2 Embolic Protection Propensity Match

Patient age

Female gender

Patient height

Patient weight

History of smoking

History of diabetes

History of hypercholesterolemia

History of peripheral vascular disease

Pre-procedure renal insufficiency

History of chronic renal dialysis

History of chronic obstructive lung disease

Family history of CAD

History of sub-acute stent thrombosis

Transfer from referring hospital

Performing center (anonymized)

Unique physician identification number

ST-segment elevation myocardial infarction on presentation

NSTEMI on presentation

Cardiogenic shock on presentation

Ejection Fraction ≤ 30%

Previously treated coronary lesion

Pre-procedure target lesion severity

Length of the coronary lesion

Diameter of the reference vessel (for vein graft lesion treated)

Risk of the coronary lesion (per NCDR)

Presence of graft lesion

Thrombectomy during PCI

Intraprocedural use of glycoprotein 2b3a inhibitor

Total number of lesions treated during index PCI

Total number of stents used

Number of coronary vessels treated

Total number of PCI during admission

Left main PCI

Emergent procedure status

Salvage procedure status

Compassionate use procedure (per Mass-DAC definition)

Fluoro-time (surrogate for procedure duration)

A.3 Vascular Closure Device Propensity Match

Age > 70yr

Female gender

Patient height

Patient weight

History of diabetes

History of hypertension

History of peripheral vascular disease

Pre-procedure renal insufficiency

History of chronic renal dialysis

History of chronic obstructive lung disease

Performing center (anonymized)

ST-segment elevation myocardial infarction on presentation

NSTEMI on presentation

Left main PCI

Number of coronary vessels treated

Intraprocedural use of glycoprotein 2b3a inhibitor

Use of bivalirudin during PCI

Use of venous sheath during PCI

Total number of PCI during admission

Intra-aortic balloon pump during procedure

Emergent procedure status

Salvage procedure status

Fluoro-time (surrogate for procedure duration)

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  • 1.Platt R, Wilson M, Chan KA, Benner JS, Marchibroda J, McClellan M. The new Sentinel Network--improving the evidence of medical-product safety. N Engl J Med. 2009 Aug 13;361(7):645–7. doi: 10.1056/NEJMp0905338. [DOI] [PubMed] [Google Scholar]
  • 2. [Accessed February 3, 2010];The Sentinel Initiative: a national strategy for monitoring medical product safety. http://www.fda.gov/downloads/Safety/FDAsSentinelInitiative/UCM124701.pdf.
  • 3.Baim DS, Mehran R, Kereiakes DJ, Gross TP, Simons M, Malenka D, et al. Postmarket surveillance for drug-eluting coronary stents: a comprehensive approach. Circulation. 2006 Feb 14;113(6):891–7. doi: 10.1161/CIRCULATIONAHA.105.569657. [DOI] [PubMed] [Google Scholar]
  • 4. [Accessed February 3, 2010];Manufacturer and User Facility Device Experience Database - (MAUDE) doi: 10.1097/BOT.0000000000001948. http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/PostmarketRequirements/ReportingAdverseEvents/ucm127891.htm. [DOI] [PubMed]
  • 5. [Accessed February 3, 2010];Massachusetts Data access center (MASS-DAC) Percutaneous coronary intervention Cohort timeline summary. http://www.massdac.org/pci.
  • 6.Matheny ME, Ohno-Machado L, Resnic FS. Monitoring device safety in interventional cardiology. J Am Med Inform Assoc. 2006 Mar-Apr;13(2):180–7. doi: 10.1197/jamia.M1908. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Arora N, Matheny ME, Sepke C, Resnic FS. A propensity analysis of the risk of vascular complications after cardiac catheterization procedures with the use of vascular closure devices. Am Heart J. 2007 Apr;153(4):606–11. doi: 10.1016/j.ahj.2006.12.014. [DOI] [PubMed] [Google Scholar]
  • 8.Matheny ME, Morrow DA, Ohno-Machado L, Cannon CP, Sabatine MS, Resnic FS. Validation of an automated safety surveillance system with prospective, randomized trial data. Med Decis Making. 2009 Mar-Apr;29(2):247–56. doi: 10.1177/0272989X08327110. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Matheny ME, Arora N, Ohno-Machado L, Resnic FS. Rare adverse event monitoring of medical devices with the use of an automated surveillance tool. AMIA Annu Symp Proc. 2007:518–22. [PMC free article] [PubMed] [Google Scholar]
  • 10.Gostin LO. National health information privacy: regulations under the Health Insurance Portability and Accountability Act. JAMA. 2001 Jun 20;285(23):3015–21. doi: 10.1001/jama.285.23.3015. [DOI] [PubMed] [Google Scholar]
  • 11.Cannon CP, Battler A, Brindis RG, Cox JL, Ellis SG, Every NR, et al. American College of Cardiology key data elements and definitions for measuring the clinical management and outcomes of patients with acute coronary syndromes. A report of the American College of Cardiology Task Force on Clinical Data Standards (Acute Coronary Syndromes Writing Committee) J Am Coll Cardiol. 2001 Dec;38(7):2114–30. doi: 10.1016/s0735-1097(01)01702-8. [DOI] [PubMed] [Google Scholar]
  • 12. [Accessed February 3, 2010];American College of Cardiology’s National Cardiovascular Data Registry’s instrument. http://www.accncdr.com/WebNCDR/ELEMENTS.ASPX.
  • 13. [Accessed May 25, 2010];Massachusetts Cardiac Study: Process of completing final public reports on PCI. http://massdac.org/sites/default/files/reports/FlowchartMassDACSteps.pdf.
  • 14. [Accessed February 3, 2010];Securing SQL Server. http://msdn.microsoft.com/en-us/library/bb283235(SQL.90).aspx.
  • 15.Rosenbaum P, Ruben D. The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika. 1983;70:41–55. [Google Scholar]
  • 16.Brookhart MA, Schneeweiss S, Rothman KJ, Glynn RJ, Avorn J, Sturmer T. Variable selection for propensity score models. Am J Epidemiol. 2006 Jun 15;163(12):1149–56. doi: 10.1093/aje/kwj149. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Cepeda MS, Boston R, Farrar JT, Strom BL. Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders. Am J Epidemiol. 2003 Aug 1;158(3):280–7. doi: 10.1093/aje/kwg115. [DOI] [PubMed] [Google Scholar]
  • 18.Newcombe RG. Interval estimation for the difference between independent proportions: comparison of eleven methods. Stat Med. 1998 Apr 30;17(8):873–90. doi: 10.1002/(sici)1097-0258(19980430)17:8<873::aid-sim779>3.0.co;2-i. [DOI] [PubMed] [Google Scholar]
  • 19.Austin PC. Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulations. Biom J. 2009 Feb;51(1):171–84. doi: 10.1002/bimj.200810488. [DOI] [PubMed] [Google Scholar]
  • 20.Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med. 1998 Apr 30;17(8):857–72. doi: 10.1002/(sici)1097-0258(19980430)17:8<857::aid-sim777>3.0.co;2-e. [DOI] [PubMed] [Google Scholar]
  • 21.O’Brien PC, Fleming TR. A multiple testing procedure for clinical trials. Biometrics. 1979 Sep;35(3):549–56. [PubMed] [Google Scholar]
  • 22.Resnic FS, Zou KH, Do DV, Apostolakis G, Ohno-Machado L. Exploration of a bayesian updating methodology to monitor the safety of interventional cardiovascular procedures. Med Decis Making. 2004 Jul-Aug;24(4):399–407. doi: 10.1177/0272989X04267012. [DOI] [PubMed] [Google Scholar]
  • 23.Matheny ME, Ohno-Machado L, Resnic FS. Risk-adjusted sequential probability ratio test control chart methods for monitoring operator and institutional mortality rates in interventional cardiology. Am Heart J. 2008 Jan;155(1):114–20. doi: 10.1016/j.ahj.2007.08.022. [DOI] [PubMed] [Google Scholar]
  • 24.Spiegelhalter D, Grigg O, Kinsman R, Treasure T. Risk-adjusted sequential probability ratio tests: applications to Bristol, Shipman and adult cardiac surgery. Int J Qual Health Care. 2003 Feb;15(1):7–13. doi: 10.1093/intqhc/15.1.7. [DOI] [PubMed] [Google Scholar]
  • 25.Steiner SH, Cook RJ, Farewell VT, Treasure T. Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics. 2000 Dec;1(4):441–52. doi: 10.1093/biostatistics/1.4.441. [DOI] [PubMed] [Google Scholar]
  • 26. [Accessed May 25, 2010];Mass-DAC Data Submission and Report Timelines. http://massdac.org/sites/default/files/PCITimelines.pdf.
  • 27.Gross TP, Kessler LG. Medical device vigilance at FDA. Stud Health Technol Inform. 1996;28:17–24. [PubMed] [Google Scholar]
  • 28.Samore MH, Evans RS, Lassen A, Gould P, Lloyd J, Gardner RM, et al. Surveillance of medical device-related hazards and adverse events in hospitalized patients. JAMA. 2004 Jan 21;291(3):325–34. doi: 10.1001/jama.291.3.325. [DOI] [PubMed] [Google Scholar]
  • 29.Maisel WH. Unanswered questions--drug-eluting stents and the risk of late thrombosis. N Engl J Med. 2007 Mar 8;356(10):981–4. doi: 10.1056/NEJMp068305. [DOI] [PubMed] [Google Scholar]
  • 30.Shah JS, Maisel WH. Recalls and safety alerts affecting automated external defibrillators. JAMA. 2006 Aug 9;296(6):655–60. doi: 10.1001/jama.296.6.655. [DOI] [PubMed] [Google Scholar]
  • 31.Resnic FS, Gross TP, Marinac-Dabic D, Loyo-Berrios N, Donnelly S, Normand SL, et al. Automated surveillance to detect postprocedure safety signals of approved cardiovascular devices. JAMA. 2010 Nov 10;304(18):2019–27. doi: 10.1001/jama.2010.1633. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Managing Risks from Medical Product Use: Creating a Risk Management Framework; Report to the FDA Commissioner from the Task Force on Risk Management; Rockville, MD: U.S. Department of Health 26. 105 Code of Massachusetts Regulations. Department of Public Health; 2001. pp. 130.1201–130.130. [Google Scholar]

RESOURCES