How real-world evidence can really deliver: a case study of data source development and use

Purpose and goals of original data collection

Electronic health record (EHR) data hold great promise for evaluating the quality and efficiency of care. Realization of the potential for EHRs requires that data be captured without interfering with clinical workflow and without requiring manual data entry into databases. Efforts to extract EHR data to date have largely depended on natural language processing and have not scaled beyond the settings in which they were developed.^1–4

In 2012, the US Food and Drug Administration (FDA) funded a demonstration performed by Mercy Health (Mercy) whereby unique device identifiers (UDIs) were incorporated into Mercy’s electronic information systems, enabling assessments of cardiovascular device performance using EHR data.⁵ The goals and design of the demonstration have been previously described.^6–8 In summary, between May and December 2012, a barcode scanning system was installed in 5 Mercy cardiac catheterization laboratories (Cath Labs) for capturing data on all consumable supplies, including coronary stents. Coronary stent barcodes were used as prototype UDIs and were linked at the patient level with clinical data from the hemodynamic system (Merge Hemo, Merge, Chicago, Illinois, USA) and Mercy’s EHR (EpicCare, Epic, Verona, Wisconsin, USA). A database termed the UDI research database (UDIR) was built that contains EHR-extracted data, UDI-associated coronary stent attributes, and data from other sources, for example, the Social Security Death Master File. The UDIR is updated weekly with EHR data enabling longitudinal follow-up for purposes of device evaluation, including safety surveillance. Details of UDIR content and structure have been published previously⁸ and initial evaluations of its utility appear in the demonstration report on the FDA’s website.⁹

Data collected

Development of the UDIR

Mercy is an integrated delivery network in Missouri, Kansas, Oklahoma and Arkansas with over 40 hospitals and a total of 4396 licensed beds. At the time of the demonstration, Mercy operated five Cath Labs at its hospitals in Joplin, Missouri; Rogers, Arkansas; Springfield, Missouri; St Louis, Missouri; and Washington, Missouri. Mercy manages supply chain information with the Infor Lawson (Infor Lawson, New York, USA) enterprise resource processing (ERP) system and has the same EHR in all its inpatient facilities and clinician practices. In addition, during the time of the demonstration, all Mercy Cath Labs used Merge Hemo as their hemodynamic software for documentation of all procedures, including stent implantation. A point of use barcode scanning inventory system (OptiFlex CL, Omnicell, Mountain View, California, USA) was installed for capturing information on all Cath Lab consumable items (including the prototype coronary stent UDIs) at the point of care.⁷

When coronary stents are brought to the Cath Lab, their prototype UDIs (and now actual UDIs) are scanned into OptiFlex CL. When a stent is implanted in a patient, its UDI is scanned and automatically uploaded into Merge Hemo and OptiFlex CL. The device is linked to the patient in both Merge Hemo and OptiFlex CL. After case completion, Cath Lab personnel transmit the case record (including UDI) from Merge Hemo to the UDIR; OptiFlexCL data are automatically transmitted to the UDIR. Conflicts in patient identification are then reconciled through integration between UDIR and Mercy’s integrated patient data mart. Additionally, a preset group of baseline patient characteristics (laboratory values, diagnoses and so on) are automatically extracted from the EHR and uploaded to the UDIR on a weekly basis. Finally, UDI-associated stent attributes are uploaded to the UDIR from the FDA’s Global UDI Database (GUDID)¹⁰ and from a supplemental database termed the Supplemental UDI Database (SUDID). The structure of the UDIR, then, supports analyses utilizing variables in three broad categories: baseline patient characteristics, coronary stent attributes and patient outcomes including adverse events.

Assessment of the UDIR

The UDIR contains data from multiple sources: EpicCare (Clinical and Billing), Merge Hemo, OptiFlex CL, GUDID, SUDID and the Social Security Death Master File. Additionally, two of Mercy’s Cath Labs (Springfield and St Louis) participate in the National Cardiovascular Data Registry’s (NCDR) CathPCI Registry into which data from all percutaneous coronary interventions (PCIs) are entered by personnel from the hospitals’ clinical quality departments. These data are available for comparison with UDIR data.

UDIR data completeness was assessed using comparisons of data within the database as extracted from each of the data sources and with data obtained from reports submitted to the CathPCI Registry. Cath Lab personnel bar code scanning compliance was also assessed because it is a major contributor to data completeness. Details of data completeness and scan compliance evaluations are included in the online supplementary appendix B and online supplementary figures 1; 2. In summary, both were found to be at an acceptable level for safety surveillance purposes.

Data resource use

Example case study

To demonstrate the functionality of the database and face validity of UDIR data, analyses were performed of ‘all comers’ receiving coronary stents from 1 November 2012 to 26 October 2013. We tracked the major adverse cardiac events of death, acute myocardial infarction (AMI), total coronary revascularization and stent thrombosis. Deaths were captured in the Social Security Death Master File and the other outcomes were found in Epic Clarity (the Epic data warehousing utility) using diagnosis and billing codes.⁹ We selected mortality as the outcome of interest because of its objectivity and data completeness and ‘drug’ as the stent attribute of interest. We hypothesized that there would be no difference in mortality among stents that eluted various drugs (drug eluting stent (DES))—or no drug in the case of bare-metal stents (BMS). During the time of the study, all implanted DES contained everolimus, paclitaxel or zotarolimus. Patient and procedure selection are shown in figure 1.

Figure 1.

Patient and procedure selection.

Our analytic plan called for the UDIR to be an active surveillance dataset and we anticipated using unadjusted data to identify safety signals as we felt it would be difficult to apply propensity scoring to prospectively captured EHR data. When a safety signal was identified, we used propensity matching methods to adjust for potential selection bias in the choice of the stent’s drug. An implanter might, for instance, have chosen a BMS for a patient in shock because of the poor prognosis. Separate analyses were conducted using propensity matching and weighting techniques. Propensity scores were calculated using covariates based on the adjustment models used in the Massachusetts CathPCI data (Mass-DAC registry) with presenting diagnosis based on International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9 CM) codes.¹¹ Twelve of the clinical covariates, as listed in table 1, were available in the UDIR database and were used in our final model.

Table 1.

Baseline characteristics prior to propensity score approaches for patients with complete data and 30 days of follow-up

Baseline characteristics (n=1445)	Stent by drug attribute		P value	Standardised difference*
	DES (n=1145)	BMS (n=300)
Female	376 (32.8%)	100 (33.3%)	0.871	−0.011
Age >65 years	600 (52.4%)	191 (63.7%)	<0.001	−0.230
Caucasian	1095 (95.6%)	286 (95.3%)	0.822	0.014
Married	800 (69.9%)	168 (56.0%)	<0.0001	0.290
Risk factors
Alcohol use	420 (36.7%)	90 (30.0%)	0.031	0.142
Illicit drug use	67 (5.9%)	28 (9.3%)	0.030	−0.132
Acute MI	400 (34.9%)	163 (54.3%)	<0.0001	−0.398
Shock	21 (1.8%)	23 (7.7%)	<0.0001	−0.277
COPD	148 (12.9%)	58 (19.3%)	0.005	−0.175
Diabetes mellitus	435 (38.0%)	102 (34.0%)	0.203	0.083
Dialysis	23 (2.0%)	8 (2.7%)	0.484	−0.044
EF <30%	24 (2.1%)	10 (3.3%)	0.208	−0.076

*Standardised difference=difference in means or proportions divided by SE; imbalance defined as an absolute value >0.10 (small effect size).

COPD, chronic obstructive pulmonary disease; EF, ejection fraction; MI, myocardial infaction.

Finally, Cox proportional hazard models were conducted to compare mortality between patients receiving BMS versus DES. Analyses were performed using SAS V.9.3 and Kaplan Meier curves were created using R V.3.1. The Mercy Hospital St Louis Institutional Review Board determined that the demonstration was exempt from review.

A total of 2250 patients were found undergoing PCI with stent implantation. Except for those patients who died within the first 30 days, a minimum of 30 days of follow-up data were available for each patient with data censored at 1 year post-PCI. The mean follow-up time was 320.9±107.0 days. After applying exclusions, there remained 1555 procedures performed on 1445 patients with an average of 1.1 implanted stents per patient. Of these patients, 300 (20.8%) had received BMS. The remaining received DES, consisting of 980 patients (67.8%) with everolimus, 148 (10.2%) with zotarolimus and 17 (1.2%) with paclitaxel stents.

The distribution of baseline characteristics between patients receiving DES and BMS is shown in table 1. Patients receiving BMS were more likely to be older (over age 65 years) and single, to have COPD, to use drugs illicitly and to be experiencing AMI and shock at baseline. Propensity score matching reduced the number of patients available for analysis to only 558 and the number of procedures to 584. Propensity score weighting, on the other hand, eliminated all the differences between groups for the covariates included in the propensity model and allowed us to keep all 1445 patients in our analysis. The results of propensity score weighting are shown in table 2. The weighted frequency for patients receiving DES stents was 299. We, therefore, selected it as our adjustment methodology.

Table 2.

Baseline characteristics after propensity score weighting for patients with complete data and 30 days of follow-up

Baseline characteristics (n=1445)	Stent by drug attribute		P value	Standardised difference*
	DES (n=299)†	BMS (n=300)
Female	97 (32.4%)	100 (33.3%)	0.739	−0.019
Age >65 years	189 (63.2%)	191 (63.7%)	0.888	−0.009
Caucasian	287 (96.0%)	286 (95.3%)	0.636	0.032
Married	168 (56.2%)	168 (56.0%)	0.955	0.004
Risk factors
Alcohol use	88 (29.4%)	90 (30.0%)	0.823	−0.012
Illicit drug use	26 (8.7%)	28 (9.3%)	0.771	−0.022
Acute MI	164 (54.8%)	163 (54.3%)	0.904	0.010
Shock	23 (7.7%)	23 (7.7%)	0.962	0.001
COPD	55 (18.4%)	58 (19.3%)	0.766	−0.024
Diabetes mellitus	102 (34.1%)	102 (34.0%)	0.979	0.002
Dialysis	9 (3.0%)	8 (2.7%)	0.787	0.021
EF <30%	10 (3.3%)	10 (3.3%)	0.972	0.001

*Standardized difference=difference in means or proportions divided by SE; imbalance defined as an absolute value >0.10 (small effect size).

†Weighted sum, Ns rounded to an integer.

COPD, chronic obstructive pulmonary disease; EF, ejection fraction; MI, myocardial infaction.

A survival analysis of coronary stents by drug attribute (figure 2) demonstrated no significant differences in unadjusted mortality at 1 year among the various DES: everolimus (27 deaths, 2.5%); zotarolimus (four deaths, 2.6%); and paclitaxel (0 deaths, 0%). We, therefore, combined DES for subsequent comparisons with BMS. Figure 3 compares the survival of the combined DES cohort with BMS and demonstrates significantly higher unadjusted 1-year mortality in patients receiving BMS (32 deaths, 10.4%) compared with DES (31 deaths, 2.5%) with separation of the curves occurring in the first 40 days and leveling off thereafter. In a surveillance dataset, this would represent a safety signal prompting further evaluation.

Figure 2.

Unadjusted survival by drug attribute.

Figure 3.

Survival by DES vs BMS. BMS, bare-metal stent; DES, drug eluting stent.

Figure 3 also contains a survival plot comparing the combined DES with BMS adjusted using propensity score weighting. It demonstrates significant attenuation of the mortality difference between DES and BMS although a statistically significant variance remains (HR for BMS 1.82, 95% CI: 1.04 to 3.19, p=0.04).

Strengths and limitations of data source

While we linked to the Social Security Death Master File to capture mortality, the UDIR itself contains EHR data on the three major drivers of device performance: the patient, the operator and the device itself. It also contains several EHR-derived safety outcomes, for example, readmissions, revascularization procedures and stent thrombosis. This construct supports analyses and surveillance capabilities that use various combinations of variables from all three EHR-based categories looking at a variety of safety endpoints. Hypothesis-driven analyses can be performed or events can be monitored longitudinally to detect unanticipated safety signals.

Although our demonstration analysis was performed on coronary stents implanted in 2012–2013, Mercy Cath Labs have continued capturing UDI as part of inventory management, thus, enabling continuation of the UDIR. The UDIR is currently undergoing revision as part of a larger initiative in which we are establishing a distributed data network with two other health systems. This requires updating code sets, for example, ICD-10, and development of a common data model to support distributed analytics. The SUDID requires continual updating to accommodate new devices with new UDIs. Additionally, analyses of more contemporary data will need to include changes in clinical practice. For instance, the use of BMS has decreased markedly since the time period of our original analysis. We will need to understand how changing device selection over time might affect the characteristics included in our propensity score model.

Using UDI as the foundation for medical device identification will permit our approach to extend to other classes of implanted devices. In this regard, Mercy is initiating UDI capture in our operating rooms enabling evaluation of surgically implanted devices. Finally, as challenges related to data quality and completeness are resolved, it would also be possible to use UDIR data in the assessment of device effectiveness as well.

A major advantage of the data source is availability of the data extracts from Mercy’s systems obviating the need for case report forms and avoiding interference with workflow. However, not all covariates are available that are used in previously published stent survival predictive models.¹¹ Further, the use of data extraction constrains our clinical data to discrete elements and therefore limits our ability to draw strong inferences as key information is often contained within clinical notes which are, at present, not available for analysis. There were also challenges in discerning the timing of events within specific days as data entry times were captured but not the event times themselves, thus making it impossible, for instance, to determine whether an AMI occurring on the same day as a PCI occurred before or after the procedure.

Data completeness presented challenges as well as we were using data collected in the routine course of care and because we required that data from stenting procedures be transmitted to the UDIR by clinical personnel after completion of the procedure. Additionally, data completeness was found to vary over time. We devised a method for tracking data completeness that enables significant data gap identification in near real time. Another concern is the possibility of incorrect or incomplete information from the Social Security Death Master File, which is becoming a less reliable source of mortality data as a result of policy changes.¹²

Our EHR longitudinal follow-up data reflected procedures and tests performed at the discretion of the clinician and were not required by protocol. Any signal related to these data can therefore only be considered ‘hypothesis generating’. We selected mortality as the outcome of interest in this first demonstration analysis because we felt we could capture virtually all deaths minimizing the risk of incomplete ascertainment that would have challenged outcomes identified only in the EHR.

One of the greatest limitations is the lack of statistical power and data asymmetry illustrated in the case study. The analysis included 1445 patients and presented challenges in our efforts to do propensity modeling producing only 554 matched pairs, due to the significant imbalance in baseline risk between those patients receiving BMS as compared with those receiving DES. Further, one of three types of DES (everolimus) comprised 86% (980) of all DES in the analysis.