Comparability of Event Adjudication Versus Administrative Billing Claims for Outcome Ascertainment in the Dual Antiplatelet Therapy (DAPT) Study: Findings from the EXTEND-DAPT Study

Abstract

Background:

Data from administrative claims may provide an efficient alternative for endpoint ascertainment in clinical trials. However, it is uncertain how well claims data compares to adjudication by a clinical events committee in trials of patients with cardiovascular disease.

Methods:

We matched 1,336 patients ≥65 years old who received percutaneous coronary intervention in the Dual Antiplatelet Therapy (DAPT) Study with the National Cardiovascular Data Registry (NCDR) CathPCI Registry linked to Medicare claims as part of the Extending Trial-Based Evaluations of Medical Therapies Using Novel Sources of Data (EXTEND) Study. Adjudicated trial endpoints were compared to Medicare claims data with ICD-9 codes from inpatient hospitalizations using time-to-event analyses, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and kappa statistics.

Results:

At 21-month follow-up, the cumulative incidence of major adverse cardiovascular and cerebrovascular events (combined mortality, myocardial infarction [MI], and stroke) was similar between trial-adjudicated events and claims data (7.9% vs. 7.2%, respectively; p = 0.50). Bleeding rates were lower using adjudicated events compared with claims (5.0% vs. 8.6%, respectively; p <0.001). The sensitivity and PPV of comprehensive billing codes for identifying adjudicated events were 65.6% and 85.7% for MI, 61.5% and 47.1% for stroke, and 76.8% and 39.3% for bleeding, respectively. Specificity and NPV for all outcomes ranged from 93.7-99.5%. All 39 adjudicated deaths were identified using Medicare data. Kappa statistics assessing agreement between events for MI, stroke, and bleeding were 0.73, 0.52, and 0.49, respectively.

Conclusions:

Claims data had moderate agreement with adjudication for MI and poor agreement but high specificity for bleeding and stroke in the DAPT Study. Deaths were identified equivalently. Using claims data in clinical trials could be an efficient way to assess mortality among Medicare patients and may help detect other outcomes, though additional monitoring is likely needed to ensure accurate assessment of events.

INTRODUCTION

Randomized controlled trials (RCTs) are considered the gold standard study design to evaluate effectiveness and safety of clinical treatments. However, RCTs have significant challenges to implementation including high financial costs, extensive time required for trial oversight, and limited follow-up.¹ One pragmatic strategy is utilization of administrative claims data with billing codes to assess study endpoints, which have traditionally required adjudication by a Clinical Events Committee (CEC). If comparable to adjudication, use of claims data may allow for event ascertainment that is cheaper, quicker, and less burdensome. Though clinical events reported in claims data have been used extensively in observational studies,^2-7 there are limited data comparing claims-based events to events adjudicated by a trial CEC.

To further investigate the performance of claims data in assessing adjudicated events for patients with coronary artery disease (CAD), we compared cardiovascular and bleeding events identified in Medicare data with adjudicated events in the Dual Antiplatelet Therapy (DAPT) Study, a randomized, placebo-controlled clinical trial comparing 30 vs. 12 months of DAPT following percutaneous coronary intervention (PCI).⁸ Data from the DAPT Study was linked to Medicare claims as part of the Extending Trial-Based Evaluations of Medical Therapies Using Novel Sources of Data (EXTEND) Study.⁹ Using EXTEND, this study compared rates of all-cause mortality, myocardial infarction (MI), stroke, and major bleeding between claims data and CEC-adjudication.

METHODS

EXTEND-DAPT Study Overview

The EXTEND Study is funded by the National Heart, Lung, and Blood Institute (1R01HL136708). An overview of the aims and methods have been previously described.⁹ The EXTEND-DAPT substudy utilized data from the DAPT Study linked to the American College of Cardiology’s (ACC) National Cardiovascular Data Registry (NCDR) CathPCI Registry® and Medicare fee-for-service beneficiary claims.

The DAPT Study was a randomized, placebo-controlled clinical trial which enrolled patients who underwent PCI and received DAPT consisting of aspirin and a thienopyridine for one year.⁸ At 12 months following PCI, patients without post-PCI ischemic or bleeding events were randomized to either placebo (12 total months of DAPT) or continued thienopyridine for another 18 months (30 total months of DAPT). Enrollment of patients took place between August 13, 2009 and July 1, 2011. The trial was conducted by the Baim Institute for Clinical Research. Studies using the EXTEND dataset have been approved by the Institutional Review Board at Beth Israel Deaconess Medical Center with a waiver of informed consent. Data are available from the corresponding author upon reasonable request.

Study Population

For this investigation, we included all U.S. patients aged 65 years or older in the DAPT Study who could be successfully linked via the NCDR CathPCI Registry to the Centers for Medicare and Medicaid Services (CMS) inpatient claims data for all fee-for-service Medicare-insured patients. Linkage between the trial and the CatchPCI Registry was performed using deterministic algorithms based on age or date of birth, sex, PCI date and stent type, hospital discharge date, and hospital identifier. These data were then linked to Medicare inpatient claims (for non-death outcomes) and the Master Beneficiary Summary File (to determine dates of death) using direct identifiers available in the CathPCI Registry. Patients who could not be linked to the CathPCI Registry due to inexact matching characteristics or who were not subsequently found in CMS-fee-for-service claims or the Medicare Master Beneficiary Summary File were excluded.

Among 11,648 patients randomized in the DAPT Study, 3,908 were enrolled in the U.S. and ≥65 years old. Of these patients, 2,558 were linked to the CathPCI Registry. Of those, 1,336 were successfully included in the linked EXTEND-DAPT Cohort (Figure 1). The primary reasons for non-match among Medicare-eligible patients were being insured by Medicare Advantage or lacking sufficient identifying information to be linked between data sources. All baseline characteristics were obtained from information collected in the DAPT Study. The study period was from time of randomization (12 months following PCI) to 21 months post-randomization (33 months following PCI).

Figure 1.

Study Cohort with Data Linking Criteria in the EXTEND-DAPT Study

Assessment of Clinical Events

This study examined major clinical events evaluated in the DAPT Study, including death, MI, stroke, major bleeding, and major adverse cardiovascular and cerebrovascular events (MACCE). MACCE was defined as a compositive of all-cause death, MI, and stroke. Clinical events in trial data were determined based on adjudication by the DAPT Study CEC, which was blinded to randomization status. The trial used pre-specified definitions of MI and stroke for adjudication.⁸ Major bleeding in this study was defined as any adjudicated event that met criteria for either moderate or severe bleeding according to the Global Utilization of Streptokinase and Tissue Plasminogen Activator for Occluded Arteries (GUSTO) classification, or Type 3 or Type 5 bleeding according to the Bleeding Academic Research Consortium (BARC).^10,11

Clinical events in administrative claims were defined based on a comprehensive list of specified International Classification of Diseases, 9^th Revision, Clinical Modification (ICD-9) diagnosis codes associated with hospitalizations. For each outcome we identified ICD-9 codes based on clinical relevance as well as prior literature (Supplemental Table I).^12-16 Given events leading to blood transfusions were included in trial definitions of bleeding, ICD-9 procedure codes for blood transfusions were also used. An event was counted if the corresponding codes were present in either the primary or secondary billing position during the hospitalization associated with the event. In a sensitivity analysis, we also assessed events using only diagnosis codes present in the primary billing position. An event in claims data was counted as a match with a CEC-adjudicated event if the hospitalization admission date occurred within 14 days of the event date determined in the trial.

Statistical Analysis

Baseline characteristics of linked and unlinked individuals are presented using percentages for categorical variables and means with standard deviations (SDs) for continuous variables. Univariate comparisons were performed using Fisher’s exact test for categorical variables and t-tests for continuous variables.

Clinical events as determined by claims billing codes and CEC-adjudicated events were compared using three different methods. First, time-to-event analyses were performed using the Kaplan-Meier method to estimate cumulative incidence of each end point. With this approach, the linked study cohort was evaluated and plotted using clinical events determined separately in claims data and by CEC-adjudication. Patients were censored either at the time of the first event for each endpoint or at the time of censoring in the trial. Log-rank tests were used to compare groups. Second, we calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each endpoint identified using claims data, with adjudicated events serving as the gold standard. Third, given that adjudicated outcomes may not always be perfectly accurate, we also determined kappa statistics to assess agreement between events. Recurrent events were included in analyses determining these values.

The selection of relevant claims codes for each endpoint was performed using three separate strategies – one which included a comprehensive list of codes and two other more parsimonious strategies. The first strategy employed a comprehensive list of ICD-9 codes for each outcome based on clinical plausibility and prior review of the literature. The second strategy used only ICD-9 codes from the full list which had either a PPV of 100% or a positive likelihood ratio (+LR) of greater than 10 for identifying a trial event (the +LR algorithm), which is similar to previously used methods.¹⁷ The third strategy utilized a recursive classification-tree decision algorithm applied to the full list of ICD-9 codes for each outcome. The recursive tree algorithm is a well-established nonparametric method to identify prediction rules using a process that involves variable selection and threshold selection to partition the sample into subgroups with similar risks of the outcome.¹⁸ Ten-fold cross validation was performed for both the +LR and recursive tree algorithms using 90% of the cohort as the training dataset and 10% as the validation dataset in ten unique iterations. Results were reported as point estimates with 95% confidence intervals (CI) using exact Clopper-Pearson confidence intervals for sensitivity and specificity, standard logit confidence intervals for PPV and NPV, and confidence intervals as defined by Fleiss et al. for kappa statistics.^{19, 20} All analyses were conducted in SAS v 9.4 (SAS Institute, Cary, NC) using a two-tailed p <0.05 to define significance.

RESULTS

Among 1336 patients in the linked EXTEND-DAPT Study, 32.7% were women and the mean (SD) age was 71.8 (5.5) years. Compared to Medicare patients who could not be linked, the study cohort had more women, lower rates of index presentation with STEMI, and higher rates of presentation with stable angina (Table 1). Adverse outcomes up to 21 months following trial randomization were similar between the linked and unlinked groups.

Table 1.

Characteristics and Outcomes of Linked and Unlinked Patients ≥65 Years Old in the EXTEND-DAPT Study

Baseline Characteristics*	Linked Patients (n = 1336)	Unlinked Patients (n = 2572)	P value
Age (mean, SD)	71.8 (5.5)	71.5 (5.3)	0.09
Female	32.7	29.5	0.04
Nonwhite Race	6.6	7.5	0.36
Hispanic/Latino Ethnicity	2.5	3.6	0.07
BMI (mean, SD)	29.6 (5.3)	29.7 (5.3)	0.76
Diabetes Mellitus	33.2	34.4	0.48
Hypertension	85.1	83.1	0.10
Current or Recent Smoking	11.3	10.8	0.70
Stroke/TIA	5.4	4.8	0.44
Congestive Heart Failure	6.4	6.8	0.73
Peripheral Arterial Disease	10.4	8.2	0.03
History of Major Bleeding	1.0	1.1	0.75
Prior PCI	35.2	37.0	0.29
Prior CABG	16.6	18.0	0.27
Prior Myocardial Infarction	21.0	22.5	0.30
Atrial Fibrillation	4.8	4.8	0.94
Index PCI Indication
STEMI	4.0	6.6	<0.001
NSTEMI	11.2	12.4	0.25
Unstable Angina	15.0	16.1	0.41
Stable Angina	45.1	40.9	0.01
Other	24.8	23.9	0.56

^*All values presented are percentages except for age and BMI. BMI = body mass index, CABG = coronary artery bypass graft surgery, NSTEMI = non-STE elevation myocardial infarction, PCI = percutaneous coronary intervention, SD = standard deviation, STEMI = ST-elevation myocardial infarction, TIA = transient ischemic attack

Comparisons of Clinical Events Using Comprehensive List of ICD-9 Codes

In time-to-event analyses of the linked cohort, there was no significant difference in the cumulative incidence of MACCE reported by claims compared with adjudicated events (7.9% vs 7.2%, log-rank p value = 0.50) (Figure 2A). All 39 deaths and death dates reported in trial data were identified using CMS data. The sensitivity of the comprehensive list of claims for detecting MACCE in follow-up was 72.9%, the specificity was 97.8%, the PPV was 77.0%, and the NPV was 97.2%. The kappa statistic for MACCE was 0.72, representing moderate agreement between claims and trial data. Detailed numbers of events are shown in Table 2.

Figure 2. Clinical Events Reported by Medicare Claims and Adjudicated Trial Data in the EXTEND-DAPT Study.

Cumulative incidence of (A) MACCE, (B) myocardial infarction, (C) stroke, and (D) major bleeding from 0 to 21 months following trial randomization in the DAPT Study (12 to 33 months following PCI). The curves for DAPT Study represent events adjudicated by the trial Clinical Events Committee and the Medicare Claims curves represents hospitalizations with events determined by primary and secondary diagnosis codes; MACCE = major adverse cardiovascular and cerebrovascular events, MI = myocardial infarction.

Table 2.

Assessment of Clinical Events Using Claims Data Compared to Adjudicated Trial Data in the EXTEND-DAPT Study

Major Adverse Cerebrovascular and Cardiovascular Events
Present in Claims (N)*	Present in Trial (N)			Sensitivity (%, 95% CI)	Specificity (%, 95% CI)	PPV (%, 95% CI)	NPV (%. 95% CI)	Kappa Coefficient (95% CI)
	Yes	No	Total
Yes	94	28	122	72.9 (64.3-80.3)	97.8 (96.8-98.5)	77.0 (69.6-83.1)	97.2 (96.3-97.9)	0.72 (0.66-0.79)
No	35	1222	1257
Total	129	1250	1379
Myocardial Infarction†
Yes	-	-	-	65.6 (52.7-77.1)	99.5 (98.9-99.8)	85.7 (73.7-92.8)	98.3 (97.6-98.8)	0.73 (0.64-0.83)
No	-	-	-
Total	-	-	-
Stroke
Yes	16	18	34	61.5 (40.6-79.8)	98.6 (97.9-99.2)	47.1 (33.9-60.6)	99.2 (98.8-99.5)	0.52 (0.37-0.68)
No	10	1302	1312
Total	26	1320	1346
Major Bleeding
Yes	53	82	135	76.8 (65.1-86.1)	93.7 (92.2-94.9)	39.3 (33.6-45.3)	98.7 (98.0-99.1)	0.49 (0.40-0.57)
No	16	1214	1230
Total	69	1296	1365

CI = confidence interval NPV = negative predictive value, PPV = positive predictive value

^*The comprehensive list of codes was used to ascertain events in claims.

^†Values for cell counts were not reported per the Centers for Medicare and Medicaid cell suppression policy.

Claims data reported a statistically similar though numerically lower cumulative incidence of MI compared to adjudicated trial data (3.7% in claims vs. 4.6% in adjudicated data; log-rank p value = 0.28; Figure 2B). The PPV of claims data for identifying MI was 85.7%; sensitivity, specificity, and NPV are shown in Table 2. The kappa statistic for MI was 0.73. The cumulative incidence for stroke was low overall, with similar rates between claims and trial data over time (2.0% in claims vs. 1.9% in adjudicated data; log-rank p value 0.78; Figure 2C). However, sensitivity and kappa statistics for claims in evaluating stroke events were modest, while PPV was poor (Table 2).

There were more major bleeding events identified in claims data compared to trial data (cumulative incidence 8.6% in claims vs. 5.0% in adjudicated data; log-rank p value <0.001; Figure 2D). Claims had moderate sensitivity (76.8%) but lower PPV (39.3%) and kappa (0.49) for major bleeding events; NPV for bleeding was high (98.7%) (Table 2). Among the 82 bleeding in events that were in claims data and not matched to adjudicated BARC Type 3 or 5 or moderate to severe GUSTO bleeding events, 15 (18.3%) matched with adjudicated BARC Type 2 bleeding events.

Use of primary diagnosis codes only resulted in much lower sensitivity but higher PPV for all outcomes compared to use of primary and secondary diagnosis codes (Supplemental Table II).

Comparisons of Clinical Events Using +LR and Recursive Tree Algorithms

After applying the +LR algorithm, a smaller subset of ICD-9 codes were selected from the comprehensive list of codes used in primary analyses (Supplemental Table III). Cumulative incidence rates of outcomes using claims identified by the +LR algorithm were similar to incidence rates determined by the comprehensive list of codes (Supplemental Table IV). Sensitivity values for events in claims data using the +LR algorithm with ten-fold cross validation were lower than using a comprehensive list of codes; specificity, PPV, NPV, and kappa statistics were similar (Supplemental Table II).

An overview of the recursive tree algorithm for assessing clinical events using diagnostic codes in claims is shown in the Supplemental Figure. Sensitivity and kappa statistics were lower with the recursive tree algorithm compared to both the comprehensive list of codes and the +LR algorithm, though the PPV for MI and bleeding were much higher (Supplemental Table II).

DISCUSSION

In this study we compared adverse clinical events identified in administrative claims data with trial events adjudicated by a CEC in the DAPT Study. Our findings show that Medicare claims data perform very well in assessing all-cause mortality and moderately well for the individual endpoint of adjudicated myocardial infarction. Claims had moderate sensitivity and poor PPV for adjudicated stroke and major bleeding events, though specificity and negative predictive values of claims for all events was high. These findings demonstrate both opportunities and important limitations in using claims data to assess major adverse events in a population with coronary artery disease.

Administrative data including claims with ICD codes have been an important source of information for observational studies, particularly those assessing cardiovascular outcomes.^2-7 There is also a growing interest by investigators and regulatory bodies in using these types of data in clinical trials. Use of claims data allows for longer follow-up than most clinical trials and may be particular suited for lower cost, registry-based pragmatic clinical trials. In Sweden, for example, the Thrombus Aspiration during ST-Segment Elevation Myocardial Infarction in Scandinavia (TASTE) trial used endpoints obtained directly from the SWEDEHEART registry and claims data without separate adjudication by a CEC.²¹ Claims data may also be useful for post-marketing surveillance of medical treatments. The U.S. Food and Drug Administration (FDA), for example, has initiated the National Evaluation System for Health Technology (NEST) program and the Sentinel Initiative to more comprehensively assess medical devices and pharmaceuticals using “real-world evidence,” including medical billing claims.^22,23 However, there is still uncertainty in how reliable claims data are for assessing clinical events.

Our study helps clarify some of the potential strengths and limitations of using claims in place of events adjudicated by a CEC in a large, randomized, regulatory placebo-controlled clinical trial. One prior study by Hlatky et al. evaluated claims in two clinical trials that were part of the Women’s Health Initiative and showed moderate agreement with adjudicated MI events (kappa = 0.71 to 0.74) with a moderate PPV of 71%.¹² Other studies have evaluated MI events from claims using investigator adjudication of observational data rather than a CEC. These showed very high PPV of claims (87.1%−96.7%) for identifying an MI event. ^{15, 24, 25} We further show that claims underestimates cumulative incidence of MI compared to adjudicated events in a randomized clinical trial. These findings are similar to a prior study showing that MI incidence is lower in claims data compared to adjudication by researchers from the TRANSLATE-ACS observational study.¹³ Underestimation of MI using claims data should therefore be considered in future trials as well as observational studies that use claims to assess MI.

Performance of claims data in identifying stroke events from our study was more modest, identifying only 61.5% of all adjudicated strokes with a PPV of 47.1%. In prior studies, the PPV of claims have been over 60% and as high as 93% in assessing stroke.^{13, 14, 26-28} The poorer performance in our study may in part be related to relatively small number of stroke events observed. Overall agreement between claims was also modest with a kappa of 0.52, which is similar to the kappa of 0.55 seen for stroke in TRANSLATE-ACS.¹³ Though cumulative incidence of stroke over time appears to be similar between claims and adjudicated events in our study, these results should be interpreted with caution given the low number of strokes.

Among all non-death outcomes assessed, claims data had the highest sensitivity for identifying adjudicated major bleeding events (76.8%) but also significantly overestimated cumulative incidence and had poor PPV (39.3%). In the TRANSLATE-ACS study, one-year incidence of bleeding from claims was lower than for adjudicated bleeding (5.0 vs. 5.4%) and agreement was very poor (kappa = 0.24 for any hospitalized bleeding).¹³ Our study utilized a much larger list of codes associated with bleeding including procedure codes for blood transfusions, which likely resulted in the relatively higher incidence of bleeding and improved agreement (kappa = 0.49) with adjudicated events. Our findings are similar to a recent study comparing Medicare claims data to physician adjudication in patients who received transcatheter mitral valve repair,¹⁶ which found a sensitivity of 84% and a PPV of 37% for adjudicated bleeding events using claims.

Our study has several implications. First, deaths were perfectly matched in trial and Medicare data, suggesting that mortality data from CMS is highly accurate and could be used to more efficiently track death of Medicare patients in clinical trials. Second, given the variable performance of claims for other adjudicated outcomes, investigators should be cautious in how such data may be used to evaluate endpoints in future clinical trials. Even so, claims successfully identified more than 60% of adjudicated events for each outcome. Third, specificity and negative predictive values of ICD codes for all outcomes we assessed was very high. Therefore using ICD codes associated with hospitalizations may provide an efficient method to screen for potential events that can then be adjudicated with further review of medical records. This is particularly relevant given that traditional trial adjudication may miss events, such as when patients cannot be contacted, are unable to provide information on medical facilities where events may have occurred, or records cannot otherwise be obtained by study personnel.¹² There could also be potential cost-savings with this type of approach, though potential missed events in claims as well as delays in obtaining claims data would need to be considered.

Fourth, our study suggests that claims-based endpoints should use both primary and secondary diagnosis codes, since primary codes alone may miss many adjudicated events. Lastly, two different analytic algorithms designed to select a more parsimonious list of codes did not consistently improve performance of claims data in assessing adjudicated events. Therefore using comprehensive lists of codes based on clinical plausibility may be the most useful strategy for identifying cardiovascular events and bleeding in the CAD population, though findings may differ in larger cohorts.

There are several important limitations of our study. Our cohort only included U.S. patients ≥65 years old enrolled in fee-for-service Medicare, and represented a small subgroup of the overall DAPT Study population. Therefore our findings may not be generalizable to other populations, including those who would not meet selection criteria of the DAPT Study. Medicare data also includes reliable and comprehensive follow-up data on hospitalizations for enrolled patients. Unlike in Medicare, patients with private insurance frequently do not have continuous coverage under one insurer for an extended duration of time. Consequently, our findings cannot necessarily be extrapolated to patients under the age of 65 with private insurance, for which accuracy of follow-up data using claims is less certain. Furthermore, we only counted hospitalizations as events and did not have data for emergency department visits or short-term observation stays which may have resulted in missed adjudicated events. The number of events in our study was also relatively small, which may have affected the performance of specific claims codes. This study used only ICD-9 codes and results may differ with ICD-10 codes. Though most diagnoses are similar between the two classifications, further studies will be needed to assess accuracy of codes currently in use. Lastly, even though generally considered the gold standard, adjudicated trial events may not always be perfectly accurate. This should be considered in the interpretation of our findings.

CONCLUSION

Use of Medicare claims to identify events in the DAPT Study had variable agreement with events identified by CEC-adjudication. The occurrence of death was equivalent between claims and trial adjudication, and there was modest agreement for MI events. Claims data performed less well in assessing stroke and bleeding events. While using claims data may be an efficient way to identify outcomes in clinical trials, additional monitoring and review may still be needed for non-death outcomes to ensure events are reported accurately.

WHAT IS KNOWN

• Randomized controlled clinical trials are needed to determine effectiveness of therapies in cardiovascular medicine, though ensuring accurate follow-up is often expensive and inefficient.

• Administrative claims data using billing codes are widely available and relatively easy to obtain for assessment of clinical outcomes, but little is known about how claims-based events compare to adjudicated events in the context of clinical trials.

WHAT THE STUDY ADDS

• Compared to adjudication of events by the Clinical Events Committee (CEC) in the Dual Antiplatelet Therapy (DAPT) Study comparing 30 vs. 12 months of DAPT after percutaneous coronary intervention, administrative claims data in Medicare had moderate agreement for myocardial infarction and poor agreement but high specificity for bleeding and stroke.

• Death events adjudicated by the DAPT Study CEC were identified equivalently in Medicare data.

• Using claims data in clinical trials could be a potentially efficient way to assess outcomes in follow-up, though additional monitoring is likely needed to ensure accurate assessment of events.

Abbreviations:

RCT

randomized control trial

CEC

Clinical Events Committee

CAD

coronary artery disease

DAPT

Dual Antiplatelet Therapy

PCI

percutaneous coronary intervention

MI

myocardial infarction

CMS

Centers for Medicare and Medicaid Services

MACCE

major adverse cardiovascular and cerebrovascular events

ICD-9

International Classification of Diseases, 9^th Revision