A Comparison of Reintervention Rates After Endovascular Aneurysm Repair Between the Vascular Quality Initiative Registry, Medicare Claims, and Chart Review

Abstract

Objectives:

The accurate measurement of reintervention after endovascular aneurysm repair (EVAR) is critical during postoperative surveillance. The purpose of this study was to compare reintervention rates after EVAR from three different data sources: the Vascular Quality Initiative (VQI) alone, VQI linked to Medicare claims (VQI-Medicare), and a gold standard of reviewing clinical chart review supplemented with telephone interviews.

Methods:

We reviewed the medical records of 729 patients who underwent EVAR at our institution between 2003 to 2013. We excluded patients without follow up reported to VQI (n=68, 9%) or without Medicare claims information (n=114, 16%). All patients in the final analytic cohort (n=547) had follow up information available from all three data sources (VQI alone, VQI linked to Medicare, and chart review). We then compared reintervention rates between the three data sources. Our primary endpoints were the agreement between the three data sources, and the Kaplan-Meier estimated rate of reintervention at 1, 2, and 3 years after EVAR. For gold standard assessment we supplemented chart review with telephone interview as necessary to assess reintervention.

Results:

VQI data alone identified 12 reintervention events in the first year after EVAR. Chart review confirmed all 12 events, and identified 18 additional events not captured by VQI. VQI-Medicare data successfully identified all 30 of these events within the first year. VQI-Medicare also documented four reinterventions in this time period that did not occur based on patient interview (4/547, 0.7%). The agreement between chart review and VQI-Medicare data at 1-year was excellent (kappa=0.93). At 3-years, there were 81 (18%) reinterventions detected by VQI-Medicare and 70 (16%) detected by chart review, for a sensitivity of 92%, a specificity of 96%, and kappa of 0.80. Kaplan-Meier survival analysis demonstrated similar reintervention rates after 3-years between VQI-Medicare and chart review (log rank p=0.59).

Conclusions:

Chart review after EVAR demonstrated a 6% one-year and 16% three-year reintervention rate, and almost all (92%) of these events were accurately captured using VQI-Medicare data. Linking VQI data with Medicare claims allows for an accurate assessment of reintervention rates after EVAR without labor-intensive physician chart review.

Introduction:

Over 30,000 patients undergo elective endovascular aneurysm repair (EVAR) in the United States each year.¹ Late results from early randomized trials and Cochrane reviews suggest that 20–30% of patients need one or more additional interventions after their initial endovascular repair,^2–4 and this need for reintervention does not appear to plateau over time.^{2, 5} These findings indicate that the number of reinterventions following elective EVAR will likely continue to rise, highlighting the need for diligent postoperative surveillance. As such, it is imperative to develop reliable and scalable means of tracking patients who have undergone EVAR, and follow their long-term outcomes.

However, the current method of follow up – relying on patients and surgeons to achieve this goal – has demonstrated limitations. Current reports suggest nearly one in three patients undergoing EVAR is lost to follow up within the first three years.^6–9 Combining data from vascular registries, such as the Vascular Quality Initiative (VQI),¹⁰ with Medicare claims¹¹ may offer a solution to this challenging problem. The VQI registry was created to allow surgeons to follow procedure specific outcomes and provide clinically relevant information to patients and physicians regarding their vascular care. However, the VQI registry is only designed to capture one-year outcomes – not long enough to provide adequate surveillance after EVAR – and follow up continues to be challenging for many surgical practices.⁹ Conversely, Medicare claims data offers the advantage of long-term follow up for a large number of patients, and can identify procedures performed at different institutions. However, it can be difficult to accurately identify patients and clinical events using the diagnostic and procedural codes implemented for billing. Furthermore, the number and type of billing codes used to identify events impacts the accuracy of event detection,¹² and has major implications for the interpretation of study findings. We hypothesized that VQI data linked to Medicare claims could provide an accurate means by which to assess outcomes in patients undergoing EVAR. To test this hypothesis, we compared the rate of reintervention found within a combined dataset of VQI registry data linked to Medicare claims (VQI-Medicare) against the rate found on retrospective chart review at our own institution.

Methods:

Cohort creation

We identified all patients who underwent EVAR at our institution from January 2003 to December 2013 using the VQI registry.¹⁰ This method has been shown on internal review to capture 98–100% of EVAR procedures performed at our institution (unpublished data). We then performed chart reviews on all patients to identify reintervention events. All reinterventions were adjudicated by two reviewers (JC, PG).

We then obtained the corresponding Medicare claims information for patients who were Medicare eligible. Medicare follow up data were available from January 2003 to December 2013. Patients identified by the VQI registry were then linked to their respective Medicare claims file. We linked patients using an indirect matching method described previously.⁵ Briefly, patients in the VQI registry were identified in Medicare claims data using a series of non-unique identifiers (e.g. procedure date, date of birth, zip code) to create unique patient-level matches between the registry and the Medicare claims file.

We then created a cohort of patients for whom follow up information was available in all 3 data sources (VQI registry alone, VQI-Medicare, and chart review). This allowed us to capture the date and type of all procedures performed after the index procedure (EVAR). From this we were able to assess the concordance of reintervention rates after EVAR between the 3 different data sources. When datasets were discordant, we conducted telephone interviews with patients to assess whether a reintervention had occurred.

Primary outcomes

Our primary outcome measure was reintervention after EVAR. We defined reintervention as any additional procedure performed following the index hospitalization to treat endoleaks, further aneurysmal degeneration, or any complications related to the original repair (e.g. femoral artery repair for access site complication). Any subsequent procedures performed for pre-existing conditions (e.g. for an endovascular repair of a popliteal artery aneurysm) were not included as reinterventions. We compared the rate of reintervention between VQI-Medicare and chart review at 1, 2, and 3 years using Kaplan-Meier estimation. We assessed the concordance of reintervention rates using Cohen’s kappa.

Medicare coding algorithm

We created a list of primary diagnosis and procedure codes using the international classification of diseases, ninth revision (ICD-9) to identify reintervention events. The initial list of codes compiled was based on prior work at our institution and others (Supplementary Table 1).^{5, 6, 13} Medicare billing code events identified within the VQI-Medicare linked dataset were then compared to the reintervention events identified on chart review to determine if billing events represented true reinterventions. If the occurrence of a reintervention was unclear, patient telephone interviews were conducted.

We then calculated the accuracy of each individual billing code by determining the percentage of billing events representing a true clinical event (e.g. a specific billing code appearing 10 times, but only representing a true reintervention 4 times, would have 40% accuracy). We determined the accuracy for all codes appearing in the first year following EVAR. If a billing code was associated with less than 50% of true clinical events, it was considered for removal from our list of codes used to represent reintervention. Details of the revisions to the coding algorithm are described in the Supplementary Appendix and Supplementary Table 2.

Statistical analysis

We report absolute numbers and percentages where appropriate. Continuous variables are represented as means with standard deviations, and categorical variables are listed as percentages. The final cohort for analysis represents the same group of patients, differing only by the data source from which reintervention events are identified; therefore, no comparative statistics on baseline characteristics were calculated. Rates of reintervention were calculated using Kaplan-Meier survival analysis with hazard function estimation. In addition, because the cohorts compared represent the same patients, and differ only in which data source was used to assess the rate reintervention, the at-risk number at each time point is the same for the datasets being compared. Concordance between the reintervention rates obtained from the 3 datasets was analyzed using Cohen’s kappa. We also calculated the sensitivity and specificity of VQI-Medicare linked data to identify a reintervention event when compared to chart review. All statistical analyses were performed using Stata version 14 software (College Station, Texas).

Human subjects protection

Medical record review and patient interviews for this study were approved by the Committee for the Protection of Human Subjects at Dartmouth. All patient personal health information was protected, records and outcomes were de-identified, and no testing or procedures were required for this study. Thus, the need for specific consent was waived. VQI and Medicare information is collected under the auspices of an Agency for Healthcare Research and Quality designated Patient Safety Organization. Therefore, this portion of the study was exempt from internal review.

Results:

Details of the analytic cohort

We identified 729 patients who underwent EVAR at our institution during the study period. We excluded 68 patients (9%) who did not have VQI follow up available, and 114 patients (16%) for whom Medicare claims data were not available (71 patients <65 years of age, 43 not matched to their respective Medicare claims file).

All (100%) of the remaining 547 patients had follow up information available from each of the 3 data sources. This group of patients formed the final analytic cohort which was used to compare rates of reintervention and examine concordance between our data sources. Cohort characteristics were typical for this patient population (Table 1), with a mean age of 75.5 years, and 22.9% being female. Hypertension and smoking history were common, and most patients were on preoperative aspirin and statin therapy. More than 90% of EVARs were performed electively.

Table 1:

Characteristics of the analytic cohort.

Variable	n=547
Age, mean (SD), years	75.5 (7.3)
Female	22.9%
BMI, mean (SD), kg/m	27.7 (5.1)
Hypertension	84.0%
Coronary Artery Disease	35.5%
Congestive Heart Failure	14.6%
Chronic Obstructive Pulmonary Disease	42.6%
History of Smoking	83.8%
Diabetes	22.1%
Creatinine >1.8 mg/dL	8.8%
Preoperative Aspirin	74.3%
Preoperative Statin	65.4%
Elective Operation	90.7%

SD, standard deviation; kg/m, kilograms per meter squared; mg/dL, milligrams per deciliter

Rates of reintervention

The Kaplan-Meier estimated 1-year rate of reintervention after EVAR using the VQI registry alone was 3%, corresponding to a total of 12 events. As VQI is designed to collect 1-year outcomes, we truncated survival estimates using the VQI registry alone at this time point. The estimated rate of reintervention found on chart review was twice that found using the VQI registry alone, showing a 1-year reintervention rate of 6%, or 30 events. The chart review rate of reintervention maintained a nearly linear increase over the study period, and was 16% at 3 years. The rate of reintervention found using VQI-Medicare was 6% at 1 year, and 18% at 3 years. This rate was calculated using a coding algorithm comprised of the codes outlined in Table 2.

Table 2:

List of ICD-9 codes used in the revised algorithm

3804	3846	3929	3971	4415	9966
3806	3864	3930	3972	4400	99660
3808	3866	3931	3974	4442	99661
3814	3868	3951	3975	44421	99662
3816	3884	3952	3976	4448	99669
3818	3886	3954	3977	44481
3834	3888	3956	3978	44489
3838	3891	3957	3979	99674
3844	3925	3958	3990	9961
3846	3926	3959	4413	99659

ICD-9, international classification of diseases, ninth-revision.

Concordance between the datasets

Using the VQI registry alone, 12 reintervention events were identified within the first year following EVAR. Chart review confirmed these 12 events, and identified an additional 18 events not captured by the VQI registry. The additional events found by chart review but not identified using the VQI registry consisted of EVAR limb thrombectomy or repair of a kinked EVAR limb (n=4), femoral artery reconstruction or femoral-femoral bypass for an occluded EVAR limb (n=4), unsuccessful reintervention procedures (n=3), proximal aortic cuff placement (n=3), coiling for type 2 endoleak (n=1), or patients who were deceased at another hospital and thought to have suffered an aneurysm-related mortality event (n=3). Full details of the reintervention procedures are described in Supplementary Table 3.

Both the baseline and revised coding algorithms of VQI-Medicare linked data captured all 30 of the events found on chart review during the first year. However, the baseline coding algorithm identified 35 additional events within the first year that did not represent a true reintervention. Changes were made in the coding algorithm to generate our revised coding algorithm as detailed in Methods above and Supplementary Appendix. These changes improved the concordance between chart review and VQI-Medicare significantly, with the revised coding algorithm now identifying only 4 events within the first year that did not represent a true reintervention based on chart review and patient telephone interview. The statistical agreement between chart review and VQI-Medicare before the coding changes was 0.63 at both 1 and 3 years, as determined by Cohen’s kappa. However, after changes to the coding algorithm, this improved to 0.93 and 0.80 at 1 and 3 years, respectively, indicating excellent agreement. Using chart review as the gold standard, the sensitivity of VQI-Medicare to identify a reintervention event improved from 90.3% to 91.9%, and specificity improved from 86.7% to 96.1% at 3 years (Figure 1).

Figure 1:

Reintervention rates and concordance between chart review and VQI-Medicare: Baseline and revised coding algorithms. VQI, vascular quality initiative, VQI-Medicare, vascular quality initiative data linked to Medicare claims. Standard error <10% for all reported statistics.

Discussion:

Our review of a single center series using multiple data sources to evaluate reinterveiton after EVAR demonstrated two important findings. First, more than one in five patients whose abdominal aortic aneurysm is treated by endovascular means can expect to undergo reintervention, and this need for reintervention does not plateau over time. Second, our study suggests that a linked clinical-claims registry may offer a scalable, reliable, accurate, and cost-effective way to provide long-term surveillance for reintervention after EVAR.

While our long-term rate of reintervention after EVAR was nearly identical to the rate reported in the EVAR-1 trial,² we also found that the rate of reintervention in our cohort was highly dependent on the codes chosen to represent events from the Medicare claims data. Initially, our coding algorithm was highly sensitive, but had poor specificity, greatly overestimating the true reintervention event rate. For example, our initial list of codes used for event detection in Medicare included ICD-9 code 3893 “venous catheterization not elsewhere classified.” We included this code because we hypothesized that it would be associated with transcaval coil embolization, a common method of treating type 2 endoleaks at our institution. This billing code appeared 21 times in our institutions Medicare data. However, it was only associated with a reintervention found on chart review in 1/21 cases, and in this case, was also associated with another ICD-9 code used for event detection. We therefore removed ICD-9 code 3893 from our list of codes used to represent reintervention events. By performing adjudications of billing codes such as this, we were able to modify our coding algorithm such that the specificity of reintervention events found using the VQI-Medicare database improved from 86.7% to 96.1% at 3 years, while maintaining high sensitivity (90.3% to 91.9% at 3 years). Similarly, the concordance of reintervention rates found using chart review versus VQI-Medicare database was high (0.93 and 0.80 at 1 and 3 years, respectively) following coding algorithm revision.

Challenges with coding accuracy at both the billing code entry,¹⁴ and research use levels,^{12, 15, 16} have been described by many investigators across specialties. Even within vascular surgery, coding algorithms to define clinical events such as stroke can be difficult to define,^{15, 17} and may have a profound impact on study results. These findings in concert with ours highlight the need for researchers to carefully select codes to represent true events when using Medicare data, and perform chart-level adjudication of billing codes to ensure accuracy.

We noted a substantial difference in the number of events detected by the VQI dataset alone and those found on chart review. These events most often represented complications related to EVAR, such as EVAR limb thrombectomy, or femoral artery reconstruction for access site complications. These procedures may have been errors in data entry (EVAR limb thrombectomy), or have been overlooked as related to the index EVAR (femoral artery reconstruction). These events represent opportunities for improvement for data entry into VQI when considering postoperative surveillance for EVAR.

Our findings have important implications. First, the cumulative incidence of reintervention after EVAR demonstrates a linear increase over time. This finding is consistent with long-term results from randomized trials,² and may account for the inferior outcomes associated with patients who are lost to follow up.⁸ Furthermore, this indicates that patients who undergo EVAR must have long-term surveillance, as the rate of reintervention does not appear to plateau. The method described in our report, which leverages registry data and Medicare claims, may be a cost-effective approach for a distributed surveillance network to evaluate EVAR performance over time. This linked registry-claims surveillance system is both sensitive and specific. It also offers a scalable mechanism which can identify reintervention events occurring at either the index or outside institutions for Medicare patients. Finally, it also offers a reliable method to monitor mortality from rupture, even after EVAR, across the United States. These attributes of the linked registry represent an important advance over VQI data taken in isolation.

Our study has limitations. It is an experience from a single center, and as such, Medicare coding trends from our institution may not be representative of those at other hospitals. This limitation highlights the need for a multi-center validation project, which we are currently undertaking. Medicare coding events were compared to retrospective chart review. The optimal comparison would be prospectively collected data with blinded evaluation of reintervention events. However, no such source is available for use with the VQI registry. Therefore, we felt that our two-reviewer retrospective method of event adjudication provided the most reliable information possible. Our cohort was limited to patients who were found in all 3 data sources, and because of this we cannot comment on coding trends for patients who are not Medicare eligible. We adjudicated chart review events against billing codes during the first year, using our findings to revise our coding algorithm. We then applied these changes to the 3 years of data. We did not feel that it was necessary to adjudicate all 3 years of events for a series of reasons. First, our Kappa concordance remained excellent (>0.8) for all years analyzed. Second, the sensitivity and specificity of VQI-Medicare compared to chart review was 92% and 96% respectively. Finally, our findings closely resembled those of randomized clinical trials. Although our sensitivity and specificity remained excellent (92% and 96% respectively), we were not able to obtain perfect 100% agreement. However, given the known limitations of claims data, perfect agreement is likely not possible.

Conclusions:

VQI data linked to Medicare claims closely mirrors chart review when evaluating reintervention after EVAR, and the rates of reintervention we found were similar to those published in randomized clinical trials. Furthermore, VQI-Medicare was 92% sensitive and 96% specific in identifying a true reintervention event. The rate of events found in Medicare claims was highly dependent on the billing codes chosen to represent those events. Only after close adjudication and iterative revisions of our coding algorithm did rates become similar, highlighting the care that must be taken when using Medicare data for clinical research. Nevertheless, VQI-Medicare represents a validated and accurate assessment of reintervention after EVAR, without the need for labor intensive chart review.