A comparative evaluation of risk-adjustment models for benchmarking amputation-free survival after lower extremity bypass

Abstract

Background

Providing patients and payers with publicly reported risk-adjusted quality metrics for the purpose of benchmarking physicians and institutions has become a national priority. Several prediction models have been developed to estimate outcomes after lower extremity revascularization for critical limb ischemia, but the optimal model to use in contemporary practice has not been defined. We sought to identify the highest-performing risk-adjustment model for amputation-free survival (AFS) at 1 year after lower extremity bypass (LEB).

Methods

We used the national Society for Vascular Surgery Vascular Quality Initiative (VQI) database (2003–2012) to assess the performance of three previously validated risk-adjustment models for AFS. The Bypass versus Angioplasty in Severe Ischaemia of the Leg (BASIL), Finland National Vascular (FINNVASC) registry, and the modified Project of Ex-vivo vein graft Engineering via Transfection III (PREVENT III [mPIII]) risk scores were applied to the VQI cohort. A novel model for 1-year AFS was also derived using the VQI data set and externally validated using the PIII data set. The relative discrimination (Harrell c-index) and calibration (Hosmer-May goodness-of-fit test) of each model were compared.

Results

Among 7754 patients in the VQI who underwent LEB for critical limb ischemia, the AFS was 74% at 1 year. Each of the previously published models for AFS demonstrated similar discriminative performance: c-indices for BASIL, FINNVASC, mPIII were 0.66, 0.60, and 0.64, respectively. The novel VQI-derived model had improved discriminative ability with a c-index of 0.71 and appropriate generalizability on external validation with a c-index of 0.68. The model was well calibrated in both the VQI and PIII data sets (goodness of fit P = not significant).

Conclusions

Currently available prediction models for AFS after LEB perform modestly when applied to national contemporary VQI data. Moreover, the performance of each model was inferior to that of the novel VQI-derived model. Because the importance of risk-adjusted outcome reporting continues to increase, national registries such as VQI should begin using this novel model for benchmarking quality of care.

In anticipation of health care reform, the Agency for Healthcare Research and Quality (AHRQ) convened a panel of experts in 2009 tasked with developing methods for better quantifying the “quality, safety, efficiency, effectiveness, and equity of care.”¹ At the same time, several agencies and professional societies have developed resources for tracking and reporting such measures. Patient Safety Organizations (PSO) provide one step toward reaching this goal shared by policymakers and physicians. The Society for Vascular Surgery (SVS) PSO was approved by AHRQ in 2011, and the resultant Vascular Quality Initiative (VQI) database tracks outcomes for vascular surgical patients at the national level. The VQI represents an important opportunity to measure and document quality of care and patient outcomes. Specifically, the VQI can provide risk-adjusted estimates of procedure-specific outcomes, which are invaluable in assessing quality and safety of patient care as well as in identifying possible opportunities for future quality improvement initiatives.

Although several existing scoring systems stratify critical limb ischemia (CLI) patients according to risk of major amputation or death, the VQI currently lacks a statistical model for risk adjustment of outcomes after lower extremity bypass (LEB). The Bypass versus Angioplasty for Severe Ischaemia of the Leg (BASIL) trial included derivation of a prediction model for 1-year and 2-year survival.² The Finland National Vascular (FINNVASC) registry was used to develop a model for predicting risk of perioperative death or major amputation.³ The Project of Ex-vivo vein graft Engineering via Transfection III⁴ (PREVENT III [PIII]) cohort was used to derive a prediction model for 1-year amputation-free survival (AFS).⁵ These prediction models (BASIL, Finnvasc, and PIII) have each been externally validated.^6–8 Their relative performance was also tested in two single-institution retrospective studies in Europe.^7,9 The relative performance of these prediction models in a multicenter cohort in the United States has not been evaluated.

An analysis of these prediction models would greatly inform the process of selecting the optimal risk-adjustment tool as the VQI continues to move forward and provide valuable deliverables for its members. Although each model used a slightly different end point, there is considerable overlap in the variables included, suggesting that they can be applied to one unified, clinically relevant end point that characterizes quality revascularization for CLI. We used the VQI data set to externally validate these three well-known risk scoring systems (BASIL, FINNVASC, PIII) for predicting 1-year AFS among CLI patients and to quantify their performance in this contemporary United States data set.

In addition, because the VQI currently lacks a statistical model for risk-adjusting outcomes after LEB, each of these models could be considered for this application. FINNVASC and PIII were intended as preoperative risk scores, but this type of regression model (or that from BASIL) may adequately provide risk adjustment in VQI. However, if none of these perform well when used for this application in VQI, then a novel model should be derived in VQI and validated externally. Because the existing models require modification from their initial designs to be tested in VQI, their discrimination may be diminished. Therefore, a fourth novel model was derived and tested using VQI data and then externally validated using multicenter data from the PIII data set. In accordance with the goals of the VQI, the goal of this study was to compare the performance of all four models for predicting 1-year AFS after LEB to inform the VQI’s selection of a risk-adjustment model for quality reporting.

METHODS

Study cohort

Prospectively collected data from the VQI database was retrospectively reviewed to identify all patients who underwent infrainguinal LEB between January 1, 2003, and December 31, 2012. The VQI is a national cooperative quality improvement initiative developed to prospectively collect data and outcomes for patients undergoing vascular surgical procedures. It consists of >1300 physicians from >300 academic and community medical centers across the United States. Details of this registry are available online (www.vascularqualityinitiative.org). Data are physician-reported at the time of operation and include preoperative, intraoperative, and in-hospital postoperative details. Follow-up data are entered at ~1 year postoperatively. All information is sent to a central data repository where it is aggregated and audited. Research analysts are blinded to patient, surgeon, and hospital identity.

The VQI registry was queried for patients who underwent LEB between 2003 and 2012 for CLI. CLI was defined as rest pain or tissue loss, or both. Lower extremity bypass included open infrainguinal revascularization procedures: femoral-popliteal bypass, femoral-tibial bypass, and distal origin bypass. Patients who underwent emergency procedures were excluded, as were patients with missing data pertaining to the end points of major amputation or death. Use of the VQI data set was reviewed by the University of Massachusetts Medical School Institutional Review Board and deemed exempt, and patient consent was not required.

Covariates examined

Patient information for >100 clinical and demographic variables (available at www.vascularqualityinitiative.org) was collected. Demographic information included age at the time of the procedure, sex, and race. Comorbidities examined included coronary artery disease (history of myocardial infarction without current symptoms, stable angina, or unstable angina or myocardial infarction within the past 6 months), chronic obstructive pulmonary disease (dependent on medication or home oxygen), congestive heart failure (by history), diabetes mellitus (diet controlled and dependent on medication), hypertension (history of hypertension or blood pressure ≥140/90 mm Hg on the preoperative evaluation), and history of tobacco use (never, <1 year prior, or current). Renal disease was categorized in three strata: normal (serum creatinine ≤1.8 mg/dL), renal insufficiency (serum creatinine >1.8 mg/dL), and dialysis dependent; serum creatinine in micromoles was reported as a continuous variable also. Height in meters (m) and weight in kilograms (kg) was collected, and body mass index (BMI) was calculated as kg/m². Weight was grouped according to quartiles: ≤66, 67–77, 78–91, and >91 kg.

History of previous coronary revascularization included coronary artery bypass graft or percutaneous coronary intervention, or both. History of previous arterial bypass and previous angioplasty or stent was included. Previous major amputation included above-knee and below-knee operations. Information on functional status included preoperative ambulation status (independent, with assistance, wheelchair-bound, or bedbound) and living situation (nursing home, or at home/homeless). Technical factors included urgency of the procedure (elective or urgent), recipient vessel (above-knee popliteal, below-knee popliteal, or tibial/other), and conduit (prosthetic or great saphenous vein). Medication use at discharge was recorded, including β-blockers, antiplatelet agents (aspirin, clopidogrel, or both), and statins. Information on the presence of infection and characteristics of ulceration was not available. Tissue loss was categorized only as present or absent.

End points

The primary end point was AFS, a composite end point defined as freedom from ipsilateral major amputation and freedom from all-cause mortality at 1-year follow-up. Major amputation was defined as any amputation above the ankle. Although the FINNVASC and BASIL models were not originally derived and validated for 1-year AFS, they were all tested using this end point. These scores have been tested for several end points other than those originally described, with acceptable performance. FINNVASC was externally validated for 1-year AFS.⁹ BASIL was externally validated for 1-year survival.⁷

Application of prediction models

The FINNVASC score required no modifications. Modifications were made to the other models in accordance with the variables available in the VQI cohort. The PIII score required no further modifications than those described in a previous publication (mPIII).⁶ The BASIL model required omission of the Bollinger score, history of transient ischemic attack or stroke, and number of ankle pressure measurements obtained because those variables are not captured in VQI. The three models were then applied to the VQI cohort, using the published point assignments (mPIII and FINNVASC) or β-coefficients (BASIL) to model 1-year AFS.

Derivation of a VQI model

A univariate screen of the covariates listed in Table I was performed, and those with P < .2 were incorporated into a multivariable model with backwards selection for 1-year AFS. The entire cohort was used to construct the model to maximize sample size because external validation was planned. Because the model’s intended use was risk adjustment rather than prediction modeling, integer scores and categories of risk were not assigned.

Table I.

Patients who underwent lower extremity bypass (LEB) for critical limb ischemia (CLI) in the Vascular Quality Initiative (VQI)

Covariates	No. (%) (N = 7750)
Demographics
Age, years
<60	1895 (24)
60–69	2334 (30)
70–79	2162 (28)
≥80	1359 (18)
Male sex	4997 (65)
Race
White	6503 (84)
Nonwhite	1247 (16)
Preoperative factors
Smoking status
Never	1394 (18)
Prior history	3187 (41)
Current	3157 (41)
Coronary artery disease
None	5105 (66)
History of MI, asymptomatic	1835 (24)
Stable angina	635 (8.2)
Unstable angina/MI ≤6 months	165 (2.1)
History of coronary revascularization	2653 (34)
Congestive heart failure	1444 (19)
Hypertension	6927 (89)
Diabetes mellitus	4333 (56)
Chronic obstructive pulmonary disease	1964 (25)
Renal function
Creatinine ≤1.8 mg/dL	6553 (92)
Creatinine >1.8 mg/dL	539 (7.6)
Dialysis dependent	593 (8.0)
Previous
Arterial bypass (any)	2566 (33)
Angioplasty or stent (ipsilateral)	2876 (37)
Major amputation (contralateral)	485 (6.2)
Living status
Home/homeless	7413 (96)
Nursing home	321 (4.2)
Ambulation status
Independent	5361 (69)
With assistance	1843 (24)
Wheelchair-bound	477 (6.2)
Bedbound	48 (0.6)
Technical factors
Urgency
Elective	5682 (73)
Urgent	2068 (27)
Prosthetic bypass conduit	2605 (34)
Single-segment great saphenous vein	4767 (62)
Target vessel
Above-knee popliteal	1381 (19)
Below-knee popliteal	2281 (31)
Tibial/other	3799 (51)
Medications at discharge
Aspirin or clopidogrel	6814 (88)
Statin	5475 (77)
β-blocker	5085 (71)

MI, Myocardial infarction.

External validation of the VQI model

The PIII data set was used for external validation to maximize generalizability to a United States population. The FINNVASC and BASIL data sets were derived from non-United States samples, unlike the VQI data set, and were not used for external validation. This cohort consisted of 1404 patients who underwent LEB using autogenous conduit for CLI between 2001 and 2003. Details of the cohort have been published elsewhere.⁴ To test the VQI model, certain variables were omitted because they were not collected in the PIII study. This included prosthetic conduit, ambulation status, procedure urgency, and congestive heart failure.

Statistical analysis

Descriptive statistics were used to characterize the cohort. Univariate analysis was performed to compare the demographic and comorbidities between those patients in the cohort and those who had been excluded for lacking end point data. One-year AFS was modeled by applying the points (or β-coefficients for BASIL) systems to construct Cox proportional hazards models. After applying the models to the VQI cohort, discrimination was tested using the Harrell c-index. Where calibration could be assessed, the Hosmer-May goodness-of-fit test was used, and if necessary, observed and expected outcomes were compared in calibration tables. Discrimination of the VQI-derived model was also assessed using the c-index. The Harrell c-index is a measure of the model’s predictive accuracy, similar to the C statistic, or area under the receiver operating characteristic curve, but appropriate for time-to-event analyses with censoring. The Hosmer-May goodness-of-fit test evaluates how well calibrated the model is by testing if a variable representing the risk groups themselves adds predictive ability. If this test is statistically significant, then the model does not perform equally well across risk groups and is not well calibrated. Analyses were conducted using SAS 9.2 software (SAS Institute Inc, Cary, NC).

RESULTS

Study cohort

We identified 7750 patients who underwent nonemergency LEB for CLI (Table I). Most patients were male (4997 [65%]), white (6503 [84%]), and 60 to 80 years old (5855 [76%]). Tobacco exposure was prevalent (6344 [82%]). More than one-third of patients (2876 [37%]). had already been treated with a prior ipsilateral endovascular intervention, and 51% (3799) of the bypasses were to a tibial vessel. Single-segment great saphenous vein was used in 4767 (62%) patients. Excluded were 340 patients (4.2% of the total cohort) for lacking end point data. Univariate analysis revealed that this excluded group was older, with more coronary artery disease, congestive heart failure, and chronic kidney disease. The effect of these differences on AFS could not be determined, but given their higher risk characteristics, it is possible that the calculated AFS may represent a slight underestimate.

AFS

The overall 1-year AFS was 74% (Fig 1). On analysis of the individual components comprising this composite end point, the 1-year freedom from major amputation was 86% and the 1-year survival was 89%.

Fig 1.

Kaplan-Meier curve shows amputation-free survival (AFS) of 74% at 1 year (N = 7750).

Application of prediction models

Modified PIII score

When the mPIII score was applied to the cohort, 7737 patients were included in the model (13 were excluded for lacking relevant covariates). The distribution of scores was similar to that seen in the validation study, with 5370 patients (69%) in the low-risk category (score 0–3). The score discriminated well based on significantly different survival between the three risk groups (log-rank P < .0001; Fig 2). The group-specific risk estimates were similar to the original published results also, with 1-year AFS of 37% in the high-risk, 65% for medium-risk, and 80% for the low-risk groups. The c-index for the model was 0.63.

Fig 2.

Kaplan-Meier curve shows amputation-free survival (AFS) over 1 year, stratified by modified Project of Ex-vivo vein graft Engineering via Transfection III (mPIII) risk group (n = 7737).

FINNVASC score

When the FINNVASC score was applied to the cohort, 7738 patients were included in the model (12 were excluded for lacking relevant covariates). Most patients were distributed between scores of 1 (2295 [30%]) and 2 (2850 [37%]). The group-specific risk estimates of 1-year AFS were 85% for score = 0, 79% for score = 1, 71% for score = 2, 63% for score = 3, and 63% for score = 4 (Fig 3). The c-index was 0.59.

Fig 3.

Kaplan-Meier curve shows for amputation-free survival (AFS) over 1 year, stratified by the Finland National Vascular (FINNVASC) registry risk score (n = 7738).

BASIL score

When the modified BASIL score was applied to the cohort, 7730 patients were included in the model (20 were excluded for lacking relevant covariates). Because this is not a discrete scoring system like the two models above that identify specific strata of risk, groups cannot be described and plotted. However, a c-index for the model can still be calculated and was 0.65. The May-Hosmer goodness-of-fit test P value was .6, and the calibration table showed sufficient agreement between observed and expected number of patients who died or underwent major amputation by 1 year within deciles of risk.

Derivation of a VQI AFS model

Univariate screen of the variables listed in Table I led to identification of 15 factors to be tested in the multivariable model (Table II). When entered into a Cox regression model of 1-year AFS, bedbound status (hazard ratio, 4.4; 95% confidence interval, 2.7–7.0), followed by dialysis dependence (hazard ratio, 2.5; 95% confidence interval, 2.1–2.9) had the largest magnitude of effect on the hazard of amputation or death (Table III). Discrimination was good with c-index of 0.71. It was well calibrated, with a nonsignificant P = .6 on Hosmer-May goodness-of-fit testing. On Cox regression modeling of 1-year survival and 1-year amputation as independent end points, a similar panel of variables remained in each model, with only modest changes in the magnitudes of effect compared with the results of modeling the composite end point (Table IV).

Table II.

Multivariable analysis of factors associated with amputation-free survival (AFS) after lower extremity bypass (LEB) in 7730 patients

Covariates	HR (95% CI)	P value
Demographics
Age, years		<.0001
<60	Referent
60–69	0.99 (0.84–1.2)
70–79	1.1 (0.95–1.3)
≥80	1.6 (1.4–1.9)
Sex, male	0.95 (0.84–1.1)	.39
Race		.01
White	Referent
Non-white	1.2 (1.04–1.4)
Preoperative factors
Indication		<.0001
Rest pain	Referent
Tissue loss	1.9 (1.7–2.2)
Smoking status		<.0001
Never	Referent
Prior history	0.92 (0.79–1.06)
Current	0.63 (0.54–0.73)
Coronary artery disease		<.0001
None	Referent
History of MI,       asymptomatic or stable	1.3 (1.2–1.5)
Unstable angina/MI ≤6 months	2.3 (1.7–3.2)
Congestive heart failure	2.0 (1.8–2.2)	<.0001
Diabetes mellitus	1.4 (1.3–1.6)	<.0001
Chronic obstructive     pulmonary disease	1.2 (1.0–1.3)	.03
Renal function		<.0001
Creatinine ≤1.8 mg/dL	Referent
Creatinine >1.8 mg/dL	1.5 (1.3–1.9)
Dialysis-dependent	3.4 (2.9–3.9)
Previous
Arterial bypass (any)	1.2 (1.1–1.4)	.001
Angioplasty or stent (ipsilateral)	1.2 (1.02–1.3)	.02
Major amputation (contralateral)	2.0 (1.7–2.4)	<.0001
Ambulation status		<.0001
Independent	Referent
With assistance	2.0 (1.8–2.3)
Wheelchair-bound	2.9 (2.4–3.5)
Bedbound	6.4 (4.2–10)
Technical factors
Prosthetic bypass conduit	1.6 (1.4–1.8)	<.0001
Single-segment great     saphenous vein	0.63 (0.56–0.71)	<.0001
Recipient vessel		<.0001
Above-knee popliteal	Referent
Below-knee popliteal	1.1 (0.88–1.3)
Tibial/other	1.5 (1.3–1.8)
Urgency		<.0001
Elective	Referent
Urgent	1.4 (1.3–1.6)
Weight, kg		<.0001
≤66	1.1 (0.94–1.3)
67–77	Referent
78–91	0.77 (0.66–0.91)
>91	0.74 (0.62–0.88)
Aspirin or clopidogrel	0.60 (0.52–0.70)	<.0001

CI, Confidence interval; MI, myocardial infarction; OR, odds ratio.

Table III.

Derivation of a novel risk adjustment model in the Vascular Quality Initiative (VQI) data set for predicting amputation-free survival (AFS) at 1 year^a

Covariates	HR (95% CI)
Age, years
<60	Referent
60–69	0.92 (0.78– 1.1)
70–79	0.90 (0.75–1.1)
≥ 80	1.2 (1.03–1.5)
Tissue loss	1.3 (1.1–1.5)
Congestive heart failure	1.4 (1.2–1.6)
Diabetes	1.2 (1.1–1.4)
Renal function
Creatinine ≤1.8 mg/dL	Referent
Creatinine >1.8 mg/dL	1.4 (1.1–1.7)
Dialysis-dependent	2.5 (2.1–2.9)
Preoperative functional status
Independent	Referent
Ambulatory with assistance	1.4 (1.2–1.6)
Bedridden	4.4 (2.7–7.0)
Wheelchair	2.2 (1.8–2.7)
Prosthetic bypass conduit	1.8 (1.6–2.1)
Distal target
Above-knee popliteal	Referent
Below-knee popliteal target	1.3 (1.1–1.6)
Tibial/pedal target	1.7 (1.4–2.1)
Urgent procedures	1.4 (1.2–1.5)
Weight, kg
≤66	1.1 (0.97–1.3)
67–77	Referent
78–91	0.82 (0.69–0.98)
> 91	0.73 (0.61–0.88)
Antiplatelet agent upon discharge	0.62 (0.54–0.73)

CI, Confidence interval; HR, hazard ratio.

^aData for 7318 patients, of which 530 events were deaths, 516 events were major amputations, and 46 were both; c-index = 0.71.

Table IV.

Variables included in the Cox regression models

Covariates	Amputation or death	Death	Amputation
Age	X	X	X
Tissue loss	X	X	X
Gender			X
Race		X
Congestive heart failure	X	X
Diabetes	X
History of MI, asymptomatic   or stable angina		X
History of peripheral   vascular intervention			X
History of major amputation			X
Creatinine	X	X	X
Unstable angina/MI   within 6 months		X
Ambulatory status	X	X	X
Single-segment great   saphenous vein			X
Prosthetic bypass conduit	X	X	X
Graft type	X		X
Urgent/elective	X	X	X
Weight	X	X
Aspirin or clopidogrel	X	X	X

MI, Myocardial infarction.

External validation

When the novel VQI model was applied to the PIII data set, the magnitudes of effect were extremely similar to those seen in the VQI data set. The c-index for the novel VQI model applied to the PIII data set was 0.68 (Table V). The Hosmer-May goodness-of-fit test indicated good calibration (P = .99).

Table V.

External validation in the modified Project of Ex-vivo vein graft Engineering via Transfection III (PREVENT III [mPIII]) data set for amputation free survival (AFS) at 1 year^a

Covariates	HR (95% CI)
Age, years
<60	Referent
60–69	0.81 (0.57–1.1)
70–79	1.5 (1.1–2.1)
≥80	2.2 (1.5–3.0)
Tissue loss	1.7 (1.2–2.3)
Diabetes	1.4 (1.1–1.9)
Renal function
Creatinine ≤1.8 mg/dL	Referent
Creatinine >1.8 mg/dL	1.4 (1.01–2.1)
Dialysis dependent	3.1 (2.4–4.0)
Distal target
Above-knee popliteal	Referent
Below-knee popliteal target	1.4 (0.87–2.3)
Tibial/pedal target	1.5 (0.96–2.4)
Weight, kg
≤66	1.1 (0.8–1.4)
67–77	Referent
78–91	0.93 (0.69–1.2)
>91	0.82 (0.59–1.1)
Antiplatelet agent upon discharge	0.83 (0.64–1.1)

CI, Confidence interval; HR, hazard ratio.

^aData for 1348 patients, of which 182 events were deaths, 119 events were major amputations, and 34 were both; c-index = 0.68.

DISCUSSION

The stated mission of the SVS PSO “is to improve patient safety and the quality of health care delivery by providing web-based collection, aggregation, and analysis of clinical data submitted in registry format for all patients undergoing specific vascular treatments.”¹⁰ As a corollary to that, these aggregated data are ideally suited for benchmarking outcomes after vascular surgical procedures, which is appealing as pay-for-performance concepts take hold. Risk adjustment is critical to any meaningful comparisons of outcomes after vascular surgical procedures. The model for risk-adjusting outcomes after LEB has not been determined yet for the SVS VQI.

Several scoring systems exist that stratify CLI patients according to risk of major amputation or death at various time points in follow-up. A recent review of three such scores (BASIL, FINNVASC, and PIII) was conducted using data from a single institution in England.⁷ These scores have not been compared in a United States population. The SVS VQI is an important resource for assessing quality of care in vascular surgery, and risk adjustment is critical to that process. Although several scoring systems stratify CLI patients according to risk of major amputation or death, the VQI currently lacks a statistical model for risk adjustment of outcomes after LEB. This analysis of existing prediction models can inform VQI’s selection of a model for quality reporting. It is important to note that two of these three previously described models (PIII and FINNVASC) were described as risk scores rather than risk-adjustment models. Our intention was not to create another model for preoperative counseling on risk prediction. Instead, we sought to define the model that most accurately risk adjusts for the purposes of benchmarking outcomes, setting an “expected” rate for comparison with that “observed.” As a result, the model was not used to assign quartiles of risk based on an integer score, and the results were therefore not presented in that way. This use of a regression analysis for risk adjustment is well described and is already in use for other end points, such as in-hospital stroke or death after carotid endarterectomy, that are reported in VQI. It is also important to note that all three models required some modification from their original descriptions to be applied to VQI. Because our goal was not to externally validate them, but rather to assess their suitability to risk adjustment in VQI, we do not believe this prohibits the comparative analysis.

Our findings indicate that existing scoring systems perform modestly for prediction of 1-year AFS. Our novel VQI-derived model incrementally improved on these established models. There is not a statistical method to define the difference in c-indices that is “significant,” so their cardinal values were compared instead. The reasons for this improved performance may be multifactorial but may reflect some differences in this data set compared with those used for the other models. This data set is the largest by several orders of magnitude and is the most contemporary of those tested. It is important to acknowledge that none of these models performed extraordinarily well, given that the highest c-index was still well below 0.8. This may suggest that these data sets do not capture variables that are most strongly predictive of AFS; specifically, no information is available on the presence of infection or characteristics of ulceration, and no data are captured regarding blood markers of systemic inflammation. Further work may be needed to improve our understanding of which factors are truly relevant for prediction of outcomes after LEB.

Alternatively, the modest performance of these models may reflect the choice of the end point modeled. AFS has been criticized as an end point for evaluating revascularization for CLI. Conte¹¹ described notable limitations to the use of AFS for comparison of revascularization strategies, because the goal of LEB for CLI is limb salvage, without an expectation that revascularization would directly affect survival. Alternatively, the SVS has adopted objective performance goals for comparison of revascularization strategies.¹² Although reasonable for the purposes of comparing techniques of revascularization, benchmarking outcomes in the VQI using all of the numerous end points described by these objective performance goals is not currently feasible.

Functional outcomes and health-related quality of life outcomes are also very valuable for patients. Given that no single end point can perfectly define quality of care for CLI, 1-year AFS remains a well-established, clinically sound measure for use in risk-stratifying outcomes after LEB. As a “hard” end point, AFS can be explained to patients in very clear terms. AFS also has the advantage over bypass patency for providing clinical outcome information, rather than information on technical success. AFS is certainly not the only important outcome after lower extremity revascularization, but it does continue to be meaningful for patients, practitioners, and payers.

As with all observational cohort studies, there are important limitations inherent to this study design. Improving the compliance with long-term follow in each of the VQI procedure modules is a recognized as an area for improvement; our study highlights this substantial unmet need. As described above, the total study cohort consisted of 7750 patients. During the follow-up period, 1482 patients died. Of those patients who did not die (n = 6268), 1413 patients had follow-up entered beyond 1 year and 2510 patients had follow-up beyond 9 months; these are not mutually exclusive groups, meaning that long-term follow beyond 9 months was performed and documented in only 62.6% of eligible patients. Acknowledging this limitation, we believe that this data set and the analysis performed provide a meaningful way in which to provide risk-adjusted outcome reporting for patients who undergo LEB for CLI.

CONCLUSIONS

This analysis demonstrated that previously described prediction models for 1-year AFS after LEB for CLI (mPIII, FINNVASC, and BASIL) perform modestly in the VQI cohort. Our VQI-derived novel model of this end point has improved performance in VQI and has been externally validated in the PIII data set. We believe that the VQI should use this novel model for risk-adjusting outcomes in the VQI LEB data set.