Patient Safety Indicators are an insufficient performance metric to track and grade outcomes of open aortic repair

Abstract

Objective:

National rankings of hospitals rely on outcomes-based evaluation to assess the performance of surgical programs, particularly those performing high-risk elective surgical procedures such as open aortic repair. Various classification systems exist for tracking outcomes, but increasingly the International Classification of Diseases, Tenth Revision-based Agency for Healthcare Research and Quality Patient Safety Indicators (PSIs) are used as a publicly reported comparison measure of hospital quality performance. We sought to critically evaluate the accuracy of the existing vehicles to assess open aortic repair outcomes in an established program.

Methods:

This is a case-control study of patients who underwent open abdominal aortic aneurysm repair at the Johns Hopkins Medical Institutions from 2004 to 2018. Patients’ characteristics and outcomes were collected as part of a prospectively maintained retrospective database. For each case, hemorrhagic, cardiac, respiratory, renal, wound, and thromboembolic complications were identified with the unique definitions used for open abdominal aortic aneurysm repair by the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) database, the Society for Vascular Surgery Vascular Quality Initiative (VQI) database, and the Agency for Healthcare Research and Quality PSI initiative.

Results:

Of the 154 patients included in the study, 79 (51.0%) were identified as having a complication as defined by the VQI, 46 (29.7%) according to the NSQIP, and 15 (9.7%) according to the PSI system (P < .001). Patients most likely to incur a complication in the PSI system were those with a pararenal or more extensive aneurysm, with baseline congestive heart failure, requiring a supramesenteric clamp (all P < .01), or with an aneurysm >6.5 cm in diameter (P = .02). The NSQIP and VQI systems both identified more postoperative hemorrhagic, respiratory, renal, and wound complications than the PSI system did (P < .05). The VQI system identified the most renal complications (52; P < .001); factors unique to incurring a complication in the VQI include use of a suprarenal clamp and performance of an aortorenal bypass procedure as part of the repair (P < .01). Particularly weak correlation was noted between the PSI system and the VQI with respect to renal outcomes (ρ = 0.163).

Conclusions:

The PSI system identified fewer important complications than either of the clinically focused databases, with the VQI capturing the most postoperative events, mostly because of its stringent definition of renal injury. We conclude that the PSI system should not form the basis of grading hospital performance in comparing clinically relevant complications of open aortic surgery programs.

Regionalization of surgical care for Medicare patients describes the practice of referring patients to a small number of high-volume centers for specific high-risk procedures, such as coronary artery bypass grafting, esophagectomy, and pancreaticoduodenectomy.¹ The concept of regionalization is one that has been discussed for years; it originally stems from the positive relationship between hospital volume and patient outcomes that was identified in the late 1990s for many complex surgical procedures, including open aortic repair.^1,2 Whereas surgeon volume and hospital volume have been proven repeatedly to correlate with outcomes for open aortic repair,^3–5 the factors mediating surgical outcomes have proved to be far more nuanced than originally thought.⁶ With the realization that surgical outcomes are mediated by multiple complex factors that extend far beyond simple hospital volume, health care has seen the increasing ubiquity of detailed surgical quality outcomes databases to track complications for specific procedures.^7–9

Nationally reported surgical outcomes have since become a keystone of quality improvement and a comparison tool for institutions. Within vascular surgery, the American College of Surgeons National Surgical Quality Improvement Program (NSQIP)⁷ and the Society for Vascular Surgery (SVS) Vascular Quality Initiative (VQI)⁹ databases are widely employed to compare clinical outcomes, but the definitions of postoperative complications vary widely between these classification systems. These systems are developed by clinicians and focused on improving inpatient care. In 2003, the International Classification of Diseases, Tenth Revision-based Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs) were designed by health care researchers as a means to capture specific adverse postoperative events and to guide allocation of Medicare funds.⁸ Whereas the measures were originally used independently, a weighted composite score (PSI-90) was later added to the system in 2014 as an institution- or program-specific performance metric that could be used for ranking and comparison.¹⁰

The clinical relevance and reliability of the PSI system as an outcomes tracking method have previously been called into question.^11–13 A general study of surgical patients of all specialties demonstrated that a large number of clinically relevant complications are not captured in the system at all, such as postoperative pneumonias, the majority of surgical site infections, and strokes.¹¹ Despite these concerns, individual and composite outcomes derived from the PSI system are increasingly publicly reported and form a core metric in Medicare and Medicaid pay-for-performance programs, in which the PSI-90 can be responsible for 30% to 35% of an institution’s score determining its allocation of Medicare payments.¹⁰ These scores can also be used as a means of ranking individual surgical programs performing high-risk surgical procedures, such as open aortic repair, and have been discussed as a new means of facilitating the regionalization of procedures within Medicare.^12,14

The relevance of the PSI system to aortic repair has been briefly touched on in the literature, where the introduction of endovascular aortic repair reduced overall PSI occurrences across a large data set.¹⁵ Likewise, PSI incurrence was noted to portend higher mortality and longer hospital lengths of stay as well as higher health care expenditures after open aortic surgery.^14,16 Despite this, the accuracy of PSIs as a comparison measure in vascular surgery has never been investigated. In this work, we sought to critically evaluate the accuracy of the existing vehicles to assess outcomes for open aortic surgery at an established complex aortic program. We use a single-institution data set to compare the concordance of the VQI, NSQIP, and PSI systems as a means of tracking complications in open aortic repair.

METHODS

This study is a case-controlled retrospective analysis performed with a prospectively maintained database used at the Johns Hopkins Medical Institutions for the internal tracking of aortic surgery outcomes. All patients who underwent open abdominal aortic aneurysm (AAA) repair between the years 2004 and 2018 were included. This investigation was reviewed and approved by the Institutional Review Board, and the need for patient consent was waived, given the retrospective and purely analytical nature of the work. The study included 154 patients with all extents of AAA, including infrarenal, juxtarenal, pararenal, and suprarenal to the extent of a Crawford type IV thoracoabdominal aortic aneurysm. No aneurysms with extent beyond the level of the diaphragm were included in this cohort of patients. Any repairs performed for aortic graft infection or its sequelae were also excluded. Conversions of failed endovascular repairs were included, as were all repairs performed on an emergent or urgent basis otherwise meeting anatomic criteria for inclusion.

The primary end point for this study was a composite outcome for any complication, which was recorded as positive for a given system if any complication within that system was incurred. Complications included in our internal database were defined according to institutional definitions, and for each case, hemorrhagic, cardiac, respiratory, renal, wound, and thromboembolic complications were identified by the unique definitions used by the American College of Surgeons NSQIP database,⁷ the AHRQ PSI initiative,⁸ and the SVS VQI database⁹ made available by each of the respective organizations. The SVS VQI database has specific criteria for complication definitions based on procedure; the open AAA repair data dictionary, which is not publicly available, was obtained to establish the VQI definitions described in this work. Of note, our institutional data set uses the Risk, Injury, Failure, Loss of kidney function, and End-stage renal disease (RIFLE) definition for acute kidney injury (AKI).¹⁷ In reclassifying complications, retrospective chart review was used to confirm the accuracy or to further refine the outcomes information recorded in the database. All unique definitions for each system of classification are fully delineated in Table I.

Table I.

Definitions of complications, by organ system, as defined by our institution, the Society for Vascular Surgery Vascular Quality Initiative (SVS VQI), the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP), and the Agency for Healthcare Research and Quality Patient Safety Indicator (AHRQ PSI) systems

Complication	Institutional	SVS VQI	ACS NSQIP	AHRQ PSI
Hemorrhagic	• Return to the OR or additional procedure for control of hemorrhage	• Quantifies RBC transfusions within 72 hours of the OR • Return to the OR or additional procedure for control of hemorrhage	• Quantifies RBC transfusions within 72 hours of the OR • Return to the OR or additional procedure for control of hemorrhage	• Return to the OR or additional procedure for control of hemorrhage
Cardiac	• Cardiac arrest • Myocardial infarction with ECG changes or cardiologist’s diagnosis or procedure • Any new-onset dysrhythmia requiring ICU readmission or procedure (cardioversion)	• Cardiac arrest • Myocardial infarction with elevated troponin or ECG changes • Any new-onset dysrhythmia requiring medication or cardioversion • Any new-onset heart failure	• Cardiac arrest • Myocardial infarction characterized by troponins 3× upper limit of normal, ECG changes, or cardiologist’s diagnosis	Not captured
Respiratory	• Pneumonia • Unplanned reintubation • Ventilator support >72 hours • Respiratory embarrassment requiring ICU readmission	• Pneumonia • Unplanned reintubation • Captures ventilator support >12 hours and >24 hours	• Pneumonia • Unplanned reintubation • Ventilator support >48 hours	• Unplanned reintubation • Ventilator support >96 hours
Renal	• Creatinine increase of 3× baseline or to a level of >3 mg/dL with a change of at least 0.5 mg/dL • New hemodialysis	• Creatinine increase of >0.5 mg/dL • New hemodialysis	• Creatinine increase of >2 mg/dL • New hemodialysis	• New hemodialysis
Wound	• Surgical site infection • Dehiscence, superficial or deep • Dehiscence, superficial or deep	• Surgical site infection • Dehiscence, superficial or deep	• Surgical site infection • Dehiscence, superficial or deep	• Any dehiscence requiring return to OR
Thromboembolic	• Any DVT or PE	Not captured	• Any DVT or PE	• Any DVT or PE
Bowel ischemia	• Any ischemic colitis, medically or surgically managed	• Any ischemic colitis, medically or surgically managed	• Any ischemic colitis, medically or surgically managed	Not captured
Neurologic	• Any cerebrovascular accident • Any spinal cord ischemia	• Any cerebrovascular accident	• Any cerebrovascular accident	Not captured
Extremity	• Limb ischemia requiring procedural or operative intervention	• Any documented limb ischemia, medically or surgically managed	• Limb ischemia requiring procedural or operative intervention	Not captured

DVT, Deep venous thrombosis; ECG, electrocardiography; ICU, intensive care unit; OR, operating room; PCI, percutaneous coronary intervention; PE, pulmonary embolism; RBC, red blood cell.

In assessing these data, it should be noted that the NSQIP and PSI databases capture only elective cases (n = 146), whereas our institutional data and the VQI system capture all cases (n = 154), yielding a discrepancy in the number of cases included in each system. In calculating complication rates within each system, only the cases that were pertinent to the system’s defined target population were included to give the most accurate portrayal of the system’s performance in its intended population. Patients were defined as symptomatic if they presented acutely with a contained rupture, abdominal or back pain attributed to the aneurysm, or new renal failure attributable to renal artery involvement in the expanding aneurysm. Not all organ systems are captured by the PSI system (neurologic outcomes, limb complications, bowel ischemia), and a number of PSI measures are not included in this analysis as they do not have corresponding outcomes in the other systems or were irrelevant to this data set (ie, unrecognized intra-abdominal injury, inpatient fall with hip fracture, pressure ulcer).

The PSI system additionally contains a composite outcome titled the PSI-90, which assigns weights to the various components to create a score for a particular program or institution.⁸ These weights are revised periodically and are based on both the incidence and harm associated with the event; currently, the greatest weights are ascribed to postoperative respiratory failure (PSI-11, 0.21), postoperative sepsis (PSI-13, 0.25), and venous thromboembolic events (PSI-12, 0.21). The sepsis metric was not included in this analysis as there were no patients in this cohort who met criteria for this event on chart review. The weighting for PSIs is fully described in Table II.

Table II.

Component weights of the Patient Safety Indicator (PSI) composite metric (PSI-90) as defined by the Agency for Healthcare Research and Quality (AHRQ) in the 2019 update

Component	Description	Harm weight	Volume weight	Component weight
PSI-3	Pressure ulcer rate	0.3080	0.0860	0.1373
PSI-6	Iatrogenic pneumothorax rate	0.1381	0.0538	0.0385
PSI-8	In-hospital fall with hip fracture rate	0.1440	0.0172	0.0128
PSI-9	Perioperative hemorrhage or hematoma rate	0.0570	0.1598	0.0472
PSI-10	Postoperative AKI requiring dialysis rate	0.3584	0.0280	0.0520
PSI-11	Postoperative respiratory failure rate	0.2219	0.1821	0.2094
PSI-12	Perioperative PE or DVT rate	0.1557	0.2543	0.2052
PSI-13	Postoperative sepsis rate	0.3102	0.1550	0.2491
PSI-14	Postoperative wound dehiscence rate	0.1441	0.0138	0.0103
PSI-15	Unrecognized abdominopelvic accidental puncture or laceration rate	0.1474	0.0500	0.0382
PSI-90	Composite sum	–	–	1.0000

AKI, Acute kidney injury; DVT, deep venous thrombosis; PE, pulmonary embolism.

The significance of associations between particular demographic and operative characteristics was assessed with Pearson χ² testing for categorical variables and Student t-test for continuous variables. Complication rates between the various classification systems were assessed with proportion testing, and strength of correlation between the various systems in capturing complications in the same patients was measured with Spearman rank correlation test and reported as a ρ value. For Spearman ρ, strength of correlation was deemed weak if ρ < 0.40 and strong if ρ > 0.59; all intermediate values were interpreted as moderate. Significance for all other analysis results described in this work was defined at a level of P ≤ .05. All data were analyzed using Stata version 15.1 (StataCorp LLC, College Station Tex).

RESULTS

Overall, 154 patients who underwent open AAA repair met criteria for inclusion. Their demographics, comorbidities, anatomy, and operative information are fully summarized in Table III. Briefly, 71.4% were male and 82.5% were white, with a mean age of 67.8 years and a normal baseline creatinine concentration (1.12 ± 0.06 mg/dL). Of these, most were current or former smokers (72.1%) with hyperlipidemia (74.0%) and hypertension (87.0%), and 46.8% had a diagnosis of coronary artery disease. Whereas the average creatinine concentration for the group was normal, 23.4% had some degree of documented chronic kidney disease before undergoing aortic repair, and 3.9% fell into the category of end-stage renal disease on preoperative dialysis.

Table III.

Demographic and operative information for study cohort

	% or mean	Frequency or SD
Total		154
Male sex	71.4	110
Race
White	82.5	127
Black	9.7	15
Other	7.8	12
Age, years	67.8	66.3–69.3
AAA size, cm	6.3	6.1–6.6
Creatinine baseline value, mg/dL	1.12	1.06–1.18
Symptomatic	26.6	41
HLD	74.0	114
HTN	87.0	134
Diabetes	12.3	19
PVD	16.2	25
CAD	46.8	72
COPD	26.6	41
CHF	9.1	14
CTD	6.5	10
Smoking history	72.1	111
CKD	23.4	36
ESRD	3.9	6
AAA class
Infrarenal	29.2	45
Juxtarenal	41.6	64
Pararenal	13.6	21
Suprarenal	15.6	24
Clamp level
Infrarenal	29.9	46
Suprarenal	28.6	44
Supramesenteric	22.1	34
Supraceliac	19.5	30
Renal flush	33.1	51
IMA reimplantation	7.8	12
Renal bypass	35.1	54
Other abdominal procedure	12.3	19
EVAR conversion	6.5	10

AAA, Abdominal aortic aneurysm; CAD, coronary artery disease; CHF, congestive heart failure; CKD, chronic kidney disease; COPD, chronic obstructive pulmonary disease; CTD, connective tissue disease; ESRD, end-stage renal disease; EVAR conversion, cases performed as an open conversion of a failed endovascular repair requiring concomitant explantation of the stent graft; HLD, hyperlipidemia; HTN, hypertension; IMA, inferior mesenteric artery; PVD, peripheral vascular disease; SD, standard deviation.

Average aneurysm diameter was 6.3 cm, with most AAAs being classified as juxtarenal (41.6%) and 70.8% with some component of renal vessel involvement (juxtarenal or more extensive involvement of the renovisceral vessels). Likewise, 70.1% required a suprarenal or higher clamp during the operation; 33.1% received protective cold renal flush. Of these patients, 35.1% underwent a concomitant bypass procedure to one or more renal vessels, and 12.3% underwent some other additional intra-abdominal procedure (eg, ventral hernia repair, cholecystectomy). Also of note, 6.5% of the procedures in this series were conversions of previously placed, failed endovascular repairs.

Table IV demonstrates a condensed version of the demographic and operative factors and their associations with postoperative complications; a full matrix of all associations can be accessed as Supplementary Tables I and II (online only). Notably, in the PSI system, the factors associated with incurring a complication were use of a supramesenteric clamp, a suprarenal or more extensive aneurysm, an aneurysm with a diameter ≥6.5 cm, and preoperative congestive heart failure (all P < .01). For the VQI, factors associated with incurring any complication were use of a suprarenal clamp, a pararenal or more extensive aneurysm, and a concomitant aortorenal bypass procedure (all P < .01). In the NSQIP, the only unique factor associated with incurring any complication was nonwhite race (P < .01). In all three complication classifications, postoperative respiratory complications were associated with preoperative chronic obstructive pulmonary disease and congestive heart failure (all P ≤ .03 for both factors); in the PSI system, these factors were not significant with respect to the respiratory end point.

Table IV.

Condensed summary of associations between demographic, comorbidity, and anatomic characteristics and postoperative complications in each classification system

	Institutional	VQI	NSQIP	PSI
Any complication
Nonwhite race	10 (.37)	11 (.23)	8 (<.01)	3 (.69)
CHF	6 (.27)	8 (.65)	7 (.08)	5 (<.01)
≥Suprarenal clamp	35 (.29)	63 (<.01)	33 (.78)	13 (.15)
≥Supramesenteric clamp	26 (.01)	45 (<.01)	23 (.13)	12 (<.01)
≥6.5 cm	17 (.26)	28 (.17)	18 (.13)	8 (.02)
≥Suprarenal extent	11 (.06)	20 (<.01)	11 (.06)	6 (<.01)
≥Pararenal extent	18 (.08)	32 (<.01)	16 (.32)	9 (<.01)
Aortorenal bypass	22 (.03)	40 (<.01)	20 (.15)	8 (.10)
Hemorrhagic
Age ≥75 years	0 (.36)	14 (.78)	4 (.05)	0 (.43)
HLD	1 (.02)	6 (.05)	6 (.22)	1 (.08)
CTD	1 (.13)	2 (.14)	2 (.06)	1 (.05)
Respiratory
Nonwhite race	7 (.06)	6 (.03)	5 (.53)	2 (.63)
COPD	11 (<.01)	8 (.03)	11 (.03)	4 (.24)
CHF	6 (<.01)	6 (<.01)	7 (<.01)	3 (.01)
PVD	6 (.13)	6 (.02)	7 (.09)	3 (.18)
≥Supramesenteric clamp	14 (.02)	10 (.06)	15 (.02)	7 (.02)
≥6.5 cm	11 (.03)	8 (.07)	12 (.01)	5 (.06)
≥Suprarenal extent	5 (.32)	4 (.27)	4 (.27)	3 (.14)
Cardiac
CAD	8 (.07)	12 (.07)	8 (.04)	–
>6.5 cm	5 (.26)	9 (.06)	6 (.02)	–
Renal
Male	11 (086)	43 (.03)	8 (.53)	2 (.37)
Nonwhite race	6 (.02)	8 (.62)	4 (.04)	2 (<.01)
Symptomatic	7 (.07)	15 (.67)	5 (.04)	1 (.37)
PVD	5 (.06)	9 (.80)	4 (.05)	1 (.21)
≥Suprarenal clamp	5 (.05)	13 (.02)	4 (.03)	0 (.54)
≥Supramesenteric clamp	10 (.03)	33 (<.01)	7 (.05)	2 (.09)
≥Suprarenal extent	5 (.05)	13 (.02)	4 (.03)	0 (.54)
≥Pararenal extent	7 (.12)	23 (<.01)	6 (.03)	0 (.36)
Aortorenal bypass	8 (.12)	28 (<.01)	5 (.28)	1 (.64)
≥7.5 cm	3 (.29)	9 (.12)	3 (.03)	1 (.06)

CAD, Coronary artery disease; CHF, congestive heart failure; COPD, chronic obstructive pulmonary disease; CTD, connective tissue disease; HLD, hyperlipidemia; HTN, hypertension; NSQIP, National Surgical Quality Improvement Program; PSI, Patient Safety Indicator; PVD, peripheral vascular disease; VQI, Vascular Quality Initiative.

Results are described as frequencies (P value). Boldface P values represent statistical significance. Unlisted associations were nonsignificant.

Table V summarizes the comparison analyses performed between the various classification systems. The VQI captured the most postoperative events, identifying a complication in 51.0% of study patients, followed by 29.7% in the NSQIP and 9.7% with PSIs. Most of the additional complications captured by the VQI were renal complications; overall, it captured 52 renal complications vs 10 and 2 by the NSQIP and the PSI system, respectively (P < .001). Beyond that, both the NSQIP and the VQI identified more postoperative hemorrhagic, respiratory, renal, and wound complications than the PSI system (P < .05) with the exception of the VQI respiratory dimension, which was not significant compared with the PSI system (P = .19). In the whole cohort, there were three deaths. For both death and venous thromboembolism, binary outcomes, there were no significant differences between the classification systems.

Table V.

Summary of complications captured in each of the respective classification systems

	Institutional	VQI	NSQIP	PSI	P			Spearman ρ
No.	154	154	146	146	vs Institutional	vs VQI	vs NSQIP	vs Institutional	vs VQI	vs NSQIP
Any complication	46	79	46	15	<.001	<.001	<.001	0.533	0.334	0.499
Hemorrhagic	4	12	10	3	.76	.03	.05	1.000	0.343	0.343
Respiratory	22	16	24	9	.02	.19	.02	0.663	0.620	0.578
Bowel ischemia	4	4	3	NC	–	–	–	–	–	–
Cardiac	11	18	10	NC	–	–	–	–	–	–
Wound	5	5	5	0	.03	.03	.02	–	–	–
Extremity	2	3	2	NC				–	–	–
Neurologic	3	1	1	NC				–	–	–
Renal	15	52	10	2	.001	<.001	.02	0.377	0.163	0.435
DVT or PE	2	NC	2	2	1.00	–	1.00	1.000	–	1.000
Death	3	3	2	2	.70	.70	1.00	1.000	1.000	1.000

DVT, Deep venous thrombosis; NC, not captured; NSQIP, National Surgical Quality Improvement Program; PE, pulmonary embolism; PSI, Patient Safety Indicator; VQI, Vascular Quality Initiative.

P values reflect proportion testing against the PSI system. P values derived by comparing NSQIP with VQI were insignificant with the exception of “any complication” and “renal” (both P < .001). Boldface P values represent statistical significance. Spearman ρ values reflect rank correlation testing against the PSI system.

In examining how well the various classification systems correlate with one another, the Spearman coefficient for VQI vs NSQIP shows moderate concordance between the two systems (ρ = 0.551) and particularly weak concordance for renal outcomes (ρ = 0.376). When the PSI system is compared with the VQI and NSQIP, the correlation is weak for the VQI (ρ = 0.334) and only moderately concordant with the NSQIP (ρ = 0.499). With respect to organ-specific dimensions, the best concordance was demonstrated with respiratory outcomes (0.578–0.620). The weakest correlation was demonstrated between VQI and PSI renal outcomes (ρ = 0.163). Correlations between specific organ system complications within the classification systems themselves were largely nonsignificant and are fully detailed in Supplementary Tables III–VI (online only). Of note, the PSI-90 calculation for this cohort using the formula from the AHRQ was 2.27%.

DISCUSSION

Overall, the results of this analysis emphasize the lack of granularity inherent in the AHRQ PSI system for identifying and recording complications in open aortic repair and its poor correlation with more clinically focused quality metrics, such as the NSQIP and VQI. Those patients shown to be most likely to incur a PSI in our study were those patients with the most severe comorbidities and anatomic complexity, including preoperative congestive heart failure, extensive aneurysm (suprarenal or higher), and large aneurysm (≥6.5 cm). By design, the PSI is a decidedly coarse measure of patient outcomes as it tends to capture those patients with specific types of complications, such as missed enterotomies, fascial dehiscence, pulmonary emboli, and unplanned reintubations. In this sense, PSI is by no means a meritless system of tracking complications. There is value in designating some events as serious preventable patient outcomes or so-called never events. However, by definition, these events will be uncommon, making robust quantitative PSI-based comparisons challenging, particularly on an institution to institution basis.

Previous research using large data sets has shown PSIs to correlate significantly with readmission and increased length of stay in open and endovascular AAA repair; however, this work required examining nearly 67,000 Medicare patients to attain statistical significance.¹⁵ By most standards, our institution falls into the category of a high-volume open aortic repair center with >20 open cases annually^3–5; however, even at that volume, we incurred only 15 true (eg, independently reviewed and confirmed) PSI instances during the course of 14 years. Thus, among high-volume centers, the rate of true PSI incurrences will be far too low to meaningfully compare single institutions even with longitudinal averaging. Use of the PSI-90 composite metric further diminishes the weight of most of these complications, particularly clinically significant ones, such as postoperative dialysis, inherently creating an extremely small range of comparison between institutions.

Independent reviewers investigating the accuracy of the PSI system using identified data have found 33% of PSIs to be inherent to the disease process (ie, a high false-positive rate) and 30% to be clinically insignificant.¹³ The low positive predictive value of PSIs as a quality metric is an issue that has been identified repeatedly in the literature.^18–21 These various studies use varying reference standards but have estimated positive predictive value of PSIs as 55% to 83% overall, universally regarded as suboptimal for public reporting and pay-for-performance evaluation, prompting some to advocate for daily review of PSI occurrences to prevent false-positive reporting and associated loss of reimbursements.²¹ Although we did not reproduce this phenomenon in our comparison analysis, this was not unexpected, given the methods and aim of our work. Our aim was to use critical review to capture true instances of complications that should be recorded in these quality databases based on their unique definitions, reflecting how complications are recorded in the NSQIP and VQI (ie, by an independent reviewer employed specifically for that purpose). The PSI system, in contrast, is based on International Classification of Diseases, Tenth Revision codes and is at the mercy of the coding and documentation practices of multiple providers with various levels of medical coding expertise, leading to great variability in reporting that can affect PSI-90 calculations.

Since the introduction of PSI-90, it has been criticized as poorly representative of hospital performance in that it unfairly penalizes hospitals for vigilant surveillance of adverse outcomes, is excessively harsh toward centers operating on sicker patients, and is by design skewed toward particular clinical areas.^10,14 Whereas many of the components, such as postoperative hemorrhage, are surgically universal, others, such as iatrogenic pneumothorax and accidental laceration, are designed to be relevant to strict abdominopelvic procedures, not thoracoabdominal, retroperitoneal, or extremity procedures. The pure code-based nature of the PSIs can lead to false captures, particularly with respect to sepsis, a relatively heavily weighted component of the PSI-90. This point is particularly germane to open aortic surgery, in which all patients postoperatively spend time in the intensive care unit, may require vasopressor support for vasoplegia, and will certainly have a transiently elevated white blood cell count and lactate level; such patients can easily be miscoded as meeting sepsis criteria even in the absence of infection. Another independent review of institutional PSIs noted a high rate of inaccurate coding for deep venous thrombosis, another high-weight component of the PSI-90, where 21% of the deep venous thromboses flagged were actually due to arterial thrombosis.²² Both of these points underscore that the PSI-90 was not designed for use in vascular surgery and should be used with extreme caution in this context.

Our analysis noted particularly poor concordance of the PSI system with the VQI (ρ = 0.334) and only moderate concordance with the NSQIP (ρ = 0.499), but it is important to consider that the PSI system was not designed with the same goals in mind as these two more clinically focused databases. In fact, even the VQI and NSQIP do not capture the same patient populations. The VQI is designed as a means of tracking outcomes for all manifestations of a single vascular procedure; that is, it captures all AAA repairs, regard-less of whether the repair was performed emergently or electively. This is in contrast to the NSQIP database, which was conceived as a system to capture events after elective surgical procedures only and was originally designed with no specialty-specific outcomes in mind. This was changed slightly in 2015 with the addition of the procedure targeted files in the NSQIP that allowed outcomes specific to vascular procedures to be tracked.⁷ Previous research delineating the differences in the VQI and NSQIP databases has emphasized that the populations captured in these databases are not the same and, second, that the outcomes measured are not necessarily directly comparable.^23,24 This is reflected in our analysis, which demonstrated only moderate correlation between the systems overall (ρ = 0.551); we found this lack of concordance was particularly stark for renal outcomes (ρ = 0.376).

Renal injury is one of the complications most relevant to aortic repair but has the greatest variation in definition between the databases (P < .001 between all systems). Need for new postoperative dialysis is known to correlate strongly with mortality, with an associated in-hospital mortality of 75% reported in one study of ruptured AAA.²⁵ New dialysis is captured as a renal complication by all three databases and is the sole renal complication captured in the PSI system; however, perioperative AKI is known to be a far more common but insidiously morbid complication of aortic repair. Previous efforts to characterize the risk of renal function decline associated with open aortic surgery have demonstrated that patients experiencing perioperative AKI after AAA repair are twice as likely to experience a permanent decline in renal function (decrease in estimated glomerular filtration rate ≥20%) at 1 year compared with those without AKI.²⁶

The NSQIP uses a creatinine concentration-based definition for AKI that requires an absolute change of 2.0 mg/dL, whereas the VQI classifies AKI in any patient having an increase in creatinine concentration of 0.5 mg/dL or more. Previous comparisons of these two databases have remarked on this, citing especially poor concordance between the two in terms of renal outcomes.²⁴ Indeed, our data corroborated this observation as the VQI unsurprisingly captured more renal complications than the NSQIP or the PSI system (52 vs 10 and 2, respectively; P < .001), with a markedly weak correlation noted between the PSI system and the VQI (ρ = 0.163). The NSQIP creatinine concentration-based definition of renal injury has previously been criticized as insufficient to identify meaningful instances of AKI that can be associated with future adverse effects. The RIFLE criteria are consensus guidelines that define AKI as any change in serum creatinine concentration ≥50%. One retrospective study of 27,841 surgical patients at a single academic institution found a 30% difference between RIFLE and NSQIP definitions of AKI.²⁷ The study also found that patients meeting criteria for RIFLE AKI had a 10 times greater risk of death compared with those without AKI, whereas the NSQIP definitions of AKI had low sensitivity for adverse postoperative outcomes. In this and other studies, changes in serum creatinine concentration as little as 0.2 or 0.3 mg/dL have been associated with adverse long-term renal outcomes.^27,28

A systematic review examining the incidence of renal injury after open aortic surgery using the Kidney Disease: Improving Global Outcomes threshold of 0.3 mg/dL cites an AKI rate of 20% to 30% with infrarenal clamping and 26% to 37% with application of a suprarenal clamp.²⁹ This is consistent with the VQI rate of renal injury identified in our cohort, which was largely suprarenally clamped (33.8%). It is difficult to envision an application of a suprarenal cross-clamp that would not result in a serum creatinine concentration bump of 0.2 or 0.3 mg/dL, calling into question whether this should be defined as a complication for juxtarenal aneurysm repair or simply accepted as inherent to repair of complex aneurysms involving the renovisceral segment. That said, a more stringent definition is likely to be more accurate for prediction of future patient outcomes when adhering to creatinine concentration-based definitions of AKI. Clearly, the magnitude of serum creatinine concentration change associated with significant morbidity and mortality is small, and tracking nuanced changes in renal function over time is beyond the scope of a database like the NSQIP or VQI.

Finally, the findings described here highlight the obvious limitations of the PSI system with respect to open aortic repair; however, it is uncommon in this day and age that the bulk of any complex aortic program is made up of open procedures. It has been well established that the employment of advanced endovascular techniques, such as multiply fenestrated or branched stent grafts to approach renovisceral aneurysms, minimizes early complications but is associated with increased late complications and need for additional procedures.³⁰ This characteristic greatly frustrates the current quality improvement systems in their ability to capture differences in outcomes for patients undergoing endovascular procedures. As a specialty, our ability to compare aortic programs will only become increasingly handicapped as complex endovascular techniques gain ground and trainees become less well versed in the open techniques required to address the late complications of such repairs. Maintaining a specialty-specific, highly sensitive outcomes tracking system in the form of the VQI will become increasingly important to safeguard against inappropriate external assessments and policy changes based on outcome measures whose genesis was not clinically based.

The strengths of this work include its use of an identified, single-institution data set, which allowed us to critically revisit the electronic medical record and recode by hand the complications according to the respective classification definitions. The nature of the referral practice at our institution grants a high degree of anatomic complexity to this moderately sized group of AAA repairs, with >70% of patients requiring a suprarenal or higher cross-clamp. Limitations include the retrospective nature of the recoding, which raises the possibility that some complications may not have been appropriately captured because of documentation errors. During the study period, our institution also experienced a transition in our electronic medical record that resulted in significant changes in perioperative documentation. It is possible that because of both these elements, some complications may have gone uncaptured. Likewise, this is a relatively limited sample of patients, which restricts our ability to perform robust multivariate analysis and to model which patient characteristics are independently associated with adverse outcomes in each of the respective quality assessment paradigms. The element of surgeon experience with respect to the repair of complex renovisceral aneurysms may have an impact on the overall complication rates reported here; however, this should have minimal impact on the comparison between the classification systems themselves.

CONCLUSIONS

In comparing three nationally used systems for tracking postoperative complications, the PSI system identified fewer important complications than either of the clinically focused databases, with the VQI capturing the most postoperative events, mostly because of its stringent definition of renal injury. We conclude that the PSI system cannot and should not form the basis of grading hospital performance in comparing clinically relevant complications among complex aortic surgery programs and should not form the basis of care regionalization for aortic repair. Lack of discrimination among programs based on PSI analysis will be increasingly relevant as aortic surgery programs evolve more complex endovascular techniques that minimize 30-day complications as long-term complications are not captured in such systems. Future investigations will be needed to determine the best way to track and to compare performance across aortic programs employing a diverse, multiplatform approach to aneurysm repair.

ARTICLE HIGHLIGHTS.

• Type of Research: Single-center, retrospective, case-control study

• Key Findings: Adding the Agency for Healthcare Research and Quality Patient Safety Indicator (PSI) definitions to 154 open aortic aneurysm repairs identified significantly fewer clinically significant postoperative events compared with either National Surgical Quality Improvement Program or Vascular Quality Initiative definitions (15 vs 46 and 79, respectively; P < .001). Particularly weak correlation was noted between the PSI system and the Vascular Quality Initiative with respect to renal outcomes (ρ = .163).

• Take Home Message: The PSI system did not accurately capture the majority of clinically significant postoperative events after open aortic aneurysm repair and should not be used to compare aortic programs or to form the basis of care regionalization.