The evaluation of three comorbidity indices in predicting postoperative complications and readmissions in pediatric urology

Abstract

Introduction

The surgical comorbidity assessment is important for patient risk stratification, counseling, and research. In adults, risk assessment indices, such as the Charlson Comorbidity Score (CCS) or Van Walraven Index (VWI), are well established. In pediatrics, however, risk assessment indices are scarce. Recently, a pediatric-specific risk assessment index, the Rhee index, was developed to discriminate mortality for pediatric general surgery patients. Currently, there is no validated risk assessment tool in pediatric urology.

Objective

We compared the performance of the CCS, VWI, and Rhee Index in discriminating postoperative complications and readmissions to the emergency room/inpatient unit after pediatric urological procedures.

Methods

We analyzed the Nationwide Readmissions Database (NRD), State Inpatient Databases (SID), and State Emergency Department Databases (SEDD). We included patients (< 18 years) who underwent the following urological procedures: ureteroneocystostomy, ureteroureterostomy, radical/partial nephrectomy, pyeloplasty, appendicovesicostomy, enterocystoplasty, vesicostomy, and bladder neck sling. Complications were identified based on definitions in the National Surgical Quality Improvement Program (NSQIP). Thirty-day emergency room admission and inpatient readmissions were extracted. Comorbidity scores were calculated using each of the three indices. We compared the performance of each index in discriminate primarily postoperative complications in the NRD and both admission types in the SID/SEDD by constructing a receiver operating characteristics (ROC). AUCs were compared using the Delong method.

This protocol was reviewed by our Institutional Review Board and deemed to be exempt.

Results

We identified a total of 8,006 patients in NRD and 6,236 patients in SID/SEDD. The Rhee index had the best performance for discriminating postoperative complications (AUC = 0.67, 95% CI 0.64–0.70) compared to CCS (AUC = 0.62, 95% CI 0.60–0.65) and VWI (AUC = 0.62, 95% CI 0.59–0.65); p < 0.01. The CCS had the best performance for discriminating 30-day inpatient readmissions (AUC = 0.63, 95% CI 0.61–0.66) than VWI (AUC = 0.54, 95% CI 0.52–0.57), and Rhee Index (AUC = 0.56, 95% CI 0.54– 0.59); p < 0.0001. All three indices had similarly poor discrimination for 30-day ER admissions: CCS (AUC = 0.52), VWI (AUC = 0.51), and Rhee Index (AUC = 0.50); p = 0.5.

Discussion

The Rhee Index had the best performance for discriminating postoperative complications, while the CCS was superior for discriminating inpatient readmissions among the three indices. Limitations to our study include inpatient-only procedures, inability to identify complications managed in clinics, omission of secondary operations, accounting for parental anxiety, and the generalizability of SID.

Conclusions

The three comorbidity indices evaluated are poor discriminators for postoperative complications, 30-day inpatient readmissions or 30-day ER admissions. A new index is needed for pediatric urology patients.

Introduction

Surgical comorbidity assessment is an integral part of patient risk stratification, counseling, and clinical research. Patient-related comorbidities can impact surgical mortality, post-surgical complications, and readmissions and, thus, influence healthcare outcome and cost [1]. It is imperative to identify at-risk patients preoperatively to provide additional counseling and care to ensure the best surgical outcome.

Among adults, comorbidity assessment is well established. The Charlson Comorbidity Score (CCS) and Van Walraven Index (VWI) are two commonly used risk assessment scores designed to discriminate mortality with good validity and reliability [2]. The CCS has been validated in clinical practice, and its modification by Romano et al. has been used extensively in administrative database research [3]. The VWI is specifically designed for administrative database use, and it is a modification of the Elixhauser score. Previous studies comparing CCS and VWI in the administrative setting found VWI to be slightly superior to CCS in discriminating both mortality and readmission [4]. By contrast, validated comorbidity indices in pediatrics are scarce. Since the majority of children do not require long-term care and mortality is rare, pediatric risk assessments have been limited to acute care settings using physiologic parameters to discriminate impending in-hospital mortality of severely ill patients [5].

Recently, Rhee et al. [6] developed a pediatric-specific multispecialty risk index using 70 medical comorbidities to discriminate mortality in mostly general surgery patients with reported receiver operating characteristics (ROC) of 0.90. However, using mortality as an outcome discriminator may not be useful for pediatric urology patients given the exceedingly low perioperative mortality rates, and, thus, a more clinically relevant end point would be complications and readmissions.

Currently, there is no validated risk assessment index specifically for pediatric urology. Therefore, we aimed to determine if the CCS, VWI, and Rhee index could be extended beyond their designed purpose to discriminate postoperative complications and readmissions after urological procedures. We hypothesized that the Rhee index would have better performance for all outcomes than VWI and CCS.

Methods

Data sources

We used a compilation of several data sources in order to examine outcomes occurring in both inpatient and emergency settings. Specifically, we used data from the Healthcare Cost and Utilization Project (HCUP): the 2013 Nationwide Readmissions Database (NRD) and 2007–2010 State Inpatient Databases (SID) and State Emergency Department Database (SEDD) from California, Florida, North Carolina, New York, and Utah were selected to provide a broad geographic overview.

NRD is specifically designed to track inpatient readmissions. It includes 21 states that are geographically dispersed and account for 49% of all US hospitalizations [7]. However, the NRD cannot link to emergency room visits.

SID includes annual, state-specific inpatient data. SEDD is an annual, encounter-level data on emergency department (ED) visits to hospital-affiliated emergency departments that do not result in admissions. Using HCUP supplemental variables for revisit analysis, SID and SEDD can be linked to track sequential visits in the ER within a given timeframe.

Since the NRD cannot link emergency room visits, we primarily evaluated how well each index can discriminate postoperative complications and modeled 30-day inpatient readmission as a secondary analysis in the NRD. To model 30-day inpatient and ED admissions, we used SID and SEDD as these two databases can be linked to track a specific patient within and across the two settings. Per HCUP requirements, we limited reporting of any events occurring in a minimum of 15 patients in either database.

Comorbidity indices

Charlson Comorbidity Index

The CCS includes 17 disease conditions that are weighted based on their relative severity. A summary score is calculated based on the sum of the weights [8]. The CCS can be adopted for administrative database research by using the Enhanced ICD-9-CM diagnosis codes listed in Quan et al. [9].

Van Walraven Index

The VWI is a modification of the Elixhauser comorbidity score [10]. The Elixhauser score includes 30 dichotomous disease variables instead of an overall index score. To overcome this limitation, Van Walraven et al. [11] derived weights to summarize the 30 variables into a single score. The possibility of having a negative comorbidity score in VWI should be emphasized. As previously explained by Elixhauser et al. [10] this is mainly a coding bias inherent to administrative database. For example, severely ill patients have so many diagnoses that the more “benign” conditions are overlooked and not coded. Conversely, a healthy patient is more likely to have a benign diagnosis coded in the absence of other more serious conditions. Thus, the presence of codes for nonthreatening diseases is a proxy for healthy patients and associated with a decreased risk of in-hospital death.

Rhee Index

The Rhee Index is a pediatric specific risk assessment tool designed to discriminate mortality for mostly general surgery patients. The index consists of a 7-point scale developed using 70 comorbid variables. It has an excellent reported AUC of 0.90 for discriminating mortality after pediatric surgery [6].

Cohort selection

We included patients less than 18 years of age who underwent inpatient urology specific procedures identified by International Classification of Diseases, 9th revision, Clinical Modification (ICD-9) codes for ureteroneocystostomy (56.74), ureteroureterostomy (56.41), radical/partial nephrectomy (55.5/55.4), pyeloplasty (55.87), appendicovesicostomy (44.52), enterocystoplasty (57.87), vesicostomy (44.51), and bladder neck sling (59.68) in both NRD and SID/SEDD.

Only patients with a single unique surgical procedure during their index hospitalization were included. Surgical encounters with multiple procedures done during a single anesthesia event (i.e., enterocystoplasty with bladder neck sling) were excluded to minimize bias in outcome discrimination.

Outcome selection

We examined three primary outcomes: postoperative complications, 30-day inpatient readmissions, and 30-day ER admissions.

Postoperative complications were identified from NRD based on definitions in the National Surgical Quality Improvement Program (NSQIP), which include surgical site infection (998.32, 998.31), peritoneal abscess (567.22), acute renal failure (584, 586), urinary tract infection (599.0), respiratory complication (997.3, 518.81, 518.82, 514, 518.4), pneumonia (481, 482, 483, 484, 485, 486, 487, 507), postoperative respiratory insufficiency (518.5), acute respiratory insufficiency or acute respiratory distress syndrome (518.82), acute respiratory failure (518.81), sepsis (790.7, 995.91), pulmonary embolism (415.1, 415.11, 415.19), postoperative cardiovascular accident (997.02), cardiac complications (997.1), myocardial infarction (410), cardiac arrest (427.5), bleeding (285.1, 998.11), and deep vein thrombosis (453.4, 453.40, 453.9). As a secondary analysis, we also modeled how well the three indices can discriminate 30-day inpatient readmission in NRD and compared this result with the ones obtained in SID.

We calculated 30-day inpatient readmissions from the date of the index procedure in the SID. Admissions after December 1, 2010, were excluded to ensure a complete 30- day readmission window. We linked SID visits to SEDD to determine if a 30-day ER admission had occurred from the initial visit.

Statistical analysis

When assessing the NRD, we used weighted descriptive statistics to describe the patient demographics while accounting for the complex survey design. For the SID we used standard descriptive statistics due to lack of weights.

Receiver operating characteristics (ROC) curves were constructed using GEE models for the SID due to variance within each hospital to change by year, and weighted logistic regression models for the NRD due to its complex survey design. We first evaluated the how well each index can discriminate in-hospital death within the NRD to evaluate how well they can perform for the factor that they were designed to evaluate. Next, we modeled our three outcomes (postoperative complications (yes/no), 30-day inpatient readmissions, and 30-day ER admissions) by fitting three models for each outcome as the primary discriminator without covariates. We chose not to account for possible confounders because our primary goal was to evaluate the discriminability of these comorbidity scores for outcomes that they were not designed to discriminate (i.e., readmissions and complications rather than mortality) as we were primarily interested in their individual discriminative strength with our outcomes of interest.

To compare the performance of the ROC curves, we used the DeLong, DeLong, and Clarke–Pearson non-parametric approach to measure discrimination between the different comorbidity indices [12]. We used the CCS as the reference model.

To ensure that excluding the multiple procedures did not significantly bias results, we performed a sensitivity analysis including patients who had undergone multiple concurrent procedures in the cohort.

An alpha of 0.05 and 95% CI were used as criteria for statistical significance. All analyses were performed using SAS, version 9.4 (SAS Institute, Cary, NC, USA).

Results

Patient demographics

A total of 8,006 patients were identified in NRD and 6,236 patients in SID. The average age was 4.9 years (SE 0.3) in NRD and 5.6 years (SE 5.1) in SID. The overall sex ratio was evenly distributed in both databases. The three most common urological procedures in both databases were ureteroneocystostomy, pyeloplasty, and nephrectomy (Table 1). The median score was approximately 0 for all three indices. In addition, the Rhee index had more patients with a score of 1 than the other two indices (Table 2).

Table 1.

Demographics

	NRD	SID
N total	8006	6236
Age (yr, mean {se})	4.9 (0.3)	5.6 (5.1)
Male	4079 (50.9%)	2961 (47.5%)*
Female	3927 (49.1%)	3151 (50.5%)*
Public Payer	3492 (43.6%)	2495 (40.0%)*
Private Payer	4094 (51.1%)	3243 (52.0%)*
Other Payer	420 (5.3%)	493 (7.9%)*
VWI (median)	-0.4	0
CCI (median)	0	0
Rhee (median)	0	0
Ureteroneocystostomy	3294 (41.1%)	2687 (43.1%)
Ureteroureterostomy	65 (0.8%)	82 (1.3%)
Pyeloplasty	2473 (30.9%)	1437 (23.1%)
Nephrectomy	1470 (18.4%)	1404 (22.5%)
Partial Nephrectomy	332 (4.1%)	263 (4.2%)
Appendicovesicostomy	123 (1.5%)	91 (1.5%)
Enterocystoplasty	239 (3%)	198 (3.2%)
Bladder neck sling	15 (0.2%)	71 (1.1%)

^*Does not equal 100% due to missing data

Table 2.

Patient distribution by risk assessment scores

NRD		0	1	2	3	4	5	≥6
CCI		6,536 (81.6%)	365 (4.6%)	765 (9.5%)	147 (1.8%)	33 (0.4%)	*	156 (1.9%)
VWI		6,849 (85.5%)	35 (0.4%)	*	419 (5.2%)	20 (0.2%)	242 (3%)	436 (5.4%)
Rhee		4410 (55.1%)	3023 (37.7%)	312 (3.9%)	191 (2.4%)	30 (0.4%)	29 (0.4%)	15 (0.2%)
SID
CCI		5,528 (81.9%)	300 (4.4%)	671 (9.9%)	92 (1.4%)	23 (0.3%)	*	127 (1.9%)
VWI		5,812 (86.1%)	18 (0.3%)	*	356 (5.3%)	22 (0.3%)	183 (2.7%)	352 (5.2%)
Rhee		3,825 (56.6%)	1937 (28.7%)	182 (2.7%)	76 (1.1%)	43 (0.6%)	25 (0.4%)	29 (0.4%)

^*Count less than 15; also for Rhee index in SID, there were 635 (9.4%) missing records

Index performance in discriminating in-hospital mortality

The Rhee index had the best performance for discriminating in-hospital death (AUC = 0.85, 95% CI 0.67–1.00) than VWI (AUC = 0.66, 95% CI 0.41–0.91) and CCS (AUC = 0.50, 95% CI 0.37–0.63); p < 0.05 (Fig.1A). The Rhee score approached the reported AUC of 0.9 in an initial analysis of pediatric general surgery [6].

Figure 1.

(A) NRD in-hospital mortality. (B) NRD postoperative complications. (C) SID inpatient readmissions.

Performance in discriminating postoperative complications/readmission in NRD We observed that as the comorbidity score increased, the rate of complications also increased across all three indices (Table 3). The Rhee Index had the best performance for discriminating postoperative complications (AUC = 0.67, 95% CI 0.63–0.71) compared with CCS (AUC = 0.62, 95% CI 0.58–0.66) and VWI (AUC = 0.61, 95% CI 0.58–0.65); p < 0.05 (Fig. 1B). As a secondary analysis, we evaluated the performance of the three indices in discriminating 30-day inpatient readmission in NRD. We found that the CCS had the best performance for discriminating inpatient readmission (AUC = 0.63, 95% CI 0.59–0.67) followed by Rhee index (AUC = 0.58, 95% CI 0.54–0.62) and VWI (AUC = 0.54, 95% CI 0.50–0.58). The performance of these three indices in discriminating 30-day inpatient readmission in NRD is overall equivalent to those obtained in SID/SEDD, as described below (Supplementary Figure).

Table 3.

Complications/Readmissions per Risk Score

NRD Score	CCS complication rate	P	VWI complication rate	P	Rhee complication rates	P
0	415 (6.3%)	<0.001	409 (6%)	<0.001	226 (5.1%)	<0.001
1 to 3	211 (16.6%)		75 (16.1%)		382 (10.8%)
4+	37 (19%)		180 (25.8%)		55 (76%)
SID Score	CCS Inpt. readmissions	P	VWI Inpt. readmissions	P	Rhee Inpt. readmissions	P
0	291 (5.7%)	<0.001	364 (6.8%)	<0.001	256 (6.5%)	<0.001
1 to 3	150 (15.7%)		35 (9.6%)		229 (10.5%)
4+	50 (31.3%)		92 (16.8%)		*(6.2%)
SID Score	CCS ED readmissions	P	VWI ED readmissions	P	Rhee ED readmissions	P
0	408 (8%)	0.03	426 (8%)	0.02	328 (8.3%)	0.13
1 to 3	99 (10.3%)		37 (10.1%)		193 (8.8%)
4+	17 (10.6%)		61 (11.2%)		* (3.1%)

^*Exact number not reported due to count less than 15

Index performance in discriminating postoperative readmissions in SID/SEDD We observed that as the comorbidity score increased, the rate of inpatient and ED admissions also increased, except for the Rhee Index (Table 3). The CCS had the best discrimination for 30-day inpatient readmissions (AUC = 0.63; 95% CI 0.60–0.65) compared with VWI (AUC = 0.54; 95% CI 0.52–0.56) and the Rhee Index (AUC = 0.56; 95% CI 0.53–0.58); p < 0.0001) (Fig. 1C). For ER admissions, all three indices had similar discriminative property: CCS (AUC = 0.52; 95% CI 0.50–0.54), VWI (AUC = 0.51; 95% CI: 0.49–0.53), and Rhee Index (AUC = 0.50; 95% CI 0.47–0.52); p = 0.5 (Fig. 1D).

Sensitivity analysis including multiple procedures in the cohort

The demographic features were similar between the analytical cohort and the sensitivity cohort (Supplementary Table S1); multiple procedures made up 3.7% of the NRD encounters and 3.9% of the SID encounters. The distribution of the comorbidity scores was similar between the cohorts (Supplementary Tables 2 and 3). Most importantly, the AUCs for all three indices in the multiple-procedure cohort were similar to those in the single-procedure cohort (Supplementary Table 4).

Discussion

Given the absence of a validated surgical risk assessment index in pediatric urology, we compared the performance of the CCS, VWI, and Rhee Index in discriminating postsurgical complications and readmission for pediatric urology patients. To the best of our knowledge, this is the first such study aimed to determine the best surgical risk assessment index for pediatric urology patients among existing indices.

We found that the Rhee Index and CCS have moderate discriminative ability for complication and inpatient readmission, respectively. However, with an AUC = 0.7 as a cutoff for good discriminative ability, none of three indices performed well for postoperative complications, ER admissions, and inpatient readmissions. Among the three indices, the Rhee Index had the best performance for discriminating postoperative complications compared with CCS and VWI. This is not unexpected given that the Rhee Index was developed using a pediatric cohort. Similarly, we hypothesized the Rhee Index would also have a better discriminative ability for both readmission types. However, interestingly, the CCS had a better performance in discriminating 30-day inpatient readmissions among the three indices tested in both NRD and SID/SEDD, and all three indices performed similarly poorly in discriminating ER readmissions. None of the indices reached an AUC greater than 0.7. In order to improve risk stratification in pediatric urology, a new comorbidity index needs to be developed.

Comorbidity indices have long been widely used in the adult population. The CCS is one of the most commonly used measures for mortality across all specialties, including urology. In addition to discriminating mortality, the CCS has also been used to discriminate post urological complications such as after percutaneous nephrolithotomy and transurethral resection of the prostate [13, 14]. It was interesting that the CCS had the worst performance within the NRD for in-hospital mortality. A possible explanation is that the CCS was developed using medical history from adult patients and thus not as robust for pediatric urology cohort in the administrative setting.

To mirror the applicability of CCS for adults, Rhee et al. [6] developed a multispecialty surgical comorbidity score for pediatric patients using the Nationwide Inpatient Sample and Kid’s Inpatient Database. The introduction of the Rhee index was a major step forward to pediatric surgical research, as it provides a validated risk assessment score for pediatric patients overall. Indeed, not surprisingly, the Rhee Index showed excellent ROC for discriminating mortality in our cohort (AUC = 0.85) similar to the original study (AUC = 0.90) [6].

Mortality is a rare event in pediatric general surgery, and even more so in pediatric urology. Thus, using mortality as a risk-adjusted outcome may be less useful among typical (i.e., low-risk) pediatric urology patients. For example, a prior analysis using NSQIP data showed that mortality for pediatric urological surgery was nearly nil, while the overall mortality for pediatric surgeries was only 0.3%. Therefore, mortality may have minimal utility as a performance indicator for children’s surgery [15] To provide better patient counseling and optimization of care prior to pediatric urological surgery, more clinically oriented risk-adjusted outcomes would be complication or readmission. For example, a prior single-institution analysis reported a 39% complication rate following lower urinary tract reconstructive surgery such as appendico-vesicostomy, especially procedures with bowel involvement were found to be associated with a higher rate of complications [16, 17].

Although complications and readmissions are more prevalent in pediatric patients than mortality, these are still relatively scarce. For example, the absolute number of readmissions/complications are low in our study, and this could be due to three factors: (1) pediatric urology patients are in general relatively healthy, and thus have fewer patient-related factors for complication or readmission (i.e., poor wound healing due to diabetes or immunosuppression); (2) the ICD-9 codes and NSQIP complications may not be specific enough for pediatric urology patients to capture the necessary complications; and (3) if a patient was readmitted into a hospital not included in our sample, then this readmission would not be captured. We suspect a combination of these factors is contributing to the low complication and readmission rates.

Recently, a pediatric surgical risk calculator was developed using NSQIP data and is designed to discriminate mortality and morbidity. This risk calculator showed a high discriminative property for mortality and modest discriminative property for complications with a c statistic of 0.97 and 0.77, respectively [18]. However, readmission was not included in the calculator. Since inpatient and ER readmission are also major components that can impact healthcare outcomes, readmission should be an important variable to consider when developing a risk assessment model. Furthermore, outpatient procedures such as urethral/testicular surgeries accounted for the majority of urological procedures included in the risk calculator model; and thus, its applicability to inpatient urological procedures is uncertain.

Our findings should be viewed in the context of study limitations. First, NRD and SID are constructed to capture only inpatient urological procedures. Thus, outpatient procedures such as orchiopexy and hydrocele repair were not included in our analysis. However, the impact of this omission is likely minimal because the low surgical risks associated with outpatient procedures. Furthermore, only those complications described in NSQIP were captured, so the total number of complications was likely underestimated. Multiple procedures in a single operation were excluded from analysis to limit bias in outcome discrimination. For example, by including multiple procedures in a single anesthesia event, it would be difficult to estimate which of the two procedures is ultimately linked to a complication or readmission. However, this would be unlikely to significantly impact the conclusions of our study and this was verified in our sensitivity analysis including multiple concurrent procedures.

An additional limitation was that complications managed in clinic are not captured. Thus, minor complications such as surgical site infection and secondary procedures such as ureteral stent removal were likely missed given the outpatient nature of these procedures.

We attempted to construct calibration curves for our outcomes using the Hosmer– Lemshow method. However, due to the low event rates in our dataset and the fact that the Hosmer–Lemshow test could only group predicted probabilities into no more than four groups, we were unable to reliably construct calibration curves.

Parental anxiety is a situation unique to pediatric surgery. Therefore, it would be expected that some ER visits did not necessarily correlate with true surgical complications, but rather parental concerns. However, this factor is highly variable, difficult to evaluate, and likely had minimal effect on our study outcome.

ICD-9 diagnosis and procedure codes are limited by the details of coding. Coding inconsistency can occur given that individuals with different clinical experiences perform these data entry. However, both NRD and SID are routinely monitored by HCUP for coding accuracy, and it likely provides a reasonably reliable representation of the study cohort. In addition, for SID, patient representation was limited to the five states from which we had data available, and it may not have provided a true representation of the US population.

We have certainly omitted other published pediatric risk assessment indices that are specific for non-surgical risk assessment. However, the main purpose of our study was to evaluate how well some of the common surgical risk assessment indices (adult or children) can discriminate outcomes in pediatric urology cohorts. Not to our surprise, none of them had good discriminative property. Our findings should serve as a cautionary tale for utilizing these comorbidity indices for studying pediatric urology procedures.

Finally, one could argue that the reason all three indices had poor discrimination is that they are designed to discriminate mortality instead of complications or readmission. However, risk indices that can discriminate complication and readmission currently do not exist for pediatric urology patients. The CCS and VWI have been previously used as discriminators for complication and readmission in the administrative setting for adults [4]. Thus, we applied the same methodology for pediatric urology patients. The poor discriminative performance of the models employing these indices motivates the need for the development of a new index that can be used to discriminate outcomes of interest in pediatric urology patients.

Conclusions

The Rhee Index had the best performance for postoperative complications compared with VWI and CCS. The CCS had the best performance for 30-day inpatient readmissions. All three indices had similarly poor performance for 30-day ER admissions with none of the AUCs surpassing 0.53. None of our discriminative models reached an AUC of 0.7. To improve risk stratification, patient counseling, and outcome research, a new risk index is needed unique to pediatric urology patients.