Identifying and Codifying Complications after Radical Cystectomy: Comparison of Administrative Diagnostic and Procedure Codes, and Clinical Chart Review

Abstract

Purpose:

To our knowledge the reliability of administrative claims codes to report postoperative radical cystectomy complications has not been examined. We compared complications identified by claims data to those abstracted from clinical chart review following radical cystectomy.

Methods:

We manually reviewed the charts of 268 patients treated with radical cystectomy between 2014 and 2016 for 30-day complications and queried administrative complication coding using 805 ICD-9/10 codes. Complications were categorized. Using Cohen κ statistics we assessed agreement between the 2 methods of complication reporting for 1 or more postoperative complications overall, categorical complications and complications stratified by the top quartile length of hospital stay and patients who were readmitted.

Results:

At least 1 or more complications were recorded in 122 patients (45.5%) through manual chart review and 80 (29.9%) were recorded via claim coding data with a concordance rate of κ = 0.16, indicating weak agreement. Concordance was generally weak for categorical complication rates (range 0.05 to 0.36). However, when examining only the top length of stay quartile, 1 or more complications were reported in 32 patients (65%) by the manual chart review and in 12 (25%) via coding data with a concordance rate of κ = −0.2. Agreement was weak, similar to the total cohort.

Conclusions:

Manual chart review and claim code identification of complications are not highly concordant even when stratified by patients with an extended length of stay, who are known to have more frequent complications. Researchers and administrators should be aware of these differences and exercise caution when interpreting complication reports.

Radical cystectomy is a complex surgery associated with a complication rate ranging between 30% and 60%.^1,2 As interest in health outcome, policy and quality oriented research has increased, administrative claims data have been used to characterize postoperative complications after cystectomy.^3,4 Although prior studies have revealed significant differences in identifying complications based on administrative codes data compared to clinical data after hysterectomy, spine surgery, oral surgery and prostatectomy, information is lacking on the reliability of claims codes to identify complications after radical cystectomy.^5–8

Administrative claims codes provide an efficient way to evaluate health care practices and outcomes across different hospital and clinical settings at a scale difficult to achieve with other data sources.^9–11 In addition to being increasingly used in research,^3,4,12,13 claims codes are key measures for a number of quality improvement and performance measurement programs.^14,15 However, because they lack clinical granularity, these codes may not be reliable measures of every type of outcome.^8,15–17 For example, postoperative complications may not be completely captured with claims.¹⁸

Reliable and consistent complication reporting is critical for comparative outcomes research, quality assessment programs and policy initiatives targeting postoperative complications. While claims codes are already being used to study cystectomy related complications in some of these contexts, reliability has not yet been established. Accordingly the objective of this study was to assess the reliability of administrative claims codes to identify complications after cystectomy.

METHODS

Data Sources and Study Population

Patients treated with radical cystectomy of bladder cancer between February 2014 and April 2016 at our institution were identified through an institutional cystectomy registry. Clinical variables, including clinical stage, the American Society of Anesthesiologists™ score, the Elixhauser comorbidity index,¹⁹ body mass index, preoperative anemia, hospital LOS and readmission were abstracted from the clinical record according to a standardized protocol. These data were then linked to health information from an integrated institutional analytics platform (a HRI system) which includes administrative claims codes.

Complications

Clinically Reported.

Postoperative complications from day 1 to day 30 after radical cystectomy were recorded and management of each complication was detailed with clinical chart abstraction. Complications were defined as any unexpected deviation from the standard postoperative course and graded according to the Clavien-Dindo classification system.²⁰ Recorded complications were then categorized into groups, including hemorrhage/bleeding; cardiac, respiratory, gastrointestinal, genitourinary, wound/hernia, thrombosis/embolic, infection/sepsis, renal failure/metabolic/dehydration/failure to thrive, neurological and miscellaneous/surgical (supplementary Appendix, https://www.jurology.com).² Related events were reduced to a single complication for each category to avoid duplicates and allow for category level analyses.

Administrative Coded Based.

HRI data include CMS reportable diagnostic and procedural ICD-9 and ICD-10 codes which are date annotated. All new inpatient and outpatient procedure and diagnostic ICD-9 and 10 codes within 30 days of radical cystectomy were organized into a searchable data set. Notably the transition to ICD-10 coding occurred on October 1, 2015, spanning the course of this study.

We collected ICD-9 and ICD-10 complication codes listed in institutional resources and used in prior publications,^4,12,21 and then used coding crosswalks to translate ICD-10 codes to the corresponding ICD-9 code.²² To ensure a comprehensive search we included diagnostic and procedure codes. For example, the ICD-9 diagnostic code 998.1 (hemorrhage complicating procedure) was included along with the ICD-9 procedure code 99.0 (transfusion of blood and blood components). A comprehensive data set of a total of 805 unique ICD-9 and 10 codes was created and categorized (supplementary table, https://www.jurology.com). Code based complications were assigned a Clavien score when possible.²⁰

Reliability Testing and Statistical Analysis

Complications were examined as binary variables to avoid counting duplicates at the patient level. Clinical review and chart abstraction were considered the reference standard to identify complications. Positive, negative, false-positive and false-negative findings were then calculated for code based complications overall (the identification of 1 or more postoperative complications) and in complication categories. As secondary analyses complications were also examined by Clavien grade in the top LOS quartile²³ in hospital readmission cases and by surgery date before or after conversion to the ICD-10. Complication frequency was compared to LOS for each method using the Spearman correlation. Cases with a recorded complication in coding data but no complication in chart review data were considered false-positive cases while those without a complication in coding but recorded as having a complication on chart review were considered false-negative cases.

Interrater agreement between code and abstraction complication reporting was examined using κ statistics, which account for agreement by chance. Generally κ agreement is interpreted as poor—less than 0.4, moderate—0.4 to 0.75 and excellent—greater than 0.75.²⁴ A heat map was constructed to visualize patterns of positive, negative, false-positive and false-negative findings across complication categories (see figure). All statistical analyses were performed with SAS®, version 9.4. Study compliance and regulation were approved and overseen by the Moffitt Cancer Center Scientific Review Committee and the Institutional Review Board. The study received Institutional Review Board approval (IRB No. MCC16526).

RESULTS

Patient Characteristics

A total of 268 patients who underwent radical cystectomy and had complete available HRI and chart abstraction data were used in analysis. Table 1 lists demographic and clinical characteristics. Median patient age was 68 years. The population had significant comorbidities (Elixhauser median score 5) and more than 66% of the cases were classified as ASA III or greater. Median LOS was 6 days and 18% of patients were classified as having prolonged LOS, defined as LOS at the 75th percentile or greater, that is 8 days or longer. Of the patients 55 (21%) were readmitted.

Table 1.

Social, demographic and perioperative information in 268 patients

Median age (IQR)	68	(60–75)
No. female (%)	60	(22)
No. male (%)	208	(78)
No. race/ethnicity (%)
Caucasian	255	(95)
African American	4	(2)
Hispanic/Latino	6	(2)
Other	7	(3)
No. surgery date (%):
Before 10/2015 (ICD-9)	199	(74)
After 10/2015 (ICD-10)	69	(26)
Comorbidity score:
Median Elixhauser (IQR)	5	(3–6)
No. ASA (%)
No. ASA I (%)	1	(less than 1)
No. ASA II (%)	89	(34)
No. ASA III (%)	163	(62)
No. ASA IV (%)	10	(4)
No. kg/m2 body mass index (%):
Less than 30	110	(62)
30 or Greater	67	(38)
No. preop anemia (%):*
Yes	135	(50)
No	133	(50)
Median ml estimated blood loss (IQR)	750	(475–1,200)
No. urinary diversion (%):
Ileal conduit	212	(78)
Neobladder	39	(15)
Colon pouch	11	(4)
Other	4	(2)
Length of stay:
Median days (IQR)	6	(5–8)
No. prolonged (%)†	49	(18)
No. readmission (%):
Yes	55	(21)
No	213	(79)
No. clinical American Joint Committee on Cancer stage (%):
I (T1NxMx or less)	72	(27)
II (T2NxMx)	152	(57)
III (T3NxMx)	29	(11)
IV (T4NxMx)	14	(5)
Unknown	1	(less than 1)

^*Hemoglobin less than 11.5 gm/dl in females and less than 13 gm/dl in males.

^†LOS 75th percentile (8 days) or greater.

Heat map of agreement in complication reporting via chart abstraction as referent vs administrative coding data for 30-day postoperative complications only in patients with complications from highest to lowest κ in descending order. Each column represents individual patient. ARF, acute renal failure. FTT, failure to thrive. Misc., miscellaneous. Yellow indicates false-negative. Orange indicates false-positive. Light gray indicates negative. Dark gray indicates positive.

Complication Reporting

The number of complications identified by chart abstraction and administrative coding varied significantly depending on the reporting method (table 2). At least 1 complication within 30 days of surgery was identified by chart review in 122 cases (46%) and a complication was identified using coding data in 80 (30%). This pattern was similar in complication categories except for wound/hernia complications, for which the same number of complications were identified.

Table 2.

Complications on postoperative days 1 to 30 by reporting method in 268 patients

	No. Chart	Review (%)	No. Coding	Data (%)
No. 1 complication or more (%)	122	(45.5)	80	(29.9)
No. complication category (%):
Hemorrhage/bleeding	9	(3.4)	10	(3.7)
Cardiovascular	4	(1.5)	16	(6.0)
Respiratory	8	(3.0)	3	(1.1)
Gastrointestinal	35	(13.1)	9	(3.4)
Genitourinary	14	(5.2)	17	(6.3)
Wound/hernia	24	(9.0)	24	(9.0)
Thrombosis/embolic	13	(4.9)	2	(0.7)
Infection/sepsis	65	(24.3)	28	(10.4)
Renal failure, acidosis, electrolyte abnormalities, dehydration, failure to thrive	26	(9.7)	21	(7.8)
Neurological	4	(1.5)	–
Miscellaneous surgical	13	(4.9)	15	(5.6)
No. complication grade (%):
Minor (Clavien 1–2)	95	(35.4)	23	(8.6)
Major (Clavien 3–5)	33	(12.3)	3	(1.1)
Unclassified	–		54	(20.1)
No complication	140	(52.2)	188	(70.1)
No. length of stay/total No. (%):
Top quartile	32/49	(65.3)	12/49	(24.5)
Bottom 3 quartiles	90/219	(41.1)	68/219	(31.1)
No. readmission/total No. (%):
Yes	53/55	(96.4)	27/55	(49.1)
No	69/213	(32.4)	53/213	(24.9)
No. surgery date/total No. (%):
Before 10/2015	93/199	(46.7)	59/199	(29.6)
After 10/2015	29/69	(42.0)	21/69	(30.4)

Table 2 shows complication frequency stratified by complication grade, LOS, readmission and surgery date. All chart abstracted complications were assigned a Clavien grade but only 26 cases could be assigned an estimated Clavien grade using codes. In cases with an assigned Clavien grade agreement was weak for minor and major complications (κ = 0.34 and 0.34, respectively). Complications correlated positively with LOS for the manual chart review (r = 0.24, p <0.0001) and for coding data (r = 0.15, p = 0.01). When examining cases in the top quartile of LOS or readmission cases following hospital discharge after cystectomy, of which each isa surrogate for high complications,^2,23,25 claims codes identified complications less frequently than chart review. For example, 25% of the top quartile LOS cases had a coded complication while in 65% a complication was identified by chart review (κ = −0.2). Similarly in 49% of 55 readmitted patients a complication was identified by claims codes compared to 96% by chart review (κ = 0.07).

Of the patients 47 (18%) had 1 or more complications within 30 days of surgery as confirmed by chart review (positive). Conversely 113 patients (42%) had no coded complication or a complication based on chart review (negative). Of the remaining cases 33 (12%) and 75 (28%) were false-positive and false-negative, respectively.

The overall accuracy of complication codes, defined by the number of positive and negative cases divided by all cases, was only 60%. Agreement between identifying complications using coding data and chart abstraction was weak overall (κ = 0.16) and across complication categories (range −0.05 to 0.36). Wound/hernia complications had the highest agreement (κ = 0.36).

The figure shows a heat map of positive, negative, false-positive and false-negative cases stratified by complication type. The highest number of positive cases was seen in the wound/hernia group. The highest number of false-negative cases occurred in the infection/sepsis category. False-positive cases were noted most commonly for genitourinary complications.

The supplementary table (https://www.jurology.com) shows the full categorical complication count. The sensitivity and specificity of claims codes to identify overall complications (ie 1 complication or more) was 38.5% and 77.4%, respectively. In the complication categories sensitivity ranged from 5.7% to 50% and specificity was greater than 90% (table 3).

Table 3.

Concordance index, sensitivity, specificity, and positive and negative predictive values of coding data vs manual chart review for complications overall and by category

					% Predictive Value
Complications	κ Concordance	Index (95% CI)	% Sensitivity	% Specificity	Pos	Neg
Overall (1 or more)	0.16	(0.06–0.28)	38.5	77.4	58.8	60.1
Category:
Hemorrhage/bleeding	0.29	(0.013–0.57)	33.3	97.3	30.0	97.7
Cardiovascular	0.18	(−0.05–0.41)	50.0	94.7	12.5	99.2
Respiratory	0.17	(−0.14–0.48)	12.5	99.2	33.3	97.4
Gastrointestinal	0.04	(−0.077–0.16)	5.7	97.0	22.2	87.3
Genitourinary	0.008	(−0.12–0.13)	7.1	93.7	5.9	94.8
Wound/hernia	0.36	(0.17–0.55)	41.7	94.3	41.7	94.3
Thrombosis/embolic	0.12	(−0.11–0.35)	7.7	99.6	50.0	95.5
Infection/sepsis	0.26	(0.13–0.39)	26.2	94.6	60.7	80.0
Acute renal failure, acidosis, failure to thrive	0.28	(0.093–0.46)	30.8	94.6	38.1	92.7
Neurological		–	–	–	–	98.5
Miscellaneous surgical	−0.05	(−0.08–−0.03)	–	94.1	–	94.9

Referent is manual chart review.

DISCUSSION

Administrative claims codes are used as outcome and quality measures but they lack clinical detail and context may limit usefulness in some cases.^16,17 Although claims codes have been used in a number of studies focused on complications after cystectomy,^3,4 there is scant evidence regarding the ability of claims codes to capture complications reliably. When comparing complications identified by claims codes vs clinical chart abstraction in a cohort of cystectomy cases, we found that agreement between the 2 approaches was poor even in clinical circumstances associated with high complication rates (eg extended LOS and readmission). Using a structured chart review as the reference standard for complication reporting,^2,26 the sensitivity for identifying any and specific types of complications by administrative claims codes was low at 5.7% to 50%. Conversely specificity was fairly high. These findings are similar to those in previous large database studies of CMS claims data compared to data from the ACS (American College of Surgeons) NSQIP® (National Surgical Quality Improved Program).^16,17 They suggest that claim codes are not reliable to identify complications due to the significant frequency of false-negative data.

Consistent with prior reports, at 24% and 13% infection/sepsis and gastrointestinal complications, respectively, were the most common complications identified by chart review.^1,2 The false-negative rates of infection/sepsis and gastrointestinal complications were high at 18% and 12%, respectively (see figure and supplementary table, https://www.jurology.com). The figure also shows the highest false-positive rates in the genitourinary and miscellaneous surgery complication categories (each 6%). However, false-positive complication codes were less common than false-negative codes across complication categories, indicating that overreporting complications with claims codes is less concerning than underreporting. The overall accuracy of complication codes was only 60%, further underscoring the limited ability of administrative codes to identify complications.

As the low κ scores reflect, agreement between identifying complications by chart review and claims data was poor regardless of complication category or substratification. Similarly, Tollefson et al reported poor concordance between administrative data and clinical outcomes following prostatectomy.⁷ Also, the results of several studies suggest suboptimal alignment of code based quality measures and clinical outcome data obtained through structured chart review.^16,27–29 Researchers who use administrative data sets to examine outcomes after cystectomy should recognize the performance limitations of claims codes.

Our results highlight an important aspect of complication reporting which cannot be fully implemented with claims codes. The majority of complications identified using coding data, that is 54 of 80 (67.5%), could not be assigned a Clavien grade due to the lack of clinical detail or context associated with the coded diagnosis, treatment or procedure. Given that twice as many complications could not be assigned a grade vs those which could be graded using coding data, it is difficult to draw meaningful conclusions regarding grade of complication.

The pitfalls of assigning complication grades are apparent in a comparison of robotic and open radical cystectomy using a hospital payer discharge database.⁴ Using ICD-9 codes to identify cystectomy complications and estimate Clavien grade, Leow et al reported fewer minor complications following robotic cystectomy and no differences in major complications between open and robotic cystectomy.⁴ Although some major complications (eg cardiac arrest) can be assigned a Clavien score based on diagnostic severity and/or a procedural code indicating intervention,³⁰ this is an uncommon exception.

Based on our findings, estimating complication grades based on administrative codes is problematic, particularly for low grade complications. For example, we categorized code 790.7 (bacteremia) as a Clavien II complication. However, the clinical severity of bacteremia is not captured by administrative codes. Bacteremia requiring intensive care unit management would be considered a major (grade III) complication according to the Dindo-Clavien classification.³⁰ Furthermore, preexisting conditions requiring a diagnosis code for postoperative treatment may be inaccurately captured as complications. All 14 false-positive cardiac complications were related to preexisting atrial fibrillation, which required treatment postoperatively and was erroneously captured as a complication when querying administrative data.

Clinical registries and large administrative data sets are important sources of health data. Accordingly they have become common resources for clinical outcomes, health services and quality assessment research. Because not all outcome and process measures are captured accurately with administrative codes, important limitations are inherent in using these data sources. In the case of complication assessment the lack of reliability and the inaccuracy of claims codes are well known.^16–18

Although they were examined previously at a general level, specific comparisons of complication assessment using claims codes vs clinical data in patients who undergo cystectomy have not been reported to date to our knowledge. Lawson et al previously described the suboptimal performance of using administrative codes to report 30-day surgical complications compared to NSQIP data.¹⁶ Furthermore, Potosky et al found that Medicare claims data have significant limitations in capturing prostatectomy complications, particularly complications not prompting an intervention.¹⁸ Our results mirror these findings and also show that codes underreport complications in clinical settings associated with high complication rates, such as prolonged LOS and readmission. Given the current efforts to focus on and reduce readmissions, the disconnect between administrative codes capturing cystectomy complications and readmission is concerning. In our analysis only 49% of readmitted patients had an associated complication code.

Our study reemphasizes that administrative code algorithms should be used cautiously when assessing complications. It also highlights the inconstancies of using hospital administrative data (CMS coding data), originally designed for billing purposes, to measure quality. Administrative claims codes have been used widely to assess outcomes and as a proxy for quality without necessary validation studies. Our findings underscore the constraints of this approach.

Several limitations should be considered. 1) Our study time frame included the transition from ICD-9 to ICD-10. While we used crosswalks to back-transform ICD-10 codes and categorize complications in groups applicable to cystectomy related complications, errors in translational and repeat categorizations could have occurred as direct ICD-9 to ICD-10 translations do not exist. To minimize the chance of translation error all ICD-10 complication codes were categorized with clinical review. Although ICD-10 codes are designed to be more specific and detailed, we did not identify different results according to ICD-9 vs ICD-10 codes.

2) Not all codes indicating complications may have been included in our code dictionary and algorithm. To minimize the risk of excluding important codes we used more than 800 ICD-9 and 10 codes. We also examined origin sources for codes denoting complications, including ICD system taxonomy for complications (eg ICD-9 900 series) and code lists used in prior publications.

3) Our analysis was limited to ICD-9 and 10 codes. HCPCS (Healthcare Common Procedure Coding System) CPT codes were not included. This exclusion may have prevented us from identifying some physician based services linked to complications. However, we believe that the risk of large omissions is low, given that diagnostic and procedural ICD codes were included.

4) Complication identification, inclusion or exclusion by clinical chart review may be subject to human judgment. Thus, misclassification error is possible. We used a standardized approach to chart review, specifying source documents for review, conditions, clinical scenarios, diagnoses and procedures associated with the development of a complication and criteria to classify and grade complications for chart review. These measures should have limited the risk of misclassification.

CONCLUSIONS

Notwithstanding possible limitations, the level of disagreement of ICD-9 and 10 codes with clinical chart review and the relatively poor performance of code based complication reporting, including underreporting and low sensitivity, show that claims codes are not reliable or complete sources for identifying complications after cystectomy. As health administrative data sources are increasingly used for research and quality comparisons, these results underscore the limitations and cautions related to using complications as outcome and/or quality measures.

Abbreviations and Acronyms

ASA

American Society of Anesthesiologists^™

CMS

Centers for Medicare and Medicaid Services

HRI

health and research informatics

LOS

length of stay