Abstract
Background:
Large studies evaluating pediatric acute kidney injury (AKI) epidemiology and outcomes are lacking, partially due to underuse of large administrative health care data.
Objective:
To assess the diagnostic accuracy of administrative health care data-defined AKI in children admitted to the pediatric intensive care unit (PICU).
Design:
Retrospective cohort study utilizing chart and administrative data.
Setting:
Children admitted to the PICU at 2 centers in Montreal, QC.
Patients:
Patients between 0 and 18 years old with a provincial health insurance number, without end-stage renal disease and admitted to the PICU between January 1, 2003, and March 31, 2005, were included.
Measurements:
The AKI was defined from chart data using the Kidney Disease: Improving Global Outcomes (KDIGO) definition (Chart-AKI). The AKI defined using administrative health data (Admin-AKI) was based on International Classification of Disease, Ninth Revision (ICD-9) AKI codes.
Methods:
Data available from retrospective chart review, including baseline and PICU patient characteristics, and serum creatinine (SCr) and urine output (UO) values during PICU admission, were merged with provincial administrative health care data containing diagnostic and procedure codes used for ascertaining Admin-AKI. Sensitivity, specificity, positive, and negative predictive value of Admin-AKI compared with Chart-AKI (reference standard) were calculated. Univariable associations between Admin-AKI and hospital mortality were evaluated.
Results:
A total of 2051 patients (55% male, mean age at admission 6.1 ± 5.8 years, 355 cardiac surgery, 1696 noncardiac surgery) were included. The AKI defined by SCr or UO criteria occurred in 52% of cardiac surgery patients and 24% of noncardiac surgery patients. Overall, Admin-AKI detected Chart-AKI with low sensitivity, but high specificity in cardiac and noncardiac surgery patients. Sensitivity increased by 1.5 to 2 fold with each increase in AKI severity stage. Admin-AKI was associated with hospital mortality (13% in Admin-AKI vs 2% in non-AKI, P < .001).
Limitations:
These data were generated in a PICU population; future research should study non-PICU populations.
Conclusions:
Use of administrative health care data to define AKI in children leads to AKI incidence underestimation. However, for detecting more severe AKI, sensitivity is higher, while maintaining high specificity.
Keywords: diagnostic codes, administrative codes, administrative database, diagnostic accuracy, sensitivity, specificity, positive predictive value, negative predictive value, critically ill children, validation
Abrégé
Contexte:
On dispose de peu d’études à grande échelle évaluant l’épidémiologie et l’évolution de l’insuffisance rénale aigüe (IRA) chez les enfants, notamment en raison d’une sous-utilisation des données administratives du système de santé.
Objectif:
Évaluer la précision diagnostique de l’IRA définie à partir des données administratives en santé chez des enfants admis aux unités de soins intensifs pédiatriques (USIP).
Type d’étude:
Une étude de cohorte rétrospective utilisant des données administratives et les données provenant des dossiers médicaux.
Cadre:
Les USIP de deux centres hospitaliers de Montréal, au Canada.
Sujets:
Ont été inclus les patients âgés de 0 à 18 ans possédant un numéro d’assurance-maladie provincial qui ont été admis aux USIP entre le 1er janvier 2003 et le 31 mars 2005 avec une insuffisance rénale non terminale.
Mesures:
L’IRA-Dos a été définie à partir des dossiers médicaux en utilisant les critères du KDIGO (Kidney Disease: Improving Global Outcomes). L’IRA-Admin a été définie à partir des données administratives en santé avec les codes d’IRA de la neuvième révision de la Classification internationale des maladies (CIM-9).
Méthodologie:
Les données tirées de l’examen rétrospectif des dossiers médicaux, soit les valeurs de créatinine sérique (SCr) et de diurèse pendant le séjour aux USIP et les caractéristiques des patients, initiales et à l’admission, ont été fusionnées aux données administratives provinciales en santé contenant les codes de diagnostic et de procédure utilisés pour établir l’IRA-Admin. La sensibilité, la spécificité et les valeurs prédictives négative et positive de l’IRA-Admin, en comparaison à l’IRA-Dos (standard de référence), ont été calculées. L’association univariée entre l’IRA-Admin et la mortalité à l’hôpital a également été évaluée.
Résultats:
Un total de 2 051 patients ont été inclus (355 ayant subi une cardiochirurgie et 1 696 non opérés). L’âge moyen des sujets à l’admission était de 6,1 ± 5,8 ans et 55 % étaient des garçons. L’IRA définie par les critères de SCr et de diurèse a été diagnostiquée chez 52 % des patients opérés et chez 24 % des patients non opérés. Pour l’ensemble de la cohorte (patients opérés ou non), l’IRA-Admin a détecté l’IRA-Dos avec une faible sensibilité, mais avec une spécificité élevée. La sensibilité s’est accrue de 1,5 à 2 fois pour chaque passage à un stade supérieur de gravité de l’IRA. Enfin, l’IRA-Admin a été associée à un taux plus élevé de mortalité à l’hôpital (13 % des patients IRA-Admin contre 2 % des patients sans IRA, p<0,001).
Limitations:
Ces résultats concernent une population de patients hospitalisés aux USIP. Des études futures devraient inclure des populations non admises aux USIP.
Conclusions:
L’utilisation des données administratives en santé pour définir l’IRA chez les enfants a mené à une sous-estimation de son incidence. Cependant, la méthode montre une plus grande sensibilité dans la détection des cas plus graves d’IRA, tout en conservant une spécificité élevée.
What was known before
Acute kidney injury (AKI) is associated with poor short- and long-term outcomes in children. There have been no studies in children evaluating diagnostic accuracy of AKI diagnostic codes from administrative databases.
What this adds
Understanding the accuracy of administrative data for identifying AKI in children, and studying their relationship with outcomes, will strongly enhance future research aimed at determining pediatric AKI health outcomes using administrative data.
Introduction
Acute kidney injury (AKI) in children admitted to the pediatric intensive care unit (PICU) is associated with increased PICU and hospital mortality, prolonged length of stay, and prolonged mechanical ventilation.1-4 Robust data in adults and emerging data in children suggest that AKI is also a risk factor for long-term renal disease and associated cardiovascular morbidity.1,2 Most published pediatric AKI research to date has consisted of retrospective chart reviews, which are expensive, time-consuming, and challenging to perform as multicenter studies.
Administrative health care databases are a rich source of population-based information, which may be used to evaluate outcomes of large populations in a cost-effective manner and have fewer logistic barriers compared to traditional prospective cohort designs.5,6 However, the extent to which research using administrative databases, which mainly use diagnostic and procedure codes, accurately capture AKI diagnosis and reflect associations of AKI with outcomes in children is unclear.
In adults, several studies have evaluated the validity of administrative database AKI diagnostic and procedure codes, by comparing administrative data-defined AKI with the reference standard definition from chart review.6 In general, administrative databases in adults have shown poor sensitivity, but high specificity to identify AKI.6 No studies have formally evaluated the diagnostic accuracy of administrative database-defined AKI in critically ill children, despite the fact that recent studies have begun using such methods to study pediatric AKI outcomes.7 Understanding the accuracy of administrative data for identifying AKI in children, and studying their relationship with outcomes, will strongly enhance future research aimed at determining pediatric AKI health outcomes.
We performed a diagnostic accuracy study of administrative health care data-defined AKI in children admitted to the PICU, using chart review as the reference standard. A secondary objective was to determine the extent to which AKI defined using administrative health care data is associated with hospital mortality, as a surrogate evaluation of prognostic value and validity.
Methods
Study Design and Population
This diagnostic accuracy study was a secondary analysis using data previously collected from a retrospective cohort study of children admitted to 2 tertiary care PICUs (Montreal Children’s Hospital [MCH] and Centre Hospitalier Universitaire de Sainte-Justine [CHUSJ]) between January 1, 2003, and March 31, 2005.8,9 Inclusion criteria for the original study were PICU admission during the study period and age 0 to 18 years on the day of PICU admission. Only the first admission during the study period per subject was used. Exclusion criteria included a diagnosis of end-stage renal disease (ESRD) (dialysis or transplant) at PICU entry, on the kidney transplant list prior to PICU admission or ESRD diagnosis made during the index hospitalization, or missing a provincial health insurance number. Subjects were further excluded from this analysis if they had neither serum creatinine (SCr) nor urine output (UO) PICU data available from chart review. Approvals from institutional research ethics boards and the Commission d’accès à l’information du Québec (CAI, provincial ethics body) were obtained and requirement for patient consent was waived.
Data Source and Collection for Retrospective Chart Review
Medical charts and electronic medical records of eligible patients were reviewed retrospectively. A standardized case report form was developed by the investigators and iteratively refined by consensus, followed by evaluation for test-retest and interrater reliability. Data collected included baseline patient demographics, PICU diagnostic and illness severity data, PICU medication and complications, and PICU and hospital outcomes. Pertinent to the current study, pre-PICU (baseline) SCr, highest daily PICU SCr, and daily UO values (in 8-hour time periods) were recorded for the purpose of AKI status determination. Baseline SCr was defined as the lowest SCr value in the 3 months before PICU admission. If baseline SCr was unavailable, it was estimated by back-calculation using a previously validated method in children.10
Data Sources and Collection for Administrative Health Care Data
Chart data described above from PICU admission was merged with data from the Régie de l’assurance maladie du Québec (RAMQ) and Med-Echo (provincial health care administrative databases), and with the Quebec Vital Statistics Registry database (containing mortality data). The database pertinent to this study is an acute care hospitalization database, including admission and discharge dates and primary diagnoses (International Classification of Disease, Ninth Revision [ICD-9]), physician information, procedures, and up to 15 secondary diagnoses (ICD-9 codes). Individual subject data for chart review were de-identified and merged with the administrative data by an external biostatistician (MD) for analysis, to protect patient confidentiality.
AKI Definition by Reference Standard Chart Review and Administrative Data
Medical chart review served as the reference standard method to define AKI. Chart-defined AKI (referred hereafter as Chart-AKI) was defined in 3 ways based on the Kidney Disease: Improving Global Outcomes (KDIGO) definition (Supplementary Table 1)11,12: using SCr alone (Chart-AKISCr), using UO criteria alone (Chart-AKIUO), and using the combined SCr and/or UO criteria (Chart-AKISCrUO). For the latter definition, if a patient fulfilled criteria for either SCr and/or UO, they were classified as having Chart-AKISCrUO. The maximum AKI stage defined by SCr or UO was used to classify Chart-AKI severity (eg, if classified as stage 2 by SCr but stage 3 by UO, the patient was assigned stage 3 Chart-AKISCrUO). We also specifically evaluated dialysis-requiring AKI by chart review. The Chart-AKI definition was based on the KDIGO definition because it is the latest AKI definition developed, is the currently internationally accepted definition, and includes pediatric-specific criteria.11,13
To define AKI using administrative health care data, we developed an algorithm including ICD-9 diagnostic and procedure codes. During our study period (2003-2005), Quebec was using ICD-9 codes only; transition to use of International Classification of Diseases, Tenth Revision (ICD-10) codes was in 2006. We began by reviewing literature for similar studies in adults, followed by “pediatric-specific” examination of these administrative data code algorithms, retaining only relevant pediatric diagnostic and procedure codes, to refine the algorithm (face validity). The algorithm evaluated for this study (Supplementary Table 2) only included ICD-9 codes. Pertinent ICD-10 codes identified from our review are shown in Supplementary Table 2 for reference and future research. We primarily evaluated 2 algorithms: one was composed of diagnostic codes related to AKI only (hereafter referred to as Admin-AKI, shown in Supplementary Table 2), and another was composed of AKI diagnostic codes and codes related to the dialysis procedures to define dialysis-requiring AKI (or Admin-AKI+D, shown in Supplementary Table 2).
Clinical Outcome
The primary clinical outcome was mortality during the index hospitalization defined using the Quebec Vital Statistics Registry database.
Statistical Analysis
Characteristics between groups were compared using student t tests, Mann-Whitney U tests, or chi-square tests, depending on data distribution. Sample size requirements were calculated for the original retrospective cohort study. For the current study, the level of precision achievable when estimating sensitivity and specificity was calculated,14 using data from a recent similar study performed in adults (17% sensitivity and 98% specificity of Admin-AKI to detect Chart-AKI)15 and known pertinent data in this cohort for the noncardiac surgery PICU patients (approximately 24% prevalence of Chart-AKISCrUO in 1696 patients with available data). This power calculation (assuming α = 0.05) revealed that sensitivity and specificity could be calculated with a ±2% and ±1% level of precision, respectively. Analyses were performed by a biostatistician using SAS statistical software, release 9.2 (SAS Institute Inc, North Carolina). Reporting was performed as per guidelines on reporting of diagnostic accuracy studies and observational cohort studies.16
We evaluated the diagnostic characteristics (sensitivity, specificity, negative, and positive predictive values) for Admin-AKI definitions to detect Chart-AKI by each of the Chart-AKI definitions (SCr, UO, or both) and by Chart-AKI severity (any Chart-AKI, Chart-AKI stage 2 or worse, stage 3 Chart-AKI). These analyses were carried out in 2 separate cohorts: (1) patients admitted to the PICU post-cardiac surgery during the hospitalization and (2) noncardiac surgery patients. The decision to analyze these 2 populations was made a priori, as their AKI disease patterns and causes differ significantly, and because health care providers may also differ significantly (eg, cardiologists and cardiac surgeons specifically caring for post-cardiac surgery patients and thus responsible for discharge summaries). Moreover, differing findings between these populations would imply different needs for knowledge translation strategies with regard to AKI diagnosis and education.17 We performed 2 additional analyses in the noncardiac surgery population only (too few events in the cardiac surgery population). First, we evaluated the diagnostic characteristics of Admin-AKI+D in children with AKI requiring dialysis by chart review. Second, we evaluated the association between AKI and mortality using univariable logistic regression for both Chart-AKI and Admin-AKI definitions (expressed as unadjusted odds ratio [OR] with 95% confidence interval [CI]). We have previously published the multivariable association of Chart-AKI with mortality in this cohort.9,10 We therefore evaluated the association of Admin-AKI with hospital mortality, adjusting for age and sex.
Results
Study Population and Characteristics
Of 2,499 eligible patients, a further 24 patients were excluded due to missing health number or repeat admissions during the study period (N = 2475, Figure 1). Of these patients, 424 patients had neither an SCr nor a UO measured during PICU admission and were excluded, leaving 2051 patients available for analysis (355 cardiac surgery and 1696 noncardiac surgery patients with either SCr or UO data available to evaluate Chart-AKISCrUO; see Figure 1 for distribution of patients available for different Chart-AKI definition analyses).
Figure 1.
Study flow diagram.
Note. Figure 1 depicts subject selection flow, leading to the final analysis population. In the lower portion of the diagram, numbers of subjects for whom measures were available for the different methods of defining AKI are provided. PICU = pediatric intensive care unit; SCr = serum creatinine; UO = urine output.
Tables 1 and 2 display characteristics of patients with vs without AKISCrUO in the cardiac surgery (Table 1) and noncardiac surgery (Table 2) cohorts. As shown, in both cohorts, variables relating to illness severity (calculated death risk, use of mechanical ventilation, vasopressors and nephrotoxic medication, longer mechanical ventilation, and prolonged length of stay) were higher in the AKISCrUO groups (Tables 1 and 2). In the cardiac surgery group only, patients with AKISCrUO were also significantly younger (Table 1).
Table 1.
Baseline and PICU Patient Characteristics of the Cardiac Surgery Study Cohort.
| Characteristic | AKISCrUO
n = 185 |
No AKISCrUO
n = 170 |
|---|---|---|
| Age at PICU admission, mean (SD) years | 3.0 (4.8)a | 3.3 (4.1) |
| Male, n (%) | 108 (58.4) | 94 (55.3) |
| CHUSJ study site (vs MCH), n (%) | 107 (57.8) | 83 (48.8) |
| PRISM calculated death rate, mean (SD) | 19.2 (18.0)a | 11.3 (11.6) |
| Invasive mechanical ventilation, n (%) | 183 (98.9)a | 151 (88.8) |
| Days of invasive mechanical ventilation, mean (SD) | 5.6 (5.3)a | 2.8 (3.3) |
| Vasopressors, n (%) | 154 (83.2)a | 104 (61.2) |
| Nephrotoxic antibiotics,b n (%) | 60 (32.4)a | 18 (10.6) |
| Steroids, n (%) | 34 (18.4)a | 16 (9.4) |
| Nephrology consultation, n (%) | 18 (9.7)a | 5 (2.9) |
| Renal replacement therapy, n (%) | 1 (0.5) | 0 (0.0) |
| Length of PICU stay, mean (SD) days | 10.5 (20.4)a | 4.7 (7.5) |
| Length of hospital stay, mean (SD) days | 26.7 (39.1)a | 17.4 (35.1) |
Note. PICU = pediatric intensive care unit; AKISCrUO = acute kidney injury defined using serum creatinine and urine output criteria; CHUSJ = Centre Hospitalier Universitaire de Sainte-Justine; MCH = Montreal Children’s Hospital; PRISM calculated death rate = the logarithmic equation calculation of death rate using the Pediatric Risk of Mortality Score.18
Denotes that there was a statistically significant difference between AKI and non-AKI group for the variable.
Nephrotoxic antibiotics for which data were collected included aminoglycosides, vancomycin, nephrotoxic anti-virals including acyclovir, and amphotericin B.
Table 2.
Baseline and PICU Patient Characteristics of the Noncardiac Surgery Study Cohort.
| Characteristic | AKISCrUO
n = 406 |
No AKISCrUO
n = 1290 |
|---|---|---|
| Age at PICU admission, mean (SD) years | 7.3 (5.9)a | 6.5 (5.9) |
| Male, n (%) | 222 (54.7) | 712 (55.2) |
| CHUSJ study site (vs MCH), n (%) | 295 (72.7)a | 817 (63.3) |
| PRISM calculated death rate, mean (SD) | 12.2 (20.7)a | 4.7 (8.8) |
| Primary PICU admission reason, n (%) | ||
| Cardiac (nonsurgical) | 31 (7.6) | 72 (5.6) |
| Trauma | 45 (11.1) | 155 (12.0) |
| Renal | 15 (3.7)a | 4 (0.3) |
| Infection (excludes bronchiolitis) | 98 (24.1)a | 252 (19.5) |
| Neurological/neurosurgical | 46 (11.3)a | 222 (17.2) |
| Gastroenterological | 33 (8.1)a | 46 (3.6) |
| Oncological (nontransplant) | 16 (3.9) | 44 (3.4) |
| Respiratory | 44 (10.8) | 126 (9.8) |
| Diabetes | 18 (4.4)a | 17 (1.3) |
| Other | 60 (14.8)a | 352 (27.3) |
| Post-operative (noncardiac), n (%) | 111 (27.3)a | 485 (37.6) |
| Invasive mechanical ventilation, n (%) | 232 (57.1)a | 526 (40.8) |
| Days of invasive mechanical ventilation, mean (SD) | 4.1 (6.1)a | 1.8 (3.5) |
| Vasopressors, n (%) | 119 (29.3)a | 90 (7.0) |
| Nephrotoxic antibiotics,b n (%) | 161 (39.7)a | 292 (22.6) |
| Steroids, n (%) | 152 (37.4)a | 346 (26.8) |
| Nephrology consultation, n (%) | 94 (23.2)a | 51 (4.0) |
| Renal replacement therapy, n (%) | 17 (4.2)a | 0 (0.0) |
| Length of PICU stay, mean (SD) days | 9.3 (35.3)a | 3.2 (7.1) |
| Length of hospital stay, mean (SD) days | 31.8 (58.6)a | 19.4 (57.9) |
Note. PICU = pediatric intensive care unit; AKISCrUO = acute kidney injury defined using serum creatinine and urine output criteria; CHUSJ = Centre Hospitalier Universitaire de Sainte-Justine; MCH = Montreal Children’s Hospital; PRISM calculated death rate = the logarithmic equation calculation of death rate using the Pediatric Risk of Mortality Score.18
Denotes that there was a statistically significant difference between AKI and non-AKI group for the variable.
Nephrotoxic antibiotics for which data were collected included aminoglycosides, vancomycin, nephrotoxic anti-virals including acyclovir, and amphotericin B.
Diagnostic Characteristics of Admin-AKI to Detect Chart-AKI in Cardiac Surgery Patients
In the cardiac surgery population, 170 of 355 (47.9%) patients developed Chart-AKISCr, 51 of 204 (25.0%) had Chart-AKIUO, and 185 of 355 (52.1%) developed Chart-AKISCrUO in the PICU. Only one cardiac surgery patient was treated with dialysis. Of the ICD-9 Admin-AKI diagnostic codes shown in Supplementary Table 2, all cardiac surgery patients with Admin-AKI attained a diagnosis by either one of the AKI codes, AKI unspecified, or a combination of both codes. None of the Admin-AKI patients had codes for abnormal renal function, Pregnancy-AKI, or glomerulonephritis (GN).
Table 3 shows that in cardiac surgery patients, Admin-AKI detected the presence of Chart-AKI with low sensitivity, but high specificity. Overall, the sensitivity of Admin-AKI to detect AKIUO was higher than for detecting AKISCr, with minimal impact on specificity (shown throughout Table 3). The diagnostic characteristics of Admin-AKI to detect AKISCrUO were extremely similar to those for detecting AKISCr (right side of Table 3).
Table 3.
Diagnostic Characteristics of Administrative Data-Defined AKI (Admin-AKI) in Post-Cardiac Surgery Patients.
| Detecting any Chart-AKI Total n = 355 for AKISCr and for AKISCrUO; n = 204 for AKIUO | |||
|---|---|---|---|
| Diagnostic characteristica | SCr criteria 170 AKI events |
UO criteria 51 AKI events |
SCr or UO criteria 185 AKI events |
| Sensitivity | 5.9% (4.6-7.6) | 13.7% (10.1-18.5) | 5.9% (4.7-7.6) |
| Specificity | 99.5% (98.6-99.8) | 98.5% (97.5-99.2) | 100% (99.2-100) |
| PPV | 90.9% (77.3-97.4) | 70.0% (54.2-82.4) | 100% (88.2-100) |
| NPV | 53.5% (51.5-55.4) | 82.0% (80.0-83.8) | 49.4% (47.5-51.4) |
| Detecting Chart-AKI stage 2 or worse | |||
| SCr criteria 64 AKI events |
UO criteria 11 AKI events |
SCr or UO criteria 64 AKI events |
|
| Sensitivity | 14.1% (10.8-18.2) | 18.2% (8.7-32.6) | 14.1% (10.8-18.2) |
| Specificity | 99.3% (98.7-99.7) | 96.7% (95.6-97.6) | 99.3% (98.7-99.7) |
| PPV | 81.8% (67.4-91.2) | 20.0% (9.6-35.5) | 81.8% (67.4-91.2) |
| NPV | 84.0% (82.5-85.4) | 96.3% (95.2-97.2) | 84.0% (82.5-85.4) |
| Detecting Chart-AKI stage 3 | |||
| SCr criteria 32 AKI events |
UO criteria 7 AKI events |
SCr or UO criteria 35 AKI events |
|
| Sensitivity | 15.6% (10.6-22.3) | 14.3% (4.0-34.1) | 14.3% (9.7-20.5) |
| Specificity | 98.1% (97.4-98.7) | 96.4% (95.2-97.2) | 98.1% (97.3-98.7) |
| PPV | 45.5% (31.7-59.8) | 100% (2.8-24.7) | 45.5% (31.7-59.8) |
| NPV | 92.2% (90.9-93.2) | 97.6% (96.5-98.3) | 91.3% (90.0-92.4) |
Note. Chart-AKI = AKI defined by medical chart review (reference standard); AKISCr = acute kidney injury defined using serum creatinine criteria; AKIUO = acute kidney injury defined using urine output criteria; AKISCrUO = acute kidney injury defined using serum creatinine and urine output criteria; SCr = serum creatinine; UO = urine output; PPV = positive predictive value; NPV = negative predictive value.
Estimates of sensitivity, specificity, PPV, and NPV are presented with 95% confidence intervals.
With increasing severity of Chart-AKI definition (eg, stage 2 or worse AKI), the sensitivity of Admin-AKI to detect Chart-AKI generally increased (to a range of 14%-18%), with slight reduction in specificity (Table 3).
Diagnostic Characteristics of Admin-AKI to Detect Chart-AKI in Noncardiac Surgery Patients
In the noncardiac surgery population, 345 of 1648 (20.9%) patients developed Chart-AKISCr in the PICU, 109 of 760 (14.3%) had Chart-AKIUO, and 406 of 1696 (24.1%) had Chart-AKISCrUO. There were 17 noncardiac surgery patients with dialysis-requiring AKI. Of ICD-9 diagnostic codes shown in Supplementary Table 2, all Admin-AKI patients attained Admin-AKI status with codes for AKI, AKI unspecified, or a combination of both codes. One patient had a diagnosis of GN, but also had a diagnostic code for AKI. No patients fulfilled Admin-AKI criteria with the abnormal renal function or Pregnancy-AKI codes.
Table 4 shows that in noncardiac surgery patients, the overall pattern of low sensitivity and high specificity of Admin-AKI to detect Chart-AKI was similar to what was seen in cardiac surgery patients. However, overall, sensitivity for detecting Chart-AKI was higher (as high as 70% for detecting stage 3 AKIUO, Table 4). Similarly, as severity of Chart-AKI increased, the sensitivity and negative predictive value for Admin-AKI to detect Chart-AKI increased while the specificity and positive predictive value decreased (Table 4).
Table 4.
Diagnostic Characteristics of Administrative Data-Defined AKI (Admin-AKI) in Noncardiac Surgery Patients.
| Detecting any Chart-AKI Total n = 1648 for AKISCr; n = 760 for AKIUO; n = 1696 AKISCrUO | |||
|---|---|---|---|
| Diagnostic characteristica | SCr criteria 345 AKI events |
UO criteria 109 AKI events |
SCr or UO criteria 406 AKI events |
| Sensitivity | 15.7% (14.3-17.2) | 20.2% (17.3-23.4) | 13.8% (12.6-15.1) |
| Specificity | 98.9% (98.7-99.1) | 95.2% (94.5-95.8) | 99.1% (98.8-99.3) |
| PPV | 79.4% (75.1-83.1) | 41.5% (36.2-47.1) | 82.4% (78.2-85.8) |
| NPV | 81.6% (80.9-82.3) | 87.7% (86.8-88.6) | 78.5% (77.8-79.2) |
| Detecting Chart-AKI stage 2 or worse | |||
| SCr criteria 163 AKI events |
UO criteria 37 AKI events |
SCr or UO criteria 178 AKI events |
|
| Sensitivity | 25.8% (23.3-28.5) | 46.0% (39.2-52.8) | 23.6% (21.3-26.1) |
| Specificity | 98.2% (98.0-98.5) | 95.0% (94.4-95.6) | 98.3% (98.0-98.5) |
| PPV | 61.8% (57.0-66.3) | 32.1% (27.1-37.6) | 61.8% (57.0-66.3) |
| NPV | 92.3% (91.8-92.8) | 97.2% (96.6-97.6) | 91.7% (91.1-92.1) |
| Detecting Chart-AKI stage 3 | |||
| SCr criteria 81 AKI events |
UO criteria 23 AKI events |
SCr or UO criteria 83 AKI events |
|
| Sensitivity | 43.2% (39.0-47.6) | 69.6% (60.4-77.4) | 42.2% (38.0-46.5) |
| Specificity | 97.9% (97.6-98.1) | 95.0% (94.3-95.5) | 98.0% (97.7-98.2) |
| PPV | 51.5% (46.7-56.2) | 30.2% (25.3-35.6) | 51.5% (46.7-56.2) |
| NPV | 97.1% (96.8-97.4) | 99.0% (98.6-99.3) | 97.1% (96.7-97.3) |
Note. Chart-AKI = AKI defined by medical chart review (reference standard); AKISCr = acute kidney injury defined using serum creatinine criteria; AKIUO = acute kidney injury defined using urine output criteria; AKISCrUO = acute kidney injury defined using serum creatinine and urine output criteria; SCr = serum creatinine; UO = urine output; PPV = positive predictive value; NPV = negative predictive value.
Estimates of sensitivity, specificity, PPV, and NPV are presented with 95% confidence intervals.
Of the 17 patients who received dialysis for AKI (determined by chart review), 14 patients were identified as having Admin-AKI+D using administrative data. The sensitivity, specificity, positive, and negative predictive values for Admin-AKI+D to detect dialysis-requiring AKI were 82.4% (95% CI = 71.8%-89.7%), 99.6% (95% CI = 99.5%-99.7%), 70.0% (95% CI = 60.0%-78.4%), and 99.8% (95% CI = 99.7%-99.9%), respectively.
Association of Admin-AKI With Hospital Mortality
Only 5 patients in the cardiac surgery population died during admission; therefore, this analysis was focused on the noncardiac surgery population only. In the noncardiac surgery population, 74 of 1696 (4.4%) patients died during index admission. Table 5 shows that regardless of how Chart-AKI was defined (using SCr, UO, or both criteria), there was a strong association of Chart-AKI with hospital mortality (Table 5). There was also a strong association between Admin-AKI and Admin-AKI+D with hospital mortality (Table 5, unadjusted ORs approximately 5 and 10, respectively, P < .0001). The association between Admin-AKI and hospital mortality remained statistically significant when adjusting for age and sex (adjusted OR = 6.0, 95% CI = 3.0-11.9).
Table 5.
Hospital Mortality in Noncardiac Surgery Patients With Versus Without AKI Defined by Chart Review (Chart-AKI) and by Administrative Health Data (Admin-AKI), With AKI Versus Non-AKI Odds Ratio.
| Definition method | Mortality n (%) with AKI | Mortality n (%) without AKIa | Odds ratio (95% CI)b |
|---|---|---|---|
| Chart-AKISCr | |||
| Any AKI | 46/345 (13.3%) | 27/1303 (2.1%) | 7.3 (4.5-11.9) |
| Stage 2 or worse AKI | 34/163 (20.9%) | 39/1485 (2.6%) | 9.8 (6.0-16.0) |
| Stage 3 AKI | 23/81 (28.4%) | 50/1567 (3.2%) | 12.0 (6.9-21.0) |
| Chart-AKIUO | |||
| Any AKI | 19/109 (17.4%) | 18/651 (2.8%) | 7.4 (3.8-14.7) |
| Stage 2 or worse AKI | 9/37 (24.3%) | 28/723 (3.8%) | 8.0 (3.4-18.5) |
| Stage 3 AKI | 8/23 (34.8%) | 29/737 (3.9%) | 13.0 (5.1-33.2) |
| Chart-AKISCrUO | |||
| Any AKI | 51/406 (12.6%) | 23/1290 (1.8%) | 7.9 (4.8-13.1) |
| Stage 2 or worse AKI | 36/178 (20.2%) | 38/1518 (2.5%) | 9.9 (6.1-16.1) |
| Stage 3 AKI | 24/83 (28.9%) | 50/1613 (3.1%) | 12.7 (7.3-22.1) |
| Admin-AKI | 12/68 (17.7%) | 62/1628 (3.8%) | 5.4 (2.8-10.6) |
| Admin-AKI requiring dialysis | 6/20 (30.0%) | 68/1676 (4.1%) | 10.1 (3.8-27.2) |
Note. AKISCr = acute kidney injury defined using serum creatinine criteria; AKIUO = acute kidney injury defined using urine output criteria; AKISCrUO = acute kidney injury defined using serum creatinine and urine output criteria; Admin-AKI+D = acute kidney injury with renal replacement therapy treatment, defined by administrative health data (diagnostic codes plus dialysis procedure codes); CI = confidence interval.
Without AKI refers to no AKI as per the definition method used for analysis (eg, for stage 2 AKI or worse, without AKI refers to patients with no AKI or stage 1 AKI, or no stage 2 AKI).
Odds ratio calculated using logistic regression. All odds ratios are statistically significant.
Discussion
Our results demonstrate that administrative health data detect the presence of chart-defined AKI with low sensitivity in both cardiac and noncardiac surgery patients. With increasing severity of Chart-AKI staging, the sensitivity and negative predictive value of Admin-AKI to detect Chart-AKI increase, with a minor reduction in specificity and larger reduction in positive predictive value. Despite its low sensitivity, Admin-AKI was highly specific and associated with hospital mortality, a well-studied outcome associated with pediatric AKI,1-4 which may support the use of administrative health care databases to study more severe AKI in children.
Diagnostic accuracy studies of AKI diagnostic and procedure codes in adult populations have shown very similar results, namely, a poor sensitivity (11%-17%), but high specificity (>98%) to identify AKI, with increasing sensitivity to detect more severe AKI.15 However, we noted some differences from previously published adult studies. Contrary to a recent, large adult study, in both cardiac surgery and noncardiac surgery patients, we found that the sensitivity of Admin-AKI to detect Chart-AKI was higher when using only the UO criteria to define Chart-AKI (vs using SCr criteria).15 A recent multinational study of pediatric AKI in the PICU showed that oliguria was more strongly associated with hospital mortality than SCr-defined AKI.19,20 Thus, in children, when oliguria is present, often leading to fluid overload and need for intervention (eg, diuretics), it is possible that the presence of AKI is more highly noted by physicians and therefore recorded in discharge summaries.
We found that with each increase in Chart-AKI severity stage, the sensitivity of Admin-AKI increased 1.5 to 2 fold; this rise in sensitivity was associated with a minimal decrease in specificity. Future studies aimed at defining highly accurate prevalence data or examining AKI outcomes for which there is little knowledge or data published should consider the fact that using Admin-AKI data will likely only reflect more severe forms of AKI and that non-Admin-AKI control groups will contain substantial proportions of patients with mild AKI. Future research should aim to identify other variables (eg, non-AKI-related diagnostic/procedure codes) associated with AKI (and non-AKI) to further help improve the sensitivity of Admin-AKI detection.
We found substantial differences between the cardiac and noncardiac surgery populations. In pediatrics, cardiac surgery patients may often be in different PICUs or even have different ICU specialty physicians caring for them. For example, surgeons tend to be highly implicated in the care of post-operative cardiac surgery patients and not noncardiac surgery patients. Cardiac surgery patients are often found to have SCr rise or oliguria very shortly after surgery (sometimes immediately post-operatively); this is well known to occur and most often transient.21,22 This commonplace immediate post-operative SCr rise or oliguria in these patients may explain why the sensitivity of Admin-AKI to detect Chart-AKI was substantially lower in the cardiac surgery patients; when AKI does occur immediately post-operatively in this patient population, it may simply seem less noteworthy to health care providers. Despite these findings, we cannot exclude the possibility that our reduced sample size of cardiac surgery patients contributed to lack of power and reduced estimates of sensitivity due to chance alone. Moreover, it was not possible in this study to determine the extent to which health care providers considered immediate post-operative SCr rise or oliguria more or less indicative of true AKI, compared with later appearance of AKI in the PICU.
To our knowledge, this is the first published detailed Admin-AKI algorithm, aimed at use in children admitted to the PICU. A recent study published on hospitalized children with nephrotoxin-associated AKI also evaluated an AKI diagnostic algorithm utilizing ICD-9 discharge diagnoses.23 This study similarly demonstrated that sensitivity for detection of AKI by SCr criteria was low (approximately 20%) and specificity was high (approximately 95%) but did not evaluate different AKI severity levels and was focused on nephrotoxin-mediated AKI. Although we only evaluated validity of ICD-9 codes due to our pre-ICD-10 study period, our study provides a more extended algorithm for consideration by future researchers, including ICD-9 and ICD-10 codes.17 Our algorithm also provides corresponding procedural codes for dialysis, thereby enhancing the sensitivity for detecting dialysis-requiring (most severe) AKI. Our study was limited to patients admitted to the PICU, the highest risk patient population for poor AKI outcomes.
This study had limitations. First, although this is one of the largest published retrospective PICU AKI cohorts, the sample size for the cardiac surgery cohort was low. This study included data from 2 hospitals within the same geographic region, limiting generalizability. Center-specific practice patterns (eg, fluid management; acute dialysis initiation practice; urinary catheter insertion and removal practice) and center-specific beliefs on what constitutes a diagnosis of AKI may impact ascertainment of both Chart-AKI and Admin-AKI. It is thus important that similar research in other geographic regions be performed to determine the extent to which our results are generalizable. The cohort included patients admitted between 2003 and 2005, because the original goal of the study was to evaluate long-term outcomes of AKI in children admitted to the PICU using administrative health care data (therefore, we chose a somewhat older cohort, to have a minimum of 5 years follow-up after PICU admission). It is conceivable that awareness of AKI, and thus inclusion of AKI as a discharge or admission diagnosis on discharge summaries, has improved in the last decade and that sensitivity of Admin-AKI may be higher today. A previous adult study from the United States showed that billing codes for AKI have increased in recent years.15 However, there have not been any specific knowledge translation or awareness enhancement interventions performed to target increased identification of AKI at our centers. Our analysis included only patients who had SCr or UO data collected during PICU admission. This likely led to a study population biased toward patients with higher illness severity. However, it is those patients who are of higher interest in terms of evaluating AKI outcomes and identifying the presence of AKI using administrative health data.
In conclusion, we have shown that using administrative health data including hospitalization admission and discharge diagnostic codes and dialysis procedural codes to identify the presence of AKI in the PICU is highly specific, but not sensitive. It is therefore important to exercise caution when using administrative data to assign AKI diagnosis and/or evaluate its association with patient outcomes and to consider the impact of this AKI ascertainment method’s strength and limitations on specific study objectives. Studies targeting more severe forms of AKI can improve sensitivity of Chart-AKI detection as high as approximately 70% and studies including noncardiac surgery patients will also have higher likelihood of better sensitivity for Chart-AKI detection. Our study highlights a probable lack of awareness of AKI as an important health care problem in PICU patients, on the part of PICU health care providers, as evidenced by the common occurrence of lack of AKI diagnostic coding in patients with mild and moderate AKI. Future research should include intervention studies aimed at increasing awareness of AKI and encouraging the identification of AKI in critically ill patients. We also found that Admin-AKI is associated with hospital mortality, albeit with somewhat lower magnitude to what is known for Chart-AKI. Although this finding supports that Admin-AKI may have prognostic value, more research is needed with larger sample sizes to determine whether this association holds in large multivariable models (considering comorbidities, acute illness severity, and treatment variables). Future research should also determine the extent to which Admin-AKI in children is associated with post-hospital discharge outcomes including health care utilization, chronic kidney disease, and mortality, to evaluate validity and prognostic value of Admin-AKI for studying AKI outcomes.
Supplemental Material
Supplemental material, Supplementary_material_DArienzo_et_al_Revision_2_Clean for A Validation Study of Administrative Health Care Data to Detect Acute Kidney Injury in the Pediatric Intensive Care Unit by David D’Arienzo, Erin Hessey, Rami Ali, Sylvie Perreault, Susan Samuel, Louise Roy, Jacques Lacroix, Philippe Jouvet, Genevieve Morissette, Marc Dorais, Jean-Philippe Lafrance, Veronique Phan, Michael Pizzi, Rahul Chanchlani and Michael Zappitelli in Canadian Journal of Kidney Health and Disease
Footnotes
Ethics Approval and Consent to Participate: Ethics approval was provided by the Research Institute of the McGill University Health Centre Pediatric Research Ethics Board (Study code 10-399-PED) and Commission d’accès à l’information du Québec (CAI, provincial ethics body #10-15-04) . Requirement for patient consent was waived.
Consent for Publication: All authors reviewed the final manuscript and provided consent for publication.
Availability of Data and Materials: Data and materials may be made available upon written request to the corresponding author.
Authors’ Note: Genevieve Morissette is now affiliated with Department of Pediatrics, Division of Intensive Care, Centre mère-enfant Soleil, Centre hospitalier de l’Université Laval, Quebec, QC, Canada.
Author Contributions: MZ contributed to research idea and study design; EH, RA, and MP contributed to data acquisition; MZ, MP, EH, DD, SP, SS, LR, JL, PJ, GM, MD, JL, VP, and RC analyzed and interpreted the data; MP, EH, and MZ contributed to statistical analysis; MZ contributed to supervision or mentorship. Each author contributed important intellectual content during article drafting or revision and accepted accountability for the overall work by ensuring that questions pertaining to the accuracy or integrity of any portion of the work are appropriately investigated and resolved.
Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by an operating research grant by the Fonds de Recherche du Québec–Santé.
Supplemental Material: Supplemental material for this article is available online.
References
- 1. Hoste EA, Clermont G, Kersten A, et al. RIFLE criteria for acute kidney injury are associated with hospital mortality in critically ill patients: a cohort analysis. Crit Care. 2006;10:R73. doi: 10.1186/cc4915. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Schneider J, Khemani R, Grushkin C, Bart R. Serum creatinine as stratified in the RIFLE score for acute kidney injury is associated with mortality and length of stay for children in the pediatric intensive care unit. Crit Care Med. 2010;38:933-939. doi: 10.1097/CCM.0b013e3181cd12e1. [DOI] [PubMed] [Google Scholar]
- 3. Bresolin N, Silva C, Halllal A, et al. Prognosis for children with acute kidney injury in the intensive care unit. Pediatr Nephrol. 2009;24:537-544. [DOI] [PubMed] [Google Scholar]
- 4. Alkandari O, Eddington KA, Hyder A, et al. Acute kidney injury is an independent risk factor for pediatric intensive care unit mortality, longer length of stay and prolonged mechanical ventilation in critically ill children: a two-center retrospective cohort study. Crit Care. 2011;15:R146. doi: 10.1186/cc10269. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Ambroggio LV, Shah SS. Administrative data: expanding the infrastructure for pediatric research. J Pediatr. 2013;162:681-684. [DOI] [PubMed] [Google Scholar]
- 6. Vlasschaert ME, Bejaimal SA, Hackam DG, et al. Validity of administrative database coding for kidney disease: a systematic review. Am J Kidney Dis. 2011;57:29-43. [DOI] [PubMed] [Google Scholar]
- 7. Sutherland SM, Ji J, Sheikhi FH, et al. AKI in hospitalized children: epidemiology and clinical associations in a national cohort. Clin J Am Soc Nephrol. 2013;8:1661-1669. doi: 10.2215/cjn.00270113. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Hessey E, Morissette G, Lacroix J, et al. Healthcare utilization after acute kidney injury in the pediatric intensive care unit. Clin J Am Soc Nephrol. 2018;13:685-692. doi: 10.2215/cjn.09350817. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Hessey E, Morissette G, Lacroix J, et al. Long-term mortality after acute kidney injury in the pediatric ICU. Hosp Pediatr. 2018;8:260-268. doi: 10.1542/hpeds.2017-0215. [DOI] [PubMed] [Google Scholar]
- 10. Hessey E, Ali R, Dorais M, et al. Evaluation of height-dependent and height-independent methods of estimating baseline serum creatinine in critically ill children. Pediatr Nephrol. 2017;32:1953-1962. doi: 10.1007/s00467-017. [DOI] [PubMed] [Google Scholar]
- 11. Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group. KDIGO clinical practice guideline for acute kidney injury. Kidney Int Suppl. 2012;2:1-138. doi: 10.1038/kisup.2012.4. [DOI] [Google Scholar]
- 12. Akcan-Arikan A, Zappitelli M, Loftis LL, Washburn KK, Jefferson LS, Goldstein SL. Modified RIFLE criteria in critically ill children with acute kidney injury. Kidney Int. 2007;71:1028-1035. doi: 10.1038/sj.ki.5002231. [DOI] [PubMed] [Google Scholar]
- 13. Sutherland SM, Byrnes JJ, Kothari M, et al. AKI in hospitalized children: comparing the pRIFLE, AKIN, and KDIGO definitions. Clin J Am Soc Nephrol. 2015;10:554-561. doi: 10.2215/CJN.01900214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Jones SR, Carley S, Harrison M. An introduction to power and sample size estimation. Emerg Med J. 2003;20:453-458. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Grams ME, Waikar SS, MacMahon B, Whelton S, Ballew SH, Coresh J. Performance and limitations of administrative data in the identification of AKI. Clin J Am Soc Nephrol. 2014;9:682-689. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg. 2014;12:1495-1499. [DOI] [PubMed] [Google Scholar]
- 17. Siew ED, Basu RK, Wunsch H, et al. Optimizing administrative datasets to examine acute kidney injury in the era of big data: workgroup statement from the 15(th) ADQI Consensus Conference. Can J Kidney Health Dis. 2016;3:12. doi: 10.1186/s40697-016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Pollack MM, Ruttimann UE, Getson PR. Pediatric risk of mortality (PRISM) score. Crit Care Med. 1988;16:1110-1116. [DOI] [PubMed] [Google Scholar]
- 19. Kaddourah A, Basu RK, Bagshaw SM, et al. Epidemiology of acute kidney injury in critically ill children and young adults. N Engl J Med. 2017;376:11-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Basu RK, Kaddourah A, Terrell T, et al. Assessment of Worldwide Acute Kidney Injury, Renal Angina and Epidemiology in critically ill children (AWARE): study protocol for a prospective observational study. BMC Nephrol. 2015;16:24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. AlAbbas A, Campbell A, Skippen P, Human D, Matsell D, Mammen C. Epidemiology of cardiac surgery-associated acute kidney injury in neonates: a retrospective study. Pediatr Nephrol. 2013;28:1127-1134. [DOI] [PubMed] [Google Scholar]
- 22. Zappitelli M, Krawczeski CD, Devarajan P, et al. Early postoperative serum cystatin C predicts severe acute kidney injury following pediatric cardiac surgery. Kidney Int. 2011;80:655-662. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Schaffzin JK, Dodd CN, Nguyen H, Schondelmeyer A, Campanella S, Goldstein SL. Administrative data misclassifies and fails to identify nephrotoxin-associated acute kidney injury in hospitalized children. Hosp Pediatr. 2014;4:159-166. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental material, Supplementary_material_DArienzo_et_al_Revision_2_Clean for A Validation Study of Administrative Health Care Data to Detect Acute Kidney Injury in the Pediatric Intensive Care Unit by David D’Arienzo, Erin Hessey, Rami Ali, Sylvie Perreault, Susan Samuel, Louise Roy, Jacques Lacroix, Philippe Jouvet, Genevieve Morissette, Marc Dorais, Jean-Philippe Lafrance, Veronique Phan, Michael Pizzi, Rahul Chanchlani and Michael Zappitelli in Canadian Journal of Kidney Health and Disease