Abstract
Background:
Although surgical complications are often included as an outcome of surgical research conducted using administrative data, little validation work has been performed. We sought to evaluate the diagnostic performance of an algorithm designed to capture major surgical complications using health administrative data.
Methods:
This retrospective study included patients who underwent high-risk elective general surgery at a single institution in Ontario, Canada, from Sept. 1, 2016, to Sept. 1, 2017. Patients were identified for inclusion using the local operative database. Medical records were reviewed by trained clinicians to abstract postoperative complications. Data were linked to administrative data holdings, and a series of code-based algorithms were applied to capture a composite indicator of major surgical complications. We used sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and accuracy to evaluate the performance of our administrative data algorithm, as compared with data abstracted from the institutional charting system.
Results:
The study included a total of 270 patients. According to the data from the chart audit, 55% of patients experienced at least 1 major surgical complication. Overall sensitivity, specificity, PPV, NPV and accuracy for the composite outcome was 72%, 80%, 82%, 70% and 76%, respectively. Diagnostic performance was poor for several of the individual complications.
Conclusion:
Our results showed that administrative data holdings can be used to capture a composite indicator of major surgical complications with adequate sensitivity and specificity. Additional work is required to identify suitable algorithms for several specific complications.
Abstract
Contexte:
Même si les complications postopératoires sont souvent incluses parmi les paramètres mesurés lors de recherches sur la chirurgie qui reposent sur les données administratives, leur validation n’a pas fait l’objet d’études. Nous avons voulu évaluer le rendement diagnostique d’un algorithme conçu pour faire le point sur les complications postopératoires majeures à partir des données administratives sur la santé.
Méthodes:
Cette étude rétrospective a inclus des malades soumis à des chirurgies à haut risque non urgentes dans 1 seul établissement en Ontario, au Canada, entre le 1er septembre 2016 et le 1er septembre 2017. Les cas ont été recrutés à partir de la base de données périopératoires de l’établissement. Les dossiers médicaux ont été passés en revue par des médecins dument formés pour extraire les données concernant les complications postopératoires. Ces données ont été reliées à des registres de données administratives, et une série d’algorithmes a été appliqué pour créer un paramètre mixte des complications chirurgicales majeures. Nous avons utilisé la sensibilité, la spécificité, la valeur prédictive positive (VPP), la valeur prédictive négative (VPN) et la justesse afin d’évaluer le rendement de notre algorithme bâti sur les données administratives, plutôt que sur les données extraites du système de tenue de dossier de l’établissement.
Résultats:
En tout, l’étude a recruté 270 patients. Selon l’analyse des dossiers, 55 % ont eu au moins 1 complication postopératoire majeure. La sensibilité, la spécificité, la VPP, la VPN et la justesse pour le paramètre mixte ont été respectivement de 72 %, 80 %, 82 %, 70 % et 76 %. Le rendement diagnostique a été faible pour plusieurs complications individuelles.
Conclusion:
Nos résultats ont montré que les registres de données administratives peuvent servir à créer un paramètre mixte des complications postopératoires majeures avec une sensibilité et une spécificité adéquates. Il faudra approfondir la recherche pour concevoir des algorithmes adaptés à diverses complications spécifiques.
Administrative databases are increasingly being used to address priorities in surgical research. Although surgical complications are often investigated as study outcomes,1–5 little work has been done to validate the algorithms used to capture these outcomes. Administrative data holdings were not designed for clinical research, and validation is required to ensure that the diagnostic and intervention codes that are used reflect the clinical state being reported. Most importantly, research findings based on unvalidated administrative data algorithms may be biased, potentially resulting in invalid conclusions.6 Validation studies can be used to enhance the ability of both investigators and readers to interpret the results of administrative data research by quantifying the degree of misclassification and its potential effect on study results.7
We conducted a single-centre chart abstraction study to evaluate the ability of administrative data to capture a composite indicator of major surgical complications, defined by a series of algorithms using hospital admission, same day surgery and physician billing records.
Methods
This retrospective study included a convenience sample of 295 patients who underwent major elective general surgical procedures at a single, large, acute care institution in Ontario, Canada, from Sept. 1, 2016, to Sept. 1, 2017. Sequential patients who underwent an eligible procedure during the study accrual period were identified using the institutional operative database and included in the study. Eligible procedures included those that involved laparotomy or laparoscopy with a planned postoperative in-patient stay and were categorized as intestinal procedures (small or large), pancreatic with or without duodenal procedures, liver resection, repair of massive ventral hernia, procedures related to the spleen, gastric procedures and other procedures (complex biliary surgery, rectal resection). Ethics approval was obtained from the Western University Health Sciences Research Ethics Board.
A clinical abstractor performed a complete review of the electronic and paper medical records of included patients up to 30 days postoperatively and identified all postoperative complications that occurred during the hospital admission. Major postoperative complications of interest included bleeding, defined by a receipt of a blood transfusion or a drop in postoperative hemoglobin of more than 20% compared with baseline; sepsis, defined as life-threatening organ dysfunction caused by a dysregulated host response to infection9; shock, defined as persistent hypotension with mean arterial pressure (MAP) of less than 65 mm Hg or need for vasopressors to maintain MAP above 65 mm Hg8; acute renal failure, defined as acute alterations in serum creatinine or urine output9; new-onset hemodialysis; pneumonia, defined as acute infection of the lung parenchyma by a pathogen, according to clinical diagnosis with supporting radiographic or microbiology evidence10; ventilator use for more than 48 hours; new-onset atrial fibrillation or flutter, documented by electrocardiography (ECG) and not previously documented in the medical record; venous thromboembolism, defined by acute deep vein thrombosis confirmed on duplex ultrasonography or pulmonary embolism confirmed by computed tomography (CT); cardiac or respiratory arrest; coma, defined by persistent Glasgow Coma Scale (GCS) score of less than 8; stroke, confirmed by CT or magnetic resonance imaging; myocardial infarction, defined by symptoms of acute myocardial ischemia or new ischemic changes confirmed by ECG in addition to an elevation in serum troponin11; and an unplanned return to the operating room, defined as any return to the operating room not foreseeable at the completion of the elective operation). The number of patients who died in hospital was also captured.
Data collected through chart abstraction were transferred to the secure environment of ICES (formerly the Institute for Clinical Evaluative Sciences) and linked to administrative data holdings using each patient’s unique health insurance number. All residents of Ontario have access to publicly funded health care through a national health insurance program, and all insured services are captured in administrative databases. Hospital admission and same-day surgery records are maintained in the Canadian Institute for Health Information’s Discharge Abstract Database and Same Day Surgery database, whereas the Ontario Health Insurance Plan database captures fee-for-service billing records. Patient age and sex were obtained from the Registered Persons Database, which contains demographic information for all Ontario residents. These data sets were linked using unique encoded identifiers and analyzed at ICES Western.
The algorithm used to identify each postoperative complication in the administrative data holdings is presented in Table 1. The preliminary code list was based in part on the algorithms published by Govindarajan and colleagues1 and supplemented through a review of the relevant coding documentation by the authorship group. After the initial assessment, a second review was conducted to consider additional codes recorded within the administrative records for our cohort, resulting in the inclusion of 8 additional diagnostic codes (Table 1). Reported baseline variables include patient age and sex, procedure type and length of hospital stay.
Table 1.
Administrative data algorithms used to define postoperative complications
| Variable | Code type | Codes |
|---|---|---|
| Atrial fibrillation or flutter | ICD-10 | I4800, I4801, I481, I483, I484, I4890, I4891 |
| Bleeding | CCI | 1LZ19HHU1A, 1LZ19HHU1J, 1LZ19HHU2A, 1LZ19HHU2J, 1LZ19HHU4J, 1LZ19HHU5J, 1LZ19HHU9A, 1LZ19HHU9J |
| ICD-10 | R58, T810 | |
| CIHI flag | BTREDBC = ‘Y’ or ‘1’ | |
| Cardiac arrest or respiratory arrest | ICD-10 | I460, I461, I469, R092 |
| Fee code | G391, G395, G521, G522, G523 | |
| Coma | ICD-10 | G931†, G9381†, R4020, R4029, S0625† |
| New-onset hemodialysis* | Fee code | G082, G083, G085, G090, G093, G095, G323, R849 |
| Myocardial infarction | ICD-10 | I210, I211, I212, I213, I214, I2140, I2141, I2142, I2149, I219, I220, I221, I228, I229, I230, I231, I232, I233, I234, I235, I236, I2380, I2381, I2382, I2388, I2389 |
| Pneumonia | ICD-10 | J120, J121, J122, J123, J128, J129, J13, J14, J150, J151, J152, J153, J154, J155, J156, J157, J158, J159, J160, J168, J170, J171, J172, J173, J178, J180, J181, J182, J188, J189, J690, J691, J698 |
| Acute renal failure | ICD-10 | N170, N171, N172, N178, N179 |
| Stroke | ICD-10 | G450, G451, G452, G453, G454, G458, G459, H341, I600, I601, I602, I603, I604, I605, I606, I607, I608, I609, I610, I611, I612, I613, I614, I615, I616, I618, I619, I630, I631, I632, I633, I634, I635, I636, I638, I639, I64 |
| Sepsis | ICD-10 | A410, A411, A412, A413, A414, A4150, A4151, A4152, A4158, A4159, A4180, A4188, A419, B377†, T814† |
| Shock | ICD-10 | A419†, I460†, I959†, R570, R571, R572, R578, R579, T794† |
| Ventilation | Fee code | G405, G406, G557, G558 |
| Venous thromboembolism | CCI | 3GT20WC, 3GT20WE, 3GT70CA, 3GT70CC, 3GT70CE, 3GT70KC, 3GT70KD, 3GT70KE, 3IM10VC, 3IM10VX, 3IM10VY, 3IM12VA, 3JY10VA, 3JY10VC, 3JY10VN, 3JY10VX, 3JY12VA, 3JY20WC, 3JY20WE, 3KR10VA, 3KR10VC, 3KR10VN, 3KR12VA, 3KX10VA, 3KX10VC, 3KX10VN, 3KX10VX, 3KX12VA, 3KX30DA, 3KX30DB, 3KX30DC, 3KX30DD |
| ICD-10 | I26, I260, I269, I801, I802, I803, I822, I828, O87102, O87104, O87109, O87802, O87804, O87809, O87902, O87904, O87909, O88201, O88202, O88203, O88204, O88209 | |
| Unplanned return to the operating room | CIHI flag | INUNPL1 = ‘Y’ or ‘1’ |
CCI = Canadian Classification of Health Interventions; CIHI = Canadian Institute for Health Information; ICD-10 = International Classification of Diseases, 10th revision.
Patients were not eligible for the new-onset hemodialysis outcome if there was evidence of hemodialysis (defined by fee codes G860, G861, G862, G863, G864, G865, G866) within the previous year.
Codes that were added following review of administrative records for our cohort.
Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and overall accuracy were used to evaluate the performance of administrative data in capturing major surgical complications, as compared with data obtained through chart abstraction. We also report 95% confidence intervals (CIs) for each metric, calculated using the exact (Clopper–Pearson) method. In addition to the primary composite outcome, we also assessed each individual complication as well as in-hospital mortality. Values representing 5 or fewer patients are suppressed owing to ICES privacy requirements. The study analyst (A.M.) had full access to the project data sets. Reporting of this study follows the Reporting of studies conducted using observational routinely-collected data (RECORD) statement.12
Results
Among the initial 295 patients, we were unable to link 11 (3.7%) to administrative data holdings, and an additional 14 (4.7%) were excluded because we could not identify a matching hospital admission or procedure record. After these exclusions, the final cohort included 270 patients, of whom 107 (40%) were women. The median age of the cohort was 64 (interquartile range [IQR] 53–73) years and the median length of hospital stay was 7 (IQR 4–12) days. The majority of patients underwent procedures involving the intestines, pancreas, liver or repair of a massive ventral hernia (Table 2).
Table 2.
Demographic and clinical characteristics of the study cohort
| Variable | No. (%)* |
|---|---|
| Age, median (IQR), yr | 64 (53–73) |
| Sex | |
| Female | 107 (39.6) |
| Male | 163 (60.4) |
| Length of hospital stay, median (IQR) | 7 (4–12) |
| Type of procedure | |
| Intestines (small or large) | 76 (28.1) |
| Pancreas (with and without duodenum) | 52 (19.3) |
| Liver | 44 (16.3) |
| Repair of massive ventral hernia | 36 (13.3) |
| Spleen | 26 (9.6) |
| Stomach | 21 (7.8) |
| Other (gallbladder, bile ducts and rectum) | 15 (5.6) |
IQR = interquartile range.
Unless indicated otherwise.
According to the data from the chart audit, 148 (55%) patients experienced at least 1 major surgical complication. The most common complication was bleeding (122, 45%), followed by sepsis (49, 18%), acute renal failure (28, 10%) and shock (28, 10%). By comparison, within administrative data only 130 (48%) patients were found to have 1 or more surgical complication. In general, estimated prevalence according to administrative data was 30%–60% lower for each of the individual complications, except for ventilator use, venous thromboembolism, and cardiac or respiratory arrest, each of which showed greater prevalence within administrative data (Figure 1).
Figure 1.
Prevalence of major surgical complications (in-line data values represent chart review data). OR = operating room.
Diagnostic performance metrics are presented in Table 3. Overall sensitivity for the composite outcome was 72% (95% CI 64%–79%), whereas specificity was 80% (95% CI 72%–87%), PPV was 82% (95% CI 74%–88%) and NPV was 70% (95% CI 62%–77%). Across the individual complications, the average sensitivity, specificity, PPV and NPV was 65% (range 36%–100%), 97% (range 91%–100%), 70% (range 22%–100%) and 95% (84%–100%), respectively. The overall accuracy of administrative data in capturing the composite outcome was 76% (95% CI 70%–81%), whereas accuracy ranged from 71% to 100% across each component complication. In-hospital mortality was well captured, with all performance metrics ranging from 92%–100%.
Table 3.
Diagnostic performance of administrative data in capturing major surgical complications and in-hospital mortality
| Outcome | Diagnostic metric (95% confidence interval) | ||||
|---|---|---|---|---|---|
| Sensitivity, % | Specificity, % | Positive predictive value, % | Negative predictive value, % | Accuracy, % | |
| Major complication | 72 (64–79) | 80 (72–87) | 82 (74–88) | 70 (62–77) | 76 (70–81) |
| Atrial fibrillation or flutter | 59 (33–82) | 99 (97–100) | 83 (52–98) | 97 (94–99) | 97 (94–98) |
| Acute renal failure | 36 (19–56) | 99 (97–100) | 83 (52–98) | 93 (89–96) | 93 (89–95) |
| New-onset hemodialysis | 100 (NR–100) | 100 (99–100) | 100 (NR–100) | 100 (99–100) | 100 (99–100) |
| Bleeding | 53 (44–62) | 85 (78–90) | 75 (64–83) | 69 (62–75) | 71 (65–76) |
| Cardiac or respiratory arrest | 56 (21–86) | 93 (89–96) | 22 (7–44) | 98 (96–100) | 92 (88–95) |
| Coma | 100 (NR–100) | 99 (97–100) | 50 (NR–93) | 100 (99–100) | 99 (97–100) |
| Venous thromboembolism | 60 (26–88) | 97 (94–99) | 43 (18–71) | 98 (96–100) | 96 (92–98) |
| Myocardial infarction | 100 (NR–100) | 100 (98–100) | 50 (NR–99) | 100 (99–100) | 100 (98–100) |
| Ventilator use for ≥ 48 hours | 95 (77–100) | 94 (91–97) | 60 (42–76) | 100 (98–100) | 94 (91–97) |
| Pneumonia | 52 (31–73) | 98 (96–100) | 75 (48–93) | 96 (92–98) | 94 (91–97) |
| Stroke | 50 (NR–99) | 100 (99–100) | 100 (NR–100) | 100 (98–100) | 100 (98–100) |
| Sepsis | 53 (38–67) | 91 (87–95) | 58 (42–72) | 90 (85–93) | 84 (80–89) |
| Shock | 50 (31–69) | 98 (96–100) | 78 (52–94) | 94 (91–97) | 93 (90–96) |
| Unplanned return to the operating room | 46 (19–75) | 100 (99–100) | 100 (54–100) | 97 (95–99) | 97 (95–99) |
| In-hospital mortality | 100 (72–100) | 100 (98–100) | 92 (62–100) | 100 (99–100) | 100 (98–100) |
NR = not reportable due to privacy requirements (all NR values ≤ 40).
Discussion
Our results showed that administrative data holdings can be used to capture a composite indicator of major surgical complications with adequate sensitivity and specificity. However, poor diagnostic performance was observed for several clinically important individual complications, such as acute renal failure, sepsis and cardiac or respiratory arrest.
For some of the individual complications, poor performance was likely owing in part to low prevalence within this specific cohort. Specifically, although sensitivity and specificity are independent of prevalence, very small event numbers likely reduced the precision of our estimates. For example, as a major health event, it would be expected that stroke is coded within administrative data sets. Using a similar algorithm and data holdings, Hall and colleagues13 examined the validity of diagnostic codes for stroke and transient ischemic attack and reported an overall sensitivity of 82%, as compared with 50% observed in the current study. With fewer than 6 cases of postoperative stroke in the current study, even a single missed case would have had a substantial effect on our results.
Other complications, such as acute kidney injury, are likely captured poorly within administrative databases. A systematic review by Vlasschaert and colleagues14 reported an overall sensitivity of 29% for acute kidney injury, with a range from 15% to 81% across the 9 studies included. Given the findings from previous research, the poor performance of our algorithm in detecting acute kidney injury was somewhat expected and may reflect a lack of consistency in how clinicians identify acute kidney injury, despite the presence of consensus definitions.8 Nevertheless, acute kidney injury is an important surgical outcome that should be considered as part of a definition of surgical complication, even if only a fraction of patients with acute kidney injury can be detected within administrative data holdings.
Our study was powered to validate the composite indicator of major complications, not the components of the composite outcome. Caution is advised when interpreting the values presented for the individual complications, particularly those with low prevalence, and further work is recommended to validate and refine each of the algorithms used in this study.
Several factors may account for what is and is not captured within administrative data holdings. From the point of care to the documentation and coding processes, there are many opportunities for error, including the misidentification of events by clinicians, incomplete or inaccurate description in medical records, misinterpretation of medical language by coding professionals and transcription and code-selection errors. Development of standardized definitions should help to improve the identification of clinically important outcomes; however, as with acute kidney injury, the creation of these definitions alone is insufficient and must be accompanied by education and other knowledge translation strategies. In terms of reducing data errors, advances in artificial intelligence should be used to identify potential issues for coding professionals to investigate, from data inconsistencies to improbable combinations of events. Such investments will help to improve the reliability and validity of administrative data research, as well as all of the other purposes for which administrative data are originally captured.
Limitations
A limitation of this study is that it included only patients from a single centre who underwent a relatively narrow range of procedures. The study is also limited by the use of a single clinician abstractor, the use of convenience sampling and the relatively small number of patients included, which likely affected the findings for some of the less prevalent conditions, such as stroke and myocardial infarction. Future studies should be conducted in multiple centres, include patients undergoing a broader range of surgical procedures and aim to refine the code-based algorithms developed in this study. The algorithms are directly applicable to research conducted using administrative data from Ontario, and results should be generalizable to patients treated at all institutions throughout the province; however, the algorithms will require adaptation for use outside of Ontario, and generalizability of the adapted algorithms will need to be investigated in future studies.
Conclusion
Our results showed that administrative data holdings can be used to capture a composite indicator of major surgical complications with adequate sensitivity and specificity. Although more work is required to validate algorithms for several specific complications, future studies using administrative data to investigate surgical complications should make use of the provided algorithms. Investigators should also be aware that, in general, administrative data will underestimate the true incidence of surgical complication and may be less accurate for specific, important complications.
Acknowledgements
This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health (MOH) and the Ministry of Long-Term Care (MLTC). This document used data adapted from the Statistics Canada Postal CodeOM Conversion File, which is based on data licensed from Canada Post Corporation or data adapted from the Ontario MOH Postal Code Conversion File, which contains data copied under license from Canada Post Corporation and Statistics Canada. Parts of this material are based on data or information compiled and provided by the Canadian Institute for Health Information and MOH. The analyses, conclusions, opinions and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred.
Footnotes
Competing interests: None declared.
Contributors: L. Allen, C. Vinden, K. Vogt and E. Walser designed the study. E. Walser acquired the data, which L. Dubois, P.M. Jones and J.A. McClure analyzed. L. Dubois, J.A. McClure and K. Vogt wrote the article, which L. Allen, P.M. Jones, C. Vinden and E. Walser reviewed. All authors approved the final version to be published.
Data sharing: The data set from this study is held securely in coded form at ICES. While legal data sharing agreements between ICES and data providers (e.g., health care organizations and government) prohibit ICES from making the data set publicly available, access may be granted to those who meet prespecified criteria for confidential access, available at https://www.ices.on.ca/DAS (email: das@ices.on.ca).
Funding: This project was supported by ICES Western and the Department of Surgery at the Schulich School of Medicine and Dentistry, University of Western Ontario.
References
- 1.Govindarajan A, Urbach DR, Kumar M, et al. Outcomes of daytime procedures performed by attending surgeons after night work. N Engl J Med 2015;373:845–53. [DOI] [PubMed] [Google Scholar]
- 2.Dubois L, Allen B, Bray-Jenkyn K, et al. Higher surgeon annual volume, but not years of experience, is associated with reduced rates of postoperative complications and reoperations after open abdominal aortic aneurysm repair. J Vasc Surg 2018;67:1717–1726.e5. [DOI] [PubMed] [Google Scholar]
- 3.Pincus D, Jenkinson R, Paterson M, et al. Association between surgical approach and major surgical complications in patients undergoing total hip arthroplasty. JAMA 2020;323:1070–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Wallis CJ, Ravi B, Coburn N, et al. Comparison of postoperative outcomes among patients treated by male and female surgeons: a population based matched cohort study. BMJ 2017;359:j4366. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Kelly EC, Winick-Ng J, McClure JA, et al. Hysterectomy in Ontario: a population-based study of outcomes and complications in minimally invasive compared with abdominal approaches. J Obstet Gynaecol Can 2019;41:1168–76. [DOI] [PubMed] [Google Scholar]
- 6.Neuman MD. The importance of validation studies in perioperative database research. Anesthesiology 2015;123:243–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Ehrenstein V, Petersen I, Smeeth L, et al. Helping everyone do better: a call for validation studies of routinely recorded health data. Clin Epidemiol 2016;8:49–51. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Singer M, Deutschman CS, Symour CW, et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016;315:801–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Mehta RL, Kellum JA, Shah SV, et al. Acute Kidney Injury Network: report of an initiative to improve outcomes in acute kidney injury. Crit Care 2007;11:R31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Mackenzie G. The definition and classification of pneumonia. Pneumonia 2016;8:14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Thygesen K, Alpert JS, Jaffe AS, et al. Fourth universal definition of myocardial infarction. J Am Coll Cardiol 2018;72:2231–64. [DOI] [PubMed] [Google Scholar]
- 12.Benchimol EI, Smeeth L, Guttmann A, et al. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) Statement. PLoS Med 2015;12:e1001885. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Hall R, Mondor L, Porter J, et al. Accuracy of administrative data for the coding of acute stroke and TIAs. Can J Neurol Sci 2016;43:765–73. [DOI] [PubMed] [Google Scholar]
- 14.Vlasschaert ME, Bejaimal SA, Hackam DG, et al. Validity of administrative database coding for kidney disease: a systematic review. Am J Kidney Dis 2011;57:29–43. [DOI] [PubMed] [Google Scholar]