To the Editor:
Within medicine, there is a growing focus on understanding the care that is provided at the end of life (1). Although the ability to capture preferences for life-sustaining treatments in routinely collected healthcare data is limited, one existing option is the International Classification of Diseases, Ninth Revision (ICD-9), code V49.86, signifying “do-not-resuscitate status.” This code may potentially be used to examine the care of patients with do-not-resuscitate (DNR) status (2, 3); to examine trends, patterns, and variation in end-of-life decision making (4); or potentially to adjust for patient preferences when examining the quality of care (5). However, performance characteristics for this code are unknown. Therefore, we performed a single-center validation study to determine the sensitivity and specificity of the V49.86 code for identifying whether hospitalized patients had a DNR status at any time during their hospitalization. We hypothesized that the code would be specific but not sensitive for the presence of DNR status.
Methods
This study was approved by the Columbia University Medical Center (CUMC) Institutional Review Board (IRB-AAAP2112 New York, NY). Written informed consent was waived. Data for this study were extracted from the CUMC Clinical Data Warehouse, a repository of electronic medical records for all hospitalizations at CUMC. We included all adult admissions (age ≥18 yr) from August 1, 2013, through August 31, 2015, including repeat hospitalizations. We collected demographic and clinical variables, including age, sex, race, whether patients were admitted to an intensive care unit (ICU), discharge destination, and all diagnoses received during the hospitalization. Although our unit of analysis was a hospitalization, we use the term patients in this letter to refer to hospitalizations. At CUMC, specific note templates are routinely used to document DNR status within the electronic medical record, whereby completion of the note automatically generates a DNR order in the patient’s chart. (DNR orders also may be entered separately.) We identified 1) whether a patient had DNR status documented either by a DNR note and/or by a DNR order and 2) whether a patient had a V49.86 code (Figure 1). Because timing of DNR orders identified with the V49.86 code may be unclear, we also determined when DNR notes and orders were placed in relation to the admission date. The reference standard for this study was the presence of a DNR note and/or a DNR order in the Clinical Data Warehouse database. To validate our reference standard, we reviewed 50 charts labeled as having documented DNR status and looked for the physical presence of a DNR note or order, as well as 50 charts labeled as not having documented DNR status to confirm absence of a note or order. We specified a priori that the reference standard would be considered adequate if the DNR note had a positive predictive value (PPV) and negative predictive value (NPV) greater than 90%. For the V49.86 code, all diagnosis fields were searched for the presence of the code, and any use was counted.
Figure 1.
Flowchart demonstrating the number of hospital admissions with documented do-not-resuscitate (DNR) status and with the International Classification of Diseases, Ninth Revision, V49.86 code billed.
We calculated sensitivity, specificity, positive and negative predictive value, positive and negative likelihood ratios, and the area under the receiver operating characteristic curve for the V49.86 code. We also calculated these characteristics for several predefined subgroups to determine if the performance of the code differed, including 1) patients who did and those who did not die during hospitalization, 2) patients who were and those who were not admitted to the ICU, 3) patients stratified by Charlson comorbidity index, 4) patients with metastatic cancer, and 5) patients with conditions for which in-hospital mortality is reported as a measure of the quality of care (pneumonia, heart failure, acute myocardial infarction [AMI], chronic obstructive pulmonary disease [COPD], and stroke). Database management and analyses were performed using SAS 9.4 (SAS Institute) and Stata 13.1 (StataCorp) software.
Results
During the study period, there were 100,910 hospitalizations at CUMC of 68,657 different patients, of whom 7,369 (7.3%) had documented DNR status and 5,428 (5.4%) had a diagnosis code of V49.86. DNR status was documented within 24 hours of admission in approximately half (54.7%; n = 4,027) of patients with a DNR order and approximately half of patients with a V49.86 code billed (54.0%; n = 2,931). Compared with the overall cohort, patients who had documented DNR status or a diagnosis code of V49.86 were more likely to be older; to have a higher number of comorbidities; to have a diagnosis of cancer, pneumonia, AMI, COPD, or stroke; to be discharged to hospice or a skilled nursing facility; or to have died during hospitalization (Table 1).
Table 1.
Characteristics of patients at Columbia University Medical Center, August 2013–August 2015
| Total |
DNR Note/Order Present |
ICD-9 V49.86 Code Present |
|
|---|---|---|---|
| n (%) (N = 100,910) | n (%) (N = 7,369) | n (%) (N = 5,428) | |
| Age, yr | |||
| 18–64 | 61,828 (61.3) | 1,222 (18.9) | 960 (17.7) |
| 65–74 | 17,267 (17.1) | 1,176 (18.2) | 917 (16.9) |
| 75–84 | 13,390 (13.3) | 1,669 (25.8) | 1,422 (26.2) |
| ≥85 | 8,424 (8.4) | 2,409 (37.2) | 2,129 (39.2) |
| Sex | |||
| Female | 58,578 (58.1) | 3,840 (59.3) | 3,261 (60.1) |
| Male | 42,332 (42.0) | 2,636 (40.7) | 2,167 (39.9) |
| Race | |||
| White | 38,442 (38.1) | 2,400 (37.1) | 2,022 (37.3) |
| Black | 12,241 (12.1) | 737 (11.4) | 608 (11.2) |
| Asian | 2,778 (2.8) | 122 (1.9) | 100 (1.8) |
| Other/declined | 47,449 (47.0) | 3,217 (49.7) | 2,698 (49.7) |
| Charlson comorbidity index | |||
| 0 | 48,567 (48.1) | 1,237 (19.1) | 894 (16.5) |
| 1–2 | 29,177 (28.9) | 2,126 (32.8) | 1,843 (34.0) |
| ≥3 | 23,166 (23.0) | 3,113 (48.1) | 2,691 (49.6) |
| Metastatic cancer | 10,830 (10.7) | 1,421 (21.9) | 1,188 (21.9) |
| Pneumonia | 1,771 (1.8) | 363 (20.5) | 326 (18.4) |
| Heart failure | 15,784 (15.6) | 2,193 (13.9) | 1,950 (12.4) |
| Acute myocardial infarction | 2,410 (2.4) | 348 (14.4) | 308 (12.8) |
| Chronic obstructive pulmonary disease | 1,196 (1.2) | 153 (12.8) | 119 (10.0) |
| Stroke | 1,921 (1.9) | 305 (15.9) | 265 (13.8) |
| Admitted to intensive care | 14,142 (14.0) | 2,084 (14.7) | 1,573 (11.1) |
| Discharge destination | |||
| Home | 61,050 (60.5) | 629 (9.7) | 524 (9.7) |
| Home with health services | 21,779 (21.6) | 1,505 (23.2) | 1,338 (24.7) |
| Skilled nursing facility | 9,203 (9.1) | 1,508 (23.3) | 1,298 (23.9) |
| Hospice | 850 (0.8) | 620 (9.6) | 576 (10.6) |
| Other facility | 861 (0.9) | 199 (3.1) | 165 (3.0) |
| Rehabilitation facility | 2,704 (2.7) | 87 (1.3) | 77 (1.4) |
| Other | 1,969 (2.0) | 22 (0.3) | 18 (0.3) |
| Died in hospital | 2,453 (2.4) | 1,905 (29.4) | 1,431 (26.4) |
Definition of abbreviations: DNR = do not resuscitate; ICD-9 = International Classification of Diseases, Ninth Revision.
The reference standard had a PPV of 100% and NPV of 100% when compared with gold standard manual chart review. The V49.86 code had high specificity (99.7% [99.6–99.7]) and moderate sensitivity (69.2% [53.3–80.5]) for identifying patients with DNR status, as well as a high PPV (94.0% [93.3–94.6]), NPV (97.6% [97.5–97.7]), and area under the receiver operating characteristic curve (0.84 [0.84–0.85]). Specificity was slightly lower for patients who died during hospitalization (93.7% [90.7–96.0]), whereas sensitivity was lower for patients without comorbidities (58.6% [55.9–61.2]). Sensitivity was modestly improved for patients with certain conditions (pneumonia, 75.7% [71.3–79.8]; AMI, 74.7% [70.1–78.9]; stroke, 74.2% [69.2–78.8]), whereas specificity was maintained (Table 2).
Table 2.
Performance characteristics of the International Classification of Diseases, Ninth Revision, V49.86 code for do-not-resuscitate status
| Sensitivity (%) | Specificity (%) | AUC | PPV (%) | NPV (%) | LR+ | LR− | ||
|---|---|---|---|---|---|---|---|---|
| All patients |
69.2 (68.2–70.3) | 99.7 (99.6–99.7) | 0.84 (0.84–0.85) | 94.0 (93.3–94.6) | 97.6 (97.5–97.7) | 198.0 (177.5–220.9) | 0.31 (0.30–0.32) | |
| Died in hospital |
67.4 (65.4–69.4) | 93.7 (90.7–96.0) | 0.81 (0.79–0.82) | 98.4 (97.6–99.0) | 33.5 (30.6–36.5) | 10.7 (7.2–15.9)) | 0.35 (0.32–0.37) | |
| Survived hospitalization |
69.9 (68.7–71.2) | 99.7 (99.6–99.7) | 0.85 (0.84–0.85) | 92.4 (91.5–93.2) | 98.3 (98.2–98.4) | 214.3 (191.3–240.1) | 0.30 (0.29–0.31) | |
| Admitted to ICU |
||||||||
| Yes |
65.5 (63.5–67.5) | 99.5 (99.4–99.6) | 0.83 (0.82–0.83) | 96.3 (95.2–97.1) | 93.7 (93.2–94.1) | 131.4 (101.7–169.7) | 0.35 (0.33–0.37) | |
| No |
70.9 (69.6–72.2) | 99.7 (99.6–99.7) | 0.85 (0.85–0.86) | 93.0 (92.2–93.8) | 98.2 (98.1–98.3) | 216.2 (191.6–244.0) | 0.29 (0.28–0.30) | |
| Charlson comorbidity index, % |
||||||||
| 0 |
58.6 (55.9–61.2) | 99.8 (99.8–99.9) | 0.79 (0.78–0.80) | 91.7 (89.7–93.4) | 98.8 (98.7–98.9) | 373.3 (296.1–470.8) | 0.41 (0.39–0.44) | |
| 1–2 |
70.7 (68.9–72.5) | 99.6 (99.5–99.6) | 0.85 (0.84–0.86) | 93.7 (92.5–94.8) | 97.4 (97.2–97.6) | 163.1 (135.8–195.9) | 0.29 (0.28–0.31) | |
| ≥3 |
72.4 (70.9–73.9) | 99.3 (99.2–99.4) | 0.86 (0.85–0.87) | 94.9 (94.0–95.7) | 95.2 (94.9–95.5) | 103.8 (87.7–122.8) | 0.28 (0.26–0.29) | |
| Metastatic cancer |
||||||||
| Yes |
71.5 (69.2–73.7) | 99.4 (99.3–99.6) | 0.85 (0.84–0.87) | 95.7 (94.4–96.8) | 95.3 (94.8–95.7) | 129.5 (98.3–170.5) | 0.29 (0.27–0.31) | |
| No |
68.6 (67.4–69.8) | 99.7 (99.6–99.7) | 0.84 (0.84–0.85) | 93.5 (92.7–94.2) | 97.9 (97.8–98.0) | 209.6 (186.0–236.1) | 0.32 (0.30–0.33) | |
| Pneumonia |
||||||||
| Yes |
75.7 (71.3–79.8) | 99.0 (98.3–99.4) | 0.87 (0.85–0.89) | 95.7 (92.9–97.6) | 93.1 (91.6–94.3) | 73.5 (43.5–124.1) | 0.25 (0.21–0.29) | |
| No |
68.8 (67.7–69.9) | 99.7 (99.6–99.7) | 0.84 (0.84–0.85) | 93.9 (93.2–94.5) | 97.7 (97.6–97.8) | 202.7 (181.3–226.7) | 0.31 (0.30–0.32) | |
| Heart failure |
||||||||
| Yes |
72.5 (70.7–74.2) | 99.1 (99.0–99.3) | 0.86 (0.85–0.87) | 94.2 (93.0–95.2) | 95.0 (94.6–95.3) | 84.2 (70.0–101.3) | 0.28 (0.26–0.30) | |
| No |
67.5 (66.2–68.8) | 99.7 (99.7–99.8) | 0.84 (0.83–0.84) | 93.9 (93.0–94.6) | 98.1 (98.0–98.2) | 254.6 (222.3–291.5) | 0.33 (0.31–0.34) | |
| Acute myocardial infarction |
||||||||
| Yes |
74.7 (70.1–78.9) | 99.2 (98.7–99.5) | 0.87 (0.85–0.89) | 94.8 (91.7–97.0) | 95.3 (94.3–96.2) | 94.2 (57.7–154.1) | 0.26 (0.22–0.30) | |
| No |
68.9 (67.8–70.0) | 99.7 (99.6–99.7) | 0.84 (0.84–0.85) | 93.9 (93.2–94.6) | 97.7 (97.6–97.8) | 202.8 (181.3–226.9) | 0.31 (0.30–0.32) | |
| COPD |
||||||||
| Yes |
67.1 (59.4–74.1) | 99.5 (98.9–99.8) | 0.83 (0.80–0.87) | 95.8 (90.5–98.6) | 94.8 (93.3–96.0) | 137.6 (57.0–332.0) | 0.33 (0.27–0.41) | |
| No |
69.3 (68.2–70.3) | 99.7 (99.6–99.7) | 0.84 (0.84–0.85) | 93.9 (93.3–94.6) | 97.7 (97.6–97.8) | 199.0 (178.3–222.2) | 0.31 (0.30–0.32) | |
| Stroke |
||||||||
| Yes |
74.2 (69.2–78.8) | 99.2 (98.7–99.6) | 0.87 (0.84–0.89) | 95.5 (92.2–97.6) | 94.7 (93.5–95.7) | 97.7 (55.4–172.2) | 0.26 (0.22–0.31) | |
| No | 69.0 (67.9–70.1) | 99.7 (99.6–99.7) | 0.84 (0.84–0.85) | 93.9 (93.2–94.5) | 97.7 (97.6–97.8) | 201.4 (180.2–225.1) | 0.31 (0.30–0.32) | |
Definition of abbreviations: AUC = area under the receiver operating characteristic curve; COPD = chronic obstructive pulmonary disease; ICU = intensive care unit; LR = likelihood ratio; NPV = negative predictive value; PPV = positive predictive value.
Discussion
In a single academic medical center, we found that the ICD-9 code V49.86 was highly specific and moderately sensitive in identifying hospitalized patients with a DNR status. The V49.86 code performed similarly to or better than other ICD-9 diagnosis codes, which generally have had high specificity but low sensitivity for identifying patients with a given condition (6–10). Performance of the code was stable across various subgroups of patients, with small variation in sensitivity for patients with certain conditions. Despite its relative accuracy, the code did not discriminate the timing of DNR status, which may limit its utility because early DNR status is more likely to reflect preexisting preferences for treatment, whereas DNR status occurring later in hospitalization may reflect patient preferences, clinical decline, and/or the quality of care delivered. Our study does have limitations because our findings arise from a single center. Also, we did not evaluate the equivalent ICD-10 code (Z66 “Do-not-resuscitate”) that is now largely used, though prior studies have demonstrated a similar performance between equivalent ICD-9 and ICD-10 codes (11–13). Given the observed performance characteristics, we recommend that the V49.86 code be used for studies designed to capture the general epidemiology of patients with DNR status (e.g., evaluating secular trends and variation in DNR orders). However, given that administrative data cannot reliably determine timing of DNR status within a hospitalization, this code should not be used for risk adjustment within a hospitalization, though prior DNR may be used for risk adjustment in longitudinal studies.
Supplementary Material
Footnotes
Supported by National Institutes of Health grants K01 HL116768 and R01 HL136660 (A.J.W.) and a Paul B. Beeson Career Development Award (K08AG051184) from the National Institute on Aging and the American Federation for Aging Research (M.H.).
Author disclosures are available with the text of this letter at www.atsjournals.org.
References
- 1.Sudore RL, Lum HD, You JJ, Hanson LC, Meier DE, Pantilat SZ, et al. Defining advance care planning for adults: a consensus definition from a multidisciplinary Delphi panel. J Pain Symptom Manage. 2017;53:821–832.e1. doi: 10.1016/j.jpainsymman.2016.12.331. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Kazaure H, Roman S, Sosa JA. High mortality in surgical patients with do-not-resuscitate orders: analysis of 8256 patients. Arch Surg. 2011;146:922–928. doi: 10.1001/archsurg.2011.69. [DOI] [PubMed] [Google Scholar]
- 3.Wenger NS, Pearson ML, Desmond KA, Brook RH, Kahn KL. Outcomes of patients with do-not-resuscitate orders: toward an understanding of what do-not-resuscitate orders mean and how they affect patients. Arch Intern Med. 1995;155:2063–2068. [PubMed] [Google Scholar]
- 4.Cheng MT, Hsih FY, Tsai CL, Tsai HB, Tsai DFC, Fang CC. Increased rate of DNR status in hospitalized end-of-life patients in Taiwan. Intensive Care Med. 2016;42:1816–1817. doi: 10.1007/s00134-016-4509-y. [DOI] [PubMed] [Google Scholar]
- 5.Walkey AJ, Barnato AE, Wiener RS, Nallamothu BK. Accounting for patient preferences regarding life-sustaining treatment in evaluations of medical effectiveness and quality. Am J Respir Crit Care Med. 2017;196:958–963. doi: 10.1164/rccm.201701-0165CP. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.McIsaac DI, Gershon A, Wijeysundera D, Bryson GL, Badner N, van Walraven C. Identifying obstructive sleep apnea in administrative data: a study of diagnostic accuracy. Anesthesiology. 2015;123:253–263. doi: 10.1097/ALN.0000000000000692. [DOI] [PubMed] [Google Scholar]
- 7.Taylor DH, Jr, Østbye T, Langa KM, Weir D, Plassman BL. The accuracy of Medicare claims as an epidemiological tool: the case of dementia revisited. J Alzheimers Dis. 2009;17:807–815. doi: 10.3233/JAD-2009-1099. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Iwashyna TJ, Odden A, Rohde J, Bonham C, Kuhn L, Malani P, et al. Identifying patients with severe sepsis using administrative claims: patient-level validation of the Angus implementation of the international consensus conference definition of severe sepsis. Med Care. 2014;52:e39–e43. doi: 10.1097/MLR.0b013e318268ac86. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Chan M, Lim PL, Chow A, Win MK, Barkham TM. Surveillance for Clostridium difficile infection: ICD-9 coding has poor sensitivity compared to laboratory diagnosis in hospital patients, Singapore. PLoS One. 2011;6:e15603. doi: 10.1371/journal.pone.0015603. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Fawzy A, Bradford M, Lindenauer PK, Walkey AJ. Identifying vasopressor and inotrope use for health services research. Ann Am Thorac Soc. 2016;13:414–418. doi: 10.1513/AnnalsATS.201508-487BC. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Jolley RJ, Sawka KJ, Yergens DW, Quan H, Jetté N, Doig CJ. Validity of administrative data in recording sepsis: a systematic review. Crit Care. 2015;19:139. doi: 10.1186/s13054-015-0847-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Jolley RJ, Quan H, Jetté N, Sawka KJ, Diep L, Goliath J, et al. Validation and optimisation of an ICD-10-coded case definition for sepsis using administrative health data. BMJ Open. 2015;5:e009487. doi: 10.1136/bmjopen-2015-009487. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Gedeborg R, Furebring M, Michaëlsson K. Diagnosis-dependent misclassification of infections using administrative data variably affected incidence and mortality estimates in ICU patients. J Clin Epidemiol. 2007;60:155–162. doi: 10.1016/j.jclinepi.2006.05.013. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.