Skip to main content
NCBI home page
PLOS One logoLink to PLOS One
. 2023 Mar 13;18(3):e0283045. doi: 10.1371/journal.pone.0283045

Validity of ICD codes to identify do-not-resuscitate orders among older adults with heart failure: A single center study

Katherine Callahan 1,*,#, Yubraj Acharya 1,#, Christopher S Hollenbeak 1,#
Editor: Keith M Harris2
PMCID: PMC10010557  PMID: 36913366

Abstract

Background

Observational research on the advance care planning (ACP) process is limited by a lack of easily accessible ACP variables in many large datasets. The objective of this study was to determine whether International Classification of Disease (ICD) codes for do-not-resuscitate (DNR) orders are valid proxies for the presence of a DNR recorded in the electronic medical record (EMR).

Methods

We studied 5,016 patients over the age of 65 who were admitted to a large, mid-Atlantic medical center with a primary diagnosis of heart failure. DNR orders were identified in billing records from ICD-9 and ICD-10 codes. DNR orders were also identified in the EMR by a manual search of physician notes. Sensitivity, specificity, positive predictive value and negative predictive value were calculated as well as measures of agreement and disagreement. In addition, estimates of associations with mortality and costs were calculated using the DNR documented in EMR and the DNR proxy identified in ICD codes.

Results

Relative to the gold standard of the EMR, DNR orders identified in ICD codes had an estimated sensitivity of 84.6%, specificity of 96.6%, positive predictive value of 90.5%, and negative predictive value of 94.3%. The estimated kappa statistic was 0.83, although McNemar’s test suggested there was some systematic disagreement between the DNR from ICD codes and the EMR.

Conclusions

ICD codes appear to provide a reasonable proxy for DNR orders among hospitalized older adults with heart failure. Further research is necessary to determine if billing codes can identify DNR orders in other populations.

Introduction

Research on issues in advance care planning (ACP) can be challenging as there are few ACP variables routinely collected in large datasets [1, 2]. Some datasets, such as the Health Retirement Study’s (HRS) Exit Interview, which began conducting surveys in 1996, contain survey data on advance care planning. However, the data collected in the Exit Interview are completed with a “proxy informant” of HRS participants who have died, and the number of cases with this information is relatively small with less than 1,000 participants per year [3]. The ability to identify ACP data in large administrative and other observational health care datasets that are easily accessible to health services researchers would substantially help advance research in ACP.

One potentially important source of data is administrative discharge data, which are collected by hospitals that use them for billing, monitoring, and reporting to organizations that measure quality of care [4]. These data provide information on hospital stays and routinely include International Classification of Diseases (ICD) codes, which reflect patient diagnoses and procedures patients receive during their hospitalization [4]. Another potential source of data is claims data, which records information from third-party payers and often includes ICD codes to document diagnoses and procedures. Since 2010 there has been an ICD code for do-not-resuscitate (DNR) orders—V49.86 in the 9th revision and Z66 in the 10th revision, which began to be widely used in the US in October 2015 [1, 5]. A DNR is a signed medical order that tells healthcare providers not to begin cardiopulmonary resuscitation (CPR) in the event of cardiac arrest [6]. In recent years, the documentation of a DNR status has been incorporated into all-encompassing legal documents known as Medical Orders for Life Sustaining Treatment (MOLST). These forms include options for other forms of life sustaining treatments such as breathing machines and feeding tubes, and regardless of how DNR status is documented for patients they are still recorded in patient charts or electronic medical records (EMR) [7]. Although a DNR is only part of the ACP process, it can be important for patients to define their resuscitation preferences as CPR is often not beneficial for individuals at end-of-life (EOL) who are already weak and frail, such as those with severe chronic and terminal illnesses such as heart failure (HF) [8]. Almost half of individuals die within the first five years of a HF diagnosis and it has an exacerbation-prone trajectory that often results in frequent hospitalizations, which contrasts with chronic, declining trajectories more common in other chronic and terminal illnesses such as dementia [9]. Researchers therefore have access to a proxy for DNR orders through the widely available ICD codes. This proxy may allow for research around DNR orders as part of the ACP process. For example, we recently used ICD-9 codes as a proxy for DNR orders to examine whether patients with DNR orders were more likely to be readmitted after being hospitalized in Pennsylvania for HF [10].

A major concern about ICD codes for DNR orders is it is not known whether they are valid proxies for actual DNR orders, and whether they reflect a patient’s resuscitation preference that is recorded in their medical record. There are several reasons that the ICD code may not reflect a patient’s actual DNR status. ICD codes are used by hospital billing coders in order to maximize revenues to the hospital. If the ICD code for DNR does not have revenue value in billing then hospital billing coders may not prioritize recording DNR in the limited slots available for entry. Under this scenario, DNRs could be routinely under-represented in administrative data. If the ICD code for a DNR order is not a reasonably accurate measure for the legal document that a patient signs to make their wishes regarding CPR known to the clinical team, then caution should be used in using it as a proxy for DNR, and previous studies that have used it should be interpreted with caution. However, if the ICD codes could be shown to be valid proxies for actual DNR status, then observational data sets such as administrative discharge data and claims data may be useful for EOL research.

There is some evidence that the ICD code for the DNR order is accurate [1, 6]. For example, one study found that that the ICD-9 code for the DNR order (V49.86) is moderately sensitive and highly specific among a cross-section of patients admitted to a hospital [1]. Another study that investigated the validity of the ICD-9 code among primary diagnoses of myocardial infraction, pneumonia, and heart failure (HF), suggested that the DNR proxy has a high sensitivity but only moderate specificity [6]. While these studies suggest that the ICD code provides a potentially promising proxy for DNR, there are several limitations. These studies include primary diagnoses that include many observations for which a DNR order is not likely to be relevant. For example, it is reasonable to think that discussing resuscitation preferences with a 16-year-old patient admitted with a distal radius fracture is not a high priority for health care providers. This in turn can affect the accuracy of the ICD code documented. In addition, these studies did not include data on the more recent ICD-10 codes that have been in widespread use in the US since 2015.

In order to address these gaps, this study sought to determine the validity of ICD proxies for DNR in elderly patients hospitalized with a primary diagnosis of HF. Our sample includes both ICD-9 and ICD-10 codes for DNR, and also explores whether use of the ICD proxy produces different estimates when used as a substitute for a gold standard measure of DNR orders found in the EMR.

Objectives

The purpose of this study was to determine whether ICD-9 and ICD-10 codes for DNR are valid proxies for the presence of a DNR that is recorded in the EMR. We utilized data from a large, academic hospital in the mid-Atlantic region and merged billing records that contain ICD codes with data regarding DNRs documented in the EMR. In addition to comparing ICD DNR proxies to actual DNR documentation, we also estimated the association between DNR and mortality and costs using both the ICD code DNR and the DNR recorded in the patients EMR in order to determine whether the ICD measure provides similar estimates of association relative to actual DNR documented in the EMR.

Materials and methods

Data source

This research used data from the Penn State Milton S. Hershey Medical Center. Billing records were obtained from the cost-accounting system (Change Healthcare, Nashville, TN), which provided ICD codes from which DNR was identified, as well as details on patient characteristics, hospital stay, and total costs. In addition, the EMR for each patient was examined, which provided physician notes that documented the patient’s treatment preferences, including a DNR order. This study was reviewed and approved by the Pennsylvania State University Institutional Review Board and informed consent was not required. Data were received de-identified from the medical center’s decision support department. In addition, two authors (KC and CH) had full access to all the data in the study and take responsibility for its integrity and the data analysis.

Cohort

Patients admitted between 2013 and 2018 with a principle diagnosis of HF were identified using the ICD-9 code of 428.xx (heart failure) and ICD-10 code of I50.xx (heart failure). Patients under the age of 65 were excluded as older patients are more likely to participate in any type of ACP [11, 12]. The final sample included n = 5,016 patients.

Outcomes

DNR proxies in ICD codes (iDNR) were identified using an ICD-9 code of V49.89 and an ICD-10 code of Z66 among secondary diagnosis codes in billing records. The actual DNR contained in the EMR (eDNR) was identified by searching the EMR, particularly physician notes, for any mention of DNR status. Secondary outcomes included in-hospital mortality and total cost of admission. Costs are estimated in the cost-accounting system using ratios of costs-to-charges at the department level and include an overhead allocation. Thus, total costs in this study represent fully-loaded operating costs for the HF admission. Furthermore, costs were adjusted for inflation to 2021 US dollars using the medical care component of the Consumer Price Index.

Covariates

Several covariates that have been shown to be associated with DNR in the existing literature were included in the analyses. To control for patient characteristics, indicators were used for age groups (65–74, 75–84, or 85+), sex (male or female), race (white or other race/ethnicity), payer (Medicare, Medicaid, commercial, or other), and hospital admission type (elective, emergent, or urgent). To control for comorbidities, we included the Charlson comorbidity index (CCI), which is a weighted sum of seventeen common comorbidities that can be identified using ICD codes [13–15]. The CCI was transformed into four categories (0, 1, 2–3, or 4+).

Statistical analysis

In order to determine whether the iDNR predicts the presence of eDNR, the sensitivity and specificity of the iDNR were estimated relative to the eDNR as the gold standard. Sensitivity is defined as the probability that a patient has an iDNR given that patient has an eDNR; specificity is defined as the probability that a patient does not have an iDNR given that the patient does not have an eDNR. We also computed the positive predictive value (PPV), which is defined as the probability that a patient has an eDNR given that they have an iDNR, and the negative predictive value (NPV), which is the probability that a patient does not have an eDNR given that they do not have a iDNR. Additionally, these measures were computed across strata of patient characteristics, including age, sex, race/ethnicity, and comorbidities.

Two univariate tests of agreement between the two DNR measures were performed [16]. First, the level of agreement between the iDNR and eDNR was estimated using a Cohen’s kappa statistic. The kappa statistic is a correlation measure with values between -1 and 1 that reflect the overall level of agreement between two predictions and is often used to measure inter-rater reliability [17, 18]. Second, systematic disagreement between the iDNR and eDNR was compared using McNemar’s test. McNemar’s test was used to investigate whether there are systematic differences in disagreement between the two measures. In this case, it tests whether the iDNR is significantly more likely to be negative when the eDNR is positive, or whether the eDNR is more likely to be negative when the iDNR is positive [19].

After estimating agreement between the two DNR measures, we examined whether the use of each measure would yield substantially different estimates of association when used as a covariate in multivariable analysis about the associations between DNR and outcomes. The outcomes that we were specifically interested in—mortality and costs—were modeled using the iDNR and eDNR as independent variables and controlling for other covariates. Finding that the two measures yielded substantially different associations would be evidence that the iDNR is not a good proxy for eDNR. Linear probability models were used to model the relationship between DNR and mortality, after controlling for the covariates described above. Linear regression models were used to model the association between total costs and DNR after controlling for the patient characteristics. Stata software (version 15.1, College Station, TX) was used for all analyses, and statistical significance was defined as p<0.05.

Results

Patient characteristics

A description of patient characteristics stratified by DNR status is provided in Table 1. The summary compares elderly HF patients without a DNR (n = 3,398) to those with an iDNR (n = 1,388) and those with an eDNR (n = 1,498) across several covariates. In this sample, elderly HF patients were more likely to have a DNR (either iDNR or eDNR) if they were older, white, female, and had Medicare as their primary insurance. There were no substantial differences between patients with an iDNR and patients with an eDNR. The largest difference observed was between CCI categories, where there were slightly more patients with an iDNR that had a score of zero compared to patients with an eDNR (63% vs. 60%). Compared to patients with DNRs, patients without a DNR were more likely to have a CCI score of one to three, yet slightly less likely to have a CCI score of zero or four or more.

Table 1. Summary statistics for elderly HF patients, stratified by DNR status in ICD codes and EMR.

Variable No DNR ICD DNR EMR DNR
(N = 3398) (N = 1388) (N = 1498)
Age
 65–74 43.73% 18.23% 18.89%
 75–84 37.32% 34.08% 33.71%
 85+ 18.95% 47.69% 47.40%
Sex
 Female 43.11% 55.12% 54.81%
 Male 56.89% 44.88% 45.19%
Race
 White 93.76% 96.76% 96.93%
 Other Race 6.24% 3.24% 3.07%
Charlson Comorbidity Index Score
 0 63.83% 63.33% 60.21%
 1 7.65% 7.49% 8.48%
 2 to 3 17.30% 16.79% 17.49%
 4+ 11.21% 12.39% 13.82%
Payor
 Commercial 19.63% 16.86% 16.69%
 Medicare 70.95% 71.97% 72.50%
 Other 9.42% 11.17% 10.81%
Open in a new tab

Measures of agreement

A 2×2 contingency table of iDNR by eDNR is shown in Table 2. Cohen’s kappa statistic for agreement between iDNR and eDNR was estimated to be 0.83, which suggests “almost perfect agreement” between the iDNR and eDNR [17]. On the other hand, the McNemar test suggested there were systematic differences in disagreement between iDNR and eDNR. In terms of disagreement, a false negative iDNR was significantly more likely than a false positive iDNR. However, as seen in Table 2, the proportion of both false negatives and false positives was small.

Table 2. Contingency table of eDNR and iDNR.

eDNR
DNR No DNR Total
iDNR DNR 1,268 120 1,388
No DNR 230 3,398 3,628
Total 1,498 3,518 5,016
Open in a new tab

Sensitivity and specificity

For the full sample (N = 5,016), sensitivity of iDNR was estimated to be 84.6%, which means that 84.6% of the individuals who have an eDNR were correctly documented as having an iDNR. Specificity of iDNR was estimated to be 96.6%, which means that 96.6% of patients who did not have an eDNR also did not have an iDNR. The PPV was estimated to be 91.4% and NPV was estimated to be 93.7%, meaning that among the entire sample of individuals with and without DNRs, the iDNR correctly identified who did and did not have a DNR in slightly over 91% of cases. Table 3 shows the sensitivity, specificity, PPV and NPV of the full sample, as well as the sample stratified by patient characteristics of iDNR relative to eDNR. Within the subgroups of patient characteristics, the highest level of sensitivity was found among female patients (87.1%) and the highest level of specificity was found among patients who had a CCI score of 2 to 3 (98.8%). On the other hand, the lowest sensitivity was found among patients who had a CCI score of four or more (79.2%), and the lowest level of specificity was among patients aged 85+ (92.5%).

Table 3. Summary of diagnostic characteristics of iDNR relative to eDNR, stratified by patient characteristics.

Patient Characteristics Sample Size Sensitivity Specificity PPV NPV
Total (N = 5016) 84.60% 96.60% 91.40% 93.70%
Age
 65–74 (N = 1795) 80.20% 98.30% 95.20% 92.10%
 75–84 (N = 1815) 85.30% 96.80% 91.90% 93.90%
 85+ (N = 1406) 85.90% 92.50% 83.00% 93.90%
Sex
 Female (N = 2336) 87.10% 96.70% 91.80% 94.60%
 Male (N = 2680) 81.70% 96.50% 90.90% 92.50%
Race
 White (N = 4753) 84.60% 96.50% 91.20% 93.60%
 Other Race (N = 263) 87.00% 97.70% 94.10% 94.60%
Charlson Comorbidity Index Score
 0 (N = 3172) 86.30% 95.60% 89.20% 94.20%
 1 (N = 391) 78.70% 98.50% 95.70% 91.60%
 2 to 3 (N = 857) 86.30% 98.80% 96.90% 94.40%
 4+ (N = 596) 79.20% 97.90% 94.30% 91.70%
Code Set
 ICD-9 (N = 1841) 81.85% 98.48% 95.81% 92.72%
 ICD-10 (N = 3175) 86.27% 95.55% 89.20% 94.23%
Open in a new tab

Mortality

The iDNR provided similar estimates of the association between DNR and mortality as the eDNR. This is shown in Table 4, which provides the results of linear probability models using both iDNR and eDNR and controlling for covariates. The estimated likelihood of dying before discharge was 11.3 percentage points higher for patients with a DNR as indicated by the iDNR (p<0.001), and 9.3 percentage points higher for patients with an eDNR (p<0.001). Additionally, there were other covariates not related to the central question of the iDNR vs. eDNR that were also associated with in-hospital mortality.

Table 4. Results of linear probability models of mortality controlling for eDNR versus iDNR as independent variables as well as other covariates.

Variable eDNR Coefficient P-Value iDNR Coefficient P-Value
DNR 0.093 <0.001 0.113 <0.001
Age
 65–74 Reference
 75–84 -0.011 0.139 -0.013 0.074
 85+ -0.022 0.007 -0.027 0.001
Sex
 Female -0.014 0.024 -0.015 0.014
 Male Reference
Race
 White Reference
 Other Race -0.012 0.375 -0.012 0.383
Charlson Comorbidity Index Score
 0 Reference
 1 0.003 0.797 0.008 0.468
 2 to 3 0.001 0.91 0.004 0.674
 4+ 0.018 0.071 0.022 0.023
Payor
 Private -0.002 0.772 -0.003 0.726
 Medicare Reference
 Other 0.107 <0.001 0.106 <0.001
Constant 0.030 <0.001 0.028 <0.001
Open in a new tab

Hospitalization costs

As seen in Table 5, iDNR and eDNR provided different estimates of the association between DNR and total hospitalization costs, however, they were in the same direction. Linear regression models suggested that, after controlling for covariates and among patients who died, having an iDNR order was associated with cost savings of $19,313 (p = 0.002) and eDNR was associated with cost savings of $39,728 (p<0.001).

Table 5. Results of linear regression models of hospital cost controlling for eDNR versus iDNR as independent variables as well as other covariates.

Variable Coefficient for EMR DNR P-Value Coefficient for ICD DNR P-Value
Died 51,510.77 <0.001 39,331.73 <0.001
DNR -668.07 0.672 -1,740.50 0.285
Died & DNR -39,727.87 <0.001 -19,312.69 0.002
Age
 65–74 Reference
 75–84 -9,477.56 <0.001 -9,346.32 <0.001
 85+ -16,159.29 <0.001 -16,036.67 <0.001
Sex
 Female -4,002.47 0.003 -4,008.08 0.003
 Male Reference
Race/Ethnicity
 Non-Hispanic White Reference
 Other Race/Ethnicity -6,017.59 0.041 -6,071.84 0.040
Charlson Comorbidity Index Score
 0 Reference
 1 -12869.1 <0.001 -13232.09 <0.001
 2 to 3 -8963.363 <0.001 -9185.259 <0.001
 4+ -9038.276 <0.001 -9379.919 <0.001
Payor
 Private -1,068.03 0.531 -1,170.47 0.493
 Medicare Reference
 Other -3,096.45 0.170 -3,505.91 0.121
Intercept 36,814.53 <0.001 37,157.06 <0.001
Open in a new tab

Among patients who died, the mean cost saving for patients with an eDNR and no iDNR was significantly greater than the cost saving for patients with an iDNR and no eDNR. Among patients who died without a DNR, costs were approximately $51,500 higher (p<0.001) in the eDNR model and approximately $39,000 higher (p<0.001) in the iDNR model. Costs were similar in magnitude for patients who had a DNR and survived.

There were other patient characteristics associated with costs. For example, patients who were older, female, non-white, had higher co-morbidity score, and had lower total hospitalization costs. Additionally, compared to those who were insured by Medicare, patients covered by other types of insurance had lower costs of approximately $3,000.

Discussion

Results from this study suggest that the ICD code for DNR is a valid proxy for the presence of a DNR order documented in the medical record in this sample of elderly HF patients. This was indicated by a high sensitivity (85%), specificity (97%), PPV (91%) and NPV (94%) relative to the gold standard of a DNR documented in the EMR. The estimated kappa statistic (κ = 0.83) also suggests there is substantial agreement between DNR in billing records and the EMR. It should be noted that the results from McNemar’s test suggest a systematic difference between false negatives and false positives; however, this test only considers the off-diagonal elements of the contingency table, which were relatively small compared to the total sample (Table 2). The sensitivity, specificity, PPV, NPV, and kappa statistic all consider the entire contingency table, and therefore together are strong indicators that the iDNR is valid. In comparison to other common ICD code proxies used in current research studies the iDNR measure performed as well or better [20–22].

However, the estimates of associations between DNR and mortality and DNR and cost using the iDNR and eDNR indicate that when using the iDNR the results need to be carefully considered. While the association between iDNR and mortality and eDNR and mortality were similar (9.3 percentage points vs. 11.3 percentage points), the associations with costs were very different ($19,313 for iDNR, and $39,728 for eDNR). Given the high sensitivity and specificity of iDNR, this finding is most likely due to the fact that there are more false negative results for iDNR, and also because cost data are usually highly skewed and have a large variance. These results suggest caution in using the iDNR as a proxy to measure DNR orders.

This study makes several contributions to the literature on DNR orders and ACP. Although the sample is restrictive, estimates of sensitivity, specificity, PPV, NVP, and agreement suggest that both the ICD-9 and ICD-10 codes are reasonable proxies for DNR orders recorded in elderly HF patient medical records. Second, the results provide confirmation of other validation studies that have examined ICD-9 codes and similarly found that the ICD code is a reasonable proxy for DNR orders [1, 6]. The results of this study indicate that for elderly HF patients estimates of association between iDNR and patient outcomes, such as mortality or costs, are similar in direction to those that would be obtained using eDNR. Additionally, the results of this study support that DNR status can be identified in large administrative databases, enabling researchers to adjust for this as a potential confounder if necessary [23].

Similar to Fonseca et al. (2018), we found the iDNR to have very high specificity at 97%. However, unlike their study, we also found the iDNR sensitivity to be high (85% vs. 69%) [1]. This may be due to differences in the sample, as we only looked at patients over the age of 65 with HF, and the Fonseca et al. study (2018) included patients at any age and admitted with any diagnosis, although the most common admission diagnoses were metastatic cancer, pneumonia, HF, acute myocardial infraction, chronic obstructive pulmonary disease, and stroke. In addition, our iDNR measure included both ICD-9 and ICD-10 diagnosis codes; Fonseca et al. (2018) only included ICD-9 codes in their proxy.

Results of this study differ somewhat from those reported in Goldman et al. (2013) who investigated the accuracy of DNR orders using administrative discharge data in California. They found that DNR orders were more likely to be recorded inaccurately as the time from admission increased. Our study used ICD codes that were determined after the patient was discharged, and includes DNR orders that could have been completed at any time during the hospital stay. One potential explanation for this difference is a difference in the definition of iDNR as DNR orders in California are measured only within 24 hours of admission. They do not capture DNR orders that may have been finalized after admission and may be less accurate for patients with longer hospitalizations [6].

One piece of evidence from our study that suggested that the iDNR might not be a good proxy for eDNR was McNemar’s test, which found significant disagreement between the two measures. This test compared the number of false positive results (N = 120) and false negative (N = 230) results and showed that the iDNR yielded systematically more false negatives. Certainly, there should be a greater expectation for false negative results since coders have a limited number of placeholders for ICD codes, and priority is given to those that result in higher reimbursement. The false positive cases are more of a puzzle. In our sample there were 120 false positive cases out of 5,016 patients, which is a relatively small proportion. Their presence raises the question, however, how a billing coder who assigns ICD codes may have indicated the presence of a DNR when we did not find such evidence (i.e., DNR in the EMR). It is possible that these may have just been simple error, it is also possible that DNR orders were counted in false positive cases when patients received a consultation with palliative care or hospice teams, which they might have interpreted as having a DNR order. Another potential explanation is that coders may have observed evidence of a DNR order for a prior hospitalization and assumed (correctly or incorrectly) that the DNR order would apply to the current hospital admission [6, 24].

Limitations

The results of this study only came from patients at a single medical center. Therefore, it is possible that the use of ICD codes for DNR orders we observed can be attributed to the practice patterns of this particular center’s billing coders as well as the specific practice patterns around ACP and DNR orders of physicians, therefore these results do not reflect practice patterns at other institutions or nationally. It is possible that different coding practices at another hospital would produce significantly different results. Additionally, the patient population at this medical center lacks racial and ethnic diversity, creating the potential for a population with greater racial diversity to have very different results.

This study only examined elderly patients with HF. iDNR may be different for younger patients with HF or for patients with other chronic and terminal illnesses. In the future it may be useful to examine cohorts that include adults aged 40 and older as the use of DNR orders increases at this age [10]. Another limitation to our study the use of only patients with a principal diagnosis of HF. It is possible that the most severely ill patients with HF have a principal diagnosis of acute respiratory failure or myocardial infraction and have HF as the secondary diagnosis [25]. Therefore, it is possible that our sample criteria excluded a part of the HF population.

There are other challenges to measuring associations between patient outcomes and DNR orders that are due to discrepancies between a patient’s wishes and care that is actually received. A patient may have a DNR order in place and still receive CPR after a cardiac event. This may occur if the clinical team is unaware of the DNR order or if the patient’s family overrides the DNR order. Family members’ desire for heroic measures may be given preference to patient’s wishes, particularly in hospital environments where lawsuits are more prevalent and if the medical care ends in death for the patient [24]. The greater the extent of these occurrences, the more the estimates of association between DNR orders and outcomes will be biased toward the null.

Conclusions

Using a sample of elderly patients with a primary diagnosis of HF at a single, large, academic medical center in the mid-Atlantic, this study examined the validity of ICD codes for DNR orders and compared their use relative to DNR status in the EMR. Results showed high sensitivity, specificity, PPV, NPV, and overall agreement. However, they showed that the use of the ICD proxy did not always provide similar estimates of association between patient outcomes and DNR status. These results suggest that the ICD codes provide a reasonable proxy for DNR orders in this sample, and open the door to further applied research on DNR orders. Additional research is needed to validate the ICD proxies using alternative coding pathways to identify HF as discussed above, as well as in other diseases and other populations. In addition, future research should continue to explore other proxies for other elements of ACP beyond DNR orders.

Data Availability

There are legal restrictions to share the de-identified data as it came from billing records and electronic medical records from Penn State Milton S. Hershey Medical Center. These restrictions have not been imposed by the Penn State Institutional Review Board, but by Penn State Milton S. Hershey Medical Center. Contact information regarding data access is: Cletis Earle - Chief Information Officer - Penn State Health (E-mail: cearle1@pennstatehealth.psu.edu).

Funding Statement

The author(s) received no specific funding for this work.

References

  • 1.Fonseca L, Walkey AJ, Ma X, Hua M. Validation of the V49.86 code for do-not-resuscitate status in hospitalized patients at a single academic medical center. Ann Am Thorac Soc. 2018;15(10):1234–7. doi: 10.1513/AnnalsATS.201804-257RL [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Sudore RL, Heyland DK, Lum HD, Rietjens JAC, Korfage IJ, Ritchie CS, et al. Outcomes that define successful advance care planning: A delphi panel consensus. J Pain Symptom Manage. 2018;55(2):245–55.e8. doi: 10.1016/j.jpainsymman.2017.08.025 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Health and Retirement Study. 2018 Exit Interviews: Data description and usage. 2020. https://hrsdata.isr.umich.edu/sites/default/files/documentation/data-descriptions/
  • 4.Andrews RM. Statewide hospital discharge data: Collection, use, limitations, and improvements. Health Serv Res. 2015;50 Suppl 1(Suppl 1):1273–99. doi: 10.1111/1475-6773.12343 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Centers for Disease Control and Prevention. International Classification of Diseases, (ICD-10-CM/PCS) Transition—Background 2015. https://www.cdc.gov/nchs/icd/icd10cm_pcs_background.htm.
  • 6.Goldman LE, Chu PW, Osmond D, Bindman A. Accuracy of do not resuscitate (DNR) in administrative data. Med Care Res Rev 2013;70(1):98–112. doi: 10.1177/1077558712458455 [DOI] [PubMed] [Google Scholar]
  • 7.Jain VG, Greco PJ, Kaelber DC. Code Status Reconciliation to Improve Identification and Documentation of Code Status in Electronic Health Records. Appl Clin Inform. 2017;8(1):226–234. Published 2017 Mar 8. doi: 10.4338/ACI-2016-08-RA-0133 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.American Heart Association. What is CPR? 2021. https://cpr.heart.org/en/resources/what-is-cpr.
  • 9.Ahluwalia SC, Enguidanos S. Advance care planning among patients with heart failure: A review of challenges and approaches to better communication. J Clin Outcomes Manage. 2015;22:1–10. [Google Scholar]
  • 10.Callahan K, Kitko L, Van Scoy LJ, Hollenbeak CS. Do-not-resuscitate orders and readmission among elderly patients with heart failure in Pennsylvania: An observational study, 2011–2014. Heart Lung. 2020;49(6):812–6. doi: 10.1016/j.hrtlng.2020.09.012 [DOI] [PubMed] [Google Scholar]
  • 11.Yadav KN, Gabler NB, Cooney E, Kent S, Kim J, Herbst N, et al. Approximately One In Three US Adults Completes Any Type Of Advance Directive For End-Of-Life Care. Health Aff (Millwood). 2017;36(7):1244–51. doi: 10.1377/hlthaff.2017.0175 [DOI] [PubMed] [Google Scholar]
  • 12.Carr D, Luth EA. Advance Care Planning: Contemporary Issues and Future Directions. Innov Aging. 2017;1(1):igx012. doi: 10.1093/geroni/igx012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Charlson M, Szatrowski TP, Peterson J, Gold J. Validation of a combined comorbidity index. J Clin Epidemiol. 1994;47(11):1245–51. doi: 10.1016/0895-4356(94)90129-5 [DOI] [PubMed] [Google Scholar]
  • 14.Charlson ME, Pompei P, Ales KL, Mackenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation. J Chronic Dis. 1987;40(5):373–83. doi: 10.1016/0021-9681(87)90171-8 [DOI] [PubMed] [Google Scholar]
  • 15.Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992;45(6):613–9. doi: 10.1016/0895-4356(92)90133-8 [DOI] [PubMed] [Google Scholar]
  • 16.Watson PF, Petrie A. Method agreement analysis: A review of correct methodology. Theriogenology. 2010;73(9):1167–79. doi: 10.1016/j.theriogenology.2010.01.003 [DOI] [PubMed] [Google Scholar]
  • 17.Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70(4):213–20. doi: 10.1037/h0026256 [DOI] [PubMed] [Google Scholar]
  • 18.McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276–82. [PMC free article] [PubMed] [Google Scholar]
  • 19.Fagerland MW, Lydersen S, Laake P. The McNemar test for binary matched-pairs data: mid-p and asymptotic are better than exact conditional. BMC Med Res Methodol. 2013;13:91-. doi: 10.1186/1471-2288-13-91 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Roumie CL, Mitchel E, Gideon PS, Varas-Lorenzo C, Castellsague J, Griffin MR. Validation of ICD-9 codes with a high positive predictive value for incident strokes resulting in hospitalization using Medicaid health data. Pharmacoepidemiol Drug Saf. 2008;17(1):20–6. doi: 10.1002/pds.1518 [DOI] [PubMed] [Google Scholar]
  • 21.Kern DM, Davis J, Williams SA, Tunceli O, Wu B, Hollis S, et al. Validation of an administrative claims-based diagnostic code for pneumonia in a US-based commercially insured COPD population. Int J Chron Obstruct Pulmon Dis. 2015;10:1417–25. doi: 10.2147/COPD.S83135 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Prins H, Hasman A. Appropriateness of ICD-coded diagnostic inpatient hospital discharge data for medical practice assessment. A systematic review. Methods Inf Med. 2013;52(1):3–17. doi: 10.3414/ME12-01-0022 [DOI] [PubMed] [Google Scholar]
  • 23.Bradford MA, Lindenauer PK, Wiener RS, Walkey AJ, Do-not-resuscitate status and observational comparative effectiveness research in patients with septic shock. Crit Care Med. 2014;42:2042–2047. doi: 10.1097/CCM.0000000000000403 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Pandit MS, Pandit S. Medical negligence: Coverage of the profession, duties, ethics, case law, and enlightened defense—A legal perspective. Indian J Urol. 2009;25(3):372–8. doi: 10.4103/0970-1591.56206 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Mehta AB, Syeda SN, Wiener RS, Walkey AJ. Epidemiological trends in invasive mechanical ventilation in the United States: A population-based study. J Crit Care. 2015;30(6): 1217–1221. doi: 10.1016/j.jcrc.2015.07.007 [DOI] [PMC free article] [PubMed] [Google Scholar]

Decision Letter 0

Keith M Harris

18 Oct 2022

PONE-D-22-21926Validity of ICD Codes to Identify Do-Not-Resuscitate OrdersPLOS ONE

Dear Dr. Callahan,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Dec 02 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Keith M. Harris, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified (1) whether consent was informed and (2) what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information.

If you are reporting a retrospective study of medical records or archived samples, please ensure that you have discussed whether all data were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. If patients provided informed written consent to have data from their medical records used in research, please include this information.

3. We note you have included a table to which you do not refer in the text of your manuscript. Please ensure that you refer to Table 2 in your text; if accepted, production will need this reference to link the reader to the Table.

4. Please include your tables as part of your main manuscript and remove the individual files. Please note that supplementary tables (should remain/ be uploaded) as separate "supporting information" files

Additional Editor Comments:

Reviewer 1 has pointed out several important factors to address. Authors are urged to take a more cautious approach to interpreting findings and stating conclusions. Please address each of the reviewer's comments and suggestions. If data is available, suggestions for expanding age range and ethnic diversity, etc. appear reasonable. The treatment of continuous variables and unweighted variables needs to be addressed according to PLOS' requirements for analyses to be conducted according to high standards. This is a valuable work, and we look forward to your response.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Major Points

1. My primary concern regarding the manuscript is the disconnect between the generalizability with which the authors discuss their results and the restrictive nature of their cohort. The authors use their results to discuss how discharge and claims data can be used to identify DNR orders in future studies. However, their cohort is exceedingly restrictive and may not be the most appropriate with which to identify patients admitted with heart failure. Their reported statistics of sensitivity, specificity, PPV, and NPV are only valid for this specific cohort and likely are not generalizable to other conditions or patient cohorts. The authors should first consider their patient selection along the below delineated lines and also discuss their results with much greater clarity about the limitations of their findings throughout the abstract and manuscript, not just in the limitations section of the discussion.

a. Age – the authors select patients who are >65 years of age. Use of DNR orders does increase with age meaning there would be greater prevalence of DNR orders in an older cohort. However, previous studies have indicated use of DNR orders begins to increase at around age 40. Use of an older cohort would result in higher sensitivity and specificity due to the higher prevalence. The authors should address this by either widening the cohort or doing a sensitivity analysis which includes patients down to age 40.

b. Cohort definition – the authors only include patients with a principle diagnosis of HF (ICD9 and ICD10) as the ‘HF’ cohort. However, previous work (Mehta et al. Journal of Critical Care. 2015.) demonstrates that the most severely ill patients with HF (i.e. those most likely to have a DNR order) may have a principle diagnosis of acute respiratory failure and a secondary diagnosis of HF (i.e. presented with hypoxia due to pulmonary edema) or a principle diagnosis of acute MI with a secondary diagnosis of HF. These alterative coding pathways may be used to increase reimbursement but still tend to identify patients with HF. The authors could consider a sensitivity analysis with a wider cohort definition or they should mention that their definition may have selectively excluded individuals most likely to have a DNR order.

c. Choice of heart failure – the authors restrict their analysis to HF only for unclear reasons. They suggest that studies that include all patients may include diagnoses for which DNR may not be relevant. However, DNR orders issued early in the course of admission tend to be more indicative of comorbid condition rather than the acute presenting illness. Moreover, other cohorts such as cancer may be even more relevant for DNR orders. The authors should provide greater justification for why HF was chosen, what biases such a choice may create, and limit their discussion about the utility of their findings to patients with HF.

2. My second major concern deals with the use of a single center and conclusion that can be drawn from it. I commend the authors on their sample size and efforts but I am concerned about any conclusions drawn from a single center on generalizable billing practices that might be heavily biased. The authors correctly mention several possible reasons why DNR orders may be systematically over or under estimated in billing codes from a given center. Each health center tends to bill in a unique fashion. Biases in terms of including or not including certain codes tend to happen at the hospital level, not the patient level. Therefore, the use of a single health system does not necessarily reduce the potential of coding bias to either over or under represent DNR orders in ICD codes. The authors should be far more specific in addressing the limitations of a single center in their discussion. Specifically, they should address the potential for different coding practices at different hospitals to result in significantly different results that what they have observed.

3. I am also concerned about the lack of racial diversity in this single center study. There is extensive evidence indicating that different racial groups have significantly different rates of ACPs and DNR orders when inpatient. Specifically, patients who identify as black are far less likely to have a DNR order when hospitalized. Therefore, a population with greater racial diversity may have very different performance metrics for iDNR vs eDNR. The authors should address this issue in the discussion.

4. The authors use and presentation of the CCI is confusing. The CCI is a continuous score ranging from 0 to 37 based on a weighted assessment of each individual comorbidity. The authors indicate that they are using the CCI but then present data in categories of 0, 1, 2-3, and 4+. The language used in the results seems to indicate that these are not categories of the scores but rather number of comorbidities. As each comorbidity carries different weights, simply counting the number of comorbidities is misguided. Moreover, when considering advanced directives, certain comorbidities are far more associated with a DNR order (e.g. cancer) than others. Therefore, simply looking at the CCI to compare differences between the iDNR and eDNR group can lead to misguided conclusions. I would suggest the authors clarify how the CCI is being used in the models and would present different rates of certain key comorbidities like cancer.

5. I am additionally confused by the use of cost as a potential outcome. The fact that patients with a DNR order have lower costs is not surprising or informative. Patients with a DNR order are less likely to receive high cost invasive therapies and, given associations with mortality, have shorter length of stays. Moreover, I am also confused by the primary exposure in the cost analysis as presented in Table 4. It appears that the primary exposure is a 4 level categorical variable with levels of Died, DNR, Died & DNR, and Lived & no DNR. This variable is very confusing as there are potentially overlapping categories and potentially groups not represented. For example, if a patient died but had no DNR does that mean they are level 1 or is there overlap between level 1 and level 3. Moreover, if the groups are mutually exclusive, is level 2 really Lived and DNR? The authors need to clarify why cost is an important outcome and how they are defining the exposure. One might consider looking at LOS rather than cost given that costs can vary widely between hospitals.

6. Several studies have shown significant changes to coding patterns associated with the shift from ICD-9 to ICD-10 in October 2015.While the authors address the validity of iDNR in several subgroups, they do not report differences in iDNR validity for ICD-9 vs ICD-10 codes. I would strongly suggest the authors do further subgroup analysis focusing on ICD-9 vs ICD-10 to show that secular changes in coding pratices and the shift from ICD-9 to ICD-10 has not changed validity.

Minor Points

1. In the introduction, the authors indicate that DNR is a legal document. However, in most states it is a hospital order that only applies to that hospitalization. Medical Orders for Life Sustaining Treatments (MOLST) forms have taken over in many states as the primary method of communicating goals of care between hospitalizations. In most states, MOLST forms are legal documents. The authors should just clarify this point about DNR orders.

2. The Methods Data Source section states that ‘both authors had full access to all the data’. There are 3 authors so I would suggest modifying the text to indicate whether all others, 2/3 authors, etc had access to the data.

3. In the results discussing mortality, the section around non-DNR drivers of mortality is a bit distracting. Knowing that “other pay sources” had the highest mortality is not related to the central question of iDNR vs eDNR.

4. Figure 1 is really a Table, not a figure.

5. Previous work has shown that DNR status can be a major confounder in comparative effectiveness and observational research (Bradford et al. CCM. 2014). Being able to identify DNR status in large administrative databases would enable to researchers to adjust for this key confounder if/when necessary. The authors should explore this fact in the discussion.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachment

Submitted filename: callahan DNR review.docx

Click here for additional data file. (16.3KB, docx)
PLoS One. 2023 Mar 13;18(3):e0283045. doi: 10.1371/journal.pone.0283045.r002

Author response to Decision Letter 0

23 Nov 2022

Response to Reviewers

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

We have made the following changes – based on PLOS ONE - Formatting Sample Main Body PDF:

� Changed all headings (level 1) to font size 18

� Changed all subheading (level 2) to font size 16

� Changed heading of “Methods” to “Materials and methods”

� Changed all subheading to be written in sentence case

� Moved tables to be included directly after the paragraph in which they are first cited

� Changed Table title to bold font

� Changed references titles to be in sentence case

� Change reference journals names to be abbreviated

We have also made the following changes – based on PLOS ONE - Formatting Sample Title Authors Affiliations PDF:

� Title changed to sentence case

� Changed author names to not include education, added provided symbols as necessary

� Added author contribution

2. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified (1) whether consent was informed and (2) what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information.

Participant consent was waived by the Pennsylvania State University Institutional Review Board. This is now noted in the Methods section.

If you are reporting a retrospective study of medical records or archived samples, please ensure that you have discussed whether all data were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. If patients provided informed written consent to have data from their medical records used in research, please include this information.

We received the data in a fully de-identified fashion; this is now mentioned in the Methods section. As mentioned above, informed consent was waived by the Pennsylvania State University Institutional Review Board.

3. We note you have included a table to which you do not refer in the text of your manuscript. Please ensure that you refer to Table 2 in your text; if accepted, production will need this reference to link the reader to the Table.

Table 2 is now referenced in the text, and it has been moved to appear directly under the paragraph where it is referenced.

4. Please include your tables as part of your main manuscript and remove the individual files. Please note that supplementary tables (should remain/ be uploaded) as separate "supporting information" files

All tables have been included as part of the main manuscript, directly under the paragraph where they are mentioned in the text.

Additional Editor Comments:

Reviewer 1 has pointed out several important factors to address. Authors are urged to take a more cautious approach to interpreting findings and stating conclusions. Please address each of the reviewer's comments and suggestions. If data is available, suggestions for expanding age range and ethnic diversity, etc. appear reasonable. The treatment of continuous variables and unweighted variables needs to be addressed according to PLOS' requirements for analyses to be conducted according to high standards. This is a valuable work, and we look forward to your response.

Each of Reviewer comments has been addressed as we describe below. We appreciate the expression of the value of the work.

Reviewer Comments:

Major Points

1. My primary concern regarding the manuscript is the disconnect between the generalizability with which the authors discuss their results and the restrictive nature of their cohort. The authors use their results to discuss how discharge and claims data can be used to identify DNR orders in future studies. However, their cohort is exceedingly restrictive and may not be the most appropriate with which to identify patients admitted with heart failure.

Language throughout the manuscript has been changed to underscore the restrictive nature of the cohort and the limited generalizability. Specifically, this has been done in the “Conclusions” section of the “Abstract”, as well as the first and third paragraph of the “Discussion”, and the “Conclusions” section.

Their reported statistics of sensitivity, specificity, PPV, and NPV are only valid for this specific cohort and likely are not generalizable to other conditions or patient cohorts. The authors should first consider their patient selection along the below delineated lines and also discuss their results with much greater clarity about the limitations of their findings throughout the abstract and manuscript, not just in the limitations section of the discussion.

a. Age – the authors select patients who are >65 years of age. Use of DNR orders does increase with age meaning there would be greater prevalence of DNR orders in an older cohort. However, previous studies have indicated use of DNR orders begins to increase at around age 40. Use of an older cohort would result in higher sensitivity and specificity due to the higher prevalence. The authors should address this by either widening the cohort or doing a sensitivity analysis which includes patients down to age 40.

We have revised our language to be more cautious about generalizability beyond a similar patient population. We no longer have access to the EMR records of the sample of patients, therefore we would not be able to expand the cohort or do a sensitivity analysis. In the “Limitations” section of the “Discussion” we have added that in the future it would be useful to look at patients aged 40 and above.

b. Cohort definition – the authors only include patients with a principle diagnosis of HF (ICD9 and ICD10) as the ‘HF’ cohort. However, previous work (Mehta et al. Journal of Critical Care. 2015.) demonstrates that the most severely ill patients with HF (i.e. those most likely to have a DNR order) may have a principle diagnosis of acute respiratory failure and a secondary diagnosis of HF (i.e. presented with hypoxia due to pulmonary edema) or a principle diagnosis of acute MI with a secondary diagnosis of HF. These alterative coding pathways may be used to increase reimbursement but still tend to identify patients with HF. The authors could consider a sensitivity analysis with a wider cohort definition or they should mention that their definition may have selectively excluded individuals most likely to have a DNR order.

This is an important point. We agree that our cohort identification strategy may have left out some patients with severe HF that have progressed to other conditions such as ARF and MI who might be discharged with a primary diagnosis of ARF or MI. However, attempting to include patients with a secondary diagnosis of HF and primary diagnosis of ARF or MI would also capture patients for whom HF was not necessarily severe and was a comorbid condition. It seems to us that only a prospective study could adequately distinguish between these types of patients. We have opted to continue with our cohort as defined, but have added language to the limitations section acknowledging this potential selection bias and appreciate the reviewer for pointing it out.

c. Choice of heart failure – the authors restrict their analysis to HF only for unclear reasons. They suggest that studies that include all patients may include diagnoses for which DNR may not be relevant. However, DNR orders issued early in the course of admission tend to be more indicative of comorbid condition rather than the acute presenting illness. Moreover, other cohorts such as cancer may be even more relevant for DNR orders. The authors should provide greater justification for why HF was chosen, what biases such a choice may create, and limit their discussion about the utility of their findings to patients with HF.

We have clarified in the second paragraph of the “Introduction” section that HF was chosen because:

� It can be important for patients to define their resuscitation preferences as CPR is often not beneficial for individuals at end-of-life (EOL) who are already weak and frail, such as those with severe chronic and terminal illnesses such as heart failure (HF).

� Specific statistics of HF and its trajectory make communication during this time even more important

2. My second major concern deals with the use of a single center and conclusion that can be drawn from it. I commend the authors on their sample size and efforts but I am concerned about any conclusions drawn from a single center on generalizable billing practices that might be heavily biased. The authors correctly mention several possible reasons why DNR orders may be systematically over or under estimated in billing codes from a given center. Each health center tends to bill in a unique fashion. Biases in terms of including or not including certain codes tend to happen at the hospital level, not the patient level. Therefore, the use of a single health system does not necessarily reduce the potential of coding bias to either over or under represent DNR orders in ICD codes. The authors should be far more specific in addressing the limitations of a single center in their discussion. Specifically, they should address the potential for different coding practices at different hospitals to result in significantly different results that what they have observed.

This has been specifically addressed in the first paragraph of “Limitations” in the “Discussion” section.

3. I am also concerned about the lack of racial diversity in this single center study. There is extensive evidence indicating that different racial groups have significantly different rates of ACPs and DNR orders when inpatient. Specifically, patients who identify as black are far less likely to have a DNR order when hospitalized. Therefore, a population with greater racial diversity may have very different performance metrics for iDNR vs eDNR. The authors should address this issue in the discussion.

We agree; the cohort is reflective of the regional population in central Pennsylvania, and is not very diverse. We have added language acknowledging this limitation in the Discussion section.

4. The authors use and presentation of the CCI is confusing. The CCI is a continuous score ranging from 0 to 37 based on a weighted assessment of each individual comorbidity. The authors indicate that they are using the CCI but then present data in categories of 0, 1, 2-3, and 4+. The language used in the results seems to indicate that these are not categories of the scores but rather number of comorbidities. As each comorbidity carries different weights, simply counting the number of comorbidities is misguided.

We agree that the language implied that these were numbers of comorbidities. We have changed the language in the “Results” section to be consistent with our treatment of CCI as categories of an index.

5. Moreover, when considering advanced directives, certain comorbidities are far more associated with a DNR order (e.g. cancer) than others. Therefore, simply looking at the CCI to compare differences between the iDNR and eDNR group can lead to misguided conclusions. I would suggest the authors clarify how the CCI is being used in the models and would present different rates of certain key comorbidities like cancer.

The CCI already assigns greater weight to comorbidities that carry a higher mortality risk. We ran some sensitivity analyses with a few of the more severe comorbidities (e.g., metastatic cancer) in the index and found that they were not more informative than the index overall. We therefore opted to include the index as a way to reflect overall comorbidity burden. This is addressed in the “Covariates” portion of the “Material and methods” section.

6. I am additionally confused by the use of cost as a potential outcome. The fact that patients with a DNR order have lower costs is not surprising or informative. Patients with a DNR order are less likely to receive high cost invasive therapies and, given associations with mortality, have shorter length of stays.

Moreover, I am also confused by the primary exposure in the cost analysis as presented in Table 4. It appears that the primary exposure is a 4 level categorical variable with levels of Died, DNR, Died & DNR, and Lived & no DNR. This variable is very confusing as there are potentially overlapping categories and potentially groups not represented. For example, if a patient died but had no DNR does that mean they are level 1 or is there overlap between level 1 and level 3. Moreover, if the groups are mutually exclusive, is level 2 really Lived and DNR?

The authors need to clarify why cost is an important outcome and how they are defining the exposure. One might consider looking at LOS rather than cost given that costs can vary widely between hospitals.

Regarding cost as a potential outcome, we disagree somewhat with the reviewer. Numerous studies have suggested that much of the resources used in the care of people in the last year of life are futile. Patients with DNR forego potentially futile care, consistent with their wishes. Using cost as an outcomes provides some quantification of those benefits. Also, a DNR is a piece of paper that expresses a patient’s wish not to receive CPR in the event of cardiac arrest. No other care should be impacted by the presence of a DNR. Patients should receive all therapies, including high cost therapies, as long as there is no cardiac event. So, the DNR is not inherently cost-saving, unless the patient has a cardiac event and the family and clinical team honor the patient’s wishes. The use of main effects plus an interaction term for DNR and Died are intended to reflect this hypothesis about cost savings and DNR. We should expect that a patient who has a DNR and dies should have cost savings since they should not receive heroic care if the death was from a cardiac event. Patients who do not have a DNR and who also die should be more costly since they should receive heroic measures. Patients who do not have a DNR and who live should have costs that are similar to patients who have a DNR and live. The combination of main effects plus interaction term captures all of these groups, with no overlap and no double counting. We agree with the reviewer, however, that our inclusion of Lived & DNR as a reference group is confusing, would imply overlap and double counting, and detracts from the main effects plus interaction approach. This reference group has been removed. Also, since this is a single center study there should be no concerns about costs varying widely between hospitals, though of course generalizability is still a concern as mentioned above.

7. Several studies have shown significant changes to coding patterns associated with the shift from ICD-9 to ICD-10 in October 2015.While the authors address the validity of iDNR in several subgroups, they do not report differences in iDNR validity for ICD-9 vs ICD-10 codes. I would strongly suggest the authors do further subgroup analysis focusing on ICD-9 vs ICD-10 to show that secular changes in coding pratices and the shift from ICD-9 to ICD-10 has not changed validity.

Agreed, the further subgroup analysis focusing on ICD-9 vs ICD-10 is now shown in Table 3.

Minor Points

1. In the introduction, the authors indicate that DNR is a legal document. However, in most states it is a hospital order that only applies to that hospitalization. Medical Orders for Life Sustaining Treatments (MOLST) forms have taken over in many states as the primary method of communicating goals of care between hospitalizations. In most states, MOLST forms are legal documents. The authors should just clarify this point about DNR orders.

We agree, and have added language in the “Introduction” section to clarify this point about DNR orders and MOLST forms.

2. The Methods Data Source section states that ‘both authors had full access to all the data’. There are 3 authors so I would suggest modifying the text to indicate whether all others, 2/3 authors, etc had access to the data.

Agreed, this had been updated, and we have added language in the “Data Source” section to reflect that two of the authors had access to the data.

3. In the results discussing mortality, the section around non-DNR drivers of mortality is a bit distracting. Knowing that “other pay sources” had the highest mortality is not related to the central question of iDNR vs eDNR.

We agreed; this has been removed from the “Results” section discussing mortality.

4. Figure 1 is really a Table, not a figure.

Agreed, Figure 1 has been changed to Table 2, and subsequent tables have been re-numbered.

5. Previous work has shown that DNR status can be a major confounder in comparative effectiveness and observational research (Bradford et al. CCM. 2014). Being able to identify DNR status in large administrative databases would enable to researchers to adjust for this key confounder if/when necessary. The authors should explore this fact in the discussion.

This had been added to the “Discussion” section.

Attachment

Submitted filename: Response to Reviewers - 11-20-22.docx

Click here for additional data file. (28.7KB, docx)

Decision Letter 1

Keith M Harris

4 Jan 2023

PONE-D-22-21926R1Validity of ICD codes to identify do-not-resuscitate ordersPLOS ONE

Dear Dr. Callahan,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Feb 18 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Keith M. Harris, PhD

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments:

Thank you for resubmitting an improved manuscript. There are just a few points to address. Please take all of Reviewer 1's points carefully and address each. None of these impact on the validity of the study, but rather the validity of the language used to describe the study. Improving the accuracy and completeness will be helpful for researchers interested in DNROs. PLoS is generous with word counts, so feel free to explain these matters.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: No

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I commend the authors for their responses to my previously submitted review. While they have improved many aspects of the manuscript, I continue to have significant reservations about the generalizability of the findings and I do not believe the authors have narrowed their conclusions and interpretation of their data. Based on their paper, iDNR may be a valid proxy for eDNR in their specific hospital for older patients with heart failure in a predominantly white population. As coding patterns vary from hospital to hospital and from one condition to another, they cannot conclude that iDNR will perform as well in another hospital or health care system. Moreover, PPV and NPV will change based on prevalence and a hospital with more non-White patients will have a very different prevalence of eDNR. The authors acknowledge this limitation in their response but I believe that the current presentation might be misleading. I would recommend two specific changes to ensure readers have a clear picture of the scope of the study. I would suggest:

1. Title - Validity of ICD codes to identify do-not-resuscitate orders. This title very much implies a generalization of the findings that is not appropriate. I would suggest a modification to something along the lines of: "Validity of ICD-codes to identify do-not-resuscitate orders among older adults with heart failure: A Single Center Study"

2. Abstract. Conclusions. "ICD codes provide a reasonable proxy for DNR orders." I would strongly argue that the authors do not have sufficient evidence to say that iDNR is a reasonable proxy for eDNR for all patients. As many readers only read the abstract, the abstract needs to be clear about the implication. I would suggest "ICD codes provide a reasonable proxy for DNR orders among hospitalized older adults with heart failure. Further research is necessary to determine if billing codes can identify DNR orders in other populations."

Minor Point

In Table 3, to more clearly state that the CCI is a score not a count of comorbidities i would change the row title to "Charlson Comorbidity Index Score"

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2023 Mar 13;18(3):e0283045. doi: 10.1371/journal.pone.0283045.r004

Author response to Decision Letter 1

31 Jan 2023

Comments to the Author

Reviewer #1: I commend the authors for their responses to my previously submitted review. While they have improved many aspects of the manuscript, I continue to have significant reservations about the generalizability of the findings and I do not believe the authors have narrowed their conclusions and interpretation of their data.

Thank you for your time reviewing the paper and making sure the findings of our study are communicated in the most appropriate way. Below we address all of concerns your review revealed.

Based on their paper, iDNR may be a valid proxy for eDNR in their specific hospital for older patients with heart failure in a predominantly white population. As coding patterns vary from hospital to hospital and from one condition to another, they cannot conclude that iDNR will perform as well in another hospital or health care system. Moreover, PPV and NPV will change based on prevalence and a hospital with more non-White patients will have a very different prevalence of eDNR. The authors acknowledge this limitation in their response but I believe that the current presentation might be misleading. I would recommend two specific changes to ensure readers have a clear picture of the scope of the study. I would suggest:

1. Title - Validity of ICD codes to identify do-not-resuscitate orders. This title very much implies a generalization of the findings that is not appropriate. I would suggest a modification to something along the lines of: "Validity of ICD-codes to identify do-not-resuscitate orders among older adults with heart failure: A Single Center Study"

We agree, the additions to the title will add further clarity. Language has been added to the title.

2. Abstract. Conclusions. "ICD codes provide a reasonable proxy for DNR orders." I would strongly argue that the authors do not have sufficient evidence to say that iDNR is a reasonable proxy for eDNR for all patients. As many readers only read the abstract, the abstract needs to be clear about the implication. I would suggest "ICD codes provide a reasonable proxy for DNR orders among hospitalized older adults with heart failure. Further research is necessary to determine if billing codes can identify DNR orders in other populations."

We agree, the language presented in the conclusion of the Abstract has been changed to reflect the wording suggested.

Minor Point

In Table 3, to more clearly state that the CCI is a score not a count of comorbidities i would change the row title to "Charlson Comorbidity Index Score"

This row title has been changed to the suggested in all tables.

Decision Letter 2

Keith M Harris

1 Mar 2023

Validity of ICD codes to identify do-not-resuscitate orders among older adults with heart failure: A single center study

PONE-D-22-21926R2

Dear Dr. Callahan,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Keith M. Harris, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Acceptance letter

Keith M Harris

3 Mar 2023

PONE-D-22-21926R2

Validity of ICD codes to identify do-not-resuscitate orders among older adults with heart failure: A single center study

Dear Dr. Callahan:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Keith M. Harris

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    Attachment

    Submitted filename: callahan DNR review.docx

    Click here for additional data file. (16.3KB, docx)
    Attachment

    Submitted filename: Response to Reviewers - 11-20-22.docx

    Click here for additional data file. (28.7KB, docx)

    Data Availability Statement

    There are legal restrictions to share the de-identified data as it came from billing records and electronic medical records from Penn State Milton S. Hershey Medical Center. These restrictions have not been imposed by the Penn State Institutional Review Board, but by Penn State Milton S. Hershey Medical Center. Contact information regarding data access is: Cletis Earle - Chief Information Officer - Penn State Health (E-mail: cearle1@pennstatehealth.psu.edu).

    Articles from PLOS ONE are provided here courtesy of PLOS

    RESOURCES