Skip to main content
NCBI home page
PLOS One logoLink to PLOS One
. 2022 Dec 13;17(12):e0279033. doi: 10.1371/journal.pone.0279033

Accuracy of the electronic health record’s problem list in describing multimorbidity in patients with heart failure in the emergency department

Brandon L King 1,#, Michelle L Meyer 1,‡, Srihari V Chari 1,‡, Karen Hurka-Richardson 1,‡, Thomas Bohrmann 2,‡, Patricia P Chang 3,‡, Jo Ellen Rodgers 4,‡, Jan Busby-Whitehead 5,‡, Martin F Casey 1,*,#
Editor: Fabrizio Pecoraro6
PMCID: PMC9747000  PMID: 36512600

Abstract

Patients with heart failure (HF) often suffer from multimorbidity. Rapid assessment of multimorbidity is important for minimizing the risk of harmful drug-disease and drug-drug interactions. We assessed the accuracy of using the electronic health record (EHR) problem list to identify comorbid conditions among patients with chronic HF in the emergency department (ED). A retrospective chart review study was performed on a random sample of 200 patients age ≥65 years with a diagnosis of HF presenting to an academic ED in 2019. We assessed participant chronic conditions using: (1) structured chart review (gold standard) and (2) an EHR-based algorithm using the problem list. Chronic conditions were classified into 37 disease domains using the Agency for Healthcare Research Quality’s Elixhauser Comorbidity Software. For each disease domain, we report the sensitivity, specificity, positive predictive value, and negative predictive of using an EHR-based algorithm. We calculated the intra-class correlation coefficient (ICC) to assess overall agreement on Elixhauser domain count between chart review and problem list. Patients with HF had a mean of 5.4 chronic conditions (SD 2.1) in the chart review and a mean of 4.1 chronic conditions (SD 2.1) in the EHR-based problem list. The five most prevalent domains were uncomplicated hypertension (90%), obesity (42%), chronic pulmonary disease (38%), deficiency anemias (33%), and diabetes with chronic complications (30.5%). The positive predictive value and negative predictive value of using the EHR-based problem list was greater than 90% for 24/37 and 32/37 disease domains, respectively. The EHR-based problem list correctly identified 3.7 domains per patient and misclassified 2.0 domains per patient. Overall, the ICC in comparing Elixhauser domain count was 0.77 (95% CI: 0.71-0.82). The EHR-based problem list captures multimorbidity with moderate-to-good accuracy in patient with HF in the ED.

Introduction

Patients with heart failure (HF) often suffer from a heavy burden of multimorbidity (i.e., the co-occurrence of two or more chronic medical conditions). On average, patients with HF have 4 to 8 chronic medical conditions in addition to their HF [1–3]. Worsening burden of multimorbidity is predictive of increased healthcare costs, polypharmacy, and risk of both hospitalization and death among patients with HF [1, 4, 5]. Potential mechanisms by which multimorbidity worsens outcomes include poor care coordination, polypharmacy and drug interactions, and conflicting medical recommendations across disease-specific guidelines [6–9]. Common chronic conditions that are recognized as complicating the care of patients with HF include anemia, sleep disordered breathing, respiratory disease, renal impairment, type 2 diabetes, thyroid disease, musculoskeletal disorders, arrhythmias, depression, and cognitive impairment [2, 10]. Despite the emerging importance of multimorbidity, there is a lack of systematic approaches to measure co-occurring chronic conditions, particularly in the setting of emergency care [11–13]. In clinical practice, physicians use the shared mental model of the ‘problem list’, a written list of medical problems requiring management, to rapidly communicate the overall burden of medical illness in a patient (Fig 1). The problem list is particularly important for emergency department (ED) physicians who routinely care for unfamiliar and undifferentiated patients. Given its importance in clinical practice, the problem list has been widely implemented in various types of electronic health records (EHRs) [14, 15]. The EHR-based problem list provides structured data that can be readily extracted in the pursuit of studying multimorbidity. Unfortunately, problem lists are not traditionally used for research due to the perception that they are mostly managed by primary care physicians and often incomplete or inaccurate [14, 16, 17]. However, advances in health informatics—including problem-based charting, clinical decision support, and linkage to payment mechanisms in the her [18–20]—as well as federal mandates for providers to demonstrate meaningful use of health information technology—have improved documentation of conditions within problem lists [21, 22]. In the context of these advances, EHR-based problem lists may provide an accurate picture of overall disease burden in multimorbid patients.

Fig 1. Depiction of the electronic health record-based problem list.

Fig 1

Open in a new tab

Used with permission of 2021 Epic Systems Corporation.

We hypothesized that problem lists may be reasonably accurate among ED patients with a previously documented diagnosis of chronic HF for two reasons: (1) temporal trends in increased use of health information technology and (2) patients with HF often have routine outpatient encounters in which EHR-based problem lists may be updated. Measurement of multimorbidity from the problem list requires the application of a categorization system. We sought to assess the accuracy of applying the Elixhauser comorbidity index, a widely studied clinical classification system traditionally applied to administrative and hospital billing data [23], that collapses thousands of diagnoses into 38 meaningful co-morbid domains.

Materials and methods

Study design

We performed a retrospective chart review study on a random sample of 200 patients with a known diagnosis of HF who presented to an ED. All patient encounters occurred at a southeastern academic medical center between January 1, 2019 and December 31, 2019. Data was obtained from the Carolina Data Warehouse for Health (CDW-H), a central data repository sourced from two electronic health records within the University of North Carolina Health Care System (UNC-HCS). The study was approved and granted a waiver of consent by the UNC Institutional Review Board (IRB #21–0002). Study reporting followed the 2015 STARD Guideline for Reporting Diagnostic Accuracy Studies [24].

Participants

Patients were eligible for inclusion based on a documented diagnosis of HF on the EHR-based problem list. A comprehensive list of International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) diagnostic codes to identify HF were based on the Agency for Healthcare Research and Quality’s (AHRQ) Elixhauser Comorbidity Software v2021.1 [25]. These codes included: I50x, I0981, I110, I130, I132, I5181, I97130, I97131, O2912x, R570, Z95811, Z97811, Z95812. Similar codes have had high specificity (98.8%), though moderate sensitivity (78.7%), in identifying patients with HF in administrative datasets [26]. More importantly, the application of the I50x diagnostic codes to EHR-based problem lists has been demonstrated to identify patients with HF with a high positive predictive value (PPV) of 96% [27]. Given that multimorbidity is predominantly a problem in older adults, we restricted our study population to patients ≥65 years old. Finally, only a patient’s index ED visit in 2019 was eligible for random selection into the study cohort (i.e., repeat ED visits were excluded). Random selection was performed, using a random number sequence generator by the R Core Team [28]. To assess the quality of the random selection, patient characteristics of all eligible patients were compared to the characteristics of patients selected for chart review.

Measuring multimorbidity

Although there is no consensus on the optimal approach to measure multimorbidity [29, 30], the Elixhauser comorbidity index has been validated for use with both ICD-9 and ICD-10-CM diagnostic codes [23, 31]. Though the Elixhauser index was originally intended to prognosticate future mortality based on 29 disease domains, the recent release of the AHRQ’s updated Elixhauser comorbidity software for ICD-10-CM diagnostic codes (v2021.1) has expanded the number of disease domains to 38, thus providing a more comprehensive description of disease burden in patients [25]. As the AHRQ Elixhauser software is routinely updated and provided at no cost to researchers, the proposed methods provide an optimal clinical classification system to apply in the study of multimorbidity.

Chart review was performed as a gold standard measurement in capturing multimorbidity at the time of the ED encounter. A research assistant (BK) was trained in structured chart review to extract data on chronic conditions from five areas in the EHR including: (1) ED provider documentation, (2) admission summary if available, (3) outpatient encounters and discharge summaries prior to the ED encounter of interest, and (4) ‘Care Everywhere’, which is a tool for health information exchange between healthcare systems. Height and weight were directly reviewed from EHR flowsheets to determine the presence of obesity. All documented conditions were mapped back to ICD-10-CM diagnostics codes and categorized into the 38 Elixhauser domains using the AHRQ’s reference files [25]. All charts with unclear documentation or conflicting documentation were reviewed by an ED physician (MFC) for final adjudication. Except for echocardiogram reports, labs and radiographic data were not directly reviewed unless unclearly documented diagnoses required further adjudication. To assess the reliability of our gold standard measurement, 15% of the study sample was reviewed by a nurse practitioner (KHR) who was similarly trained in structured chart abstraction. Both chart reviewers were unblinded to the EHR-based problem list as most provider notes used pre-populated templates that extract data directly from the problem list. /par In parallel, multimorbidity was measured in the problem list by extracting ICD-10-CM codes tied to chronic conditions on the EHR-based problem list. Codes had to have a designation of active on the non-hospital problem list (i.e., presumably requiring outpatient management) at the time of the ED encounter. All active ICD-10-CM diagnostic codes were then entered into the Elixhauser comorbidity software for categorization into the 38 disease domains (including HF). The data analyst (TB) and research assistant (SVC) who measured multimorbidity from the problem list were blinded to the process and results of the chart review.

Finally, the AHRQ Elixhauser software calculates two summary indices, based on a patient’s multimorbidity, to prognosticate a patient’s risk of in-hospital mortality and readmission [32]. The weights were extracted from the software and used to calculate each index based on the data abstracted from the chart review. Similarly, the AHRQ software automatically calculated the indices based on the ICD-10-CM codes provided from the EHR-based problem list.

Analysis

Descriptive statistics were used to characterize prevalence of each domain and distribution of disease burden as measured by chart review. To assess the reliability of the chart review, the prevalence-adjusted bias-adjusted kappa (PABAK) was calculated for each domain across the two chart reviewers (BK, KHR). In comparing the accuracy of the EHR-based problem list against chart review, we report the sensitivity, specificity, PPV, and negative predictive value (NPV) for each Elixhauser domain. Overall agreement between chart review and the EHR-based problem list was assessed by measuring the intraclass correlation coefficient (ICC) on three summary measures of multimorbidity including: (1) Elixhauser domain count, (2) Elixhauser readmission index, and (3) Elixhauser in-hospital mortality index. To obtain the ICC, we used a two-way, mixed effects models to obtain ICCs on consistency of agreement. We also report ICCs on absolute agreement as they provide complementary information [33]. Thresholds for reporting PABAK and ICC (values less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.90 represent poor, moderate, good, and excellent agreement, respectively) were based on established guidelines [34, 35]. Bland-Altman analysis was performed to further characterize the degree of agreement between the two approaches in measuring Elixhauser domain count [36]. Finally, diagnostic accuracy by threshold classification was performed using receiver operating characteristic (ROC) curve analysis. A gold standard binary classification of Elixhauser domain count was created using the presence of multimorbidity (i.e., presence of 2 or more additional chronic conditions in accord with prior studies) as observed in the gold standard chart review [11]. Similar, Elixhauser readmission and mortality indices were dichotomized into high or low scores based on the median value observed in the study population. Area under the curve (AUC) is reported as the measure of accuracy of the EHR-based problem list in characterizing the Elixhauser domain count, readmission index, and mortality index in accord with the binary classifications discussed above. All statistical analyses were performed in Stata MP/16.1 and SAS 9.4.

Results

Participants

A total of 66,600 ED encounters between January 1, 2019-December 31, 2019 were assessed for study eligibility. After identifying index visits by older patients (ie, ≥65 years old) with HF, there were 1,130 ED encounters eligible for random selection for further chart review (S1 Fig). In the study cohort, the mean age was 78.3 years (standard deviation [SD] 9.2) and 52.5% patients were female (Table 1). Most patients were white (65%), non-Hispanic (95.5%), and used Medicare as their primary payor (95.5%). The most prevalent HF phenotype observed in the study cohort was HF with preserved ejection fraction (60%). Patient characteristics in the study cohort were similar to those found in 1,130 ED encounters eligible for random selection (S1 Table).

Table 1. Demographics of study sample and level of multimorbidity by subgroup.

N (%) Mean number of Elixhauser domains (SD) Minimum-maximum number of domains
Total 200 (100%) 5.4 (2.1) 1–10
Age
65–69 48 (24%) 5.6 (2.1) 2–9
70–74 27 (13.5%) 6.2 (2.0) 3–9
75–79 35 (17.5%) 5.5 (2.1) 1–9
80–84 32 (16%) 5.1 (2.1) 2–10
85+ 58 (29%) 5.1 (2.1) 1–10
Gender
Female 105 (52.5%) 5.6 (2.1) 1–10
Male 95 (47.5%) 5.3 (2.1) 1–10
Race
White 130 (65%) 5.6 (2.1) 1–10
Black 57 (28.5%) 5.4 (2.0) 2–10
Other 13 (6.5%) 4.5 (1.9) 2–8
Ethnicity
Non-Hispanic 191 (95.5%) 5.5 (2.1) 1–10
Hispanic 9 (4.5%) 4.8 (1.4) 3–8
Primary Payer
Medicare 191 (95.5%) 5.4 (2.1) 1–10
Non-Medicare 9 (4.5%) 5 (2.1) 2–8
HF Phenotype
Reduced EF 65 (32.5%) 5.4 (2.0) 1–9
Preserved EF 120 (60%) 5.6 (2.2) 1–10
Unspecified 15 (7.5%) 4.7 (1.9) 2–9
Open in a new tab

SD = standard deviation; HF = heart failure; EF = ejection fraction

Multimorbidity by chart review (Gold standard)

There was a mean number of 5.4 Elixhauser domains identified in each patient with HF (SD 2.1; range: minimum 1 to maximum 10; Table 1). The mean Elixhauser readmission and mortality index scores were 18.0 (SD 7.5) and 19.0 (SD 13.2), respectively. The distribution of multimorbidity followed similar patterns when stratified by patient demographics including age group, race, ethnicity, and primary insurer. The top 10 most prevalent domains included: uncomplicated hypertension (90%), obesity (42%), chronic pulmonary disease (38%), deficiency anemia (33%), diabetes with chronic complications (30.5%), cerebrovascular disease (28%), depression (27%), valvular disease (26%), moderate renal failure (25.5%), and peripheral vascular disease (24%; Table 2). Among 37 possible domains, interrater reliability of structured chart review across reviewers was found to be moderate, good, and near perfect in 5, 8, and 18 domains, respectively (PABAK ranged between 0.67 to 1.0; S2 Table). PABAK was indeterminate for 6 domains as perfect alignment in either positive or negative cases results in a denominator of 0 in the calculation of kappa.

Table 2. Diagnostic accuracy of measuring Elixhauser comorbid domains from the EHR-based problem list for the ten most prevalent disease domains.

Prevalence by chart review (gold standard) Sensitivity Specificity PPV NPV
Elixhauser domain 1 % (N) % % % %
(95% CI) (95% CI) (95% CI) (95% CI)
Cerebrovascular disease 28% (56) 57.10% 99.30% 97.00% 85.60%
(43.2%-70.3%) (96.2%-100.0%) (84.2%-99.9%) (79.4%-90.6%)
Chronic pulm. disease 38% (76) 76.30% 99.20% 98.30% 87.20%
(65.2%-85.3%) (95.6%-100.0%) (90.9%-100.0%) (80.6%-92.3%)
Deficiency anemias 33% (66) 69.70% 97.80% 93.90% 86.80%
(57.1%-80.4%) (93.6%-99.5%) (83.1%-98.7%) (80.3%-91.7%)
Depression 27% (54) 51.90% 98.60% 93.30% 84.70%
(37.8%-65.7%) (95.1%-99.8%) (77.9%-99.2%) (78.4%-89.8%)
Diabetes with chronic comp. 30.5% (61) 85.20% 97.80% 94.50% 93.80%
(73.8%-93.0%) (93.8%-99.6%) (84.9%-98.9%) (88.5%-97.1%)
Moderate renal failure 25.5% (51) 68.60% 98.70% 94.60% 90.20%
(54.1%-80.9%) (95.2%-99.8%) (81.8%-99.3%) (84.5%-94.3%)
Obesity 42% (84) 28.60% 97.40% 88.90% 65.30%
(19.2%-39.5%) (92.6%-99.5%) (70.8%-97.6%) (57.7%-72.4%)
Peripheral vascular disease 24% (48) 77.10% 96.70% 88.10% 93.00%
(62.7%-88.0%) (92.5%-98.9%) (74.4%-96.0%) (87.9%-96.5%)
Uncomplicated hypertension 90% (180) 77.80% 100.00% 100.00% 33.30%
(71.0%-83.6%) (83.2%-100.0%) (97.4%-100.0%) (21.7%-46.7%)
Valvular disease 26% (52) 80.80% 98.60% 95.50% 93.60%
(67.5%-90.4%) (95.2%-99.8%) (84.5%-99.4%) (88.5%-96.9%)
Open in a new tab

EHR = electronic health record; PPV = positive predictive value; NPV = negative predictive value

Accuracy of EHR problem list

Extraction of multimorbidity data using ICD-10-CM diagnostics codes from the problem list identified a mean of 4.1 Elixhauser domains per patient with HF (SD 2.1; range: minimum 0 to maximum 11). The mean Elixhauser readmission and in-hospital mortality index scores of 16.4 (SD 6.7) and 19.5 (SD 11.7), respectively.

The diagnostic accuracy of the EHR problem list as compared to chart review—as measured by sensitivity, specificity, PPV, and NPV—for the ten most prevalent domains is presented in Table 2. Among the top ten most prevalent domains, sensitivity ranged widely from 28.6% to 80.8%. Most conditions were above 75% sensitive with the exceptions of cerebrovascular disease (57.1%), deficiency anemia (69.7%), depression (51%), and obesity (28.6%). Conversely, reported specificities occurred within a narrow range of 96.7% to 100%. Correspondingly, the PPV of the ten most prevalent conditions ranged from 88.1% to 100%. The NPV also varied widely ranging between 33.3% to 93.8%. However, most domains had a NPV above 85% with the exceptions of uncomplicated hypertension (33.3%), obesity (65.3%), and depression (84.7%).

Similar variability (or lack thereof) was seen in the sensitivities and specificities among lower prevalence disease domains (S3 Table). In total, 13 domains had a PPV lower than 90% including: obesity (88.9%), peripheral vascular disease (88.1%), mild liver disease (87.5%), seizure disorder (85.7%), diabetes without chronic complications (81.8%), peptic ulcer disease (75%), paralysis (75%), other neurologic disorders (72.7%), weight loss (69.2%), complicated hypertension (54.5%), psychoses (33.3%), blood loss anemia (28.6%), and cancer-solid tumor without metastasis (28.6%). None of the lower prevalence domains had a NPV below 90%.

Perfect alignment between chart review and the problem list was observed in 35 patients (17.5%; Table 3). The EHR-based problem list correctly identified 3.7 Elixhauser domains per patient. Conversely, there was an average of 2 misclassified (i.e., either false positive or false negative) domains per patient. The average number of misclassified domains was observed to increase among patients with higher multimorbidity (i.e., higher domain count as reported by chart review). In measuring the agreement on Elixhauser domain count between chart review and EHR-problem list, we observed a consistency ICC of 0.77 (Table 4). Similar consistency ICCs were reported in the comparison of Elixhauser readmission (ICC 0.81) and in-hospital mortality index (ICC 0.76). The absolute agreement ICCs was observed to drop to 0.62 in assessment of Elixhauser domain count, but no significant change in the absolute ICC was found for the Elixhauser index on readmission (ICC 0.79) and in-hospital mortality (ICC 0.75). Discrepancies between absolute and consistency ICC suggest the presence of non-negligible bias [33]. The occurrence of bias is further supported by the Bland-Altman analysis on Elixhauser domain count which demonstrated a mean difference of 1.4 domains (95% limits of agreement: -1.4 to 4.1) between chart review and EHR-based problem list measurement. The Bland-Altman analysis further demonstrated that only 6 (3.0%) subjects were outside the 95% limits of agreement (S2 Fig), which is within the expected 5% outside the limit of agreement. Lastly, ROC curve analyses yielded AUCs of 0.89, 0.86, and 0.83 in classifying the Elixhauser domain count, readmission index, and mortality index (S3 Fig).

Table 3. Overall accuracy of the EHR-based problem list stratified by level of multimorbidity reported in chart review.

Elixhauser domain count by chart review (gold standard)
1 2 3 4 5 6 7 8 9 10 Total
Patients (N) 2 13 25 33 31 32 24 24 14 2 200
Proportion of patients with all domains captured by HER problem list (%) 100% 62% 24% 24% 19% 13% 13% 4% 7% 0% 20%
Proportion of patients with perfect agreement between chart review and EHR problem list (%) 100% 54% 24% 24% 13% 13% 13% 0% 7% 0% 18%
Average number of correctly classified domains by EHR-based problem list (%) 1 1.5 1.8 2.4 3.5 4.4 5.2 5.1 6.4 8.5 3.7
Average number of missed domains by EHR-based problem list* 0 0.5 1.2 1.6 1.5 1.6 1.8 2.9 2.6 1.5 1.7
Average number of misclassified domains by EHR-based problem list* 0 0.6 1.6 1.8 2 1.9 2.1 3.3 2.9 2 2
Open in a new tab

*Rate of misclassified domains = false negative rate + false positive rate; Rate of missed domains = false negative rate alone; EHR = Electronic Health Record System

Table 4. Comparison Elixhauser multimorbidity measures by chart review and EHR-based problem list.

Multimorbidity meausrement Comparison of measurements
Chart review (gold standard) EHR-based problem list Mean difference Consistency ICC Absolute ICC
Mean (SD) Mean (SD) ICC (95% CI) ICC (95%CI)
Elixhauser domain count 5.4 (2.1) 4.1 (2.1) 1.3 0.77 (0.71–0.82) 0.62 (0.09–0.82)
Elixhauser readmission rate 18.0 (7.5) 16.4 (6.7) 1.6 0.81 (0.75–0.85) 0.79 (0.70–0.85)
Elixhauser in-hospital mortality index 19.0 (13.2) 19.5 (11.7) -0.5 0.76 (0.69–0.81) 0.75 (0.69–0.81)
Open in a new tab

ICC = intraclass correlation coefficient; CI = confidence interval; SD = standard deviation; EHR = electronic health record

Discussion

This study shows that multimorbidity can be measured by applying the Elixhauser classification system to ICD-10-CM diagnostic codes stored in EHR-based problem lists. The EHR-based problem list performed with moderate-to-good accuracy with positive predictive value and negative predictive value of using the EHR-based problem list greater than 90% in 24/37 and 32/37 disease domains respectively. However, there was a clear bias towards undermeasurement as the problem list failed to identify 1.7 domains per patient. This result was largely driven by poor capture of two high prevalence disease domains including: obesity (NPV 65.3%) and uncomplicated hypertension (NPV 33.3). Despite these issues, the proposed methodology performed well overall as supported by high ICCs, Bland-Altman analysis (which only found 3% of subjects outside the 95% limits of agreement), and high AUCs observed in the ROC curve analyses.

Our findings have important implications for both practicing ED physicians and researchers. Use of the EHR-based problem list is particularly advantageous to ED physicians, whom have little familiarity with the patients they treat and often must make time-sensitive treatment decisions. Moreover, many ED patients (such as those with dementia) cannot self-report a complete medical history; a situation whereby ED physicians are nearly fully dependent on data in the EHR-problem list. The high specificities reported in this study demonstrate that reported diseases on the problem list generally represent true positives and may not require further verification through extended chart review. Regardless, ED physicians should be aware of underreporting of comorbid conditions and incorporate alternative methods—such as patient interview, medication reconciliation—to obtain a complete medical history. Secondly, physicians should be aware of specific patterns in underreporting of high prevalence conditions (i.e., obesity, depression, and hypertension) and seek to augment their documentation on EHR-based problem lists. Similar patterns in poor EHR capture of obesity and hypertension have been described elsewhere in the medical literature [17, 37], though the underlining cause remains unknown.

With regards to the research community, our findings support the use of ICD-10-CM diagnostic codes from EHR-based problem lists to serve as a foundation in describing multimorbidity. The proposed technique offers the inherent advantage of measuring multimorbidity at the point of care prior to any interventions being done in the ED. From a quality measurement perspective (i.e., measuring adherence to evidence-based guidelines), the method clearly delineates multimorbidity data that was available to physicians and advanced practice providers prior to performing any interventions. Furthermore, the described methodology—given its reliance on discrete, EHR-based variables—can readily be incorporated into more complex EHR-based implementation interventions, such as best practice alerts or embedded risk stratification calculators. Such interventions have been shown to augment clinical workflow and improve guideline adherence [38]. Regardless, researchers should be aware of limitations in poor capture of obesity and hypertension. Augmented EHR-phenotypes incorporating both ICD-10-CM codes from the problem list as well extraction of additional discreet data elements—such as documented height & weight, laboratory results, radiographic findings, or ambulatory medications—could be incorporated into algorithms to improve the sensitivity and specificity in detecting the disease domains. Additionally, future algorithms to describe multimorbidity may incorporate advanced machine-learning methods, which can account for both structured data elements in the problem list as well as unstructured data captured in clinical notes.

Limitations

First, we validated use of the EHR-based problem list in a narrow study population, namely older adult patients with HF at a single academic institution. This was consistent with our study hypothesis that patients with HF would have routine healthcare encounters with outpatient physicians, and thus a well-maintained problem list. Regardless, our findings will have to be validated in a broader population of ED patients and various types of hospitals. Secondly, we observed interrater variability in our gold standard measurement with moderate and good agreement in 5 and 7 domains, respectively. The gold standard relied on clinician raters as they can perform scoping chart reviews, though variability among clinicians is common and difficult to remove in highly controlled settings such as randomized trials [39]. Despite these challenges, our raters had near perfect agreement in the majority (18/37) of the domains. Thirdly, the AHRQ Elixhauser is missing some notable disease domains, specifically coronary artery disease and hyperlipidemia, that are relatively important in assessment of multimorbidity among older patients with HF. We believe these domains would be excellent additions to future iterations of the AHRQ comorbidity software. Finally, gold standard measurement of chronic conditions did not rely upon stringent guideline-based disease definitions (e.g., pulmonary function testing to confirm obstructive pulmonary disease). The average chart review was completed in an hour, thus use of stringent definitions for all the 37 disease domains examined was not feasible in this retrospective study. Further, we used information that is readily available to ED providers during health encounters. In this regard, the gold standard implemented in this study pragmatically reflects how physicians accrue multimorbidity data in clinical practice.

Conclusion

Multimorbidity measurement from EHR-based problem lists can be performed with moderate-to-good accuracy among patients with HF in the ED. The methods do have some limitations with clear underreporting of two high prevalent disease domains (uncomplicated hypertension and obesity). Further work is needed to validate these findings in a broader array of ED patients and various types of hospitals.

Supporting information

S1 Table. Comparison of demographics of full population of ED patients with HF population as compared to selected random sample.

HF = Heart Failure.

(PDF)

Click here for additional data file. (41.5KB, pdf)
S2 Table. Reliability in measuring multimorbidity by structured chart review.

*Indeterminate kappa due to complete agreement on either positive cases or negative cases resulting in a 0 in the denominator; PABAK = prevalence-adjusted, bias-adjusted kappa.

(PDF)

Click here for additional data file. (54.3KB, pdf)
S3 Table. Diagnostic accuracy of measuring Elixhauser comorbid domains from the EHR-based problem list for all 37 domains.

EHR = electronic health record; PPV = positive predictive value; NPV = negative predictive value.

(PDF)

Click here for additional data file. (66.3KB, pdf)
S1 Fig. Flow diagram of selection of patients for the study cohort.

(PDF)

Click here for additional data file. (69.8KB, pdf)
S2 Fig. Bland-Altman plots comparing Elixhauser domain count across chart review and EHR-based problem list.

(PDF)

Click here for additional data file. (78.7KB, pdf)
S3 Fig. ROC analysis.

(PDF)

Click here for additional data file. (539.7KB, pdf)
S1 Data

(XLS)

Click here for additional data file. (154.5KB, xls)

Acknowledgments

We appreciate Hongtu Zhu and Huaying Qiu for providing feedback on the statistical approach.

Data Availability

All relevant data are within the paper and its Supporting information files.

Funding Statement

BK received a grant from the ISTEM NHLBI T35 supported by grant number T35-HL134624 from the National Institutes of Health. URL: https://www.niddk.nih.gov/research-funding/process/apply/funding-mechanisms/t35 No - The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

  • 1. Manemann SM, Chamberlain AM, Boyd CM, Gerber Y, Dunlay SM, Weston SA, et al. Multimorbidity in Heart Failure: Effect on Outcomes. J Am Geriatr Soc. 2016;64(7):1469–74. doi: 10.1111/jgs.14206 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. Gimeno-Miguel A, Gracia Gutierrez A, Poblador-Plou B, Coscollar-Santaliestra C, Perez-Calvo JI, Divo MJ, et al. Multimorbidity patterns in patients with heart failure: an observational Spanish study based on electronic health records. BMJ Open. 2019;9(12):e033174. doi: 10.1136/bmjopen-2019-033174 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Chamberlain AM, St Sauver JL, Gerber Y, Manemann SM, Boyd CM, Dunlay SM, et al. Multimorbidity in heart failure: a community perspective. Am J Med. 2015;128(1):38–45. doi: 10.1016/j.amjmed.2014.08.024 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Kennel PJ, Kneifati-Hayek J, Bryan J, Banerjee S, Sobol I, Lachs MS, et al. Prevalence and determinants of Hyperpolypharmacy in adults with heart failure: an observational study from the National Health and Nutrition Examination Survey (NHANES). BMC Cardiovasc Disord. 2019;19(1):76. doi: 10.1186/s12872-019-1058-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Soley-Bori M, Ashworth M, Bisquera A, Dodhia H, Lynch R, Wang Y, et al. Impact of multimorbidity on healthcare costs and utilisation: a systematic review of the UK literature. Br J Gen Pract. 2021;71(702):e39–e46. doi: 10.3399/bjgp20X713897 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Tinetti ME, Bogardus ST Jr., Agostini JV. Potential pitfalls of disease-specific guidelines for patients with multiple conditions. N Engl J Med. 2004;351(27):2870–4. doi: 10.1056/NEJMsb042458 [DOI] [PubMed] [Google Scholar]
  • 7. Piette JD, Kerr EA. The impact of comorbid chronic conditions on diabetes care. Diabetes Care. 2006;29(3):725–31. doi: 10.2337/diacare.29.03.06.dc05-2078 [DOI] [PubMed] [Google Scholar]
  • 8. Turner BJ, Hollenbeak CS, Weiner M, Ten Have T, Tang SS. Effect of unrelated comorbid conditions on hypertension management. Annals of internal medicine. 2008;148(8):578–86. doi: 10.7326/0003-4819-148-8-200804150-00002 [DOI] [PubMed] [Google Scholar]
  • 9. Lagu T, Weiner MG, Hollenbeak CS, Eachus S, Roberts CS, Schwartz JS, et al. The impact of concordant and discordant conditions on the quality of care for hyperlipidemia. J Gen Intern Med. 2008;23(8):1208–13. doi: 10.1007/s11606-008-0647-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. tewart S, Riegel B, Boyd C, Ahamed Y, Thompson DR, Burrell LM, et al. Establishing a pragmatic framework to optimise health outcomes in heart failure and multimorbidity (ARISE-HF): A multidisciplinary position statement. Int J Cardiol. 2016;212:1–10. doi: 10.1016/j.ijcard.2016.03.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Ho IS, Azcoaga-Lorenzo A, Akbari A, Black C, Davies J, Hodgins P, et al. Examining variation in the measurement of multimorbidity in research: a systematic review of 566 studies. Lancet Public Health. 2021;6(8):e587–e97. doi: 10.1016/S2468-2667(21)00107-9 [DOI] [PubMed] [Google Scholar]
  • 12. Harrison C, Britt H, Miller G, Henderson J. Examining different measures of multimorbidity, using a large prospective cross-sectional study in Australian general practice. BMJ Open. 2014;4(7):e004694. doi: 10.1136/bmjopen-2013-004694 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Violan C, Foguet-Boreu Q, Flores-Mateo G, Salisbury C, Blom J, Freitag M, et al. Prevalence, determinants and patterns of multimorbidity in primary care: a systematic review of observational studies. PLoS One. 2014;9(7):e102149. doi: 10.1371/journal.pone.0102149 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Klappe ES, de Keizer NF, Cornet R. Factors Influencing Problem List Use in Electronic Health Records-Application of the Unified Theory of Acceptance and Use of Technology. Appl Clin Inform. 2020;11(3):415–26. doi: 10.1055/s-0040-1712466 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Simons SM, Cillessen FH, Hazelzet JA. Determinants of a successful problem list to support the implementation of the problem-oriented medical record according to recent literature. BMC Med Inform Decis Mak. 2016;16:102. doi: 10.1186/s12911-016-0341-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Wright A, Feblowitz J, Maloney FL, Henkin S, Bates DW. Use of an electronic problem list by primary care providers and specialists. J Gen Intern Med. 2012;27(8):968–73. doi: 10.1007/s11606-012-2033-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Wang EC, Wright A. Characterizing outpatient problem list completeness and duplications in the electronic health record. J Am Med Inform Assoc. 2020;27(8):1190–7. doi: 10.1093/jamia/ocaa125 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Wright A, Pang J, Feblowitz JC, Maloney FL, Wilcox AR, McLoughlin KS, et al. Improving completeness of electronic problem lists through clinical decision support: a randomized, controlled trial. J Am Med Inform Assoc. 2012;19(4):555–61 doi: 10.1136/amiajnl-2011-000521 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19. Wright A, McCoy AB, Hickman TT, Hilaire DS, Borbolla D, Bowes WA 3rd, et al. Problem list completeness in electronic health records: A multi-site study and assessment of success factors. Int J Med Inform. 2015;84(10):784–90. doi: 10.1016/j.ijmedinf.2015.06.011 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Li RC, Garg T, Cun T, Shieh L, Krishnan G, Fang D, et al. Impact of problem-based charting on the utilization and accuracy of the electronic problem list. J Am Med Inform Assoc. 2018;25(5):548–54. doi: 10.1093/jamia/ocx154 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. Nair A, Dreyfus D. Technology alignment in the presence of regulatory changes: The case of meaningful use of information technology in healthcare. Int J Med Inform. 2018;110:42–51. doi: 10.1016/j.ijmedinf.2017.11.012 [DOI] [PubMed] [Google Scholar]
  • 22.HealthIT.gov. Health IT Policy Committee: Recommendations to the National Coordinator for Health IT 2015. cited 2021 Oct 19.
  • 23. Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care. 1998;36(1):8–27. doi: 10.1097/00005650-199801000-00004 [DOI] [PubMed] [Google Scholar]
  • 24. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. doi: 10.1136/bmj.h5527 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Agency for Healthcare Quality and Research. Elixhauser Comorbidity Software, Version 3.7. cited 2021 Jan 11.
  • 26. Tison GH, Chamberlain AM, Pletcher MJ, Dunlay SM, Weston SA, Killian JM, et al. Identifying heart failure using EMR-based algorithms. Int J Med Inform. 2018;120:1–7. doi: 10.1016/j.ijmedinf.2018.09.016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27. Blecker S, Katz SD, Horwitz LI, Kuperman G, Park H, Gold A, et al. Comparison of Approaches for Heart Failure Case Identification From Electronic Health Record Data. JAMA Cardiol. 2016;1(9):1014–20. doi: 10.1001/jamacardio.2016.3236 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.R Core Team 2022. cited 2022.
  • 29. Diederichs C, Berger K, Bartels DB. The measurement of multiple chronic diseases–a systematic review on existing multimorbidity indices. J Gerontol A Biol Sci Med Sci. 2011;66(3):301–11. doi: 10.1093/gerona/glq208 [DOI] [PubMed] [Google Scholar]
  • 30. Johnston MC, Crilly M, Black C, Prescott GJ, Mercer SW. Defining and measuring multimorbidity: a systematic review of systematic reviews. Eur J Public Health. 2019;29(1):182–9. doi: 10.1093/eurpub/cky098 [DOI] [PubMed] [Google Scholar]
  • 31. Simard M, Sirois C, Candas B. Validation of the Combined Comorbidity Index of Charlson and Elixhauser to Predict 30-Day Mortality Across ICD-9 and ICD-10. Med Care. 2018;56(5):441–7. doi: 10.1097/MLR.0000000000000905 [DOI] [PubMed] [Google Scholar]
  • 32. Moore BJ, White S, Washington R, Coenen N, Elixhauser A. Identifying Increased Risk of Readmission and In-hospital Mortality Using Hospital Administrative Data: The AHRQ Elixhauser Comorbidity Index. Med Care. 2017;55(7):698–705. doi: 10.1097/MLR.0000000000000735 [DOI] [PubMed] [Google Scholar]
  • 33. Liljequist D, Elfving B, Skavberg Roaldsen K. Intraclass correlation—A discussion and demonstration of basic features. PLoS One. 2019;14(7):e0219854. doi: 10.1371/journal.pone.0219854 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2):155–63. doi: 10.1016/j.jcm.2016.02.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276–82. doi: 10.11613/BM.2012.031 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8(2):135–60. doi: 10.1177/096228029900800204 [DOI] [PubMed] [Google Scholar]
  • 37. Mattar A, Carlston D, Sariol G, Yu T, Almustafa A, Melton GB, et al. The prevalence of obesity documentation in Primary Care Electronic Medical Records. Are we acknowledging the problem?. Appl Clin Inform. 2017;8(1):67–79. doi: 10.4338/ACI-2016-07-RA-0115 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38. Shanafelt TD, Dyrbye LN, Sinsky C, Hasan O, Satele D, Sloan J, et al. Relationship Between Clerical Burden and Characteristics of the Electronic Environment With Physician Burnout and Professional Satisfaction. Mayo Clin Proc. 2016;91(7):836–48. doi: 10.1016/j.mayocp.2016.05.007 [DOI] [PubMed] [Google Scholar]
  • 39. Miskulin DC, Athienites NV, Yan G, Martin AA, Ornt DB, Kusek JW, et al. Comorbidity assessment using the Index of Coexistent Diseases in a multicenter clinical trial. Kidney Int. 2001;60(4):1498–510. doi: 10.1046/j.1523-1755.2001.00954.x [DOI] [PubMed] [Google Scholar]

Decision Letter 0

Fabrizio Pecoraro

Transfer Alert

This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present.

13 May 2022

PONE-D-22-09414Accuracy of the Electronic Health Record’s Problem List in Describing Multimorbidity in Patients with Heart Failure in the Emergency DepartmentPLOS ONE

Dear Dr. Casey,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Jun 27 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Fabrizio Pecoraro

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified (1) whether consent was informed and (2) what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information.

If you are reporting a retrospective study of medical records or archived samples, please ensure that you have discussed whether all data were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. If patients provided informed written consent to have data from their medical records used in research, please include this information.

3. Please update your submission to use the PLOS LaTeX template. The template and more information on our requirements for LaTeX submissions can be found at http://journals.plos.org/plosone/s/latex.

4. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. 

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

5. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Casey et al. present a study to evaluate a multimorbidity ‘problem list’ via electronic health records (EHRs) for patients with heart failure (HF) admitted in emergency departments (ED). The population sample was drawn from patients admitted in the ED of a southeastern US academic medical center. The inclusion criteria were determined on the basis of ICD-10 and demographics. An problem list across multiple disease domains was defined and multimorbidity was measured as gold standard via a 4-steps thorough chart review (plus additional materials if needed, e.g., echocardiography reports) performed by two trained persons and further checked by MD and nurse practitioner. The AHRQ Elixhauser (ICD-10) comorbidity score was calculated by manually reviewing ICD-10 codes. Domain based, and overall concordance indices for the cart review were calculated (i.e., PABAK, ICC), and then PPV and NPV of the gold standard with Elixhauser scores (three summary indices).

Results show a large range of variability in both the concordance indices and the diagnostic accuracy when stratifying by domain. In particular, there was poor capture of obesity and hypertension. The authors conclude that the diagnostic accuracy of the EHR-based multimorbidity problem list for patients with HF in ED.

Overall, the manuscript is well-written and well-structured, with balance among the presented sections.

However, there are concerns about the study rationale/premise, and robustness of analysis for which the conclusive claims or the clinical relevance might be hampered, diminished.

Methodological issues:

1. Study sample. First, ED/HF patients would possibly present as a selected population with comorbidity bias in one or more domains; the authors acknowledge that in part, but it may be a major driver. Second, the random sample was 200 over 1,130 eligible patients in the EHR data extraction. Even if chart review might have been too cumbersome for the whole sample, at least comparison with demographics and other descriptive statistics should have been made.

2. Most of EHR systems nowadays provide automated calculation of comorbidity index, usually the Charlson’s. The way in which the calculation is performed can vary, e.g., the time of lookback for finding a chronic condition, or the number of times a condition is diagnosed to confirm, and can be associated to a measurement error, population variance. The Elixhauser calculation presented here might have affected in the same way. The authors should have tried to consider a fully automated score provided by the EHR system.

3. The moderate PABAK and ICC in certain domains also indicates is substantial variability in clinical determinations, and this happens in parallel with the Elixhauser score determination, likely affecting the resulting diagnostic accuracy.

4. Given the continuous scales, a threshold analysis could have been useful (e.g., ROCs).

Clinical significance issues:

The work --as is-- stands as a mere exercise to compare imperfect, resource-intensive measures. The authors acknowledge the limitations. Yet, possibly, a predictive-based development based on these preliminary results could provide the translational relevance that does not emerge here.

Reviewer #2: Thank you for a neat study. This would be improved by including:

1. The means of random chart selection.

2. Assessment of key demographics (which ought to be readily suitable for automated statistical count) between the sample and the full non-excluded population - such as age, gender, ethnicity. This would confirm that the sample was representative in practice.

3. Discussion of the hypothetical savings which could be obtained in ED time if applied as a routine practice - e.g. saved time per patient; patients per day.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Professor Emeritus Michael Rigby

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2022 Dec 13;17(12):e0279033. doi: 10.1371/journal.pone.0279033.r002

Author response to Decision Letter 0

5 Sep 2022

Reviewer #1: Casey et al. present a study to evaluate a multimorbidity ‘problem list’ via electronic health

records (EHRs) for patients with heart failure (HF) admitted in emergency departments (ED). The

population sample was drawn from patients admitted in the ED of a southeastern US academic medical

center. The inclusion criteria were determined on the basis of ICD-10 and demographics. An problem list

across multiple disease domains was defined and multimorbidity was measured as gold standard via a 4-

steps thorough chart review (plus additional materials if needed, e.g., echocardiography reports)

performed by two trained persons and further checked by MD and nurse practitioner. The AHRQ

Elixhauser (ICD-10) comorbidity score was calculated by manually reviewing ICD-10 codes. Domain

based, and overall concordance indices for the cart review were calculated (i.e., PABAK, ICC), and then

PPV and NPV of the gold standard with Elixhauser scores (three summary indices).

Results show a large range of variability in both the concordance indices and the diagnostic accuracy

when stratifying by domain. In particular, there was poor capture of obesity and hypertension. The

authors conclude that the diagnostic accuracy of the EHR-based multimorbidity problem list for patients

with HF in ED.

Overall, the manuscript is well-written and well-structured, with balance among the presented sections.

However, there are concerns about the study rationale/premise, and robustness of analysis for which

the conclusive claims or the clinical relevance might be hampered, diminished.

AUTHORS Response: Thanks for the positive feedback on our writing. We recognize the limitations of the

proposed methodology, specifically that 1.7 domains were not identified per patient when using the

EHR-based problem list. Although the proposed methodology for measuring multimorbidity has clear

limitations, the method does form the basis of a promising foundation which can be built upon by

extracting additional data elements (e.g., lab values, radiographic findings) to improve overall

accuracy. We have updated the manuscript text to acknowledge this feedback (see page 9-10), which

we very much agree with.

Methodological issues:

1. Study sample. First, ED/HF patients would possibly present as a selected population with comorbidity

bias in one or more domains; the authors acknowledge that in part, but it may be a major driver.

Second, the random sample was 200 over 1,130 eligible patients in the EHR data extraction. Even if

chart review might have been too cumbersome for the whole sample, at least comparison with

demographics and other descriptive statistics should have been made.

AUTHORS Response: We have included a new supplemental table that compares the demographics of the 1,130

eligible patients to the 200 patients randomly selected for chart review. We believe the values are

comparable across the two groups (reflecting that random selection of charts for review was

performed correctly; see Supplemental Table 1).

2. Most of EHR systems nowadays provide automated calculation of comorbidity index, usually the

Charlson’s. The way in which the calculation is performed can vary, e.g., the time of lookback for finding

a chronic condition, or the number of times a condition is diagnosed to confirm, and can be associated

to a measurement error, population variance. The Elixhauser calculation presented here might have

affected in the same way. The authors should have tried to consider a fully automated score provided by

the EHR system.

AUTHORS Response: We have investigated whether our healthcare system’s EHR includes any measures of

comorbidity such the Elixhauser or Charlson index. Unfortunately, our current EHR build does not

include any measures of comorbidity. Thank you for this suggestion.

3. The moderate PABAK and ICC in certain domains also indicates is substantial variability in clinical

determinations, and this happens in parallel with the Elixhauser score determination, likely affecting the

resulting diagnostic accuracy.

AUTHORS Response: We included a discussion on this topic in our limitation section (see page 10). Thank you for

this suggestion.

4. Given the continuous scales, a threshold analysis could have been useful (e.g., ROCs).

AUTHORS: We maximized study power by using continuous scales, rather than thresholds.

Unfortunately, the idea of multimorbidity measurement as a standalone measurement is still

relatively nascent. We are unaware of any consensus agreements on meaningful thresholds that could

be used to guide an ROC analysis comparing domain count and Elixhauser index scores. We do

appreciate this suggestion, but do not see viable approach to implementing a meaningful ROC

analysis.

Reviewer #2

Clinical significance issues:

The work --as is-- stands as a mere exercise to compare imperfect, resource-intensive measures. The

authors acknowledge the limitations. Yet, possibly, a predictive-based development based on these

preliminary results could provide the translational relevance that does not emerge here.

AUTHORS Response: See above. We have expanded some language in our manuscript to highlight use of

additional structured data elements (such as, laboratory values, radiographic findings) and advanced

methods (such as machine-line) which incorporate unstructured data in capturing the burden of

multimorbidity (see page 10).

Reviewer #2: Thank you for a neat study. This would be improved by including:

1. The means of random chart selection.

AUTHORS Response: We have a included a description and citation on the random sequence generator used to

select charts in our methods. Thank you for this suggestion.

2. Assessment of key demographics (which ought to be readily suitable for automated statistical count)

between the sample and the full non-excluded population - such as age, gender, ethnicity. This would

confirm that the sample was representative in practice.

AUTHORS Response: See above commentary on inclusion of new Supplemental Table 1.

3. Discussion of the hypothetical savings which could be obtained in ED time if applied as a routine

practice - e.g. saved time per patient; patients per day.

AUTHORS Response: We have expanded on the implications for ED physicians (see page 9), though with the

caveats that the EHR-problem list is not a perfect technique. We are not aware of any studies on the

exact time ED physicians spend obtaining medical history per patient, and thus are unable to quantify

a meaningful hypothetical savings. We do appreciate this suggestion.

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file

naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliation

s.pdf

AUTHORS Response: We have reviewed the guidelines and updated the body of text accordingly.

2. Please provide additional details regarding participant consent. In the ethics statement in the

Methods and online submission information, please ensure that you have specified (1) whether consent

was informed and (2) what type you obtained (for instance, written or verbal, and if verbal, how it was

documented and witnessed). If your study included minors, state whether you obtained consent from

parents or guardians. If the need for consent was waived by the ethics committee, please include this

information.

If you are reporting a retrospective study of medical records or archived samples, please ensure that you

have discussed whether all data were fully anonymized before you accessed them and/or whether the

IRB or ethics committee waived the requirement for informed consent. If patients provided informed

written consent to have data from their medical records used in research, please include this

information.

AUTHORS Response: We have updated this section appropriately.

3. Please update your submission to use the PLOS LaTeX template. The template and more information

on our requirements for LaTeX submissions can be found at http://journals.plos.org/plosone/s/latex.

AUTHORS Response: We have reviewed the guidelines and updated the body of text accordingly.

4. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial

Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you

received for your study in the ‘Funding Information’ section.

AUTHORS Response: We have updated this section appropriately.

5. In your Data Availability statement, you have not specified where the minimal data set underlying the

results described in your manuscript can be found. PLOS defines a study's minimal data set as the

underlying data used to reach the conclusions drawn in the manuscript and any additional data required

to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data

set be made fully available. For more information about our data policy, please see

http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as

either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs,

or accession numbers within your revised cover letter. For a list of acceptable repositories, please see

http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially

identifying patient information must be fully anonymized.

AUTHORS Response: We have created a deidentified dataset to be uploaded that will be uploaded to our

supplement in accord with PlosOne’s policies. Some variables such as gender and race were removed

due to internal policies to ensure protection of private health information. These variables can be

made available upon author request.

Attachment

Submitted filename: Response to reviewers.pdf

Click here for additional data file. (124KB, pdf)

Decision Letter 1

Fabrizio Pecoraro

28 Sep 2022

PONE-D-22-09414R1Accuracy of the Electronic Health Record’s problem list in describing multimorbidity in patients with heart failure in the emergency departmentPLOS ONE

Dear Dr. Casey,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. As reported by Reviewer 1, some additional efforts should be made to investigate some of the issues previously highlighted by him/her, in particular with the domain determination (e.g., with a design change to assess robustness) and with the diagnostic accuracy. I suggest to update the paper on the basis of the Reviewer 1's comments before resubmit it to PLOS ONE. 

Please submit your revised manuscript by Nov 12 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Fabrizio Pecoraro

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have provided reasonable responses to my comments. However, rather than addressing the limitations of their paper, they have merely acknowledged them. I think that some additional effort should be made to investigate some of the issues previously highlighted, in particular with the domain determination (e.g., with a design change to assess robustness) and with the diagnostic accuracy. Computational phenotyping is an iterative process, but at the moment the paper still falls shorts of its premise and objective. Nonetheless, if the authors elaborate more on the domain and diagnostic performance, the manuscript could be published.

Reviewer #2: Thank you for addressing previous reviewer feedback, and providing a suitably improved and clearer manuscript.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2022 Dec 13;17(12):e0279033. doi: 10.1371/journal.pone.0279033.r004

Author response to Decision Letter 1

2 Nov 2022

As reported by Reviewer 1, some additional efforts should be made to investigate some of the issues

previously highlighted by him/her, in particular with the domain determination (e.g., with a design change

to assess robustness) and with the diagnostic accuracy. I suggest to update the paper on the basis of the

Reviewer 1's comments before resubmit it to PLOS ONE.

AUTHORS: We have incorporated a ROC curve analyses into our manuscript (pg 6, 9). Thank you

for this suggestion as we believe it has made our manuscript more complete.

Attachment

Submitted filename: Response to reviewers.pdf

Click here for additional data file. (124KB, pdf)

Decision Letter 2

Fabrizio Pecoraro

29 Nov 2022

Accuracy of the Electronic Health Record’s problem list in describing multimorbidity in patients with heart failure in the emergency department

PONE-D-22-09414R2

Dear Dr. Casey,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Fabrizio Pecoraro

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: No additional comments. No additional comments. No additional comments. No additional comments. No additional comments.

Reviewer #2: The additional material enhances an already acceptable paper. Thank you for your careful considerations.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

Acceptance letter

Fabrizio Pecoraro

2 Dec 2022

PONE-D-22-09414R2

Accuracy of the electronic health record’s problem list in describing multimorbidity in patients with heart failure in the emergency department

Dear Dr. Casey:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Fabrizio Pecoraro

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Table. Comparison of demographics of full population of ED patients with HF population as compared to selected random sample.

    HF = Heart Failure.

    (PDF)

    Click here for additional data file. (41.5KB, pdf)
    S2 Table. Reliability in measuring multimorbidity by structured chart review.

    *Indeterminate kappa due to complete agreement on either positive cases or negative cases resulting in a 0 in the denominator; PABAK = prevalence-adjusted, bias-adjusted kappa.

    (PDF)

    Click here for additional data file. (54.3KB, pdf)
    S3 Table. Diagnostic accuracy of measuring Elixhauser comorbid domains from the EHR-based problem list for all 37 domains.

    EHR = electronic health record; PPV = positive predictive value; NPV = negative predictive value.

    (PDF)

    Click here for additional data file. (66.3KB, pdf)
    S1 Fig. Flow diagram of selection of patients for the study cohort.

    (PDF)

    Click here for additional data file. (69.8KB, pdf)
    S2 Fig. Bland-Altman plots comparing Elixhauser domain count across chart review and EHR-based problem list.

    (PDF)

    Click here for additional data file. (78.7KB, pdf)
    S3 Fig. ROC analysis.

    (PDF)

    Click here for additional data file. (539.7KB, pdf)
    S1 Data

    (XLS)

    Click here for additional data file. (154.5KB, xls)
    Attachment

    Submitted filename: Response to reviewers.pdf

    Click here for additional data file. (124KB, pdf)
    Attachment

    Submitted filename: Response to reviewers.pdf

    Click here for additional data file. (124KB, pdf)

    Data Availability Statement

    All relevant data are within the paper and its Supporting information files.

    Articles from PLOS ONE are provided here courtesy of PLOS

    RESOURCES