Abstract
Background
Interventions to reduce readmissions following acute heart failure hospitalization require early identification of patients. The purpose of this study was to develop and test accuracies of various approaches to identify patients with acute decompensated heart failure (ADHF) using data derived from the electronic health record.
Methods and Results
We included 37,229 hospitalizations of adult patients at a single hospital in 2013–2015. We developed four algorithms to identify hospitalization with a principal discharge diagnosis of ADHF: 1) presence of one of three clinical characteristics; 2) logistic regression of 31 structured data elements; 3) machine learning with unstructured data; 4) machine learning with both structured and unstructured data. In data validation, Algorithm 1 had a sensitivity of 0.98 and positive predictive value (PPV) of 0.14 for ADHF. Algorithm 2 had an area under the receiver operating characteristic curve (AUC) of 0.96, while both machine learning algorithms had AUCs of 0.99. Based on a brief survey of three providers who perform chart review for ADHF, we estimated providers spent 8.6 minutes per chart review; using this this parameter, we estimated providers would spend 61.4, 57.3, 28.7, and 25.3 minutes on secondary chart review for each case of ADHF if initial screening was done with algorithms 1, 2, 3, and 4, respectively.
Conclusion
Machine learning algorithms with unstructured notes had best performance for identification of ADHF and can improve provider efficiency for delivery of quality improvement interventions.
Keywords: phenotype, electronic health record, heart failure, hospitalization
Acute decompensated heart failure (ADHF) is among the most common reason for hospitalizations among older adults in the United States.1 Hospitalizations for heart failure are associated with high rates of readmission, many of which may be preventable.2 As a result, initiatives such as Medicare’s Hospital Readmissions Reduction Program have focused on decreasing the number of readmissions following a hospitalization with a principal discharge diagnosis of heart failure.3 Hospitals have responded to these policies by targeting patients hospitalized for ADHF with inpatient interventions including medicine reconciliation, patient and family education, heart failure order sets or protocols, involvement of multidisciplinary teams, and scheduling outpatient follow up prior to discharge.4–6 Many of these intervention target patients early during hospitalization.
In order to target patients hospitalized for ADHF, a rapid method is needed identify them during hospitalization. Although most assessments of quality of care or readmission rates related to heart failure have relied on identification using discharge diagnosis codes,7,8 these codes are only documented after the patient is discharged. A multidisciplinary approach to prevention of readmission requires early identification of patients with ADHF. Indeed, one recent study suggested that a care plan intervention coupled with use of natural language processing (NLP) to identify hospitalized heart failure patients may lead to improvement in post-discharge outcomes.9 However, there have been limited evaluations of the comparative advantage of advanced approaches to identify patients hospitalized for ADHF with more conventional methods based on important clinical factors that have also been shown to improve provider efficiency.10
We recently developed a series of algorithms to identify presence of chronic heart failure during hospitalization.11 We found that algorithms derived from analysis of free text from clinical notes had best performance and could be used for quality improvement efforts such as problem list enhancement. However, more targeted algorithms are needed to guide expensive, resource intensive interventions to identify patients hospitalized for acute decompensated heart failure. Our goal was to develop and compare algorithms of increasing complexity to identify hospitalizations with a principal discharge diagnosis of heart failure. Given the emphasis that hospitals currently place on patients with ADHF, we focused developing models with high sensitivity to avoid missed opportunities for care improvement; this approach assumed that secondary chart review by providers may be necessary to confirm a diagnosis in clinical practice. To determine the potential benefit of each algorithm in hospital delivery, we estimated the time needed for secondary review by providers to confirm the hospitalization was for ADHF following initial screening with each algorithm.
Methods
We performed a retrospective study of hospitalizations at Tisch Hospital, the primary acute care hospital at NYU Langone Medical Center, using data obtained from the electronic health record (EHR, Epic, Epic Systems, Verona, WI). We included all hospitalizations for patients ≥18 years admitted on or after January 1, 2013 and discharged by February 28, 2015. We excluded hospitalizations that were less than 24 hours. The cohort was similar to the one used in developing algorithms to identify patients with chronic heart failure,11 although we did not include patients hospitalized at the Hospital for Joint Diseases in the current study and these patients were included in the prior study. Additionally, in the current study we developed new algorithms to identify patients with ADHF while algorithms in the prior study11 were developed to identify all hospitalized patients with chronic heart failure.
We randomly divided our dataset into 75% model development and 25% validation sets. The primary dependent variable was ADHF defined by a principal discharge diagnosis using standard International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes (402.01, 402.11, 402.91, 404.01, 404.03, 404.11, 404.13, 404.91, 404.93, and 4288,12).
Potential structured predictor variables included demographics, laboratory results, vital signs, problem lists diagnoses, and heart failure related medications. For laboratory results and vital signs, we included an indicator of presence or absence of results and the value. We also included an indicator of the presence of an echocardiogram but did not include specific results including ejection fraction (EF) which were reported in note form. Problem list diagnoses were those that were an active problem on the EHR problem list at the second night of hospitalization and included heart failure, acute myocardial infarction, and atherosclerosis; problem list diagnoses need not be related to the primary reason for hospitalization. We also included variables of a prior discharge diagnosis of heart failure, both as a principal discharge diagnosis and as a secondary diagnosis. Medications included both inpatient and active outpatient therapies for a loop diuretic, an angiotensin converting enzyme (ACE) inhibitor or angiotensin receptor blocker (ARB), a beta-blocker, an evidence-based heart failure beta-blocker, and an aldosterone antagonist. We used unstructured data from echocardiogram reports, chest imaging reports, and admission, physician progress, and consult notes. We included variables up to the second midnight of hospitalization; this time frame was chosen as we wanted to identify cases early during hospitalization and a stay of two midnights is generally considered the minimum time necessary to warrant a hospitalization.13
We developed four algorithms for identification of a principal discharge diagnosis of heart failure at the second midnight of hospitalization. The first algorithm was the presence of one or more of the three following characteristics: heart failure on the problem list; inpatient loop diuretic use, or BNP≥500 pg/ml. This algorithm was based on a screening tool currently used by the heart failure transitions team at our hospital. The second algorithm used logistic regression using structured variables thought to be clinically relevant by two clinicians with expertise in heart failure (SB and SDK). The third algorithm used a machine learning approach with the unstructured data. The fourth algorithm used a machine learning approach and combined both structured and unstructured data elements for patient classification.
To understand the potential use of the algorithms in clinical practice, we calculated the number of hospitalizations that would be identified as positive by each algorithm for each true positive case of ADHF. We then estimated average time needed to perform secondary screening of positive charts (i.e. both true positives and false positives) identified by each algorithm for each true positive case.
We performed a brief survey of nurse practitioner (NP) and physician assistant (PA) providers at our hospital to estimate the time needed for chart review. We approached all three providers from heart failure transitions team who were known to have performed EHR chart review for ADHF. All three providers agreed to be surveyed and verbal consent was obtained. Providers were asked to respond to the following questions based on recall of usual work in clinical practice: 1) the average time needed to review a new chart to determine whether the patient had ADHF, and 2) the average time needed to review all charts for ADHF on days in which they were reviewing charts. As we found a discrepancy between reported average review time per chart (first question) and the average review time per day (second question), we reconciled these by deriving review time per chart based on the second question. We then estimated the parameter for time of chart review as the mean of this value and reported average review time per chart.
Statistical Analysis
We calculated the mean number of new hospitalizations per day for ADHF during the study period by dividing the total number of hospitalizations by the total number of days in the study period.
Classification algorithms were developed in the development set using a principal heart failure discharge diagnosis as the dependent variable. We used a logistic regression model with structured data elements as the independent variables for algorithm 2. For algorithm 3, we developed a machine learning algorithm using L1-regularization logistic regression using free text. In this approach, we identified all individual words that occurred ten times or more in notes or reports. Of the 36,463 word that satisfied this criterion, each individual word was considered a potential variable for the model. L1-regularization, which includes a penalty term to reduce overfitting, used variable selection to select individual words for the final model.14 Algorithm 4 was developed using L1-regularization logistic regression in consideration of both free text used in algorithm 3 (i.e. 36,463 words) and all data elements (i.e. 31 structured variables) used in algorithm 2.
We calculated sensitivity and PPV for each algorithm in the development set and then validation set using the principal discharge diagnosis of heart failure as the gold standard. As our goal was to minimize false negatives, we set our thresholds above which a hospitalization was classified as for heart failure based on a sensitivity of 0.98 in the development set and again in the validation set. These thresholds only applied to algorithms 2–4, which provide a continuous-valued prediction; we also calculated area under the receiver operative characteristic (AUC) curve for these three algorithms as well as specificity and negative predictive value (NPV) for all algorithms.
We determined the number of hospitalizations that would be identified as positive by each algorithm–thus necessitating secondary chart review–for each true positive case of heart failure. We calculated this as the sum of the true positives (TP) and false positives (FP) for each algorithm in the validation set. As PPV=TP/(TP+FP) and we were interested in total positive charts per TP, we calculated the sum of TP and FP for each algorithm to be 1/PPV.
We estimated time needed for secondary chart review for each true positive (TP) of ADHF with initial screening with each algorithm. First, we surveyed NP and PA providers about review time per chart and review time per day; for providers who responded with a range of times, we used the mean of the range. Second, we calculated a mean reported review time per chart and a mean reported review time per day. Third, we estimated the “derived review time per chart based on review time per day” by dividing mean reported review time per day by the average number of charts needing review per day. To estimate the number of charts per day, i.e. the denominator of the “derived review time per chart based on review time per day,” we multiplied the total positives per TP for algorithm 1 by the average daily number of ADHF cases during the study period; algorithm 1 was used for initial heart failure screening at our hospital. Fourth, we averaged the “derived review time per chart based on review time per day” and the mean reported review time per chart to obtain the parameter of review time per chart. Fifth, we multiplied this parameter by the number of hospitalizations identified as positive for each TP with each algorithm to estimate the time needed for secondary chart review for each TP with each algorithm.
Results
Of 37,229 hospitalizations included in the study, 1,294 (3.5%) carried a principal discharge diagnosis of heart failure (Table 1). Patients hospitalized for heart failure were of older age (76.2 vs. 60.8 years), were more likely to be black/African-American race (14.3% vs. 7.7%), and were more likely to be Medicaid beneficiaries than patients hospitalized with another principal discharge diagnosis. Among patients hospitalized for heart failure, 69.9% had heart failure on the problem list, 91.3% had a prior echocardiography, 79.8% had an inpatient diuretic, and 91.9% had a BNP checked by two midnights of hospitalization (Table 1). There were a total of 2,346,779 notes or imaging reports included in the study, with a mean of 123.9 for patients with a principal discharge diagnosis of heart failure and 60.8 for other patients. We calculated a mean of 1.6 new hospitalizations for acute heart failure per day during the study period.
Table 1.
Characteristics of 37,229 hospitalizations, by principal discharge diagnosis
| Characteristic | Principal Diagnosis of Heart Failure (n=1,294) | All other Hospitalizations (n=35,935) |
|---|---|---|
| Age, Mean (SE) | 76.2 (13.0) | 60.8 (19.3) |
| Female | 44.4 | 49.3 |
| Black/African American Race | 14.3 | 10.2 |
| Hispanic/Latino Ethnicity | 7.7 | 7.5 |
| Medicaid | 26.8 | 20.5 |
| Heart failure in problem list | 69.9 | 8.0 |
| Prior diagnosis of any heart failure | 40.3 | 7.3 |
| Prior diagnosis of principal heart failure | 23.7 | 2.0 |
| Prior echocardiography | 91.3 | 39.3 |
| Inpatient diuretics | 79.8 | 13.4 |
| Outpatient diuretics | 74.5 | 16.5 |
| Inpatient ACE inhibitors or ARB | 35.1 | 22.8 |
| Outpatient ACE inhibitors or ARB | 48.2 | 31.3 |
| Inpatient beta blockers | 58.9 | 38.2 |
| Outpatient beta blockers | 73.4 | 35.1 |
| Inpatient heart failure beta blockers | 41.3 | 11.5 |
| Outpatient heart failure beta blockers | 54.9 | 18.0 |
| Inpatient aldosterone antagonist | 17.2 | 2.3 |
| Outpatient aldosterone antagonist | 22.0 | 4.5 |
| Systolic blood pressure, Mean (SE) | 123.0 (21.1) | 123.5 (18.7) |
| Diastolic blood pressure, Mean (SE) | 65.3 (13.8) | 68.3 (13.2) |
| Creatinine, Mean (SE) | 1.8 (1.5) | 1.1 (1.2) |
| Sodium, Mean (SE) | 138.2 (4.4) | 138.2 (3.9) |
| BNP | ||
| <500 | 3.4 | 24.3 |
| 500–999 | 5.9 | 12.7 |
| 1000–4999 | 34.2 | 33.9 |
| 5000–9999 | 20.4 | 12.2 |
| 10000–19999 | 18.4 | 7.8 |
| ≥20000 | 17.7 | 9.1 |
| Any Systolic blood pressure | 99.7 | 99.7 |
| Any diastolic blood pressure | 99.7 | 99.7 |
| Any creatinine | 99.9 | 98.8 |
| Any sodium | 99.9 | 98.8 |
| Any BNP | 91.9 | 17.6 |
| Acute MI in problem list | 6.8 | 2.5 |
| Atherosclerosis in problem list | 35.2 | 14.1 |
| Discharge diagnosis of heart failure (primary or secondary) | 100.0 | 15.1 |
| Principal discharge diagnosis of heart failure | 100.0 | 0.0 |
Values are percent unless otherwise noted. ACE=angiotensin converting enzyme, ARB= angiotensin receptor blocker, BNP=B-type natiuretic peptide , MI=myocardial infarction
The first algorithm for identification of hospitalizations for ADHF, defined as the presence of one of three clinical characteristics (heart failure on the problem list, an inpatient diuretic, or BNP≥500), was associated with a sensitivity of 0.98 and PPVs of 0.13 and 0.14 in the development and validation sets, respectively (Table 2).
Table 2.
Performance characteristics of four algorithms for classification of acute decompensated heart failure
| Development Set | Validation Set | ||||||
|---|---|---|---|---|---|---|---|
| Algorithm | Description | AUC | Sensitivity | PPV | AUC | Sensitivity | PPV |
| 1 | Presence of ≥1 clinical characteristic* | 0.98 | 0.13 | 0.98 | 0.14 | ||
| (0.97–0.99) | (0.13–0.14) | (0.96–0.99) | (0.12–0.15) | ||||
| 2 | Logistic regression with structured data | 0.96 | 0.98 | 0.14 | 0.96 | 0.98 | 0.15 |
| (0.96–0.96) | (0.97–0.99) | (0.14–0.15) | (0.95–0.97) | (0.97–0.99) | (0.13–0.16) | ||
| 3 | Machine learning using notes and imaging reports | 0.99 | 0.98 | 0.42 | 0.99 | 0.98 | 0.30 |
| (0.99–0.99) | (0.96–0.98) | (0.40–0.44) | (0.98–0.99) | (0.95–0.99) | (0.27–0.33) | ||
| 4 | Combination of structured and unstructured data | 0.99 | 0.98 | 0.43 | 0.99 | 0.98 | 0.34 |
| (0.99–0.99) | (0.96–0.98) | (0.41–0.45) | (0.98–0.99) | (0.95–0.99) | (0.31–0.37) | ||
The second algorithm, which used logistic regression and structured data elements to identify patients hospitalized for ADHF, included 31 data elements in the final model. Variables with the strongest association with a principal discharge diagnosis of heart failure included heart failure in the problem list, inpatient diuretic use, and an elevated BNP (Appendix Table 1). While a prior principal discharge diagnosis of heart failure was associated with a current principal discharge diagnosis of heart failure, prior diagnosis of a secondary heart failure was inversely associated with a current principal discharge diagnosis ( Appendix Table 1). The logistic regression model was associated with an AUC of 0.96 and PPV of 0.15 when the sensitivity was fixed at 0.98 (Table 2).
Appendix Table 1.
Classifiers of acute decompensated heart failure, using logistic regression of structured data (algorithm 2)
| Characteristic | Beta coefficient |
|---|---|
| Age | 0.01 |
| Female | −0.17 |
| Black/African American Race | 0.24 |
| Hispanic/Latino Ethnicity | 0.09 |
| Medicaid | 0.26 |
| Heart failure in problem list | 1.36 |
| Prior diagnosis of any heart failure | −1.02 |
| Prior diagnosis of principal heart failure | 0.65 |
| Prior echocardiography | 0.64 |
| Inpatient diuretics | 1.66 |
| Outpatient diuretics | 0.22 |
| Inpatient ACE inhibitors or ARB | 0.03 |
| Outpatient ACE inhibitors or ARB | 0.14 |
| Inpatient beta blockers | −0.64 |
| Outpatient beta blockers | 0.27 |
| Inpatient heart failure beta blockers | 0.24 |
| Outpatient heart failure beta blockers | 0.28 |
| Inpatient aldosterone antagonist | 0.34 |
| Outpatient aldosterone antagonist | 0.30 |
| Systolic blood pressure | 0.00 |
| Diastolic blood pressure | 0.01 |
| Creatinine | 0.04 |
| Sodium | 0.02 |
| BNP (reference group: no BNP) | |
| <500 | 0.93 |
| 500–999 | 1.97 |
| 1000–4999 | 2.35 |
| 5000–9999 | 2.64 |
| 10000–19999 | 2.92 |
| ≥20000 | 3.05 |
| AcuteMI in problem list | −0.08 |
| Atherosclerosis in problem list | −0.20 |
The third algorithm, in which we used machine learning on unstructured notes, included 427 variables of individual single words in the final model. As shown in Appendix Table 2, the top predictors were related to hospitalizations for heart failure and included “bnp,” “diuresis,” “chf,” and “exacerbation;” the strongest negative predictive term was “ivf.” This model had an AUC of 0.99 in validation and a PPV of 0.30 when setting the sensitivity at 0.98. Algorithm 4, in which we used machine learning on both structured and unstructured data, included 432 elements in the final model. The top predictors in this model were the free text terms of “diuresis,” “bnp,” and “chf;” the top structured data elements–representing the ninth, twelfth, and thirteenth most influential predictors overall–were presence of a bnp laboratory result, heart failure on the problem list, and inpatient diuretic administration (Appendix Table 3). This algorithm had an AUC of 0.99 and a PPV of 0.34 when the sensitivity was 0.98 in the validation set. In the validation set, the specificities of the four algorithms were 77.2%, 79.1%, 91.5%, 93.0%, respectively, while the NPV was 99.9% for all algorithms.
Appendix Table 2.
All 427individual free text words used in classification of acute decompensated heart failure, using a machine learning algorithm on unstructured data (algorithm 3).
| Free text feature | Beta coefficient |
|---|---|
| bnp | 0.9996 |
| diuresis | 0.8901 |
| chf | 0.8785 |
| ivf | −0.6475 |
| hf | 0.6437 |
| exacerbation | 0.6375 |
| failure | 0.5401 |
| lasix | 0.4925 |
| vent | −0.3913 |
| pleural | 0.3899 |
| injury | −0.3731 |
| wts | 0.3598 |
| bumex | 0.3508 |
| icd | 0.3372 |
| ppi | −0.3319 |
| described | −0.3289 |
| severely | 0.3031 |
| fluids | −0.2976 |
| shortness | 0.2886 |
| decompensated | 0.2859 |
| found | −0.2815 |
| admit | −0.2779 |
| diuresed | 0.2703 |
| ef | 0.2693 |
| allergies | −0.2635 |
| bladder | −0.2506 |
| risk | −0.2506 |
| salt | 0.2466 |
| class | 0.2434 |
| ns | −0.2422 |
| steroids | −0.2325 |
| ef | 0.2278 |
| obtain | 0.2278 |
| impression | −0.2269 |
| maintain | −0.2217 |
| npo | −0.2211 |
| dilated | 0.2174 |
| bases | 0.2103 |
| sepsis | −0.2057 |
| 500 | 0.2056 |
| scds | −0.2023 |
| dyspnea | 0.2016 |
| throughout | −0.1999 |
| afebrile | −0.1962 |
| possible | −0.1949 |
| pa | 0.1941 |
| breakfast | 0.1936 |
| breathing | 0.1921 |
| phos | 0.186 |
| crackles | 0.1835 |
| bb | 0.1813 |
| gram | −0.1792 |
| planned | −0.178 |
| guarding | −0.1771 |
| urgent | −0.1745 |
| home | −0.1733 |
| dry | 0.1727 |
| digoxin | 0.1705 |
| friend | −0.1697 |
| 160 | −0.1687 |
| q4h | −0.1679 |
| initial | 0.164 |
| prilosec | −0.1633 |
| affect | 0.1626 |
| control | −0.1624 |
| during | −0.16 |
| pneumonia | 0.158 |
| sore | −0.1567 |
| po | −0.156 |
| amendments | −0.1552 |
| ivc | 0.1546 |
| hpi | −0.1517 |
| sounds | 0.1517 |
| orthopnea | 0.1508 |
| cardiomegaly | 0.1505 |
| pelvic | −0.1489 |
| diurese | 0.1488 |
| dictated | 0.1485 |
| troponin | 0.1481 |
| resident | −0.148 |
| nontender | 0.1453 |
| reyentovich | 0.1451 |
| 1300 | 0.1424 |
| allergies | −0.1403 |
| asthma | −0.1388 |
| mi | −0.1384 |
| dm | 0.1379 |
| prednisone | −0.1378 |
| done | 0.1377 |
| glimepiride | 0.1357 |
| tolerance | 0.1347 |
| assistance | 0.1344 |
| tele | 0.1327 |
| 40mg | 0.1317 |
| age | −0.1306 |
| light | −0.1306 |
| worsening | 0.1304 |
| back | −0.1302 |
| gait | −0.1301 |
| general | −0.1295 |
| rapid | −0.1273 |
| intervention | −0.1255 |
| region | −0.1236 |
| compliance | 0.1219 |
| id | −0.1205 |
| having | −0.1196 |
| 49 | −0.1191 |
| 103 | 0.1178 |
| assess | 0.1175 |
| jvp | 0.1172 |
| probnp | 0.1159 |
| sulfate | −0.1151 |
| pitting | 0.115 |
| pressure | 0.1136 |
| atherosclerotic | 0.1136 |
| setting | −0.1135 |
| paroxysmal | −0.1099 |
| sitting | 0.1085 |
| subjective | −0.1076 |
| 702 | 0.1075 |
| free | −0.1068 |
| filed | 0.1056 |
| biv | 0.1045 |
| 1l | 0.1038 |
| culture | −0.1034 |
| units | −0.1031 |
| ctab | −0.1029 |
| 127 | −0.1025 |
| lesion | −0.1022 |
| hr | −0.1013 |
| rashes | 0.1006 |
| iii | 0.0999 |
| tachypneic | 0.0999 |
| milrinone | 0.0993 |
| mellitus | 0.0984 |
| dyspneic | 0.0983 |
| within | −0.0981 |
| auscultation | −0.0976 |
| 123 | −0.0973 |
| 5 | 0.0954 |
| stage | 0.0954 |
| subsequent | −0.0951 |
| rt | −0.0949 |
| likely | −0.094 |
| fatigue | 0.094 |
| peripheral | −0.0937 |
| ending | −0.092 |
| function | 0.0919 |
| proceed | −0.0912 |
| operative | −0.0911 |
| nd | −0.0908 |
| worsened | 0.0902 |
| pleasant | −0.0901 |
| recurrent | −0.0887 |
| furosemide | 0.0878 |
| lead | 0.087 |
| continues | 0.0868 |
| 101 | −0.0855 |
| neurological | −0.0855 |
| et | −0.0854 |
| aox3 | −0.0851 |
| 114 | −0.084 |
| several | 0.0826 |
| wean | 0.0822 |
| increased | 0.0821 |
| episodes | −0.0819 |
| contrast | −0.0804 |
| imaging | −0.0803 |
| administration | 0.0802 |
| doing | −0.0788 |
| doe | 0.0788 |
| consultation | −0.0784 |
| annular | 0.0771 |
| bilitot | −0.077 |
| notable | −0.0763 |
| also | −0.0763 |
| uc | −0.0758 |
| potential | −0.0757 |
| mildly | −0.0757 |
| 212 | −0.0752 |
| treat | −0.0741 |
| gout | −0.074 |
| asymmetric | 0.074 |
| cough | 0.0733 |
| strict | 0.0731 |
| gain | 0.0719 |
| yo | −0.0695 |
| bibasilar | 0.0694 |
| please | −0.0688 |
| weights | 0.068 |
| flat | 0.0677 |
| greater | 0.0676 |
| aggressive | 0.0663 |
| tab | −0.0659 |
| 85 | 0.0659 |
| attack | 0.0649 |
| pelvis | −0.0647 |
| sternal | 0.0644 |
| end | 0.0642 |
| wd | −0.0635 |
| cardiologist | 0.0634 |
| o2 | 0.0623 |
| 78 | 0.0622 |
| nebs | −0.0612 |
| severity | −0.0609 |
| enlarged | 0.0608 |
| comfortable | 0.0604 |
| goal | 0.0603 |
| mod | 0.0588 |
| trace | −0.0585 |
| sq | −0.058 |
| ct | −0.0576 |
| presents | 0.0574 |
| trop | 0.0573 |
| appropriate | −0.0555 |
| cream | 0.0554 |
| os | 0.0554 |
| prominence | 0.0551 |
| stretcher | −0.0545 |
| q8h | −0.0544 |
| carvedilol | 0.0541 |
| little | −0.0538 |
| dx | −0.0538 |
| labs | −0.0537 |
| apex | 0.0536 |
| increasing | 0.0531 |
| started | −0.0529 |
| ppm | −0.0525 |
| compared | 0.0524 |
| anti | −0.0523 |
| infarct | 0.0522 |
| congestive | 0.0521 |
| systems | −0.0517 |
| ultrasound | −0.0515 |
| via | −0.0514 |
| unit | −0.0514 |
| drainage | −0.0513 |
| 150 | −0.0511 |
| ray | 0.0505 |
| close | −0.0503 |
| day | −0.0499 |
| improved | 0.0497 |
| examined | −0.0493 |
| ongoing | −0.0492 |
| sxs | 0.0492 |
| position | −0.0488 |
| rhonchi | −0.0484 |
| bedside | −0.0482 |
| 100 | −0.048 |
| aiss | 0.048 |
| xray | 0.0478 |
| 138 | −0.0475 |
| 2 | −0.0473 |
| screening | −0.0467 |
| frank | −0.0467 |
| 51 | 0.0464 |
| nc | 0.0455 |
| mediastinal | −0.0452 |
| ace | 0.0452 |
| vanco | −0.045 |
| icu | −0.045 |
| trauma | −0.045 |
| change | −0.0446 |
| 38 | −0.0445 |
| 424 | −0.0436 |
| ventricle | 0.0431 |
| eyes | −0.0426 |
| rule | 0.0426 |
| 144 | 0.0425 |
| aorta | −0.0419 |
| when | −0.0418 |
| 350 | 0.0416 |
| mouth | −0.0414 |
| returned | −0.0412 |
| could | −0.0409 |
| 22 | −0.0407 |
| appearing | −0.0405 |
| lv | 0.0402 |
| penicillins | 0.0401 |
| my | −0.0396 |
| elderly | −0.0392 |
| cell | −0.0391 |
| safety | 0.039 |
| dated | 0.0389 |
| marital | 0.0387 |
| mass | −0.0383 |
| movement | −0.0371 |
| dysfunction | 0.0371 |
| arterial | −0.0369 |
| hold | −0.0367 |
| cholesterol | 0.0367 |
| transferred | −0.0365 |
| prn | −0.0364 |
| lle | −0.0359 |
| meds | −0.0354 |
| tte | 0.0354 |
| plt | −0.035 |
| 99 | −0.035 |
| leads | 0.0349 |
| cor | −0.0348 |
| weeks | 0.0341 |
| inferior | −0.034 |
| caliber | −0.0339 |
| maintained | 0.0334 |
| stenosis | −0.0327 |
| obstruction | −0.0327 |
| small | 0.0323 |
| mcv | 0.032 |
| questionable | −0.0318 |
| tortuous | −0.0317 |
| plavix | −0.0316 |
| pacing | −0.0315 |
| asa | −0.0314 |
| overload | 0.0313 |
| wnl | −0.031 |
| ws | −0.0303 |
| were | −0.0295 |
| established | −0.0293 |
| night | −0.0293 |
| swelling | 0.0292 |
| temporal | −0.029 |
| 73 | −0.0288 |
| cardiovascular | −0.0285 |
| regular | −0.0279 |
| 80 | −0.0278 |
| phlegm | 0.0277 |
| induced | −0.0276 |
| stomach | −0.0271 |
| gastrointestinal | −0.0269 |
| 94 | 0.0268 |
| pvd | −0.0267 |
| legs | 0.0262 |
| 2006 | −0.0257 |
| cardiomyopathy | 0.0255 |
| chloride | −0.0254 |
| 0 | −0.0254 |
| ac | −0.0249 |
| removal | −0.0249 |
| teeth | 0.0242 |
| pulm | 0.0237 |
| fistula | 0.0233 |
| spironolactone | 0.0232 |
| arrived | −0.0229 |
| 15 | −0.0228 |
| study | 0.0228 |
| necessary | −0.0227 |
| afternoon | 0.0227 |
| much | 0.0215 |
| going | −0.0204 |
| healthy | 0.0204 |
| hrs | −0.0202 |
| addendum | −0.02 |
| inhaler | −0.0192 |
| beta | 0.0191 |
| strength | −0.0186 |
| md | −0.0179 |
| sem | 0.0178 |
| 75 | −0.0172 |
| more | 0.017 |
| rom | −0.0169 |
| pmd | −0.0165 |
| fair | −0.0164 |
| through | −0.0163 |
| sexually | −0.0161 |
| notified | 0.0152 |
| ckd | 0.015 |
| staff | −0.0149 |
| daughter | 0.0144 |
| medicine | −0.0143 |
| stopped | 0.0143 |
| restriction | 0.0139 |
| somewhat | −0.0138 |
| dizziness | −0.0135 |
| awake | −0.0134 |
| breast | −0.0133 |
| needs | 0.0133 |
| stool | −0.012 |
| psychiatric | 0.012 |
| check | 0.0119 |
| grossly | −0.0113 |
| urine | −0.0113 |
| overall | −0.0107 |
| hepatitis | 0.0104 |
| atrovent | −0.0103 |
| intolerance | 0.0103 |
| which | −0.0101 |
| 133 | −0.0099 |
| nt | 0.0099 |
| nebulization | −0.0098 |
| fib | −0.0097 |
| ii | 0.0097 |
| prophylaxis | −0.0091 |
| probably | 0.0091 |
| murmurs | 0.009 |
| echocardiography | 0.0089 |
| trig | 0.0085 |
| spouse | 0.0082 |
| sclerae | −0.0081 |
| angioplasty | −0.0078 |
| organomegaly | 0.0078 |
| chol | 0.0066 |
| 170 | −0.0065 |
| exertion | 0.0062 |
| keep | 0.0061 |
| groin | −0.0059 |
| cannot | 0.0055 |
| 90 | −0.0053 |
| septal | 0.0053 |
| lbs | 0.0052 |
| ersd | 0.005 |
| consolidation | −0.0049 |
| 146 | 0.0047 |
| limited | −0.0045 |
| ua | −0.0039 |
| basename | 0.0037 |
| post | −0.0036 |
| arrhythmia | −0.0031 |
| decrease | −0.0031 |
| evidence | −0.0026 |
| load | −0.0024 |
| routine | −0.0022 |
| twice | −0.0021 |
| 3d | −0.002 |
| dysuria | 0.0018 |
| meals | 0.0014 |
| cooperative | −0.0004 |
| 63 | −0.0004 |
Appendix Table 3.
All 432 features for classification of acute decompensated heart failure, using a machine learning algorithm on both structured and unstructured data (algorithm 4). Features with a * are structured data elements; all others are individual free text words from unstructured data.
| Characteristic or free text feature | Beta coefficient |
|---|---|
| diuresis | 0.8772 |
| bnp | 0.8576 |
| chf | 0.7207 |
| hf | 0.6239 |
| ivf | −0.6103 |
| exacerbation | 0.596 |
| injury | −0.4181 |
| failure | 0.4087 |
| Any BNP* | 0.405 |
| wts | 0.3886 |
| vent | −0.3735 |
| Heart failure in problem list* | 0.37 |
| Inpatient diuretic* | 0.3673 |
| pleural | 0.3567 |
| lasix | 0.3456 |
| ppi | −0.3353 |
| severely | 0.3289 |
| icd | 0.3221 |
| fluids | −0.3209 |
| BNP≥20000* | 0.3037 |
| described | −0.2972 |
| bumex | 0.2889 |
| decompensated | 0.2801 |
| found | −0.2765 |
| wbc | −0.2758 |
| diuresed | 0.2718 |
| deformities | −0.2582 |
| admit | −0.2578 |
| shortness | 0.2522 |
| risk | −0.2506 |
| bladder | −0.2506 |
| salt | 0.2466 |
| class | 0.2434 |
| ns | −0.2422 |
| steroids | −0.2325 |
| obtain | 0.2278 |
| ef | 0.2278 |
| impression | −0.2269 |
| maintain | −0.2216 |
| npo | −0.2211 |
| dilated | 0.2174 |
| bases | 0.2103 |
| sepsis | −0.2057 |
| 500 | 0.2056 |
| scds | −0.2023 |
| dyspnea | 0.2016 |
| throughout | −0.1999 |
| afebrile | −0.1962 |
| possible | −0.1949 |
| pa | 0.1941 |
| breakfast | 0.1935 |
| breathing | 0.1921 |
| phos | 0.186 |
| crackles | 0.1836 |
| bb | 0.1813 |
| gram | −0.1793 |
| planned | −0.178 |
| guarding | −0.177 |
| urgent | −0.1744 |
| home | −0.1732 |
| dry | 0.1727 |
| digoxin | 0.1705 |
| friend | −0.1696 |
| 160 | −0.1687 |
| q4h | −0.1679 |
| initial | 0.164 |
| prilosec | −0.1633 |
| affect | 0.1626 |
| control | −0.1624 |
| during | −0.16 |
| pneumonia | 0.1579 |
| sore | −0.1567 |
| po | −0.156 |
| amendments | −0.1552 |
| ivc | 0.1546 |
| hpi | −0.1517 |
| sounds | 0.1517 |
| orthopnea | 0.1508 |
| cardiomegaly | 0.1505 |
| pelvic | −0.1488 |
| diurese | 0.1488 |
| dictated | 0.1485 |
| troponin | 0.1481 |
| resident | −0.148 |
| nontender | 0.1453 |
| reyentovich | 0.1452 |
| 1300 | 0.1424 |
| allergies | −0.1404 |
| asthma | −0.1388 |
| mi | −0.1384 |
| dm | 0.1379 |
| prednisone | −0.1378 |
| done | 0.1377 |
| glimepiride | 0.1357 |
| tolerance | 0.1347 |
| assistance | 0.1344 |
| tele | 0.1327 |
| 40mg | 0.1317 |
| light | −0.1306 |
| age | −0.1305 |
| worsening | 0.1304 |
| back | −0.1302 |
| gait | −0.1301 |
| general | −0.1294 |
| rapid | −0.1272 |
| intervention | −0.1255 |
| region | −0.1236 |
| compliance | 0.122 |
| id | −0.1205 |
| having | −0.1196 |
| 49 | −0.1191 |
| 103 | 0.1178 |
| assess | 0.1175 |
| jvp | 0.1172 |
| probnp | 0.1158 |
| sulfate | −0.1151 |
| pitting | 0.1149 |
| pressure | 0.1137 |
| setting | −0.1135 |
| atherosclerotic | 0.1135 |
| paroxysmal | −0.1099 |
| sitting | 0.1085 |
| subjective | −0.1076 |
| 702 | 0.1075 |
| free | −0.1068 |
| filed | 0.1056 |
| biv | 0.1044 |
| 1l | 0.1038 |
| culture | −0.1034 |
| units | −0.1032 |
| ctab | −0.1029 |
| 127 | −0.1026 |
| lesion | −0.1022 |
| hr | −0.1014 |
| rashes | 0.1006 |
| iii | 0.0999 |
| tachypneic | 0.0999 |
| milrinone | 0.0993 |
| mellitus | 0.0984 |
| dyspneic | 0.0983 |
| within | −0.0981 |
| auscultation | −0.0976 |
| 123 | −0.0972 |
| 5 | 0.0955 |
| stage | 0.0954 |
| subsequent | −0.0951 |
| rt | −0.0949 |
| likely | −0.0941 |
| fatigue | 0.094 |
| peripheral | −0.0937 |
| ending | −0.092 |
| function | 0.0919 |
| proceed | −0.0912 |
| operative | −0.0911 |
| nd | −0.0909 |
| pleasant | −0.0901 |
| worsened | 0.0901 |
| recurrent | −0.0887 |
| furosemide | 0.0878 |
| Creatinine* | 0.0875 |
| lead | 0.0869 |
| continues | 0.0868 |
| neurological | −0.0856 |
| 101 | −0.0855 |
| et | −0.0854 |
| aox3 | −0.0852 |
| 114 | −0.084 |
| several | 0.0826 |
| wean | 0.0821 |
| increased | 0.0821 |
| episodes | −0.0819 |
| contrast | −0.0803 |
| imaging | −0.0802 |
| administration | 0.0802 |
| doing | −0.0788 |
| doe | 0.0788 |
| consultation | −0.0784 |
| annular | 0.0771 |
| bilitot | −0.077 |
| notable | −0.0763 |
| also | −0.0761 |
| uc | −0.0758 |
| potential | −0.0757 |
| mildly | −0.0756 |
| 212 | −0.0752 |
| treat | −0.0741 |
| gout | −0.074 |
| asymmetric | 0.074 |
| cough | 0.0733 |
| Inpatient bblocker* | −0.0731 |
| strict | 0.0731 |
| gain | 0.0719 |
| yo | −0.0695 |
| bibasilar | 0.0694 |
| please | −0.0687 |
| weights | 0.068 |
| flat | 0.0677 |
| greater | 0.0676 |
| aggressive | 0.0664 |
| tab | −0.0659 |
| 85 | 0.0659 |
| attack | 0.0649 |
| pelvis | −0.0648 |
| sternal | 0.0644 |
| end | 0.0642 |
| wd | −0.0636 |
| cardiologist | 0.0634 |
| o2 | 0.0623 |
| 78 | 0.0622 |
| nebs | −0.0613 |
| severity | −0.0609 |
| enlarged | 0.0608 |
| comfortable | 0.0604 |
| goal | 0.0603 |
| Prior hospitalization with principal diagnosis of heart failure* | 0.0593 |
| mod | 0.0588 |
| trace | −0.0585 |
| sq | −0.058 |
| ct | −0.0576 |
| presents | 0.0574 |
| trop | 0.0573 |
| appropriate | −0.0555 |
| cream | 0.0554 |
| os | 0.0554 |
| prominence | 0.0551 |
| stretcher | −0.0545 |
| q8h | −0.0543 |
| carvedilol | 0.0542 |
| little | −0.0538 |
| dx | −0.0538 |
| labs | −0.0536 |
| apex | 0.0536 |
| increasing | 0.0531 |
| started | −0.0529 |
| ppm | −0.0525 |
| compared | 0.0524 |
| anti | −0.0523 |
| infarct | 0.0522 |
| congestive | 0.052 |
| systems | −0.0517 |
| ultrasound | −0.0515 |
| drainage | −0.0514 |
| via | −0.0514 |
| unit | −0.0514 |
| 150 | −0.0511 |
| ray | 0.0505 |
| close | −0.0503 |
| day | −0.0499 |
| improved | 0.0497 |
| ongoing | −0.0492 |
| examined | −0.0492 |
| sxs | 0.0492 |
| position | −0.0488 |
| rhonchi | −0.0484 |
| bedside | −0.0482 |
| 100 | −0.048 |
| aiss | 0.048 |
| xray | 0.0478 |
| 138 | −0.0475 |
| 2 | −0.0473 |
| screening | −0.0468 |
| frank | −0.0467 |
| 51 | 0.0462 |
| nc | 0.0455 |
| mediastinal | −0.0452 |
| ace | 0.0452 |
| vanco | −0.0451 |
| trauma | −0.045 |
| icu | −0.045 |
| change | −0.0446 |
| 38 | −0.0446 |
| Echo* | 0.044 |
| 424 | −0.0436 |
| ventricle | 0.0431 |
| rule | 0.0427 |
| eyes | −0.0426 |
| 144 | 0.0425 |
| when | −0.0419 |
| aorta | −0.0419 |
| 350 | 0.0416 |
| mouth | −0.0414 |
| returned | −0.0412 |
| could | −0.0409 |
| 22 | −0.0407 |
| appearing | −0.0405 |
| lv | 0.0402 |
| penicillins | 0.0401 |
| my | −0.0396 |
| elderly | −0.0394 |
| cell | −0.0391 |
| safety | 0.039 |
| dated | 0.0389 |
| marital | 0.0387 |
| mass | −0.0383 |
| movement | −0.0371 |
| dysfunction | 0.0371 |
| arterial | −0.0368 |
| hold | −0.0367 |
| cholesterol | 0.0366 |
| transferred | −0.0365 |
| prn | −0.0365 |
| lle | −0.0359 |
| Sodium* | −0.0355 |
| meds | −0.0354 |
| tte | 0.0354 |
| plt | −0.035 |
| 99 | −0.035 |
| leads | 0.035 |
| cor | −0.0348 |
| weeks | 0.0341 |
| inferior | −0.034 |
| caliber | −0.0339 |
| Outpatient evidence based bblocker* | 0.0337 |
| maintained | 0.0334 |
| stenosis | −0.0328 |
| obstruction | −0.0327 |
| small | 0.0322 |
| mcv | 0.032 |
| tortuous | −0.0318 |
| questionable | −0.0318 |
| plavix | −0.0316 |
| pacing | −0.0315 |
| asa | −0.0313 |
| overload | 0.0313 |
| wnl | −0.031 |
| ws | −0.0303 |
| were | −0.0295 |
| established | −0.0293 |
| night | −0.0293 |
| temporal | −0.029 |
| 73 | −0.0288 |
| cardiovascular | −0.0285 |
| regular | −0.0278 |
| 80 | −0.0278 |
| phlegm | 0.0277 |
| induced | −0.0276 |
| Inpatient ACE/ARB* | −0.0276 |
| stomach | −0.0271 |
| gastrointestinal | −0.0269 |
| 94 | 0.0268 |
| pvd | −0.0267 |
| legs | 0.0262 |
| 2006 | −0.0257 |
| cardiomyopathy | 0.0255 |
| chloride | −0.0254 |
| 0 | −0.0253 |
| ac | −0.0249 |
| removal | −0.0249 |
| pulm | 0.0237 |
| arrived | −0.0229 |
| 15 | −0.0229 |
| study | 0.0228 |
| necessary | −0.0227 |
| afternoon | 0.0227 |
| much | 0.0215 |
| going | −0.0204 |
| healthy | 0.0204 |
| hrs | −0.0202 |
| addendum | −0.0201 |
| inhaler | −0.0192 |
| beta | 0.0191 |
| strength | −0.0186 |
| sem | 0.0179 |
| md | −0.0177 |
| 75 | −0.0173 |
| more | 0.0171 |
| rom | −0.0168 |
| pmd | −0.0165 |
| fair | −0.0164 |
| through | −0.0163 |
| sexually | −0.0162 |
| notified | 0.0153 |
| staff | −0.0149 |
| medicine | −0.0144 |
| daughter | 0.0144 |
| stopped | 0.0143 |
| somewhat | −0.0139 |
| restriction | 0.0138 |
| dizziness | −0.0136 |
| awake | −0.0134 |
| breast | −0.0133 |
| needs | 0.0132 |
| stool | −0.012 |
| check | 0.0119 |
| grossly | −0.0113 |
| urine | −0.0113 |
| overall | −0.0107 |
| hepatitis | 0.0104 |
| atrovent | −0.0103 |
| intolerance | 0.0103 |
| which | −0.0101 |
| nt | 0.01 |
| 133 | −0.0099 |
| nebulization | −0.0098 |
| fib | −0.0097 |
| ii | 0.0097 |
| prophylaxis | −0.0091 |
| probably | 0.0091 |
| echocardiography | 0.009 |
| murmurs | 0.0089 |
| spouse | 0.0082 |
| sclerae | −−0.0081 |
| organomegaly | 0.0079 |
| angioplasty | −0.0078 |
| chol | 0.0066 |
| 170 | −0.0065 |
| age* | 0.0064 |
| exertion | 0.0062 |
| keep | 0.0061 |
| groin | −0.0059 |
| Outpatient Aldosterone antagonist* | 0.0058 |
| cannot | 0.0055 |
| 90 | −0.0053 |
| lbs | 0.0052 |
| consolidation | −0.0049 |
| 146 | 0.0047 |
| limited | −0.0046 |
| ua | −0.0039 |
| post | −0.0036 |
| basename | 0.0036 |
| decrease | −0.0031 |
| arrhythmia | −0.0031 |
| evidence | −0.0026 |
| Diastolic Blood Pressure* | −0.0024 |
| load | −0.0024 |
| routine | −0.0022 |
| twice | −0.0021 |
| 3d | −0.002 |
| Systolic Blood Pressure* | 0.0019 |
| meals | 0.0014 |
| cooperative | −0.0006 |
| 63 | −0.0004 |
Based on the PPV of the algorithms in the validation set, we calculated that algorithms 1-4 would identify a total of 7.1, 6.7, 3.3, and 2.9 hospitalizations as being positive for each true positive case of heart failure; these values represent the total number of hospital charts needed for secondary chart review for each true positive case. Given the high sensitivity of these algorithms, 98% of all heart failure cases would be identified and confirmed by the gold standard of secondary provider review.
The heart failure NP and PA providers reported a mean of 11.7 (range 10–15) minutes spent on reviewing each charts for heart failure. Providers also reported a mean of 65 (range 53–90) minutes spent on reviewing charts for heart failure per day; based on this rate, we derived a review time per chart of 5.5 minutes, assuming an average of 11.7 charts reviewed per day during the study period. Averaging the reported time per chart (11.7 minutes) and the derived review time per chart based on reported review time per day (5.5 minutes), we estimated that providers spent 8.6 minutes per chart. Using this parameter for review time per chart, the time providers would spend performing chart review for each true positive case of ADHF would be 61.4, 57.3, 28.7, and 25.3 minutes if initial screening was done with algorithms 1, 2, 3, and 4, respectively (Figure).
Figure.
Average time needed for secondary screening by providers to confirm each true positive case of acute decompensated heart failure diagnosis following initial screening with one of four automated algorithms: presence of 1 of 3 clinical characteristics (algorithm 1), logistic regression of structured data (algorithm 2), machine learning of unstructured data (algorithm 3), and machine learning of a combination of structured and unstructured data (algorithm 4). Average time was calculated: as estimated time per chart based on provider survey multiplied by the sum of true and false positives when the true positive equals one.
Discussion
Reducing readmissions is a priority for hospital systems and, as a result, hospitals have put substantial efforts in improving outcomes for patients hospitalized for heart failure. Such efforts have broadly been categorized into three domains: inpatient care, discharge processes and transitional care, and general quality improvement efforts such as performance feedback. Inpatient care initiatives include dedicated heart failure teams, electronic order sets, inpatient education, and provider reminders for evidence based therapies.5 Transitional care interventions include early discharge planning, providing medications at time of discharge, scheduling follow up with outpatient provider, pharmacist counseling, and post-discharge phone calls.5,6 The majority of these interventions require early identification of appropriate patients during hospitalizations.10
In order to accomplish any inpatient care intervention and many of discharge and transitional care interventions, patients with acute decompensated heart failure need to be easily identified well before discharge. We developed multiple computable phenotypes15,16 for ADHF early during hospitalization. A simple algorithm based on three clinical characteristics that has been utilized by the heart failure team at our institution demonstrated high sensitivity as intended. However, this algorithm also had a very low PPV, necessitating the team to perform a significant amount of secondary chart review for validation. A second algorithm that relied on a linear combination of 31 structured data elements was similarly limited by high number of false positives. Conversely, two machine learning algorithms that utilized unstructured text from provider notes and imaging reports significantly increased PPV as compared to the algorithms using only structured data.
Nonetheless, even these top performing machine learning algorithms would require a secondary chart review to confirm a diagnosis of AHDF in clinical practice. Given the potential high costs associated with not intervening on patients hospitalized for ADHF, our goal was to capture nearly all of these patients with these algorithms. While the machine learning algorithms had a near perfect AUC, this measure may not reflect classification ability at the extremes. As a result, only one-third of those hospitalizations identified as ADHF by our best algorithm were actual cases when we set the sensitivity at 98%. Nonetheless, with the machine learning algorithm on structured and unstructured data, practitioners would need to review three charts for each correctly identified patient with ADHF; this compares favorably to the seven charts needed to review for each true positive case with algorithms using structured data. As we estimated that our practitioners spend 8.6 minutes per chart, this improved performance of algorithm 4 as compared to algorithm 1 would reduce the amount of time spent on chart review by 36.1 minutes per case of ADHF. The overall time savings with algorithm 4 is dependent on the volume of heart failure patients–in the hospital. At our hospital, we observed 1.6 hospitalizations for ADHF per day during the study period. As a result, the implementation of the more advanced algorithm could save nearly an hour of provider time every day or the equivalent of 0.17 of a full time (40 hour) equivalent. Given this benefit, we will be implementing this algorithm to run at our hospital; the algorithm will be used to build a daily EHR-based screening list for our heart failure team.
The machine learning algorithms performed as well, if not better, in terms of AUC when compared to similarly developed algorithms to identify patients with any heart failure–acute or chronic in the hospital.11 However, the algorithms for acute decompensated heart failure appeared to have lower PPV than those from the prior study. These differences in performance may be related to differences in prevalence, as there are about four times as many patients are hospitalized with any heart failure as with acute decompensated heart failure.12 Given the volume of patients with any heart failure coupled with the intensity of resources used to prevent readmissions following hospitalizations for ADHF, hospitals likely prefer targeting interventions to patients hospitalized with a principal discharge diagnosis of heart failure. Thus the algorithms developed for our study will be useful for hospitals focused on preventing readmissions in response to policy and payer efforts such as the Hospital Readmissions Reduction Program.3
Our study has limitations that deserve mention. The study used data from a single hospital so results may not be applicable to other institutions. First, while our approaches could be easily replicated, the algorithms that use unstructured data may need to be calibrated for individual institutions as language and documentation can vary from site to site. Given potential differences in data across institutions, machine learning algorithms may not see similar performance improvement at other sites. Second, our gold standard for ADHF was not based on provider chart adjudication but rather discharge diagnosis, an imperfect measure. It is possible that limitations of the algorithms may be partly related to limitations of this gold standard; for instance, some of the false positives could be related to true cases of ADHF that have another discharge diagnosis. Nonetheless, the discharge diagnosis is currently used for quality measurement3,8 so is of primary interests to hospitals, despite its limitations. Third, ICD-10 codes have replaced ICD-9 codes in most countries which may limit generalizability; however, the algorithms can be easily validated using these newer codes. Fourth, we surveyed only three providers to assess time spent on chart review. These were the total number of providers performing such chart review at our institution at the time, but their approach to chart review was not standardized and their workflow may not be generalizable to other institutions. Fifth, provider responses were based on recall and thus subject to recall bias, and which may partly explain the potential inconsistency between reported time spent on individual chart review and reported time spent reviewing charts per day.
Given current incentive structure to reduce readmissions following hospitalizations for ADHF,3 early identification of patients with ADHF is needed to initiate interventions for readmission reduction. We found that using traditional approaches with structured data can accurately identify nearly all patients with ADHF but are limited by a large number of false positives. Machine learning algorithms with unstructured notes and radiology reports can significantly reduce false positives, thereby improving provider efficiency for delivery of quality improvement interventions. Nonetheless, our results suggest that some amount of secondary chart review is necessary for interventions designed to target all patients hospitalized with ADHF.
Highlights.
Machine learning improves identification of heart failure patients over automated approaches that rely on a few structured variables
Improvement in identification may be related to use of free text from clinical notes
Initial screening with a machine learning algorithm can reduce time needed for providers to perform chart review, thus improving efficiency for care
Acknowledgments
This work was supported by the Agency for Healthcare Research and Quality (AHRQ) grant K08HS23683.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- 1.Pfuntner A, Wier LM, Stocks C. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville (MD): 2013. Most Frequent Conditions in U.S. Hospitals, 2011: Statistical Brief #162. [Google Scholar]
- 2.Yancy CW, Jessup M, Bozkurt B, et al. 2013 ACCF/AHA Guideline for the Management of Heart Failure: A Report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2013;62(16):e147–239. doi: 10.1016/j.jacc.2013.05.019. [DOI] [PubMed] [Google Scholar]
- 3.Centers for Medicare & Medicaid Services. [Accessed 11/23/16];Readmissions Reduction Program. Available at: https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program.html.
- 4.Bradley EH, Sipsma H, Horwitz LI, et al. Hospital strategy uptake and reductions in unplanned readmission rates for patients with heart failure: a prospective study. Journal of general internal medicine. 2015;30(5):605–611. doi: 10.1007/s11606-014-3105-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Kociol RD, Peterson ED, Hammill BG, et al. National survey of hospital strategies to reduce heart failure readmissions: findings from the Get With the Guidelines-Heart Failure registry. Circulation. Heart failure. 2012;5(6):680–687. doi: 10.1161/CIRCHEARTFAILURE.112.967406. [DOI] [PubMed] [Google Scholar]
- 6.Vasilevskis EE, Kripalani S, Ong MK, et al. Variability in Implementation of Interventions Aimed at Reducing Readmissions Among Patients With Heart Failure: A Survey of Teaching Hospitals. Academic medicine : journal of the Association of American Medical Colleges. 2016;91(4):522–529. doi: 10.1097/ACM.0000000000000994. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Keenan PS, Normand SL, Lin Z, et al. An administrative claims measure suitable for profiling hospital performance on the basis of 30-day all-cause readmission rates among patients with heart failure. Circ Cardiovasc Qual Outcomes. 2008;1(1):29–37. doi: 10.1161/CIRCOUTCOMES.108.802686. [DOI] [PubMed] [Google Scholar]
- 8.Bonow RO, Ganiats TG, Beam CT, et al. ACCF/AHA/AMA-PCPI 2011 Performance Measures for Adults With Heart Failure: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Performance Measures and the American Medical Association-Physician Consortium for Performance Improvement. Circulation. 2012;125(19):2382–2401. doi: 10.1161/CIR.0b013e3182507bec. [DOI] [PubMed] [Google Scholar]
- 9.Evans RS, Benuzillo J, Horne BD, et al. Automated identification and predictive tools to help identify high-risk heart failure patients: pilot evaluation. Journal of the American Medical Informatics Association : JAMIA. 2016 doi: 10.1093/jamia/ocv197. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Banerjee D, Thompson C, Bingham A, Kell C, Duhon J, Grossman H. An Electronic Medical Record Report Improves Identification of Hospitalized Patients With Heart Failure. J Card Fail. 2015;22(5):402–405. doi: 10.1016/j.cardfail.2015.12.006. [DOI] [PubMed] [Google Scholar]
- 11.Blecker S, Katz SD, Horwitz LI, et al. Comparison of Approaches for Heart Failure Case Identification From Electronic Health Record Data. JAMA Cardiol. 2016;1(9):1014–1020. doi: 10.1001/jamacardio.2016.3236. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Blecker S, Paul M, Taksler G, Ogedegbe G, Katz S. Heart failure-associated hospitalizations in the United States. J Am Coll Cardiol. 2013;61(12):1259–1267. doi: 10.1016/j.jacc.2012.12.038. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Centers for Medicare and Medicaid Services. [Accessed 11/23/16];Fact Sheet: Two-Midnight Rule. Available at: https://www.cms.gov/Newsroom/MediaReleaseDatabase/Fact-sheets/2015-Fact-sheets-items/2015-07-01-2.html.
- 14.Ng AY. Feature selection, L 1 vs. L 2 regularization, and rotational invariance. Paper presented at: Proceedings of the twenty-first international conference on Machine learning; 2004. [Google Scholar]
- 15.Mo H, Thompson WK, Rasmussen LV, et al. Desiderata for computable representations of electronic health records-driven phenotype algorithms. Journal of the American Medical Informatics Association : JAMIA. 2015;22(6):1220–1230. doi: 10.1093/jamia/ocv112. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Shivade C, Raghavan P, Fosler-Lussier E, et al. A review of approaches to identifying patient phenotype cohorts using electronic health records. Journal of the American Medical Informatics Association : JAMIA. 2014;21(2):221–230. doi: 10.1136/amiajnl-2013-001935. [DOI] [PMC free article] [PubMed] [Google Scholar]