Deriving comorbidities from medical records using natural language processing

Abstract

Extracting comorbidity information is crucial for phenotypic studies because of the confounding effect of comorbidities. We developed an automated method that accurately determines comorbidities from electronic medical records. Using a modified version of the Charlson comorbidity index (CCI), two physicians created a reference standard of comorbidities by manual review of 100 admission notes. We processed the notes using the MedLEE natural language processing system, and wrote queries to extract comorbidities automatically from its structured output. Interrater agreement for the reference set was very high (97.7%). Our method yielded an F1 score of 0.761 and the summed CCI score was not different from the reference standard (p=0.329, power 80.4%). In comparison, obtaining comorbidities from claims data yielded an F1 score of 0.741, due to lower sensitivity (66.1%). Because CCI has previously been validated as a predictor of mortality and readmission, our method could allow automated prediction of these outcomes.

Introduction

Extracting comorbidity information is crucial for identifying phenotype–genotype associations, studying clinical outcomes because of the confounding effect of comorbidities,^1–3 and patient cohort identification because it helps identify cohorts with certain outcomes.^{4 5} As composite representations of burden of disease and patient complexity, comorbidity measures have also been related to health resource use, mortality and readmission.^{6 7}

Numerous methods exist to measure comorbidities in clinical research including the Charlson comorbidity index (CCI), cumulative illness rating scale, index of coexisting disease, and Kaplan index.⁸ Among these, various versions of the CCI are most extensively studied.^{8 9} The index encompasses 19 medical conditions, each weighted between 1 and 6 based on the relative risk of their association with 1-year mortality. The summed CCI score, calculated by adding the individual scores for each item, has proved to be an accurate predictor of mortality, disability, readmission and length of stay.^10–12

For most applications, the CCI score is calculated by manual record review or using claims data, typically coded using the International Classification of Diseases, 9th Version (ICD-9).^13–15 The former approach is costly and the latter introduces biases due to coding errors, heterogeneous coding conventions, and the granularity of the coding system.¹⁶ In addition, previous research has suggested that manually extracting comorbidity information from medical records is superior to the use of ICD-9 codes¹⁷ and claims data are not available until after discharge time.

Most of the information needed for comorbidities must be extracted from narrative reports, particularly from the ‘history of present illness’ and ‘past medical history’ sections. Previous efforts in automating the extraction of comorbidity information from narratives are limited to specific domains¹⁸ or rudimentary methodologies such as keyword search.¹⁹ However, it should be possible to utilize natural language processing (NLP) to build a generalizable method for extracting comorbidities from electronic health records (EHR). Therefore, we aim to develop an effective automated and generalizable NLP-based method that derives comorbidities from narrative records.

Methods

We used a 15-item modified version of the CCI (table 1), which we will hereafter refer to as ‘the index’. It is different from the original Charlson index⁹ in that it combines mild, moderate and severe liver disease into one category, combines diabetes mellitus with and without complications into one category, and excludes metastatic solid tumors. To formulate the index, we performed a preliminarily manual review of randomly selected notes and included in the index only those categories that could be accurately captured. We found that the notes frequently lacked sufficient information to distinguish the severity of liver disease or diabetes, and were not explicit regarding tumor staging. Thus, in order to have a reliable gold standard comparator, we collapsed or excluded the relevant categories.

Table 1.

Items included in the modified Charlson comorbidity index, and their relative scores

	Item	Points	POStrain	TP	FN	FP	TN
1	Myocardial infarction	1	8	5	2	8	55
2	Congestive heart failure	1	10	14	4	3	49
3	Peripheral vascular disease	1	3	6	1	6	57
4	Cerebrovascular disease	1	9	10	2	8	50
5	Dementia	1	3	11	1	0	58
6	Chronic pulmonary disease	1	11	5	9	0	56
7	Connective tissue disease	1	3	2	0	2	66
8	Peptic ulcer disease	1	3	2	2	2	64
9	Liver disease (mild, moderate or severe)	1	3	4	1	1	64
10	Diabetes mellitus (with/without complications)	1	12	23	3	0	44
11	Hemiplegia	2	2	3	1	2	64
12	Renal disease	2	10	20	1	4	45
13	Leukemia	2	2	1	1	0	68
14	Lymphoma	2	2	1	1	1	67
15	AIDS/HIV	6	1	1	0	2	67

Fourth column (POS_train) indicates total number of positive cases in the training set (N = 30), and columns 5–8 show the number of true positive (TP), false positive (FP), false negative (FN) and true negative (TN) assignments by the automated method in the test set (N = 70), in comparison to manual review. The total number of positive cases in the test set can be easily calculated by adding the values for TP and FN in each row.

We randomly selected 100 admission notes for patients admitted at the New York-Presbyterian Hospital during 2009–12, each corresponding to a unique patient. Two physicians annotated the notes to record the presence or absence of each of the 15 items in the index, creating a ‘reference standard’. We measured agreement between two coders using a subset of 30 notes. We then processed all the notes using the MedLEE NLP system, which handles negations, interprets the level of certainty associated with the concepts in the notes, and normalizes the concepts to concept unique identifiers (CUI) in the unified medical language system.²⁰ We excluded findings attributed to patient's family and only included ‘diseases and conditions’ by excluding medications, laboratory results, procedures and measurements, resulting in a table containing ongoing conditions of each patient represented as CUI. We then used a randomly selected subset of 30 subjects for training and the remaining 70 for testing.

Using knowledge we developed queries to determine the comorbidities. Each query contained a list of CUI representing the pertinent item in the score (for an example, see table 2). Queries were developed by consensus between the first two authors. For each disease or condition in the index, we included CUI representing that disease or condition (eg, ‘diabetes mellitus’) as well as those that imply the problem exists (eg, ‘diabetic foot’). We manually evaluated and refined the queries using the training set. Finally, we applied these queries to the test set to evaluate the accuracy of the automated method against the reference set. Concepts included in each query can be found in the supplementary data (available online only).

Table 2.

The list of concepts included in the query for diabetes mellitus, and their preferred names in the unified medical language system

CUI	Preferred name
C0011849	Diabetes mellitus
C0011854	Diabetes mellitus, insulin-dependent
C0011860	Diabetes mellitus, non-insulin-dependent
C0206172	Diabetic foot
C1456868	Diabetic foot ulcer
C0011880	Diabetic ketoacidosis
C0011881	Diabetic nephropathy
C0011884	Diabetic retinopathy
C0011882	Diabetic neuropathies
C1720297	Disorder associated with type I diabetes mellitus
C2919365	Macroalbuminuric diabetic nephropathy

The concepts are represented using their unique identifiers (CUI) in the unified medical language system.

In addition, we compared our method with an approach that determined comorbidities using the ICD-9 codes recorded at discharge. First, the accuracy of each method was calculated based on the proportion of patient–condition pairs that were correctly classified. For example, if a patient was correctly classified as having diabetes mellitus this patient–condition pair was counted towards true positives, and when diabetes mellitus was wrongly ruled out it was counted towards false negatives, and so on. We calculated the sensitivity, specificity and F1 score to measure performance, and also compared the overall agreement of each approach with the reference set using Cohen's κ. In addition, we compared the summed CCI score of each method versus that of the reference standard using a standard t test. All the analyses were performed on the test set (N=70). Our study was designed with 80% power to detect a minimum difference in mean CCI scores of 0.5 between manual and automated methods in the test set. To select a minimum difference of 0.5 we referred to previous studies suggesting that differences in CCI scores of at least 1.0 were necessary for clinically important differences in outcomes.^{21 22}

Results

Overall, 53% of patients were women, and the average age was 63.6 years (range 1–98 years, SD 23.5 years). Based on manual review, each of these patients had on average 2.12 of the index conditions reported in their admission note (range 0–5, SD 1.51). Interrater agreement for manual review was high (97.8%) with the Cohen's κ equal to 0.923 (95% CI 0.876 to 0.970). Reasons for disagreement included miscoding and different interpretation of uncertain results (eg, ‘possible CHF’ was coded as positive by one coder and negative by the other).

Within the test set, the NLP-based method yielded a sensitivity of 81.2%, a specificity of 95.3%, an F1 score of 0.761, and high agreement with the reference set (Cohen's κ 0.723). Among the 15 index items, the best results were obtained for diabetes (sensitivity 100% and specificity 85%), and the poorest for chronic pulmonary diseases (sensitivity 36% and specificity 100%). The latter was a relatively uncommon diagnosis, and low sensitivity was due to a missing CUI (‘chronic obstructive airway disease’), which was not included in our query and did not occur in our training data but occurred in the test data. The summed CCI score was not significantly different in the automated versus manual calculations (mean CCI score of 2.9 for automated method vs 2.6 for the manual approach, p=0.329) (figure 1).

Figure 1.

Scatter plot of summed Charlson comorbidity index scores calculated for the test set (N=70), using the automated method versus those calculated by manual chart review. The size of circles is in proportion to the number of patients for whom the respective scores were calculated by each method.

Compared with the reference standard for the test set, comorbidity data extracted using the ICD-9 codes recorded at the time of discharge had a sensitivity and specificity of 66.1% and 97.6%, respectively (F1 score 0.741). While the NLP-based method was more sensitive and slightly more accurate than the claims-based model, the summed CCI score was not significantly different when calculated using the claims data versus the reference set (p=0.109) or the NLP-based method (p=0.251).

Discussion

We were able to calculate a modified version of the CCI with high accuracy using a combination of NLP and tailored queries. Our method was able to calculate this summed score with minimal error, which can potentially allow automated predictions of patients’ outcomes including mortality, disability, readmission and length of stay.^{6 7}

Automating the CCI score using EHR was studied previously. Singh et al¹⁹ used a keyword search to find terms pertinent to each item on the CCI score, and then used an Excel formula to calculate the score. While they achieved high accuracy compared to manual data extraction, their method of correcting for negations was basic and their success could also be attributed partly to the availability of a specific query builder for their EHR system. In comparison, our method does not depend on a specific EHR system and uses NLP to deal with negations, temporality and certainty. In another study, Chuang et al¹⁸ used an approach that utilized the MedLEE NLP system for processing of the notes; however, their method did not gain a high accuracy on the test set and was evaluated only regarding a specific domain (pneumonia patients). However, their reference standard was based on ICD-9 codes, which may not be an ideal choice of reference standard (see below). In comparison, our method achieved a higher accuracy, probably because we used a different gold standard (manual chart review) and a more current version of the MedLEE NLP system, with improvements in disambiguating abbreviations and normalizing the terms to CUI.

Our findings indicate that obtaining comorbidity information from claims data is less effective, primarily because of a lower sensitivity. This can be due to situations in which chronic conditions (such as a history of myocardial infarction) were in the note but not in the claims data. Under-reporting of chronic comorbidities in the claims data has been demonstrated in previous research.^{23 24} These are necessary for the CCI score and the higher sensitivity of the NLP method provides a more complete list of comorbidities using medical records. Previous research has also shown that claims data is subject to coding errors and granularity issues.^{25 26} In addition, deriving comorbidities from claims data is only possible retrospectively, because these data become available after the patient's discharge, hindering the utility of claims data for comorbidity information in applications such as reducing readmission, when it is ideal to identify patients with higher chances of readmission before discharge so that they could be managed appropriately. Nevertheless, because both NLP-based and claims-based approaches showed high accuracy in capturing comorbidity information, we recommend using them as complementary sources of information whenever applicable.

Our study has limitations. While our sample size was statistically large enough, the small size of the training set for query development contributed to the low sensitivity of at least one of the queries (ie, chronic pulmonary diseases). Because we randomly divided the notes into training and test sets, the distribution of positives and negatives among the different conditions was not balanced in the two sets. For some conditions (eg, HIV) there were more positive cases in the training set than expected, and for others (eg, dementia) there were fewer. However, the majority of the terms in the queries were mainly determined using clinical knowledge and consensus and therefore the choice of terms for inclusion in the queries minimally depended on the number of training cases. While we achieved good results using a small training set, it is likely that we could achieve higher sensitivity and better overall performance by using a larger sample size. However, caution is needed to ensure that the larger sample size would not lead to ‘statistically significant’ differences that are not of clinical value. In addition, we only used admission notes in the current study, but intend to expand our model to use other types of notes in the future. Finally, we used a modified version of the CCI. This is not the first study to use a modified version of the CCI, as multiple previous studies have found that various modifications of the CCI predict outcomes across different populations.^27–29 Nevertheless, direct conclusions about the utility of our index in clinical studies cannot be inferred from the current finding, and the modification could have affected the performance of our method. Future research will include exploring ways to improve further the accuracy of our approach, demonstrating the utility of this approach in clinical studies, and reducing confounding in studies of patient cohort identification.

Conclusion

We were able to derive comorbidities automatically from narrative admission notes with high accuracy. Our method has a higher sensitivity and accuracy than determining comorbidities from claims data, and has an additional advantage of utility in prospective studies that need to identify phenotypes from medical records and correct for the confounding effects of comorbidities.