Extracting Semantic Lexicons from Discharge Summaries using Machine Learning and the C-Value Method

Abstract

Semantic lexicons that link words and phrases to specific semantic types such as diseases are valuable assets for clinical natural language processing (NLP) systems. Although terminological terms with predefined semantic types can be generated easily from existing knowledge bases such as the Unified Medical Language Systems (UMLS), they are often limited and do not have good coverage for narrative clinical text. In this study, we developed a method for building semantic lexicons from clinical corpus. It extracts candidate semantic terms using a conditional random field (CRF) classifier and then selects terms using the C-Value algorithm. We applied the method to a corpus containing 10 years of discharge summaries from Vanderbilt University Hospital (VUH) and extracted 44,957 new terms for three semantic groups: Problem, Treatment, and Test. A manual analysis of 200 randomly selected terms not found in the UMLS demonstrated that 59% of them were meaningful new clinical concepts and 25% were lexical variants of exiting concepts in the UMLS. Furthermore, we compared the effectiveness of corpus-derived and UMLS-derived semantic lexicons in the concept extraction task of the 2010 i2b2 clinical NLP challenge. Our results showed that the classifier with corpus-derived semantic lexicons as features achieved a better performance (F-score 82.52%) than that with UMLS-derived semantic lexicons as features (F-score 82.04%). We conclude that such corpus-based methods are effective for generating semantic lexicons, which may improve named entity recognition tasks and may aid in augmenting synonymy within existing terminologies.

INTRODUCTION

Over the last two decades, there have been many efforts to develop natural language processing (NLP) methods, resources, and systems for understanding clinical text in electronic health records¹. Semantic information contained within lexicons is useful for almost all aspects of NLP tasks, such as named entity recognition (NER), word sense disambiguation, and syntactic parsing. For example, MedLEE (Medical Language Extraction and Encoding System)^2,3, one of the most comprehensive clinical NLP systems, primarily relies on semantic patterns of clinical text. Therefore, it is critical for MedLEE to accurately identify semantic categories of words and phrases in clinical notes (also called semantic tagging). Semantic lexicons that map clinical terms to their associated semantic categories such as “disease” and “medication”, are valuable resources for various semantic tagging approaches, including both dictionary-based lookup methods and machine learning based named entity recognition (NER) algorithms.

One way to generate clinical semantic lexicons is to utilize existing knowledge sources in the medical domain, e.g., medical terminologies. The Unified Medical Language System (UMLS)⁴ is one of the most comprehensive knowledge sources in the medical domain. The Metathesaurus of UMLS contains concepts from over one hundred medical ontologies, and each concept is assigned to at least one of the 133 semantic types as defined by the Semantic Network in UMLS. Thus, the UMLS Metathesaurus is a very useful resource for semantic lexicons. Some studies showed that the UMLS had good coverage of concepts for certain types of clinical notes, e.g., emergency medicine chief complaints⁵. However, other studies have demonstrated that the UMLS does not adequately represent concepts from other corpora ^6,7. In Friedlin and Overhage’s study⁷, they showed that a large portion of concepts found in radiology x-ray reports, discharge summaries, and admission notes were poorly represented in the UMLS Metathesaurus as pre-coordinated concepts and post-coordination was often required. Furthermore, NLP researchers have tried to construct semantic lexicons from the UMLS Metathesaurus^8,9 and evaluated its uses in medical language processing¹⁰. Friedman et al.¹⁰ showed that the performance of MedLEE dropped significantly when UMLS-derived lexicons were used to replace MedLEE’s original semantic lexicons, which were generated by manual review of clinical corpus.

Another method for building semantic lexicons is to extract terms from a relevant corpus. Manual review generates high-quality lexicons but is very costly and time-consuming. Automated or semi-automated methods for building semantic lexicons from corpora have been investigated in the general English domain¹¹. Few of them have been applied to clinical corpus to build semantic lexicons for medical text processing. A relevant task is term extraction for medical vocabulary development. For example, Zeng et al.¹² investigated different corpus-based methods to extract terms for a consumer health vocabulary, and Harris and colleagues¹³ developed a method using both frequency and syntactic information to extract terms in the domain of functioning, disability, and health. Term extraction usually consists of two steps: 1) to extract candidate terms from a corpus; 2) to determine which terms should be included in a vocabulary (also called “termhood determination”). Different methods such as statistical-based¹⁴, linguistic-based¹⁵, and hybrid approaches such as the C-value method¹⁶ have been developed to extract and rank the most promising terms from corpus for further human review. C-value, the method selected in this study, is described in more detail below. Although some term extraction studies have been aimed at extracting terms in a specific class (e.g., protein binding terms¹⁷ or risk factors of Metabolic Syndrome¹⁸), most term extraction studies have not linked extracted terms to specific semantic types. NER is another closely related task, which identifies word boundaries of specific types of named entities (analogous to semantic types) in text. Partially due to the lack of annotated corpus, limited studies have been done on machine learning based clinical NER methods. A prominent example is the 2010 i2b2 clinical NLP challenge¹⁹, which had its goal as extraction of three types of entities: Problems, Treatments, and Tests from discharge summaries. However, terms extracted from the i2b2 NER task cannot be used as lexical terms immediately, as their boundaries were defined by specific syntactic rules. For example, they need to be complete noun phrases; therefore articles and possessives were included in those terms. The complete documentation for these rules is available in the annotation guideline of the challenge (https://www.i2b2.org/NLP/Relations/).

In this study, we proposed a new method for building semantic lexicons from clinical corpora. Our method combines a machine learning based NER method with the C-Value termhood determination algorithm. We applied the method to a 10-year corpus of discharge summaries and built a semantic lexicon for three semantic groups (Problem, Treatment, and Test) used in the 2010 i2b2 challenge. Our evaluation showed that 84% new terms extracted from the corpus were meaningful clinical terms and the corpus-derived semantic lexicons improved the performance of a clinical NER system.

METHODS

We developed a corpus-based method for building semantic lexicons in the clinical domain. We applied the method to a 10-year collection of all typed and dictated discharge summaries generated at Vanderbilt University Hospital (VUH) to derive a list of lexical terms associated with three semantic groups: Problem, Treatment, and Test. We denoted this term collection LEX-DSUM. Meanwhile, we also derived lexicons from the UMLS Metathesaurus for the same three semantic groups, called LEX-UMLS. To assess the quality of corpus-derived lexicons, we compared those two lexicon files and manually reviewed 200 randomly selected lexical terms that were in LEX-DSUM and not in LEX-UMLS. In addition, we applied both lexicon files as dictionary sources to a machine learning based NER system for the concept extraction task of 2010 i2b2 NLP challenge, and evaluated their effects on performance respectively.

Data Sets

We used two data sets in this study. One is an unlabeled corpus of VUH discharge summaries dated between 2000 and 2009. These notes were selected from the Synthetic Derivative (SD), a de-identified copy of VUH electronic medical records²⁰. There were over 600,000 notes in total. The other data set was the set of notes derived from the 2010 i2b2 clinical NLP challenge¹⁹ (https://www.i2b2.org/NLP/Relations/). The i2b2 data set contains 826 clinical notes (including discharge summaries and some progress notes) from three institutions: Partners HealthCare, Beth Israel Deaconess Medical Center, and University of Pittsburgh Medical Center. Each clinical note was manually reviewed and entities for “Problem”, “Treatment” and “Test” were annotated according to the guideline. In the challenge, 349 annotated notes were used for training, and the remaining 477 notes were used for testing. The test set contained 45,009 entities.

The corpus-based method for building semantic lexicons

Our method consists of two steps: 1) extract candidate terms for each semantic type; 2) determine the inclusion of candidate terms. Figure 1 shows an overview of the method. In step 1, we applied a Conditional Random Field (CRF) based NER system to the corpus of VUH discharge summaries, generated candidate terms for each semantic type, and normalized candidate terms based on a set of rules. In step 2, the C-Value algorithm was used to score and rank multi-word terms and the UMLS was used to determine single-word terms. Using a heuristic cutoff, we finalized terms that were included in our corpus-derived semantic lexicon file (LEX-DSUM).

Figure 1.

Overview of the two-step corpus-based method for building semantic lexicons.

Candidate Term Extraction:

We have developed a hybrid clinical NER system called MedNET (Medical Named Entity Tagger)²¹ based on the CRF algorithm, which can identify Problem, Treatment, and Test entities from clinical text. When features from existing NLP systems such as MedLEE and KnowledgeMap Concept Identifier (KMCI)²² were used, MedNET achieved a best F-score of 0.8391 in the concept extraction task of 2010 i2b2 NLP challenge, which was ranked second among 20+ participating teams in the challenge. In this study, we used MedNET to identify candidate terms of Problem, Treatment, and Test from the 10-year discharge summaries. Specifically, we built the MedNET classification model based on 349 annotated notes in the training set of the i2b2 challenge. We used all features described in ²¹, except outputs from MedLEE and KMCI. We then processed the entire VUH discharge summary corpus using MedNET and collected all entities for Problem, Treatment, and Test, as predicted by MedNET, which formed the pool of candidate terms. As we mentioned previously, i2b2 guideline required that entities had to be complete noun phrases or adjective phrases. Thus, articles and possessives were often included in candidate terms. We defined a set of rules to normalize candidate terms, by removing articles, possessives, etc. For example, in the sentence “An approximately 1 cm lesion on the left was noted”, the candidate term predicted by MetNET was “an approximately 1 cm lesion on the left”, which was then normalized to “lesion on the left”. The final outputs of Step 1 were these normalized candidate terms, each of which was associated with one of the three semantic groups: Problem, Treatment, and Test.

Termhood determination:

In this step, we separated normalized candidate terms into two categories: single-word vs. multi-word terms, and applied different strategies for inclusion. For a single-word term, we would include it if the term with its associated semantic type was also found in the UMLS derived lexicon file. The assumption is that the UMLS has good coverage on single-word clinical concepts. The challenge of termhood determination is mostly when considering multi-word terms. For example, our candidate terms contained phrases such as “chest pain”, “chronic chest pain”, “new chest pain”, and “complaints of chest pain”. How can we determine that we should include first two terms into our lexicon, but not the latter two terms? In this study, we determined the inclusion of multi-word terms by scoring and ranking each of them using the C-Value approach¹⁶. The C-Value defines a measure based on frequency of a term and its longer terms. For a candidate multi-word term a, if it is a string of maximum length, its C-Value is determined by its length and frequency in the corpus; if a is a part of longer candidate term, C-Value considers the length of a, the frequency of a, and the frequency of longer strings that contain a. Equation 1 shows how to calculate C-Value for a term a. A high C-value means that the term is important and we may consider including it to the domain vocabulary. We calculated C-Value for every normalized candidate multi-word term and ranked them in a descending order. We set a relative high cutoff value to ensure the quality of corpus-derived semantic lexicons, selecting a C-Value of 20 as the cutoff value for inclusion after a brief review of the rankings. Thus, all candidate terms with C-Value<20 were excluded. Lastly, we combined these single-word terms determined by UMLS with multi-word terms determined by the C-Value to form the final semantic lexicon file (LEX-DSUM).

$$\mathit{CValue}(a)=\{\begin{array}{l}{\text{log}}_2\left|a\right|.f(a),\mathit{if} a \mathit{is} \mathit{not} \mathit{nested}\hfill \\ {\text{log}}_2\left|a\right|.(f(a)-\frac{1}{P(T_a)}\sum _{b\in T_a}f(b)),\mathit{if} a \mathit{is} \mathit{nested}\hfill \end{array}$$

Equation 1. C-Value calculation for candidate term a, where f (a) is the frequency of term candidate a, | a | is the length of a, T_a is the set of extracted candidate terms that contain a, and P(T_a) is the total number of longer candidate terms that contain a.

The UMLS-derived semantic lexicons

In addition to the corpus-based semantic lexicon (LEX-DSUM), we also constructed a semantic lexicon file from the UMLS Metathesaurus (2011AA version), to serve as a baseline for the comparison. We defined a mapping between UMLS Semantic Types and three groups (Problem, Treatment, and Test) in this study. Table 1 shows the details of the mapping. UMLS terms associated with UMLS Semantic Type given in Table 1 were assigned to the corresponding semantic category (e.g., Problem for a “Disease or Syndrome” term). We called this UMLS derived semantic lexicon file “LEX-UMLS”. UMLS terms with other semantic types were ignored.

Table 1.

Grouping semantic types in UMLS to generate semantic lexicons for Problem, Treatment, and Test.

Category	Semantic types in UMLS
Problem	Pathologic Function, Disease or Syndrome, Mental or Behavioral Dysfunction, Cell or Molecular Dysfunction, Congenital Abnormality, Acquired Abnormality, Anatomical Abnormality, Neoplastic Process, Sign or Symptom, Injury or Poisoning, Pathologic Function, Experimental Model of Disease, Virus, Bacterium, Finding
Treatment	Therapeutic or Preventive Procedure, Medical Device, Steroid, Biomedical or Dental Material, Biological Active Substance, Pharmacologic Substance, Antibiotic, Clinical Drug, Drug Delivery Device
Test	Laboratory Procedure, Laboratory or Test Result, Diagnostic Procedure, Organism Attribute, Organism function, Physiologic Function, Mental Process

Evaluation

To evaluate the quality and usefulness of LEX-DSUM, we conducted two studies. The first evaluation was to assess the quality of LEX-DSUM. We randomly selected 200 new terms that were in LEX-DSUM, but not in LEX-UMLS. A physician manually reviewed these 200 terms to determine if a term was clinically meaningful and if the semantic category was correct. To decide if a term had a clinical meaning and how it was represented in the UMLS, we defined three categories: 1) false positive terms – should not be included; 2) new terms that are lexical variants of existing UMLS concepts; and 3) new terms that are either new concepts or precoordinated terms that can be represented by post-coordination of existing UMLS concepts. To facilitate the assignment of these categories to terms, we mapped all 200 terms to the UMLS concepts using KMCI, which has been used for a decade to map Vanderbilt clinical notes to UMLS CUIs²². The physician manually reviewed KMCI’s outputs and assigned each term to one of the three categories. In addition, terms not immediately mapped to UMLS concepts were manually interrogated for possible UMLS matches by querying a database of UMLS concepts.

In addition, we also evaluated the usefulness of the corpus-derived lexicon in the concept extraction task of 2010 i2b2 NLP challenge. The MedNET system²¹ that we developed for the i2b2 challenge allows users to supply a dictionary containing semantic information of words or phrases. The system then runs a lookup program to use the lexicon’s semantic labels in the CRF classifier. In this evaluation, we kept all other settings of MedNET identical to the i2b2 challenge, but supplied either LEX-DSUM or LEX-UMLS for feature generation. In addition, we also included the result when no semantic lexicon was used for MedNET. The classification models were built using the 349 training notes and performance was evaluated using the test set of 477 notes in the challenge. For each set of features either from LEX-DSUM or LEX-UMLS, we reported the micro-averaged Precision, Recall, and F-measure for all concepts using the exact matching method as defined in the challenge¹⁹. To assess whether the mean F-measures of both NER systems (using LEX-DSUM or LEX-UMLS) are statistically significantly different, we further conducted a statistical test based on results from bootstrapping. From the test set, we randomly selected 1000 sentences with replacement for 200 times and generated 200 bootstrapping data sets. For each bootstrapping data set, we evaluated and measured F-measures for both LEX_DSUM and LEX_UMLS based NER systems. We then used Wilcoxon signed rank test,²³ a nonparametric test for paired samples, to determine if the difference between F-measures from LEX-DSUM and LEX-UMLS based NER systems is statistically significant (p-value < 0.05). In addition, we recorded the running time of the dictionary lookup program on 1000 notes with each semantic lexicon.

RESULTS

Statistics of two lexicon files

From the 10-year discharge summaries corpus, we generated a semantic lexicon of 70,896 terms from VUH discharge summaries. Table 2 shows counts of terms from the UMLS-derived and corpus-derived semantic lexicon files. The LEX-UMLS contained 2,702,400 terms, which was much larger than the number of terms in LEX-DSUM (70,896). Using an exact string matching approach, we found that a large portion of terms (44,957/70,896=63.4%) in LEX-DSUM were not in LEX-UMLS, indicating many new terms.

Table 2.

Comparison between LEX-UMLS and LEX-DSUM

	Number of terms in LEX-UMLS	Number of terms in LEX-DSUM	Number of terms in both LEX-DSUM and LEX-UMLS	Number of terms in LEX-DSUM, but not in LEX-UMLS
Problem	1,050,929	38877	13616	25261
Treatment	1,588,303	15832	8938	6894
Test	63,168	16207	3385	12822
All	2,702,400	70896	25939	44957

Manual Evaluation of New Terms

To assess the quality of new terms detected from the corpus, 200 terms were randomly selected from the pool of 44957 new terms. Manual review revealed that 34 (17%) terms were incorrect and should not be included in the semantic lexicons. The remaining 166 (83%) of the predicted terms were clinically meaningful terms. Among them, 117 (59%) were new terms, either terms not in the UMLS or new pre-coordinated terms. An additional 49 (25%) were lexical variants of existing UMLS concepts. Thirty-four terms (17%) were judged to be not either a problem, treatment, or test. Of note, some of these were actually valid clinical concepts of other semantic types, such as “neck of the gallbladder.” Table 3 shows the results of manual review of 200 new terms. For those clinical meaningful terms, the accuracy of semantic tags was 152/166 (92%).

Table 3.

Result of analysis on generated lexicon from the corpus.

Category	Count (out of 200)	Example
New terms (precoordinate d terms or new concepts)	117 (59%)	degree + clinical concept: “severe psoriasis”, “full exam” body location + clinical concept : “ fracture of the left scapula”, “lower extremity fasciotomies” combination of multiple clinical concepts: “cardiac or pulmonary disease”, “distal tibia and fibula fractures” new concepts: “thrombus formation”, “road rash” other pattern e.g. “scant bibasilar crackles”, “parkland tubal ligation”
New lexical variants of existing UMLS concepts	49 (25%)	change order of words e.g. “sputum afb”, “pelvic thrombophlebitis” morphological variant e.g. “peak flows” abbreviation e.g. “ab pain”
False positive terms	34 (17%)	incorrect word grouping: “atenolol tenormin”, “meals lansoprazole”, slight increase”, “renal insufficiency creatinine” valid clinical terms, not a problem, test, or treatment: “neck of the gallbladder”, “arterial access site”

Evaluation of Semantic Lexicons on i2b2 Concept Extraction Task

Table 4 shows the performance of MedNET on the test set of the concept extraction task of 2010 i2b2 NLP challenge, when different semantic lexicons were used as dictionaries for semantic features. The baseline performance of MedNET without semantic lexicon features was 81.07% (F-measure), and its performance was improved when either lexicon file was used. The MedNET system achieved a better F-measure (82.52%) using the corpus-derived lexicon (LEX-DSUM) than with the LEX-UMLS lexicon (82.04%). The Wilcoxon signed rank test based on bootstrapping data showed that the improvement of F-measure (LEX-DSUM over LEX-UMLS) was statistically significant (p-value < 0.05). Given the large number of entities in the test set (45,009), an increase of 0.48% in F-measure was a large improvement. Notably, the file size of LEX-DSUM (3.13MB) was much smaller than LEX_UMLS (149.2MB), which reduced the running time of the dictionary lookup program from 52.78 seconds to 3.55 seconds, for 1000 notes.

Table 4.

Effects of semantic lexicon on the 2010 i2b2 concept extraction task.

Semantic Lexicon	Precision	Recall	F-measure	Lexicon Size (MB)	Lookup Program Running Time for 1000 notes (s)
No Lexicon	84.87%	77.61%	81.07%	N/A	N/A
LEX-UMLS	85.12%	79.17%	82.04%	149.2	52.78
LEX-DSUM	85.28%	79.93%	82.52%	3.13	3.55

DISCUSSION

In this study, we developed a new method for building semantic lexicons from clinical corpora. We combined a CRF-based NER system with the C-Value termhood determination algorithm and extracted lexicons from discharge summaries for three semantic groups: Problem, Treatment and Test. Manual review showed that most discovered new terms were indeed valid clinical entities. Most new terms represented either pre-coordinated clinical terms or alternate lexical forms not found in current medical terminologies (as represented in the UMLS). This method and lexicon could represent a great resource for clinical NLP. We also demonstrated the usefulness of the corpus-derived lexicon in an NER task. The classifier using the corpus-derived lexicon achieved a better F-measure than the one using the UMLS-derived lexicon, and contained only 2.62% as many terms. Consequently, it was nearly two orders of magnitude faster as well.

Although the lexicon file from UMLS contained near three million terms, only a small portion actually occurred in the VUH discharge summaries based on exact string matching. Our analysis showed that only 89,811 UMLS terms could be exactly or partially mapped to candidate terms predicted by MedNET from discharge summaries. This finding indicates a gap between existing terminologies and clinical corpora. Many terms collected in the UMLS may never occur in clinical text and many terms appearing in clinical text may not be in the UMLS. Medical terminologies are not designed for text processing purposes and linguistic variants may not be of interest to a specific terminology. Our manual analysis showed that many new terms in LEX-DSUM were either not contained within the UMLS or represented by post-coordination of existing UMLS concepts. These findings are consistent with those by Friedlin and Overhage⁷.

Our study also demonstrated a few advantages of the corpus-derived semantic lexicon. First of all, it had better coverage than the UMLS-based lexicon, which could be important for many NLP tasks. As shown in Table 4, the size of lexicon file derived from the corpus was much smaller than that of the UMLS-based lexicon file. Although additional terms in LEX-UMLS do not hurt the mapping performance for this application, it increases the search space and costs time. Our results showed that the running time for a lookup program was decreased by 93.3% when we switched the dictionary from LEX-UMLS to LEX-DSUM, which could be very important for real-time clinical NLP applications.

We analyzed errors in semantic lexicons. Errors in semantic category assignment were often caused by ambiguity of clinical entities. The current MedNET system does not have advanced semantic disambiguation modules. Even though we used a high cutoff value for term inclusion, we still had some false positive terms, such as “new symptoms.” Such terms occur because of their high corpus frequency. As “new symptoms” is not contained within longer strings, its score is determined by its frequency and length, according to the C-Value equation. To resolve this, we may consider heuristic rules to re-define the score calculation, such as penalizing words (such as “new”) that modify various entities. We also noticed that some errors were associated with the i2b2 annotation guideline. For example, the guideline included unspecific entities such as “above-stated issues”, which was annotated as an incorrect entry in our evaluation.

This study has limitations. Although our method is generalizable, we focused on three high-level semantic groups, for which a previously annotated corpus is available for training the model in step 1. When we move to build lexicons for fine-grained semantic types, additional semantic annotation will be needed. Another solution is to investigate semi-supervised methods for building semantic lexicons from corpus, as described in ¹¹. In this study, we heuristically determined the cutoff value for C-Value based on our observation. We plan formally investigate C-Value optimization methods for clinical text. We used MedNET that was trained on i2b2 training set to generate LEX-DSUM from VUH discharge summaries, which may correlate to the high performance of LEX-DSUM in the evaluation of the i2b2 concept extract task. However, the test set was independent and used for evaluating performance only. Therefore we believe the improvement of LEX-DSUM in Table 4 was mainly due to the quality of the corpus-based lexicon.

CONCLUSION

In this study, we developed a new method that combines the machine learning based NER method with the C-Value termhood determination algorithm to build semantic lexicons from clinical corpus. Using the method, we built a lexicon for three semantic groups (Problem, Treatment, and Test) from a corpus of discharge summaries. Our evaluation showed that the method could extract new concepts that were not contained within the UMLS and the corpus-derived semantic lexicons improved the performance of a clinical NER system, indicating the potential benefits for clinical NLP research.