Evaluating the Impact of Dictionary Updates on Automatic Annotations Based on Clinical NLP Systems

Abstract

Concept encoding, which maps text spans to concepts in standard terminologies, is a critical component in clinical natural language processing (NLP) systems to allow semantic interoperability with other clinical applications. A majority of clinical NLP systems adopt dictionary or lexicon based approaches and the performance of concept encoding is often evaluated using a human created gold standard generated with reference to the most up-to-date standard terminologies available at the time of gold standard creation. With the advance of medical science, standard terminologies or dictionaries can evolve. However, it remains unknown whether the dictionary updates will impact the performance of concept encoding. In this study, we evaluated the annotation performance of two clinical NLP systems, cTAKES and MedXN based on updated dictionaries to gain further insights. Specifically, we compared the automatic annotation results with previously manually generated gold standards. The results of our study demonstrate the annotation changes based on dictionary updates in clinical NLP systems and that it is necessary to do temporal management for gold standards, which raises the need for appropriate terminology management tools for back version compatibility to update gold standards.

Introduction

The widespread adoption of electronic health records (EHRs) has led to an unprecedented increase in the volume of valuable information that is stored in unstructured clinical notes. A great amount of information, including medication information, is largely embedded in the clinical free text. To fully leverage the benefits of the EHR, tremendous efforts have been dedicated to develop clinical natural language processing (NLP) systems over the past decade, with the aim of extracting various types of information from clinical notes and mapping and encoding the extracted information to standard terminologies. Existing clinical NLP systems such as MedLEE¹, MetaMap², cTAKES³, and MedEx⁴ have been developed and applied on a variety of clinical information extraction tasks to facilitate patient care and clinical research^5,6. The clinically meaningful representations of medical concepts are the key for health analytic applications.

The development of NLP systems requires knowledge about words. To be practical and habitable, NLP systems must be furnished with a substantial lexicon, which covers a realistic vocabulary and provides the specific kinds of linguistic knowledge required for certain applications⁷. In the clinical domain, a majority of existing NLP systems are dictionary-or lexicon- based for their named entity recognition (NER) components. Specifically, they employ pattern matching based on a dictionary look-up to determine what information to extract and how to encode such extracted information. MedLEE (Medical Language Extraction and Encoding System), one of the earliest and most comprehensive clinical NLP systems, parsed the input text using a predefined semantic lexicon¹. MedEx, a medication information extraction system, utilized drug lexicon files generated from RxNorm in the semantic tagging step⁴. MetaMap, a system developed by the National Library of Medicine to map biomedical free text to UMLS (Unified Medical Language System) Metathesaurus concepts, performed the parsing process based on the SPECIALIST lexicon². CLAMP (Clinical Language Annotation, Modeling, and Processing), a highly customized NLP system, has a dictionary-based named entity recognition (NER) component with a comprehensive lexicon collected from multiple resources such as UMLS⁸. cTAKES, a widely applied NLP system developed by Mayo Clinic, detects and extracts named entities from clinical notes using a dictionary compiled from a subset of UMLS⁹. MedXN, also developed by Mayo Clinic, uses RxNorm as its dictionary to extract and normalize medication information¹³. As a further example, both rule-based and hybrid systems in the 2009 i2b2 NLP challenge built their vocabularies from publicly available knowledge resources¹⁰. For most teams, the recognition of medication names was primarily dictionary-based. It has been observed that the quality of dictionary might have a direct impact on the quality of the results¹¹. It is very clear that an accurate and up-to-date dictionary plays an important role in medical concepts recognition and extraction.

The robustness of clinical NLP systems is typically evaluated using high-quality gold standards¹², which are often generated through manual annotation based on the up-to-date dictionaries. One of the benefits of using terminologies is that they provide a standardized way for annotation, allowing for high inter-annotator agreement and reducing ambiguities through normalization. As medical science evolves, however, dictionaries must also be changed so as to remain up to date. One example of such a change is that some concepts become obsolete and new concepts can be introduced. It thus follows that these changes could possibly also influence the annotation performance of NLP systems.

In this study, we evaluated the impact of updated dictionaries on the annotations produced by two NLP systems: MedXN¹³ and cTAKES⁹. Specifically, we updated the base dictionaries for MedXN and cTAKES with the newly released version of UMLS and RxNorm. We then compared the automatic annotations generated by cTAKES on medical entities (including anatomical site, disorder, procedure, and medication) and MedXN on various levels of medication information with the gold standards. Conclusions were drawn through error analysis.

Methods

The method in this study consisted of the following steps: (1) dictionary updates, (2) automatic annotation by cTAKES and MedXN, (3) comparison of annotation results with gold standards, (4) results analysis. The overall method is shown in figure 1.

Figure 1.

The overall method

Materials and datasets

MedXN, a UIMA (Unstructured Information Management Architecture) based NLP system, is designed not only to extract medication names and other related attribute information (e.g., dosage, strength, frequency, route, duration, form) but also normalize the extracted information to the most appropriate RxNorm name (RxCUI)¹³. Using RxNorm as its dictionary, MedXN applied a dictionary lookup method using the Aho-Corasick algorithm to match medication name to its corresponding RxCUI and attributes to corresponding RxNorm term types, such as SCD, SBD. The version of RxNorm bundled within the latest public release of MedXN is the 2013 RxNorm release. To update MedXN’s dictionary, the most recent release of RxNorm (06/04/2018) was obtained from the NLM’s official website. As a standardized nomenclature of medications, RxNorm organizes and presents clinical drugs and drug delivery devices based on 11 external source vocabularies. Medication strings from various medication sources (e.g., DrugBank, NDFRT) with the same meaning were linked by concepts. RxNorm comes as a set of nine Rich Release Format (RRF) files, each of which contains a specific subset of information. We used the RXNCONSO.RRF table for dictionary update in this study, which provided information on concepts, concept names, and their sources.

cTAKES is a UIMA based system which consists of different modules including a sentence boundary detector, tokenizer, normalizer, part-of-speech tagger, shallow parser, and named entity recognizer. Specifically, the NER component implements a terminology-agnostic dictionary look-up algorithm with a noun-phrase look-up window. The dictionary was generated from a subset of the UMLS that included SNOMED-CT, SNOMED-CT-US, and RxNorm. SNOMED-CT, one of its largest source vocabularies, was first introduced in 2003 and has been a source vocabulary for the UMLS since UMLS version 2004AA. SNOMED-CT is the most comprehensive clinical terminology in the world, encompassing more than 300,000 concepts. As SNOMED-CT is distributed with the UMLS, and because the cTAKES distributable package includes a dictionary generator explicitly for generating from the UMLS, we used the UMLS as the source for updating cTAKES’ dictionary. The most recent release of the UMLS as of the time of writing (2018AA) was downloaded from the NLM official website.

For MedXN, the development set was used for calibration of updated dictionaries and test set for evaluation. The development set consisted of 659 manually annotated medication mentions along with their attributes from 159 randomly selected clinical notes. The test set contained 397 mentions of medications from 26 clinical notes. Both development and test sets were originally annotated using the 2013 release of RxNorm.

The evaluation dataset for cTAKES consisted of 151 randomly selected clinical notes with named entity annotations in over 15 UMLS categories¹⁴. The gold standard was manually annotated using the UMLS 2006AD release. We further limited the reference standards to only the source vocabularies SNOMED-CT (and its variants) and RxNorm.

Dictionary update

MedXN uses five look-up dictionaries: medication name dictionary, full medication name dictionary, RxCUI-represented full name dictionary, dose form dictionary, and false medication dictionary¹³. The medication name dictionary contained medication names compiled from RxNorm ingredients and brand names with ‘IN’ (e.g., ‘Aspirin’), ‘PIN’ (e.g., ‘Acetylsalicylate Sodium’), ‘MIN’ (e.g., ‘Acetaminophen / Aspirin’), and ‘BN’ (e.g., ‘Anacin’) RxNorm term types. In order to fully capture the lexical and semantic variants of medication names, we included the medication strings that have the same RxCUIs as the above medications. In addition, the manually compiled abbreviations of medication names were included. The full medication name dictionary contained the following RxNorm term types: SCDC (e.g., ‘Aspirin 1.5 MG/ML’), SCDF (e.g., ‘Aspirin Oral Solution’), SCD (e.g., ‘Aspirin 975 MG Oral Tablet’), SBDC (e.g., ‘Aspirin 1.5 MG/ML [Platet]’), SBDF (e.g., Aspirin Oral Solution [Solprin]), SBD (e.g., ‘Aspirin 81 MG Oral Tablet [Heartline]’), and SY (e.g., ‘Acetylsalicylic Acid’). On the basis of the full medication name dictionary, the RxCUI-represented full name dictionary was generated by replacing the drug names and dose forms with RxCUIs. The dose form dictionary is a list of RxNorm dose forms generated from the RXNCONSO.RRF table with “DF” term type. The false medication dictionary contains potentially false medication names, which are the medication strings in RxNorm from other sources but are highly ambiguous English words, such as “his”, “date”.

MedXN was first tested on the development set for system calibration and refinement. Specifically, based on the annotation results in the development set, we removed highly ambiguous words and abbreviations from the medication name dictionary and also added such words into the false medication dictionary.

Since cTAKES includes a dictionary generator that uses the UMLS, we used a UMLS installation with the SNOMEDCT, SNOMEDCT-US, and RxNorm source vocabularies with the default set of UMLS semantic types (T019->T034, T037, T040->T050, T056->T061, T109, T110, T114->T119, T121->T131, T184, T190, T191, T195->T197, T200, and T203).

Evaluation

We compared the new dictionary with the old dictionary of MedXN regarding the number of medication names, full medication names (e.g., SCD), dose forms, and false medications. Specifically, the first three metrics were determined based on the number of RxCUIs contained within the dictionary. The last one was counted based on the unique number of medication strings.

The test set was used for MedXN annotation performance evaluation. Precision, recall, and F-measure were used as evaluation metrics. We evaluated the annotation performance of MedXN on medication names and other related attributes, including dosage (e.g., ‘2’ in ‘2 tablets’), strength (e.g., ‘30 mg’), form (e.g., ‘tablet’, ‘capsule’), route (e.g., ‘oral’, ‘topical’), frequency (e.g., ‘daily’, ‘twice a day’), duration (e.g., ‘for a month’), DrugRxCUI, and NormRxCUI, where DrugRxCUI means the RxCUI for drug alone (e.g., RxCUI of ‘Aspirin’ (=1191) from ‘Aspirin 81 mg oral tablet’) and NormRxCUI stands for the RxCUI for full medication information (e.g., 243670 for ‘Aspirin 81 mg oral tablet’). We calculated precision, recall, and F-measure at two levels: exact match and partial match. The exact match requires that both span and string match while partial match allows for both string and overlapped span match. Error analysis was conducted to identify the source of errors. We also compared the annotation performance of MedXN after dictionary with that using the old dictionary.

For cTAKES annotation evaluation, we focused on four semantic categories of named entities: anatomical site, procedure, disorder, and medication. Precision, recall, and F-measure were used as evaluation metrics. Manual review was conducted for error analysis of incorrectly annotated named entities in four categories to further identify the sources of errors. Since the performance of cTAKES based on the old dictionary is not available, our evaluation did not include the comparison between system performances based on old and new dictionaries.

Results

MedXN

The results of the dictionary comparison are shown in Table 1. From Table 1, we can see that the new dictionary has a relatively large increase in “IN” and “PIN” term types, the number of which is nearly triple of those in the old dictionary. On the other hand, the number of “SBDF” items decreased compared with the old dictionary. The comparison results showed that the updated dictionary has a larger and more comprehensive coverage for medication names.

Table 1.

Comparison results of the new and old dictionary of MedXN

	Medication name				Full medication name							Dose form	False meds
	IN*	BN*	PIN*	MIN*	SCD*	SCDC*	SCDF*	SBD*	SBDC*	SBDF*	SY*
Old dictionary	5017	15777	1572	3739	33249	25712	13904	17707	18883	15460	4114	155	103
New dictionary	17662	27646	5738	6277	36188	27129	14790	18830	19009	14690	6841	176	199
Percentage increase	2.52	0.75	2.65	0.68	0.09	0.06	0.06	0.06	0.007	-0.050	0.66	0.14	0.93

^*IN is ingredient, BN is brand name, PIN is precise ingredient, MIN is multiple ingredients, SCD is semantic clinical drug, SCDC is semantic clinical drug component, SCDF is semantic clinical drug form, SBD is semantic brand name, SBDC is semantic branded drug component, SBDF is semantic branded drug form, SY is synonym.

The annotation performance of MedXN after the dictionary update is shown in Table 2. We compared the results with the previous study¹³, the comparison results are also shown in Table 2, where minus sign (-) denotes decrease and plus sign (+) means increase in the performance. According to the results in Table 2, MedXN achieved a good performance in annotating medication name, with the F-measure of 0.904. It performed poorly in detecting duration information, achieving the F-measure of 0.667, which is mainly due to inexact span match. However, compared with the previous results, MedXN with the new dictionary had a higher F-measure in both exact (0.667) and partial match (0.750) of duration.

Table 2.

The annotation performance of MedXN with updated dictionaries on the test data

Type	Exact match			Partial match
	Precision	Recall	F-measure	Precision	Recall	F-measure
Medication	0.892 (-0.033)	0.917 (-0.010)	0.904 (-0.022)	0.931 (-0.051)	0.951 (-0.016)	0.941 (-0.034)
Dosage	0.825 (-0.085)	0.842 (-0.085)	0.833 (-0.086)	0.825 (-0.085)	0.842 (-0.087)	0.833 (-0.086)
Strength	0.832 (-0.125)	0.864 (-0.079)	0.848 (-0.095)	0.842 (-0.137)	0.876 (-0.066)	0.859 (-0.101)
Form	0.785 (-0.159)	0.840 (-0.114)	0.812 (-0.142)	0.785 (-0.159)	0.840 (-0.142)	0.812 (-0.151)
Route	0.876 (-0.098)	0.888 (-0.060)	0.882 (-0.066)	0.897 (-0.093)	0.909 (-0.051)	0.903 (-0.071)
Frequency	0.766 (-0.074)	0.748 (-0.092)	0.757 (-0.083)	0.842 (-0.112)	0.822 (-0.125)	0.832 (-0.119)
Duration	0.728 (-0.085)	0.615 (0.034)	0.667 (0.086)	0.818 (0.104)	0.692 (0.104)	0.750 (0.105)
DrugRxCUI	0.887 (-0.028)	0.912 (-0.003)	0.899 (-0.016)	0.921 (-0.006)	0.940 (0.003)	0.931 (-0.001)
NormRxCUI	0.836 (-0.070)	0.683 (-0.079)	0.752 (-0.076)	0.866 (-0.040)	0.699 (-0.126)	0.773 (-0.091)

We conducted an error analysis through manually reviewing the false negative and false positive mentions of medication names. The results of the error analysis are listed in Table 3. In total, there were 55 false positive and 34 false negative drug mentions.

Table 3.

Error analysis for MedXN on automatic annotations of medication information

	Total	Source of error
		Gold standard	Incomplete matching	Dictionary	New medications	System
False positives	55	24	8	12	11	0
False negatives	34	13	9	10	0	2

The sources of errors for false positives are analyzed as follows. The first source of error is from the gold standard, which is attributed to changes of RxCUIs, annotation granularity (span) and misspellings. Firstly, our gold standard was created in 2013, and some RxCUIs have been changed or retired since then. For example, the RxCUI for “Hyzzar” was changed from “823960” to “217681” and the RxCUI 284905 for “Tylenol Arthritis” has been retired. Secondly, the granularity of human annotation will influence the results. In the gold standard, “Nasonex nasal spray” was annotated as a medication name. However, this medication string does not exist in updated dictionaries. This also applies to annotations of drugs such as “Imdur sustained release” and “Hyzaar 100-25”. Highly detailed and granular annotation in gold standards can potentially lead to the inaccurate match with automated annotations. Thirdly, misspelled words in the gold standard resulted in mismatches with automated annotation using updated dictionaries. For instance, “Aspirin” was misspelled as “Aspir” and “Tramadol” was misspelled as “Tramado” in the gold standard, while results from automated annotation were “Aspirin” and “Tramadol”. Therefore, the automated annotations cannot be matched to the gold standards.

The second source of error is incomplete match, i.e., resulting from the separate extraction of ingredient and brand names. For example, instead of extracting “Lisinopril [PRINIVIL/ ZESTRIL]” as a whole, MedXN detected them separately as follows: “Lisinopril [PRINIVIL/” and “ZESTRIL”.

The third source of error came from the dictionaries, including abbreviations we failed to generate based on expert knowledge, medication strings not existing in the dictionaries, and also some mentions of drugs that can be confused with other categories of entities, such as lab test (e.g., staphylococcus aureus). Firstly, due to the flexibility of human language, updated dictionaries cannot capture all the orthographic and morphological variations of medication names. For example, the medication string variations under the concept “807279” cannot capture the string “(Td) Tetanus-Dipth Toxoid-Td”. Secondly, updated dictionaries failed to generate the abbreviations for some drugs appearing in the test set. For example, “Triamterene-HCTZ” is the abbreviation for “triamterene-hydrochlorothiazide”, which was not covered by the new dictionaries.

The last source of error was caused by new medications which were not annotated in the gold standard but detected by updated dictionaries, such as “Lubriderm”, “Eucerin”, “Calcium 600 + D”. Also, some of the false positives were mentions of lab results such as “staphylococcus aureus”, appearing in the following text: “CULTURE, BACTERIA: STAPHYLOCOCCUS AUREUS , 4+ , MRSA”.

The sources of errors for false negatives came from issues of gold standard, incomplete matching, and dictionary, similar as mentioned above. Beyond these three types of errors, another error source for false negatives is from the system itself, which means MedXN system failed to detect and recognize the drug names.

cTAKES

Our gold standard consisted of 8,365 mentions of four categories of named entities, among which 50.33% are mentions of “Anatomical Site”, 30.40% are “Disorder”, 16.40% are mentions of “Procedure”, and only 2.87% are “Medications”. The named entity annotation performance of cTAKES regarding the four categories is shown in Table 4. Among the four categories, “Anatomical site” has the highest F-measure of 0.725. The precision of “Anatomical site” is over 0.88. However, the “Medications” category has the lowest precision score and also the lowest F-measure. The F-measure for four categories is 0.664.

Table 4.

The annotation performance of cTAKES with the updated dictionary on the test data in terms of four categories

	Precision	Recall	F-measure
Disorder	0.617	0.654	0.635
Anatomical site	0.881	0.616	0.725
Procedure	0.667	0.417	0.513
Medications	0.354	0.742	0.479
Overall	0.745	0.598	0.664

Through analyzing the false negatives annotations by cTAKES, we found that there were three types of error sources. Firstly, some medical entity strings including some abbreviations that do not exist in the current version of SNOMED-CT, which explains the low recall score (Recall: 0.598). In total, there were 1,319 distinct false negative mentions, among which 955 (72.40%) entity strings were not covered by the new dictionary (2018AA). Specifically, among these 955 false negative strings, 63 are abbreviations, such as “aso” which stands for “arteriosclerosis obliterans”, “cta” means “computed tomography angiography” or “CT angiography”. Secondly, there exist some discontinuous annotations in the gold standard (e.g., “muscle...weakness” in “muscle tenderness and weakness”). It is hard for cTAKES to produce the same discontinuous annotations as well as category assignment. Thirdly, some CUIs were not in use anymore. There were 826 distinct CUIs in the false negatives, among which 5 were obsolete. Even though this is a small number, it still caused problems for accurate annotation.

A total of 471 distinct false positive mentions of entities were found, among which 230 were mentions in “Disorder”, 81 in “Procedure”, 84 in “Anatomical site” and 76 in “Medications”. The sources of errors are as follows. Firstly, some entities were incorrectly annotated by cTAKES. For example, the text “difficulty” and “possible” were annotated as “Disorder” category. In the “Medication” category, some general English words such as “air”, “fat”, and “solid” were falsely annotated. Secondly, some entities were not annotated in the gold standard by human annotators. For example, if “carcinoma” appeared in the note for several times, only some of the mentions were annotated by human experts. Thirdly, some entities annotated by cTAKES with new dictionaries did not exist in the SNOMED-CT of UMLS 2006AD version, implying that the new dictionary has detected some new information. For example, “Bisphosphonate therapy” was entered into SNOMED-CT in 2017 and “Aaccular aneurysm” in 2009.

A total of 240 mentions of medications were in the gold standard, among which 209 were annotated using RxNorm. Among the false negatives, there were 21 distinct CUIs, 11 out of which were from RxNorm. Analyzing the false negative mentions of medications, we found out that most of the errors in RxNorm and SNOMED-CT were caused by abbreviations. For example, “ai” is the abbreviation for “Aromatase Inhibitors”, “er” for “Estrogen Receptors”, “th” for “Thyroid Hormones”. The existence of such abbreviations make for it hard for accurate concept encoding. Another source of error came from the misspellings. Misspellings are common in clinical notes due to the time-constraint nature of clinical setting. For example, “Arimidex” was misspelled as “Arimdex” and “Anastrozole” was misspelled as “Anastrzole”. Annotators made some inferences when annotating misspelled medication names. However, the new dictionary failed to detect and annotate these misspellings.

Discussion

In this study, we updated the dictionaries of two clinical NLP systems cTAKES and MedXN to investigate the impact of the dictionary updates on their resulting annotation performances. We evaluated cTAKES on a medical entity annotation task and MedXN on a medication information annotation task using previously generated gold standards. We did thorough error analysis to gain insights on the underlying reasons behind the discrepancies of the system performance after dictionary updates. Our results show that the automatic annotation performance based on clinical NLP systems relies on two factors: the dictionaries (i.e., coverage, newly introduced concepts, etc.) and gold standards. Our study showed that it is inevitable for the latest dictionaries to miss some old concepts. Since standard dictionaries or terminologies are evolving, dictionaries, gold standards, and concept encoding results need to be chronically managed. Even though standard terminologies such as RxNorm and SNOMED-CT have provided information on the changes of concepts, they are not incorporated within clinical NLP systems. In addition, the increase of most term types in MedXN new dictionary compared with the old dictionary does not necessarily guarantee the complete and accurate information extraction because some new medications have been added and old medications have been changed or deleted. Therefore, we consider that the back version compatibility of dictionaries is very important for capturing most complete and accurate annotations through clinical NLP system.

There is an increasing amount of evidence showing that information embedded in clinical notes hold great potential for decision support, evidence-based medicine and active pharmacovigilance, which is critical for improving patient safety and delivering high quality health care. Therefore, accurately annotated concepts captured in text serve as the semantic foundation of downstream analytics. It is critical that concepts are clearly characterized, flexibly represented, and adequately reflect the target domain. However, our findings show that gold standard is another main source of error. The reason may lie in the fact that a major challenge with concept annotation is the lack of common consensus in choosing both the text span(s) and the concept identity, even among human annotators. Factors that contribute to the difficulty include the natural diversity of inter-specialty and/or inter-person interpretations, insufficient utility for assisting well-informed decision, and the questionable common preference for a mono-perspective annotation. Except for these subjective reasons, dictionary change is another factor influencing gold annotation quality. Since generating gold standards for different NLP purposes by human experts is expensive and time-consuming, existing gold standards can be utilized. However, one potential problem might be that previously created gold standard cannot fully reflect the current status of terminologies, resulting lower coverage and mismatches and adversely impacting the evaluation results. For the sake of time, resource saving and validity of system evaluation, it is necessary to update the gold standards based on the most up-to-date information of terminologies. Managing terminology may help to improve gold standard generation.

Conclusion

In this study, we compared the automatic annotations produced by cTAKES and MedXN with gold standards to gain further insights on how the dictionary updates will impact the annotation performance of clinical NLP systems. The results of our study show that the coverage of dictionaries and also the temporal quality of the gold standards will largely influence the annotation performance. With the appropriate terminology management tools for back version compatibility, the gold standard update can be done in a semi-automated fashion.