Clinical named-entity recognition: A short comparison

Abstract

The adoption of electronic health records has increased the volume of clinical data, which has opened an opportunity for healthcare research. There are several biomedical annotation systems that have been used to facilitate the analysis of clinical data. However, there is a lack of clinical annotation comparisons to select the most suitable tool for a specific clinical task. In this work, we used clinical notes from the MIMIC-III database and evaluated three annotation systems to identify four types of entities: (1) procedure, (2) disorder, (3) drug, and (4) anatomy. Our preliminary results demonstrate that BioPortal performs well when extracting disorder and drug. This can provide clinical researchers with real-clinical insights into patient’s health patterns and it may allow to create a first version of an annotated dataset.

I. Introduction

Healthcare organizations and goverment agencies have successfully adopted electronic health records (EHRs), which has created an explosion of clinical data available for research [1]. EHRs are mostly composed of unstructured free text, this free text often records critical information. The conversion of EHRs to structured data is labor-intensive, expensive, and can not successfully capture all relevant information.

The use of natural language processing (NLP) techniques allows to decrease the time and the human intervention needed to obtain critical information from free text which has positively been impacting biomedical and clinical research [2], [3]. On the other hand, clinical research has become difficult to measure due to the absence of available gold standard datasets, which are mainly manually created. To ensure a high quality of gold standard datasets, an inter-annotator agreement is calculated between the experts that are requested to independently annotate the data. This makes the creation of these datasets a costly process. Biomedical named-entity recognition (BioNER) is a very important task in NLP that aims to automatically recognize and classify biomedical entities; such as genes, disease names, medication names, procedures, proteins, chemicals, among others; from biomedical text. Different BioNER systems have been proposed to extract clinical concepts from text, including MedLEE [4], MetaMap [5], MetaMap Lite [6], KnowledgeMap [7], Apache cTAKES [8], HiTEX [9], MedTagger [10], CLAMP [11], QuickUMLS [12]. There is a wide range of BioNER systems in the literature. Overviews of these existing bioNER applications have been provided in [3], [13], [14]. However, there is a lack of comparison studies providing information to choose the most suitable tool for a specific entity extraction task in the clinical domain, for instance the extraction of treatments, disorders, drugs, procedures, anatomy. The ultimate goal of our project is to demonstrate the feasibility to create a clinical gold standard dataset using an ensemble of the more appropriate annotation systems, and to recommend the most suitable tool for the annotation of a specific clinical entity type (i.e., categories). In sum, the aim of this study is to compare three state-of-the-art annotation systems used successfully in the clinical and biomedical domain. We selected four categories for the evaluation: (1) drug (2) disorder, (3) procedure, and (4) anatomy.

II. Methods

A. Data

We used the publicly available Medical Information Mart for Intensive Care (MIMIC-III) database [15], [16]. For this study, we compiled a subset of 1,000 clinical notes from the table “NOTEEVENTS” which contains deidentified notes, including nursing and physician notes, ECG reports, imaging reports, and discharge summaries. The single rule to preprocess the extracted clinical notes was to delete null entries.

B. Annotation systems

We implemented the following three applications:

1. BioPortal¹: is a web portal that provides access to a library of more than 1,100 biomedical ontologies and terminologies. BioPortal enables community participation in the evaluation and evolution of ontology content by providing features to add mappings between terms, to add comments linked to specific ontology terms and to provide ontology reviews [17], [18]. The open biomedical annotator provides REST Web service for the annotation task.

2. CLAMP²: is a Java-based clinical language annotation, modeling, and processing toolkit. CLAMP provides different state-of-the-art NLP modules such as entity recognition, entity linking, normalization. It also provides an integrated development environment with user-friendly graphic user interfaces to allow users to quickly build customized NLP pipelines for individual applications [11].

3. ScispaCy³: is a specialized Python NLP library for processing biomedical, scientific, and clinical texts which leverages the spaCy library⁴, used and evaluated on several NLP tasks such as part-of-speech tagging, dependency parsing, named entity recognition, and sentence segmentation [19].

C. Data analysis

We evaluated 1,000 clinical notes in terms of the following aspects: A) a general overview of all annotated entities for the four clinical categories; B) an overview of the entities extracted per category; and C) implementation requirements.

III. Experimental results

A. General overview

To evaluate the annotation of all entities of all categories with the three systems, we first evaluated the number of entities extracted by each application as shown in Table I. Then, we studied the co-identification (i.e., overlapping) of different entities by the three applications over the dataset. Fig. 1 illustrates the distribution of distinct entities extracted by the three different applications, as well as the percentage of entities extracted by more than two applications. As shown in Fig. 1, 10% of distinct entities were extracted by the three applications for the four categories.

TABLE I.

Number of entities extracted by the three applications.

	Total entities	Distinct entities
BioPortal (B)	190,898	5,632
CLAMP (C)	55,258	6,291
ScispaCy (S)	157,597	3,613
B ∪ C ∪ S	403,753	11,777

Fig. 1.

Distribution of the 11,777 distinct entities annotated with BioPortal, CLAMP, and ScispaCy.

B. Extraction of distinct entities per category

We also evaluated the extraction of the four categories: drug, disorder, procedure, and anatomy. Fig. 2 illustrates the distribution of distinct entities extracted per category with the three applications and all entities extracted in common between the three systems.

Fig. 2.

Distribution of distinct entities extracted per category with BioPortal, CLAMP, and ScispaCy.

C. Implementation requirements

We also mention some details needed to set up the three applications. A registration is needed to use BioPortal. It allows 15 queries per second per IP address. The limit of characters per query is 7,250. CLAMP and ScispaCy need an UMLS account. In general, ScispaCy requires less time for execution followed by CLAMP.

IV. Discussion

As shown in Table I, BioPortal extracted the highest number of entities (190,898) from the 1,000 clinical records. CLAMP got the lowest number of entities (55,258). However, CLAMP extracted more distinct entities (6,291) than BioPortal and ScispaCy. Since CLAMP is based on machine learning algorithms, it seeks to extract only relevant entities from clinical records. Also, 10% of distinct entities were extracted by the three applications. Moreover, CLAMP performed better when extracting distinct procedure entities (2,023), see Fig. 2. BioPortal performed better when extracting distinct disorder (2,050) and drug (719) entities. ScispaCy obtained better results over the distinct anatomy entities only (19). We also investigated that ScispaCy performs better with more training.

V. Conclusion

Our preliminary comparison of annotation systems consisted of three annotation tools over a sample of 1,000 clinical notes from MIMIC-III database. We extracted four entity categories. In general, BioPortal performed better than CLAMP and ScispaCy. In the near future, we will investigate the feasibility of creating a first version of an annotated dataset using the common entities extracted by two or more applications. Finally, further exploration of the annotation tasks is planned with: (a) additional state-of-the-art annotation applications, such as QuickUMLS, Apache cTAKES, and MetaMap; (b) an annotated dataset; and (c) a bigger sample from the MIMIC-III database.