An overview of Biomedical Entity Linking throughout the years

Abstract

Biomedical Entity Linking (BEL) is the task of mapping of spans of text within biomedical documents to normalized, unique identifiers within an ontology. This is an important task in natural language processing for both translational information extraction applications and providing context for downstream tasks like relationship extraction. In this paper, we will survey the progression of BEL from its inception in the late 80s to present day state of the art systems, provide a comprehensive list of datasets available for training BEL systems, reference shared tasks focused on BEL, discuss the technical components that comprise BEL systems, and discuss possible directions for the future of the field.

1. Introduction

Biomedical entity linking (BEL), also known as normalization or grounding, is a natural language processing (NLP) task dealing with the mapping of spans of text within biomedical documents to normalized, unique identifiers within an ontology. Associating these spans, known as mentions, with a discrete concept allows information extracted from the text to be easily filtered and aggregated. In this paper, we will survey the progression of BEL from its inception in the late 80s to present day state of the art systems, provide a comprehensive list of datasets available for training BEL systems, reference shared tasks focused on BEL, describe the technical components that comprise BEL systems, and discuss where the field needs to go from here.

To conduct this systematic review, we queried PubMed for all full-text articles published between 1980 and August 2022 which contained any of the phrases “entity linking”, “entity normalization”, “concept linking”, or “concept normalization” in the abstract or title. This search yielded 134 results, of which, the earliest was published in 1985. We retrieved 132 of these for further review, excluding two results which corresponded to a book chapter and a corrigendum. After review, we excluded another 72 results for reasons such as the abstract or title incidentally containing one of the search phrases without referring to the NLP task (40), the primary focus of the paper being a pipeline which includes a non-novel BEL component (19), the paper mentioning entity linking in passing (11), the paper being a review which did not introduce any original contributions (3), and the paper focusing on non-biomedical applications of entity linking (2). A PRISMA diagram showing how we filtered the results of our search is shown in Figure 1. We have included some additional publications related to biomedical entity linking in the following sections which were not included in the PubMed results, but which we felt had substantively contributed to the field.

Figure 1:

PRISMA flow diagram of publications returned from a PubMed search for full-text articles published between 1980 and August 2022 which contained any of the phrases “entity linking”, “entity normalization”, “concept linking”, or “concept normalization” in the abstract or title.

2. History

2.1. Early Work

In the late 1980’s, medical literature was expanding rapidly, but physicians were unable to search it effectively due to unfamiliarity with the Medical Subject Headings (MeSH) vocabulary used to index citations in the MEDLINE database [1]. This impediment motivated the initial work on BEL. To improve search efficacy for non-expert users, two physicians at Massachusetts General Hospital proposed MicroMeSH in 1987, an “intelligent search assistant” for searching the MEDLINE database, which used a synonym, acronym, and abbreviation dictionary to map users’ search queries to a list of possible MeSH terms with wildcard matching [1]. The idea was later expanded to facilitate the MeSH indexing of articles directly with systems such as CLARIT (1991) [2], SAPHIRE (1995) [3], OSCAR4 (2011) [4], and MetaMap (2001) [5]. These subsequent systems used linguistic rules, patterns, and dictionaries to map concept mentions to MeSH terms. MetaMap became the backbone of the Medical Text Indexer (MTI) [6] in 2004. Today, the National Library of Medicine (NLM) at the National Institutes of Health (NIH) employs MTI as the automated first-line indexer for over 350 journals.

Application of BEL to clinical text was not far behind indexing publications. CHARTLINE (1992) [7] and MedLEE (1995) [8] used similar dictionary matching techniques to extract and link entities in clinical reports to the Unified Medical Language System (UMLS). REX (2006) [9], by physicians Friedlin and McDonald, linked mentions from clinical notes to ICD-9-CM codes to facilitate medical record coding and included the novel feature of negation recognition to mitigate false positives for negative mentions (i.e. patient denies smoking). Friedlin later adapted his REX system to identify adverse drug reactions (ADR) mentioned on drug labels and link them to the Medical Dictionary for Regulatory Activities (MedDRA) with a system called SPLICER [10]. Shortly after Friedlin’s publications, Savova et al. [11] also released an end-to-end clinical NLP system called cTAKES (2010), which included an entity linking component. QuickUMLS [12] (2016) addressed the computational performance limitations of its predecessors by using an approximate dictionary matching algorithm, CPMerge, to achieve higher F1 scores than both MetaMap and cTAKES while requiring only a fraction of their runtime. RysannMD [13] similarly created a fast and accurate system which used a probabilistic model based on the individual tokens in each mention to predict concept mappings.

For developing the first generation of BEL systems, which relied exclusively on dictionary matching techniques and jointly performed NER and entity linking, researchers generally annotated their own training data from scratch. This changed in the mid-2010s with the release of prominent entity linking corpora, such as the ShARe/CLEF eHealth Challenge corpus[14] and the NCBI dataset [15] which provided a set of linked mentions out of the box. For the first time, researchers could model BEL as an independent task, limiting the scope of their work to matching a mention assumed to be an entity to its corresponding concept. This allowed for more complex perturbations of pre-extracted mentions, which would have been combinatorially intractable when considering a document in its entirety. D’Souza and Ng [16] broke ground with an influential sieve-based method that attempted to match mentions to concepts through ten progressively fuzzy layers of morphological permutations. Figure 2 illustrates potential layers in a rule-based BEL pipeline. Rather than permuting the limited set of mentions in an attempt to match concepts in the much larger ontology, Liu, et al. [17] created their own semantic lexicon based on knowledge from the UMLS and information mined from a large clinical corpus to maximize the probability of extracting mentions from a corpus which correspond to concepts in their MedLEx. Leal et al. [18] applied a rule-based similarity approach to the ShARe/CLEF dataset by searching for matches by minimizing Levenshtein distance to SNOMED-CT candidates and resolving ties by choosing the SNOMED-CT concept with the lowest Information Content (IC) [19]. While these systems were more sophisticated than their predecessors, they still shared many of the core limitations of the earliest work. Rule-based systems are generally fast, but they are unable to consider semantic meaning, so they struggle when linking mentions that require either context (i.e. does “depression” refer to a mood disorder or a sunken area?) or when vernacular for describing a concept is too lexically diverse (i.e. how many ways can you say “inadequate oral intake”?).

Figure 2:

Possible steps in a rule-based BEL pipeline where the system attempts to match progressively more permuted versions of the initial mention.

2.2. Modern Era

While dictionary-based clinical NLP methods remain popular for production implementation because of their interpretability and configurability [20], learning-based methods have largely replaced them in informatics research because of their superior performance. This paradigm shift transitioned BEL from a matching problem to a mapping problem requiring successful systems to numerically represent mentions and concepts and train models to connect them. One of the best-known early attempts at applying machine learning to BEL was DNorm [21], which used TF-IDF representations of mentions and concepts to train a linear classifier to score pairs of mention and concept representations. DNorm demonstrated a nearly 10 point gain in F-measure performance over existing rule-based baselines, becoming the defacto baseline for subsequent systems. The author later incorporated DNorm into a joint NER and BEL model called TaggerOne [22], which considered the results of two scoring functions in semi-Markov models that determined both the mention boundaries of the entity and linked it to the appropriate concept.

The first round of deep learning techniques applied to BEL represented tokens with static vector representations of words (such as TF-IDF and word embeddings [23]) and used architectures like CNN and BiLSTM to demonstrate improvement over classical machine learning (ML) baselines like DNorm [24, 25, 26]. The emergence of deep contextual embeddings, such as ELMo[27] and BERT[28], effected a sea change in natural language processing research, and BEL research has been no exception. While some researchers still investigate using static embeddings as their primary form of representation, all current state of the art systems use some form of deep contextualized embeddings, with BERT encoders pre-trained on clinical and/or biomedical text being the clear favorites [29, 30, 31]. As with classical ML BEL, both binary [32] and multi-class [33] classification models are popular, but the improved quality of representations and the ability to train the encoder has opened up other options as well, like similarity-based ranking [29] and clustering [31]. Figure 3 illustrates typical steps in a machine learning-based BEL pipeline and lists some of the configurable options for each step.

Figure 3:

Typical steps in a machine learning-based BEL pipeline.

3. Applications

The Apache Unstructured Information Management Architecture^™ (UIMA) framework [34] is an interoperability platform developed to handle software systems that process large amounts of text. Its advantage is the plug-and-play aspect allowing different components to be pipelined together. The framework has been ported from general English to process large amounts of clinical and biomedical text. CLAMP [35], cTAKES [11], Leo [36], MedTagger [37], and NOBLE Coder [38] all utilize the UIMA framework. One key component to each of these frameworks with respect to this review is the addition of a BEL component into their information extraction pipelines. A typical pipeline includes components that 1) initially extract specific entity types (e.g., Diseases, Drugs) from the text, 2) determine the relationship between the entities (e.g., Treats, Reason), and 3) link the entities to their respective concept in an ontology (e.g., the UMLS). The BEL component normalizes synonymous terms (e.g., Heart Attack and Myocardial Infarction) allowing information across documents to be analyzed regardless of their lexical diversity. However, as with relation extraction, error propagation [39] becomes a challenge in real-world environments where any error that occurs when identifying the entities is propagated to downstream tasks including both the identification of the relations between the entities and the linking of those entities to their respective concepts.

These system have been used to develop information extraction pipelines to address use cases centered around the extraction of specific types of information from clinical notes [20]. For example, mapping clinical entities in notes to Fast Healthcare Interoperability Resources (FHIR) standards [40] to supplement discrete electronic health record data for purposes such as cohort identification and clinical monitoring. Another example includes automatically assigning International Codes for Diseases (ICD-9-CM/ICD-10-CM) to clinical records for automated billing [41, 42]. These codes are typically utilized for billing purposes but can also provide salient disease or symptom information about the patient [42].

4. Datasets

The set of biomedical corpora annotated for BEL continues to increase every year and this task continues to become a prominent research interest. Important dimensions for diversity of these datasets are the domain of the text corpus, target ontology for linking, and the types of entities being linked. Scientific literature, the original BEL domain, remains popular, with corpora often annotating broad ranges of biomedical concepts mapped to MeSH terms or UMLS concepts. Several BioCreative challenges have published corpora in this domain focused on niche entities like genes or chemicals, which sometimes map to smaller ontologies. Clinical domain datasets are often targeted to entities which provide clinical utility such as disorders, problems, tests, and treatments. These are generally mapped to either the UMLS or ICD codes. Other sources for datasets include online social media such as Tweets and discussion forum posts, as well as drug packaging labels, and Wikipedia. There is a particular research interest in using BEL to link adverse drug events (ADE) to either MedDRA or the UMLS. We identified at least seven datasets that have been curated for the sole purpose of linking drugs and ADEs. Table 1 shows for each dataset, the document type, entity types, the target ontology, the number of documents in the dataset, the number of mentions, and number of unique mentions (when provided).

Table 1:

Biomedical Entity Linking Datasets

Domain	Doc Type	Citation	Date	Entity(ies)	Ontology	Doc Count	Mentions	Unique Concepts
Scientific Literature	Biomedical Abstract	GENIA [43]	2003	Biomedical (broad)	MeSH	2,000	93,293	–
		NCBI Disease [15]	2014	Disorder	MeSH	793	6,892	790
		MedMentions [44]	2019	Biomedical (broad)	UMLS	4,392	352,496	34,724
		MM-ST21pv [44]	2019	Biomedical (broad)	UMLS	4,392	203,282	25,419
		PubMedDS [45]	2021	Biomedical (broad)	MeSH	13,197,430	57,943,354	44,881
		RegEl [46]	2022	DNA Regulatory Elements	Various	419	8,369	2,947
	Biomedical Article	BC5CDR [47]	2016	Chemical, Disorder	MeSH	1,500	10,227	–
		CRAFT [48]	2016	Biomedical (broad)	Many–	97	–	–
		BioNLP-2019 [49]	2019	Bacteria Biotope	NCBI	392	7,232	1,072
		PhaarmaCoNER [50] (ESP)	2019	Chemical, Drug	UMLS	1,000	7,624	–
		BC7NLMCHEM [51]	2021	Chemical	MeSH	150	38,342	2,064
	Multi Source	Quaero [52] (FRA)	2014	Biomedical (broad)	UMLS	2,538	26,407	5,796
		Mantra [53]	2014	Biomedical (broad)	UMLS	1,450	5,530	3,780
	Figure Caption	BC6BioID [54]	2017	Gene,Chemical	ChEBI,UniProt	17,883	133,003	7,652
Clinical	Clinical Note	ShARe/CLEF [14]	2013	Disorder	UMLS	431	19,557	1,871
		CUILESS2016 [55]	2018	Disorder	UMLS	431	5,397	1,738
		n2c2 2019 [56] (Luo, 2019)	2019	Problem, Test, Treatment	UMLS	100	10,919	3,792
		MADE [57]	2019	ADE, Drug, Indication	MedDRA	1,089	43,000	–
		Cantemist [58] (ESP)	2020	Oncology	ICD-O	1,301	16,030	850
		BRONCO [59] (DE)	2021	Oncology	ICD-10, OPS, ATC	200	11,124	4,027
Social Media/Online Literature	Drug Label	TAC2017 [60]	2017	ADE	MedDRA	200	26,488	–
	Tweets	Twitter ADR [61]	2015	ADE, Indication	UMLS	1,784	1,693	–
		SMM4H-17 [62]	2017	ADE	MedDRA	25,678	–	–
		TwADR-L [63]	2016	ADE	SIDER?	1,436	–	273
	Drug Forum	DailyStrength ADR [61]	2015	ADE, Indication	UMLS	6,279	4,929	–
		CADEC [64]	2015	ADE,Disorder,Drug	AMT,MedDRA,SNOMED	1,253	9,111	3,591
		PsyTAR [65]	2019	ADE,Disorder	UMLS	891	7,414	1,671
		COMETA [66]	2020	Biomedical (broad)	UMLS	–	20,000	3,645
	Wikipedia	WikiMed [45]	2021	Biomedical (broad)	UMLS	393,618	1,067,083	57,739

International Classification of Diseases for Oncology (ICD-O); Operationen und Prozedurenschlüssel (OPS); Anatomical Therapeutic Chemical Classification System (ATC);

5. Performance Comparison

In Table 2, we compare the performance of various extant systems on six BEL datasets (BC5CDR Disease [47], BC5CDR Chemical [47], CADEC [64], NCBI Disease [15], n2c2 2019 [56], and ShARe/CLEF [14]) in terms of accuracy. These datasets were chosen because of their relative popularity and the number of authors choosing to evaluate their systems using accuracy. We chose accuracy as our common metric because it is reported for a plurality of systems. All results were reported by the respective authors, so it’s important to note that results may not be directly comparable due to differences in evaluation techniques. For example, Miftahutdinov and Tutubalina [67] evaluated their system using cross validation on the entire corpus rather than only the test partition. Some authors choose to remove conceptless annotations [68, 67]. Also, some systems [29] only require their systems to map mentions to a correct synonym for predictions to be considered correct, whereas other systems require the more stringent criteria of mapping to the correct concept ID [68, 69]. The latter specification generally results in lower accuracy because systems must solve the additional challenge of disambiguation.

Table 2:

Performance Comparison of Extant Systems

	BC5CDR (d)	BC5CDR (c)	CADEC	NCBI Disease	n2c2 2019	ShARe/CLEF
Chen, et al. [70]	-	-	-	-	82.1	-
D’Souza, et al. [16]	-	-	-	84.7	-	90.8
Ji, et al. [71]	-	-	-	89.1	-	91.1
Lee, et al. [72]	-	-	65.0	-	-	-
Li, et al. [73]	-	-	-	86.1	-	90.3
Liu, et al. (SAPBERT) [30]	93.5	96.5	-	92.3	-	-
Limsopatham and Collier [63]	-	-	81.41	-	-	-
Kalyan, et al. [74]	-	-	82.6	-	-	-
Miftahutdinov and Tutubalina (2018) [67]	-	-	88.8	-	-	-
Miftahutdinov, et al. (2021) [75]	75.8	83.8	-	-	-	-
Mondal, et al. [76]	-	-	-	90.0	-	-
Niu, et al. [26]	-	-	84.7	-	-	-
Pattisapu, et al. [77]	-	-	76.7	-	-	-
Phan, et al. [78]	90.6	95.8	-	87.7	-	-
Schumacher, et al. [68]	-	-	-	-	-	62
Silva, et al. [79]	-	-	-	-	80.6	-
Sung, et al. [29]	93.2	96.6	-	91.1	-	-
Tutubalina, et al. (2018) [25]	-	-	70.1	-	-	-
Wright, et al. [80]	88	-	-	87.8	-	-
Xu, et al. (2020) [81]	-	-	87.5	-	83.6	-
Xu and Miller (2022) [82]	-	-	-	-	85.3	91.3
Zhang, et al. (KRISSBERT) [69]	90.7	96.9	-	89.9	80.2	90.4
Zhao, et al. [83]	-	-	-	88.2	-	-

Comparison of reported accuracies on six popular BEL datasets. The BC5CDR dataset contains partitions corresponding to disease (d) and chemical (c) normalization, which are often evaluated separately.

6. Shared Tasks

There have been a number of shared tasks focused on BEL, starting with the inaugural BioCreative challenge in 2004 [84]. Table 2 shows the different tasks that have been organized over the years. We classify these tasks into three categories based on the type of text that was annotated as outlined in the previous section. Within each category, the tasks are ordered based on their date. The table also includes the document source, entities and the associated ontology.

The majority of shared tasks focus on scientific literature with the early BioCreative tasks mapping a broad class of biomedical entities to concepts in the MeSH ontology[84]. Since that time, new shared tasks have been developed every four years or so, expanding from abstracts to full text, and incorporating new entity types. The clinical shared tasks began in 2013 [14] focusing on disorders with the most recent task [56] expanding to include both treatments and tests. The social media shared tasks both happened in 2017 and focused on adverse drug reactions(ADR).

7. Technical Discussion

All BEL systems are a pipeline of various components and techniques which can be mix and matched to fit a practitioner’s data and use case. Some potential applications of BEL are discussed in Section 3. In this section we will discuss the major categories of techniques, how they work, and where they’ve been applied.

7.1. Preprocessing

Many BEL publications make no mention of any pre-processing of the input corpus prior to training. Whether this step is implied or simply omitted is not entirely clear, but where mentioned, many systems follow standard pre-processing steps such as converting all text to lowercase and removing punctuation. Authors frequently correct spelling on the NCBI Disease dataset, for which D’Souza, et al. [16] curated a corpus-specific dictionary to this end, but we have not seen a generalizable tool in use for other datasets. Two additional common steps are expanding abbreviations to full form using the Abbreviation Plus Pseudo-Precision (Ab3P)[89] tool and separating composite mentions into distinct parts (i.e. “BRCA1/2” into “BRCA1” and “BRCA2”) using the SimConcept[90] tool. Finally, it is common practice to append the mentions from the training set to the synonym dictionary when evaluating performance on the test set [16, 29]. However, some have questioned whether this results in an unfair evaluation given the frequent overlap of mentions between training and test datasets [91].

7.2. Mention/Concept Representation

Rule-based systems represent mentions using tokens[5, 16], in other words, actual human-readable words and phrases. These representations can do fairly well given that many mentions are morphologically similar to known synonyms of their corresponding concept, but this technique has a real upper bound when mentions differ significantly from documented synonyms, and as Blair, et al. [92] note, synonym coverage for biomedical entities is far from complete. Representing mentions numerically opens up a world of possibilities for choosing sophisticated learning algorithms. The simplest such representation is Term Frequency-Inverse Document Frequency (TF-IDF) vectors, used in the first machine learning-based BEL system, DNorm[21]. This technique scores tokens with a ratio its frequency in a mention by its overall frequency in the set of concept synonyms. While this technique is intuitive, it fails to capture semantic meaning and shares many shortcomings with token representation. Word embeddings, which project tokens into a latent semantic vector space, do address the problem capturing semantic meaning. The first iteration of such techniques, led by Word2Vec[23], created static vector representations of tokens which effectively aggregated the contextual usage of a given token within a corpus and embedded it in the semantic space. For the first time, word embeddings allowed us to mathematically compare the similarity of two given tokens without requiring any additional knowledge. The improved quality of these representations correlated with a higher quality output from the systems which incorporated them. The primary downside to these static representations is that they cannot capture the nuance of words that have different meanings in different contexts. Deep contextualized embeddings such as ELMo[27] and BERT[28] capture not only aggregate semantic meaning, but also take into account a token’s context within a specific sentence. These techniques provide unquestionably state of the art embedding quality embeddings, which are the foundation of all the current top performing BEL systems. However, quality comes at a computational cost and generating deep contextualized embeddings at any practical scale requires access to a GPU. The final major category of representations is graph-based techniques, such as concept vectors. Node2Vec [93], as employed by Ferré, et al. [94] in their CONTES system, models concepts in an ontology as nodes in a graph and relationships between concepts as edges, it then generates a vector space which embeds concepts such that connected nodes in the graph correspond to closeness within the vector space. CONTES used these concept vectors only to represent concepts, and learned a mapping between the semantic space representing mentions and the ontology space generated by Node2Vec. They also note that this technique may not scale well to large ontologies.

7.3. Linking Algorithms

The crux of any BEL system is the algorithm which links the representation of a mention to a concept in the target ontology. The most basic implementation of this mapping is a dictionary lookup, which checks if the mention is an exact match of some known concept synonym. To increase recall, systems [16] may create morphological permutations of the mention and check if the permutations match any known synonyms, but the expression of natural language is diverse and any system which generates enough blind permutations to achieve respectable recall will inevitably generate a huge number of false positives. But there is still a place for morphological feature extraction in sophisticated BEL systems, some have used Lucene search to select a small set of candidate concepts prior to using deep learning techniques to make a final prediction [95].

Learning algorithms train systems find mappings between mentions and concepts in a vector space, which allows them to achieve both higher recall and precision. BEL systems incorporating classical machine learning started with linear classifiers to learn positive and negative correlations between tokens in mentions and concept synonyms [21]. As the quality of word representations improved and access to GPUs became widespread in the 2010s, deep learning techniques such as CNN [63], RNN [63], GRU [25], and BiLSTM [96] came into vogue. Other systems have trained lesser known learning algorithms such as RankSVM [33] and TreeLSTM [97], but neither of these have achieved widespread adoption.

As expected, using a BERT for BEL performs quite well. Typically, researchers use BERT classifiers [30], but sequence-to-sequence translation models have been explored as well [98]. Other models have leveraged the high quality of BERT embeddings to rely on simple similarity measures to perform their mapping [29], training only the encoder and omitting a secondary neural architecture entirely. PageRank, an algorithm originally designed for scoring the relevance of search engine results, has been used to link entities when using graph-based representations [99].

One technique uncommon in BEL that deserves more attention is clustering, which Angell, et al. [31] employed following candidate generation by creating an affinity graph with mention-mention and mention-concept connections for all mentions and candidates in a given document. They iteratively pruned connections in the graph to create clusters until each cluster contained exactly one concept linked one or more mentions. This approach is especially helpful for disambiguating mentions of generic phrases which corresponded to entities described more specifically elsewhere in the document and yielded the current state of the art performance for few-shot entity linking.

7.4. Training Techniques

In addition to the building blocks described in the previous sections, we noted several training techniques commonly employed by successful BEL systems. The most common of these is a two step process in which a system first uses a high-recall technique to select a small pool of candidate concepts from the target ontology, followed by a higher precision technique to select a single concept for prediction out of the pool of candidates. The algorithms used for candidate generation vary widely, but recurring solutions include search engine-style algorithms like bag-of-words retrieval function BM25 [33] or lucene [95], similarity of mention representations [76, 29], and edit distance [99]. A related strategy for narrowing the range of possible candidates is to predict the semantic type of the mention and only consider candidates of the predicted semantic type. The MedType [45] system was created to perform this type of semantic type prediction in entity linking pipelines. Another way that semantic types have been used to augment BEL pipelines is to train the prediction step to rank all candidates with the correct semantic type over those with the wrong semantic type [95, 81], as opposed to loss functions which only consider the top-ranked candidate. External knowledge bases such as Wikipedia [100, 30] have also shown promise as valuable sources of information for inclusion in BEL systems.

The state of the art SAPBERT model [30] attributed its success to a self-alignment pre-training strategy in which only difficult positive and negative examples for a given gold concept in each mini-batch are used for training. The subsequent multi-similarity loss function simultaneously pushes negative examples away from the gold concept, while pulling the positive examples closer. Finally, it is also common to perform entity linking jointly with other NLP tasks, in particular, named entity recognition [83, 101, 22].

7.5. Multilingual-based Approaches

Entity linking in non-English corpora presents additional challenges and several non-English corpora[52, 58, 50] exist to train systems to tackle these challenges. The most straightforward approach is to link directly from the source documents to an ontology in the same language. This can work well if the ontology has good coverage, but in the UMLS, there are many times more English synonyms available than those in non-English target language, even in the best cases (Spanish and French with more than six times and twenty-four times respectively[102]). Non-uniform distribution of non-English synonyms does allow that there are cases in which this strategy could still work for specific languages and problems, such as identifying disorders in Italian clinical notes[103], but for other languages and use cases, the scarcity of target language synonyms can be a insurmountable obstacle for this strategy. A naive approach for overcoming these challenges is to simply translate the non-English mentions into English using standard translation software and perform BEL on the translations. This works reasonably well, but is limited by the quality of the translation, which may struggle to properly translate medical jargon[103]. Roller, et al., 2018[104] combined these two approaches sequentially, first looking for a match for a given mention in the target language UMLS, then English language UMLS, and finally searching English UMLS for the translation of the mention. Deep learning-based approaches[26] favoring encoder models learning a direct mapping from non- English mentions to English synonyms[105] have performed well. The current best performing model for multilingual BEL adapts the SAPBERT[30] system to map mentions in any language to language-agnostic CUIs in the UMLS. This system augments the cross-lingual links between CUIs by leveraging the titles of Wikipedia articles available in multiple languages where the article title can be mapped to the UMLS for at least one language. The authors found that performance for a given language generally correlated with its similarity to English, likely because more general translation knowledge could be incorporated into the model[102].

8. Discussion

A fundamental limitation of BEL is that treating the task as a classification problem with a learning-based approach requires the output space to be at minimum equal to the number of concepts to be predicted. While this works well when the output space tends to be small [63], these approaches struggle as the size of the taxonomy increases [95], particularly with concepts that have only a few example mentions in the training data. While current state of the art accuracies greater than 90% on many of the most common BEL datsets would seem to indicate that the problem is largely solved, Tutubalina, et al. [91] found that approximately 80% of entity mentions in the test datasets they analyzed were either duplicated within the test set or replicated exactly in the training dataset. Because many systems [16, 95] add training mentions to their synonym dictionaries used for inference against test data, this unrealistically inflates the actual abilities of a system to link mentions in a corpus with higher variability. They supported this hypothesis by creating a “refined” version of five popular BEL datasets, removing all duplication of mentions in the test sets, and comparing a state of the art BEL system’s performance on the original and refined test sets. Their results showed a substantial performance impact from the de-duplication, indicating that developing effective solutions to BEL as a zero or few-shot learning problem is an area ripe for future improvements. Developing effective techniques for distant [77] and self-supervision [69] will be crucial to scaling BEL systems to perform well when linking mentions to concepts which are dissimilar to annotated data.

The development of non-English BEL corpora [52, 58, 50] and recent multilingual systems [105, 102] are a great start for expanding BEL to be a truly multilingual task, but BEL performance on non-English texts trails far behind the state of the art performance on English texts, especially for languages which are absent or severely underrepresented in the UMLS. More work is needed specifically to develop more non-English BEL corpora and to find new strategies for overcoming the difficulties of mapping underrepresented languages to the UMLS.

Newman-Griffis, et al. [106], demonstrate that existing BEL datasets do not sufficiently capture the ambiguity resulting from unique strings mapping to multiple possible CUIs in the UMLS. Polysemy, where a word can have multiple senses, can harm the generalizability of models when the training data exposes models to only one sense a word, erroneously causing it to appear unambiguous. This phenomenon can be especially prevalent in datasets which annotate only narrow slices of clinical entities, such as diseases. In such a case, “cold” may be annotated several times in reference to a viral infection, but never as a relative perception of temperature, though both senses of the word may appear in actual notes. Another source of ambiguity common to telegraphic clinical language is metonymy, in which one concept is used as shorthand for a related concept. Without properly understanding context, BEL systems can easily conflate devices for procedures (i.e. “stent”), substances for lab measurements (i.e. “potassium”), and diagnoses for symptoms. A final source of ambiguity can result from the level of specificity in the annotation, such as whether an instance was noted to be a sequela, whether multiple were specified (i.e. “injuries” vs. “injury”), or hierarchical specificity (i.e. “hemiplegia” vs. “left hemiplegia”). They recommend developing ambiguity-focused datasets to train systems to capture a more nuanced contextual understanding of ambiguous mentions.

A subsequent paper by Newman-Griffis [107] introduces the research paradigm of “translational NLP”, in which basic and applied NLP research inform one another. Under this paradigm, we can see a potential way to mitigate at least one class of ambiguity. When researchers query structured clinical data, which may include the discretized results of a BEL algorithm, they rarely search for a single concept in isolation, rather they curate a set of concepts [108], often hierarchically related, which correspond to a more general clinical phenotype. In such cases, hierarchical classification errors could in many cases be close enough to the gold concept to still be included in the correct phenotype. Evaluating performance with respect to ontological similarity rather than solely considering a binary measure of whether a prediction exactly matches the gold concept could be a productive line of inquiry for future BEL research.

9. Conclusions

In this paper, we reviewed previous work on BEL providing an overview of the progression of historical approaches (Section 2) and providing a reference for the BEL datasets (Section 4) and shared tasks (Section 6) that have been developed. We then discussed salient challenges and opportunities for future work, highlighting four areas specifically:

• Reported results are inflated by overlap between training and test mentions and duplication within test datasets. Evaluating systems performance on datasets without the benefit of overlap and duplication makes it clear that there is much work to be done with BEL as a zero or few-shot learning problem.

• BEL performance on non-English mentions is significantly lower than on English, especially for those languages absent or severely underrepresented in the UMLS.

• Current BEL datasets do not sufficiently capture the ambiguity resulting from unique strings mapping to multiple distinct concepts.

• Alternative performance metrics like ontological similarity should be explored in order to develop systems which best meet real world use cases.

Table 3:

Entity Linking Shared Tasks

Domain	Year	Task	Document Source	Entity Type(s)	Ontology
Scientific Literature	2004	BC I (1b)[84]	MEDLINE	Fly, mouse, and yeast genes	Organizer provided
	2006	BC II (1b)[85]	MEDLINE	Human genes	EntrezGene
	2010	BC III GN[86]	PMC full text	Genes	EntrezGene
	2016	BC V CDR (3a)[47]	PubMed	Chemicals, diseases, chemical-disease interactions	MeSH
	2017	BC VI Bio-ID (1)[54]	Figure captions	Genes, chemicals, cell type, subcellular location, tissue, organism
	2019	BioNLP 2019 (1)[49]	PubMed	Microorganism, habitat, phenotype	NCBI, OntoBiotope
	2021	BC VII NLMCHEM (1b)[51]	PubMed	Chemicals	MeSH
clinical	2013	ShARe/CLEF 2013 (1b,2)[14]	Clinical records	Disorders	SNOMED CT
	2014	SE-2014 (7b)[87]		Disorders	SNOMED CT
	2015	SE-2015 Task 14 (1,2a)[88]		Disorders	SNOMED CT
	2019	2019 n2c2 (3)[56]		Problems, treatments, tests	SNOMED CT, RxNorm
	2019	PharmaCoNER[50]	Clinical records	Drugs, chemicals	SNOMED CT
	2020	IberLEF CANTEMIST-NORM[58]	(ESP)	Tumor morphology	ICD-O
social Media	2017	SMM4H 2017 (3)[62]	Twitter	ADRs	MedDRA
	2017	TAC 2017[60]	Drug labels	ADRs	MedDRA

BioCreative (BC); SemEval (SE); Task/Track number in parentheses