Edge Weight Updating Neural Network for Named Entity Normalization

Abstract

Discriminating the matched named entity pairs or identifying the entities’ canonical forms are critical in text mining tasks. More precise named entity normalization in text mining will benefit other subsequent text analytic applications. We built the named entity normalization model with a novel edge weight updating neural network. We, next, verify our model’s performance on NCBI disease, BC5CDR disease, and BC5CDR chemical databases, which are widely used named entity normalization datasets in the bioinformatics field. We also tested our model with our own financial named entity normalization dataset to validate the efficacy for more general applications. Using the constructed dataset, we differentiate named entity pairs. Our model achieved the highest named entity normalization performances in terms of various evaluation metrics. Our proposed model when tested on four different datasets achieved state-of-the-art results.

Introduction

The text mining technology is undergoing a rapid evolution thanks to the exponential growth in the number of text-rich documents available online, and as a result, it is being widely applied in a range of domains such as finance and bioinformatics. Text mining aims to extract the information from documents to derive valuable insights. Documents subject to analysis contain many named entities, which are proper names that denote unique objects such as organizations, products, persons, and locations. The technique used to extract named entities from documents is called named entity recognition (NER, henceforth). Furthermore, named entity normalization (NEN, henceforth) involves matching extracted named entities with homogeneous identity and is pivotal for text mining tasks.

More specifically, in the biomedical domain, disease names and chemicals in drugs often have different surface forms while sharing the same concept. Types of named entities with different surface forms that share same concept can be divided into following categories: (1) synonyms, (2) abbreviations, (3) acronyms, (4) different combinations of punctuations and alphabets, (5) descriptive phrases, and (6) possible NER parsing errors. For example, “hepatomegaly" and “liver enlarged" do not have matching strings but the two disease names have identical meanings, and thus, these two named entities are synonyms. Biomedical named entities have a wide variety of different surface forms compared with entities from other text sources. More accurate named entity normalization techniques will potentially improve the quality of downstream tasks. Moreover, matching entity pairs such as “International Business Machines" and “IBM", which are examples of acronyms, are very critical in financial text mining applications. Linking entities with the same identity enables accurate sentiment analysis on firms and products. Furthermore, evaluation of news impacts on the stock market requires the connection between news articles and related firms. Given the wide range of named entities in bioinformatics and finance documents, the total number of tokens to be calculated for text clustering and classification is enormous.

The early NEN models explored knowledge-based or rule-based approaches [1–4]. Generating the rules for named entity matching based on domain knowledge is valid only for the dataset in which the corresponding rules are already created. The rule-based models are not robust for the neologisms. In order to overcome the disadvantage that the rule-based model is not robust, models based on machine learning have been introduced [5–7]. However, machine learning models are limited to specific fields such as bioinformatics NEN and chemical engineering NEN due to lack of NEN datasets in other domains. Our research aims to construct fully automated NEN model that can be applied to various other domains. To test our model’s robustness on different domain, we also apply the NEN dataset in finance.

An automated named entity normalization model reduce the burden of hand-mined information extraction tasks. Clear linkage between entities with different forms, such as abbreviations and acronyms, aid in more accurate sentiment analysis. The named entity normalization model also benefits the creation of more comprehensible classifying and clustering documents. The primary contributions of our study are (1) constructing better performing NEN model using an Edge Weight Updating Neural Network and (2) applying our proposed model to bioinformatics NEN and financial NEN tasks.

The proposed method, that is, the edge weight updating neural network, consists of four parts: (1) ground truth entity graph construction, (2) similarity-based entity graph construction, (3) edge weight updating neural network training, and (4) edge weight updating neural network inferencing. The main concept behind the Edge Weight Updating Neural Network is to minimize the Ground Truth Entity Graph’s edge weight distributions and the Similarity-Based Entity Graph’s edge weight distributions. By minimizing the edge weight distributions on the two graphs, entity embeddings capture more accurate information on semantic similarity between matching entities.

Our proposed model is evaluated on three widely used bioinformatics datasets (NCBI Disease, BC5CDR Disease, and BC5CDR Chemical) and its performance is compared with other cutting-edge models. Furthermore, to validate the efficacy of our proposed model in general NEN tasks, we construct a financial NEN dataset with state-of-the-art NER using BERT [8]. Using the constructed dataset, we propose the deep learning model to solve more practical financial NEN tasks. Out dataset incorporates major challenges in entity matching: (1) synonyms, (2) abbreviations, (3) acronyms, (4) different combinations of punctuations and alphabets, (5) descriptive phrases, and (6) possible NER parsing errors. Compare with other recent NEN models, our proposed model shows higher accuracies in all datasets used in the experiments, and our model is tested with not only bioinformatics NEN datasets but also financial NEN datasets, which verifies the efficacy in general NEN tasks.

The remainder of this paper is organized as follows. Section 2 describes related work. The structure for our proposed model is described in Sect. 3. Experiment settings for testing model performances are provided in Sect. 4. Section 4.1 presents an overview of dataset we used for evaluations. A brief explanation of pre-constructed NEN datasets from the bioinformatics domain is given in this section. Furthermore, Sect. 4.1 in Experiment Settings(subsection 4) provides the overview of preprocessing for data and financial NEN dataset construction with examples. In Sect. 5, we present the details regarding the qualitative and quantitative analyses we conducted on the models. Finally, in Sect. 6, we present our conclusions.

Related Work

Bioinformatics, chemical engineering, and materials science domain actively adopt cutting-edge deep learning frameworks for NEN tasks. According to Cho et al. [9], various products exist for recognizing and normalizing named entities in biomedical fields such as ProMiner [3] and MetaMap [10]. DNorm [11] and TaggerOne [12] also used machine learning models such as pairwise ranking scoring and semi-Markov models, respectively, for NEN processing. In genetic engineering, GenNorm [13] and GNAT [14] are used to normalize the gene names. ChemSpot [15] uses Conditional Random Field for NER and NEN tasks in chemical engineering. Weston et al. [16] developed MatScholar [16] python repository to perform general NLP tasks on material science texts, which includes entity normalization.

The above researches and products used NEN datasets concentrated on specific domains. ShARe/CLEF [17] is one of the widely used NEN datasets for bioinformatics that is made up of clinical notes. The NCBI [18] dataset contains PubMed abstracts for disease name normalization tasks. TAC2017ADR [19] aims to link identical drug labels. The BC2GM [20], BioNLP09 [21], and BioNLP-OST19 [22] datasets deal with genes, proteins, and bacteria, respectively. In chemical engineering, SCAI [23] and IUPAC [24] are available for researches on chemical name matching. Similar to chemical names, Weston et al. [16] developed a dataset for material engineering to normalize entities to a canonical form.

Applying machine learning algorithms in the financial domain is gaining increasing attention. One major branch is stock movement forecasting using various deep learning mechanisms [25, 26]. Thanks to the rapid developments of unstructured data processing techniques, researches on applying text mining techniques to the financial fields have increased in number. In their study, Gupta et al. [27] illustrated the trends for applying text mining in finance. Among many related text mining applications in finance, NEN can be applied to various financial researches and financial practices. In preprocessing for applying text mining techniques to solve real-world problems, NER and NEN models are performed preemptively. However, the NEN dataset for the financial domain is scant and there is a need for developing a dataset targeting the financial NEN.

Many researchers have developed targeted datasets for more general NEN tasks in domains such as user comments, product description, and financial invoices. For example, in their study, Jijkoun et al. [28] used user comments from newspaper websites. Sun et al. [29] performed normalization of product entity names, for which the dataset was developed by the authors. The study conducted by Francis et al. [30] on financial invoices is the most relevant one to our study. However, Francis et al. focused on insurance, telecommunications, banking, and tax companies using the following entities: International Bank Account Number (IBAN) of the beneficiary, invoice number, invoice date, and due date [30]. The focus of our study is on more general financial entity normalization, which covers entities from all financial sectors. Previous studies using the datasets illustrated above used various machine learning and deep learning models.

There are similarities between the string matching methodologies in various other fields and NEN researches. Sun et al. [29] proposed NEN for product names using a pre-constructed product entity linkage dictionary. In semantic string matching, Siamese Neural Networks are widely used [31–33]. Krivosheev et al. [34] used Siamese Graph Neural Network for company name normalization. We need to extend NEN on company names to NEN on a wide range of product names and legal entities. Siamese RNN model successfully apprehends the morphological similarity between strings [35]. Niu et al. [36] applied Attention mechanisms for medical concept normalization. Furthermore, the evolution of Transformer-based models capacitate the adoption pre-trained language models such as BERT [8] for entity linking problems [37].

The major development in recent NEN researches is as follows. D‘Souza et al. [1] proposed an early NEN model using a rule-based model, which requires comparatively more human input when generating the rules. The model is static and, thus, there is a possibility that new rules need to be created when applying the model to other datasets. NEN models that use more advanced machine learning and deep learning techniques can be more effective. Leaman et al. [12] used semi-Markov model, Li et al. [38] used word-level CNN model, and Wirght and Dustin [39] and Phan et al. [40] models based on BiGRU and BiLSTM. However, BERT achieved state-of-the-art performance in many general text mining and natural language processing (NLP) challenges. Compared with the four models illustrated above, the most recent researches such as the BERT ranking model [6] and BioSyn [7] takes full advantage of the BERT model by training the model based on BERT embeddings. The BERT Ranking model [6] used ranking-based objective function and BioSyn [7] used Synonym Marginalization techniques as the objective function for training. Our proposed model optimizes BERT embedding vectors with named entity graph’s edge weight updating neural network. Our proposed model successfully captures the ground truth linkage between named entity graphs, achieving the highest accuracies. Previous NEN researches focus mainly on the NEN dataset from a specific domain. To test the efficacy of our model in more general NEN tasks, we evaluate our model with NEN datasets from both the bioinformatics domain and financial domain.

Many NEN researches explore semi-supervised learning models. Our proposed model is motivated by one of the leading semi-supervised models on images, Edge-Labeling Graph Neural Network for Few-shot Learning [41] (EGNN). The major difference between EGNN and our model is that EGNN labels an edge for each round of training but our model updates edge weights for top K connected entities. By capturing more node and edge information simultaneously for each round of training, the proposed model shows better performance compared with other NEN models.

Proposed Method

Our proposed model, Edge Weight Updating Neural Network, consists of four major parts.

1. Ground Truth Entity Graph construction.

2. Similarity-Based Entity Graph construction.

3. Edge Weight Updating Neural Network training.

4. Edge Weight Updating Neural Network inferencing.

The basic idea behind Edge Weight Updating Neural Network is to minimize the Ground Truth Entity Graph’s edge weight distributions and the Similarity-Based Entity Graph’s edge weight distributions. The detailed flow diagram of Edge Weight Updating Neural Network is represented in Fig. . Our motivation for constructing Edge Weight Updating Neural Network is to provide more positive and negative samples for training at once. Training the model to minimize the ground truth data and reconstructed data is more widely used in the deep learning models in computer vision such as GAN [42]. We apply these ideas to text mining and create the model that is trained to minimize the edge weight distributions of the ground truth entity graph and that of the similarity-based entity graph. In Fig. 1, AWS is our query entity. First, with the vanilla BERT encoder, unrelated entities such as Apple, Inc. and NYSE might have higher similarity scores(edge weight between two entities) than the entity Amazon AWS. Then, the similarity-based graph is constructed with the given query entities according to the current similarity scores. The graph’s edge weight distribution is compared with the ground truth entity graph’s edge weight distributions. Entity embeddings are trained with Kullback–Leibler divergence [43] loss between two graphs. After iterating trough these steps, the BERT encoder will be trained to calculate the two entities’ ground truth similarity score.

Fig. 1.

Diagram of edge weight updating neural network with the query entity, AWS

We use the BERT model for the named entity embeddings. There are two main reasons for choosing the BERT model for the named entity embeddings. There exist various pretrained BERT models that serve the specific purposes such as BioBERT [44] for bioinformatics documents, FinBERT [45] for finance documents, and PatentBERT [46] for patent documents. Compared to other language embedding models such as Word2Vec [47], Glove [48], and Fasttext [49], BERT model’s WordPiece tokenizer is more robust handling the out of vocabulary problems. The number of out of vocabulary entities using the pretrained language models listed above are tabulated in Table .

Table 1.

Number of out of vocabulary entities using pretrained language models

Model (total # entities)	Word2Vec	Glove	Fasttext	BERT-based models
NCBI disease (73,181)	8469	7991	7279	0
BC5CDR disease (73,548)	8526	8077	7319	0
BC5CDR chemical (407,428)	87,859	70,683	70,091	0
Financial NEN dataset (24,195)	3871	4300	3615	0

Detailed steps for constructing the Ground Truth Named Entity Graph, building the Similarity-Based Entity Graph, and training and inferencing the Edge Weight Updating Neural Network are presented in Sects. 3.1, 3.2, 3.3 and 3.4, respectively.

Ground Truth Entity Graph Construction

Ground Truth Entity Graph constructions are based on entity mentions in each dataset and their mapping concept IDs. Figure  demonstrates the steps for building the graph.

For the NEN corpus, each entity is annotated with one or more concept IDs. For example in Fig. 2, entities A, B, and C share the same concept ID, ID_1. Then, entities A, B, and C are fully connected in the entity graph. Other entity pairs, D - E (concept ID: ID_2) and F - G (concept ID: ID_3) are linked. The training dataset for each NEN corpus has query entities with corresponding concept ID. If query entity Q has a concept ID of ID_1, then, query entity Q will be linked to entities A, B, and C in the pre-constructed graph. As the constructed graph is the ground truth graph, each edge weight in the graph is 1.

Fig. 2.

Ground truth named entity graph construction

We iterate all the entities in training sets that include the referencing dictionary entity table and the query entity table. Graph created by the following steps above is the Ground Truth Entity Graph which is the reference or the target graph the Similarity-Based Entity Graph will try to match.

Similarity-Based Entity Graph Construction

For each query entity, Similarity-Based Entity Graph is constructed as follows. Graph edges are calculated using BERT embedding vector similarities. We use BioBERT [44] for bioinformatics NEN corpus’ initial BERT embeddings and the original BERT [8] for financial NEN corpus’ initial BERT embeddings.

For example, in Fig. , let query entity Q has size of 768 (vector length of BERT embeddings), $Embe{d}_Q=(X_{Q1}{X}_{Q2}\cdots X_{Q768})$. Similarly, BERT-based entity embeddings in the dictionary set are also denoted as $Embe{d}_{\mathit{entity}}=(X_{entity1}{X}_{entity2}\cdots X_{entity768})$. The BERT embedding has a fixed length of 768, so our embedding vectors have a vector length of 768.

To calculate the edge weights based on entity similarities, we calculate inner products between query entities and dictionary entities. Since the calculation time for the matrix multiplication is fast, computing the similarity between every query entities and dictionary entities are relatively less time consuming. The largest dataset, BC5CDR Chemical, with the matrix size of (407,454 * 768) by (1317 * 768) takes about 0.77 s on CPU and 0.27 s on GPU. The fastest calculation time on CPU is NCBI Diesease, (72,887 * 768) by (1,587 * 768), which take 0.20 s. The fastest calculation time on GPU is Finance NEN dataset, (20,071 * 768) by (20,071 * 768), which take 0.042 s. $<,>$ is the notation for inner product and $Si{m}_Q$ is the set of similarities between query entity Q and all the entities in a dictionary; then the similarity between each query entity and each dictionary entity calculation is expressed as Eq. 1,

$$\begin{array}{c}\hfill Si{m}_Q=\{Sim\mid Sim=<Embe{d}_Q,Embe{d}_D>forD\in Dictionary\}\end{array}$$

where,

$$\begin{array}{cc}\hfill Embe{d}_{Q\in QueryEntities}& =(X_{Q1}{X}_{Q2}\cdots X_{Q768}),\hfill \\ \hfill Embe{d}_{D\in Dictionary}& =(X_{D1}{X}_{D2}\cdots X_{D768})\hfill \end{array}$$ 1

We normalize the similarity score by dividing the maximum similarity score in each query entity’s similarity score set, $Si{m}_Q$. For Similarity-Based Entity Graph, top K edges based on similarity score are selected. Highlighted blue region in entity similarity table for query entity Q in Fig. 3 demonstrates the edge weight determination steps when $top\_k=5$. Mathematically, edge weights are calculated using Eq. 2.

$$\begin{array}{cc}& ConnectedEdgeWeight=\{Weigh{t}_{\mathit{Edge}}\mid Edge\in ConnectedEdge\}\hfill \end{array}$$

where,

$$\begin{array}{cc}\hfill \mathit{ConnectedEdge}& =argma{x}_{top\_k}\{Weigh{t}_Q\},\hfill \\ \hfill Weigh{t}_Q& =\left\{W,e,i,g,h,t,\mid ,W,e,i,g,h,t,=,\frac{\mathit{Sim}}{Ma{x}_Q},,f,o,r,,S,i,m,\in ,S,i,m_Q\right\},\hfill \\ \hfill Ma{x}_Q& =max\{Si{m}_Q\}\hfill \end{array}$$ 2

For each training epoch, which is illustrated in Sect. 3.3, edge weights are updated. Updated entity embedding vectors generate new similarity scores that alter the edge weights in the graph.

Fig. 3.

Entity matching graph based on entity similarity construction

Edge Weight Updating Neural Network Training

Fig. 4.

Minimizing the edge weight distributions in edge weight updating neural network for query entity Q

The main concept of Edge Weight Updating Neural Network is to minimize the difference between the edge weights’ discrete distribution for each query entity in the Ground Truth Entity Graph and the Similarity-Based Entity Graph. The Similarity-Based Entity Graph is dynamic. For the each iteration in the training phase, the Similarity-Based Entity Graph is reconstructed with the updated BERT model’s parameters from the previous traning iteration. As illustrated in Sect. 3.2, edge weights are calculated by entities’ embeddings. In each training epoch in Edge Weight Updating Neural Network, baseline BERT model’s parameters are optimized to mimic the ground truth edge weight distributions.

Figure  shows the training process of our proposed model for the number of connected edges in the Similarity-Based Entity Graph is 5 ($top\_k=5$). Following the example in Sect. 3.2, query entity Q is connected to dictionary entities A, B, C, D, and F, and edge weights are 0.8, 0.9, 0.6, 0.7, and 0.5, respectively. Given the Ground Truth Entity Graph in Sect. 3.1, the truth edge weights for connected edges between query entity Q and dictionary entities, A, B, C, D, and F are 1, 1, 1, 0, and 0, respectively.

In training procedures, BERT parameters are tuned to make edge weights distributions in Similarity-Based Entity Graph closer to the ground truth edge weight distributions. We use Kullback–Leibler Divergence Loss [43](KL divergence loss, henceforth) for training our model. As edge weight distribution is discrete, we normalize the edge weights using the Softmax function.

We denote graph as G, entity as V, and edge as E. The Ground Truth Entity Graph and the Similarity-Based Entity Graph are denoted as $G_{\mathit{GT}}=(V_{\mathit{GT}},E_{\mathit{GT}})$ and $G_{\mathit{Sim}}=(V_{\mathit{Sim}},E_{\mathit{Sim}})$, respectively. The adjacency matrices for Ground Truth Entity Graph and the Similarity-Based Entity Graph are denoted $GT\_A$ and $Sim\_A$. $P_{Sim\_Edg{e}_Q}$ is the discrete distribution of edge weights of Q in Similarity-Based Entity Graph. $P_{GT\_Edg{e}_Q}$ is the discrete distribution of edge weights of Q in the Ground Truth Entity Graph. Our KL divergence loss is calculated using Eq. 3.

$$\begin{array}{c}\hfill Loss=KL(P_{GT\_Edg{e}_Q}||P_{Sim\_Edg{e}_Q})=P_{GT\_Edg{e}_Q}·log\left(\frac{P_{GT\_Edg{e}_Q}}{P_{Sim\_Edg{e}_Q}}\right)\end{array}$$

where,

$$\begin{array}{cc}& P_{GT\_Edg{e}_Q}=Softmax(GT\_Edg{e}_Q),\hfill \\ \hfill & P_{Sim\_Edg{e}_Q}=Softmax(Sim\_Edg{e}_Q),\hfill \\ \hfill & GT\_Edg{e}_Q=\{GT\_A_{query,d}\mid d\in argma{x}_{top\_k}\{Sim\_A_Q\}\},\hfill \\ \hfill & Sim\_Edg{e}_Q=\{Sim\_A_{query,d}\mid d\in argma{x}_{top\_k}\{Sim\_A_Q\}\},\hfill \\ \hfill & \text{and,}{A}_Q\text{is the edge weight vector connected to the given query entity node}Q\hfill \end{array}$$ 3

We use an Adam optimizer with weight decay [50], and set the batch size to 16 and the number of connected edges in the Similarity-Based Entity Graph to 30 ($K=30$) for all datasets we test. We train our model for 50 epochs. The best scores are reported in Sect. 4.

Edge Weight Updating Neural Network Inferencing

Fig. 5.

Inferencing the edge weight distributions in edge weight updating neural network for query entity Q

First, fine-tuned BERT embeddings illustrated in Sect. 3.3 are used to embed unseen query entities in test sets. With newly computed BERT embedding vectors, we repeat the steps in Sect. 3.2 to construct the new Similarity-Based Entity Graph. For each query entity, a dictionary entity with the highest edge weights is returned as a synonym. Figure  demonstrates the inferencing process of the Edge Weight Updating Neural Network.

Experiment Settings

Dataset

Named Entity Normalization Datasets in Bioinformatics

Most NEN researches are from the bioinformatics domain. To test our model’s performance with other NEN models, we select three of the most used bioinforinmatics NEN datasets: NCBI Disease [18] and two datasets from Biocreinative V CDR (BC5CDR, henceforth) [51].

Three datasets summarized below contains bioinformatics-related entity mentions with unique concept IDs. The main goal of these datasets is to identify the mentions that share the same concept IDs. We follow NEN preprocessing convention for the datasets below, in which the mentions that do not exist in the concept dictionary are eliminated [40]. Bioinformatics NEN datasets usually consist of train, development, and test sets. Following previous studies, we use train and development sets for training our model. Test sets are used for evaluations.

NCBI Disease [18] NCBI Disease corpus provides disease mentions in different surface forms. Disease mentions in this dataset are extracted from 793 PubMed abstracts containing a total of 6,892 disease mentions, which are mapped to 790 unique disease concepts. Disease concepts are annotated by Medical Subject Headings (MeSH) and Online Mendelian Inheritance in Man (OMIM). Disease mentions sharing the same disease concept are considered synonyms. Table shows detailed statistics of the NCBI Disease corpus.

Biocreative V CDR Disease and Biocreative V CDR Chemical [51] The BC5CDR corpus is organized for challenging tasks of disease named entity recognition and chemical-induced disease relation extraction. The BC5CDR corpus consists of 1500 PubMed articles with 4409 annotated chemicals, and 5818 disease and 3116 chemical-disease interactions [51]. The dataset contains disease mention corpus and chemical mention corpus. Disease mentions are mapped into the MeSH IDs similar to the NCBI Disease corpus. Chemical mentions are annotated using the Comparative Toxicogenomics Database (CTD) [52]. Mentions that share the same disease concept and chemical concept based on MeSH ID and CTD ID are considered synonyms. Detailed statistics of both BC5CDR Disease corpus and BC5CDR Chemical corpus are illustrated in Table 2.

Table 2.

Data statistics of three bioinformatics NEN datasets

	# of documents			# of mentions (entities)
	Train	Dev	Test	Train	Dev	Test
NCBI disease	592	100	100	5134	787	960
BC5CDR disease	500	500	500	4182	4244	4424
BC5CDR chemical	500	500	500	5203	5347	5385

Named Entity Normalization Datasets in Finance

There are no publicly open financial NEN datasets available; therefore, we constructed our own financial NEN dataset to test the performance of our proposed model in NEN tasks other than the bioinformatics domain.

Overview We construct the dataset for the financial NEN task from the annual reports (Form 10-K) of Standard and Poor’s 500 listed companies (https://github.com/sjeon7/Financial-Named-Entity-Normalization-Dataset). We aim to build the dataset that fulfills the need for financial NEN; the dataset includes (1) synonyms, (2) abbreviations, (3) acronyms, (4) different combinations of punctuations and alphabets, (5) descriptive phrases, and (6) possible NER parsing errors. A detailed explanation of primary data sources, data preprocessing steps, and dataset construction procedures are as follows. Figure  demonstrates the overall flow diagram for NEN dataset construction.

Fig. 6.

Flow diagram of the overall dataset construction

Data Source We gather the year 2019’s Form 10-Ks (published early 2020) of S &P500 companies from the U.S. firms and Exchange Commission (SEC) website,1 which is open to the public. We parse the business section of each 10-K documents from 496 companies. The business section of 10-K is considered the self-identity of firms and presents the information of main products, competitors, partners, and laws affecting the business. Among the sections in 10-K, this section contains the most number of entities. Out of 496 companies’ business section, 67,792 sentences were parsed.

Data Preprocessing For NER in financial documents, we implement the BERT NER model [8] using Huggingface’s2 Python repository. Huggingface’s NER model is trained using CoNLL-2003 NER dataset [53]. The outputs of the BERT NER model are WordPiece tokens that we have to link together with specified rules that will be circumstantially described below. There are four types of entity types: persons (PER), organizations (ORG), locations (LOC) and miscellaneous names (MISC), and one outside the named entity tag (O) in the CoNLL-2003 dataset. We detect entities with ORG and MISC tags. For the year 2019 S &P500 firms’ 10-K, we parse a total of 41,593 named entities.

Dataset Construction With named entities recognized illustrated in Sect. , we construct the financial named entity normalization dataset. As mentioned in Sect. 4.1.2, our focus is to build a NEN dataset to meet the need for general text mining in finance; the dataset includes (1) synonyms, (2) abbreviations, (3) acronyms, (4) different combinations of punctuations and alphabets, (5) descriptive phrases, and (6) possible NER parsing errors. We hand label a total of 7155 unique named entities into 2600 groups; with each group sharing the same identity. Table shows three examples in our dataset for types of named entities that need to be normalized.

Table 3.

Example of financial named entity normalization dataset

	Named entity	Matching named entity
Synonyms	Coca-Cola®	Coca-Cola
	COVID-19 pandemic	COVID-19
	iPhone 11 Pro Max	iPhone®
Abbreviations	Baker Hughes company	Baker Hughes Co
	Comcast corporation	Comcast corp
	Qualcomm incorporated	Qualcomm Inc
Acronyms	Amazon Web Services	AWS
	Bank of New York Mellon	BNY Mellon
	New York Stock Exchange	NYSE
Combinations of punctuations	Apple, Inc	Apple Inc
	Walmart U. S	Walmart U. S
	Booz Allen & Hamilton	Booz Allen Hamilton
Descriptive phrases	EY (formerly Ernst & Young)	Ernst and Young
	Securities Exchange Act of 1934	(The exchange act)
	Facebook (including Instagram)	Facebook®
NER parsing errors	Disney channel-the	Disney channel
	Full throttle®-a	Full throttle®)
	Keystone-our	Keystone foods

• Synonyms There exist entities with the suffix “^®" or “™". “Coca-Cola^®" and “Coca-Cola" are the same entity. In addition, “COVID-19 Pandemic" and “COVID-19" should be linked. We generalize the product model numbers in which “iPhone 11 Pro Max" and “iPhone^®" are considered identical entities.

• Abbreviations Most abbreviations occur for abridging “Company" to “Co.", “Corporation" to “Corp.", and “Incorporated" to “Inc.".

• Acronyms Acronyms are one of the most challenging NEN tasks. There are multiple abbreviations that are included in financial documents. We avoided matching acronyms if there are multiple original entities can be assigned. For example, “Advanced Development Programs (ADP )" and “Automatic Data Processing, Inc. (ADP)" both share the same acronyms, “ADP", but these should not be linked together.

• Combinations of punctuations The different combinations of punctuations problems can be solved using rule-based approaches. However, there are many entities with a combination of punctuations. “,", “.", and “ &" are commonly found and used interchangeably.

• Descriptive phrases In parsed named entity, an entity with descriptive phrases can be frequently found. With or without descriptive phrases, the root or the identified entity is invariable.

• NER parsing errors No NER models and entity concatenation models are perfect. If NER is conducted manually, there are possible human errors too. According to our dataset, one common error model makes is appending the following token after “-" token. NER parsing error correction is one of the important targets our NEN model aims to achieve.

Table 4.

Statistics of the financial named entity normalization dataset

	Train	Development	Test	Total
# of Identical entity groups	1710	800	90	2600
# of Positive pairs	4598	2466	3761	10,825
# of Negative pairs	7902	2534	3739	14,175
# of Pairs total	12,500	5000	7500	25,000

Hand-matched entity pairs are labeled positive. We also added negatively labeled pairs in which two entities have no relationship. A total of 25,000 pairs with 10,825 positive matching pairs and 14,175 negative pairs are created. We separate entity groups for a train set, development set, and test set in which there are no overlapping groups. This eliminates possible training bias, especially when training the model with entities’ graph topology. Table shows the statistics of our financial NEN dataset. For training the models, we use train and development set for training and test set for evaluation similar to bioinformatics datasets.

Experiment Settings: Named Entity Normalization in Bioinformatics

We compare our proposed model’s performance with seven different biomedical NEN models. The accuracy score presented in this study is excerpted from original papers. A summary of each model is illustrated in Table .

Table 5.

Models used in bioinformatics NEN datasets evaluations

Models	Descriptions
Sieve-based [1]	This is one the earliest NEN papers. The research was conducted with 10 Sieve, which is mostly a rule-based approaches. Many published post this research follow similar preprocessing steps
Taggerone [12]	Taggerone used the semi-Markov model for both NER and NEN tasks. Taggerone was originally validated on the NCBI Disease and BC5CDR corpus
CNN Ranking [38]	CNN Ranking model used a word-level deep learning approach for NEN. This research did not perform better than the previous model, Taggerone. However, it was the first study that applied deep learning to NEN tasks
NormCo [39]	NormCo used BiGRU, which is considered to be a better performing deep learning model with text data. NormCo achieved similar accuracy scores with significantly fewer parameters
BNE [40]	BNE introduced two-level BiLSTM to capture both character-level and word-level information of biomedical entities, achieving increased NEN performance
BERT ranking [6]	BERT Ranking model is based on Transformer-based embeddings that use the pre-trained BERT [8], BioBERT [44], and ClinicalBERT [54] for their entity embeddings. For each entity, candidate concepts were retrieved and three different BERT models are fine-tuned to rank and to capture the ground truth concepts
TripletNet [55]	The concept of TripletNet [56] for semi-supervised learning was introduced for NEN tasks. This study uses CNN for entity embedding and shared CNN parameters are trained with TripletNet structure
BioSyn [7]	BioSyn uses BioBERT for entity embeddings and trained with Synonym Marginalization. Marginal Maximum Likelihood (MML) is the objective function for Synonym Marginalization

Experiment Settings: Named Entity Normalization in Finance

The dataset we used is covered in Sect. 4.1.2. Table shows each model used in NEN in Finance is tested. BioSyn is one of the state-of-the-art NEN model and the model’s code is opened to public. We modified BioSyn for NEN dataset for finance domain and compared the performance. The experiments are conducted using Intel Core-i9-10940X CPU with 128GB memory and three NVIDIA GeForce Titan RTX GPU. To avoid possible biases caused by exogenous variables, we use the same setting for all models if applicable.

Table 6.

Models used in finance NEN datasets evaluations

Models	Descriptions
Edit Distance [57]	Edit Distance is suitable for basic NEN tasks for linking “Apple Inc" and “Apple Inc.". However, Edit Distance can only capture the superficial morphological similarity between two entities. In our experiment, we calculate the Edit Distance between two entity pairs and train a simple classifier to determine the equivalence of two entities
BERT [8]	BERT is a state-of-the-art model for various NLP tasks. However, for our specific tasks, the BERT model has a limitation on capturing morphological similarity between entity pairs. We use pre-trained BERT vectors with size 768 and train a simple MLP classifier with batch size 4096 to determine the linkage between entity pairs
Siamese GCN [58]	We use the entity graph illustrated in Sect. 3.2 and we use a pre-trained BERT vector for each entity node vector. 2-layer Siamese GCN is used in our experiment with 256 hidden nodes for the first GCN layer and 16 hidden nodes for the second GCN layer. GCN requires more epochs for training so we trained for 120 epochs for the full dataset (full batch: 17,500 entity pairs). The learning rate for ADAM optimizer for GCN is 0.01
Siamese BiLSTM [59]	For Character Level Siamese BiLSTM model training, we one-hot encoded the characters entity strings with unique 85 tokens. We stack two BiLSTM layers. The BiLSTM cells in the first layer return 64 dimension hidden states output and the BiLSTM cells in the second layer return 16 dimension hidden states output. To prevent overfitting, we train the BiLSTM model for 12 epochs. The BiLSTM model is trained with a learning rate of 0.001. Embedding dimension, 16, is the same as GCN
BioSyn [7]	The detailed model description is illustrated in Table 5

For the pairwise NEN tasks, we take slightly different approach for BioSyn and our proposed model. The convention for the performance test in NEN tasks in bioinformatics is more similar to retrieval tasks. In bioinformatics NEN performance test, the model retrieve most similar entity from the candidate concepts (mentions). For each trial, the model scores if the retrieved named entity’s concept ID matches the query entity’s concept ID. However, for the pairwise NEN tasks, the model will retrieve the top-k most similar entities for both entities in the pair. We test BioSyn and our proposed model with the top-k of 1 and 3. The model scores if there exists the overlapping concept IDs in two groups of retrieved entities. For top-1, the model will recommend the one most similar entity and their concept IDs will be compared, and for top-3, the model will recommend the three most similar entities and will be scored by the presence of any overlapping concept IDs in two entity groups.

Results

We conduct both quantitative and qualitative analysis. For NCBI Disease, BC5CDR Disease, and BC5CDR Chemical datasets, we compare our proposed model’s score with previous researches. Bioinformatics datasets are reported by top one recommendation accuracy. Given the biomedical entity in the train set, entities are matched with the most similar entities in datasets. If the query entity and target entity share the same concept ID, it is considered correct. The financial NEN dataset is a pairwise NEN matching corpus. For evaluations on the financial NEN dataset, models that are used in evaluations distinguish whether two named entity pairs share identical meanings or not. We also perform the qualitative analysis to assess models’ weaknesses.

Quantitative Analysis: Bioinformatics

Table 7.

Bioinformatics named entity normalization performance test

	NCBI disease	BC5CDR disease	BC5CDR chemical
Sieve-based [1]	84.7	84.1	90.7
Taggerone [12]	87.7	88.9	94.1
CNN ranking [38]	86.1	–	–
NormCo [39]	87.8	88.0	–
BNE [40]	87.7	90.6	95.8
BERT ranking [6]	89.1	–	–
TripletNet [55]	90.0	–	–
BioSyn [7]	90.7	92.9	96.6
BioSyn with TF-IDF [7]	91.1	93.2	96.6
Proposed model	91.7	93.4	96.7
Proposed model with TF-IDF	92.1	93.7	96.7

Bold faced numbers indicate the best scores and underlined numbers indicate the second best scores

Table shows a performance comparison between our proposed model and previous state-of-the-art models. The best scores are boldfaced and the second best scores are underlined. We also train and report our model’s performance with TF-IDF vectors added to the vanilla embedding vectors that is illustrated on the previous state-of-the-art model, BioSyn [7]. For three bioinformatics datasets, our proposed model achieved the highest accuracy. Our model showed the highest performance increase by 1.0% in the NCBI Disease corpus. For BC5CDR Disease and BC5CDR Chemical corpus, the performance increase compared the previous state-of-the-art model is 0.5% and 0.1%, respectively.

The NCBI Disease corpus is a comparatively harder dataset based on the performance of other models. We conclude that there there is a significant to increase the accuracy in a relatively lower performing dataset. The previous model already performs excellently on the the BC5CDR corpus with accuracy scores of 93.2–96.6%. Especially, the earlier NEN models on BC5CDR Chemical dataset already achieved over 90% and the updates on the performance is decreased yearly, the improvement on this specific dataset might reached the plateau.

Quantitative Analysis: Finance

Table 8.

Precision, Recall, F-score, Accuracy of models

	Precision (%)	Recall (%)	F-score (%)	Accuracy (%)
Edit distance	43.12	63.35	51.31	62.60
BERT	62.92	82.17	71.27	76.81
Siamese GCN	79.00	82.16	80.55	82.56
Siamese BiLSTM	75.96	89.98	82.38	85.15
BioSyn [7] by top1	99.73	77.77	87.39	88.75
BioSyn [7] by top3	99.38	89.79	94.34	94.60
Proposed model by top1	99.82	90.88	95.14	95.35
Proposed model by top3	99.68	98.03	98.85	98.85

Bold faced numbers indicate the best scores and underlined numbers indicate the second best scores

Table shows the performance of each model we test. The evaluation metrics are expressed as follows

$$\begin{array}{cc}\hfill \mathit{precision}& =\frac{\mathit{tp}}{tp+fp}\hfill \\ \hfill recall& =\frac{\mathit{tp}}{tp+fn}\hfill \\ \hfill F1& =2·\frac{precision·recall}{precision+recall}\hfill \end{array}$$ 4

False positive indicates that two entities should not be matched, but our proposed model decided to link two entities. False negative indicates that two entities should be matched, but our proposed model failed to link two entities.

For practical use in the NEN model in the finance domain, a model with higher precision should be rewarded more. In practice, a model with higher precision will reduce the burden for practitioners’ tasks by giving more reliable entity-matching results. A model with higher precision will reduce time double-checking the validity entity pairs marked as matched.

Edit Distance had the lowest score along with all performance evaluation indicators. Graph Convolutional Network we use for the experiments adopts the BERT vector as entity node features. BERT and GCN have a similar recall, but GCN has higher precision, which brings higher F-score and accuracy compared with BERT. Our proposed model achieved the highest precision, F-score, and accuracy. Among all the models, our proposed model is the only model with a precision score over 90%. Therefore, our proposed model is the most suitable for practical use.

Qualitative Analysis

Error Analysis

In error analysis, entities for which accurate recommendations are not made are reported. Through error analysis, we aim to recognize the pattern of cases where recommendations are not properly made.

Table 9.

Error analysis on three biomedical NEN datasets

	Query Entity	Retrieved Synonym Entity
NCBI Disease	Encephalopathy	Aids encephalopathy
	Nail dystrophy	Twenty nail dystrophy
	cdm	cdmd
	Copper overload	Copper deficient
	g m2 gangliosidosis	g m2 gangliosidosis type ii
BC5CDR Disease	Lung mass	Liver mass
	Hypoactivity	Hyperactivity
	htn	htx
	Thrombocytopenia type ii	Thrombocytopenia 2
	Chronic liver disease	Chronic hepatitis
BC5CDR Chemical	Inorganic as	Chemicals inorganic
	Alcohol nicotine	Alcohol nicotinyl
	dph	ddph
	naoh	Natrolite
	myo inositol 1 phosphate	myo inositol 1 3 6 triphosphate

Table lists the errors in three bioinformatics NEN datasets. Our proposed model achieves approximately 90% accuracy for all three datasets. However, finding the synonyms for short abbreviations such as “cdm", “htn", and “dph" seems relatively harder. In addition, if there exist longer overlapping strings, the performance of the model is degraded.

Table 10.

Error analysis on financial NEN dataset

Entity 1	Entity 2
False positive
Park Hyatt	(Marriott, Hyatt, Hilton and AccorHotels)
AT & T acquisition	AT & T Corp. ’ s (ATTC)
Bureau of Indian affairs (BIA)	Balanced budget Act of 1997 (BBA)
Garmin corporation	Garmin GTN Xi
Windows server	Microsoft teams
False negative
(LTE)	4 G
EU member state	European Union Member States
RPS	Renewable Portfolio Standards (RPS)
737-800 Boeing 737	(B737)
Cyber Security Regulation	Privacy and Cyber Security Regulation

Financial NEN datasets are constructed using entity pairs. Our model predicts whether two entity pairs are matched or not. Table is divided into false positive lists and false-negative lists. By examine the false-positive lists, entities with similar meanings or with matching strings are often predicted positive while the actual label is negative.

We also examine the false negatives. Matching named entities with parenthesis and abbreviations is the part where our model’s prediction is relatively unstable. Entity pairs such as “LTE" and “4 G" can be one of the most difficult to predict as positive because the intrinsic meaning of “LTE" and “4 G" requires common sense. Even our model is based on BERT, which captures the semantic meaning from the sentences where named entities are excerpted, using the common sense beyond the information presented in surrounding sentences can be limited.

Named Entity Normalization Result According to Training Progresses

Table 11.

Named entity normalization result for Epoch 0, Epoch 1, and Epoch with highest accuracy

	Epoch 0	Epoch 1	Epoch with Highest Accuracy
NCBI disease: c2 deficiency
Top 1	c2 deficiency	c2 deficiency	c2 deficiency
Top 2	c3 deficiency	c6 deficiency	c2 deficient
Top 3	t2 deficiency	c3 deficiency	Hereditary c2 deficiency
Top 4	c5 deficiency	c2 deficient	type ii c2 deficiency
Top 5	cpox deficiency	c4 deficiency	type i c2 deficiency
BC5CDR disease: failing left ventricle
Top 1	Tumor cerebral ventricle	Dysfunction left ventricular	Left sided heart failure
Top 2	Cerebral ventricle tumor	Left sided heart failure	Heart failure
Top 3	Tumors cerebral ventricle	Remodeling left ventricular	Cardiac failure
Top 4	Syndrome slit ventricle	Hypertrophy left ventricular	Heart failure left sided
Top 5	Ventricle tumor cerebral	Outflow obstruction left ventricular	Right sided heart failure
BC5CDR Chemical: vincristine sulfate
Top 1	Vincristine sulfate	Vincristine sulfate	Vincristine sulfate
Top 2	Sulfate vincristine	Vincristine	Vincristine
Top 3	Vinblastine sulfate	Voacristine	Sulfate vincristine
Top 4	Sulfate vinblastine	Leurocristine	Vincristin
Top 5	Riboflavin 3 sulfate	Ergocristine	Vincristin medac
Financial NEN: Polo Ralph Lauren Children
Top 1	Pinky swear foundation	Polo Golf Ralph Lauren	Polo Ralph Lauren
Top 2	Bath & body works Canada	Polo Ralph Lauren	Polo Ralph Lauren Children, Chaps
Top 3	Ticketmaster North America	Siemens Medical Solutions USA	Polo Golf Ralph Lauren
Top 4	LIP-BU TAN	Mojo Networks, Inc	Lilly International
Top 5	Coca-Cola life	Polo Ralph Lauren Children, Chaps	Polo / Lauren Company, LP

Bold-underlined entities are the entities with the same concept ID as the query entity

As the training epochs increase, recommendations become more accurate. We randomly selected entities from four datasets we tested. Top 5 recommendations for the selected entities are provided for epoch 0, epoch 1, and epoch with best result in Sects. 5.1 and 5.2.

Table shows how recommendations change as training progress. Entities after each dataset are the examples excerpted (c2 deficiency, failing left ventricle, vincristine sulfate, and Polo Ralph Lauren Children), and bold-underlined entities are the entities with the same concept ID as the query entity. Throughout the datasets, at epoch 0, the recommended entities differ greatly from the concept ID of the query entity. As the model is trained, the recommendation becomes more accurate in epoch 1. At the epochs in which the highest accuracy for the datasets is achieved, true synonyms for query entities are successfully selected.

Based on our experiments, our proposed model has the highest precision, recall, F1 score, and accuracy. Qualitative analysis shows that our proposed model also gives the most stable results by achieving over 98% on the four evaluation metrics.

Conclusion

We introduce Edge Weight Updating Neural Network for NEN. NEN to match extracted named entities with homogeneous identity is pivotal for many text mining tasks. We tested our model on three widely used NEN datasets, NCBI Disease, BC5CDR Disease, and BC5CDR Chemical. We also generated the NEN dataset for the finance domain. Next, we verify our model’s performance for general NEN applications.

The main contribution of this study are as follows. Our proposed model successfully links named entities with the same meanings with different surface forms. The proposed model performs best among previous NEN models. We test our model not only for bioinformatics datasets in which NEN researches are more active but also for financial NEN datasets. According to the performance of the NEN corpus in two distinct fields, our proposed model proves the efficacy for general NEN applications.

Similar to many other NEN models, the performance of linking named entities with abbreviations is comparatively lower. Matching abbreviations more accurately is one of the future works. The neural network model with our proposed Edge Weight Updating objective function performs better than other models. Providing the more general guideline for the number of training epochs and increasing the training stability is one of the future research topics.

Declarations

Conflict of Interest

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.