deepBioWSD: effective deep neural word sense disambiguation of biomedical text data

Abstract

Objective

In biomedicine, there is a wealth of information hidden in unstructured narratives such as research articles and clinical reports. To exploit these data properly, a word sense disambiguation (WSD) algorithm prevents downstream difficulties in the natural language processing applications pipeline. Supervised WSD algorithms largely outperform un- or semisupervised and knowledge-based methods; however, they train 1 separate classifier for each ambiguous term, necessitating a large number of expert-labeled training data, an unattainable goal in medical informatics. To alleviate this need, a single model that shares statistical strength across all instances and scales well with the vocabulary size is desirable.

Materials and Methods

Built on recent advances in deep learning, our deepBioWSD model leverages 1 single bidirectional long short-term memory network that makes sense prediction for any ambiguous term. In the model, first, the Unified Medical Language System sense embeddings will be computed using their text definitions; and then, after initializing the network with these embeddings, it will be trained on all (available) training data collectively. This method also considers a novel technique for automatic collection of training data from PubMed to (pre)train the network in an unsupervised manner.

Results

We use the MSH WSD dataset to compare WSD algorithms, with macro and micro accuracies employed as evaluation metrics. deepBioWSD outperforms existing models in biomedical text WSD by achieving the state-of-the-art performance of 96.82% for macro accuracy.

Conclusions

Apart from the disambiguation improvement and unsupervised training, deepBioWSD depends on considerably less number of expert-labeled data as it learns the target and the context terms jointly. These merit deepBioWSD to be conveniently deployable in real-time biomedical applications.

INTRODUCTION

With recent advances in biomedicine, we see a massive amount of biomedical text data being generated every day. To gain knowledge from these data, developing natural language processing (NLP) tools that mine them accurately within a reasonable time is crucially important. NLP components that include named entity recognition programs,¹ syntactic parsers,² and relation extractors^3,4 build the foundation of many high-level biomedical information extraction and knowledge discovery applications.^5–8 Also, it is shown that the biomedical text data such as scientific articles,⁹ clinical narratives,¹⁰ and health-related social media posts,¹¹ abound with ambiguous terms (hereafter, instead of saying ambiguous word we use ambiguous term because a [biomedical] conceptual unit that we try to disambiguate can be represented by a series of words; as in malignant B-cell lymphoma or benign B-cell lymphoma for the target ambiguous term B-cell lymphoma). In the lowest level, surrounded by this innate ambiguity, all other components and the full biomedical application will suffer if it is not resolved properly.

A word sense disambiguation (WSD) algorithm attempts to predict the correct sense of a term within a context given a set of candidates. For example, in the sentence “Ca intakes in the United States and Canada appear satisfactory among young adults,” the sense set for Ca consists of Canada ($s_1$), California ($s_2$), calcium ($s_3$), and cornu ammonis ($s_4$) and the goal is to predict the correct sense $s_3$ for this specific occurrence of Ca. It is shown that this automatically identifying the intended sense of ambiguous words improves the performance of clinical and biomedical applications such as medical coding and indexing,^12,13 detection of adverse drug event,¹⁴ automatic medical reporting,^15,16 and other secondary uses of data such as information retrieval and extraction,¹⁷ and question-answering systems.¹⁸ These capabilities are becoming essential tasks due to the growing amount of information available to researchers, the transition of healthcare documentation and patient-practitioner interaction toward electronic health records and automatic expert systems, and the push for quality and efficiency in health care.

Supervised machine learning WSD algorithms typically build 1 separate classifier for each ambiguous term, which will be trained solely on the instances of that term. That is, to train an accurate WSD model, a large amount of annotated instances are needed, the curation of which will be expensive and labor-intensive particularly in health informatics.^19,20 Recent studies in the biomedical domain incorporate expert-involved active learning techniques to accelerate the labeling process of this training data.^21,22 Nevertheless, considering the multiclassifier design of the traditional supervised WSD models, the real-world implementation of them in the domain is still impracticable.

We introduce a 1-size-fits-all deepBioWSD architecture for disambiguation of biomedical text data, a deep learning–based model that unifies all disambiguation classifiers into 1 single network. In a supervised manner, this network will be trained on all existing instances of the ambiguous terms as 1 group of training data in which sense-context pair and $s_{\text{i}}\epsilon \{1.0, 0.0\}$ constitute the input and the output, respectively. While the network encodes the shared information among all instances, for a given training-instance, it learns the senses of the unlabeled terms in the context and the sense of the labeled center term at the same time. To this end, our architecture employs a bidirectional long short-term memory network (BLSTM), and works with neural sense embeddings, which can be pretrained.

Supervised WSD in biomedicine

Jimeno-Yepes et al²³ prepared the National Library of Medicine’s MSH WSD dataset in 2011 with naive Bayes accuracy of 93.84% (NB [these abbreviations are used during evaluation of the WSD algorithms]). Later, traditional discriminative models with rigorous linguistic and biomedical specific features were used for WSD evaluation.^24,25 To avoid an intense feature engineering, recently, the state-of-the-art accuracy of 95.97% was achieved by Jimeno-Yepes²⁶ using unigrams and word embeddings with support vector machines (SVM_Yepes); they also reported the accuracy of 94.87% for their long short-term memory networks (LSTMs). In another supervised model, Antunes and Matos²⁷ used bag-of-words as local features and word embeddings as global features and reported accuracy of 95.6% when SVM classifiers were employed (SVM_Ant-Mat). To eliminate an extreme need for extensive amount of annotated data to train classifier of each term, Sabbir et al²⁸ recently developed a knowledge-based model at the cost of accuracy (92.24%, KN). In another recent knowledge-based study, Duque et al²⁹ reported accuracy of 71.52% on MSH WSD for their system called Bio-Graph that employs a PageRank algorithm to work with occurrence graphs built from Medline abstract to address WSD (Bio-Graph).

Neural embeddings for WSD

With recent interests in training neural word embeddings from large raw corpora,^30–32 several studies included pretrained word embeddings in their WSD models, some of which were concerned with biomedical text.^33–36 Lately, computation of sense embeddings has gained the attention of researchers as well where their importance in the WSD tasks has been investigated;^37–40 however, the mapping of these hardly interpretable inducted sense embeddings to a sense inventory (eg, the Unified Medical Language System [UMLS]) has been the main bottleneck for their wider employment in WSD systems.⁴¹ In the deepBioWSD model, first, we build our sense embeddings using the UMLS text definition of senses; then, these embeddings initialize our BLSTM network before training.

Bidirectional LSTM

LSTMs address the vanishing gradient problem in RNNs by incorporating gating functions into their state dynamics (see Supplementary Appendix).⁴² Standard Recurrent Neural Networks (RNNs) and LSTMs, however, have restrictions as the future input information cannot be reached from the current state, so, a Bidirectional LSTM fuses 1 forward and 1 backward LSTM.⁴³ In WSD, this means we are able to encode the information of both preceding and succeeding words with respect to a pivotal ambiguous term. Kågebäck and Salomonsson³⁵ proposed a partially shared multiclassifier WSD model with BLSTMs that employed word embeddings (BLSTM_Kåg-Sal). In our previous work, we developed a single-classifier WSD model with just 1 BLSTM network (BLSTM_Pes-etal);³⁶ this model, however, uses 2 separate word and sense spaces for the context and center words, which caused inconsistency and worse performance. As we will see, the deepBioWSD network is only dependent on sense space for both center and context terms, an architectural improvement over BLSTM_Pes-etal network for better sense prediction, faster training, and less dependency on expert-labeled data. Other existing BLSTM-based WSD algorithms are Seq2Seq-inspired models, which typically underperform conventional supervised WSD models.^44–46

Zero-shot learning

Zero-shot learning (ZSL) aims at predicting labels for instances that belong to classes that were not directly seen during training.^47,48 The underlying secret ensuring the success of ZSL is to find an intermediate semantic representation to transfer the knowledge learned from seen classes to unseen ones.⁴⁹ The scalability of the model is of utmost importance since a large amount of unlabeled data is generally present and can be received by interaction with the environment,⁵⁰ which is the case in medical informatics. We show deepBioWSD with a unitary and uniform network architecture that it offers benefits from ZSL; that also, in turn, prevents the “cold start” problem (ie, when a model cannot draw any inferences as it has not yet gathered sufficient information related to a subject matter or application; hence, training of the model from scratch with sufficient amount of labeled data seems inevitable) that exists in other supervised WSD algorithms.

Experimental data

Unified Medical Language System

The UMLS (https://www.nlm.nih.gov/research/umls/) is a terminology integration system that contains Metathesaurus and SPECIALIST Lexicon. The Metathesaurus holds ∼3.4 million biomedical and clinical concepts (hereafter, we use concept and sense [of a term] interchangeably) by maintaining their hierarchical relationships. Each concept has a unique identifier called CUI (Concept Unique Identifier), a set of representative terms, and a text definition. The Metathesaurus provided us with the sense sets of the ambiguous terms. The SPECIALIST Lexicon resource contains information about common English vocabulary and biomedical terms by offering tools for language processing. We used its programs to demarcate terms in the contexts; in our early example, the United States is an unambiguous term (CUI: C0041703) consisting of 3 words, and satisfactory is a single-word ambiguous term (C0205410, C1547307). The latest UMLS release 2018AA was used in the study. This release covers >83 000 ambiguous representative terms.

Medline abstracts

Medline includes over 20 million citations of life sciences and biomedical articles from 1966 to the present. Combined with the UMLS concept definitions, we employed Medline 2013 bigram-list (https://mbr.nlm.nih.gov/Download/) to create our sense embeddings.

Validation datasets

We employed the MSH WSD dataset (https://wsd.nlm.nih.gov/collaboration.shtml) for the evaluation of WSD algorithms.²³ This dataset provides 37 888 instances for 203 ambiguous terms (including abbreviations) that take 2–5 senses (∼100 instances per each sense are provided). Prepared from Medline, every instance of a target ambiguous term is manually annotated with a CUI within the sense set of that term. For example, an instance of Ca is labeled with either C0006823 (Canada), C0006675 (California), C0006754 (calcium), or C3887642 (cornu ammonis); while every instance of the target term lymphogranulomatosis takes the sense C0036202 (benign lymphogranulomatosis) or C0019829 (malignant lymphogranulomatosis).

MATERIALS AND METHODS

Pretraining of sense embeddings

Inspired by studies for (high-dimensional) distributed representation of biomedical concepts,^51–53 and low-dimensional vector representation of words,^54,55 we pretrained UMLS sense embeddings in 6 steps as depicted in Figure 1 . In essence, the second-order computation of vector representation of concepts prevents the issue of sparsity (of word features) in the first-order vector representation of their definitions, pointwise mutual information statistically defines the degree of relevance between each biomedical concept and its (second-order) word features, and latent semantic analysis aims at condensing the final high-dimensional vectors to a size proper for a deep neural network. These steps briefly explained below are executed in advance to compute sense embeddings of the UMLS concepts before training our deepBioWSD network which they initialize (see Supplementary Appendix for further details).

Figure 1.

The figure represents different steps in our unsupervised method to generate low-dimensional sense embeddings for the Unified Medical Language System (UMLS) concepts. These embeddings initialize of disambiguation deep neural network. C: concept; f: new feature; LSA: latent semantic analysis; PMI: pointwise mutual information; S: salient feature; SOC: second-order co-occurrence; W: word feature.

Step 1—Bigrams and Medline words co-occurrence matrix.

We built a co-occurrence matrix from the bigram-list of Medline abstract. This matrix is symmetric and sparse, and represents the contextual information of the Medline words.

Step 2—UMLS concept definition extension and definition matrix.The definition extension of concepts by their immediate concepts’ in an ontology/thesaurus enriches their semantic.^51,56 When applied to the UMLS concepts, words in the extended definitions have associated co-occurrence vectors from Medline computed in step 1. For every (extended) definition, the definition matrix stores the frequency of these word features.

Step 3—Second-order co-occurrence (SOC) matrix.To build a SOC vector of a concept, we first summed the Medline co-occurrence vectors of the content words in that concept’s extended definition, and then normalized the result vector by the number of words in the definition. In other words, we took the centroid of the vectors associated with each word in the definition, and then normalized the result to uniformly treat the different size definitions.

Step 4—Pointwise mutual information (PMI) on SOC matrix. Not all word features associated with a concept are equally important. PMI, as in equation 1, statistically measures the level of association between the concepts (their associated words; ie, $wor{d}_i$) and the word features (ie, $wor{d}_j$), instead of naive consideration of word feature frequency cutoff threshold.^57,58 Once PMI values are calculated—with respect to the (frequency) probabilities of (co-)occurrences of these words, our validation set helps to set a low cutoff threshold for the removal of irrelevant features. We applied the add-1 smoothing technique to the SOC matrix in advance to avoid bias toward infrequent occurrences.⁵²

$$PM{I}_{sense}\left(wor{d}_i,wor{d}_j\right)=log\frac{p\left(wor{d}_i,wor{d}_j\right)}{p\left(wor{d}_i\right)\times p\left(wor{d}_j\right)}$$ (1)

Step 5—Latent semantic analysis (LSA) on PMI-on-SOC matrix—LSA, given by equation 2, uses a singular value decomposition algorithm that resulted 2 square and unitary matrices $\mathit{U}$ an ${\mathit{V}}^T$, and a non-negative diagonal matrix $\Sigma$ that held singular values on its diagonal in a nonincreasing order.⁵⁹

$$PMI\_on\_SOC=\mathit{U\Sigma }{\mathit{V}}^T$$ (2)

Step 6—Reducing the rank of singular values—Having equation 3, we truncated the singular value decomposition to 100 for low-dimensional representation of UMLS concepts. Determined by our validation set, smaller embedding sizes yielded worse WSD results, and higher dimensions did not improve the accuracy and just increased the training time.

$$sense\_embeddings=\mathit{U}{\Sigma }_{100}$$ (3)

deepBioWSD network definition

In contrast to other supervised WSD networks, in which a softmax layer with a cross-entropy or hinge loss is often parametrized to select the corresponding weight matrix and bias vector for every sense of an ambiguous term, our network shares parameters over all senses. Given an instance and the position of a target term, the deepBioWSD network computes a probability distribution over candidate senses of that term.

The architecture of our network consists of 7 layers (Figure 2 ). Due to the replacement of the conventional softmax layer with a linear (regression) layer, we imposed a modification to the input. That is, apart from the contextual features, the sense for which we want to discover whether the given context is meaningful will be provided as input. For an ambiguous term with the sense set $\{s_1,\dots ,s_{\text{n}}\}$, the network runs $n$ times (for every sense) and the highest-confidence sense would be selected. In lower layers, to determine proximity of the senses and the given context, after computing cosine similarities of each candidate sense (embedding) with the senses of the context terms, the sequential result of the cosine similarities between the correct sense and the surrounding context communicate a pattern-like information that our BLSTM layer encodes—which consequently yields higher confidence in the upper regression layer; however, for the incorrect senses, this premise of homogeneity and proximity does not hold (ie, negative samples). Several studies already incorporated the idea of sense-context cosine similarities in their WSD models.^36,60–62 Nevertheless, the context terms, which are determined by the SPECIALIST Lexicon during the disambiguation process, can be ambiguous themselves. To deal with their ambiguity, just before the cosine layer, a pooling layer is devised, the result of which learns the senses of the ambiguous terms appeared in the context. This means the network takes gradients with respect to (shared) sense embeddings of both the target term and the context terms at the same time.

Figure 2.

The figure illustrates our 1-size-fits-all deepBioWSD network which treats all center ambiguous terms (and their instances) uniformly. The emb represents embedding size$, l_i$ is the current candidate sense (or label) under investigation, and ${\mathit{X}}^{\mathit{j}}$ is the jth term in the context (left or right). Besides training on the center terms, the embeddings of the context terms would be updated and learned (ie, disambiguated) during training. LSTM: long short-term memory network; UMLS: Unified Medical Language System.

UMLS Concept Embedding Layer. For one instance, the input of the network consists of a sense and a list of (left and right) context terms, which paired together form a list of context components. For context$D$, which encompasses an ambiguous term with the sense set of$\{s_1,\dots ,s_{\text{n}}\}$, the embedding layer weights for the examined input sense $s_{\text{i}}$, is determined by equation 4. Then, this input is copied to |$D$| positions of the context to form the first pair of the context-components and set the same embedding weights in the layer.

$${\mathit{l}}_{\mathit{i}}={\mathit{W}}_{\mathit{s}}^{\mathit{l}}.{\mathit{v}}_{\mathit{s}}\left({\mathit{s}}_{\mathit{i}}\right), i\in \{1,\dots ,\text{ n}\}$$ (4)

where, ${\mathit{l}}_{\mathit{i}}\in {\mathbb{R}}^{100}$ and ${\mathit{v}}_{\mathit{s}}\left({\mathit{s}}_{\mathit{i}}\right)$ is the 1-hot representation of the sense. A 1-hot representation is a vector with the dimension ${\mathit{V}}_{\mathit{s}}$ consisting of $\left|{\mathit{V}}_{\mathit{s}}\right|-1$ zeros and a single one that indicates the index of a sense in a look-up table; the ${\mathit{V}}_{\mathit{s}}$ size is equal to the number of CUIs in the UMLS. The ${\mathit{W}}_{\mathit{s}}^{\mathit{l}}$ is the shared look-up table for the center terms and context terms; it is initialized with the sense embeddings that we computed in advance. Equation 4 have the effect of picking the column (ie, a sense embedding) from ${\mathit{W}}_{\mathit{s}}^{\mathit{l}}$ corresponding to that sense.

Regarding a context term input, which form the second pair of a context component, at position $k$ in the same context $D$ the embedding weights are determined by

$${\mathit{x}}^k=\left[{\mathit{l}}_1, \dots , {\mathit{l}}_{\mathit{n}}\right] \text{ε}{\mathbb{R}}^{100\times m}, k\in \left\{-\frac{\left|D\right|}{2},\dots ,-2,-1,1,2,\dots ,\frac{\left|D\right|}{2}\right\}$$ (5)

where, ${\mathit{l}}_{\mathit{i}}$ is set by equation 4, and $k$ is the position of the term in the context (left or right) while |$D$|/2 = 125 is a hyperparameter of the network (padding or truncating was applied wherever needed). $m$ is another hyperparameter that typically should be equal to the size of the largest sense set; however, in the experiments of the study we observed an inverse relationship between the sense set size and the occurrence frequency of the terms, therefore we limited $m$ to be 5. This means only those terms in the context were inputted to the network that had the sense set of size 5 or less (ie, some infrequent terms were ignored). This resulted a faster convergence with no accuracy loss. For those terms with the sense set size of <5, a generic embedding vector of very large negative numbers was employed to fill in the void senses; this helped maxpooling consider only the sense embeddings of a context term.

Pooling and Flattening Layer. Here, max operation is applied over all rows per each context term’s sense embeddings, denoted as maxpool(5) in Figure 2. After maxpooling, each context term is represented with a 100-dimensional global feature vector. We also flattened the result column vector into a row vector as an integrated part of the maxpooling layer; that is, at position $k$ in the context, the pooling and flattening layer gives ${\overline{\mathit{l}}}^k\in {\mathbb{R}}^{1\times 100}$ for a target term sense and ${\overline{\mathit{x}}}^k\in {\mathbb{R}}^{1\times 100}$ for the predicted context term sense. Despite the intuitive use case of maxpooling to deduce the proper sense of a context term, experimentally it worked better than averagepooling.

Cosine Layer. In |$D$| positions of context components, the cosine similarities between the embedding vector of the examined sense and the maxpooled of the context terms are calculated. Computed by equation 6, the results are 2 row-vectors of size |$D$|/2 each containing the cosine similarities of the context components of their side:

$$\begin{array}{c}{\mathit{c}}_{\mathit{left}}=\left[{\mathit{c}}_{\mathit{l}}^{\mathit{k}}, \dots , {\mathit{c}}_{\mathit{l}}^{-1}\right]\in {\mathbb{R}}^{1\times |D|/2}, k\in \{-\frac{|D|}{2},\dots ,-2,-1\}\\ {\mathit{c}}_{\mathit{right}}=\left[{\mathit{c}}_{\mathit{r}}^1, \dots , {\mathit{c}}_{\mathit{r}}^{\mathit{k}}\right]\in {\mathbb{R}}^{1\times |D|/2}, k\in \{1,2,\dots ,\frac{|D|}{2}\}\\ c^i=cosine\left({\overline{\mathit{l}}}^i,{\overline{\mathit{x}}}^i\right)=\frac{{\sum }^{\stackrel{¯i}{\mathit{l}}}\odot {\overline{\mathit{x}}}^i}{\Vert {\overline{\mathit{l}}}^i\Vert \times \Vert {\overline{\mathit{x}}}^i\Vert }, k\in \{1,\dots ,k\}\end{array}$$ (6)

Bidirectional LSTM Layer. With 1 forward and 1 backward LSTM networks, we have a left context–dedicated LSTM network that receives the cosine similarities from left to right, and right context–dedicated LSTM network that receives the cosine similarities from right to left. ${\mathit{c}}_{\mathit{left}}$ and ${\mathit{c}}_{\mathit{right}}$are the inputs of these networks; their outputs are the vectors ${\mathit{h}}_{\mathit{left}}\in {\mathbb{R}}^{1\times 50}$ and ${\mathit{h}}_{\mathit{right}}\in {\mathbb{R}}^{1\times 50}$, each encoding the received information from one side of the target ambiguous term (50 is another hyperparameter).

Concatenation Layer. This layer concatenates the output row vectors of the BLSTM layer:

$${\mathit{h}}_{\mathit{merge}}=\left[{\mathit{h}}_{\mathit{left}},{\mathit{h}}_{\mathit{right}}\right]\in {\mathbb{R}}^{1\times 100}$$ (7)

Fully Connected Layer.Further, for a better representation, a hidden fully connected layer ${\mathit{h}}_{\mathit{fc}}$ is devised which is:

$${\mathit{h}}_{\mathit{fc}}=ReLU({\mathit{h}}_{\mathit{merge}} . {\mathit{W}}_{\mathit{h}}+{\mathit{b}}_{\mathit{h}})\in {\mathbb{R}}^{1\times 50}$$ (8)

where, $ReLU$ is rectified linear unit function;⁶³${\mathit{W}}_{\mathit{h}}\in {\mathbb{R}}^{100\times 50}$ and ${\mathit{b}}_{\mathit{h}}\in {\mathbb{R}}^{1\times 50}$are the weights and bias for the hidden layer. The result of this layer embeds the input sequence into a vector of size 50.

Regression for Classification Layer.This layer outputs a single value that is computed by:

$${\widehat{y}}_{s_i}={\mathit{h}}_{\mathit{fc}}.{\mathit{W}}_{\mathit{out}} +{\mathit{b}}_{\mathit{out}}, s_i\in \{s_1,\dots ,s_n\}$$ (9)

where, ${\mathit{h}}_{\mathit{fc}}$comes from the previous layer, and ${\mathit{W}}_{\mathit{out}}$and ${\mathit{b}}_{\mathit{out}}$are the weights and bias of this linear layer.

During network training, for an instance with its given context and the correct sense as inputs, ${\widehat{y}}_{s_i}$is set to be 1.0, whereas for the same context with the incorrect senses it is set to be 0.0. During testing, however, among all the senses, the output of the network for a sense that gives the highest value of ${\widehat{y}}_{s_i}$ is considered as the true sense of the ambiguous term. In other words, the correct sense is:

$$\underset{s_i}{argmax}\{{\widehat{y}}_{s_1},\dots ,{\widehat{y}}_{s_n}\}, s_i\in \{s_1,\dots ,s_n\}$$ (10)

By applying softmax to the results of the estimated values $\{{\widehat{y}}_{s_1},\dots ,{\widehat{y}}_{s_n}\}$, we can represent them as probabilities. This facilitates interpretation of them especially when deepBioWSD is benefiting from an active learning setting where intricacy and importance of 1 instance are measured.

The final recommended hyperparameters of the network which were determined during the validations are provided in Supplementary Appendix.

Unsupervised collection of training data

Considering the uniform structure of deepBioWSD, we also aimed at collecting more training data on which deepBioWSD could be pretrained. For this purpose, we employed Entrez Direct (https://www.ncbi.nlm.nih.gov/books/NBK179288/) to automatically gather data from PubMed. So, we devised a query management scheme that benefited from the notion of polyonymy of a concept (polyonomy is the employment of multiple names for the same concept): besides ambiguous representative terms, usually, one concept has other representative terms that are unambiguous (eg, lymphogranulomatosis vs malignant lymphogranulomatosis). By sending queries to PubMed for these unambiguous terms, we obtain abstracts for which we already know the true sense. It allowed us to create samples of unsupervised instances in a large quantity (see Supplementary Appendix). For each (unambiguous) sense query, we only considered the first 500 instances retrieved from PubMed (excluding the MSH WSD instances). A total of 180 175 instances were automatically prepared as PubMed returned <500 abstracts for some sense queries.

RESULTS

Sense similarity of pretrained embeddings

We will represent our method for pretraining of sense embeddings (see subsection Pre-training of sense-embeddings) plays an important role in sense prediction. This method organized the concepts based on their semantics (a physical library would be an appropriate analogy for this attempt), which will be later on introduced to the deepBioWSD network for a better and faster training. Table 1 represents a (cosine) sense similarity example for the ambiguous term CP(computed over the pretrained sense embeddings; ie, books in the library). Each column header represents one sense of CP, and the listed terms below are the closest UMLS concepts to that meaning of CP. In the table, instead of unfamiliar sense CUIs, the selected representative terms of the concepts are shown. Providing just 1 example here, we observed other senses followed the same sense similarity organization in the sense embedding space as well (see Supplementary Appendix for more examples).

Table 1.

Sense similarity for candidate senses of the ambiguous term CP

Cerebral Palsy	Propionibacterium acnes	Cleft Palate
Convulsion	Staphylococcus	Glossoptosis
Spastic syndrome	Propionibacterium	Cleft Lip
Muscle Dystonia	Stomatococcus	Omodysplasia
Dysdiadochokinesis	Micrococcus	Congenital Megacolon
Choreoathetosis	Flavobacterium	Ectromelia
Quadriplegia	Neisseriaceae	Polydactylism
Trismus	Acidovorax	Teething
Hemiplegia	Abiotrophia	Congenital Aniridia
Muscle Hypertonia	Paenibacillaceae	Omphalocele
Muscle Spasticity	Helicobacter	Syndactyly

First WSD experiment: direct learning from center terms

Between-all-models comparisons: Table 2 compares the deepBioWSD with the other WSD algorithms. Despite those for which we already had the accuracy results on MSH WSD dataset, BLSTM_Kåg-Sal and BLSTM_Pes-etal were reimplemented with their best hyperparameters chosen, a few of which were slightly different from their original papers (eg, different context size). What we report here for deepBioWSD is based on 10-fold validation experiments we conducted after considering training, validation, and test splits; other models might not necessarily follow this strategy.

Table 2.

Accuracy results for MSH WSD dataset

Method	Algorithm	Macro Accuracy (%)	Micro Accuracy (%)
Unsupervised	Bio-Graph	71.52	–
	KB	92.24	–
	deepBioWSDwith random embeddings	92.16	91.93
	deepBioWSDwithpretrained embeddings	92.67a	92.51a
Supervised	NB	93.84	–
	SVMAnt-Mat	95.60	–
	LSTM	94.87	94.78
	SVMYepes	95.97	95.81
	BLSTMKåg-Sal	95.64	95.47
	BLSTMPes-etal	95.53	95.39
	deepBioWSDwith random embeddings	93.88	93.71
	deepBioWSDwithpretrained embeddings	96.14	95.96
	deepBioWSDpretrained unsupervised w/osenseembdgs	96.64	96.47
	deepBioWSDpretrained unsupervised w/senseembdgs	96.82a	96.64a

^aWe observe deepBioWSD outperforms other measures in both supervised and unsupervised WSD settings.

Supervised.Instances of every 203 term in MSH WSD data were divided into 10 nonoverlapping folds in which 1 fold was put aside for a final testing in a 10-time validation. Training on the rest of the 9 folds, we first randomly selected 5% as a validation set to tune hyperparameters and to find the proper number of epoch the network needed to train. After hyperparameters were chosen, the final model was trained on the whole training set (including the validation set), and then was evaluated on the 203 test data folds taken out already. In the experiments, macro and micro accuracies were considered for hyperparameter tuning as well as for the final evaluation of the test data (see Supplementary Appendix). After computing the test results of the all 10 times of validation, their average was considered as the results of the models.

Unsupervised.After finding the proper structure of the network, we experimented with 2 scenarios. First, the network was trained on the automatically collected data where the MSH WSD instances made the test data. Second, the network was pretrained on these unsupervised data and then it was retrained and evaluated according to the supervised layout described previously.

These results indicate the importance of pretrained sense embeddings initializing the network. Their influence, however, is inconsiderable when the network is pretrained on the unsupervised training data. In that case, the network produces sense embeddings from scratch, and the final updated embeddings are the byproduct of the network. Overall, deepBioWSD’s single network architecture outperforms unsupervised KB and (multiclassifier) supervised WSD algorithms in the biomedical WSD task. Regarding training time, deepBioWSD also showed better efficiency (see Supplementary Appendix).

Within-our-model comparisons: We also studied if the flow of cosine similarities between a true sense and its preceding and succeeding terms (their senses) carried a sequential information that 1 BLSTM could encode and learn from. Therefore, according to what Table 3 represents, we introduced some changes in the input or in the structure of the network to verify that. We observed if we reverse the sequential flow of information into our BLSTM, we shuffle the order of the context terms, or replace our LSTMs with 2 fully connected networks of the same size 50, the achieved results were notably less than our original structure. Expectedly, due to a variable size of the original contexts (which forced padding/truncation), replacement of LSTMs with BLSTMs had negative effects.

Table 3.

deepBioWSD with other architectural settings

Algorithm	Macro Accuracy (%)
Full network in Figure 2	96.82a
BLSTM with reverse directions in Figure 2	93.86
BLSTM with a shuffled context	91.98
Fully connected layers instead of BLSTM layer	95.23
BLSTM on the left and BLSTM on the right	95.81

^aWe observe the presence and the correct direction of the BLSTM have an impact on the WSD accuracy.

Second WSD experiment: indirect learning from context terms

Considering ZSL, we also experimented if training on 1 target term’s instances led to indirect insights into other terms. As an example, assume we are training the ventricles instance, “Coronal measurements of both ventricles were similar when obtained by US and MRI images”; having ventricles (meaning cerebral ventricles here) as the target ambiguous term, we gain knowledge about the context terms as well, including US and MRI. In a new US instance, this insight helps the network to predict if US means United States or ultrasonography.

To investigate indirect learning, we randomly divided 203 number of MSH WSD terms into 10 nonoverlapping folds, and then held (instances of) 1 of the folds for testing (as unseen data) and the rest for training (10-time validation). We selected 5% of the training set as a validation set to tune hyperparameters. The final network was trained on the whole training set and then was evaluated on the test set (averaged the individual test results on the unseen target terms). Table 4 represents the average of the 10 times of validation. These results clearly represent the influence of pretrained sense embeddings on the predictions. More importantly, we observe, when deepBioWSD is not directly trained on 1 term’s instances, the preserved statistical information learned from the context (and its maxpooled embeddings) guides the network for more accurate sense prediction of that term when located at the center. Furthermore, with the current state of the network, the model will not suffer from the cold start problem because the model has been gaining the momentum, and with less number of training data needed, it will be fully trained on unseen terms in short order as well.

Table 4.

Accuracy results for indirect learning from the context terms

Stage	Supervised Setting	Macro Accuracy (%)	Micro Accuracy (%)
Before Training	deepBioWSDwith random embeddings	49.37	49.53
	deepBioWSDwithpretrained embeddings	65.46a	65.73a
After Training	deepBioWSDwith random embeddings	67.32	66.92
	deepBioWSDwithpretrained embeddings	82.08a	81.85a

^aWe observe when deepBioWSD is initialized with pretrained embeddings it performs significantly better than the random initialization of the embeddings for the same task of WSD.

Except for BLSTM_Pes-etal, for which the results of this experiment were completely random (in all cases), we could not envision and conduct the experiment for the other supervised algorithms due to their multiclassifier design.

DISCUSSION

The deepBioWSD introduces an unorthodox WSD network in which all conceptual pieces of the biomedical domain (ie, pivotal and contextual terms) are designed to be interconnected—pieces that constantly communicate information to solve the jigsaw puzzle of WSD. The network, however, found 2 types of instances challenging. First, when the syntactic structure—with similar semantic theme—surrounding the candidate senses were very similar (eg, veterinary assistant and veterinary medicine for the ambiguous term veterinary). Second, when the senses are semantically so close that they share the same immediate parent in the UMLS, or 1 term directly subsumes the other sense (as in senses for borrelia, heregulin, and HGF) (see Supplementary Appendix).

We let MeSH and SNOMED CT demarcate the context terms (following the previous studies).^60,64 We found however by adding more vocabularies from the UMLS, fewer context terms will be ignored during prediction as the model will be inclusive of more biomedical terms or senses. For example, the term 12-step program appeared frequently in the context of AA when it meant Alcoholics Anonymous (another meaning is amino acid); however, 12-step program belongs to neither MeSH nor SNOMED CT, whereas the National Cancer Institute ontology (NCI) covers it. This consideration of more vocabularies was helpful, as it slightly improved the results with a smaller context size needed. Nonetheless, with more vocabularies, the possible number of senses one term can take grows, which to some extent offsets the advantage of a smaller context size.

CONCLUSIONS

One future work direction can be consideration of other unsupervised biomedical sense embedding methods in the model. Adding an attention mechanism to the network architecture might further improve the disambiguation results as well.⁶⁵ Also, more comprehensive and systematic study for the collection of unsupervised training data is needed. The model can also be evaluated on an extrinsic task with real-world applications (eg, Clinical Information Extraction).¹⁷

FUNDING

The study has been funded in part by NSERC CREATE grant and the grant from Poland’s National Scientific Center available to SM, and in part by NSERC Discovery Grants available to SM and MS.

AUTHOR CONTRIBUTORS

AhP conceived of the presented idea, developed the theory, and investigated the network architecture. AhP and AlP performed the computations and implementation. SM and MS provided guidelines as to the test data, and verified the analytical methods. SM encouraged AhP to investigate unsupervised training of the model. SM and MS supervised the findings of this work. All authors discussed the results and contributed to the final manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Conflict of interest statement. The authors confirm that there is no competing interests.