Extracting entities with attributes in clinical text via joint deep learning

Abstract

Objective

Extracting clinical entities and their attributes is a fundamental task of natural language processing (NLP) in the medical domain. This task is typically recognized as 2 sequential subtasks in a pipeline, clinical entity or attribute recognition followed by entity-attribute relation extraction. One problem of pipeline methods is that errors from entity recognition are unavoidably passed to relation extraction. We propose a novel joint deep learning method to recognize clinical entities or attributes and extract entity-attribute relations simultaneously.

Materials and Methods

The proposed method integrates 2 state-of-the-art methods for named entity recognition and relation extraction, namely bidirectional long short-term memory with conditional random field and bidirectional long short-term memory, into a unified framework. In this method, relation constraints between clinical entities and attributes and weights of the 2 subtasks are also considered simultaneously. We compare the method with other related methods (ie, pipeline methods and other joint deep learning methods) on an existing English corpus from SemEval-2015 and a newly developed Chinese corpus.

Results

Our proposed method achieves the best F1 of 74.46% on entity recognition and the best F1 of 50.21% on relation extraction on the English corpus, and 89.32% and 88.13% on the Chinese corpora, respectively, which outperform the other methods on both tasks.

Conclusions

The joint deep learning–based method could improve both entity recognition and relation extraction from clinical text in both English and Chinese, indicating that the approach is promising.

INTRODUCTION

Extracting clinical entities and their attributes, which includes 2 subtasks of clinical entity or attribute recognition and clinical entity-attribute relation extraction, is a fundamental task of natural language processing (NLP) in the medical domain. The 2 subtasks are usually tackled sequentially in a pipeline framework, where clinical entities and attributes are recognized at first, and then relations between clinical entities and attributes are extracted. The clinical entity or attribute recognition is a typical named entity recognition (NER) task, treated as a sequence labeling problem. A large number of state-of-the-art machine learning methods, such as conditional random fields (CRF),¹ structural support vector machines,² and deep learning methods,³ have been proposed for NER. The entity-attribute relation extraction is usually recognized as a classification problem, and a variety of machine learning methods, such as support vector machines and deep learning methods, have been proposed for this task.

One issue of the pipeline framework is that errors from entity recognition are unavoidably passed to relation extraction. To avoid error propagation, joint learning methods have been proposed to recognize named entities and their relations simultaneously in recent years, and have shown better performance than pipeline methods. Among them, joint deep learning methods, such as Miwa and Bansal’s method,⁴ are state-of-the-art methods up-to-date. Inspired by Miwa and Bansal’s method, we propose a novel joint deep learning method for clinical entity or attribute recognition and entity-attribute relation extraction. The method considers relation constraints between clinical entities and attributes, and introduces a combination coefficient to leverage the 2 subtasks. Experimental results on an existing English clinical corpus from SemEval-2015 Task 14 and a newly developed Chinese corpus show that the proposed method outperforms other existing state-of-the-art methods on both corpora, indicating the joint deep learning approach is promising in clinical NER and relation extraction tasks across languages.

RELATED WORK

Clinical entity or attribute recognition

Early systems for clinical entity recognition are often rule-based, such as MedLEE.⁵ In the past decade, machine learning methods, such as CRF⁶ and structural support vector machines,⁷ have been proposed for clinical NER and shown better performance on several annotated clinical corpora.^1,8,9 Clinical entity recognition is solved as a sequence labeling problem by these machine learning methods. More recently, deep learning methods have been widely used for clinical entity recognition and achieve better performance with minimum feature engineering. For example, Jagannatha and Yu¹⁰ applied long short-term memory (LSTM) and gated recurrent unit to recognize clinical entities such as medications, diseases and the associated attributes, and obtained better performance than CRF. Liu et al³ presented a clinical entity recognition system based on bidirectional long short-term memory LSTM-CRF (Bi-LSTM-CRF), which is competitive with other state-of-the-art systems on the corpora of the 2010, 2012 and 2014 i2b2 NLP challenges.

Clinical relation extraction

Relation extraction in the clinical domain is usually recognized as a classification problem that needs to determine whether there is a relation between a pair of clinical entities or entity-attribute and what is the type of the relation if there exists any. Machine learning methods using various features, including lexical, semantic, syntactic, and medical domain ontology features,^11,12 such as support vector machines,^10,13 have been applied to clinical relation extraction. Recently, deep learning methods, such as convolutional neural networks (CNNs)^14,15 and recurrent neural networks,¹⁶ such as LSTM,^17,18 have been proposed for clinical relation extraction and achieved some successes. For example, Luo¹⁹ applied LSTM to extract relations between clinical concepts from clinical notes, which was comparable to the state-of-the-art systems of the i2b2/VA relation classification challenge. Luo et al²⁰ further proposed segment CNNs, which identified relations between 2 concepts by simultaneously learning separate representations of text segments in a sentence: preceding, concept1, middle, concept2, and succeeding. Segment CNNs obtained state-of-the-art performance on the i2b2/VA challenge corpus.

Joint clinical entity recognition and relation extraction

There have been few studies on joint entity recognition and relation extraction in the medical domain. However, a number of joint learning methods have been proposed for entity recognition and relation extraction in the open domain in recent years. They may fall into 2 categories: joint learning methods relying on feature engineering^21–24 and joint deep learning methods. The former ones include linear programming–based methods,^22,23 the joint graphical method,²³ and the incremental joint method.²⁴ The latter ones handle entity recognition and relation extraction in a unified deep learning framework. For example, Miwa and Bansal⁴ proposed a novel end-to-end model by stacking Bi-LSTM for entity recognition and bidirectional tree-structured LSTM for relation extraction. This model was able to jointly represent both entities and relations with shared parameters and used entity information in relation extraction via entity pretraining and scheduled sampling. It relies on a good syntactic parser, which may not exist in some specific domains. Zheng et al²⁵ presented a hybrid neural network which contains an encoder-decoder Bi-LSTM (Bi-LSTM-ED) module to recognize entities and a CNN module to extract relations. The contextual information of entities obtained in BiLSTM-ED is shared with the CNN module to improve the relation classification. Zheng et al²⁶ proposed a novel tagging scheme to convert entity recognition and relation extraction into a sequence labeling task. By designing a new tag schema to present the information of entities and the relationships simultaneously, the model could directly extract triplets composed of 2 entities and their relation, not entities and relations separately. However, this tag schema has one disadvantage that it cannot handle the case that one entity has multiple relations with different entities.

MATERIALS AND METHODS

Tasks and datasets

English task and corpus

The SemEval-2015 challenge organized a task about clinical entity recognition and template slot filling (ie, Task 14), which was similar to clinical entity and attributes extraction. However, besides clinical entity recognition, the challenge focuses on normalizing attribute values instead of extracting mentions of attributes from text. Therefore, we construct a new corpus based upon the dataset of SemEval-2015 Task 14 by converting cues in the text into mentions of attributes and linking them with clinical entities as relations. In our study, the 298 clinical records in the training set of SemEval-2015 Task 14 are also used as training set, and the 133 clinical records in the development set of SemEval-2015 Task 14 are used as test set. To avoid relations span more than 1 sentence, we just simply split all records into sentences by consecutive “\n”s.

Chinese task and corpus

The Chinese entity-attribute extraction task is to extract information as shown in Figure 1, in which there are 3 clinical entities with 2 attributes in the given Chinese sentence “患者四肢皮肤有划伤, 双侧肱二、三头肌反射、膝腱反射正常” (There are scratches on patient’s arms and legs; bilateral biceps and triceps reflexes and patellar reflexes are normal.), that is, a problem (“划伤” [scratches]), 2 lab tests (“双侧肱二、三头肌反射” [bilateral biceps and triceps reflexes] and “膝腱反射” [patellar reflexes]) and 2 attributes(“四肢皮肤” [arms and legs] and “正常” [normal]), the first attribute “四肢皮肤” modifies the problem “划伤” (denoted by attrOf), and the second attribute “正常” modifies the 2 lab tests (denoted by valueOf).

Figure 1.

Example sentence of Chinese clinical entities with their attribute extraction.

Following a similar guideline as the one used in the SemEval task, we annotate a Chinese corpus of 290 manually de-identified clinical documents (241 admission notes and 49 discharge summaries) that are randomly selected from a Tier 3A hospital in China. Two Chinese annotators with medical background are recruited for the annotation after appropriate training. A small number of documents (n = 10) are annotated by both annotators to calculate the interannotation agreement, which is 0.85 for entities and 0.87 for relations measured by kappa. In this study, 192 admission notes are used as training set and the remaining 49 admission notes and 49 discharge summaries are used as test set. The statistics of the corpora used for clinical entity or attribute recognition and entity-attribute relation extraction are listed in Table 1, in which #* is the number of asterisks.

Table 1.

Statistics of the corpora used for clinical entity or attribute recognition and entity-attribute relation extraction

Corpus			Training	Test	Total
English	#Document		298	133	431
	#Sentence		2534	1714	4248
	#Entity	disorder	10 720	7693	18 413
	#Attribute	negation	1512	1054	2566
		body	2160	1382	3542
		severity	916	412	1328
		change	397	269	666
		uncertain	800	408	1208
		conditional	300	273	573
		subject	76	99	175
		generic	79	78	157
	#Relation	attrOf	8013	5360	13 373
Chinese	#Document		192	98	290
	#Sentence		6150	2804	8954
	#Entity	problem	10 758	5198	15 956
		labtest	5427	3230	8657
		procedure	993	752	1745
		medicine	503	468	971
	#Attribute	value	4475	2459	6934
		negation	4773	1744	6517
		body	2238	757	2995
		temporal	803	362	1165
		severity	409	175	584
		change	150	83	233
		uncertain	136	116	252
	#Relation	attrOf	9608	3841	13 449
		valueOf	4347	2426	6773

System description

Inspired by Miwa and Bansal’s method,⁴ we propose a novel joint deep learning method to extract clinical entities with attributes from clinical text, denoted by JBCB (joint learning method using Bi-LSTM-CRF for entity or attribute recognition and Bi-LSTM for entity-attribute relation extraction). It consists of 4 components: (1) sentence encoding, (2) entity or attribute recognition, (3) entity-attribute relation (ie, <attribute, modify, entity>) extraction, and (4) joint learning. Figure 2 gives the architecture of JBCB through an example, where a negation “无 (ie, “no”)” is an attribute of (denoted by attrOf) a problem “浊音 (ie, “dullness”)”. The following sections present each component in detail.

Figure 2.

Architecture of the proposed joint deep learning method. LSTM: long short-term memory.

Sentence encoding

Following previous studies,²³ we use Bi-LSTM^27,28 to encode an input sentence, including the following 3 layers. First, an input layer converts an input sentence into a sequence of words’ embeddings by dictionary lookup. Each word’s embeddings includes 2 parts: word embedding and part-of-speech (POS) tag embedding. Second, the Bi-LSTM layer takes the word embedding sequence of the input sentence as input and returns another sequence that represents context information of each word of interest at every position from 2 parallel directions respectively. Third, the concatenation layer concatenates the outputs of forward and backward LSTMs at every position. For example, given an input sentence s=w₁w₂…w_n with each word w_t (1⩽t⩽n) represented as x_t, which is the concatenation of the word embedding and POS tag embedding (ie, x_t=[emb(w_t); emb(p_t)], where “;” is concatenation operation]), Bi-LSTM produces 2 sequences from forward and backward directions h_f=h_f1h_f2… h_fn and h_b=h_b1h_b2… h_bn. The 2 output sequences are then concatenated into h=h₁h₂… h_n, where h_t=[h_ft; h_bt]. In the case of w_t, an LSTM unit, composed of 3 multiplicative gates: an input gate, a forget gate and an output gate, modulating how much the current, the previous, and output information should be included in the current time step t (as shown in Figure 3), takes x_t, the previous hidden state vector h_t-1 and the previous cell vector c_t-1 as input and computes the current hidden state vector h_t and the current cell vector c_t based on the following formulas:

$$i_t=\sigma (W^{\left(i\right)}{x}_t+U^{\left(i\right)}{h}_{t-1}+b^{\left(i\right)})$$ (1)

$$f_t=\sigma (W^{\left(f\right)}{x}_t+U^{\left(f\right)}{h}_{t-1}+b^{\left(f\right)})$$ (2)

$$o_t=\sigma (W^{\left(o\right)}{x}_t+U^{\left(o\right)}{h}_{t-1}+b^{\left(o\right)})$$ (3)

$$u_t=\mathit{tanh}(W^{\left(u\right)}{x}_t+U^{\left(u\right)}{h}_{t-1}+b^{\left(u\right)})$$ (4)

$$c_t=i_t\odot u_t+f_t\odot c_{t-1}$$ (5)

$$h_t=o_t\odot \hspace{0.17em}\mathit{\text{tan}}\hspace{0.17em}h\left(c_t\right),$$ (6)

where$\sigma \left(·\right)$, $\tanh \left(·\right),\mathrm{}$ and $\odot$ denote the logistic function, hyperbolic tangent function, and element-wise multiplication, respectively; W and U are weight matrices; and b are bias vectors.

Figure 3.

Structure of a long short-term memory unit.

Entity or attribute recognition

Clinical entity or attribute recognition is recognized as a sequence labeling problem, and the “BILOU” tag schema is used to represent the boundaries of entities, where B/I/L denote the beginning, inside, and last words of an entity, respectively; U denotes a single-word entity; and O denotes that a word does not belong to any entity. For example, the sentence “患者四肢皮肤有划伤, 双侧肱二、三头肌反射、膝腱反射正常” in Figure 1 is represented as “患/O 者/O 四/B-body 肢/I-body 皮/I-body 肤/L-body 有/O 划/B-problem 伤/L-problem , /O 双/B-labtest 侧/I-labtest 肱/I-labtest 二/I-labtest 、/I-labtest 三/I-labtest 头/I-labtest 肌/I-labtest 反/I-labtest 射/L-labtest 、/O 膝/B-labtest 腱/I-labtest 反/I-labtest 射/L-labtest 正/B-value 常/L-value”.

A 2-layer neural network, as shown on the top left of Figure 2, is applied for clinical entity or attribute recognition. It takes hidden state sequence h=h₁h₂…h_n as input, and predicts the most possible tag sequence z=z₁z₂…z_n as follows:

$$z=\underset{y}{\mathrm{argmax}}\frac{\hspace{0.17em}\text{exp}\hspace{0.17em}\left(\mathit{score}\right(h,y\left)\right)}{{\sum }_{y^{\mathrm{\text{'}}}}\hspace{0.17em}\text{exp}\hspace{0.17em}\left(\mathit{score}\right(h,y^{\mathrm{\text{'}}}\left)\right)},$$ (7)

where $\mathit{score}\left(h,y\right)={\sum }_{t=1}^n{E}_{t,{\mathrm{}y}_t}+{\sum }_{t=0}^{n-1}{T}_{{\mathrm{}y}_t{y}_{t+1}}$, $E_{t,{\mathrm{}y}_t}=w_{y_t}^{\mathrm{T}}{h}_t$ is the score of emiting ${\mathrm{}y}_t$ at the t-th time, and $T_{{\mathrm{}y}_t{y}_{t+1}}$ is the score of transitioning from ${\mathrm{}y}_t$ to ${\mathrm{}y}_{t+1}$. To keep consistent with Miwa and Bansal’s method, we add a hidden layer before the CRF layer.

Entity-attribute relation extraction

A 3-layer neural networks are proposed for entity-attribute relation classification, called Bi-LSTM (the right part in Figure 2). It first takes a pair of a target entity and a target attribute with context between them as input at the first layer, and obtains the corresponding representation using Bi-LSTM mentioned in section Sentence Encoding Each input word w_t is represented by [h_t; l_t] (h_t is the t-th item of the sentence encoding mentioned in section Sentence Encoding, and l_t is the predicted tag embedding of w_t), the corresponding representation of context fragment between the target entity and attribute is h_fr=[h_fr→; h_fr←] (h_fr→ and h_fr← are the last outputs of LSTM from forward and backward directions), the target entity and attribute are represented by the average of hidden state vectors of words they contain, that is, $h_{r1}=\sum _{j\in {\mathit{EA}}_1}{h}_j$ and $h_{r2}=\sum _{j\in {\mathit{EA}}_2}{h}_j$, where EA₁ and EA₂ represent indices of the words in the first and second entity or attribute. In this way, the target entity and attribute pair with context is represented by h_r = [h_fr; h_r1; h_r2]. At the second layer, h_r is fed into a transform function to obtain a hidden state $h^{\mathit{relation}}$, and then the probability of a possible relation between the target entity and attribute $p_r^i$(1⩽i⩽N_r) is calculated by the softmax function at the third layer as follows:

$$h_r^{\mathit{relation}}=\mathit{tanh}(W^{\mathit{rh}}{h}_r+b^{\mathit{rh}})$$ (8)

$$y_r=W_r{h}_r^{\mathit{relation}}+b_r$$ (9)

$$p_r^i=\frac{\mathit{exp}\left(y_r^i\right)}{{\sum }_{j=1}^{N_r}\mathit{exp}\left(y_r^j\right)},$$ (10)

where W are the weight matrices, b are bias vectors, and N_r is the total number of entity-attribute relation types including null that indicates no relation.

In a specific domain, there are constraints to limit relations between entities and attributes. For example, in the clinical domain, a “value” can be an attribute of a “labtest,” but cannot be an attribute of a “problem.” In this study, we further utilize clinical entity-attribute relation constraints to filter <attribute, modify, entity> candidates that could not appear. We list all constraints in Supplementary Appendix A.

Joint learning for entity with attribute extraction

In this study, we adopt cross-entropy as loss function. The loss functions for entity or attribute recognition L_e and entity-attribute relation extraction L_r can be written as:

$$L_e=-\sum _{i=1}^{\left|D_s\right|}\sum _{t=1}^{\left|S_i\right|}{p}_t^i\mathrm{l}\mathrm{og}{p}_t^{i^{\mathrm{\text{'}}}}$$ (11)

$$L_r=-\sum _{i=1}^{\left|D_r\right|}{p}_r^i\mathrm{l}\mathrm{og}{p}_r^{i^{\mathrm{\text{'}}}},$$ (12)

where $\left|D_s\right|$ is the number of sentences in the training set, $\left|S_i\right|$ is the length of sentence $S_i$, $p_t^i$ the probability of the gold standard label of w_t in the i-th sentence,$\mathrm{}{p}_t^{i^{\text{'}}}$is the probability of the predicted label of w_t, $\left|D_r\right|$ is the number of entity-attribute relations in the training set, $p_r^i$ is the gold standard label of the i-th entity-attribute relation in the training set, and $p_r^{i^{\text{'}}}$is the probability of the predicted label for the i-th entity-attribute relation. For joint learning, L_e and L_r are combined together as the final loss function:

$$L=\mathrm{\alpha }{L}_e+(1-\alpha )L_{r,}$$ (13)

where $\alpha$ is the combination coefficient. The larger$\alpha$ is, the greater influence entity or attribute recognition task has; otherwise, the greater influence entity-attribute extraction task has.

Experiments

We compare the proposed method with the following 2 pipeline baseline methods, BCR and BCB, and a joint method JBB:

1. BCR uses Bi-LSTM-CRF for clinical entity or attribute recognition and a simple rule-based method for entity-attribute relation extraction. The rule-based method combines each attribute with the nearest entity.

2. BCB uses Bi-LSTM-CRF for clinical entity or attribute recognition and Bi-LSTM for entity-attribute relation extraction.

3. JBB is a variant of Miwa and Bansal’s method that uses Bi-LSTM instead of the bidirectional tree-structured LSTM for relation extraction.

In addition, we investigate the effects of combination coefficient $\alpha$ and clinical entity-attribute relation constraints on joint learning methods.

Preprocessing

The Stanford CoreNLP toolkit 3.8²⁹ is utilized for preprocessing, including tokenization, POS tagging, etc.

Evaluation

The performance of methods for clinical entity or attribute recognition and entity-attribute relation extraction are measured by standard micro-averaged precision (P), recall (R), and F1 score (F1).

Parameters

The hyperparameters of all deep learning models are set the same as follows except combination coefficient $\alpha$, which is optimized by searching from 0.1 to 0.9 with an increment of 0.1 (the optimal value of $\alpha$ is 0.4 on the English corpus and 0.6 on the Chinese corpus):

• Embedding dimensions: dim (emb ($w_t$)) =50, dim (emb ($p_t$)) = dim (emb ($d_t$)) = dim (emb ($l_t$)) =25

• – Learning rate: 1e-6

• – Regularization parameter: 1e-6

• – Dropout probability: 0.3

• – Bi-LSTM(encoding): dim ($h_{\mathit{ft}}$)= dim (${\mathrm{h}}_{\mathit{bt}}$)=100, dim ($h_t^{\mathit{entity}}$)=100

• – Bi-LSTM(entity-attribute relation extraction): dim (h_fr→)= dim (h_fr←) =100, dim ($h_r^{\mathit{relation}}$)=100

Chinese word embeddings are learned from all data of the 2017 China conference on knowledge graph and semantic computing (CCKS) clinical named entity recognition (CNER) challenge,³⁰ by word2vec,³¹ English word embeddings are learned from the MIMIC-III (Medical Intervention Mart for Intensive Care III) clinical database³² by word2vec, and the other embeddings are randomly initialized. We first randomly divide the training set into 2 parts (80% for training and 20% for validation to optimize parameters), then train models on the whole training set using the optimized parameters.

RESULTS

The performance of different pipeline and joint methods is shown in Table 2, where the highest P, R, and F1 on the 2 corpora are highlighted in boldface. Overall, the joint methods show better performance than the pipeline methods on the 2 corpora. For example, the best joint method (ie, JBCB) achieves the highest F1 of 89.32% on clinical entity or attribute recognition and the highest F1 of 88.13% on entity-attribute relation extraction on the Chinese corpus, outperforming the best pipeline method (ie, BCB) by 0.48% and 1.03%, respectively. Similar to previous works, Bi-LSTM-CRF outperforms Bi-LSTM for NER in our implementations of both pipeline and joint learning methods.³³

Table 2.

Comparison of different pipeline and joint methods

Corpus	Method	Entity or attribute recognition			Entity-attribute relation extraction
		P	R	F1	P	R	F1
English	BCR	0.7708a	0.7171	0.7430	0.3028	0.5320a	0.3859
	BCB	0.7708a	0.7171	0.7430	0.6342	0.3989	0.4897
	JBB	0.7280	0.6901	0.7085	0.6678a	0.3993	0.4998
	JBCB	0.7504	0.7389a	0.7446a	0.6647	0.4034	0.5021a
Chinese	BCR	0.8924	0.8843a	0.8884	0.8125	0.8681	0.8394
	BCB	0.8924	0.8843a	0.8884	0.8537	0.8891a	0.8710
	JBB	0.8716	0.8693	0.8705	0.8919a	0.8652	0.8783
	JBCB	0.9027a	0.8839	0.8932a	0.8836	0.8790	0.8813a

BCB: a pipeline method using Bi-LSTM-CRF for clinical entity or attribute recognition and Bi-LSTM for entity-attribute relation extraction; BCR: a pipeline method using Bi-LSTM-CRF for clinical entity or attribute recognition and a simple rule-based method for entity-attribute relation extraction; F1: F1-score; JBB: joint learning method using Bi-LSTM for entity or attribute recognition and Bi-LSTM for entity-attribute relation extraction; JBCB: joint learning method using Bi-LSTM-CRF for entity or attribute recognition and Bi-LSTM for entity-attribute relation extraction; P: precision; R: recall.

^a the best performance.

The detailed results of JBCB for each entity or attribute type and relation type are shown in Table 3. On the Chinese corpus, among the 4 types of entities, JBCB performs best on lab test, with an F1 of 99.83%, and worst on procedure, with an F1 of 77.46%. For attribute recognition, it performs best on negation, with an F1 of 97.57%, and worst on severity, with an F1 of 44.16%. On the English corpus, JBCB achieves an F1 of 79.95% on disorder, and performs best on severity, with an F1 of 77.43%, and worst on generic, with an F1 of 21.24%, among the 8 attribute types.

Table 3.

Detailed results of JBCB for each entity or attribute type and entity-attribute relation type

Corpus	Subtask	Type	P	R	F1
English	Entity or attribute recognition	disorder	0.7767	0.8237	0.7995
		negation	0.7543	0.7894	0.7714
		body	0.6030	0.5640	0.5828
		severity	0.7564	0.7932	0.7743
		change	0.5382	0.6543	0.5906
		uncertain	0.4428	0.4363	0.4395
		conditional	0.5890	0.5092	0.5462
		subject	0.7394	0.7354	0.7374
		generic	0.3429	0.1538	0.2124
	Entity-attribute relation extraction	attrOf	0.6647	0.4034	0.5021
Chinese	Entity or attribute recognition	problem	0.8676	0.8683	0.8680
		labtest	0.9438	0.9328	0.9383
		procedure	0.8490	0.7174	0.7746
		medicine	0.8333	0.6545	0.7777
		value	0.9350	0.9240	0.9294
		negation	0.9732	0.9782	0.9757
		body	0.8769	0.9036	0.8900
		temporal	0.9105	0.8149	0.8601
		severity	0.5113	0.3886	0.4416
		change	0.6615	0.5181	0.5811
		uncertain	0.6891	0.7069	0.6979
	Entity-attribute relation extraction	valueOf	0.9128	0.9151	0.9140
		attrOf	0.8696	0.8562	0.8605

F1: F1-score; JBCB: joint learning method using Bi-LSTM-CRF for entity or attribute recognition and Bi-LSTM for entity-attribute relation extraction; P: precision; R: recall.

In addition, we investigate the effects of combination coefficient $\mathrm{\alpha }$ and clinical entity-attribute relation constraints on the 2 joint deep learning methods by removing each one or both on the Chinese corpus. From Table 4, we can see that when combination coefficient or clinical entity-attribute relation constraints is removed, F1 decreases more or less on clinical entity or attribute recognition and entity-attribute relation extraction, clinical entity or attribute recognition is more sensitive to the combination coefficient, and clinical entity-attribute relation extraction is more sensitive to the entity-attribute relation constraints. For example, when the combination coefficient is removed (ie, $\mathrm{\alpha }$ is set to 0.5), the F1 of JBB on NER decreases 0.4%, while that on relation extraction decreases no more than 0.1%. When clinical entity-attribute relation constraints are removed, the F1 of JBB on NER decreases 0.15%, while that on relation extraction decreases 0.47%. Even so, the joint methods are still superior to the corresponding pipeline methods shown in Table 2.

Table 4.

Effects of combination coefficient $\mathrm{\alpha }$ and clinical entity-attribute relation constraints on joint deep learning methods on the Chinese corpus

Settings	Entity or attribute recognition			Relation classification
	P	R	F1	P	R	F1
JBB	0.8716	0.8693	0.8705	0.8919	0.8652	0.8783
— Combination coefficient	0.8664	0.8667	0.8665	0.8901	0.8655	0.8776
— Relation constraints	0.8685	0.8696	0.8690	0.8772	0.8700	0.8736
— ALL	0.8676	0.8687	0.8682	0.8749	0.8703	0.8726
JBCB	0.9027	0.8839	0.8932	0.8836	0.8790	0.8813
— Combination coefficient	0.8967	0.8862	0.8914	0.8797	0.8780	0.8788
— Relation constraints	0.8871	0.8947	0.8918	0.8818	0.8708	0.8763
— ALL	0.8901	0.8636	0.8918	0.8823	0.8686	0.8754

F1: F1-score; JBB: joint learning method using Bi-LSTM for entity or attribute recognition and Bi-LSTM for entity-attribute relation extraction; JBCB: joint learning method using Bi-LSTM-CRF for entity or attribute recognition and Bi-LSTM for entity-attribute relation extraction; P: precision; R: recall.

DISCUSSION

Similar to previous works,^20,21 the joint deep learning methods outperformed pipeline methods for clinical entity with attributes extraction in this study. The advantage of the joint deep learning lies in that some errors in one subtask may be fixed by another subtask. Two examples of errors in BCB fixed by JBCB are given in Supplementary Appendix B.

The reason why we replace the bidirectional tree-structured LSTM of Miwa and Bansal’s method for relation extraction by Bi-LSTM is that there is no public parser performing well enough for clinical text. Using Miwa and Bansal’s method, we obtain an F1 of 86.96% on NER and an F1 of 86.20% on relation extraction on the Chinese corpus, lower than JBB by 0.09% and 1.63%. Please see Supplementary Appendix C for the detailed information. Besides, CNN-based methods are not investigated in this study for relation extraction, because it is proved by previous works that LSTM and CNNs can obtain comparable performance in relation extraction tasks.³⁴ As CRF takes full advantages of dependencies across output tags for NER, the methods adopting Bi-LSTM-CRF for NER outperform that adopting Bi-LSTM as shown in Table 2.

The reason why the clinical entity-attribute relation constraints can improve the joint deep learning models is that they filter out a large number of negative relation samples (ie, an entity-attribute pair of interest without relation). For example, when the constraints are not considered, 17 of 20 samples for 3 entities and 2 attributes in the sentence in Figure 1 are negative. All the negative samples can be filtered out by the constraints. When the constraints are added into the joint deep learning methods, more than 90% negative samples in the training set are filtered out.

The reason why the difference in performance on the 2 corpora is large is likely that the distance (number of words in English and characters in Chinese) between the entity and attribute in a relation in English clinical text is much longer than in Chinese clinical text. The distance between an entity and one of its attributes in English clinical text is 4.15, while 2.19 in Chinese clinical text.

To assess whether the F1 scores between methods are statistically significantly different, we conduct the following statistical test. From the test set, we randomly select 1000 sentences with replacement for 100 times, thus to generate 100 bootstrapping datasets. We then calculate F1 scores for any 2 different methods on each dataset, and apply the Wilcoxon signed rank test,³⁵ a nonparametric test for paired samples, to determine whether the F1 scores between 2 methods are statistically significant (P < .05). For example, the statistical testing between JBCB and BCB shows that the difference between F1 scores from these 2 methods on entity-attribute relation extraction is statistically significant, with a P value of 3.90 × 10⁻¹⁸.

Although the proposed joint deep learning method (eg, JBCB) performs better than other related methods in comparison, it also outputs a number of errors. We look into the errors on the Chinese corpus and find the following. First, for NER, misrecognition of entities or attributes accounts for over 36% of all errors, while around 34% entities or attributes are correctly recognized which are labeled with incorrect semantic types. Moreover, around 30% entities or attributes are partially recognized which are labeled with correct semantic types. Second, in the case of relation extraction, more than 50% relations are not extracted because of misrecognized entities or attributes; about 43% relations are wrongly extracted because of incorrect entities or attributes; no more than 7% relations are not extracted even though the corresponding entities/attributes are correctly recognized. The main reasons for entity or attribute recognition errors in the system are ambiguities from numbers (the example of Error Type I), types (the example of Error Type II), and language expressions (the example of Error Type III). Most missed relations are related to entities or attributes of small number and may suffer from data imbalance. Therefore, most entity-attribute relation extraction errors are caused by entity or attribute recognition errors. Error samples are listed in Supplementary Appendix D.

In the future work, there may be 2 directions for further improvement: (1) integrating neural language models such as BERT (Bidirectional Encoder Representations from Transformers)³⁶ into our method, which leverages the attention mechanism and obtained state-of-the-art performance on a set of benchmark NLP tasks in the open domain; and (2) considering all entity-attribute pairs in a sentence at the same time for clinical entity-attribute relation extraction.

CONCLUSION

In this study, we propose a novel joint deep learning method to extract clinical entities with attributes. It integrates Bi-LSTM-CRF and Bi-LSTM together in a unified framework and introduces constraints to limit clinical entity-attribute relations and a combination coefficient to leverage entity or attribute recognition and entity-attribute relation extraction. Experiments conducted on 2 corpora prove its effectiveness. Moreover, the proposed method is also suitable for similar tasks in other domains.

FUNDING

We gratefully acknowledge Beijing Baidu Netcom Science Technology Co., Ltd.

ACKNOWLEDGMENTS

This work is supported in part by grants: National Natural Science Foundations of China grant nos. U1813215, 61876052, and 61573118 (to BT); Special Foundation for Technology Research Program of Guangdong Province grant no. 2015B010131010 (to XW); Strategic Emerging Industry Development Special Funds of Shenzhen grant nos. JCYJ20170307150528934 (to XW) and JCYJ20180306172232154 (to BT); and Innovation Fund of Harbin Institute of Technology grant HIT.NSRIF.2017052 (to BT).

AUTHOR CONTRIBUTIONS

The work presented here was carried out in collaboration among all authors. XS, YY, YX, and BT designed the study, developed the methods, and conducted experiments on the Chinese dataset; XS, ZJ, and YZ contributed to the experiments on the English dataset; BT, XW, QC, and HX provided guidance and reviewed the manuscript critically. All authors have approved of the final manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

HX and The University of Texas Health Science Center at Houston have research-related financial interests in Melax Technologies, Inc.