Extraction of Temporal Information from Clinical Narratives

Abstract

The existence of massive quantity of clinical text in electronic medical records (EMRs) has created significant demand for clinical text processing and information extraction in the field of health care and medical research. Detailed clinical observations of patients are typically recorded chronologically. Temporal information in such clinical texts consist of three elements: temporal expressions, temporal events, and temporal relations. Due to the implicit expression of temporal information, lack of writing quality, and domain-specific nature in the clinical text, extraction of temporal information is much more complex than for newswire texts. In spite of these difficulties, to extract temporal information using the annotated corpora, few research works reported rule-based, machine-learning, and hybrid methods. On the other hand, creating the annotated corpora is expensive, time-consuming, and demands significant human effort; the processing quality is inevitably affected by the small size of corpora. Motivated by this issue, in this research work, we present a novel method to effectively extract the temporal information from EMR clinical texts. The essential idea of this method is first to build a feature set appropriately for clinical expressions, followed by the development of a semi-supervised framework for temporal event extraction, and finally detection of temporal relations among events with a newly formulated hypothesis. Comparative experimental evaluation on the I2B2 data set has clearly shown improved performance of the proposed methods. Specifically, temporal event and relation extraction is possible with an F-measure 89.98 and 67.1% respectively.

Introduction

In the last several decades of computational linguistics research, automatic identification and information extraction in natural language text is an active research area [22, 29, 45]. With the introduction of TIMEX extraction in MUC-6, TIMEML with its annotation guidelines and corpora for the processing of natural language text was developed [13, 28]. The developed corpora facilitate the researchers to identify and extract the events and TIMEX expressions and enable them to process the time orientation in a text documents. After that, the three TempEval challenges have marked significant advances in the field of processing the natural language text and temporal information extraction [38–40].

The existence of clinical text in the recent electronic medical records (EMRs) has received a great level of recognition among researchers due to the need for medical professionals to access patient medical details and history from a temporal perspective. This is to analyze the progressions of disease and effect of treatment procedure. Although the EMRs contain significant clinical information, the exploitation of EMRs clinical text still remains in its infancy due to technical challenges and the nature of such text [11, 12]. Figure 1 shows an example of a clinical narrative found in EMRs with highlighted temporal information.

Fig. 1.

Clinical narrative from EMR, an example

Most importantly, clinical text is usually written in natural language with free format that contains plenty of clinical terminology and domain-specific information. It is worth noting that clinical text from EMRs is distinguished from non-clinical text by its domain-specific nature, abbreviations and short form, which significantly increases the difficulty of clinical text processing [4, 23]. To access the valuable knowledge from encoded longitudinal clinical narratives is intrinsically challenged by issues of text representation [10, 23]. Temporal information representation emerges as the most important form since all the clinical events and medical information are noted with time stamps and indirectly follow a temporal order in the nurse narratives, doctor daily notes, and discharge summaries [32]. Thus, the loss of chronological order of clinical events could have negative impact on disease diagnosis, and lead to serious medical errors as well as affect the patient treatment procedures and results [2].

Therefore, many research groups have been working on temporal reasoning and temporal information extraction in clinical narratives [46, 48, 50]. Wang et al. provides the state-of-art of developed clinical applications using EMRs [44]. This survey provided us a more concrete understanding of the provided potential solutions and the utilization of clinical text processing in the clinical domain. One potential application arising from processing clinical narratives in EMRs are clinical decision support systems, which can be applicable in areas include clinical decision support [5, 42] prognosis, disease monitoring, adverse drug effects [20], and drug development. On the other hand, Trivedi et al. have developed a visualization tool with various components that helps to analyze the clinical records for domain experts and end users [37]. This tool allows to send feedback if the recognition of the analyzed data is poor. Apart from this, Yang et al. developed the clinical information extraction and retrieval system which uses electronic health records (EHRs) to generate queries [43].

“What is the patient health status after the treatment?”

The above questions can only be answered after understanding the temporal orientation of clinical text related to the patient. Therefore, being able to identify and extract temporal information (expressions, events, and relations) is a very crucial phase and it becomes the priority task for structured representation of clinical narratives from EMRs.

To chronologically arrange all clinical events as they occurred, all the clinical events and expressions should be based on the raw clinical text. Moreover, the expressions in the clinical text are frequently written by utilizing the relative time. Furthermore, without specifying or referring the order and time, the clinical events and relations are frequently mentioned. For instance, consider the following sentences taken from clinical text are as follows:

“The patient was admitted to the hospital for surgery on 2015-11-22. Her renal function also remained stable in the perioperative period. She was discharged home in stable condition on postoperative day six.”

“The patient met the orthopaedist. He diagnosed that the patient mostly had compartment syndrome.”

The above examples show that the temporal events, expressions, and relations are mentioned with implicit durations, relative time, and order. To parse this temporal medical concepts and arrange the information in their order of occurrence, significant medical domain knowledge is required.

Even though humans can interpret the temporal information in the clinical text such as events and their relations, in machine-learning and clinical natural language processing, it is a non-trivial task [34]. Despite these difficulties, to extract temporal information, some of the reported research works have established different methods based on the existing annotated corpora [31, 33, 35]. However, all of these works were done based on the existing small number of annotated temporal corpora such as informatics for integrating biology and the bedside (I2B2) temporal relations dataset,1 semantic evaluation (SemEval) challenge 20152 and SemEval challenge 2016 clinical datasets,3 whereas abundant amount of unannotated clinical narratives are available for research such as Multiparameter Intelligent Monitoring in Intensive Care (MIMIC II), MIMIC III clinical database,4 and other resources.

Although many EMRs are freely available for research, only a very little amount of clinical text has been annotated with temporal information. To build the annotated corpus manually, it demands significant domain knowledge, considerable manpower, and time. To extract the temporal expressions, we developed a hybrid model with novel features by using HeidelTime system [26]. To boost the predictive model performance in information extraction, semi-supervised learning (SSL) aims to exploit the massive amount of unannotated data [49]. By considering this advantage into account, to automatically extract the temporal events, we proposed a novel two-stage framework by exploiting the massive unannotated data, with minimal manual intervention. This novel approach helps to reduce the annotation time, cost, and manual effort. We developed this framework and reported preliminary experiments results in [27]. In this work, to extract temporal relations, we propose a Naïve Bayesian classifier approach for classifying relationships after generating candidate pairs by proposing novel hypothesis and adoption of dependency parsing method. We will discuss about candidate pairs generation and classification in later section. Thus, our contributions in this work are as follows:

• We establish a novel semi-supervised framework for extracting temporal events in the clinical narrative text, with increasing the number of records and various window size. This approach is based on the novel idea of gradually extending the training corpus by adding annotated data obtained from unannotated clinical text.

• A new assumption on generating and identifying the potential candidate pairs from list of temporal events or expressions is formulated, that can appropriately relate events/expressions in clinical text. The effective candidate pairs generation helps to improve the relation classification performance.

The remainder of this paper is structured as follows. The state-of-the-art temporal information extraction across domains and various representation of clinical text data is briefly summarized in Section 2. In Section 3, we discuss the proposed methods of temporal information extraction, which includes temporal expressions, temporal events extraction, and temporal relation classification. In Section 4, we discuss on empirical evaluation and summarize our observations. Finally, in Section 5, we provide the discussion of the findings followed by conclusions and in Section 6.

Background

In recent years, along with the development and wide implementation of EMRs, availability of abundant clinical text has obtained a wide attention among researchers for temporal information extraction. Many research works has been reported on various topics of temporal reasoning and representation in the clinical data [19, 46, 48, 50]. To exploit the extracted temporal information in medical research and care, few studies have been initiated in natural language processing (NLP) on clinical text (applications by utilizing temporal information) [5, 20, 42]. Clinical decision support systems [5] and detection of adverse drug events [20] have become possible due to the availability of temporal information.

To accelerate the extraction of temporal information, the I2B2 challenge5 provides an annotated corpus on shared temporal relation tasks [32, 33, 35]. Followed by I2B2, SemEval challenge 2015 [6], and SemEval challenge 2016 [7] made efforts to generate an annotated corpus to expedite the researches of temporal information extraction. Bethard et al. [6] created this annotation corpus on clinical text based on domain-specific THYME-TimeML guidelines.6 This THYME-TimeML guidelines are adopted from ISO-TimeML guidelines [29]. Semantic information are added to the initial document, which is called as annotation [14, 30]. This is useful for various kinds of information extraction.

To extract temporal expressions, the well-established HeidelTime has proven very powerful [35]. HeidelTime is a Java-based multilingual, cross-domain temporal expressions tagging system for newswire text data. The system was developed mainly based on TimeML corpus and uses the general text data, but it performs very poorly with clinic notes as the temporal expressions in clinical narratives are very different from newswire text [24]. The shortcomings of HeidelTime inspired us to extend and modify it appropriately for temporal expressions in clinical text, as described later (Section 3.2).

Tang et al. developed a hybrid method by combining the conditional random fields (CRFs) and support vector machine (SVM) methods [35], and reported the extraction of the temporal events and event features by using this method. This hybrid method is an extension of [25] for the named entity recognition (NER) tagger system and Tang et al. used this method to extract the temporal events and the partial event features [35]. Various combinations of features (Lexical and UMLS (unified medical language system)) were utilized for event extraction in the NER tagger system. A pipeline approach was developed by Jindal et al. to extract events and expressions from clinical narratives [16]. They extracted the time spans in the first stage, and afterwards, the types of events were parsed. An integer quadratic program (IQP) was implemented for the temporal event extraction and HeidelTime was adopted to extract the temporal expressions [16]. However, the pipeline approach has some limitations. Firstly, performance decreases in event extraction was caused by drop in accuracy of finding the attributes. Secondly, in comparison to previous systems, the performance of the feature extraction also degraded [33, 35]. Lastly, developing hypothesis and rules are complex and takes more time.

In case of temporal relation classification, various candidate generation and relation classification approaches have been attempted in existing studies. Chambers et al. [9] used Naive Bayes classifier to classify the temporal relation between the events from the TimeBank corpus. Chang et al. developed a hybrid system, which consists of rule-based and maximum entropy (ME)-based approaches to generate the candidate pairs [47]. Finally, they have proposed an algorithm that integrates the candidate pairs from two approaches. Even though they have developed the generation process of the candidate pairs by using the ME method, the processing steps are not very transparent and developing rules for each category by analyzing the clinical text is ambiguous and time-taking. In their most recent work, pathology notes of patients with colon cancer and brain cancer from the SemEval 2017 challenge are studied [21].

In contrast to the previous work, Tang et al. addressed the candidate generation problem with a formulated hypothesis based on the dependency parsing approach [35]. In that study, they have used different strategies for each category of temporal relation (relation between events and section times, intra-sentence, and inter-sentence). The hypothesis and approach used to generate the candidate pairs for inter-sentence and intra-sentence temporal relation classifier, respectively, fail to capture some potential candidate pairs. Though the researchers have developed various candidate pairs generation and classification methods, it is still an unresolved problem. In this manuscript, we try to change this by formulating a new approach to candidate pair generation, subsequently improving temporal relation and classification performance.

Proposed Method

We approached the problem of grounding temporal information from clinical text with three different tasks: temporal expression extraction (Section 3.1), temporal event extraction (Section 3.2), and temporal relation extraction (Section 3.3). Figure 2 shows the graphical representation of elements in temporal information extraction from clinical narratives.

Fig. 2.

Elements of temporal information extraction

Temporal Expressions Extraction

Generally, temporal expressions are defined with sequences of phrases and words, which denotes notion of time or time span such as frequency and duration or the points of time such as date and time. The clinical narrative includes the special temporal expression types such as quantifiers and prepostexp [24]. The different types of temporal expressions found in clinical text are shown in Table 1. According to the I2B2 annotation guidelines, annotation of temporal expressions (TIMEX3) and determining the span on timeline for temporal expressions is straightforward. The annotated dataset have marked the entire temporal expression with TIMEX3 tags. The surrounding prepositions phrases of temporal expressions are not included usually in the TIMEX3 span. However, automatically identifying the explicit and implicit time expressions in clinical narratives without manual intervention is a non-trivial task. To accomplish our objective, we use a hybrid approach that combines conditional random fields and adopts the state-of-the-art from the newswire domain (HeidelTime system) for temporal expressions extraction from clinical narratives.

Table 1.

Temporal expressions types in clinical text

Temporal expression type	Example
Date	Evening of 2017-11-10, yesterday, Tuesday
Time	2:00 PM, before 10 mins
Frequency	Twice per month, weekly
Duration	More than 30 mins, past three days
Prepostexp	Postoperatively, post-surgery
Quantifier	Thrice, two times

Based on a deep survey of the literature, we have concluded that the HeidelTime system is one of the top performing systems of temporal expressions from non-clinical text. Therefore, we apply HeidelTime on clinical narratives in order to develop and train our model. In this process, HeidelTime7 identifies the three attributes ID, TYPE, and VALUE for the temporal expressions. The HeidelTime system tags the frequency expressions as SET. In our proposed method, we replaced this SET type expressions with FREQ expression tags before generating the features. As previously mentioned, the HeidelTime system is not effective at extracting temporal expressions from clinical text. An example of temporal expressions encountered in clinical text is given in the following lines:

“The patient is admitted before 11-21 with type 1 diabetes mellitus, who is four months post cadaveric kidney transplantation and now has good graft function. She presents for cadaveric pancreas transplantation. She was discharged home in stable condition on postoperative day six.”

In the sentences above, both implicit temporal expressions (such as early on 09-01, four months, twice a day) and one prepostexp expression (postoperative day six) are found. For the extraction of the prepostexp expression and partial date expression such as before 11-21, HeidelTime can not be used. Thus, a new framework with a novel feature set and an adapted version of HeidelTime features is used for the extraction the temporal expressions in clinical text.

Our proposed framework for temporal expression extraction in clinical narratives is shown in Fig. 3 . In our proposed framework, we use the obtained time expressions tags from the HeidelTime system as one of the input features. Along with these features, we generated and used the proposed features to develop the CRF for extracting temporal expressions. Selection of the feature set is based on our experimental evaluation. We train and evaluate our proposed model with various combinations of feature sets to chosen the best performing one. The proposed framework is used the Table 2 features to extract temporal expressions.

Fig. 3.

Framework for temporal expression extraction

Table 2.

Temporal expressions extraction: list of features

Serial number	Name of the features
i	Word
ii	POS tag of word
iii	Lemma of word
iv	Chunking
v	Is numeric
vi	HeidelTime tags
vii	Next word
viii	Next two words
ix	Previous word

Our proposed framework involves three steps to temporal expressions from raw clinical narratives. A detailed explanation of each step is given below:

• Step 1: We use HeidelTime on raw clinical narratives to extract temporal expressions. The temporal expressions is identified and annotated with three attributes: TYPE, MOD, and VAL attributes by HeidelTime system. It tags the frequency expressions as SET (which is not present in clinical narratives); thus, we manually checked and replaced SET expression type by FREQUENCY expressions. Finally, we converted all the tagged expressions as a feature called HeidelTime tags.

• Step 2: We generate a lexical feature set from raw clinical narratives by using GENIA tagger for lemmatization, part-of-speech (POS), chunking. We furthermore used NLTK package from Python for language models such as unigram, bigram features. Finally, we generate the “Is numeric” feature programmatically.

• Step 3: After the successful generation of features, we train the CRF model to extract temporal expressions.

After the successful development of our proposed method, we encode the temporal expressions from clinical narratives. Let us consider the following sentence to show the extracted temporal expressions through our developed model: “The patient is admitted <TIMEX3 > early on 09-01 </TIMEX3 > with type 1 diabetes mellitus, who is <TIMEX3 > four months </TIMEX3 > post cadaveric kidney transplantation and now has good graft function. She was discharged home in stable condition on <TIMEX3 > postoperative day six </TIMEX3 >.” The experimental evaluation of the proposed framework for event extraction is reported in [26] and discussed in Sections 4.3 and 5.1 in detail.

Extraction of Temporal Events

In the past several years, probabilistic models have been used for information processing and extraction. A conditional random field (CRF) is a widely used undirected probabilistic model, which assigns a label sequence to target sequence [18]. This probablistic model is often used for sequential data parsing or labeling, such as biological sequences or natural language text. It is evident from the literature that the advantages of CRF outperforms other methods in temporal information extraction [33]. Therefore, we focus on semi-supervised CRF by utilizing both annotated and unannotated clinical narrative text to extract temporal events. The semi-supervised CRF approach has been successfully applied to the processing of biomedical text before [15].

Events defined in clinical text generally refer to medical concepts and their relationships, such as clinical observations, symptoms occurred, disorder names, findings, and disease diagnosis described by medical professionals, medications prescribed by doctors, and the patient’s personal care or treatment procedures. Temporal event denotes the clinical event with the notion of time. Temporal events in clinical narratives have been divided into six groups according to I2B2 annotation guidelines. Standard types of temporal events are problem (disease and symptoms), test, treatment, evidential, clinical department, and occurrence.

Let us consider the following example sentences to understand more about temporal events in clinical narratives: “The patient presents for cadaveric pancreas transplantation. Her diabetes mellitus has been complicated by retinopathy and nephropathy as well as peripheral neuropathy. She takes 14 units of NPH insulin twice a day .”

In the above example, the event consists of medical terminologies and medication details as highlighted. Event extraction from such a clinical narrative requires the extensive domain-specific knowledge as discussed earlier. Moreover, obtaining annotation corpus is expensive, time-consuming, and requires quality of manual effort and domain knowledge. These constraints are affecting the creation of annotation corpus, which limits to few hundred records. Reported researches have used the supervised CRF and other various methods to extract the temporal events by using this limited number of annotated clinical text [33, 35]. By considering this limitations, to extract the temporal events, we propose a new semi-supervised framework. The graphical representation of proposed semi-supervised framework is shown in Fig. 4 for temporal events extraction. This framework is composed of two stages. In the first stage (stage I), a supervised linear-chain CRF8 model is developed to identify the temporal events by using the generated features sets, which is described in Sections 3.2.1 and 3.2.2 in detail. In the second stage (stage II), the semi-supervised CRF model is developed by utilizing the abundant unannotated data along with the generated feature sets. Thus, the annotated corpora is gradually increased. This approach helps to increase the performance of the temporal event extraction. We believe that this is the first proposed semi-supervised framework for temporal events extraction in clinical text. The Algorithm 1 illustrates the steps of the semi-supervised conditional random field model.

Fig. 4.

Graphical representation of the proposed framework for temporal event extraction

The second stage of the proposed semi-supervised framework is achieved with two tasks. First task is selection of unannotated data, which helps to select the information rich patient records from the available abundant medical records, and second task is feature generation and feature mapping, which helps to extend the labels for selected unannotated training data from annotated data. Finally, the successful generation of input features and feature mapping on the training data; we trained our target model (step 5) by using the Mallet semi-supervised CRF tool.9

Our model could automatically detect and annotate the temporal events of the example sentence as follows:

“The patient <EVENT> presents </EVENT> for <EVENT> cadaveric pancreas transplantation </EVENT>. <EVENT > Her diabetes mellitus </EVENT> has been complicated by <EVENT> retinopathy </EVENT> and <EVENT > nephropathy </EVENT> as well as <EVENT> peripheral neuropathy </EVENT>. She takes 14 units of <EVENT> NPH insulin </EVENT> twice a day.”

Selection of Unannotated Training Data

The selection of unannotated training data in first task from our proposed semi-supervised framework demands a essential strategy to select diverse training data. As previously mentioned, large collection of unannotated patient records are available for research; however, we cannot conclude that all disease groups (based on the international classification of diseases codes formulated by the WHO) are equally present in these unannotated patient records and that they contain rich and significant temporal information. In this situation, to fairly select the diverse patient records with rich temporal information, we leveraged machine-learning algorithm by considering its advantage. This smart approach selected unannotated patient records fairly rather than random selection. Moreover, this method supports to isolate diverse and rich patient records from massive unannotated patient records. To select the unannotated training records, first we should group all the unannotated patient records according to disease similarities by incorporating the domain knowledge.

There are various document clustering methods has been proposed in the existing researches [1]. The documents grouping or clustering using latent Dirichlet allocation (LDA) is based on latent topics. These topics are constructed over the probability distribution of words. However, in this method, we should initialize the number of latent topics in advance and the topic constructs over the distribution of words [8]. Based on the topics learned, the document clustering will be carried out with similarity measures. Here in our work, deciding the number of topic in advance by incorporating domain knowledge (International Classification of Diseases (ICD) codes) is more challenging and difficult in clinical narratives. Therefore, documents clustering using LDA leads to complex and time-consuming task to select unannotated training data for semi-supervised approach.

On the other hand, K-means clustering is used widely for text document grouping and suitable for larger datasets with less number of iterations [1]. Therefore, we chose the basic and prevailing K-means clustering algorithm to group the unannotated training records since we are grouping the patient records based on disease groups from ICD codes.

To accomplish the clustering, we exploited the popular and effective K-means clustering algorithm [17] with term frequency-inverse document frequency (TF-IDF) for clinical text. After clustering, we successfully obtained K number of clusters. Each cluster contains similar patient records and dissimilar from other clusters. We selected N number of sparse and diverse patient records from each cluster K based on our heuristic analysis. The selected unannotated data is included with the annotated data for next step in feature generation and mapping.

Feature Generation and Mapping

In order to train the semi-supervised CRF model and evaluate the performance, a list of feature groups was generated in the step 4.

Feature Generation

Based on the importance of each feature, the feature sets are selected. In the first group, lexical features are included, for example, the word, base form, POS tags, surrounding words, chunking, and language model. In the second group of feature set, only lemmatization of the word is considered. From the UMLS metathesaurus,10 the third group of feature set is generated, and it consists of semantic groups and concept unique identifiers (CUI) of each phrases.

• Lexical and syntactic features: GENIA tagger11 was used to create the lexical features. For example, POS tags, lemmatization, chunking the phrases. We have used the Python programming language with the NLTK package12 to generate the language model, POS tags and nearby words from the raw clinical text. In the label detection, the n-gram language model performs the significant role.

• Metamap features: The clinical text contains plenty of medical terminologies. In order to reliably detect the medical concepts from the clinical text, external medical knowledge is required. Thus, UMLS system was utilized by accessing the metamap tool to extract the important features such as CUI and semantic group.

Feature Mapping and Grouping

In order to develop an effective training steps, the labels for the unannotated dataset are extended. Different from newswire text, clinical text has the advantage that the words are grouped based on UMLS semantic grouping, which we use for our words and features, as well. Based on heuristic analysis, from the 133 semantic groups found in UMLS, 35 predominantly contributing groups are selected, such as diagnostic procedures, signs and symptoms, and we omit the rest of the semantic groups such as embryonic structure, cell, and entity functional concepts since these groups are not contributing significantly.

We trained semi-supervised the CRF model on our new training data (both annotated and unannotated), after the successful completion of feature generation and feature mapping in step 4. Subsequently, our developed model takes the advantage of the influence of unannotated data and achieves high accuracy in event extraction. The experimental evaluation of the proposed method for event extraction is reported elsewhere [27], but will be mentioned again in detail in the evaluation Section 4.3 and discussion Section 5.2.

Temporal Relation Extraction

Detection of temporal relations between a pair of events/expressions or events/events is very significant in temporal ordering of clinical events and medical applications [20]. Learning temporal relations from clinical narratives is a key task for temporal reasoning and representation. Moreover, relation detection is considered as a classification problem in temporal relation annotated corpus. However, understanding and identifying this implicit temporal order of events and/or expressions is very ambiguous yet a crucial task in the clinical domain. Therefore, many relation classification approaches have been proposed based on the available temporal relation annotated corpora [47].

Let us consider the following sentence for understanding the importance of learning temproal relations for temporal reasoning: “The patient was discharged home on postoperative day six with stable condition and excellent pancreas graft function.”

The sentence above apparently discusses about improvement of patient’s health status and the steps followed by the hospital. Therefore, learning the implicit temporal relation between the pair of events and expressions is a key to understanding the change of a patient health status.

To annotate the temporal relations in clinical text, there are various interval logic has been proposed. For example, Allen proposed an interval-based temporal logic for defining the relationship between two events [3]. In contrast to the interval temporal reasoning, a point of time reasoning has been proposed by M. Vilain and H. Kautz [41]. Among these representations, Allen’s interval logic is more suitable for representing the temporal events and temporal expressions found in clinical text [12], [48]. In Allen’s interval logic, there are 14 kinds of relations. At least one type of relationship should exist between any pair of events or expressions. In this work, we only use three relationships (AFTER, BEFORE, and OVERLAP) between pairs of events or expressions.

The proposed approach for temporal relation extraction consists of two tasks. They are as follows:

• Generating candidate pairs from extracted temporal events and expressions

• Classifying the temporal relationship between the pair of events/expressions

Figure 5 shows a graphical illustration of these two tasks.

Fig. 5.

Framework for temporal relation classification

Candidate Pair Generation

Candidate pair generation from the list of extracted events and expressions is a fundamental task for temporal relation classification. In our proposed approach, we have devised a new hypothesis for candidate pair generation based on the attributes of events and expressions along with the dependency parsing approach. In the proposed assumption, the temporal relationship exist between events or expressions that has an event types (PROBLEM, TEST, TREATMENT, EVIDENTIAL, DATE) in a sentence as shown below. The event type is identified and classified based on domain knowledge system in UMLS and HeidelTime. If the mixture of event types is in a sentence, it is considered the candidate pairs. The following hypothesis formulated based on our heuristic analysis:

• a) PROBLEM,PROBLEM

• b) PROBLEM and PROBLEM

• c) PROBLEM and DATE

• d) TREATMENT of PROBLEM or TREATMENT for PROBLEM

• e) TEST for PROBLEM

• f) EVIDENTIAL from TEST

• g) TREATMENT and DATE

• h) EVIDENTIAL and DATE

The following example sentences explain the mixture of event/expression types as explained in the above hypothesis:

1. The patient diabetes mellitus has been complicated by retinopathy, nephropathy, and peripheral neuropathy (assumption a, b).

2. The report shows that the patient takes 14 units of NPH insulin to control blood sugar twice a day (assumption c, d, g, h).

3. Oral glucose tolerance tests show that the patient has type 1 diabetes (assumption e, f).

We adopted the dependency parsing strategy as explained in [35] to generate the candidate pairs from the list of events and expressions. However, this approach is unable to discover all possible candidate pairs. Therefore, along with existing approaches, we come up with the above newly formulated assumption to discover the potential candidate pairs as shown in Figs. 6 and 7. Finally, we integrated the candidate pairs from both approaches and eliminate duplicates. This method helps to discover almost all possible candidate pairs from clinical narratives. After successfully generating the candidate pairs with target labels from annotated dataset, we move to the temporal relation classification task.

Fig. 6.

An example of candidate pair generation for intra-sentence temporal relation classification

Fig. 7.

An example of candidate pair generation for inter-sentence, cross-section, and temporal relation classification (highlighted the pairs generated by proposed hypothesis)

Temporal Relation Classification

We use the Naive Bayes Classifier implemented in the Weka toolkit13 to train the classifier for temporal relation classification. Basically, Naive Bayesian classifier is well-known for classification problem. In this approach, we estimate the probability distribution of target label of temporal relationship between the candidate pairs, Y, f : X → Y or equally P(Y|X), where X = {X₁, X₂, X₃ ... X_n} denotes the features of the raw clinical text. This is the preferred way of developing the classifier model P(Y|X) based on the training data and estimate the P(X|Y) and P(Y). The proposed framework for temporal relation classification is illustrated in Fig. 5. We perform the following steps for training of the classifiers and estimation of the distribution probability to predict unknown labels in intra-sentence, inter-sentence, and cross-section relationships.

1. Step 1: Preprocessing of the training and testing data

2. Step 2: Generating feature sets consisting of dependency-related features, position information, time-related features, event-related features, and distance between two events/expressions for three different classifiers (inter-sentence, intra-sentence and cross section)

3. Step 3: Training the classifier models using Naive Bayesian classifier with the prepared training data

4. Step 4: Estimating the probability distribution for training data using the developed model

5. Step 5: Evaluating the established classifier models with test data for inter-sentence, intra-sentence, and cross-section relationships

Let us consider the following sentence in clinical narrative from Fig. 1 processed through our proposed classification method: “The patient was admitted with type 1 diabetes mellitus who is four months post cadaveric kidney transplantation and now good graft function.”

As a result, we obtain the following output: <EVENT1=type 1 diabetes mellitus | EVENT2=postcadaveric kidney transplantation | Temporal relation=OVERLAP >

<EVENT2=postcadaveric kidney transplantation | TIMEX3=four months | Temporal relation=AFTER >

<EVENT2=postcadaveric kidney transplantation | EVENT3=good graft function | Temporal relation=OVERLAP >

After the achieving of generated features, we trained our proposed Naive Bayesian and LIBLINEAR SVM classification algorithm on candidate pairs generated by the dependency parsing approach and candidate pairs generated by proposed hypothesis. By performing the experiment with different experimental settings, a baseline result and a best result is obtained. The experimental result analysis of our proposed method for temporal relation classification is discussed in Sections 4.3 and 5.3 in detail.

Experimental Evaluation

The purpose of this experiment is to evaluate the efficacy of our developed model for temporal information extraction from clinical narratives that was introduced in Section 3. To conduct the experiment, we used the I2B2 annotated data set as a key data resource.

Dataset and Experimental Design

The I2B2 shared task on temporal relation challenge provides the preprocessed and annotated data for temporal information research. This annotated dataset was generated by following the I2B2 guidelines [32]; this annotated dataset of clinical text was created. We used this annotated dataset for our experimental evaluations. The annotated dataset of temporal relations challenge includes the training data of 180 annotated records and testing data of 120 annotated records. This training data contain nearly 10,000 sentences. The annotated training set consists of 16468, 2368, and 33660 temporal events, expressions, and relations, respectively. The testing set contains 13594, 1820, and 27530 temporal events, expressions, and relations, respectively.

In case of the preprocessing work, we have not done any specific preprocessing step in our proposed methods since we used the completely preprocessed and annotated data from I2B2 shared datasets. They have already done the de-identification and noise removal of training and testing data.

A semi-supervised framework was developed for temporal event extraction by using the 900 unannotated training data along with the annotated data. For this purpose, we used the unannotated data from I2B2 medication and relation challenges. This data consists of 85,000 sentences or more. Table 3 summarizes the different combination of feature sets in experimental settings and evaluated systems for temporal event extraction with number of training records.

Table 3.

Experimental settings for temporal event extraction and evaluation

	Lexical	Syntactic	UMLS	Data selection	No. of records
Baseline	∙				180
CRF (run1)	∙	∙			180
CRF (run2)	∙	∙	∙		180
Semi-CRF (run3)	∙	∙	∙	Random selection	300
Semi-CRF (run4)	∙	∙	∙	K-means clustering	500
Semi-CRF (run5)	∙	∙	∙	K-means clustering	900

Baseline

According to the first stage of the reported framework, temporal events are parsed by using the supervised CRF model. This model was developed with lexical features consisting of POS tags, base form, chunking and BIO-events (beginning, inside, and outside of event) as target label. In order to improve the performance and accuracy, we exploited the UMLS by accessing metathesaurus tool to incorporate the domain knowledge. Semantic group features and metamap CUI from UMLS help to increase the accuracy of the temporal event extraction as visible in Table 5. The proposed semi-supervised CRF model with the abovementioned features was trained on the I2B2 training data and unannotated data. At last, our model is tested on I2B2 testing data.

Table 5.

Evaluation results: extraction of temporal events

Temporal events	Method	Precision	Recall	F-measure
Baseline [26]	Supervised CRF	69.7	78.1	73.66
Run1 [26]	Supervised CRF + partial features	85.27	72.53	78.39
Run 2 [26]	Supervised CRF + all features	85.52	77.56	81.34
Run3 [27]	Semi-supervised CRF (SSCRF)	86.41	82.25	84.21
Run4 [27]	SSCRF	91.56	88.14	89.76
Run5, this work	SSCRF with increased no. of records	91.68	88.36	89.98
Best system from I2B2 [35]	CRF+ features	93.74	86.79	90.13

Evaluation Metrics and Results

Our models were evaluated with three measures, namely precision, recall, and F-measure, which are defined as follows:

$$\text{Precision}\left(P\right)=\frac{|\text{Sys.Output}\cap \text{Annotated.Corpora}|}{\left|\text{Sys.Output}\right|}$$ 1

$$\text{Recall}\left(R\right)=\frac{|\text{Sys.Output}\cap \text{Annotated.Corpora}|}{\left|\text{Annotated.Corpora}\right|}$$ 2

$$F-\text{measure}=2\times \frac{P\times R}{P+R}\mathrm{.}$$ 3

The results of our proposed approach for temporal expressions, events extraction and relation classification on I2B2 testing data is discussed in Tables 4, 5, and 6, respectively.

Table 4.

Evaluation results: extraction of temporal expression

Temporal expressions	Method	Precision	Recall	F-measure
HeidelTime system	Baseline [26]	77.62	79.80	78.69
HeidelTime system + features	Hybrid method [26]	81.53	79.11	79.95

Table 6.

Evaluation results of temporal relation classification

Temporal relations	Precision	Recall	F-measure
Baseline (SVM)	71.42	39.54	50.90
Dependency - SVM (overall)	77.17	50.32	60.91
Dependency + hypothesis - SVM (overall)	81.43	53.75	64.76
Baseline (Naive Bayes)	83.39	32.54	46.81
Naive Bayes (intra-sentence)	74.4	75.6	74.2
Naive Bayes (inter-sentence)	66.3	66.4	66.34
Naive Bayes (cross section)	69.2	72.4	70.7
Dependency - Naive Bayes (overall)	67.12	63.76	65.39
Dependency + hypothesis-Naive Bayes (overall)	66.8	68.4	67.1

The proposed framework of the temporal expression extraction with the newly added feature set achieved an F-measure of 79.95, which is slightly better than the HeidelTime system. Our system is able to extract special prepost expressions from clinical narratives. The precision increased considerably, whereas recall does not show much difference. Therefore, the overall performance is slight improved compared to the HeidelTime system as shown in Table 4.

In order to establish the semi-supervised method, we applied the K-means clustering algorithm on the unannotated clinical narratives to select the rich data. Thirty clusters were generated in this process and 30 records were selected from each clusters. At the end of this process, 900 sparse unannotated training records were generated along with abundant temporal information. From these 900 records, we used 300 records initially for training of the event extraction model. Consequently, the size of annotated data is increased, whereas unannotated data is added gradually from the selected unannotated records. Finally, we checked the stability of our developed event extraction model with the testing data.

The proposed semi-supervised framework with K-means data selection method achieved an F-measure of 89.98% for temporal event extraction. We experimented with various window size using semi-CRF and conferred this best results for window size {− 3, 3}. This F-measure is closer to the top performing system based on the hybrid approach [35], which is shown in Table 5. Due to the addition of unannotated dataset to the annotated data in the developed semi-supervised CRF, the model claims a great improvement in precision, recall, and F-measure. Apart from the performance and accuracy, the reliability and stability of the developed model were tested by applying it on the testing dataset as shown in Fig. 8. Our approach is flexible regarding addition of unannotated training data, and the corpus size can be easily increased. Subsequently, the accuracy and stability of the system is increased by reducing time and effort. Finally, we accomplished our objective in automatically annotating the unannotated clinical narratives with temporal events, which reduces time, cost, and manual effort. A very high F value of 89.98 is achieved.

Fig. 8.

Performance of the proposed semi-supervised CRF model on temporal event extraction: random selection (a, c), K-means clustering (b, d) for number of iterations and number of training records respectively

We initially developed and evaluated the baseline classification model using gold standard annotated data from I2B2 temporal relations. From Table 6, we can notice the improved performance of our developed classifiers (inter-sentence, intra-sentence, and cross section) on the generated candidate pairs and features. Moreover, the overall F-measure of temporal relation classification is increased from 46.81 to 67.1. From these results, it can be understood that fluctuating recall and biased precision are affecting the classification performance. However, the recall of our proposed method has increased drastically compared to the baseline model, and the biased precision has stabilized. These effects are due to the incorporation of the effective candidate pair generation hypothesis.

Discussion

The obtained results were analyzed extensively, i.e., which kind of events, expressions, and relations are difficult to be detected by our method.

Extraction of Temporal Expression

Six types of temporal expressions are annotated in I2B2 dataset.The temporal expression extraction performance of our method was manually analyzed by comparing our annotation results with the reference annotations in the I2B2 dataset. We found out that the main source of errors originated from expression recognition. We list out the summary below:

• Unclear expressions: A large number of the date expressions (such as “the night prior,” “that time,” “this time,” and “the day prior to admission”) are ambiguous to recognize. However, these expressions are fairly important. It is quite difficult to identify and recognize these types of expression. For these kind of expressions, our framework has limitation.

• Complicated frequency expressions: Short forms, such as t.i.d, b.i.d, qd, are mostly used for the frequency expressions. The bag-of-words model will be utilized in our future implementations to detect the above expressions. Moreover, duration expressions and frequency expressions are almost identical with the similar words except minute difference. For example, “three months” denotes the duration where as “every three months” belongs to frequency.

Temporal Event Extraction

We manually analyzed the temporal event extraction performance of our developed model from stage 1 very carefully by comparing our annotation results with I2B2 testing dataset. It was found out that the main source of errors is from event recognition. Our findings are listed below:

• Inadequacy of annotated datasets is a main reason for the reduced performance in the temporal event detection. This issue is addressed in stage 2 of semi-supervised framework with the utilization of unannotated dataset.

• Partial identification of events: Some medical symptoms and disease name are very lengthy. It will not be identified with the best window size of the developed model. For example, “evolution of the right posterior cerebral artery infarction,” “acute pain in pancreas and a 1 cm cyst in the right lobe of the liver.” This type of events were partially identified by our method. This issue consequently affected the performance.

The size of the annotated dataset is small and it could not cover all possible events compared to vast amount of unannotated clinical narratives available. Also, some of the events are very long and it is difficult to identify them effectively by using CRF.

The above limitations we was overcome by processing unannotated data in stage 2 of our method. This proposed approach helps to significantly improve the performance. In stage 2, to select the training data K-means clustering was utilized and feature mapping was introduced to merge the annotated and unannotated training dataset.

Our experiment was carried out with a gradual increment of training records along with the increased iterations. The performances of existing supervised methods is normally affected by new testing data as the number of annotated data is limited, while our proposed method allows gradual addition of unannotated data which helps to identify new events effectively. Therefore, the performance of event extraction is steadily increasing in each step. Moreover, the recall is increased significantly and the precision is increased slightly due to the addition of unannotated clinical narrative as shown in Table 5. From this, we concluded that unannotated data contributed to increase the accuracy of temporal event detection as expected. We planned to use CRFs with word embeddings as in [36] to extract very long events in future.

Temporal Relation Classification

Though the proposed method could improve the performance of temporal relation classification (precision and recall), it still has room for improvement on generating candidate pairs. Some of the pairs generated by our proposed approach do not have any relationships. Hence, in the future, we plan to focus on finding new strategies for candidate pairs generation and subsequently improve the precision, recall, and F-measure of temporal relation classification.

Conclusions

Due to the large availability and ambiguity of temporal information in clinical text, computers have difficulties in understanding and processing temporal information from EMRs automatically. In this work, we introduce a novel approach to parse clinical text from patient records, and output structured representations of the temporal information for further automated processing. This requires three tasks: For temporal expression recognition, we proposed a novel feature set along with the adoption of the HeidelTime system, whereas a novel semi-supervised framework was developed for extracting temporal events from clinical narrative text by extending the training data. For temporal relation classification, we formulated a new hypothesis for generating candidate pairs and exploited the Naive Bayes classifier. Identification of potential of generated candidate pairs helps to increase the performance of relation classification between events or expressions. The achieved results for temporal expressions, temporal events and relationship classification by our proposed approach have been discussed in detail. The performance and stability of the semi-supervised method for event extraction is significantly increased compared to the baseline model due to the addition of unannotated data, our novel assumption, incorporation of metamap features from UMLS. In our future work, we plan to investigate the expansion of lexical feature sets by using medical dictionary data and semantic ontologies for event extraction while considering the clinical variation and abbreviation ambiguity. Additionally, we are planning to expand the candidate generation heuristics along with the proposed hypothesis to enhance the performance of temporal relation classification.

Funding Information

This work is partially supported by Japan Ministry of Education, Culture, Sports, Science and Technology scholarship and the Vietnam National University at Ho Chi Minh City under the grant no. B2015-42-02.