Cohort selection for clinical trials using hierarchical neural network

Abstract

Objective

Cohort selection for clinical trials is a key step for clinical research. We proposed a hierarchical neural network to determine whether a patient satisfied selection criteria or not.

Materials and Methods

We designed a hierarchical neural network (denoted as CNN-Highway-LSTM or LSTM-Highway-LSTM) for the track 1 of the national natural language processing (NLP) clinical challenge (n2c2) on cohort selection for clinical trials in 2018. The neural network is composed of 5 components: (1) sentence representation using convolutional neural network (CNN) or long short-term memory (LSTM) network; (2) a highway network to adjust information flow; (3) a self-attention neural network to reweight sentences; (4) document representation using LSTM, which takes sentence representations in chronological order as input; (5) a fully connected neural network to determine whether each criterion is met or not. We compared the proposed method with its variants, including the methods only using the first component to represent documents directly and the fully connected neural network for classification (denoted as CNN-only or LSTM-only) and the methods without using the highway network (denoted as CNN-LSTM or LSTM-LSTM). The performance of all methods was measured by micro-averaged precision, recall, and F1 score.

Results

The micro-averaged F1 scores of CNN-only, LSTM-only, CNN-LSTM, LSTM-LSTM, CNN-Highway-LSTM, and LSTM-Highway-LSTM were 85.24%, 84.25%, 87.27%, 88.68%, 88.48%, and 90.21%, respectively. The highest micro-averaged F1 score is higher than our submitted 1 of 88.55%, which is 1 of the top-ranked results in the challenge. The results indicate that the proposed method is effective for cohort selection for clinical trials.

Discussion

Although the proposed method achieved promising results, some mistakes were caused by word ambiguity, negation, number analysis and incomplete dictionary. Moreover, imbalanced data was another challenge that needs to be tackled in the future.

Conclusion

In this article, we proposed a hierarchical neural network for cohort selection. Experimental results show that this method is good at selecting cohort.

INTRODUCTION

Cohort studies are an important type of medical research used to evaluate associations between diseases and exposures. Cohort selection that selects patients who meet predefined criteria is a necessary prerequisite for cohort studies. The most common cohort selection, namely phenotyping, is based on electronic health records (EHRs), which can be extremely time-consuming and challenging due to the complexity of the criteria and the diversity of EHR data. Matching the criteria for cohort selection need to access and analyze multiple documents across different subsystems of a patient. The data include structured tables (such as laboratory results and medications) and unstructured text (such as admission notes, radiology reports, progress notes, and discharge summaries). Extracting meaningful information from the mix of structured and unstructured data is not easy because of the presence of ungrammatical text, abbreviations, misspellings, local dialectal phrases, and quantitative information in various formats. Moreover, cohort selection requires further information consolidation.

In the past decades, a large number of studies have been proposed for cohort selection using EHR and various phenotypes, such as cancer,¹ diabetes,² heart failure,³ smoking status,⁴ obesity,⁵ etc., from different data sources including clinical notes, imaging data, insurance claim data, etc., have been considered in selection criterion. These methods may fall into the following 3 categories: rule-based methods,^6–11 machine learning-based methods,^12,13 and hybrid methods.^14,15

The rule-based methods usually need medical experts to design intricate rules based on cohort selection criterion, such as logical constraints like (hemoglobin < 10 and age > 50). A representative work on rule-based methods was developed by the Electronic Medical Records and Genomics (eMERGE)¹⁶ consortium. The eMERGE consortium developed a series of algorithms to define phenotypes and implement and validate rules for a number of diseases. Some complex rules such as nested Boolean logic and negation, cardinality constraints, and temporal logic were included in the eMERGE project. The Phenotype KnowledgeBase (PheKB)¹⁷ repository integrated the algorithms from eMERGE as well as from other studies. The rule-based methods are effective but have their own drawbacks. First, it is almost impossible to apply each existing algorithm to new data directly because of various data formats. Second, the existing algorithms only cover a small part of diseases or phenotypes. Developing a rule-based system for new diseases and phenotypes is a very time-consuming and laborious process.

Machine learning methods that have the ability to automatically learn patterns from data have attracted more and more attention. For instance, Mani et al¹⁸ compared classical machine learning methods such as k-Nearest-Neighbors (KNN), Naïve Bayes (NB), decision tree (DT), random forest (RF), Classification and Regression Tree (CART), support vector machine (SVM) and logistic regression (LR) for type 2 diabetes risk forecasting. Zheng et al² compared all these methods except CART to identify type 2 diabetes on another corpus. Recently, a few researchers have attempted to introduce deep learning methods for cohort selection and obtained promising results.¹⁹

Hybrid methods combine rule-based methods and machine learning methods in a number of ways. The 2 typical ways are: 1) features extracted by rules are fed into machine learning methods together with other features; 2) rule-based methods are used as a post-processing module to fix errors caused by machine learning methods. For example, Xu et al¹⁴ proposed a hybrid method to detect patients with colorectal cancer, using the machine learning method for colorectal cancer case determination on the free text and rule-based method on structured data.

The 3 categories of methods usually use natural language processing (NLP) techniques to extract useful information such as clinical entities, quantitative information, temporal information, and so on. The most frequently used NLP tools are MetaMap,²⁰ HITex,⁴ CUIMANDREef,³ REX,²¹ EPiDEA,²² MedLEE,²³ cTAKES,²⁴ and Negex.²⁵

Although lots of methods have been proposed for cohort selection, it is still difficult to compare them with each other due to the lack of public data. The Integrating Biology and the Bedside (i2b2) center organized 2 shared tasks on smoking detection²⁶ and obesity identification⁵ in 2006 and 2008, respectively. The rule-based methods performed just as well as other methods in the i2b2 2006 challenge and outperformed other methods in the i2b2 2008 challenge. The limitation of the 2 challenges is that they only considered 1 phenotype. In 2016, a challenge on symptom severity prediction from neuropsychiatric clinical records was organized,²⁷ and the ensemble system of association rules and machine learning methods won the challenge. The machine learning methods used in the ensemble system include SVM, RF, NB, Adaboost, and deep neural network.

In 2018, the national NLP clinical challenges (n2c2), which has a track (track 1) on cohort selection for clinical trials considering 13 criteria (ie, drug abuse), was launched. In this challenge, the organizer provided the de-identified mental health records (MHRs) of 288 patients for system development and test. We participated in this challenge and proposed a hierarchical neural network. Inspired by Yang et al,²⁸ we used a hierarchical network to represent documents based on sentence representation. In the hierarchical network, a convolutional neural network (CNN) and a long short-term memory (LSTM) network were used for sentence representation, a highway network²⁹ was applied to adjust information flow, a self-attention mechanism³⁰ was adopted to reweight sentences, LSTM was used for document representation, and a weighted cross entropy was used as loss function to relieve the data imbalance problem. Experimental results showed that our proposed hierarchical neural network was powerful for cohort selection for clinical trials.

MATERIALS AND METHODS

Data sets

The organizers of the n2c2 2018 challenge provided a corpus composed of MHRs of 288 patients manually annotated with indicators of 13 predefined selection criteria. The corpus was split into a training set of MHRs of 202 patients and a test set of MHRs of 86 patients. The predefined selection criteria are described in Supplementary Table 1 in detail. In this corpus, each patient had 2-–5 records. Figure 1 lists the data distribution on the 13 selection criteria, some of which data distribution is extremely imbalanced, such as ALCOHOL-ABUSE, ASP-FOR-MI, DRUG-ABUSE, ENGLISH, KETO-1YR, MAKES-DECISIONS, and MI-6MOS.

Figure 1.

Data distribution on cohort selection criteria.

Data processing

The task of cohort selection for clinical trials is to determine which criteria a patient satisfies among the 13 predefined criteria according to the patient’s MHRs. In order to tackle this task in a unified framework, we processed the MHRs of each patient as shown in Figure 2. First, we extracted section information, tables, and temporal information from the MHRs. We followed previous studies³¹ to extract section information and temporal information. To extract information from tables, we fixed their position by finding table heads followed by consecutive short lines (eg, “—-“) and consecutive blank rows, and then obtained items and values by splitting the sentence by 2–4 consecutive spaces. The table heads included “data,” “labs,” “labs/studies,” “chemistries,” “selected recent labs,” “data: pending,” “database,” “chemistry,” “laboratory,” “labs/rads,” “laboratory data,” “lab results,” “laboratories,” “lab data,” and “results and diagnostic testing.” To represent table information, we converted items with number values into items with text values such as high, low, normal, or unknown. In this way “HCT 35.6” was converted into “HCT L” (the normal value of HCT is 36.0–46.0. Second, we designed simple rules based on a key word dictionary to extract sentences related to each criterion (called clues) and ranked them in chronological order. The clues were sentences containing some key words we collected as shown in Supplementary Table 2. Finally, we obtained 12 235 clues and each clue was represented by (temporal information, section information, clue) such as (“2071/02/13”,“general situation”, “that he has had renal insufficiency in the past”).

Figure 2.

Data processing flow.

Task definition

We recognized the cohort selection for clinical trials as 13 binary classification tasks corresponding to the 13 criteria. Formally, each patient p is labeled with $y_i$ corresponding to the i-th criterion, where$\mathrm{}i=0,\mathrm{}1,\dots ,\mathrm{}12$ and $y_i\in \{0,\mathrm{}1\}$. The clues we extracted for each selection criterion form an instance $E=s_1{s}_2\dots \mathrm{}{s}_k\dots$, where $s_k$ is a clue containing $m$ words $w_1^k{w}_2^k{\dots w}_m^k$. All clues in E are ranked in chronological order.

Hierarchical neural network

Following Yang et al.'s work ^[28], we used the hierarchical neural network called CNN-Highway-LSTM and LSTM-Highway-LSTM, as shown in Figure 3 to classify an instance into 0 or 1, where 0 stands for “not met” and 1 for “met”. The neural network includes 5 components: (1) a neural network for sentence representation using CNN or LSTM, (2) a highway network to adjust information flow, (3) a self-attention neural network to reweight sentences, (4) a neural network for document representation network using LSTM, and (5) a fully connected neural network for instance classification. In the following sections, we present the components one-by-one in detail.

Figure 3.

Overview architecture of the hierarchical neural network for instance classification.

Sentence representation

For clue $s_k$, each word $w_j^k\mathrm{}(j=0,\mathrm{}1,\dots ,\mathrm{}m)$ was represented with a d dimensional word embedding. We applied CNN or LSTM to obtain the representation of $s_k$, denoted by $v_k$. In CNN, convolution operation was applied on $s_k$ using g sliding filters of different size l, and max pooling was adopted for down-sampling. In LSTM, we used the following bidirectional LSTM to represent $s_k$:

$$\overrightarrow{h_t^k}=\overrightarrow{\mathit{LSTM}}\left(\left[w_t^k,{\overrightarrow{h}}_{t-1}^k\mathrm{}\right]\right),t=1,\mathrm{}2,\mathrm{}\dots ,\mathrm{}m$$ (1)

$$\overleftarrow{h_t^k}=\overleftarrow{\mathit{LSTM}}\left(\left[w_t^k,\mathrm{}{\overleftarrow{h}}_{t+1}^k\right]\right),t=m,\mathrm{}m-1,\dots ,\mathrm{}1,$$ (2)

where ${\overrightarrow{h}}_t^k$ and ${\overleftarrow{h}}_t^k$ are hidden states of $w_t^k$ from forward and backward directions. The 2 last hidden states were concatenated together to form the representation of $s_k$, that is, $v_k=\left[{\overrightarrow{h}}_m^k,\mathrm{}{\overleftarrow{h}}_m^k\right]$.

Highway network

We used a highway network to adjust information flow as follows:

$$u_k=\mathrm{}H\left(v_k,\mathrm{}{W}_H\right)·T\left(v_k,\mathrm{}{W}_T\right)+v_k·C(v_k,\mathrm{}{W}_C),$$ (3)

where $u_k$ is the highway output of the input $v_k$, $W_H$, $W_T,$ and $W_C$ are parameters, H is a transformation function, in our case, the ReLU function. T is a transformation gate:

$$T\left(v_k,\mathrm{}{W}_T\right)=\mathrm{}\frac{1}{1+e^{-({\left(W_T\right)}^T{v}_k+b^T)}},$$ (4)

C is a carrying gate usually set to $C=1-T.$

Self-attention

As different clues may be of different importance, we reweighted all clues using self-attention as follows:

$${\alpha }_s=\frac{\exp \left(u_k^T{u}_s\right)}{{\sum }_s\exp \left(u_k^T{u}_s\right)}$$ (5)

$$u_k^{\mathrm{\text{'}}}={\sum }_s{\alpha }_s{u}_k,$$ (6)

where ${\alpha }_s$ is the weight of $u_s$, and $u_k^{\mathrm{\text{'}}}$ is the output of the input $u_k$.

Document representation

To take advantage of the temporal relationship among the clues of an instance (also called document in this study) $E$, we used the following LSTM to represent the instance:

$$h_k=\mathit{LSTM}\left(\left[h_{k-1},\mathrm{}{u}_k^{\mathrm{\text{'}}}\right]\right),$$ (7)

where $h_k$ is the hidden state at time $k$. The last hidden state of the LSTM is used as the representation of $E$, denoted by $D$.

Classifier

We fed the representation of instance $E$ (ie, $D$) into a multi-layer perceptron network for classification:

$$y_i^{\mathrm{\text{'}}}=\mathit{argmax}\left(\mathit{softmax}\right(W_{d}D+b_d\left)\right)$$ (8)

where $W_d$ is a weight matrix, $b_d$ is a bias vector, and $y_i^{\mathrm{\text{'}}}\mathrm{}$is the label for the i-th criterion predicted by our model.

As the data distribution on some cohort selection criteria is extremely imbalanced, we leveraged class weights when model training. For the i-th criterion, the weight of class c is calculated by

$$c{w}_c=\mathrm{}\frac{\sum _{c\in \{0,1\}}\mathit{count}\left(c\right)}{2\mathrm{*}\mathit{count}\left(c\right)},$$ (9)

where $c{w}_c$ is the weight of class c, $\mathit{count}\left(c\right)$ is the number of samples in $c$.

Then, weighted cross entropy was used as classification loss $L$, which can be written as$:$

$$L=-\sum _{y_i}c{w}_{y_i}·p_{y_i}·\mathit{log}{p}_{y_i^{\mathrm{\text{'}}}},$$ (10)

where $p_{y_i}$ is the probability of label $y_i$.

EXPERIMENTS

The baseline methods used in our study were the following variants of our method (CNN-Highway-LSTM or LSTM-Highway-LSTM): (1) CNN-only or LSTM-only that only contains CNN or LSTM to represent documents directly and multi-layer perceptron for classification; (2) CNN-LSTM or LSTM-LSTM that contains all components except the highway network. We used the same hyperparameters for all models. The word embedding dimension (d) was set to 200, the sizes of sliding filters in CNN (l) were set to 3 and 5, the number of filters of the same length in CNN (g) was set to 150. the dimension of hidden states in LSTM was set to 100, the learning rate was set to 0.001, and the dropout probability was set to 0.5. The epoch was tuned from 1 to 80 via 5-fold cross-validation on the training set. We implemented all methods using Tensorflow, and pretrained the word embeddings on the “NOTEEVENTS” of MIMIC III using the Word2vc tool (https://radimrehurek.com/gensim/models/word2vec.html) with default parameters. All models were evaluated on the independent test set, and their performance was measured by micro-averaged precision (P), recall (R), and F1 score (F), calculated by the office tool provided by the challenge organizers.

RESULTS

Supplementary Table 3 shows the overall performance of LSTM-Highway-LSTM and its variants. LSTM-Highway-LSTM achieved the highest micro-averaged F1 score of 0.9021, much higher than that of the baseline methods. The F1-score difference between LSTM-Highway-LSTM and the baseline methods ranges from 1.53% to 4.97%. Both the highway network and the LSTM for document representation showed performance improvement. For example, for LSTM-Highway-LSTM, the improvement gain from the highway network is 1.53% in F1-score (LSTM-Highway-LSTM vs LSTM-LSTM), and the improvement gain from both the highway network and the LSTM for document representation is 5.96% in F1-score (LSTM-Highway-LSTM vs LSTM-only). The methods using LSTM for sentence representation outperform the methods using CNN. However, CNN-only performs better than LSTM-only.

It should be stated that the best F1-score of our submission for the challenge is 0.8855 achieved by an ensemble method of CNN-only and CNN-Highway-LSTM. The ensemble strategy is to select better results from them on each criterion.

Moreover, we further investigated the performance of CNN-Highway-LSTM and LSTM-Highway-LSTM on each selection criterion. From Supplementary Table 4, we can see that LSTM-Highway-LSTM performs better than CNN-Highway-LSTM on all cohort selection criteria except ADVANCED-CAD and DRUG-ABUSE. Among 13 criteria LSTM-Highway-LSTM does not perform very well on ABDOMINAL, ALCOHOL-ABUSE, ASP-FOR-MI, DRUG-ABUSE, KETO-1YR, MAKES-DECISIONS, and MI-6MOS, with an F1-score lower than 0.8000.

DISCUSSION

In this study, we proposed a hierarchical neural network for cohort selection for clinical trials and evaluated it on the corpus of the n2c2 challenge in 2018. Our proposed method achieves promising performance, but does not perform well on some cohort selection criteria, especially on the imbalanced criteria. Among 7 criteria on which the F1-score of LSTM-Highway-LSTM is lower than 0.8000 as mentioned above, 6 criteria are extremely imbalanced. Nevertheless, the data imbalance problem is partly relieved by introducing class weights on all the 6 imbalanced criteria except ALCOHOL-ABUSE. We checked the ALCOHOL-ABUSE results and found that the clues varied dramatically in the description. The average improvement gain from class weights is more than 9% in F1-score for the other 5 imbalanced criteria.

The reason why CNN-only outperforms LSTM-only may be that a certain number of documents are too long. In the corpus of the n2c2 2018 challenge, there are totally 3744 = 288*13 documents (ie, instances) composed of 12 235 sentences (ie, clues). Of the documents, 770 do not contain any sentence. The average length of the documents is near 100, and the max length of the documents exceeds 2000. For the same reason, we introduced hierarchical architecture to represent documents. It is easy to understand that LSTM-Highway-LSTM performs better than CNN-Highway-LSTM when documents are split into sentences with an average length of 24, since LSTM has the advantage of representing sentences over CNN.

To further understand the errors of our model, 20 wrongly classified examples are randomly selected and analyzed. Misunderstanding of clues causes most errors. There are 3 cases as follows:

(1) word ambiguity. For example, “cr” in (“2093/07/09,”“medication,”“toprol xl (metoprolol succinate extended release) 100mg tablet cr 24hr take 1.5 tablet(s) po qd”) is wrongly recognized as “creatinine” instead of “chromium.”

(2) negation. For example, “MI” should be negative in (“2104/03/15,” “health maintenance,” “to obs for ro mi”) as “ro” stands for “rule out.”

(3) number. For example, it is difficult to determine whether “1.5” is high or not in (“2069/06/27,” “labs,” “bun and creatinine of 36 and 1.5, which is”) as no clear reference value is provided for creatinine.

There also a few errors caused by the use of an incomplete dictionary used for clue extraction, such as the diet supplement dictionary. Because of the incomplete diet supplement dictionary, no clue can be extracted for patients taking diet supplements.

There may be 3 major future directions for further improvement: (1) searching more effective methods to tackle imbalanced data, (2) using NLP techniques to help us understand clues better, and (3) expanding incomplete dictionaries.

CONCLUSION

In this article, we proposed a 5-layer hierarchical neural network for cohort selection for clinical trials. In this network, LSTM and CNN were individually used to represent sentences, the highway network was used to adjust information flow, the self-attention mechanism was used to reweight sentences, LSTM was used to represent documents in chronological order, and weighted cross entropy was used as classification loss to relieve the class imbalance problem. Experiments on a benchmark data set show that the proposed method is effective for cohort selection for clinical trials, and the highway network, self-attention mechanism, and weighted cross entropy also demonstrate their advantages.

FUNDING

This paper is supported in part by grants: NSFCs (National Natural Science Foundations of China) (U1813215, 61876052 and 61573118), Special Foundation for Technology Research Program of Guangdong Province (2015B010131010), Strategic Emerging Industry Development Special Funds of Shenzhen (JCYJ20170307150528934 and JCYJ20180306172232154), Innovation Fund of Harbin Institute of Technology (HIT.NSRIF.2017052).

AUTHOR CONTRIBUTIONS

The work presented here was carried out in collaboration between all authors. YX, XS, and BT designed the methods and experiments. YX and XS conducted the experiment. All authors analyzed the data and interpreted the results. YX and BT wrote the paper. All authors approved the final manuscript.

PATIENT CONSENT

Obtained.

PROVENANCE AND PEER REVIEW

Not commissioned; externally peer reviewed.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Conflict of interest statement

None declared.