MERIT: Minimal SupErvision Through Label Augmentation for Biomedical RelatIon ExTraction

Abstract

Relation Extraction (RE) is an important task in extracting structured data from free biomedical text. Obtaining labeled data needed to train RE models in specialized domains such as biomedicine can be very expensive because it requires expert knowledge. Thus, it is often the case that RE models need to be trained from relatively small labeled data sets. Despite the recent advances in Natural Language Processing (NLP) approaches for RE, training accurate RE models from small labeled data is still an open challenge. In this paper, we propose MERIT, a simple and effective approach for label augmentation that automatically increases the size of labeled data while introducing a moderate labeling noise. We performed extensive experiments on three benchmarks biomedical RE data sets. The results demonstrate the effectiveness of MERIT compared to the baseline.

Introduction

Relation Extraction (RE) is defined as classifying a type of relationship between a pair of entities occurring in a text passage. For example, given the sentence “Ciprofloxacin has some effect on Pyelonephritis”, we can infer the relation “May Treat” between two entities Ciprofloxacin and Pyelonephritis, and form a triplet (subject, relation, object). The extracted triplets from the text can be used for knowledge base population, question answering, or information retrieval. Recent advances in deep learning allow training very accurate¹² models^{41,19,10,39,24} when a large amount of human-annotated training data is available. However, collecting training data is a costly and human-intensive process. In some specialized domains such as information extraction from biomedical documents, human annotation is particularly challenging and costly because it can only be done by biomedical experts. Therefore, RE training data in the biomedical domain can often be very small and result in RE models with low accuracy.

Weak or distant labeling is a popular approach for addressing label scarcity issues in many applications, including RE^18,4. Distant supervision was applied previously to heuristically align entities to a given knowledge base (KB) with little annotation effort²⁶. However, that approach requires a KB and is not able to take into consideration the context of entity co-occurrence. In weak labeling, labeling rules are used by string matching to automatically provide labeled-data^{29,42,18,23,31,30}, which are often more efficient than using a KB for distant supervision⁴². However, exact string matching limits the generalizability of the rules², and thereafter causes low coverage of data and labeling noise. To tackle the labeling noise issue, data programming^31,30 aims to annotate the corpus by fitting a model to resolve any disagreements among the rules. Approaches to address the coverage of the labeling rules include differentiable soft-assignment of the rules to an unlabeled portion of the corpus^42,32,25. Although these approaches provide higher coverage of the rules, they still suffer from labeling noise. Moreover, generating labeling rules is a costly and inexact process and depends on the skill of a user to convert their knowledge into useful rules. There have also been efforts in reducing annotation costs in biomedical informatics by effective visual interfaces for annotation⁸ and semi-structured annotation¹⁶. However, these solutions are less applicable to RE domain.

As an alternative to weak labeling, we propose a simple and efficient approach, MERIT, to automatically increase the number of labels given a small, labeled data set. Our main observation is that the nearest neighbors of a sentence representing a particular relation between entities are likely to⁴³ represent the same relation. Thus, given a labeled sentence, we transfer its label to all its neighboring sentences. An open question to be studied in this paper is defining the neighborhood in the RE task, which requires us to define an appropriate distance measure and an appropriate distance threshold. The benefit of MERIT is that it does not require any expert knowledge, is easy to implement, and is computationally inexpensive. The proposed approach can be combined with other approaches dealing with data labeling scarcity such as weak labeling and active learning with uncertainty-based sampling^34,37,38 and clustering-based sampling^27,35.

Complementing recent efforts on synthetic training data augmentation^28,11 and other types of label augmentation³⁵, we show that MERIT successfully exploits the unlabeled portion of corpus to augment the limited available hand-labeled data with high-quality weak labels, which results in a more accurate RE model. We perform extensive experiments on three benchmark biomedical relation extraction datasets.

Task Formulation and Background

Given a sentence S_i and an entity pair (e_subj, e_obj), where e_subj is the subject entity and e_subj is the object entiry, the RE task can be transformed to a classification problem. A classifier can be built to map the relation between the subject and object entities into predefined relation types from the set of relations {R ∪ NA}, where NA denotes that there is no relation between a pair of entities.

In order to represent the candidate relation pair from an input sequence, we replace the relevant entity names with their semantic types followed by the standard preprocessing steps for RE⁶. an example of input representation to the RE model is “We further show that @PROTEIN$ directly interacts with @PROTEIN$ and Rpn4.”

We fine-tune SciBERT model³ followed by a linear classification layer added on top of it to predict the relation type of a candidate pair. In other words, given sequence S_i = (W₁, … , W_n), where W_i is the ith token in the sequence, we feed S_i into SciBERT, and retrieve the hidden state representation of the sequence (called the CLS) along with the word representations,

$$\left(h_{CLS},h_1,\mathrm{...},h_n\right)=SciBERT\left(w_1,\mathrm{...},w_n\right),$$

where h_i is the representation of ith token in a d dimensional space. As typical with BERT⁶, we use h_CLS, corresponding to the aggregate representation of a sequence, as an input into the classification layer. A softmax layer is added to output probabilistic label for the sentence,

$$z_i=softmax\left(W^t{h}_{CLS}\right),$$

where W^t is a matrix of the learnable parameters for linear layer, and Z_i is the probability vector assigned to each relation type.

Methodology

The intuition behind our approach, MERIT, stems from the assumption that neighbors of every data point (based on a distance-based metrics) are creating a local community that shares the same label. We hypothesize that if such a local community exists for a labeled data point x_i, we can transfer its label to its neighbors. We refer to the process of automatically assigning labels to neighbors of labeled data points as the weak labeling. As a result, we augment the training labels by utilizing expert supervision to generate high-quality weak labels, which can be further used in boosting the performance of any supervised RE model.

There are two key parameters in our approach that impact quality of weak labeling: 1) feature representation, and 2) distance threshold. Figure 1 illustrates the proposed approach. It depicts the local community in 2-dimensional space.

Figure 1.

Label augmentation through local community search. The data points annotated by X denote strong label, while data points highlighted with bold color are the weak labels.

There are several choices to embed RE data points as vectors such as by semantic/static-based embeddings^7,6. However, these representations are too general and are not explicitly designed to improve quality of weak labeling. Therefore, we aim to generate an embedding that captures a semantic representation explicitly aiming to improve quality of weak labeling. We apply a dependency parse tree on a sequence to extract the shortest dependency path (SDP) between two entities. As it has been shown in previous work^20,40, SDP can boost the performance of RE models and provide strong hints about the relation between entities.

Let T be a rooted parse tree corresponding to sequence S_i. Given a pair of entities (e_subj, e_obj), SDP can be defined as a minimum set of tokens that can be reached from esubj to eobj through the dependency tree T. For example, in the sentence “Chemical caused a dose-dependent reduction in plasma Gene”, the terms (caused, reduction) are the SDP tokens between entities (Chemical, Gene).

In order to integrate the SDP information into the feature space, we take the average embedding of SDP tokens and concatenate them with the entity representations based on SciBERT³

$$h_i=concat\left(\frac{1}{k}{\displaystyle {\sum }_{j=1,j\in SDP}^k{h}_j,h_{subj},h_{obj}}\right),$$

where h_i corresponds to the final representation for sentence S_i.

The second key decision in our framework is to define local communities for data point x_i according to the described feature representation. We consider a threshold θ, under which x_j cannot be within a local community of x_i. For optimized calculations of local community around each data point, we cluster the data using the k-means6 clustering algorithm. Since we just search for local communities around each data point, k-means clustering reduces the search space by an order of magnitude. Finally, we compute the cosine similarity as the distance metric to identify the local communities. The equation below demonstrates the corresponding function:

$$LocCommunity(x_i,x_j)=\{x_j,\begin{array}{c}ifdistance(x_i,x_j)\rangle \theta \\ \varnothing ,otherwise.\end{array}$$

Experiments

First, we describe the characteristics of the datasets that we used. Next, we explain the experimental design. Finally, we discuss the results.

Datasets: We evaluate our approach on three benchmark biomedical RE datasets. The characteristics of these datasets are shown in Table 1. We used the same train, development, and test splits during model development for ChemProt and DDI datasets as described in the previous work17,14. For the PPI dataset, we utilized AIMed corpus5 and performed 5-fold cross-validation due to the lack of standard train and test split. ChemProt and DDI tasks are multi-class classification problems. For ChemProt, there are 6 different relation types to capture the interaction between chemical and protein. In addition, the DDI contains 5 relations that correspond to the interaction between drugs. The labels for the DDI task are advice, effect, int, mechanism, and negative. PPI task is a binary classification problem to extract human protein interactions.

Table 1.

Statistics of the dataset used for label augmentation.

Dataset	Train	Validation	Test	#relations
ChemProt	18k	11.2k	15.7k	6
DDI	22k	5.5k	5.7k	5
PPI	5.2k	-	583	2

Evaluation metric and experiment design: We report standard precision (P), recall (R), and F1-score (F1) for binary classification, and micro-P, micro-R, and micro-F1 for multi-class classification. The evaluation metrics are as follows:

$$P=\frac{TP}{TP+FP},R=\frac{TP}{TP+FN},F1=\frac{2*P*R}{P+R},$$

where TP refers to True Positive, FP to False Positive, and FN to False Negative. For multi-class classification, the TP in micro-P and micro-R is the number of all positive class examples (we consider NA type as the negative label).

Micro-F1 calculates the total number of TPs, FNs, and FPs. We fine-tuned SciBERT for RE task. We set the maximum sequence length to 200, batch size to 16, distance similarity threshold to 0.9, learning rate to 2e-5, and epochs to 10. The remaining hyperparameters were the default values.

Results: We compared the effectiveness of our approach with the random sampling baseline (RS). We ran experiments for different labeling budget sizes [100, 200, 500] to acquire expert annotations. In both experiments, we randomly sampled from the corpus and trained the RE model on the collected samples. Compared to the RS baseline, MEIRT has the added benefit of leveraging the weak labels along with the strong labels. This experiment evaluated the effectiveness of MERIT as an extension to any supervised RE model. Figure 2 demonstrates a significant improvement of our approach over the RS baseline. Due to the random selection, we conducted all the experiments in 3 independent runs and report the average performance. The results show that in a scenario where limited annotated data were available (100), F1 score is higher by about 0.20 compared to the RS baseline. In addition, as we increased the hand-labeled data (up to 500), MERIT outperformed the baseline by up to 36% in the F1 score. This shows that the weak labels did not damage the performance when larger labeled datasets were available.

Figure 2.

Comparison of our approach with RS baseline on three benchmarks biomedical RE datasets.

Ablation Study: To further evaluate the effectiveness of each parameter in our approach, we performed an ablation study that measured the impact of distance threshold and feature representations. We conducted the experiment with budget of 200 with different threshold values in the range of {0.7, 0.75,0.80, 0.85, 0.9, 0.95} on the three datasets. As Figure 3 illustrates, as the threshold decreased from 0.95 to 0.7, more weak labels were added. This in turn led to more noise in the labels, hence damaging the performance when the threshold was too low. We found that 0.90 was the optimal threshold value for all three datasets, as a good tradeoff between number of weak labels and their quality. This parameter can be further optimized during the learning process, but this was left for future research.

Figure 3.

The impact of threshold on the final performance. We use 200 labeling budgets to perform this experiment.

We also explored the impact of several different embeddings for distance function calculation. We performed the following experiment on ChemProt dataset with a labeling budget of 200. We considered the following choices for the embedding:

• CLS (dimension L) is an aggregate representation of all the tokens in a sentence;

• ent-avg (L) is the average embedding of the entities in a sentence;

• ent-sdp-avg (L) is the average embedding of the entities and SDP tokens;

• ent-concat (2L) is concatenation of the embeddings of two entities in a sentence;

• ent-words-between (3L) is concatenation of the embeddings of the two entities along with the average representation of all the words between two entities;

• ent-concat-sdp-avg (3L) is a concatenation of the embeddings of the two entities along with the average representation of SDP tokens.

As shown in Figure 4, both ent-concat-sdp-avg and ent-sdp-avg representations outperformed non-sdp-based representations. This highlights the fact that SDP provides necessary information for the distance function to combine similar examples in computing the local community. The ent-concat-sdp-avg representation increased the final F1 score by 163% over CLS and 52% over ent-avg and ent-concat embeddings. In addition, the result for entity-words-in-between-based embedding shows that adding unnecessary context was not beneficial for training. However, it still outperformed the F1 score of the baseline CLS 27%. In addition, averaging the embedding performed slightly worse than concatenation (as illustrated in Figure 4).

Figure 4.

Comparison with different feature representations on ChemProt dataset with 200 labeling budgets.

Conclusion

Obtaining labeled data needed to train RE models in biomedical domain can be very expensive because it requires expert knowledge. Thus, RE models often need to be trained from relatively small labeled datasets. Despite the recent advances in Natural Language Processing (NLP) approaches for RE, training accurate RE models from small data is still an open challenge. In this paper, we proposed MERIT, a simple and effective approach for label augmentation that automatically increases the size of labeled data while introducing a moderate labeling noise. To represent an entity pair, MERIT finds the shortest dependency path between the entities and concatenates the average embedding of the words in the shortest path to the embedding of the entities. Our experiments on three benchmarks biomedical RE data sets showed that the proposed representation results in superior accuracy. They further show that the proposed weak labeling results in higher accuracy on all three RE datasets for a range of experimental conditions, including the varying number of labeled examples.