Predicting Transition Words Between Sentence for English and Spanish Medical Text

Abstract

Transition words add important information and are useful for increasing text comprehension for readers. Our goal is to automatically detect transition words in the medical domain. We introduce a new dataset for identifying transition words categorized into 16 different types with occurrences in adjacent sentence pairs in medical texts from English and Spanish Wikipedia (70K and 27K examples, respectively). We provide classification results using a feedforward neural network with word embedding features. Overall, we detect the need for a transition word with 78% accuracy in English and 84% in Spanish. For individual transition word categories, performance varies widely and is not related to either the number of training examples or the number of transition words in the category. The best accuracy in English was for Examplification words (82%) and in Spanish for Contrast words (96%).

Introduction

Even with the increased use of video and multi-media, text remains an important tool for patient education¹. However, creating well-written, understandable text is difficult and requires appropriate vocabulary use, correct grammar, and good writing style. One important component for well-written text is the use of transition words to help the reader follow the flow of ideas.

Transition words (also referred to as connectives) are words or phrases that connect linguistic units in a text². For example, single term transition words include “however, then, also, but, thus, so, therefore, and, still, furthermore, rather, or, meanwhile”, and multi-term examples include “even so, in conclusion, other than, as a result, in addition, in the meantime”. In this work, we will use the phrase “transition word” to refer to both single and multi-term transition words. Transition words are categorized according to their role, e.g., “initiating a topic, adversative, comparison, results”³. Several different categorizations have been proposed depending on the subfield or goal⁴. In this work, we use 16 categories paralleled across English and Spanish. Table 1 gives a list of the different category types and including examples in English.

Table 1:

The number of transition words for each of the 16 categories in English and Spanish along with example transition words in English.

	Number of words
Category	English	Spanish	Example English connector words in category
Agreement	35	38	again, in the first place, not only, similarly, as a matter of fact
Cause	36	37	granted that, as long as, so long as, on condition that
Conclusion &Consequence	53	71	in conclusion, in brief, in summary, to sum up, all in all
Contrast	11	11	however, in contrast, nevertheless, yet, on the other hand
Emphasis	4	5	as a matter of fact, in fact, actually, indeed
Exemplification	55	60	for example, for instance, to illustrate, notably, in general
Explanation	5	5	that is to say, that is, namely, in other words, put differently
Importance & Order	6	8	most importantly, primarily, above all, most significantly
Linking	7	9	as for, with respect to, regarding, with regard to, as far as
Location	38	41	in the middle, to the left, alongside, behind, across,
Opposition	38	42	different from, of course, but, even so, then again, while
Particularization	4	7	in particular, particularly, more specifically, specifically
Results	5	7	as a result, as a consequence, therefore, thus, accordingly
Sequence	22	17	first, first of all, to begin with, for one thing, moreover
Similarity	5	5	likewise, similarly, correspondingly, in the same way, also
Time	63	60	at first, then, afterwards, later, to begin with, meanwhile
Total	387	423

Transitions words play an important role in text flow, especially for introducing a new topic for a reader to learn about. The words provide logical connections, help convey the structure of an argument, show information flow, improve coherence and, because of these, affect reader comprehension^4-7. Additionally, their use can be an indicator of text type. Several studies have found that their frequency differs depending on the type of text, such as news, conversation, fiction, or academic prose, with the latter containing the most transition words^2,3.

In this paper, we focus on the task of predicting transition words in medical text. In general, transition words can occur at the beginning of a sentence or embedded in a sentence to link clauses. We examine the former, i.e. predicting transition words at the beginning of a sentence. For example, given the pair of sentences

After the second year of growth hormone therapy, beneficial bone growth decreases. ____ GH therapy is not a satisfactory long term treatment.

we aim to predict that a transition word should occur at the beginning of the second sentence where the underscore is, specifically, a Results transition category word. In the text that this example was derived from, ‘therefore’ was used, however, other transition words from the same category could have also been used, e.g. ‘thus’ or ‘as a result’. We pose two variants of this prediction task. First, predict if any transition word should be used at the beginning of a sentence. Second, predict if a particular category of transition word should be used.

Previous work has shown how transition words signal relationships between and among the idea hierarchy in text using manual approaches^8,9. We view the problem as a supervised learning problem. We introduce a new dataset for transition word prediction in medical texts with data in two languages, English and Spanish. Using this dataset, we establish a baseline using a feedforward neural network trained using backpropogation and provide initial analyses on the difficulty of predicting the individual categories. We conclude with a number of possible future directions.

Background

We focus on transition words in the context of text simplification of medical text. The goal of text simplification is to transform texts into simpler variants that are easier to understand by a broader audience without losing content. Our project’s goal is to accomplish this transformation by providing a text editor that guides writers in the simplification process. The tool is developed using data-driven algorithms that are trained on large sets of text and evaluated in user studies to show evidence-based outcomes. Since transition words play an important role in text comprehension, e.g., it has been shown that not understanding them limits text comprehension¹⁰, making the transition between sentences explicit may assist readers with text comprehension since it helps clarify the logical connections in a text^11-14.

Simplifying text to increase reading comprehension for medical and healthcare information is an important task since it may help increase health literacy of patients and their family. Each year, chronic diseases afflict more people and treatments are becoming increasingly more complex, often requiring the need for participatory medicine¹⁵ where patients take an active role in their healthcare. However, the simplification task is difficult because of the complexity of the topics; the lack of large parallel corpora, particularly in the medical domain; and the need for simplifications that do not omit important information or introduce errors.

Historically, readability formulas were popular for guiding simplification^16,17 and a large, if somewhat older, non-computational group of projects focused on applying readability formulas such as the Flesch-Kincaid grade level formula¹⁸. The formulas are easily accessible since they are available in common text editing software (e.g., Microsoft Word), online (e.g., http://www.readabilityformulas.com), or through commercial avenues. This research trend is declining since using readability formulas has not been successful. Studies showing a positive effect of their use on reader comprehension are rare. Readability formulas do not identify what aspects of a text are difficult, do not provide writing alternatives, and use features that are rudimentary, e.g., equate word length with difficulty. These formulas do not incorporate current knowledge about the reading process, have limited scientific basis, and are not helpful as writing guidelines¹⁹. They also do not leverage modern resources that are currently available, e.g., large corpora with term frequencies to suggest terms with high familiarity as a replacement for difficult terms.

In recent years, computational approaches to improving the readability and understandability of health-related text have started to dominate. Several approaches have been tested as part of rule-based simplification algorithms. For example, when simplifying single words, term frequency (often measured by the Google Web Corpus²⁰) is used to replaceterms with easier synonyms²¹. Several lexical simplification approaches have been suggested that use either hand-crafted rewrite rules²² or learned rules^23,24.

In addition to rule-based simplification, there is also a health research stream using machine learning approaches. The algorithms vary from traditional machine learning to evaluate different features^25,26 to newer deep learning neural networks trained on parallel corpora²⁷. Evaluations vary, including descriptive comments on outcomes²², subjective evaluations by experts or laypersons²⁸, and user studies measuring impact through reading comprehension tasks²⁹.

While good progress has been made recently on text simplification algorithms, particularly with data-driven approaches, the output is still not of good enough quality to be used in real-world applications. Instead, for our project, we focus on human assisted simplification by using automated approaches to help guide content creators in simplifying medical text. For transition words, we plan to identify possible locations in the text where transition words could be used and suggest the category of the transition word, along with a ranked set of candidate transition word options. This paper is the first step towards this goal by evaluating the performance of supervised approaches to the transition word prediction problem.

Dataset creation

We created two datasets for the transition word prediction task: one in English and one in Spanish. English and Spanish are the two most commonly spoken languages in the U.S. and examining the task in two languages allowed us to compare how prediction performance differs across languages. Both datasets were based on medical Wikipedia articles using a large list of transition words. B oth the English and Spanish datasets are publicly available online along with the transition word lists used to create them.¹

Corpora

Since the frequency of transition words differs depending on the topic of the corpus³ it is important to train on a representative corpus. We aim to simplify health educational text and created a corpus from general information on common diseases. We downloaded all articles in English and Spanish Wikipedia tagged with the category “Diseases_—and_—disorders”. The English corpus contains 636 articles for a total of 1.3M words. The Spanish corpus contained 493 articles for a total of 637K words. We used these two corpora to derive our transition word examples.

English Dataset

To generate our transition word prediction datasets, we first identified a list of transition words in English from different online educational and writing sites listing transition words. We combined information and created 16 categories of transition words containing 387 transition words. Table 1 gives an overview of the categories with the number of transition words in both languages in the category along with example English words for each category.The categories range in size from as large as 63 words (Time) to as small as 4 (Emphasis and Particularization).

To generate the classification examples, we considered every pair of adjacent sentences within the corpus. If the second sentence in a pair contained one of the transition words, then it was marked as a positive example, otherwise, it was marked as a negative example, i.e. when there was not a transition word between the two sentences. This results in a dataset of positive examples, <s₁,s₂, c>,and negative examples, <s₁,s₂>, where s₁. is a sentence, s₂ is the sentence following s₁, and c is one of the connectors.

Table 2 shows the total number of examples extracted from the corpus. Overall, just under a quarter of the sentence pairs had connectors in them and were marked as positive. We then further processed the dataset to ensure that each sentence in each example contained at least one noun. This led to the removal of a few inappropriate pairs and a final set of more than 70K examples.

Table 2:

Dataset Description

	English	Spanish
Wikipedia Corpus	636	493
Sentence Pairs
Raw Dataset
Positive Example:	16,315	7,178
Negative Examples:	53,938	20,466
Total	70,253	27,644
Prepared Dataset
Positive Example:	16,285	7,140
Negative Examples:	53,859	20,290
Total	70,144	27,430

Spanish Dataset

Guided by the English transition word list, we had a native, expert Spanish speaker generate a comparable list for Spanish. In most cases, there was a corresponding phrase in Spanish, however, there were a few exceptions. In some cases, this resulted in a reduction of the number of words, for example when there was not a good equivalent in Spanish or if several English transition words corresponded to the same one in Spanish. In other cases, there were multiple equivalent Spanish words for a single English word. The final Spanish transition word list contained the same 16 categories with 423 transition words, slightly more than the English equivalent.

Similar to the English dataset, we generated the dataset in Spanish by considering all consecutive sentences in our corpus and creating positive, i.e., those containing a transition word in the first position of the second sentence, and negative examples. Because there are fewer texts available in Spanish Wikipedia covering the same topics, our Spanish corpus is smaller as is the number of positive and negative examples. The sentence pairs were then further preprocessed in a similar manner as English to ensure each sentence contained at least one noun. The resulting dataset contains approximately 27K sentence pairs (Table 2).

Predicting Transition Words

Given a pair of adjacent sentences, the goal is to predict whether a transition word should be used between the sentences, specifically, at the beginning of the second sentence. We view this problem as a supervised classification problem and use a feedforward neural network trained on examples of sentence pairs to make both the binary prediction of whether a transition word should be used or not, as well as a more fine-grained prediction for each of the categories of transition words.

Features

Transition words are used to indicate how information in the first sentence relates to information in the second sentence. To capture this, we extracted features based on the nouns in the two sentences, which are often the content-bearing words in the sentence. We selected the first five nouns in each of the sentences (s₁ and s₂) using the Stanford CoreNLP toolkit³⁰. For each noun, to generalize beyond its lexical form, we then extracted a 300-dimension word embedding. The embeddings were acquired from large publicly available sets using the FastText pre-trained model.² We chose this particularly embedding model since it is publicly available, it is based on a large corpus, and is available in both English and Spanish.

Based on these word embeddings, we extracted 3,601 features:

• The 300-dimension word embedding for each noun (300 for each of the 5 nouns in s₁ and the 5 nouns in s₂ for a total of 3,000 features). If a sentence contained less than five nouns, those features were given an empty value.

• The average vector of the noun word embeddings per sentence (300 features for s₁ and 300 features for s₂).

• The cosine similarity between the average word embeddings of the sentence pairs (1 feature).

Classifier

Since the task is new, we chose to use a classifier with well-known properties and used a feedforward neural network. We used the R RSNNS package with backpropagation learning, a 0.1 learning rate, and 1 hidden layer. To try and avoid overfitting, we tuned the number of hidden nodes and the number of training iterations on a development set. To pick the number of hidden nodes, we trained the network using 5, 10, and 50 hidden nodes and picked the best performing model based on the accuracy on the development set. We picked the number of training iterations as the point where the standard squared error started to increase on the development set. Parameter selection was done per task.

Experiments

We examined two prediction tasks in both English and Spanish. The first task is to predict whether any connector should be used (Any-vs-None), i.e. the binary prediction between those examples that have a connector and those that don’t. The second task is, for each connector type, to predict whether that connector type should occur or no connector should occur, i.e. 16 separate binary prediction tasks.

Experimental Setup

To create a more balanced dataset, for each task, we downsampled the number of negative examples so that the proportion of examples was 1/3rd positive and 2/3rds negative, resulting in a majority baseline of 66.67% by always predicting no connector. For the Any-vs-None task, this resulted in 16,285 positive and 32,570 negative examples for the English task and 7,140 positive and 14,280 negative examples for the Spanish task. For the per category prediction task, we did not classify any category that had fewer than 100 positive examples. Tables 4 and 5 show the number of examples for each of the categories in English and Spanish, respectively. There were five categories in English and five in Spanish with fewer than 100 examples that were ignored.

Table 4:

English transition words classification per category (Ex = Example count, A = Accuracy, P = Precision and R = Recall).

Transition	Ex.	A	P	R
Word	(Ct)	(%)	(%)	(%)
Agreement	2,055	69.89	60.11	31.24
Cause	2,885	67.11	52.65	18.82
Conc. & Cons.	1,377	73.64	67.94	40.81
Contrast	1,202	67.83	57.13	20.46
Emphasis	52
Exemplification	671	81.52	78.85	62.28
Explanation	20
Imp. & Order	9
Linking	18
Location	1,984	68.16	56.04	29.13
Opposition	2,488	66.53	55.09	18.85
Particularization	74
Results	301	72.55	63.74	43.87
Sequence	703	74.77	66.94	49.48
Similarity	224	64.29	34.69	11.60
Time	2,222	67.18	54.75	19.67

Table 5:

Spanish transition words classification per category (Ex = Example count, A = Accuracy, P = Precision and R = Recall)

Transition	Ex.	A	P	R
Word	(Ct)	(%)	(%)	(%)
Agreement	867	63.94	44.16	26.76
Cause	1,647	68.73	57.55	31.33
Conc. & Cons..	360	67.22	52.95	31.39
Contrast	524	96.50	95.24	94.27
Emphasis	42
Exemplification	250	80.40	72.20	67.20
Explanation	22
Imp. & Order	24
Linking	47
Location	350	66.00	51.59	21.43
Opposition	983	84.91	85.58	66.93
Particularization	19
Results	78
Sequence	650	62.31	43.32	20.62
Similarity	404	64.60	41.94	17.05
Time	865	73.64	63.86	54.34

We used 10-fold cross validation for all experiments and randomly selected one fold of the training portion as the development set for parameter tuning during training. We evaluated the models using accuracy, precision, and recall.

Any-vs-None Classification

Table 3 shows the results for the Any-vs-None classification task. In both English and Spanish, the accuracy is well above the baseline of 66.67%. This is particularly encouraging given how many different transition words there are in this task: sentence pairs that benefit from transition words do have marked differences from those that do not use them. Across all metrics, performance was better in Spanish than in English, even though the Spanish dataset had fewer examples and the number of individual transition words was larger. In both languages, the precision and recall numbers were similar indicating that there wasn’t a strong bias of the classifier towards either the positive or negative class. Given how varied the transition words are and the

Table 3:

Any-vs-None classification results for the two languages.

	Accuracy(%)	Precision(%)	Recall(%)
English	77.97	66.21	69.27
Spanish	83.57	75.09	76.29

Per Category Classification

Tables 4 and 5 show the results for individual transition categories for English and Spanish, respectively.

Accuracy: Overall, the accuracy for most categories is lower than for the All-vs-None results, with the exception of the Exemplification category in English and the Contrast and Oppositions categories in Spanish. In a few categories, the classifier was unable to perform better than the majority baseline (two in English and four in Spanish). The per category transition prediction problem is generally much more difficult than just predicting whether a transition word should be used or not.

To try and understand the cause of the variation in performance per category, we calculated a one-tailed Pearson correlation between accuracy and the number of examples, the number of transition words in a category, and the number of examples per transition category. There were no significant correlations found for accuracy either in English or Spanish:

• The number of examples is not related to accuracy, i.e., having a larger data set (since we keep the ratio between positive and negative examples identical) did not benefit the classifier.

• The number of possible transition words was not correlated to accuracy, i.e., having a wider variety of positive examples did not correlate with accuracy.

• The average number of examples per transition category (since the number of transition words varies for the different categories) did not correlate with accuracy, i.e., having more examples per individual transition category did not matter.

Precision: Generally, precision per category was around 50-60% for many of the categories in both languages, though there are some exceptions, particularly the best performing categories in the two languages. The lowest precision was the same category in both English (35%) and Spanish (41%), Similarity.

Similar to accuracy, we calculated the correlation between precision and the same three dataset metrics for both English and Spanish. In this case, we found one significant positive correlation between precision and the number of different transitions words in a category (r = 0.555, p = 0.038) for the English data, i.e. precision was better for transition list categories with more words in them. However, taking Bonferroni correction for performing multiple statistical tests, the correlation would not be considered significant. There were no significant correlations for the Spanish data.

Recall: Overall, recall was the lowest of the three evaluation metrics. This hints at the complexity of the problem: identifying all of the occurrences of transition word usage is difficult. English, in particular, had several categories with very poor recall scores: Similarity (12%), Cause (19%), Contract (20%), Time (20%) and Opposition (19%). Similar to precision, the lowest performing category for recall in both languages was Similarity.

There were no significant correlations found for English or Spanish between recall and the number of examples, number of transitions words, or number of examples per transition word.

Discussion

In this paper, we have introduced the new problem of predicting transition word usage between adjacent sentence pairs. We created a dataset consisting of medical text examples divided into 16 categories of transition words in both English and Spanish, which is publicly available. Using this dataset, we examined two classification tasks: predicting whether any transition word should be used and predicting the category of the transition word. As a first approach, we used a feedforward neural network with word embedding features trained using backpropogation. We found strong results for the general transition word prediction task (English accuracy of 78% and Spanish of 84%, over a majority baseline of 67%), but more sporadic results for predicting individual connectors with accuracies as high as 97% and as low as 64%.

There are a number of possible directions for future research using this dataset. We viewed the prediction of each category as an independent binary prediction task. This allowed us to understand the difficulty of the transition word prediction task per category and to compare across English and Spanish. For practical applications, this problem should be viewed as a multi-class classification problem where the goal is to predict which of the 16 categories should be used, combined with the general prediction task (either as a separate step or as part of the multi-class prediction problem). Even given the category, there is still the question of which transition word to use. For use in a simplification tool, this then becomes a ranking problem among the possible transition words within a category.As a first pass to aid analysis, we made some simplifying assumptions in the classification setup that should also be explored. First, we downsampled the negative examples to create a balanced dataset across all examples. For real application, this imbalance will have to be handled appropriately by the classifier. In particular, more evaluation needs to be done to examine how well the classifiers perform when applied to real texts where predictions must be made between every pair of sentences. Second, we ignored predicting on five of the transition categories with less than 100 examples. More data would likely fix this problem, particularly if the dataset were expanded beyond medical-related text.

The per category prediction results were low for many categories. Correlation analysis showed that this was the case regardless of the number of examples available or the number of transition words in the category. Recall, in particular, was generally low across all categories, suggesting that more work needs to be done to identify features that can better differentiate the broad usage of connector words. As an example, Exemplification had consistently strong results. We speculate that the performance was better than other categories because of the nature of the notion of exemplification which is perhaps less abstract than others (i.e., cause) and followed by more predictable and more local structure (an enumeration, rephrasing or repetition) than others. Other categories, however, may require more information for prediction.

Finally, in linguistics and education, the importance of transition words is well established and recognized, however, they have received little attention in the medical informatics and natural language processing communities and we believe they can be useful for a variety of tasks. For example, they can help with summarization by hinting at introductions, ordering, contrast and conclusions. For fact checking and information extraction tasks they can be used to identify and add structure to unstructured information sources. In general, transition words provide guidance about the flow of ideas and can be leveraged by many discourse-level tasks. We hope that the dataset, including the categorized list of transition words, along with the initial classification attempts helps motivate such uses.