Automatic Generation of Conditional Diagnostic Guidelines

Abstract

The diagnostic workup for many diseases can be extraordinarily nuanced, and as such reference material text often contains extensive information regarding when it is appropriate to have a patient undergo a given procedure. In this work we employ a three task pipeline for the extraction of statements indicating the conditions under which a procedure should be performed, given a suspected diagnosis. First, we identify each instance in the text where a procedure is being recommended. Next we examine the context around these recommendations to extract conditional statements that dictate the conditions under which the recommendation holds. Finally, corefering recommendations across the document are linked to produce a full recommendation summary. Results indicate that each underlying task can be performed with above baseline performance, and the output can be used to produce concise recommendation summaries.

Introduction

Understanding which diagnostic procedure should be performed on a patient when presented with a set of symptoms or a suspected diagnosis is a difficult task that is at the heart of the diagnostic process. Often, the details surrounding when a test is appropriate are extraordinarily nuanced, making this a potential area for clinician error. As errors in this regard can put unnecessary burden on the patient and lead to higher costs, there is a potential role for automated systems or curated resources to help guide this clinical decision making.

Reference material text, such as the disease workup pages found on the Medscape reference website (http://emedicine.medscape.com/), provide clinicians with an overview of the recommended diagnostic course for a disease. Although these resources can be invaluable for clinicians in training or for casual reference, a higher-level summary of which procedures are appropriate for the diagnosis of a given disease would be beneficial for quick reference or for machine interpretation. Unfortunately, diagnostic recommendation involves more than simply declaring that a given procedure is appropriate in all cases, leading many existing resources to either give extensive detail or summarize generally over broad disease categories.

In practice, diagnostic recommendation is about understanding not just if a procedure is a necessary part of the diagnostic process, but when. For instance, consider the following passage describing the use of echocardiography for the diagnosis of acute coronary syndrome:

Echocardiograms play an important role in the setting of ACS. Regional wall-motion abnormalities can be identified with this modality, and echocardiograms are especially helpful if the diagnosis is questionable. An echocardiogram can also help in defining the extent of an infarction and in assessing overall function of the left and right ventricles.

This passage not only gives an indication that echocardiography is an important diagnostic modality, but it also outlines specific circumstances under which it is helpful (if the diagnosis is questionable, in defining the extent of an infarction, in assessing overall function of the left and right ventricles). In this work we explore the automatic extraction of these conditional recommendation statements from reference material text, towards the generation of diagnostic guideline summary tables. We model diagnostic guideline generation as a series of three tasks: 1) mention- level recommendation detection, 2) conditional relation extraction, and 3) across sentence entity linking. Our results show that we can perform each task with above baseline performance, and that their output can be combined to produce concise procedure recommendation summary charts.

Related Work

While automatic extraction of conditional diagnostic guidelines – to the best of our knowledge – is a new task, its subtasks are closely related to a set of well-defined natural language processing tasks that have been applied in clinical settings, including sentiment analysis, event extraction, coreference resolution/entity linking, and text summarization.

Sentiment analysis. The recommendation identification task that we examine in this work is similar to the sentiment analysis task. Sentiment analysis, also called opinion mining, aims to determine an author’s attitude towards a specific topic or the overall contextual polarity of a document. There has been a lot of existing work on mining reviews or other expressions of opinion on the Web¹. Particularly, in the medical informatics domain, Wallace et al.² has conducted a large-scale sentiment analysis of online physician reviews using a statistical model guided by a small amount of annotated data. The three-way recommendation classification problem proposed in this paper is an ensemble of two typical sentiment analysis tasks: polarity classification and subjectivity identification. The former is to determine whether the expressed opinion is positive or negative (recommended vs. not recommended), whereas the latter is to determine whether a mention contains subjective information at all (recommended/not recommended vs. unspecified) and which part of the text is subjective¹. Existing approaches to sentiment analysis include those based on opinion lexicons (e.g. ontologies, semantic networks), statistical models (e.g. support vector machines, latent semantic analysis), and combinations of the two³. In this work we present a hybrid approach to recommendation identification, where a set of recommendation keywords (lexicon) is derived from development data and used in conjunction with other word, concept, and NLP-derived features for supervised classification.

Event extraction. Event extraction aims to extract information about entities and the role they play in an event. The second task in our pipeline, conditional relation extraction, can be viewed as an event extraction task, where we aim to determine whether there is a conditional relation between a (recommended) procedure and a certain constraint such as a patient’s condition. There has been a substantial amount of work on event extraction published in the past decade. A typical example is molecular event extraction from biomedical literature⁴. Another example in the medical informatics domain is the recent work of Botsis et al.⁵ on vaccine adverse event extraction from vaccine safety reports. To extract events, we need to identify the trigger/anchor word that signifies an event, and to figure out the role of each theme argument involved in the event. The presence of a candidate trigger may not necessarily suggest a relation between entities, and the problem becomes challenging when multiple potential triggers appear in the same text unit⁶. Most existing approaches to event extraction decompose the task to independent classifications of events and arguments. For instance, the winner of the BioNLP’09 shared task on event extraction⁷ first extracts a list of candidate triggers and then determines for each pair of triggers and entities whether one is a theme of the other (with around 50% F-score).

Coreference resolution. The entity grouping task we explore has parallels with several related but subtly different tasks, such as coreference resolution, entity linking, and normalization. Coreference resolution determines whether two concepts are linked by an equivalence relation. For example, in the following two sentences ‘ Color Doppler valve analysis during transesophageal echocardiography (TEE) can be used to …‘ and ‘In patients with poor transthoracic echocardiographic images, TEE may be used to …‘, the two mentions of procedures are equivalent as they refer to the same entity. A number of systems have been developed for coreference resolution in clinical texts^8,9, including those based on handcrafted rules (e.g. keywords, regular expressions, spelling correction, abbreviation expansion), supervised learning (e.g. maximum entropy classifier, support vector machines, hidden Markov model), and a mix of the two, with best-reported F-score of around 70%.

Entity linking, the task of linking each mention found in the text to a structured database such as UMLS, is done as part of UMLS concept extraction and as such is a well-studied task within the medical informatics literature^10,11. Because linking mentions back to a database implicitly groups them by the database categories, entity linking is similar to both the coreference task and the entity grouping task examined here. However, unlike these tasks, entity linking is constrained by the structure of the database, potentially leading to grouping concepts at a level inappropriate for the given task¹².

Text summarization. The production of conditional diagnostic guidelines across a document is one form of a text summarization task. The majority of related work in the medical informatics domain focuses on extractive summarization of electronic health records¹³, where summaries are created by borrowing phrases or sentences from the original input text. In clinical summarization, normalization of words to (groups of) concepts has only recently been investigated, primarily based on well-defined ontologies¹⁴. Our recommendation extraction pipeline built on top of clinical concept extraction sheds interesting light on identifying and aggregating similar information at a more abstract level^15,16.

Material and Methods

We model the production of conditional diagnostic guidelines as a pipeline of three tasks. First, each mention of a diagnostic procedure in the text is labeled as either suggesting that the procedure should be performed, suggesting that it should not be performed, or giving no recommendation one way or another. Second, we link each those mentions labeled as positive or negative recommendations with conditional statements explaining the scope of this diagnostic recommendation. Finally, we perform a coreference task to link mentions from different sentences, producing a set of document-level recommendation statements for a given diagnostic procedure.

Data. The Medscape website provides disease-centered documents containing explanations and guidelines for the presentation, diagnosis, and treatment of these diseases. To build our dataset, we extracted the text from a set of 33 Medscape diagnostic workup pages describing cardiac ailments. Annotation was then performed to identify procedure mentions in the text and to label whether the given mention is recommending that the procedure be performed. To ease the annotation process, mentions of procedures were initially detected by simple string matching against all concepts with procedure-related semantic types (e.g., “laboratory procedure”) from the Unified Medical Language System (UMLS). Although this process provided an initial set of mentions for annotations and annotators were asked to use this as a guideline for which procedures were considered, annotators had the option to correct cases where the initial mention identification was incorrect. As a second aspect of annotation, procedure mentions were linked with conditional statements that dictated when the procedure should be undertaken. Conditional statements were represented by the head of its syntactic phrase, usually a preposition. Finally, annotators grouped together all mentions in the document that referred to the same procedure. Annotators has linguistic but not medical background. Although annotators were not medical experts, practicing clinicians were on hand to clarify any uncertainty. The brat tool was used to produce the annotations¹⁷.

Figure 1 shows an example of this annotation style. The annotations in the figure combine in different ways to produce a gold standard for our three tasks. For the recommendation task, each procedure (e.g., enhanced external counterpulsation) is given a recommendation label of either unknown (no positive or negative recommendation stated in the sentence), recommended, or not recommended. For the conditional task, each trigger-procedure pair in the sentence is a candidate conditional statement, with annotators linking together true conditionals (e.g., in-EECP). No explicit example of the entity linking task is given in Figure 1. However, since the linking task compares all procedures in the document, the two procedures in Figure 1 would be checked to see if they refer to the same procedure in the given context (in this case they do not).

Figure 1.

Annotation of a single sentence with a conditional recommendation.

Overall, a total of 926 procedure mentions appeared in our dataset over a span of 2382 total sentences. Of those, 295 were recommendation statements. A total of 1103 mention-trigger pairs were present, 174 of which were marked as true conditional statements. Additionally, a small set of unannotated data from the same domain was examined for the development of features and algorithms. This data was used for development only and was not considered in training or evaluation.

Recommendation Identification. Because we wish to associate procedures with conditional recommendation statements, we perform the recommendation annotation task for each individual mention of a procedure in the text. This allows us to predict that a procedure is recommended in some instances while not recommended in others, a distinction that would be lost in document-level recommendation classification. We model the recommendation identification task as a three way (Recommended, Not Recommended, Unspecified) mention-level supervised classification task. To learn a classification model we derive features from the surrounding context in three broad categories: concept features, word features, and NLP-derived features.

Concept Features. To extract concept-based features, we first attempt to link each procedure to a UMLS concept. To do so, we use a proprietary concept extraction procedure based on identifying longest common subsequence matches between the text and candidate UMLS concepts. Although standard concept extraction procedures such as MetaMap and cTAKES exist, we chose our concept extraction procedure because internal tests suggested that it outperformed the freely available alternatives^18–20. Once concepts are identified, several features are extracted. For the concepts associated with the procedure of interest, the UMLS concept unique identifier (CUI) and its semantic and category names are retained as features. Additionally, features are produced from the CUIs of all other concepts present in the sentence. Features from concepts in the sentence were differentiated from those associated with the procedure concept. Finally, a Boolean feature captured whether or not any of the other concepts in the sentence were also procedures.

Word Features. Following previous text classification work, several feature sources were based simply on the presence of certain words in the sentence. First, a small set of both positive and negative recommendation keywords (e.g., recommend, critical, prognostic) was derived by examining a held out set of development data. One feature indicated whether any of the positive recommendation keywords were present, while another indicated the presence of negative keywords. A Boolean feature indicated whether the surrounding sentence mentioned the disease that was the theme of the document. The three words before and after the mention text were captured, as well as their relative position. Unigram and bigram features were extracted for all words in the surrounding sentence.

NLP-derived features. The final set of features required more linguistic interpretation. The Stanford CoreNLP toolkit (Version 3.4.1)²¹ was used to obtain a syntactic parse of the sentence and to assign part-of-speech labels. Using this information, the main verb of the sentence and any adjectives modifying the mentioned procedure were extracted as features. Positional features were extracted that indicated whether the procedure was in a list or conjunction with other procedures. Finally, the dependency parse of the sentence was used to determine if the mention was governed by a negation modifier.

Conditional Identification. Given a procedure mention with a positive or negative recommendation, the conditional identification task attempts to extract any part of the surrounding sentence that puts constraints on that recommendation. In order to obtain a set of candidate conditional snippets, we extract a series of trigger words that could possibly be the syntactic head of a conditional phrase. A list of possible trigger terms was generated by starting with all prepositions found in the document collection then examining development data to supplement and refine this list. Once a set of candidate triggers is identified, the task is to make a binary decision for each mention-trigger pair as to whether the phrase headed by the trigger word is a conditional statement modifying the procedure recommendation. As with recommendation identification, we model this task under a supervised learning framework.

Since many of the features used for recommendation identification potentially have discriminative value for conditional identification, the conditional identification feature vector starts by incorporating modifications of these features. This feature set is then expanded to include features capturing the relationship between the conditional phrase and the procedure and those based on the conditional phrase itself. A summary of the feature set is shown in Table 1.

Table 1.

Conditional extraction feature set.

Feature Type	Description	Features
Concept Features	Features based on UMLS concepts extracted from the text.	procedure CUI, semantic name, category name; Concepts in sentence; Concepts in conditional, other procedure concepts
Word Features	Features based on the words in the sentence	keywords; disease in sentence; words near procedure; words in conditional; trigger word; unigrams; bigrams; normalized number of words in conditional
NLP-based features	Features based on linguistic analysis	verbs; modifying adjectives; positional features; trigger governed by other trigger words
Relationship features	Features capturing the relationship between the procedure and trigger	procedure is in the conditional; normalized distance from trigger to procedure; procedure and trigger have same syntactic parent

Entity Grouping. In order to turn a set of conditional recommendation statements into a recommendation chart, mentions that refer to the same conceptual entity must be linked together. While linking each mention to an entry in an existing knowledge base such as UMLS implicitly groups them, this linking is often imperfect due to differences in word form, part of speech, or granularity between what is given in the knowledge base and what is expected in the document context. For instance, while the terms echocardiogram and echocardiography each have their own UMLS concept, they will often be used interchangeably in procedure recommendation statements. Similarly, if the term echocardiography appears after the term transthoracic echocardiography in the text they may be used to refer to the same procedure, even though normally one is a subset of the other. Given this, linking to UMLS is an insufficient method for our entity grouping task, which could be seen as more similar to traditional coreference resolution^9,22. However, unlike most coreference tasks that consider pronominal instances, in this work we are only concerned with grouping explicit (non-pronominal) mentions in the text.

We developed a new algorithm that iteratively groups mentions of procedures into clusters based on their string similarity. Given two mentions of procedures: S =< s₁s₂ …s_M > of M words, and T =< t₁t₂… t_N > of N words, we define their longest common prefix as LCF(S, T) =< p₁p₂ … p_L >, where L is the largest subset of words from S that found a partial match in T, and p_i is a partial match of a word s_i ∈ S to a word in T. A word s_i in S is said to partially match a word t_j in T, if the length of their longest common prefix p_i is above the threshold: $\frac{\left|p_i\right|}{\max \left\{\left|s_l\right|\left|t_j\right|\right\}}\ge \tau$. When = 1.0, this reduces to finding exact matches to words of S. We used relatively large τ (e.g. 0.9) for recognizing words with different forms but sharing the same root and meaning. Similar to the longest common subsequence matching problem, LCF cam be computed using dynamic programming in quadratic time relative to the length of the sequences. Two procedure mentions are linked together if their LCF equals either of the two, making it possible to group coarse- and fine-grained concepts such as echocardiography and transthoracic echocardiography under the same category.

Results

Although we examine the performance of each task separately, the training and evaluation of the recommendation and conditional identification tasks follow the same basic framework. In each case, the LIBSVM package (Version 3.17)²³ was used to train a Support Vector Machine (SVM) classifier using a linear kernel. Other kernel types were examined as well, with no impact on performance. Performance was then assessed via leave-one-out cross validation. Conversely, since our entity grouping method is based on predefined rules, no model training was required, and evaluation was done directly on the entire dataset.

The results for the recommendation identification task are given in Table 2. As to our knowledge no other systems for mention-level procedure recommendation exist, we have no existing baselines for comparison. We thus compare to a bag-of-words baseline where only the unigrams and bigrams in the sentence are considered as features. As shown in the table, the proposed system is able to significantly outperform the baseline when using the entire feature set. To get a better understanding of the contribution of each features source, a feature ablation study was also conducted. As shown, removing any of the feature sources causes a loss in performance, suggesting that each feature type played a role in the prediction. However, removing NLP-based or concept-based features caused relatively modest reductions in performance. Conversely, the word-based features were shown to be most critical, as without them the classifier yielded worse than baseline performance.

Table 2.

Results of the recommendation identification task.

	Precision	Recall	F-measure
Baseline	0.649	0.575	0.610
Our system	0.718	0.675	0.696
Minus Word-based	0.559	0.585	0.571
Minus Concept-based	0.690	0.642	0.665
Minus NLP-based	0.713	0.656	0.683

For the conditional identification task, we present two views of the performance. To understand how well the classifier performs the task in a vacuum, we present the results of the task (and an associated baseline) when ground truth recommendation labels are known. However, since in a true usage scenario it is necessary to perform the recommendation identification task first, we also present overall performance numbers for a pipelined system that performs conditional identification on the output of the recommendation identification task. Table 3 gives the results of the conditional recommendation task. Because a bag-of-words baseline would not give the classifier any information relating the procedure to the conditional, we instead chose a baseline trained on all of the word-based features given in Table 1, giving it a wider range of information. In the case where the ground truth labels are known, the learned system significantly outperforms the baseline. Similarly, the pipelined approach also outperforms the baseline, even though performing the tasks in sequence propagates the errors of the recommendation task and significantly hurts performance.

Table 3.

Results of the conditional identification task.

	Precision	Recall	F-measure
Baseline	0.509	0.477	0.493
Conditional Identification Only	0.709	0.644	0.675
Pipelined System	0.581	0.518	0.547

The goal of the entity grouping task is to assign each mention to one of several non-overlapping sets, making evaluation less straightforward. This is a problem common to general coreference resolution approaches, where several evaluation metrics have been proposed. In this work we adopt one of the more frequently applied evaluation metrics, b-cubed score²⁴. B-cubed score calculates a local precision and recall score for each mention based on the overlap between its predicted set and the ground truth set, and then produces final precision and recall numbers by aggregating over all mentions.

Table 4 shows the results of the entity grouping task. We report only a single result for our system, as the entity grouping task in not dependent on the output of the other two tasks, leading results to be the same in both the pipelined and non-pipelined scenarios. As shown, overall entity grouping performance was good, with F-measure of 0.938. We present two baselines as a means of comparison. The first baseline presents performance when grouping was done by simple string match. To get a sense of the performance of UMLS entity linking, we ran our concept extraction procedure to perform entity linking as the second baseline. Unsurprisingly, the baseline methods performed well on instances in which the terms were the same or quite similar, but less well in cases of linguistic variation. In both cases, the baseline method produced very high precision but poor recall, and produced overall F-measure significantly worse than the proposed method.

Table 4.

Results of the entity grouping task.

	Precision	Recall	F-measure
String Match Baseline	0.998	0.621	0.766
Entity Linking Baseline	0.990	0.667	0.798
Our System	0.924	0.953	0.938

Discussion

In this work we examined the recommendation identification, conditional linking, and entity grouping tasks separately, with the intent that they be combined into a larger diagnostic summary generation pipeline. This is consistent with many other multitask extraction scenarios, where pipelined approaches have been shown to be effective²⁵. However, this pipelined approach potentially ignores interdependencies between the tasks that may be exploited to improve performance. In particular, it is clear that the conditional identification task is dependent on the output of the recommendation identification task, but it might also be the case that recommendations could be better identified if the presence of relevant conditional statements were detected. To explore this option, we attempted a pilot study in which the two tasks were combined with a Markov Logic Network^26,27 based approach. Although we do not go into details about this approach here, ultimately the results indicated that the joint model produced results comparable but not superior to the presented pipelined approach. While this attempt at tighter task integration was unsuccessful, future work may wish to further examine methods that tie the tasks together.

Once each task has been performed, a diagnostic recommendation chart can be produced from the output. Several minor heuristics were applied to decide on the layout and presentation of the charts. The name used to label each procedure was selected from one of the ways in which it appeared in the text, with the most general name that mapped to a UMLS procedure given precedence. Procedures were ordered on the chart based on the order of their first appearance in the document. Conditional statements were similarly sorted by their original document ordering. To extract the conditional statement itself, the syntactic parse of the sentence was used to identify the largest phrase headed by the trigger term. Finally, a simple heuristic was used to determine if a procedure was generally (unconditionally) recommended. If at least two recommendation statements appeared for the procedure that were not linked to a conditional and none of these statements disagreed (i.e., recommended vs. not recommended), the procedure was marked generally recommended or generally not recommended, as applicable.

Figure 2 shows an example of an automatically generated summary. As shown, in many instances the chart is able to present a concise statement about the conditions under which each procedure should be performed. While the figure illustrates that the tasks undertaken here have the potential to produce relatively clean recommendation tables, it also highlights some of the potential sources for error. First and most critically, because of the nature of the automatic generation process several of the recommendations and conditionals are incorrect. For instance, the table contains the nonsensical statement that myocardial perfusion imaging is “recommended in other testing modalities”. Because the cost of giving an incorrect recommendation is potentially severe, automatically generated summaries would need to be accompanied by human oversight in a true human usage scenario, but may still have beneficial use cases for machine interpretation. A less severe error is the extraction of statements that, while not quite incorrect, provide no extra conditional information. For instance, Figure 2 states that echocardiography is “recommended in the setting of ACS”, which gives no further conditional information in this context. Finally, we note that some additional errors arise from parsing errors, where the extracted conditional contains more or less information than the ground truth.

Figure 2.

Example automatically generated diagnostic recommendation chart. Incorrect extractions are given in gray.

To get a rough understanding of the overall quality of the automatically generated summaries, we performed a small qualitative study. For each of the 33 diseases we randomly generated a recommendation chart using either the baseline, system, or gold standard methods. The generated charts were then shown to a medical resident, who examined their correctness and coherence and assigned a rating to each one on a 1 (best) to 4 scale. Results of this study suggested that the quality of the system generated charts was at least comparable to those generated by the gold standard, with average ratings of 1.72 and 1.50, respectively. However, because of the small scale of the study, we caution that while these results suggest that the automatically generated summaries have potential, a more robust analysis would be required to fully understand the overall quality.

While the tasks undertaken in this work can be used collaboratively to produce diagnostic guideline summaries, they each have the potential to play other roles in medical text extraction. Recommendation classification can be used separately to identify procedure recommendations at a high-level, and has overlap with other information extraction tasks of interest, such as the extraction of diagnoses and findings from the electronic health record^28,29. Similarly, conditional extraction has parallels with other relation extraction tasks, such as biological event extraction^30–32, semantic relation labeling^33,34, and the identification of treatment relations³⁵. Finally, coreference resolution and entity linking are frequently applied to extract and link clinical terms from various sources of unstructured text, although approaches for reference text may be different than those made on clinical notes³⁶.

Conclusion

Understanding the scenarios in which a diagnostic procedure should be performed is a difficult and nuanced task, and one in which there is a potential role for NLP-based automation and guidance. In this work we examined the extraction of conditional recommendation statements, as a first look at diagnostic recommendation summarization. This was done by examining the assignment of mention-level recommendation labels, linking these mentions with conditional recommendation statements, and grouping them based on their underlying referent. Our results show that each of these tasks can be performed with above baseline accuracy, and the resulting output can be combined into concise summaries of the diagnostic procedure. While our automatically generated summaries are likely to contain more noise than those generated from gold standard human annotations, they represent a first step toward the automated extraction and understanding of the diagnostic workflow.

While the conditional summaries produced here attempt to give an overview of when procedures are appropriate, they do not impose a relative order on when each procedure should be performed. As such, the current recommendation summaries still require that some inference be performed by the clinician to choose the most appropriate test for a given scenario. While this does not necessarily diminish their usefulness to human readers, it can be a potentially problematic omission for machine interpretation. Given this, future work will focus on the expansion of this summarization framework towards a fully formed flowchart of diagnostic procedures that attempts to give a conditional roadmap to the entire diagnostic process.