Automating the Transformation of Free-Text Clinical Problems into SNOMED CT Expressions

Abstract

An important function of the patient record is to effectively and concisely communicate patient problems. In many cases, these problems are represented as short textual summarizations and appear in various sections of the record including problem lists, diagnoses, and chief complaints. While free-text problem descriptions effectively capture the clinicians’ intent, these unstructured representations are problematic for downstream analytics. We present an automated approach to converting free-text problem descriptions into structured Systematized Nomenclature of Medicine – Clinical Terms (SNOMED CT) expressions. Our methods focus on incorporating new advances in deep learning to build formal semantic representations of summary level clinical problems from text. We evaluate our methods against current approaches as well as against a large clinical corpus. We find that our methods outperform current techniques on the important relation identification sub-task of this conversion, and highlight the challenges of applying these methods to real-world clinical text.

INTRODUCTION

As the healthcare industry increasingly embraces the promise of new data-driven approaches, the challenges of managing and organizing complex patient data become more pronounced.¹ Even before the deployment of the first electronic health record (EHR), healthcare organizations struggled to establish a structured, organized, and standard representation of patient data.² A major advance in this area came in the 1960s when Dr. Weed proposed orienting the data in the patient record around a list of current conditions, or the “problem list.”³ This emphasis on centralizing and enumerating relevant clinical problems enabled patient information to be consumed in a more systematic way, and helped to standardize physicians’ interaction with the patient record.⁴ A major advantage of this problem-oriented approach is that concise descriptions of clinical problems can summarize and emphasize sections of the larger clinical note narrative. These short phrases describing diagnoses and other patient issues are not limited to the problem list, however. “Summary level” descriptions of clinical problems are also found in diagnosis statements, chief complaints, and reasons for visit,^5,6 and all provide a concise way of expressing pertinent patient conditions.

There are a variety of ways in which these summary level problem descriptions are captured. Free-text is the most expressive form of these problem summaries, capable of capturing the clinical state directly as intended by the clinician.⁷ While clinician-friendly, this unstructured representation presents significant problems for downstream analytics.⁸ In contrast, a problem may be represented as codes chosen from a controlled terminology. This applies more structure, but limits expressiveness.⁹ Even if codified capture is the goal, many systems still allow for free-text entry as a backup if the correct code cannot be readily found.^10,11 These competing representational priorities introduce a fundamental optimization problem in representing these entries – free-text maximizes usefulness for clinicians,^12,13 while structured and codified forms are more amenable to data analytics,¹⁴ standardization activities,¹⁵ and EHR secondary use.⁵

In this study we introduce a method to minimize this conflict between structured and unstructured forms by proposing a framework for converting free-text clinical problem descriptions to codified, structured formats using Natural Language Processing (NLP) techniques. The advantage of structured representations to downstream analytics primarily motivates this effort.¹⁴ By leveraging deep learning methods, we aim to automatically translate text-based problems into Systematized Nomenclature of Medicine – Clinical Terms (SNOMED CT) Expressions,¹⁶ a structured representation capable of capturing the semantics of summary level problem descriptions in a computable way.

BACKGROUND & RELATED WORK

The selection and use of a controlled vocabulary to codify free-text clinical problems has been an active area of research.^17,6 In particular, SNOMED CT¹⁸ has been shown both in principle and in practice to be an effective standard for capturing the semantics of these clinical conditions.^19–21 Generally, clinical problems can be represented using SNOMED CT concepts in one of two ways:²²

• Pre-Coordinated Concept: A concept represented as an atomic unit with a single identifier. Example: 370221004|Severe asthma (disorder)|

• Post-Coordinated Concept: A concept represented as the composition of multiple pre-coordinated concepts that in aggregate define the intended semantics.

  Example: 195967001|Asthma (disorder)| + 24484000|Severe (severity modifier)|

Although pre-coordination has the advantage of simplicity,²³ even summary level problem descriptions are often too expressive to be captured by a single, pre-coordinated SNOMED CT concept. Elkin et al. found that SNOMED CT could only represent 51.4% of problem list entries without composition compared to 92.3% with composition.²⁴ Liu also found that composition was necessary, observing that 53% of summary level data required two or more SNOMED CT concepts.⁵

Post-coordinated concepts can be readily represented in SNOMED CT via the SNOMED CT Compositional Grammar,¹⁶ a formal specification for representing SNOMED CT post-coordinated expressions. For example,^† “Severe asthma” can be represented via the following expression consisting of a main focal concept optionally qualified by attribute/value pairs:

195967001|Asthma (disorder)|:

246112005|Severity (attribute)| = 24484000|Severe (severity modifier)|

Previous work on converting text to SNOMED CT expressions has focused on the identification and classification of the attribute relationships – for example, what (if any) SNOMED CT attribute best describes the relationship between “Severe” and “asthma.” One general approach to this task is to iteratively learn the lexical patterns around how entities relate for a given relationship type.²⁵ Miñarro-Giménez et al. utilized this technique to fit extracted problem list concepts into learned SNOMED CT relationship patterns.²⁶ This work leveraged the fact that lexical patterns in pre-coordinated SNOMED CT terms are known and relatively predictable.²⁷

Kate proposed a different approach to this task – not as a relation extraction task between two concepts within the context of a sentence, but as an attempt to identify if a relationship holds between the entire problem phrase and the concept, or “relation identification.”²⁸ To illustrate the difference, take the “Severe asthma” example above. The Min˜arro-Gime´nez et al. approach would attempt to find the relationship between “Severe” and “asthma,” whereas Kate would take the entire phrase “Severe asthma” and attempt to determine which SNOMED CT concepts relate and how. Kate used a Support Vector Machine (SVM)²⁸ model trained separately for each relationship type to determine if a given relationship held between a concept and the full text entry. Our contributions in this study are focused on extending Kate’s work in the following ways: First, we present an end-to-end process for converting free-text summary level problem descriptions to SNOMED CT expressions, enumerating the sub-tasks and incorporating additional NLP techniques such as dependency parsing. Next, we leverage deep learning techniques to increase relation identification performance as compared to the SVM model. Finally, we begin an initial evaluation of model performance in real-world scenarios using summary level problem descriptions extracted from clinical notes.

METHODS

At a high level, our methods are broken into three sequential processing steps as shown in Figure 1. First, given a summary level problem description (in text form), the relevant biomedical concepts are recognized and extracted. Next, one of the extracted concepts is chosen as the main semantic focal point, or the focus concept of the expression. Following this, the remaining concepts are attached to the focus concept by inferring their role in the expression as a whole. This relation identification task is a critical step in the formation of the SNOMED CT expression and constitutes the bulk of our contributions to this research area. These three steps are explained in detail below.

Figure 1:

Overall steps to convert summary level problem text to a SNOMED CT expression. The processing steps are executed sequentially from left to right with arrows indicating where output from one step feeds into a subsequent step. The output of each processing step is shown linked to the portion of the SNOMED CT expression to which it contributes.

Concept Extraction

The first step in converting a free-text summary level problem description to a SNOMED CT expression is to extract a list of all relevant concepts from the text. To accomplish this, we leveraged MetaMap, a named-entity recognition tool developed by the National Library of Medicine (NLM) to extract Unified Medical Language System (UMLS) concepts from text.²⁹ An example output of the MetaMap application given the input problem “Venous varicosities in lower extremities with recent thrombophlebitis” is shown below:

C0226813:Vein of lower extremity [Body Part, Organ, or Organ Component]

C0042345:Varicosities (Varicosity) [Disease or Syndrome]

C0332185:Recent [Temporal Concept]

C0040046:Thrombophlebitis [Disease or Syndrome]

In this example, the extracted concepts are shown with their UMLS Concept Unique Identifiers (CUIs) and textual descriptions. As our goal is to construct SNOMED CT expressions, we must additionally map the UMLS concepts to SNOMED CT. By configuring MetaMap to match only on SNOMED CT terms, we ensured that each returned UMLS concept encompassed at least one SNOMED CT concept. If the UMLS concept included a single SNOMED CT concept, a direct mapping was made. There are, however, cases where multiple SNOMED CT concepts are incorporated into a UMLS concept.³⁰ In these cases, all matching SNOMED CT concepts were considered.

Focus Concept Selection

Choosing the focus concept given the list of extracted SNOMED CT concepts is the next step. Here we utilized dependency parsing to align the root node of the problem dependency tree with an extracted MetaMap concept, a technique inspired by Spasić’s use of dependency trees to determine the semantic similarity of clinical terms.³¹ First, dependency parsing was conducted on the input clinical problem description. Next, the word with the ROOT dependency was compared to all extracted MetaMap concepts. Finally, if one of the extracted MetaMap concepts was triggered by the root word, that concept was then chosen as the focus concept.^‡ Figure 2 shows the dependency parse with the root word varicosities, which is then matched to the relevant concept. We used the spaCy open-source NLP toolkit for dependency parsing along with specifically trained biomedical models from the scispaCy project.³²

Figure 2:

Extracting the focus concept from summary level problem descriptions using dependency parsing. The focus concept is selected via alignment to the ROOT of the dependency parse.

Relation Identification

Identifying the relationships between the problem text and the extracted concepts is the next step. We build primarily on the relation identification task definition as described by Kate²⁸ and formalize it for our purposes as such: Given an input summary level problem description and a concept extracted via MetaMap, compute the appropriate SNOMED CT attribute (or relationship type) to connect them. Figure 3 outlines the high-level steps of the relation identification algorithm. First, the problem text and the text of each concept extracted via MetaMap are input into a classifier, the details of which are described further in the following sections. Next, the classifier outputs a probability for each SNOMED CT attribute type indicating the likelihood that a particular relationship holds between the problem text and the extracted concept. Finally, candidate relationships are pruned based on the stated domain and ranges of the SNOMED CT Machine Readable Concept Model (MRCM) Attribute Range Reference Set.³³ For example, if the extracted MetaMap concept of interest is 80195002|Structure of venous system of lower extremity (body structure)|, any SNOMED CT attribute with a range that is incompatible would be removed (such as 424226004|Using device (attribute)|, whose range is limited to children of 49062001|Device (physical object)|), as shown in Figure 3. After the pruning, the SNOMED CT attribute with the highest remaining probability is chosen.

Figure 3:

Steps to identify relationships between concepts extracted from a free-text summary level problem description.

A Deep Learning Approach to Relation Identification. Deep learning architectures have shown promise in a variety of NLP tasks,³⁴ and in this study we compare two popular models for the relation identification classifier. We first consider a Bidirectional Long Short-Term Memory (BiLSTM)^35,36 deep learning architecture. At its base level, a BiLSTM is a specialized type of Recurrent Neural Network (RNN),³⁷ an artificial neural network architecture that processes information sequentially, factoring in previous input at each current step. This makes RNNs specifically applicable to NLP tasks as text is processed much like a human would – reading words sequentially and inferring the semantics of the current word based on the previous ones.³⁸ The Long Short-Term Memory (LSTM) facet of the architecture allows for finer control over what information is retained and forgotten by employing more sophisticated feedback loops layered on top of the RNN framework.³⁵ The bidirectional extension to the LSTM completes our architecture, allowing context to be built not only forward but in the reverse direction as well.³⁹ In general, the LSTM family of models has shown promising results for NLP relationship extraction tasks.⁴⁰

Convolutional Neural Networks (CNN)⁴¹ are another deep learning architecture. Like LSTMs, CNN models also can recognize spatially related features of the data. For these models, input is filtered via sliding windows which are then pooled to create a subsampled representation of the input sequence. Although used heavily for image processing, CNNs have shown promise in a variety of NLP related tasks including relationship classification.⁴² In this study, we compare both models for our relationship identification task.

Both of our deep learning architectures use embedding models based on Bidirectional Encoder Representations from Transformers (BERT), a context-aware language model with state-of-the-art performance on a variety of NLP tasks.⁴³ For our experiments we leveraged Clinical BERT,⁴⁴ a pre-trained BERT model fine-tuned on a clinical text corpus.

Architecture. The high-level architecture and data flow for both the BiLSTM and CNN classifiers were similarly structured. First, two inputs are passed to the Input layer – the full problem text and the text of the extracted concept. Next, each input is passed to an Embedding layer to create a vector representation of the text input using the BERT model. The vectorized input is then processed by either a BiLSTM with 100 hidden units or a CNN with two convolutional layers. Both models were configured for 20% dropout to avoid overfitting. Finally, a fully-connected Dense layer with a softmax activation function is used to output the probabilities for each SNOMED CT attribute type.

Training. In concordance with Kate’s approach,²⁸ stated concept relationships from SNOMED CT US Edition, September 2018 Release 18 were used to train the classifier. Training set construction began with all SNOMED CT stated concept-to-concept relationships excluding 116680003|Is a| relationships. The reasoning for excluding “Is a” relationships is that once the concepts of the expression are known (see the concept extraction step), any “Is a” relationship for these concepts can be directly inferred from the SNOMED CT hierarchy. Next, because our classifier inputs are text, we use the SNOMED CT concept labels for training. As SNOMED CT concepts may contain multiple labels, given each relationship we created training records for all possible pairs of source and target labels. Finally, we excluded relationship types with less than 125 instances, leaving a total of 1,526,043 training records and 78 relationship types available for training. All experiments below used this data set for training, and Experiments 1 & 2 used a held-out portion of this set for testing. We refer to this data set as the SNOMED CT Relationship data set.

Evaluation

System performance was measured for both the Focus Concept Selection and Relation Identification steps of the architecture. The Concept Extraction phase was not directly evaluated, as for this task MetaMap was used without modification (see Reátegui et al.⁴⁵ for a recent analysis of MetaMap performance on clinical text). Four experiments were conducted to evaluate model performance.

Experiment 1: First, both the BiLSTM and CNN models were compared against a baseline Naïve Bayes⁴⁶ model. All models were trained and tested on the same SNOMED CT Relationship data set. Data was prepared for this experiment by partitioning 25% of the SNOMED CT Relationship set for testing and 75% for training. The Naïve Bayes, BiLSTM, and CNN classifiers were all then trained and tested on the same data, with the exception of a further 20% of the training data being withheld from the BiLSTM and CNN models for validation. For testing, we recorded the F1 scores for each individual attribute as well as overall averages for all classifiers.

Experiment 2: Next, we compared our results to previously reported results of Kate’s SVM model.²⁸ We followed Kate’s evaluation procedures in order to replicate his experiment using the best performing model from Experiment 1: For each attribute, 5000 relationships of the desired type were randomly selected from the SNOMED CT Relationship set along with an equal number of negative examples. Given this test set, the ability of the classifier to correctly determine whether or not the chosen relationship was present was recorded.

Experiment 3: To evaluate relation identification model performance in real-world scenarios and test generalizability to different data sets, we utilized a large data set of summary level clinical problems extracted from a Mayo Clinic corpus of over 14 million clinical documents. This corpus has been extensively analyzed by Liu et al. and is a rich source of diverse summary level problem descriptions.⁵ Three trained annotators examined a random subset of 401 summary level clinical problem descriptions from this corpus. First, the annotators were asked to select the word or words that represented the focus concept of the problem. Next, the annotators we tasked with connecting the focus to relevant modifiers using one of twenty-one relationship types. These relationship types were chosen via analysis of common attributes across prominent clinical problem models including the Fast Healthcare Interoperability Resources (FHIR) - Condition resource,⁴⁷ Clinical Element Model (CEM) - ClinicalAssert model,⁴⁸ and openEHR - Problem/Diagnosis archetype.⁴⁹ For more information, see Goossen⁵⁰ for details regarding these models. Only relationships supported by >20 annotations in the test corpus are evaluated in this study. This test corpus was then used to evaluate both the Focus Concept Selection and Relation Identification parts of the pipeline. The three pair-wise Cohen’s kappa inter-annotator agreement values for each annotator pair were 0.78, 0.85, 0.84 for the focus concept annotations and 0.76, 0.76, 0.82 for the relationship annotations.

Experiment 4: Finally, we evaluated how effective the dependency parse-based method is at locating the focus concept of the clinical problem. Using the test data set described in Experiment 3, we evaluated the ability to correctly identify the focus using three spaCy dependency parse models: (1) general-purpose English, (2) scispaCy biomedical, and (3) a custom model based on scispaCy biomedical fine-tuned with 150 manually-annotated clinical problem text examples.

RESULTS

Experiment 1: Overall results of the BiLSTM and CNN models compared to the Na¨ıve Bayes baseline for the SNOMED CT relation identification task are shown in Table 1. While both deep learning models significantly outper-formed the baseline, the BiLSTM model slightly outperformed the CNN model. Because of this result, downstream experiments focus on the BiLSTM model.

Table 1:

Comparing overall SNOMED CT relation identification model performance.

	Accuracy	F1 score (macro avg)	F1 score (weighted avg)
Naïve Bayes (baseline)	0.720	0.460	0.665
CNN + Clinical BERT	0.886	0.822	0.880
BiLSTM + Clinical BERT	0.888	0.851	0.888

Experiment 2: Figure 4 shows the results of the BiLSTM classifier compared to the SVM classifier results as reported by Kate.²⁸ Note we did not evaluate 116680003|Is a| relationships, so we do not fully correspond to Kate’s results. Also, Kate reported two results for 363698007|Finding site| based on two different domains/ranges. The F1 score reported in Figure 4 represents the highest of the two scores.

Figure 4:

Comparing the BiLSTM + Clinical BERT model for SNOMED CT relation identification to the SVM model as reported by Kate²⁸ for five SNOMED CT attributes.

Experiment 3: The results of the evaluation of the BiLSTM classifier against the annotated test corpus are shown in Table 2. This table contrasts two main data points: (1) the Clinical Text F₁ score, which measures the classifier’s ability to predict the correct relationship type given the focus and a modifier in the clinical text corpus, and (2) the SNOMED CT Relationship F₁ score, which is the corresponding value derived from Experiment 1 for the given attribute. The F₁Δ value shows the difference between the two scores, illustrating the difference in performance when testing on relatively predictable SNOMED CT terms vs. real clinical text. The eight relationships supported by >20 annotations in the clinical text test dataset are displayed.

Table 2:

Comparing BiLSTM relation identification scores using two different test data sets: real-world clinical text (Clinical Text F₁ score) and relationships from SNOMED CT (SNOMED CT Relationship F₁ score). The F₁Δ value equals Clinical Text F₁ score minus SNOMED CT Relationship F₁ score.

Attribute	Clinical Text Precision	Clinical Text Recall	Clinical Text F1 score	SNOMED CT Relationship F1 score	F1 Δ
Severity	1.000	0.880	0.936	0.882	0.054
Laterality	0.990	0.950	0.970	0.999	-0.029
Clinical course	1.000	0.800	0.889	0.994	-0.105
Finding site	0.956	0.790	0.865	0.988	-0.123
Due to	0.528	0.487	0.507	0.830	-0.323
Has interpretation	0.577	0.714	0.638	0.992	-0.354
Following	0.733	0.306	0.431	0.837	-0.406
Associated with	0.579	0.134	0.218	0.738	-0.520

Experiment 4: Finally, Table 3 shows the effectiveness of using a dependency parse-based method for selecting the focus concept of a problem description. For each dependency parse model the accuracy is shown, where accuracy in this context reflects the number of times the model selected the same focus span as the human annotators over the 401 total entries in the test set.

Table 3:

Evaluating the performance of the dependency parse-based method for selecting the focus concept of the clinical problem. Three different dependency parse models were evaluated.

Model	Accuracy
Default spaCy English (baseline)	0.68
ScispaCy Biomedical	0.75
ScispaCy Biomedical + fine-tuning	0.91

DISCUSSION

Overall, both the CNN and BiLSTM significantly outperformed the Na¨ıve Bayes classifier in identifying relationships between two SNOMED CT concepts, as shown in Table 1. This comparison provides evidence that a deep learning architecture is a viable approach and can outperform a simple Na¨ıve Bayes baseline. The BiLSTM did also outperform the CNN model slightly. The difference was most evident in the F1 macro avg score, which is important as this metric gives equal weight to each relationship type and disregards any SNOMED CT relationship class imbalance.

Figure 4 shows that a deep learning approach can outperform SVM classifiers at the relation identification task with SNOMED CT relationships. The BiLSTM scored higher for four of the five attributes tested, while the SVM out-performed the BiLSTM for one attribute: “Has active ingredient.” It is worth noting that we cannot directly compare Kate’s results 28 with our BiLSTM model beyond these five attributes listed in Figure 4. Overall F1 scores are not comparable because Kate’s model was trained and tested using the top 14 SNOMED CT attributes only, while our overall F1 scores (see Table 1) are derived from a classifier trained and evaluated on 78 relationship types. Also, Kate’s overall results factor in scores for 116680003|Is a| attributes – relationships that we omitted.

Table 2 highlights the challenges that come with applying these methods to clinical text. For several attributes, relation identification was significantly worse against real-world clinical problem descriptions compared to SNOMED CT relationships (as shown by F₁Δ). These differences are not unexpected – the SNOMED CT text used for training is relatively structured,²⁷ but actual clinical problem descriptions are not. As shown, performance degradation for several relationship classes is pronounced, with a highly negative F₁Δ score signifying low model generalizability from the SNOMED CT text corpus to the clinical text.

Finally, the focus concept selection results in Table 3 show not only that a dependency parse-based method of focus concept selection is an effective technique, but that using a domain-specific pre-trained model does boost performance noticeably. Even more, these results indicate that even minimal fine-tuning (150 manual annotations) can have a fairly large impact on overall performance.

CONCLUSION

The goal of this work was to present an end-to-end system for converting unstructured summary level problem descriptions into SNOMED CT expressions. Our contribution focused primarily on introducing a new deep learning method for relation identification between concepts and problem phrases. We show that our method outperforms current approaches to identifying relationships between clinical phrases and SNOMED CT concepts, a fundamental part of building SNOMED CT expressions. We also show that a model trained exclusively on SNOMED CT stated relationship text does not transfer to clinical text without performance degradation.

LIMITATIONS

Our study has several limitations. First, there is no available gold-standard test set for evaluating the full conversion of text-based problem descriptions to SNOMED CT expressions – we must evaluate individual steps of the pipeline independently. Furthermore, such a test set is challenging to construct as there may exist more than one way syntactically to represent the same conceptual expression. Also, it has been shown that codification of problems by physicians is subject to considerable variation.⁵¹ All these factors together make quantitative evaluation of this task difficult.