POETenceph - Automatic identification of clinical notes indicating encephalopathy using a realist ontology

Abstract

Identifying inpatients with encephalopathy is important. The disorder is prevalent, often missed, and puts patients at risk. We describe POETenceph, natural language processing pipeline, which ranks clinical notes on the extent to which they indicate the patient had encephalopathy. We use a realist ontology of the entities and relationships indicative of encephalopathy in clinical notes. POETenceph includes a passage rank algorithm, which takes identified disorders; matches them to the ontology; calculates the diffuseness, centrality, and length of the matched entry; adds the scores; and returns the ranked documents. We evaluate it against a corpus of clinical documents annotated for evidence of delirium. Higher POETenceph are associated with increasing numbers of reviewer annotations. Detailed examination found that 65% of the bottom scoring documents contained little or no evidence and 70% of the top contained good evidence. POETenceph can effectively rank clinical documents for their evidence of encephalopathy as characterized by delirium.

Introduction

POETenceph detects encephalopathy in inpatient notes from all clinical and ancillary domains. Encephalopathy is a broad term for brain disease, damage or malfunction¹. It is characterized by an altered mental state. Delirium is a sudden encephalopathy or state of confusion, which is associated with hyperactivity, somnolence or both. It is characterized by an acutely altered mental state, also referred to as an acute change in mental status (ACMS). We will be using delirium as the encephalopathy of interest and ACMS as a synonym for delirium.

Encephalopathy Identification

Identifying patients with encephalopathy that manifests, as delirium is important because it is prevalent, often missed^2–4, and the longer it persists the greater the risk to the patient. Importantly, some delirium is treatable. Practice guidelines for preventing and dealing with delirium have been created^2,5–7.

Patients who experience delirium have longer hospital stays, more ICU days, and frequently are on mechanical ventilation longer than non-delirium patients.^6,8–11 They are also more likely to require post-hospital institutionalization. Studies show that patients with persistent delirium have a 2 to 4-fold higher chance of dying within the next year.^8,9,12 All of this extra care is expensive. Delirium has been estimated to cost the U.S. healthcare system between $38 billion and $152 billion annually.^7,13

Delirium is estimated as being missed in the majority of older, delirious patients.^2,3,14 These misses may be due to unfamiliarity with psychiatric conditions and their treatments and the prevalence of the hypoactive forms. Wancata et al. found that non-psychiatrists’ diagnostic sensitivity in detecting general psychiatric disorders ranged from 31.3–89.5%.¹⁵ Delirium is particularly difficult because its symptoms fluctuate throughout the day. To correctly diagnose it, a clinician must look at the time course of mental status changes, the history of predisposing illness, any medications taken, any alcohol or other substance withdrawal, and pertinent environmental changes.¹⁶ It is probable that age and vision impairment are correlated with a missed diagnosis because physicians are over-looking symptoms they believe are explained by a known condition (age-related decline and perception problems, respectively).

This tendency, over-looking symptoms due to erroneous ascription, points to the need for clinical decision support. Automatically identifying documents with evidence of encephalopathy written about patients with no pertinent diagnostic codes is important to the hospital as well as the clinician and patient. Encephalopathy increases the complexity of patient care, which should be reflected in hospital reimbursement and case mix statistics.

Automated Evidence Identification

The automated identification of clinical evidence has the potential to improve clinical decision support for delirium. In cases where a physician has recognized delirium symptoms, but has wrongly ascribed them, it will be important for clinical decision support systems to identify those symptoms in order to re-direct the thinking of the clinical team. In all but a very few electronic health record systems (EHRs), the description of symptoms are found in free-text notes. ICD-9 codes and other structured text reflect current and “working” diagnoses assigned to inpatients, but symptom extraction requires natural language processing (NLP).

Our lab has developed an NLP pipeline¹⁷ to extract “disorder mentions.” Disorder mention is used in NLP to refer to all terms related to bodily malfunction. It includes signs and symptoms (sometimes grouped as clinical findings), diseases, diagnoses, and syndromes. In this paper, we will use the term “evidence” instead of disorder mention. Evidence seems a more accurate term when discussing EHR data that suggest a diagnosis because it does not imply, necessarily, that a decision has been made about what the data signify.

We have adapted the UtahPOET NLP pipeline to find evidence of encephalopathy (POETenceph) by including a realist ontology of encephalopathy.

Ontologies

Ontologies are used to incorporate world knowledge (e.g., semantic information) into NLP systems. An ontology includes the concepts, relationships, and rules governing their interactions.¹⁸ Most biomedical informaticists are familiar with the National Library of Medicine (NLM) Unified Medical Language System’s (UMLS) Metathesaurus and Semantic Network¹⁹, an amalgamated ontology of medical terms from multiple, diverse source vocabularies.

The UMLS Metathesaurus is not built using realist principles. Realist ontologies are designed to ensure that the concepts represented reflect real-world entities.²⁰ The most important example of this principal from the current system is the distinction between an EHR entry and a disorder mention. If a clinician writes, “the patient has dementia and is unresponsive,” UMLS Metathesaurus mappings would be to the disorder mentions “dementia” (C0497327) and “unresponsive to stimuli” (C0857494). However, the terms “dementia” and “unresponsive” are ambiguous. Dementia could refer to Alzheimer’s disease (C002395) or any one of the other 581 related concepts in the UMLS Metathesaurus. Unresponsive could refer to “unresponsive to questioning” (C2188201) or any one of the other 35 related concepts. This term-use ambiguity is generally acknowledged. However, there is a second type of ambiguity that is conflated into the idea of a disorder mention, the clinician’s knowledge. If we assume that the sentence is linking unresponsiveness to dementia and that the clinician knows that “unresponsive to questioning” is a symptom of dementia, but “unresponsive to stimuli” is not, then we can assume the reference is to “unresponsive to questioning.” However, if our second assumption is false and the reference is to “unresponsive to stimuli,” then we should alert the clinician to their mistake in ascribing the symptom to dementia.

Realist ontology development forces us to differentiate the two types of ambiguity because we must separate what is happening in the patient, from what the clinician is thinking, and what is written in the EHR. These three things are different types of real world entities. What is happening in the patient is a level 1 entity (a real thing), what the clinician is thinking is level 2 (a mental representation), and what is written in the record is level 3 (a written representation).²¹

One alternative to including an ontology is to train machine learning classifiers to classify clinical records into diagnostic categories. This method has not been generalizable across datasets²² probably because evidence is expressed in many different ways and structure of the documents differs. Therefore training a generalizable classifier would require a tremendous amount of training data. There would also be no guarantee that the resulting classifier reflects accurate biomedical knowledge. Although it would reflect the consensus of the clinicians whose documents it was trained with, there is no guarantee this is accurate.

Ontology and Evidence Rating

The ontology can be used to find evidence and then rate it’s association with a diagnosis of encephalopathy in the form of delirium. To find evidence is a straightforward dictionary match including lexical variants. One common way to match lexical variants is to use the UMLS Metathesaurus. We employ this technique.

To rate the association between the matched ontology concept and the diagnosis of encephalopathy, is a process of determining the semantic similarity or relatedness between the matched concept and the diagnosis. Semantic similarity is based on the separation between concepts in the hierarchical structure, while relatedness can use relationships other than hierarchical. See Batet, et al.²³ and Pedersen, et al.²⁴ for discussions of different approaches to calculating semantic similarity and relatedness. In broad terms, there are two groups of semantic relatedness algorithms, those based on an ontology and those calculated from corpus statistics.

Corpus statistics alone may not lead to good semantic similarity judgments because the use of language cannot be used to understanding the meaning of language. It is the problem that psychologists refer to as the “symbol grounding” problem²⁵. In essence, words in a system cannot derive meaning from other words because words are arbitrary symbols with no intrinsic meaning. Using a realist ontology, we ground our words by attaching them to entities in the world. The resultant ontology is then used to judge semantic relatedness.

To make this judgment, we combine a path-length metric from the ontology with information content (IC) measures following the example of Pedersen, et al.²⁴ and others^23,26,27. We use a simple measure of centrality for path length to determine semantic relatedness, since we have a central point representing complete knowledge (a diagnosis of encephalopathy). For IC, we used the number of term matches with and without stemming found in the UMLS Metathesaurus. IC is generally done with corpus statistics, but we reason that uniqueness within the UMLS Metathesaurus is a better indicator of uniqueness within the field. The final IC indicator we use in term length. We reason that longer terms are only included if the information they impart is necessary.

Methods

Evaluation

To evaluate of our approach, we will look at two system aspects. The first is POETenceph’s ability to recognize EHR entries as evidence of encephalopathy in the form of delirium. The second system aspect will be its ability to combine the evidence to rank clinical documents based on the likelihood the patient was experiencing delirium.

We will compare the POETenceph output to 100 de-identified clinical documents from the Pittsburgh dataset²⁸. The corpus was created by author YS. The full description of the corpus creation is in Shao, et al.²⁹ The documents were identified as related to delirium using topic modeling. Two subject matter experts (former nurses including coauthor CW) annotated the mentions related to delirium in each document. The annotation process is also described in Shao, et al. Annotation was covered under University of Utah IRB_00043685.

We show the overall results compared to each reviewer individually and to the annotations matched between reviewers as well as the unique annotations across reviewers. With a condition as difficult to define as delirium, we think that each expert’s contribution should be considered. Therefore, for the detailed look at the POETenceph results we classified the amount of evidence based on the number of unique annotations across the two reviewers. There was a tendency for reviewer 1 to annotate only a few concepts. Documents were categorized as showing little or no evidence (up to 3 annotations or delirium denied in the text), some evidence (delirium implied in the text), evidence against delirium (another condition stated in the text), and good evidence (5 or more annotations or delirium stated in the text).

Since POETenceph is being developed to augment a word-matching system that searches for the words “enceph” and “deliri,” with proximity negation, we will also look at the occurrences of those words in the annotated corpus.

UtahPOET pipeline

The UtahPOET pipeline (Figure 1) is described in detail in Doing-Harris, et al.¹⁷ It is built in Apache UIMA³⁰ and has the common NLP pipeline structure. Pre-processing includes sentence splitting, tokenization, and part-of-speech tagging. We add a preprocessing strep for POETenceph that identifies label-value pairs (e.g., “Ca 8.6”) using regular expressions (Figure 1, section F). The regular expressions match noun-number pairs.

Figure 1.

The UtahPOET pipeline. Zoom in for form detail.

UtahPOET is unique in that it separates well-formed sentence (i.e., prose) from ill-formed sentences (i.e., nonprose) so that nonprose sentences can be re-split and prose sentences can be dependency parsed to facilitate processing. Non-prose sentences are split at line breaks because their internal cohesion is not important. Prose sentences are dependency parsed to allow more appropriate mapping of concepts, whose constituent words are separated in the text and to attach attributes (negation, experiencer, uncertainty, generic) to the concepts. For example, “infection in her uterus” mapped to “infection of uterus.” Generic mentions are like “psychiatry” in the sentence “patient sent to psychiatry for evaluation.”

Multi-word terms and their attributes are found in nonprose by adjacency. We reason that nonprose sentences are telegraphic and unlikely to include long-distance dependencies. Long-distance dependencies can only be understood by recognizing clausal structure.

Post-processing (Figure 1, section K) specific to the SemEval competition is not used here. However, the removal of terms erroneously matched to disorders is retained (“sinus”, “tongue”, “blood”, “ear”, “a.m.”, “dr.”, “md”, “dr”, “D. D.”, “m”, “pm”, “he”, “mr.”, “CT”, “P”, “Ht”, “Eyes”, “DATE”, “T”). We also add a post-processing step in which any disorder mention found in a label-value pair is ignored. Instead of the remainder of the post-processing, we search the text of the identified disorders against the Encephalopathy ontology to determine whether they are evidence of encephalopathy.

Ontology of Encephalopathy Characterized by Delirium (OECD)

The main points of interest in the OECD are the level 3 (written representation) EHR entry, the level 2 (mental representation) clinical entities (i.e., disease picture, clinical picture, clinical finding, and diagnosis).

The upper level of a realist ontology contains the entities continuant and occurrent. Continuant subsumes the entities dependent continuant with generically dependent continuant and specifically dependent continuant and independent continuant. Explaining these entities is beyond the scope of this paper.

Figure 2 shows some of the IAO Information content entities we propose. The EHR entry entity is a textual entity from the Information Artifact Ontology (IAO)³¹, which is an information content entity, which is a generically dependent continuant. The clinical entities we propose all fall under the Ontology of General Medical Science (OGMS)³² entities disease picture, clinical picture, clinical finding, and diagnosis. These entities are data items, which are also information content entities. A diagnosis of encephalopathy is a diagnosis of mental disease from the Ontology of Mental Disease.³³ A diagnosis of mental disease is an OGMS diagnosis. The clinical findings that we propose are either OGMS clinical history findings or OGMS physical examination findings. We use the diagnostic criteria described in a across several journal articles^{5,6,16,34–37} to associate clinical findings, clinical pictures, disease pictures and diagnoses. We do not attempt to address any physical aspects of the patient, only the clinician’s interpretation of the patient.

Figure 2.

A) Proposed entities for the Ontology of encephalopathy characterized by delirium are in the red box. All other entities are from the Information Artifact Ontology. B) Proposed clinical picture entities.

We created the ontology in stages. The first stage used automated extraction of UMLS Metathesaurus concepts from sentences hand-culled from medical records because they related to Delirium. We reconciled these concepts against the diagnostic criteria for delirium as stage two. Stage three is putting the ontology into BFO format. Stage four will include an expansion of the ontology with terms identified by our ontology management system SEAM and the terms identified in this study. The final stage will be obtaining expert agreement on the final ontology.

POETenceph

We incorporate the OECD by creating two Apache Lucene indices. The first index contains the unique ontology identifiers and labels (i.e., ontology key) associated with each of the ontology entities. Centrality scores are calculated for each entity by starting with the diagnosis (encephalopathy) and associated disease pictures (encephalopathy and urinary tract infection), which are set to centrality of -1. As you travel out across relationships from the disease picture, each level adds 1 to the centrality. The scores are normalized with division by the longest path length. Example centrality, diffusion, and length scores are listed in Table 1.

Table 1.

Example centrality, diffusion, and length scores for OECD entities.

Entity	Centrality	Diffusion	Length
Delirium	0.0	1 – 0.002	0.1
Encephalopathy	0.0	1 – 0.01	0.1
Mental Disease	0.875	1 – 0.05	0.1
UTI	−0.125	1 – 4.2 E - 4	0.3
Memory Impairment	0.625	1 – 5.6	0.2
Hyperactive delirium	0.625 words	1 – 6.3 E-6	0.2

We designed the second index to accommodate the variability in language used by clinicians and to measure diffusion. It was created by matching all of the terms in the ontology (EHR entry instances and entity labels) to the UMLS Metathesaurus. Matches were restricted to the clinical source vocabularies and disorder or medication semantic types. In this index each term was associated with the ontology key of the search term that returned it, and its measures of diffusion, centrality, and length (see Table 1 for examples). Diffusion was the number of matched terms returned. Once all the terms have been searched the maximum number of matches was used to normalize the diffusion of each term, keeping it between 0 and 1. The result is subtracted from 1 because less diffuse is better.

The final aspect of the term that we used to calculate its score was the length of the term again normalized by the length of the longest term. The final score is the mean of the centrality, diffusion, and length. We score the document with the sum of the scores for the evidence that is positively asserted (not negated), experienced by the patient, and not generic.

Results

The most important test of POETenceph is determining whether high rated and low rated documents are correctly categorized. In the first set of figures we show two ways of looking at the human annotations across the documents. Figure 3a shows the POETenceph scores as they compare to each reviewer individually. Figure 3b shows the POETenceph scores compared to the annotations matched across the reviewers and the unique annotations found when combining the responses of the two reviewers. In both images it can be seen that the trend of the POETenceph scores follow those of the reviewers. It is apparent that they follow reviewer 2 more closely than reviewer 1 and the unique annotations better than the matched annotations. We find this preference for reviewer 2, who also led the number of unique annotations, was due to her annotating a higher number of pieces of evidence. Visual inspection indicated that her annotations were valid. In one instance (file #44), she annotated “Change in mental status,” “sluggish,” and “combative” that reviewer 1 did not.

Figure 3.

Comparison of scores from POETenceph and Unique Annotations: A) the bottom 20 files. B) The top 20 files. ✪ indicates the text states no delirium, ★ indicates that the text states delirium present.

In a more detailed examination, we looked at the top 20 and bottom 20 documents for the amount of evidence highlighted by the human reviewers (see Figure 4). In the bottom 20, 13 showed little or no evidence of Encephalopathy or Delirium (65%) within this group 1 had controlled encephalopathy and 1 had controlled delirium. One document showed some evidence of delirium (5%), and 6 showed good evidence (30%). The 20 highest rated documents showed almost the same pattern 14 had good evidence for delirium (70%), 3 had evidence of another condition (15%), 1 had controlled delirium (5%) and 2 had little or no evidence (10%). Three high scoring documents had evidence for related conditions (depression, alcohol withdrawal, and kidney disease), but without delirium.

Figure 3.

Comparison of scores from POETenceph and Reviewer results: A) Each reviewer separately. B) The annotations matched between the reviewers and the unique annotations across the two reviewers.

Looking at the word search for “enceph” we find only 11 documents contained the string, 3 from the low scoring documents 3 showing little evidence and 1 showing some evidence. Four of the high scoring documents contained the string “enceph,” 3 had good evidence of delirium and 1 had some evidence.

Matching the string “deliri” we find more coverage. Sixty-four of the 100 documents contain the string. It is found in only 9 of the 20 lowest scoring documents and only 4 of those contain little or no evidence, 3 contain good evidence and 2 contain some evidence. Of the 20 highest scoring documents 17 contain the string “deliri.” Two of the mentions contain little or negated evidence, 2 contain other conditions with delirium stated as not being present. Two high scoring documents with good evidence for delirium did not contain the string.

We then compared the POETenceph annotations with the expert annotations. We again looked at 21 documents across a range of scores from low to high. We found two obvious problems because we built UtahPOET to identify disorder mentions, POETenceph does not find medication or instrument mentions. These entities exist in the ontology but comparison to the ontology is done after UtahPOET has run.

Looking only at the direct evidence for delirium, POETenceph consistently misses “mental status changes,” “unable to follow commands,” “unresponsive,” “agitated,” and “not communicate.” Mental status changes, unresponsive, and agitated are simply missed by UtahPOET. Unable, follow and communicate are identified as concepts by UtahPOET, but not as evidence by POETenceph. The other class of evidence that is missed is evidence that is not found in the UMLS Metathesaurus. For example, “screams alternating with drowsiness,” “minimally verbal,” or “not clear mentally” are not found there.

POETenceph identified 249 positively asserted pieces of evidence, 64 are agreed-upon by at least one reviewer (26%). Eighty-six (35%) annotations are problems from UtahPOET, including repeated erroneous mappings of “M],” “man,” “block,” “he” (12 times), and “ca” (16 times). The remaining 98 we think are related to encephalopathy, but missed by the reviewers either because their task was to locate direct evidence (39%). This task orientation may also explain why only 5 of the 30 useful negated pieces of evidence were agreed-upon by a reviewer. POETenceph erroneously negated five others. Twenty-five of the negated evidence findings were errors like those found in the positively asserted evidence and therefore not useful.

The annotators found 107 pieces of evidence missed by POETenceph. Twenty-one were medications or instrument mentions, 3 are not disorders (Psychiatry, ammonia, potassium), 47 were pure misses (e.g., “agitated,” “acute psychotic event,” and concepts including “mental status”) and 38 are not in the UMLS Metathesaurus.

Discussion

We found that the POETenceph performed the task of ranking documents based on the amount of evidence of encephalopathy in the form of delirium well. Higher scores indicated documents that contained more reviewer annotations. The scores tracked particularly well with reviewer 2 and the total number of unique annotations in a document. Seventy percent of the top 20 and bottom 20 scoring documents were appropriately placed. Only 15% of the documents in top group and 20% in the bottom were completely misplaced. These findings indicates that using a realist ontology combined with measures of centrality, diffusion and length, it is possible to calculate scores for evidence in a document that corresponds well with expert judgments.

By restricting our search to documents without diagnostic codes for encephalopathy, we will be able to identify evidence that should be brought to the attention of clinicians and medical coders. This evidence can be presented to clinicians to encourage them to re-evaluate the patient or the billing codes whichever is appropriate.

This performance is an improvement over string search. The string “enceph” only returns 11 of the documents two of which would be completely misplaced. The string “deliri” returns many more of the documents because it was used in the creation of this document corpus, but the documents are unranked so although only 6 contain little or no evidence there is no indication of which 6. Also 3 documents with good evidence would be missed.

This study alone does not prove the utility of realist ontologies. It does indicate that they are useful. Having an ontology for the construction of a diagnosis from clinical findings that is separate from the UMLS Metathesaurus allows us to exploit the Metathesaurus’ ability to represent term use. We can also change the interactions among the diagnostic elements without advocating for changes to the Metathesaurus.

The most obvious problem exposed in this study is that UtahPOET is restricted to matching disorder mentions whereas clinical evidence also comes from medication and instrument mentions. There are other improvements to UtahPOET that would aid performance of POETenceph. We need to address the erroneous mapping of small strings like “he” and “M],” as well as the missed mappings of “agitated,” “mental status” concepts, and “depressive.” Our next step will be to address the problems with UtahPOET.

In another project we are working on formalizing the Encephalopathy ontology. This formalization will include fixes to the current associations between the entities as well as the extension into level 1 entities. The formalization process requires input from clinicians to determine the settled science linking level 1 entities, the state of the practice linking level 2 entities and the imprecision of clinician language when transcribing their thoughts and impressions into the EHR. We will be investigating the use of probabilities on the relation between EHR entries and mental representations. Once these extensions have been added to the ontology we will need to reexamine how we calculate centrality.

For concepts not found in the UMLS Metathesaurus from the SemEval challenge we developed a structure SVM component for UtahPOET. We did not have training data to train the component for the equivalent pieces of evidence nor did we try the component with its current training because it does not currently work well. In future work, we will upgrade the component and obtain more training data for it.

The final area of investigation that this study has spurred is the relationship between disorders and body locations. Currently we determine each of these concepts separately. However, there must be a method to use one as a sanity check against the other. It seems that if a body location is found it should be used to restrict the possible matches to its attached disorder. For example, and that some disorders carry a body location with them “angina pectoralis” is only in the chest. We may be able to extend this idea to the mapping of strings into the ontology.

In future work we will also investigate the balance of the three measures, which go into our scores. It is possible that the current balance gives terms length too much impact on the final score for its informational content. It may be better to weight centrality higher than length and possibly also diffusion.

Limitations

The test corpus had only documents that had been previously identified as relevant to delirium. Therefore, we may have our encephalopathy detection set too loose. Because we are in no danger of missing delirium mentions, we cannot be sure we would catch them all. Our next evaluation will contain a predetermine mixture of cases.

We are also limited in that we did not enforce agreement between the annotators. We do not believe that it is necessary for all expert reviewers to agree on each and every annotation. Individuals will always have a slightly different opinion on what constitutes important evidence for conditions as difficult to define as delirium. Forcing arbitrary agreement would create the illusion of a strict definition. We do think that having several reviewers would allow us to choose annotations favored by the majority. A majority opinion may be more indicative of the prevailing medical consensus. In future work, we will strive to have five reviewers for each file, although that will have implications for the number of files we will be able to have reviewed.

Conclusion

An effective document ranking system can be created by combining a disorder-identification NLP pipeline with a realist ontology. Comparing disorders to ontology entities and scoring the matches based on diffusion, centrality and the term length can assess the evidentiary value of disorder mentions.