Detecting Unplanned Care From Clinician Notes in Electronic Health Records

The text-mining methods the authors describe can be applied to automatically review free-text clinician notes to detect unplanned episodes of care mentioned in these notes.

Abstract

Purpose:

Reduction in unplanned episodes of care, such as emergency department visits and unplanned hospitalizations, are important quality outcome measures. However, many events are only documented in free-text clinician notes and are labor intensive to detect by manual medical record review.

Methods:

We studied 308,096 free-text machine-readable documents linked to individual entries in our electronic health records, representing care for patients with breast, GI, or thoracic cancer, whose treatment was initiated at one academic medical center, Stanford Health Care (SHC). Using a clinical text-mining tool, we detected unplanned episodes documented in clinician notes (for non-SHC visits) or in coded encounter data for SHC-delivered care and the most frequent symptoms documented in emergency department (ED) notes.

Results:

Combined reporting increased the identification of patients with one or more unplanned care visits by 32% (15% using coded data; 20% using all the data) among patients with 3 months of follow-up and by 21% (23% using coded data; 28% using all the data) among those with 1 year of follow-up. Based on the textual analysis of SHC ED notes, pain (75%), followed by nausea (54%), vomiting (47%), infection (36%), fever (28%), and anemia (27%), were the most frequent symptoms mentioned. Pain, nausea, and vomiting co-occur in 35% of all ED encounter notes.

Conclusion:

The text-mining methods we describe can be applied to automatically review free-text clinician notes to detect unplanned episodes of care mentioned in these notes. These methods have broad application for quality improvement efforts in which events of interest occur outside of a network that allows for patient data sharing.

Introduction

Unplanned episodes of care, whether hospital readmission, emergency department (ED) evaluation, or unplanned hospitalization, are inconvenient for patients, are potential markers of unsafe or inefficient practices, and are disincentivized by value-based payment methodologies.^1–5 Hospitals and health systems that provide care for Medicare beneficiaries are incentivized to reduce rehospitalizations.⁶ Detection of these rehospitalizations, which may occur at multiple facilities that do not routinely share patient-level data, is difficult. Current methods to detect rehospitalizations, unplanned episodes of care, and the symptomology associated with unplanned visits include manual chart abstraction, query of claims databases, and analysis of chief complaint lists. These approaches for understanding the nature of unplanned care among patients with cancer are labor intensive, and can be incomplete.^2,7–9

We conducted a quality-improvement project to reduce variation in care for cancer patients at our institution. One of the quality improvement goals we sought to achieve was a reduction in unplanned care episodes, including ED visits and unplanned hospitalizations. We found that many unplanned episodes occurred outside of our institution, and were mentioned in subsequently generated free-text clinician notes. However, these episodes were underascertained by relying only on structured or coded entries in our electronic health record (EHR) system.¹⁰ We believe that by including textual notes in ascertaining unplanned care episodes, we can obtain a more accurate estimate of the true rate of unplanned care.

Typical applications of text mining in oncology have focused on the discovery of novel risk factors and cancer pathways from the scientific literature.¹¹ There is a growing body of clinical applications that includes colorectal cancer screening,^12,13 identification of recurrence,¹⁴ and lymphoma classification.¹⁵ Information extraction methods are typically designed for processing large clinical corpora in real-time, and can draw from both structured and unstructured data sources. Our hypothesis is that information extraction techniques can significantly improve the ability to use EHR data to measure outcomes related to postdiagnosis morbidity in patients with cancer.

We evaluate the performance of information extraction on two tasks—identifying external ED visits and identifying the most significant symptoms at the visit—using a text-processing framework that has been previously used to monitor for adverse drug events,¹⁶ learn drug-drug interactions,¹⁷ identify off-label drug use,¹⁸ test clinical hypotheses,¹⁹ identify new medical insights,²⁰ and generate phenotypic fingerprints²¹ as well as build predictive models.²² For the purpose of our study, we defined “unplanned care” as an emergent medical event that results in ED care, unplanned inpatient care, or a trip to an outpatient urgent care center.

Our approach combines internal coded data with outcomes documented within free-text clinical notes to detect external unplanned care visits reported by patients and their caregivers. We calculate the positive predictive value (PPV) of our text-mining pipeline for detecting noncoded unplanned visits occurring at outside facilities. Also, we demonstrate that free-text documentation and coded information can be combined into a unified, structured representation that can be used to enhance quality of care assessment. In addition, we profile the most significant symptoms, and co-occurring symptoms documented within ED notes to provide insights into the multisymptomatic nature of patients seeking emergent care after diagnosis of breast, GI or thoracic cancer.

Methods

The analytic cohort included patients who first presented to the Stanford Cancer Institute (SCI) from January 1, 2010, through December 31, 2013, and had at least one International Classification of Diseases (ed 9) code in their medical record of breast (174.9), GI (150.1), or thoracic cancer (151.9). The SCI provides multidisciplinary cancer care at one site on the Stanford campus, and is a component of Stanford Health Care (SHC), an academic medical center that provides outpatient, inpatient, and since 2014, primary care. During the time of our analysis, SHC and SCI used the Epic EHR (Epic Systems, Verona WI).

We defined two follow-up observation periods for measuring the percentage of patients with any unplanned care. Both periods started with the initial visit to SCI; the first is a 90-day follow-up period, which includes 3,318 patients (breast n = 1,486; GI n = 1,198, thoracic n = 634) with at least 90 days of post-initial visit time, and the second is defined by the subset of patients in the 3-month cohort, which includes 2,510 patients (breast n = 1,131; GI n = 910, thoracic n = 469) with 1 year or more of follow-up time. This definition does not distinguish between patients with metastatic and nonmetastatic disease, but defines a cohort that excluded patients who had one or two visits for second opinions.

Using EHR data extracted from the SHC data warehouse, we linked basic demographics and diagnostic criteria with 308,096 clinical notes to conduct a retrospective study of unplanned care for new diagnoses of breast, GI, or thoracic cancer from 2010 through 2013. Our data included clinical notes documented as part of a visit, and any other machine-readable free-text document, including telephone encounter logs and communication letters that are sent from SHC providers to clinicians outside of the SHC system who are involved with a patient's care.

Detecting External Unplanned Care

To develop the methods for detecting external unplanned care visits, we used de-identified clinical notes for patients with breast cancer who were diagnosed before 2010 and patients who had received bone marrow transplant (International Classification of Diseases ed 9 Z94.81) from the research data warehouse at the Stanford School of Medicine.²⁴ In this stage, we defined the type of concepts we aim to extract (shown at the top of Figure 1) and the heuristic we use to select candidate events for postprocessing. This methods development research is conducted under an approved institutional review board protocol.

Figure 1.

Text processing pipeline for detecting unplanned care events from clinician notes. See main text for a detailed explanation.

An overview of our pipeline is illustrated in Figure 1. The first step involved processing the textual notes using string matching on a dictionary of relevant terms. For example, in Figure 1, for the text snippet S-1, the text processing step identified mentions of “presented,” “Santa Clara Valley Medical Center,” “emergency room,” as well as their document offset, or at what position within the note they were found.

In the second step, and indexed at time zero by the first SCI visit, we constructed a candidate event matrix for each 30-day postdiagnosis period. The purpose of the matrix data structure is to abstract outcome-relevant information and provide a more succinct representation for further analysis.

In order to populate the matrix with candidate events, we required an emergent care event term (eg, “urgent care,” “ED” or “ER,” highlighted red in Figure 1) to co-occur near an emergent care location term (in blue) and within the offset window. The offset window, set to the length of 75 characters to the right and left of the emergent care event term (average word length in English is 5.1 characters), defined the scope of the local context, or “snippet,” analyzed for each candidate event and was tuned during the system development process.

For each entry in the candidate event matrix, any dictionary terms that appeared near a tagged emergent care event term (eg, “urgent care,” “ED” or “ER,” highlighted red in Figure 1) and their document offset were stored. Figure 1 shows an example of how three retrieved text snippets, S-1, S-2, and S-3, would be processed by our system and represented in the n × i candidate event matrix, where n is the number of candidate event (rows) and i is the number of terms (columns) in our dictionary. Each candidate event snippet has an emergent care event term in red, and a location term in blue.

In the third step, rule-based filtering techniques were applied to the candidate event matrix to detect negated events and the event experiencer, as well as to perform concept disambiguation, which is described later in this section. The filtering techniques, which seek to increase PPV and maintain the same level of sensitivity, are similar to the algorithm used in the widely used ConTexT software, and we integrated ConTexT's terminology for negation detection.²⁴

The initial filtering step applied negation and experiencer rules to each candidate event in the matrix. Then, for remaining candidate events, we used word sense disambiguation techniques to automate the process of identifying which sense of a word (ie, meaning) was used in a sentence. These techniques examined the position of concept modifiers, or “suppressors,” relative to the position of the emergent care concept that was mentioned in the clinical note. The intuition is that by leveraging other textual cues near the mention of the emergent care term, we can assign more certainty to event detection. For example, snippet S-3 in Figure 1 shows the detection of the suppressor term “HER” (highlighted in gray). Because the event is not preceded by a presence of an event trigger (eg, “presented” or “admitted”), the detection of the suppressor provides the key piece of information that allows us to associate the detected term “ER” with the concept “estrogen receptor” instead of “emergency room” and hence remove this candidate event from the final set of results.

To measure the percentage of patients with unplanned care after their first visit to SCI, in the last step, we normalized all events to a representation that consisted of a patient identifier, cohort, and the note creation time. This new structured representation allowed for integrating all data at the patient level for outcome reporting. If a patient experienced multiple unplanned events within the study period, the event that occurred closest to the first SCI visit was used to indicate the time the patient experienced an unplanned care episode.

Evaluating the Detection of Unplanned Episodes of Care

To evaluate the performance of the text-mining pipeline, we manually reviewed 750 randomly selected entries in the candidate event matrix to create a reference set. Of 750 candidate events, 511 were labeled as affirmative and 239 as negative by our text-mining pipeline. For the same set of 750 candidate events, a reviewer manually labeled each instance as affirmative or negative, while blinded to the label assigned by the text-mining pipeline. A snippet supporting each of the 750 candidate events detected, a unique identifier for the patient and the note, the date of first visit to SHC, and the note type where shown to the manual reviewer to make their assignment. We use this reference set to estimate PPV and evaluate the performance of text mining non-SHC unplanned events.

ED Symptomology

To profile the symptomology of patients with cancer in need of unplanned care, we extracted mentions of the “symptom/disorders” concept from SHC ED notes. Similar to our unplanned event detection, the symptom data can be represented in a contiguous set of 30-day candidate event matrices. In the case of symptom detection, during postprocessing, we analyzed the local context around each symptom/disorder term for modifiers related to negation, patient history, and the experiencer (patient or family member). To address duplication issues related to symptom mentions, we only allowed a distinct set of symptoms to be assigned to a patient for each ED visit.

To assist in the mapping of terms that occurred with low frequency to higher level concepts, (eg, “right-arm pain” to “pain”), we used a symptom hierarchy based on MedDRA-preferred terms. Although MedDRA is a system designed for regulatory activities (eg, drug safety), it has relevance to symptom surveillance more broadly. In contrast to other widely used terminologies, it is crafted to support the aggregation of cases for reporting purposes.

To determine how often, and what types of, symptoms cluster together, we calculated individual frequencies and the co-occurrence rate of all possible symptom pairs and triples.

Results

For the study periods, our text-mining pipeline identified 612 non-SHC unplanned events reported by patients, their caregivers, and in some cases by external providers. Based on the evaluation of our text-mining algorithm, we estimate the PPV of text mining non-SHC unplanned events at 92%.

The majority (53%) of affirmative mentions occurred in clinic Progress Notes. The second most informative note type was Telephone Encounters (21%). Although the text-mining algorithm allows generic terms such as “outside hospital” as the location, among visits with named facilities, most visits were at EDs and urgent care clinics in the same county as SHC.

In the first thr3ee months of follow-up, we increased our ability to determine the true rate of patients with one or more unplanned care visits by 32% (15% using coded data; 20% using all data) by examining textual notes (Figure 2). For patients with 1 year of follow-up, the detection rate increased by 21% (23% using coded data; 28% using all data; Figure 3). Figures 2 and Figure 3 also show the temporal progression of each cohort's unplanned care rate from their first visit to SCI through the duration of the follow-up window, in 30-day snapshots. For each study period, non-SHC events, which occurred in the absence of any SHC unplanned episodes or events that were documented before an SHC unplanned episode, were included in the rate calculations.

Figure 2.

Percentage of patients (N = 3,318: breast n = 1,486; GI n = 1,198; thoracic n = 634) with unplanned care up to 30 days after first visit measured by coded data versus information extraction (IE).

Figure 3.

Percentage of patients (N = 2,510: breast n = 1,131; GI n = 910; thoracic n = 469) with unplanned care up to 1 year after first visit measured by information extraction.

Emergent Care Reasons

Of the 1,263 patients in our study with any unplanned care episodes, 724 had SHC Emergency Provider notes with their symptoms documented. From these notes, we extracted 179 distinct symptom/disorder terms. Pain was the most prevalent symptom reported in the notes of all patients, regardless of cohort. Overall, pain was detected in 75% of ED visits. The three most frequently specified body locations were abdominal (37%), back (17%), and shoulder (4%). After pain, the most frequent symptom documented was nausea (54%), followed by vomiting (47%), infection (36%), fever (28%), and anemia (27%).

Based on all patients' ED notes with symptom mentions, Table 1 shows the prevalence of the most frequent symptom pairs and triples at ED visits. In addition to symptoms and disorders known to occur in ED presentations, we also detected mental and behavioral health issues, including anxiety (12%), emotional distress (12%), other mental health issues (13%), and behavioral health issues (10%).

Table 1.

Prevalence of Top Pairs and Triplets of Symptoms

Group	Symptom Complex	Support
All (N = 1,263)	Nausea, pain, and vomiting	0.35
	Infection and pain	0.29
	Hypertension and pain	0.27
	Anemia and pain	0.22
	Fever and pain	0.22
Breast (n = 400)	Nausea, pain, and vomiting	0.34
	Infection and pain	0.29
	Hypertension and pain	0.29
	Dyspnea and pain	0.21
	Anemia and pain	0.21
GI (n = 550)	Nausea, pain, and vomiting	0.39
	Diarrhea and pain	0.24
	Fever and pain	0.23
	Infection and pain	0.31
	Hypertension and pain	0.26
Thoracic (n = 313)	Nausea, pain, and vomiting	0.30
	Dyspnea and pain	0.30
	Cough and pain	0.27
	Infection and pain	0.28
	Cough and pain	0.24

Discussion

Our analysis of unplanned care visits using a text-mining framework demonstrates the feasibility of profiling patient cohorts with unplanned care, the rates of symptoms and disorders they present with, and the associated care setting in which care was sought. Although a large body of previous work on quantifying emergency care visit rates and the symptoms associated with cancer care exists, it is limited by the coverage of claims-based measurement.^2,5,7,8 Also, it is often the case that one chief complaint is not enough to explain the complex morbidity of an immunocompromised patient with cancer seeking unplanned care. Our approach demonstrates high fidelity in examining the symptomatology of patients who seek unplanned care and detects precise concepts such as “epigastric pain” or “aphasia” that can be missed in analyses done on the basis of coded data or chief complaints.

Analyzing the symptomology of patients allows us to identify those symptoms that can be the focus of interventions for preventing unplanned care; such as prophylactic antiemetic agents to prevent or mitigate chemotherapy-induced nausea and vomiting^25,26 and opioids to manage cancer pain.^27,28 Specifically, our framework demonstrates a feasible and practical way to estimate prevalence rates of symptoms, and enables relative comparisons between cohorts that would otherwise be time consuming to perform through retrospective chart reviews.

Our text-mining framework also provides a practical way to determine the prevalence of the symptoms associated with unplanned care. Our analysis indicates that when a symptom is mentioned, the most frequently reported conditions are pain, nausea, and vomiting, and that in combination they co-occur in more than one third of all ED notes for our patient population. Following the top three symptoms are infections and fever. Cachexia and non–treatment-related complications also appear; overall, the top symptoms documented in ED notes postdiagnosis are consistent with high prevalence of pain and chemotherapy-induced toxicities.

Limitations of Clinical Text Mining

Extracting unplanned visit information from clinical notes presents several challenges. Independent of the algorithm is an organization's ability to index and retrieve clinical notes as machine-readable documents. Our pipeline cannot mine handwritten or scanned clinical documentation.

The information extraction tools use manually curated terminologies and rule-based techniques. The evaluation process revealed that our concept dictionary was not comprehensive for all locations and suppressor concepts. The rule used to identify candidate events, based on the proximity of an emergent care and a location term, can be too strict. Some non-SHC events are found in text that is outside the local context window.

Additional limitations relate to evaluation, which is restricted to PPV. The infeasibility of manual record review for all the patients limits our ability to report sensitivity and specificity, both of which require reviewing all the patients to find the false negatives and confirm the true negatives. This issue affects all information extraction methods applied to large document collections.²⁹

Finally, the generalizability of any text-mining algorithm needs to be assessed at alternative institutions. Although emergent care terms are less institution specific and therefore more portable, the location terminology is custom to SHC and is learned from the data. Similar approaches have to be used to derive a custom terminology for an alternative institution.

Conclusion

An understanding of the prevalence and the precipitating events that lead to emergent care in patients with cancer can inform the design of intervention strategies to avoid unplanned visits.³⁰ However, assessing the true nature of unplanned care with structured EHR data alone cannot reliably estimate all provider use rates, or the complex symptomology of patients with cancer in need of emergent care.

Additionally, given the increased availability of data in electronic form, it is crucial to develop practical and automated approaches to ascertain global views of quality of care such as rates of unplanned visits. Developing such approaches can lead to a better understanding of the quality of cancer care and inform the development of novel interventions that can improve current care delivery.

Our work demonstrates the feasibility and utility of clinical text mining for the purpose of quality measurement, and the potential of text analytics to support a new paradigm of health outcome measurement.^13,31 The ability to extract key health and medical information from unstructured free-text clinical notes is a critical component for a high-value cancer care delivery system. Although information gaps may still exist for many patients, the application of text-mining methods can improve the assessment of morbidity outcomes for patients with cancer.

Ethics Statement

The methods for detecting ED visits from provider notes were developed under an approved institutional review board protocol (No. 24883) for developing methods for the learning health system using our institution's clinical data warehouse. We apply those methods on minimally necessary data with limited sharing at Stanford Hospital under a project for quality improvement of health care under guidance provided by SHC Compliance and Privacy Department.

Authors' Disclosures of Potential Conflicts of Interest

Disclosures provided by the authors are available with this article at jop.ascopubs.org.

Author Contributions

Conception and design: Suzanne Tamang, Manali I. Patel, Douglas W. Blayney, Julie Kuznetsov, Nigam Shah

Administrative support: Julie Kuznetsov

Provision of study materials or patients: Yohan Vetteth, Douglas W. Blayney

Collection and assembly of data: Suzanne Tamang, Samuel G. Finalysen, Julie Kuznetsov

Data analysis and interpretation: Suzanne Tamang, Nigam Shah

Manuscript writing: All authors

Final approval of manuscript: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Detecting Unplanned Care From Clinician Notes in Electronic Health Records

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or jop.ascopubs.org/site/misc/ifc.xhtml.

Suzanne Tamang

Employment: Oracle Health Sciences (I)

Stock or Other Ownership: Oracle Health Sciences (I)

Manali I. Patel

No relationship to disclose

Douglas W. Blayney

Stock or Other Ownership: Abbott Laboratories, Amgen, Bristol-Myers Squibb, Express Scripts, Johnson & Johnson, UnitedHealthcare, Google, International Business Machines, Oracle Corporation, Physician Resource Managment

Consulting or Advisory Role: Clinical Oncology Advisory Group, Physician Resource Management,, SOBI, UnitedHealthcare, Bristol-Myers Squibb, Astra Zeneca, Genentech/Roche, Pfizer, Taho

Julie Kuznetsov

No relationship to disclose

Samuel G. Finalysen

No relationship to disclose

Yohan Vetteth

No relationship to disclose

Nigam Shah

Stock or Other Ownership: Kyron

Consulting or Advisory Role: Kyron

Travel, Accommodations, Expenses: Apixio