Identifying Goals of Care Conversations in the Electronic Health Record Using Natural Language Processing and Machine Learning

Abstract

Context

Goals-of-care discussions are an important quality metric in palliative care. However, goals-of-care discussions are often documented as free text in diverse locations. It is difficult to identify these discussions in the electronic health record (EHR) efficiently.

Objectives

To develop, train, and test an automated approach to identifying goals-of-care discussions in the EHR, using natural language processing (NLP) and machine learning (ML).

Methods

From the electronic health records of an academic health system, we collected a purposive sample of 3,183 EHR notes (1,435 inpatient notes and 1,748 outpatient notes) from 1,426 patients with serious illness over 2008–2016, and manually reviewed each note for documentation of goals-of-care discussions. Separately, we developed a program to identify notes containing documentation of goals-of-care discussions using NLP and supervised ML. We estimated the performance characteristics of the NLP/ML program across 100 pairs of randomly partitioned training and test sets. We repeated these methods for inpatient-only and outpatient-only subsets.

Results

Of 3,183 notes, 689 contained documentation of goals-of-care discussions. The mean sensitivity of the NLP/ML program was 82.3% (SD 3.2%), and the mean specificity was 97.4% (SD 0.7%). NLP/ML results had a median positive likelihood ratio of 32.2 (IQR 27.5–39.2) and a median negative likelihood ratio of 0.18 (IQR 0.16–0.20). Performance was better in inpatient-only samples than outpatient-only samples.

Conclusion

Using NLP and ML techniques, we developed a novel approach to identifying goals-of-care discussions in the EHR. NLP and ML represent a potential approach toward measuring goals-of-care discussions as a research outcome and quality metric.

Introduction

Communication about goals of care is an important aspect of palliative and end-of-life care, and has been associated with improved patient and family outcomes as well as reduced intensity of treatment near the end of life (1–6). However, despite the importance of such communication, discussion and documentation of goals of care remains inadequate in many settings (1, 7–11). Wide-ranging deficiencies in timely communication and documentation of goals of care have been reported in both inpatient and outpatient settings (2, 11–13). Even when goals-of-care discussions result in completion of an advance care planning document, lack of standardization in how such documentation is stored in the electronic health record (EHR) poses a barrier to communicating patients’ goals across the continuum of care (14). To promote the implementation of goals-of-care discussions for all patients with serious illness, it is important to be able to readily identify the timing and content of goals-of-care discussions (15–18). However, an efficient means of identifying goals-of-care discussions in the EHR for patients with serious illness does not yet exist.

Natural language processing (NLP) methods enable computers to process and analyze textual data written in human languages (19). Although early NLP approaches were constrained to using rule-based methods developed iteratively by humans to classify textual data into understandable categories, the advent of machine learning (ML) methods that rely on automated computational techniques to classify data have allowed NLP to extract and measure more complex constructs (19). Inductive supervised machine learning is a family of ML techniques that utilizes statistical methods to derive rules that best explain the classification of training data—i.e., data independently labelled to reflect outcomes. These rules may subsequently be used to predict the classification of new data (20).

NLP and ML have been used to automatically extract a wide variety of data from unstructured clinical records (21, 22). While many clinical NLP efforts have been limited to rule-based measures of diagnoses, symptoms, and clinical events (23–28), more recent efforts have incorporated ML techniques to infer diagnoses of cerebral aneurysms (29), measure patient-reported symptoms in patients with cancer (30), and enhance predictive modeling of mortality and long ICU stays among critically ill patients (31). In palliative care, rule-based NLP algorithms have been used to measure quality indicators and goals-of-care discussions for patients undergoing palliative surgery (32, 33), outpatients with COPD (34), and patients who died of cancer (35, 36); and, deep learning algorithms have been used to measure the documentation of patient preferences and communication among patients admitted to the intensive care unit (37, 38). However, additional work is needed to be able to accurately and efficiently identify documentation of goals-of-care discussions in the EHR across the continuum of serious illness. In this report, we describe the development and preliminary performance characteristics of a novel approach using NLP/ML to identify goals-of-care discussions for outpatients and inpatients with serious illness.

Methods

Data Sources

We assembled a retrospective sample containing clinical notes gathered from inpatients and outpatients with serious illness at the two teaching hospitals and affiliated clinics of UW Medicine, a large academic health system. Notes were collected from multiple sources: (a) all notes from participants in two randomized controlled trials (RCTs) of palliative care interventions, which had been previously abstracted for documentation of goals-of-care discussions (39, 40); (b) a convenience sample of notes written by palliative care specialists who had designated each note as containing, or not containing, goals-of-care discussions for the purpose of clinical service metrics; and, (c) clinical notes from a random sample of patients with serious illness (41, 42) (Table 1). We purposively selected these sources of EHR notes to enrich our data for notes containing goals-of-care discussions, which constitute a small proportion of randomly sampled EHR notes. All notes were extracted from the UW Medicine clinical EHR platforms (Cerner Millennium, Cerner Corp., North Kansas City, MO; and, EpicCare, Epic Systems Corp., Madison, WI) as plain text. The study was conducted as part of a quality improvement initiative to measure the prevalence of documented goals-of-care discussions for patients with serious illness at UW Medicine.

Table 1.

Composition of sample dataset.

Data source	Determination of goals-of-care discussions	Notes with goals-of-care discussion, n	Notes without goals-of-care discussion, n	Total notes, n
Outpatient clinical notes written by non-palliative providers for 537 patients with serious illness enrolled in RCT of communication-priming intervention, 2012–2016 (39)	Abstraction of any discussions of advance care planning, referral to hospice, or referral to palliative care	212	1,070	1,282
Inpatient clinical notes for 168 ICU patients enrolled in RCT of communication facilitators, 2008–2013 (40)	Abstraction of textual documentation of goals-of-care discussions	53	— a	53
Clinical notes from 421 patients evaluated by palliative care specialists, 2012–2017 b	Goals-of-care discussions indicated in structured quality-improvement metrics; confirmed by investigator review of subset of 139 notes	424	31	455
Clinical notes from random sample of 300 patients with serious illness who receive care through UW Medicine, 2014–2016 c	Abstraction of textual documentation of goals-of-care discussions	0 d	1,393	1,393
Total		689	2,494	3,183

^aGoals-of-care-negative notes for this cohort were not recorded during this trial.

^bContains 446 inpatient notes and 9 outpatient notes.

^cCriteria: age ≥ 18 years, cared for at UW Medicine between May 1, 2014 and August 1, 2016; with one or more of the nine Dartmouth Atlas chronic health conditions by ICD coding (41, 42); and receiving regular care through UW Medicine as defined by at least one non-surgical inpatient visit at a UW Medicine facility in the preceding two years, or two outpatient visits at the same UW Medicine facility in the preceding 32 months with at least one visit in the preceding two years. Contains 936 inpatient notes, and 457 outpatient notes.

^dTwo-phase abstraction of this dataset identified no goals-of-care discussions.

Outcome Assessment and Labelling

Toward the goal of measuring conversations about goals of care, we adopted an operational definition of goals-of-care documentation as a provider note including any of the following: (1) advance care planning activities (discussion of values, goals, and preferences in consideration of future care), (2) completion of advance directives or Physician Order for Life-Sustaining Treatment (POLST) forms, or (3) referral to hospice or subspecialty palliative care services (Supplemental Appendix 1). (Subspecialty palliative care consultations in our system are most frequently requested for elucidation of goals-of-care). For the 1,335 notes from RCTs (39, 40), we evaluated the abstraction protocols from the original trials and mapped previously coded elements to this study’s definition of goals-of-care documentation, with input from the principal abstractor in both trials (PDT). For the 455 palliative care specialist notes, note authors designated each note as containing or not containing documented goals-of-care discussions at the time of note entry using a structured data collection instrument attached to each note. For the 1,393 notes collected from a random sample of patients with serious illness, two research coordinators abstracted each note using a similar definition of goals-of-care discussions, which identified no goals-of-care discussions. Because notes from the sample of palliative care specialist notes and from the random sample of patients with serious illness were not initially abstracted using the same protocols as the RCTs, two study investigators (PDT, RYL) re-abstracted 239 notes sampled from both corpuses using a study-specific protocol adapted from one of the RCTs (39) (Supplemental Appendix 1), finding excellent agreement between these notes’ initial designations and subsequent re-abstraction (94% agreement, kappa 0.87).

Natural Language Processing and Machine Learning

Following preprocessing of notes to remove non-textual templated content, we randomly partitioned the dataset into 100 pairs of training sets and test sets. Each training set consisted of a random sampling of 80% of the notes in the dataset, and corresponded to a test set consisting of the remaining 20% of notes in the dataset. We used NLP to tokenize each note into unigrams (i.e. tokens of 1 word length), excluding common stop words and negation terms. For each training set, we then trained an inductive supervised ML classifier, using the scikit-learn regularized logistic regression classifier with an L2 penalty and the default penalty parameter of 1, with a training feature set consisting of the count of each NLP unigram in the training set (43). Subsequently, we applied each trained ML classifier to its corresponding test set to compute the predicted probability of goals-of-care documentation for each note in the test set. A predicted probability of greater than (or less than) 0.5 was used to classify test notes as having (or not having) goals-of-care documentation for the purpose of estimating the sensitivity, specificity, and likelihood ratios of the NLP/ML classifier as compared to the clinician-abstracted gold-standard. We also summarized the performance of the NLP/ML classifier by the area under the receiver operating curve (AUC). This process was repeated for each of the 100 randomly partitioned sample sets. We reported summary statistics of performance metrics observed across all 100 random partitions.

To explore NLP/ML performance characteristics in the inpatient and outpatient settings, we separated the sample set into inpatient-only and outpatient-only sample sets, and then proceeded to train and test the NLP/ML program within these subsets using the same approach.

Software

All programming was performed in Python v3.6 (Python Software Foundation, python.org). Natural language processing was performed using the Natural Language Toolkit v3.2.4 (NLTK Project, nltk.org) (44), and machine learning was implemented using the scikit-learn library v0.19.0 (scikit-learn developers, scikit-learn.org) (43).

Results

The composition of the sample set is shown in Table 1. Overall, we identified 689 goals-of-care-positive notes across all four data sources, with 265 identified from prior RCTs, and 424 identified by note authors and confirmed by manual review. The remaining 2,494 notes in the dataset, including all of the 1,393 notes collected from a random sample of 300 seriously ill patients over a 27-month period, did not contain goals-of-care discussions. The sample set contained 1,435 inpatient notes, of which 468 contained goals-of-care discussions; and 1,748 outpatient notes, of which 221 contained goals-of-care discussions (Table 2).

Table 2.

Performance characteristics of NLP/ML by note type.

Note types	Sensitivity	Specificity	LR+	LR−	AUC
(note counts)	Mean (SD)	Mean (SD)	Median (IQR)	Median (IQR)	Mean (SD)
All notes	82.3% (3.2%)	97.4% (0.7%)	32.2 (27.5–39.2)	0.18 (0.16–0.20)	0.943 (0.015)
Inpatient notes (468 GOC+, 967 GOC−)	97.1% (1.8%)	99.4% (0.6%)	194.7 (175.7–∞) a	0.03 (0.02–0.04)	0.996 (0.005)
Outpatient notes (221 GOC+, 1527 GOC−)	52.0% (6.7%)	96.8% (1.2%)	17.7 (12.0–22.5)	0.51 (0.45–0.54)	0.826 (0.040)

^aIn the analysis of inpatient notes, 47 of 100 random splits resulted in 100% specific test-set performance, leading to an LR+ estimate of infinity.

Over 100 random partitions of the sample set, the observed sensitivity of NLP/ML ranged from 72–93%, with a mean sensitivity of 82.3% (SD 3.2%); and, the observed specificity of NLP/ML ranged from 95–99%, with a mean specificity of 97.4% (SD 0.7%). NLP/ML classifiers had a median positive likelihood ratio (LR+) of 32.2 (IQR 27.5–39.2, range 17.9–81.0), and a median negative likelihood ratio (LR–) of 0.18 (IQR 0.16–0.20, range 0.07–0.30). The AUC ranged from 0.91–0.98, with a mean AUC of 0.94 (SD 0.01). NLP/ML classifiers specific to inpatient records had better test performance compared to NLP/ML classifiers that were specific to outpatient records (mean sensitivity 97.1% vs. 52.0%; mean specificity 99.4% vs. 96.8%) (Table 2).

Discussion

By applying NLP/ML to a sample of inpatient and outpatient EHR notes, we were able to develop and train an automated program that identified documented goals-of-care discussions. While NLP techniques have been efficacious in identifying a variety of distinct clinical entities (23–30), identifying goals-of-care discussions—a broad construct most often represented by descriptive free-text rather than precise signifiers—poses unique challenges to machine approaches (45). Most published studies that report the use of NLP to identify goals-of-care discussions are rule-based and focus on specific diseases or care contexts (30, 32–36); two studies specifically examining ICU patients’ records have used ML to identify goals-of-care discussions (37, 38). Our work demonstrates the feasibility of NLP/ML as a potential approach for identifying goals-of-care discussions in the EHR across a broad range of clinical contexts. Additionally, our findings contribute to the growing application of NLP/ML for extracting increasingly complex constructs from the EHR (46–48).

Stakeholders in palliative care research have given high priority to metrics that identify discussions between patients, families and providers about goals of care and advance care planning (15, 16, 49). Although there has been progress in measuring advance directive documentation—a metric rife with its own challenges (14, 34)—doing so ultimately falls short of measuring documented goals-of-care conversations, which are often held outside the context of advance directives (9–11, 15). Our approach of using NLP/ML to identify documentation of goals-of-care discussions brings the palliative care research community closer to the goal of quantifying and characterizing goals-of-care discussions across diverse clinical settings. Potential applications of this approach include the identification of target populations for palliative care interventions who have not had documented goals-of-care discussions, measuring outcomes for interventions designed to increase goals-of-care discussions, providing data to guide and evaluate quality improvement programs to improve communication, and developing clinician-facing EHR tools to assist clinicians in locating prior goals-of-care documentation (14). Each of these potential applications may prioritize different performance characteristics to maximize utility. For example, programs that aim to locate goals-of-care notes for point-of-care clinicians might prioritize high positive predictive value to avoid wasting clinicians’ time, while programs to identify potential study participants without prior goals-of-care discussions might prioritize high negative predictive value to avoid excluding eligible patients.

We encountered challenges in deciding what constituted documentation of a goals-of-care discussion. We elected to adopt a broad-ranging definition that encompassed many common elements of advance care planning encountered during prior clinical trials of patients with serious illness (39, 40). We also elected to exclude stand-alone documentation of code status from our definition, as these data often do not reflect the conversations we aimed to measure. Because the appropriate definition of a goals-of-care discussion differs over the research question being asked, we propose that future approaches adopt multidimensional definitions that specify different elements and types of goals-of-care documentation in order to maximize the adaptability of the end product.

Notably, in the entirety of the most generalized corpus of notes sampled—consisting of 1,393 notes authored over a 27-month period for a random sample of 300 patients with serious illness—we found no notes containing goals-of-care discussions (Table 1). This contrasted with the prevalence of goals-of-care discussions among patients enrolled in randomized trials of communication interventions (39, 40), and suggests that HER-documented goals-of-care discussions are quite rare even among patients with serious illness. This finding highlights both a clinical opportunity for improvement, and the utility of an automated approach to identifying goals-of-care documentation.

Our study has several limitations. First, our data were purposively sampled to enrich for goals-of-care documentation, and are not representative of EHR notes from a random sample of patients with serious illness. Although sensitivity, specificity, likelihood ratios, and AUC are often assumed to be inherent characteristics of the test, our observed performance is likely to be better than expected real-world performance in the general EHR. Second, because the dataset was purposively sampled, we are unable to report prevalence-dependent performance characteristics such as positive or negative predictive values. Owing to the low prevalence of goals-of-care discussions in the EHR, even algorithms with very high positive likelihood ratios may not achieve satisfactory positive predictive value. Third, our algorithm may have selected predictors unlikely to be associated with goals-of-care documentation in other samples or settings, leading to overly optimistic estimates of performance. Because our training data did not specifically identify which words or phrases of a given note reflected goals-of-care documentation, the algorithm may have selected predictors whose association with the outcome was specific to our data. Additionally, because our training data incorporated notes from four unique data sources, the algorithm may have selected predictors (e.g. predictors of RCT participation) whose association with goals-of-care documentation is mediated by the data source. Future studies should consider employing word- or phrase-level annotation of training data (e.g. identifying the specific portions of a note containing goals-of-care documentation), constructing ontologies to describe goals-of-care discussions, and exploring novel approaches to address biases in training data that differ from those present in real-world data (50). Fourth, because our sample was limited to notes from a single academic health system, our algorithm may not generalize to other health systems. Although this challenge is shared by many clinical applications of NLP/ML, ongoing advances in novel computational linguistic techniques are likely to facilitate generalization of NLP/ML to settings beyond the source of training data (51), which holds great promise for this application of NLP/ML. Fifth, our NLP/ML program demonstrated only modest performance in the analysis of outpatient records, suggesting that there is room for improvement. We hypothesize that goals-of-care discussions in the inpatient setting are more likely to reflect “in-the-moment” treatment decisions (52) occurring during medical crises or near the end of life; documentation of these discussions may be lengthier than outpatient advance care planning discussions, and may be easier for NLP/ML to identify. Despite these limitations, we believe our findings provide an important proof of concept for the use of NLP/ML in identifying goals-of-care discussions and provide direction for future studies to improve this approach.

Conclusion

Using natural language processing and machine learning techniques, we developed and evaluated a novel approach to identifying EHR documentation of goals-of-care discussions with performance characteristics that suggest proof of concept as well as room for improvement. Further study is needed to refine and validate this approach, particularly for identifying outpatient goals-of-care discussions. NLP and ML represent a novel and exciting approach toward measuring goals-of-care discussions as a research outcome and as a quality metric for palliative care.

KEY MESSAGE.

In this pilot study, natural language processing and machine learning achieved 82% sensitivity and 97% specificity for identifying documented goals-of-care discussions in medical records of patients with serious illness. The results suggest the potential of automated approaches to measuring goals-of-care discussions as a research outcome and quality metric.