Mixed-methods evaluation of three natural language processing modeling approaches for measuring documented goals-of-care discussions in the electronic health record

Abstract

Context:

Documented goals-of-care discussions are an important quality metric for patients with serious illness. Natural language processing (NLP) is a promising approach for identifying goals-of-care discussions in the electronic health record (EHR).

Objectives:

To compare three NLP modeling approaches for identifying EHR documentation of goals-of-care discussions and generate hypotheses about differences in performance.

Methods:

We conducted a mixed-methods study to evaluate performance and misclassification for three NLP featurization approaches modeled with regularized logistic regression: bag-of-words (BOW), rule-based, and a hybrid approach. From a prospective cohort of 150 patients hospitalized with serious illness over 2018–2020, we collected 4,391 inpatient EHR notes; 99 (2.3%) contained documented goals-of-care discussions. We used leave-one-out cross-validation to estimate performance by comparing pooled NLP predictions to human abstractors with receiver-operating-characteristic (ROC) and precision-recall (PR) analyses. We qualitatively examined a purposive sample of 70 NLP-misclassified notes using content analysis to identify linguistic features that allowed us to generate hypotheses underpinning misclassification.

Results:

All three modeling approaches discriminated between notes with and without goals-of-care discussions (AUC_ROC: BOW, 0.907; rule-based, 0.948; hybrid, 0.965). Precision and recall were only moderate (precision at 70% recall: BOW, 16.2%; rule-based, 50.4%; hybrid, 49.3%; AUC_PR: BOW, 0.505; rule-based, 0.579; hybrid, 0.599). Qualitative analysis revealed patterns underlying performance differences between BOW and rule-based approaches.

Conclusion:

NLP holds promise for identifying EHR-documented goals-of-care discussions. However, the rarity of goals-of-care content in EHR data limits performance. Our findings highlight opportunities to optimize NLP modeling approaches, and support further exploration of different NLP approaches to identify goals-of-care discussions.

INTRODUCTION:

Goals of care have been defined as the central aims of medical care for a patient, which are informed by their values and goals and used to guide clinical decisions.¹ High-quality, patient-centered goals-of-care discussions and their documentation are important components of care for patients with serious illness.^2–5 Goals-of-care discussions are associated with improved patient- and family-centered outcomes,^4,6 increased goal-concordant care,^7,8 lower healthcare costs,⁹ and lower intensity of end-of-life care.^4,10 However, goals-of-care discussions often occur late in the course of serious illness or do not occur at all.^11–13 Although there is ongoing debate about the value of advance care planning,^14,15 the occurrence and quality of goals-of-care discussions for patients with serious illness remains an important quality metric and research outcome for palliative care.¹⁶

The electronic health record (EHR) presents rich opportunities for examining process and outcome measures in palliative care, including the occurrence and quality of documented goals-of-care discussions.^17–19 However, because such documentation is typically written as unstructured free-text, it is often challenging to extract this data from the EHR.^19–21 Manual review—the current “gold standard”—is costly and profoundly time-intensive.^22,23 Natural language processing (NLP), which uses computational techniques to identify constructs in free-text data,²⁴ is a promising approach for measuring outcomes using real-world EHR data.^25,26 NLP is often combined with supervised machine learning (ML), wherein predictive models are trained to recognize labeled free-text examples of a given construct.²⁷ NLP has been used to assess quality indicators in cancer patients undergoing palliative surgery²⁸ and to measure documentation of ICU patients’ care preferences.²⁹

Our research group recently developed an NLP algorithm to measure documented goals-of-care discussions in a purposively enriched dataset of EHR notes.³⁰ However, the optimal modeling approach for this task and performance of NLP in data collected from a prospective cohort have not been reported.³⁰ To inform future development, we conducted a hypothesis-generating mixed-methods study of three different NLP modeling approaches for identifying documented goals-of-care discussions in a large dataset of EHR notes collected from a prospective cohort of patients hospitalized with serious illness.

METHODS:

Data Source

This study utilized data from a randomized trial, Project to Improve Communication About Serious Illness – Pilot Study,³¹ which enrolled 150 hospitalized patients of age ≥ 18 with chronic life-limiting illness³² between December 2018 and February 2020 at two large teaching hospitals at UW Medicine. Both the trial and this study were approved by the University of Washington Institutional Review Board (STUDY00004821, STUDY00011002).

All EHR notes for the enrollment hospitalization authored by a physician or advance practice provider were coded using a standardized protocol to identify documented goals-of-care discussions (Appendix I).³³ Although EHR metrics for palliative care and serious illness communication have been described in various settings,^{19,28,34–37} there is presently no consensus on how to identify documented goals-of-care discussions in the inpatient setting. Available evidence suggests that goals-of-care discussions and their documentation exist along a spectrum, from routine code status verification to detailed explorations of patients’ values and goals as they relate to end-of-life care.^38–42 Toward the aim of the parent trial (which examined the effects of a communication-priming intervention³¹), we adopted a pragmatic definition of documented goals-of-care discussions for this study that captured goals-of-care discussions as typically documented by both palliative care and non-palliative care clinicians, whilst excluding routine admission code status discussions. Using the framework for serious illness communication adopted by the American College of Physicians High Value Task Force and other groups,^1,5,41–44 we pragmatically defined documented goals-of-care discussions for this study as documentation of conversations with patients or surrogate decision-makers that either (1) explored a patient’s overarching values and goals for the purpose of guiding medical decision-making, or (2) explored a patient’s preferences about specific life-sustaining treatments or strategies relevant to their serious illness (such as mechanical ventilation, chemotherapy, or transfer to intensive care). As verification and documentation of code status is ubiquitous among patients admitted to UW Medicine medical services (e.g. “Full code”, “DNR/intubation OK”, “DNR/DNI, confirmed by [surrogate] on admission”), we only considered code status documentation to represent a documented goals-of-care discussion if it included additional documentation of values, goals, or preferences about treatments other than those offered for cardiopulmonary arrest. Discussions with inpatient physicians that led to the completion of new advance care planning or POLST documents while hospitalized, or that led to referral to specialty palliative care to further clarify goals of care, were classified as “implied or impending goals-of-care discussions” and coded separately; notes with these codes were also classified as “positive” to facilitate detection of discussions that might not have been documented separately. For this initial coding, four trained chart abstractors met twice-weekly with investigators over a three-month period to discuss and review coded goals-of-care discussions; all instances of disagreement were resolved by consensus. Randomization was concealed from abstractors until data collection was complete.

Natural Language Processing

Following preprocessing of notes to remove non-textual templated content, we developed three different featurization approaches for use in supervised machine learning: (1) bag-of-words (BOW) models, in which each note is represented by the frequency distribution of words that comprise it, weighted by term frequency–inverse document frequency (tf–idf);^45,46 (2) rule-based models, in which each note is represented by binary matches with a series of expert-defined textual search patterns indicative of goals-of-care documentation, such as the term “goals of care” (Appendix II); and, (3) hybrid models, which concatenate features from BOW- and rule-based approaches.^27,47 Each featurization approach was then used to train and test supervised ML classifiers using logistic regression with L2 regularization, in a leave-one-group-out cross-validation procedure (LOGO-CV) detailed below.⁴⁸ Within each LOGO-CV fold, we used five-fold cross-validation to optimize the regularization parameter over a grid search toward maximal area under the precision-recall curve.⁴⁹ Each leave-one-group-out model was then refit to the entirety of its available training data with the optimized hyperparameter value and used to predict the held-out patient’s data. NLP preprocessing and term-weighting were performed using the Natural Language Toolkit v3.2.4 (NLTK Project, nltk.org).

Quantitative Analysis

NLP models for classifying individual clinical notes were trained and tested using leave-one-group-out cross-validation (LOGO-CV).⁴⁸ In this approach, 150 unique ML classifiers were trained on notes from 149 patients and tested on all notes from the patient left out of the training set. Machine-predicted classification scores for each patient-test set were pooled across all 150 folds to facilitate performance analyses. Pooled classification scores were compared against the human-abstracted gold standard to estimate sensitivity (recall), specificity, positive predictive value (PPV; precision), negative predictive value (NPV), area under the receiver operating characteristic curve (AUC_ROC), and area under the precision-recall curve (AUC_PR). Although conventional diagnostic tests are often evaluated using the AUC_ROC (which summarizes sensitivity and specificity across the range of discrimination thresholds), tests designed to identify rare outcomes require particularly high specificity to achieve useful PPV.⁵⁰ For such tests, the informativeness of the AUC_ROC may be suboptimal, as much of the area measured represents discrimination thresholds of no practical utility. Accordingly, we report and analyze PPV (precision), sensitivity (recall), and area under the precision-recall curve (AUC_PR) for each approach.^49,51,52

To evaluate performance at the level of the patient-hospitalization (in which each patient is classified as having, or not having, a documented goals-of-care discussion during the index hospitalization), we compared the union of each patient’s note-level ML predictions to the union of their human-abstracted note classifications in a similar fashion.

Machine learning was implemented using the scikit-learn library v0.19.0 (scikit-learn developers, scikit-learn.org).⁵³ Receiver operating characteristic (ROC) and precision-recall (PR) analyses were conducted using Stata v17.0 (StataCorp, stata.com) and the prcurve package.⁴⁹

Qualitative Analysis

For qualitative analysis of machine-misclassified notes, we reviewed notes misclassified by rule-based and BOW approaches. We did not examine notes misclassified by the hybrid approach because it combines features, and likely error patterns, of the other two approaches. We purposively sampled a corpus of 60 misclassified notes (30 false-positive and 30 false-negative) ensuring diversity of misclassifying modeling approach (BOW vs. rule-based vs. both), note type, and number of unique patients represented. We also examined 10 true-positive notes correctly classified by both approaches for comparison. Toward the aim of generating hypotheses underlying misclassification, we used content analysis to examine potentially-explanatory note characteristics and linguistic features, using investigator triangulation to ensure trustworthiness.^54,55 Based on the underlying theory of existing frameworks in serious illness communication^{1,5,33,41–44,56,57} augmented by immersive review of an initial corpus of 10 notes, investigators (AU, RYL, JRC) created an initial set of structured codes representing known aspects of documented goals-of-care discussions and the semantics that are used to document them; these codes were then iteratively revised through a process of consensus into a coding framework (Appendix III) that was applied to the entire corpus of notes. Coded data were then iteratively reviewed alongside note characteristics and NLP predictions to generate hypotheses underlying performance differences between modeling approaches. A subset of 18 notes (25.7%) were co-coded independently by another investigator (JT) to ensure reproducibility of the codes with over 90% agreement. Disagreements were resolved through consensus.

RESULTS:

Table 1 summarizes the baseline characteristics of the 150 hospitalized patients enrolled in this study. Patients had a median age of 61 years; 56% were male, and 71% were non-Hispanic and white. Using automated EHR queries, we amassed a dataset of 4,391 clinical notes corresponding to each patient’s entire enrollment hospitalization. Among these 4,391 notes, 99 notes (2.3%) for 32 patients contained documented goals-of-care discussions identified by manual review.

Table 1.

Baseline characteristics of prospective cohort from which data was collected

Characteristic	Total sample (N=150)
Patient age, years, median (IQR)	61 (16)
Patient sex, n (%)
Male	84 (56)
Female	66 (44)
Race/ethnicity, n (%)	(n=149)
White, non-Hispanic	107 (71)
White, Hispanic	9 (6)
Black	19 (13)
Asian	2 (1)
Native American	2 (1)
Mixed race/ethnicity	10 (7)
No. of chronic life-limiting illnesses, n (%)
0 a	10 (7)
1	64 (43)
2	34 (23)
3	21 (14)
≥ 4	21 (14)
Chronic life-limiting illnesses (not mutually exclusive)
Cancer with poor prognosis	25 (17)
Chronic lung disease	46 (31)
Coronary artery disease	43 (29)
Congestive heart failure	52 (35)
Peripheral vascular disease	21 (14)
Chronic kidney disease, moderate-to-severe	56 (37)
Chronic liver disease, severe	25 (17)
Diabetes with end-organ damage	21 (14)
Dementia	6 (4)

Abbreviations: IQR, interquartile range.

^aPatients greater than 80 years of age were included in the study regardless of presence or absence of a chronic life-limiting illness.

Quantitative Analysis

Table 2a presents note-level performance metrics for BOW, hybrid, and rule-based modeling approaches as estimated by LOGO-CV at a nominal sensitivity (recall) of 70%. In ROC curve analysis, all three algorithms demonstrated ability to discriminate for presence of goals-of-care content (AUC_ROC: BOW, 0.907; rule-based, 0.948; hybrid, 0.965; Figure 1a). However, analysis of PPV (precision) and sensitivity (recall) demonstrated a marked trade-off between these two performance characteristics (Figure 1a). At thresholds favoring PPV over sensitivity, BOW demonstrated greater PPV than rule-based or hybrid approaches; whereas, at thresholds favoring sensitivity over PPV, hybrid and rule-based approaches demonstrated greater PPV than the BOW approach. The area under the precision-recall curve (AUC_PR), which summarizes PPV (precision) and sensitivity (recall) across discrimination thresholds,^58,59 was modestly higher for the hybrid approach (0.599) than the BOW (0.505) and rule-based (0.579) approaches.

Table 2.

Performance characteristics of NLP/ML by modeling approaches

(a) Note-level classifiers, N = 4,391 (99 positive)
Modeling approach	AUCROC a	AUCPR b	Performance at nominal sensitivity [recall] of 70% c
			Sensitivity [Recall]	Specificity	PPV [Precision]	NPV
BOW	0.907	0.505	0.697	0.917	0.162	0.992
Hybrid	0.965	0.599	0.697	0.983	0.493	0.993
Rules	0.948	0.579	0.707	0.984	0.504	0.993
(b) Patient-level classifiers, N = 150 (32 positive)
Modeling approach	AUCROC a	AUCPR b	Performance at nominal sensitivity [recall] of 70% c
			Sensitivity [Recall]	Specificity	PPV [Precision]	NPV
BOW	0.927	0.817	0.688	0.949	0.786	0.918
Hybrid	0.932	0.796	0.688	0.915	0.688	0.915
Rules	0.903	0.768	0.688	0.932	0.733	0.917

Abbreviations: BOW, bag-of-words; AUC, area under the curve; ROC, receiver operating characteristic curve; PR, precision-recall curve; PPV, positive predictive value; NPV, negative predictive value

^aAUC_ROC is calculated using non-parametric analysis of pooled LOGO-CV classifier predictions.

^bAUC_PR is numerically calculated from pooled LOGO-CV classifier predictions using cubic splines.⁴⁹

^cSensitivity [recall], specificity, PPV [precision] and NPV vary over discrimination thresholds (see Figure 1). Performance metrics are shown for the observed discrimination threshold at which observed sensitivity [recall] is closest to the specified value of 70%; additional values at other thresholds are presented in Supplementary Table S1.

Figure 1.

Receiver operating characteristic and precision-recall curves, by modeling approach.

Table 2b presents performance metrics for patient-level classifiers defined by the union of a patient’s note-level predictions. In examining modeling approaches at the patient level, the three approaches demonstrated comparable performance (Figure 1b).

Qualitative Analysis

We assembled a purposive sample of 70 notes, representing 34 patients. Of these 70 notes, 43 were primarily authored by a resident physician or fellow, 20 by an attending physician, and 7 by nurse practitioners or physician assistants. Among the 40 sampled notes with a documented goals-of-care discussion, 17 commented on specific treatment preferences, 8 commented on overarching values and goals for medical care, and 5 commented on a patient’s global understanding of their illness; 13 contained a new referral to subspeciality palliative care, hospice, or comfort care, and 7 indicated discussion or completion of a new advance care planning document. Among the 30 sampled notes that lacked a documented goals-of-care discussion (i.e. notes misclassified by NLP as false-positive), 13 notes referenced a prior goals-of-care discussion and 11 notes referenced a planned goals-of-care discussion that had not yet occurred.

Among notes containing goals-of-care documentation that were missed by one or both NLP modeling approaches, four patterns potentially underlying misclassification emerged from our review (Figure 2a):

Figure 2.

Qualitative features of NLP-misclassified notes.

Values and goals are a key signal for BOW- and rule-based identification of goals-of-care discussions.

Although many documented goals-of-care discussions in our dataset contained clear descriptions of patients’ values and goals, there were others in which documentation focused on exchange of prognostic information and decisions regarding preferences for life-sustaining treatments or strategies. Both BOW and rule-based modeling approaches misclassified many notes that omitted language describing the values and goals that would ideally inform such treatment decisions.

“Long discussion with patient at bedside. He had many appropriate questions. We discussed plan of care and estimated trajectory. Ultimately, he is concerned that he will not get better and that there is nothing else we can do to help him. I provided education and discussed options like [treatment] should that become necessary. He was reassured by this.” (Hematology-oncology progress note, false-negative by both BOW and rule-based models)

Discussions about advance care planning documents are difficult to capture by BOW- and rule-based NLP.

We observed in our data that goals-of-care discussions that centered on advance care planning documents, such as Physician Orders for Life Sustaining Treatment (POLST), were difficult to identify by both BOW and rule-based approaches.

“POLST filled out in conversation with patient and her friend. Confirmed DNR, limited additional interventions, antibiotics when life can be extended, and long-term feeding tube if needed.” (Medicine progress note, false-negative by both BOW and rule-based models)

Although terms indicative of advance care planning documents were represented in our rule-based and hybrid models (Appendix II), we observed that terms such as “POLST” (Physician Orders for Life-Sustaining Treatment), “advance directive”, and “DPOA” (Durable Power of Attorney) were frequently present in EHR passages that were not intended to document goals-of-care discussions. Often, these terms occurred in standardized checklists at the end of daily progress notes, such as the following example:

“VTE prophylaxis:

Risk Level: at risk

Treatment(s) ordered: enoxaparin

Contacts: [name] DPOA, sister: [phone number]

Code status: DNR. Will need POLST on discharge to SNF”

(Medicine progress note, true-negative by both BOW and rule-based models)

Widespread use of these terms in the EHR may have limited their specificity in identifying documented goals-of-care discussions.

BOW models miss goals-of-care content embedded within lengthy records.

Many goals-of-care discussions were documented in interim and discharge summaries, presumably to assist in communication with other healthcare providers. Although this type of documentation was often quite detailed, and inclusive of key phrases such as “goals of care”, BOW models often missed such content when it was embedded in lengthy records that were predominated by non-goals-of-care content. Our rule-based models were less susceptible to this type of misclassification. In the following example, the goals-of-care discussion comprises 3% of the words in this 3600-word note.

“Goals of care: A GOC meeting was held with [patient], her partner of 30 years, and the renal fellow. [Patient] expressed a desire to maintain a life-prolonging approach to care. She would like to spend as much time as possible with her partner... They enjoy being together at home, playing games and watching TV. She also asked that we prioritize pain management.” (Medicine interim summary, false-negative by BOW models, true-positive by rule-based models)

Our rule-based models had difficulty identifying code status discussions substantiated by values or goals.

In our data, there were many instances of documentation focused closely on treatment preferences for cardiopulmonary arrest, often isolated from the acute clinical context. These discussions reflect the current standard of confirming patients’ code status upon admission, but the extent to which their documentation is substantiated by the patient’s underlying values and goals is highly variable. As these discussions were rarely labeled in the EHR as “goals of care” or “family meetings” by note authors, rule-based models often misclassified such notes as false-negative.

“She has done extensive thinking about intubation and does not want to be intubated if her respiratory status worsens, even it if means dying. She wishes to do everything possible to prevent getting to that point including escalating respiratory support and antibiotics. If she worsens despite maximal respiratory and medical support, she wishes to be made comfortable and allowed to die peacefully.” (Geriatrics consult note, false-negative by rule-based models, true-positive by BOW models)

Among notes misclassified as false-positive by one or both NLP approaches, two patterns emerged in our qualitative review that applied to both modeling approaches (Figure 2b):

References to the past or the future are hard to distinguish using BOW- or rule-based NLP.

Because the evaluated modeling approaches are largely naïve to grammar, both the BOW and rule-based models over-called many planned goals-of-care discussions and references to previous discussions.

“Planning to have meeting with the patient and her partner on [date]. Will discuss patient goals, trajectory, and ongoing concerns.” (Medicine progress note, false-positive by both BOW and rule-based models)

BOW- and rule-based NLP misidentified related constructs that do not pertain to overarching goals of care.

Both BOW and rule-based models over-called notes that documented closely related concepts such as shared decision-making or trade-offs between treatments and side effects.

“Thinks his pain is under adequate control today and continues to voice his preference for enough medication to allow him to feel comfortable enough to go for walks, but not so much that he is ‘dopey’.” (Palliative Care progress note, false-positive by both BOW and rule-based models)

DISCUSSION:

Reliable and automated identification of goals-of-care discussions documented in the EHR may provide a powerful tool for research and quality improvement. Through this mixed-methods comparison of three NLP/ML approaches, we identified salient differences in performance and generated hypotheses about the patterns underlying these differences. One of the key reasons why NLP is potentially useful for measuring documented goals-of-care discussions is that such documentation is very infrequent in the EHR. In our real-world sample of 4,391 clinical notes collected from hospitalized patients with serious illness, only 99 (2.3%) contained goals-of-care content by manual review. The rarity of goals-of-care content in the EHR poses unique challenges for the development and evaluation of NLP algorithms designed to measure it.⁶⁰ Our quantitative findings are also an illustrative example of how the widely-recognized metric of AUC_ROC may not fully represent real-world performance of classifiers designed to identify rare outcomes.^49,51,52

Our study does not prove the superiority of any single modeling approach that we examined. However, our observation that hybrid models had the highest note-level AUC_ROC and AUC_PR supports ongoing efforts to further develop and optimize approaches that combine machine-derived features and expert-defined rules.²⁷ When compared to rule-based methods, NLP models that use machine-derived features have the theoretical advantage of being able to identify predictive features that extend beyond the imagination of human experts, thereby improving performance. However, training data for rare constructs such as documented goals-of-care discussions is both sparse and costly to acquire. In these situations, a hybrid approach that combines machine-derived features with expert-defined rules allows the designer to specify heuristics that can improve both performance and generalizability. For these reasons, hybrid NLP modeling approaches are becoming increasingly common in the biomedical sciences.⁶¹

We qualitatively identified several patterns among notes misclassified by the rule-based and BOW modeling approaches, which suggest potential strategies for improving performance. Both BOW and rule-based approaches missed goals-of-care discussions that lacked clear documentation of patients’ values and goals. As many such discussions centered around exchange of prognostic information or preferences about life-sustaining treatments, incorporation of additional linguistic features into rule-based and hybrid approaches (e.g. words indicative of life-sustaining treatments located within contexts suggestive of patient/family communication) may help capture documentation of these discussions. Both BOW and rule-based modeling approaches also missed discussions about advance care planning documents completed in the hospital, as many inpatient notes contained templated, context-free references to such documents (e.g. “POLST 12/14/10”) without indication of an incident goals-of-care discussion. Ideally, this would be addressed by the creation of EHR structures or registries that capture incident advance care planning documents, obviating the need for NLP to measure this outcome. BOW models also frequently missed goals-of-care discussions embedded within lengthy notes, which demonstrates the limitations of classifying EHR text using note-level word frequencies. This deficiency may be addressed by changing the unit of analysis: analyzing text at the paragraph, sentence, or phrase level may facilitate accurate detection of clustered positive predictors within lengthy records.

It was reassuring that many of the notes in the false-positive sample contained documentation of constructs related to goals-of-care, even if this content did not meet our outcome definition. Additionally, while both modeling approaches often misclassified references to past and future goals-of-care discussions, this type of misclassification may be less problematic in applications where outcomes are aggregated over longer periods of time. Notably, several notes across both the false-negative and false-positive samples contained goals-of-care content that was difficult for human reviewers to reliably characterize, even through a process of consensus. This finding highlights the inherent challenges in dichotomizing the presence or absence of goals-of-care discussions in the EHR, as actual goals-of-care conversations are multifaceted and vary over dimensions of context, timing, depth, content, and execution.

Although the methods we evaluated are common and accessible to many clinical NLP developers,^26,27 more sophisticated NLP approaches such as deep learning may address many of the challenges we describe. In deep learning-based NLP, words are represented mathematically as embeddings sensitive to position, syntax, and semantics (all aspects that are only partially representable in rule-based NLP, and not represented in BOW); and, models are constructed using neural network-like architectures that rely on fewer mathematical assumptions than traditional ML methods.^62,63 Deep learning-based NLP has been deployed in a small number of studies that measure palliative care outcomes, including one group’s broader definition of documented serious illness conversations which includes standalone documentation of code status.^29,35,64 We believe that the parallels between several of our key findings and the theoretical advantages of deep learning supports exploration of deep learning-based NLP for this task. However, because adopting deep learning-based approaches incurs substantial costs—larger quantities of training data, and more extensive computational resources (the most notable of which is model training time)—we also believe that future studies should quantitatively compare the performance, generalizability, and scalability of deep learning approaches to traditional NLP/ML approaches for identifying EHR documentation of goals-of-care discussions.⁶³

Our study has several limitations. First, our dataset of over 4,000 notes contained only 99 goals-of-care discussions for 32 patients, which aligns with previous studies showing a low prevalence of goals-of-care discussions in hospitalized patients with serious illness.^13,34,65 This small sample of positive notes may not reflect the diversity of goals-of-care documentation in the EHR, which may threaten generalizability. The small sample of positive notes also precluded testing of a single trained ML classifier on a held-out test set. While the LOGO-CV approach is well-accepted, predictions pooled from distinct LOGO-CV classifiers may lead to biased estimates of performance.⁶⁶ Second, our sample of notes was derived from participants enrolled in a clinical trial at two hospitals in a single academic health system. Because the prevalence and content of goals-of-care documentation differs across patient populations and health systems, our findings may not generalize to other patient populations or health systems. This challenge is shared by many clinical NLP applications and reinforces the need both for external validation and for novel methods that address bias and facilitate generalization of NLP algorithms to settings beyond the training data.^67,68 Third, our findings are specific to the NLP methods we evaluated, and may not generalize to other NLP featurization or modeling methods. Finally, while our qualitative analysis revealed potential etiologies for differences in performance between NLP modeling approaches, our findings are hypothesis-generating and should be confirmed by further study.

CONCLUSION:

Through this mixed-methods comparison of three NLP modeling approaches to identify EHR documentation of goals-of-care discussions, we observed differences in performance and generated hypotheses about misclassification patterns underlying these differences. Our quantitative findings suggest that future efforts using NLP to identify rare outcomes should continue to explore hybrid approaches, which may have advantages over BOW and rule-based approaches, particularly in smaller datasets. Our qualitative findings support incorporation of additional linguistic features that identify information exchange and treatment preferences, and suggest that refining the unit of analysis may lead to performance gains. Our findings also support exploration of deep learning-based NLP strategies to improve performance. Finally, our findings highlight the inherently multi-dimensional nature of goals-of-care discussions. We believe that NLP continues to represent a promising approach towards measuring documented goals-of-care discussions as a quality metric and research outcome.

KEY MESSAGE:

This study suggests NLP has potential for measuring EHR documentation of goals-of-care discussions for quality improvement and research. We use mixed methods to generate hypotheses for how to further optimize NLP modeling approaches toward this task.