Abstract
Advance care planning discussions seek to guide future serious illness care. These discussions may be recorded in the electronic health record by documentation in clinical notes, structured forms and directives, and physician orders. Yet, most studies of advance care planning prevalence have only examined structured electronic health record elements and ignored data existing in notes. We sought to investigate the relative comprehensiveness and accuracy of ACP documentation from structured and unstructured electronic health record data sources. We evaluated structured and unstructured advance care planning documentation present in the electronic health records of 435 patients with cancer drawn from three separate healthcare systems. We extracted structured advance care planning documentation by manually annotating written documents and forms scanned into the electronic health record. We coded unstructured advance care planning documentation using a rule-based natural language processing software that identified advance care planning keywords within clinical notes and was subsequently reviewed for accuracy. The unstructured approach identified more instances of advance care planning documentation (238, 54.7% of patients) than the structured advance care planning approach (187, 42.9% of patients). Additionally, 16.6% of all patients with structured advance care planning documentation only had documents that were judged as misclassified, incomplete, blank, unavailable, or a duplicate of a previously entered erroneous document. Advance care planning documents scanned into electronic health records represent a limited view of advance care planning activity. Research and measures of clinical practice with advance care planning should incorporate information from unstructured data.
Keywords: Advance Care Planning, Electronic Health Records, Palliative Care, Research Design, Natural Language Processing
INTRODUCTION
Many health care systems use documentation of advance care planning (ACP) activity as a measure of quality for serious illness care. Relatedly, the Centers for Medicare & Medicaid Services has created billing codes for ACP and incorporated them into the Healthcare Effectiveness Data and Information Set measures.(1, 2, 3) Yet, despite considerable effort to increase and accurately measure ACP activity, few studies have compared methods for evaluating its prevalence in the electronic health record.
Advance care planning, the “process that supports adults at any age or stage of health in understanding and sharing their personal values, life goals, and preferences regarding future medical care,” endeavors to “help ensure that people receive medical care that is consistent with their values, goals, and preferences during serious and chronic illness.”(1) This definition reflects the evolution of ACP from its traditional focus on the completion of advance directive documents and orders towards a practice that fosters care consistent with patient goals through ongoing support and communication.(4, 5, 6)
This welcome change emphasizing a comprehensive communication process also poses a challenge for ACP research and quality improvement. The traditional method for ascertaining ACP documentation rates relies on easily accessible structured EHR fields, such as standardized documents and forms scanned into the EHR or the use of billing codes. While such documentation is an important source of information about a patient’s wishes or ACP conversations that may have occurred, they only capture preferences at one time point, providing an incomplete and sometimes inaccurate picture of the ACP process.(7, 8) Additionally, such documentation often occurs only late in the disease process and may not capture earlier, and potentially more effective, communication practices which could do more to align end of life care with patients’ goals.(9, 10, 11, 12, 13, 14) Further, previous work has shown that patients with completed structured documents and orders often lack accompanying documentation of any ACP discussion.(15) As such, the presence of an ACP form attached to the EHR may not be a reliable method for determining whether high quality ACP has occurred.
ACP documentation found within unstructured or semi-structured EHR fields, such as clinical notes, may contain the most up-to-date, richest information about the ongoing communication regarding patients’ goals and values. However, for clinicians, researchers, and policymakers alike, such unstructured documentation may be too disorganized and time-consuming to review.(7, 15, 16, 17, 18) With modern processing power, it has become feasible to use rule-based natural language processing (NLP) to expeditiously identify and extract ACP documentation from a large number of clinical notes.(7, 19, 20, 21, 22, 23). While the ability of NLP to ascertain ACP conversations in the free-text of clinical notes is limited by what is documented in the notes and is not integrated into clinical workflows, the rapid review of free-text facilitated by NLP may better capture a patient’s ongoing ACP process.(24)
To better understand how different types of ACP documentation in the EHR may capture different aspects of ACP (defined broadly as above) and therefore influence the estimated prevalence of ACP across a patient population, we compared standardized ACP documentation from scanned forms to free-text ACP documentation ascertained through rule-based NLP free-text analysis of clinical notes in a sample of patients from three U.S. healthcare systems participating in a multisite pragmatic trial.(22)
METHODS
Sample and Data Sources
The patients whose data was included in this project were those seen in oncology clinics during the pilot period (March 2018-March 2019) of the Advance Care Planning: Promoting Effective and Aligned Communication for the Elderly (ACP PEACE) clinical trial (UG3AG060626). ACP PEACE is a pragmatic, stepped-wedge cluster randomized trial which examines the impact of a communication skills training and patient video decision aids on rates of ACP documentation in the EHR. The pilot period—which consisted of a six-month usual care period and a six-month intervention period—was necessary to develop processes and organizational structure to support the ACP PEACE trial and pilot test the study intervention.(22) Adults aged 65 and older at the time of their visit to one of the study clinics and who were identified to have advanced cancer (utilizing International Classification of Disease codes) were eligible for inclusion in the ACP PEACE trial.(22) All patients who met the eligibility criteria were included in the trial, and informed of their participation via broadcast notifications, unless they opted-out. ACP documentation was extracted from each health system’s EHR. Of the three large US healthcare systems that were involved in the study, two of the healthcare systems used Epic (Madison, WI) and the other used AllScripts (Chicago, IL) for the outpatient setting and Sunrise for the inpatient setting.
This work was approved by the Institutional Review Board at Dana-Farber Cancer Institute as an analysis for the ACP PEACE trial.(22) The Standard for Quality Improvement Reporting Excellence (SQUIRE 2.0) guidelines were followed.(25)
Standardized ACP Documentation
For purposes of this paper, standardized ACP documentation refers to forms and documents which included ACP (e.g., a structured data tab for an advance directive or health care proxy form) recorded in the EHR prior to the end of the pilot period. To identify standardized ACP documentation, experts in palliative care, oncology, and informatics from each site’s research and clinical teams identified the locations within the EHR where these documents are stored.(8) A full description of the methods used to identify standardized ACP documentation is detailed in previously published paper, though a brief overview is present below.(8)
Preidentified instances of standardized ACP documentation were extracted from the EHR and reviewed by local research teams to determine accuracy. Documents were then classified into three categories: (1) “Correct”— those that accurately reflected the structured information included in the EHR; (2) “Incorrect”— those which did not reflect the structured information included in the EHR (e.g., a form was left incomplete but scanned into the EHR); (3) “Duplicate”— the same document was scanned more than once into the EHR.(8) Documents sorted into the “Correct” category included: Advance Directive/Description of End-of-Life Wishes, Medical Orders for Life Sustaining Treatment/Out of Hospital Code Status, Post-Mortem Instructions, and Health Care Proxy forms. Documents sorted into the “Incorrect” category includes those which were determined to be Incomplete (blank or partially completed), Reported as Asked, but Not Complete, Reports as Available, but Document not Present, or a Wrong Document (i.e., Consent Form, Procedural Safety Checklist).
The final dataset of standardized ACP documentation included information regarding the name, type, and accuracy of all standardized ACP documents included in the EHR for the patient population.
Free-Text ACP Documentation
In this paper, free-text ACP documentation refers to documentation in the free-text of clinical notes regarding goals-of-care conversations, hospice use, palliative care involvement, and limitations on life-sustaining treatments (Supplemental Table 1). Clinical notes from both inpatient and outpatients authored during the study pilot period were considered. Examples of reviewed notes include history and physical, consult, progress, and discharge notes. Free-text ACP documentation was ascertained through the use of ClinicalRegex (https://lindvalllab.github.io/clinical-regex/), a regular expression NLP software developed by the Lindvall Lab at Dana-Farber Cancer Institute in Boston, Massachusetts.(26) Performance and validation of ClinicalRegex’s ability to extract free-text ACP documentation from each site’s EHR is outlined in a previously published paper.(7) Prespecified keywords and phrases associated with ACP documentation at each of the three sites was crafted and validated. Templated phrases (e.g., smart phrases in Epic) that document ACP-related topics were not counted as they may be populated by referencing the existence of a structured ACP document rather than evidence a conversation. When tested against a gold standard dataset constructed via manual chart review, the rule-based NLP software produced F1 scores ranging from 0.84 to 1.00. Additionally, upon further review, the authors of the aforementioned previously published paper indicated that instances of ‘false positives’ (i.e., the rule-based NLP discovering documentation the manual reviewers did not) were often the result of the manual reviewers missing the keyword and not the NLP discovering a true ‘false positive.’ (7)
Though ClinicalRegex was used to identify keywords and phrases associated with ACP documentation, a human annotator reviewed each clinical note identified by ClinicalRegex to determine which keywords and phrases represent actual instances of ACP documentation (‘true positives’) and which were not (‘false positives’). True positives were instances in which the free-text documentation was indicative that an ACP conversation had occurred. All the human annotators received training on how to discern true positives for ACP documentation through a series of meetings with a senior investigator. During these meetings the criteria for a true positive was discussed and coding examples were reviewed. The full keyword library and definitional guidelines for identifying ACP documentation in the free-text of clinical notes has been published elsewhere (see Supplemental Table 1 for an adapted version). (7) Patients were considered to have free-text ACP documentation so long as they had at least one free-text documentation that was indicative of an ACP conversation as per the aforementioned methodology, regardless of whether that documentation was related to a goals-of-care conversation, hospice use, palliative care involvement, or limitations on life-sustaining treatments.
Identifying ACP Documentation
The goal of this work was to compare prevalence of standardized ACP documentation to free-text ACP documentation. To accomplish this, ACP documentation was ascertained among the patient sample in the following three ways:
Solely using standardized ACP documentation derived from scanned forms and documents.
Solely using free-text ACP documentation derived from clinical notes.
Using both standardized ACP documentation and free-text ACP documentation.
To compare the applicability of each type of documentation, we considered the number of patients identified in addition to the accuracy and breadth of documentation found.
RESULTS
Patients and Data
A total of 435 patients were included from the three ACP PEACE trial study sites.(22) The median age was 72 years and 44 (10.2%) were either Black or Latino (Table 1). From the total population of 435 patients, 118 (27.1%) did not have a single instance of discoverable standardized or free-text ACP documentation.
Table 1.
Study Population Demographics.
| Characteristic | Site 1 | Site 2 | Site 3 | Overall |
|---|---|---|---|---|
| Number of Patients, N | 74 | 233 | 128 | 435 |
| Age (median), Median (IQR) | 73 (70 – 76) | 71 (68 – 75) | 74 (70 – 80) | 72 (69 – 77) |
| Gender, n (%) | ||||
| Female | 34 (45.9) | 53 (22.7) | 69 (53.9) | 156 (35.9) |
| Race/Ethnicity, n (%) | ||||
| White | 65 (87.8) | 218 (93.6) | 70 (54.7) | 353 (81.1) |
| Black | 6 (8.1) | 1 (0.4) | 25 (19.5) | 32 (7.4) |
| Latino | 0 (0.0) | 2 (0.9) | 10 (7.8) | 12 (2.8) |
| Asian | 0 (0.0) | 3 (1.3) | 9 (7.0) | 12 (2.8) |
| Other | 0 (0.0) | 3 (1.3) | 8 (6.2) | 11 (2.5) |
| Missing | 3 (4.1) | 6 (2.6) | 6 (4.7) | 15 (3.4) |
| Language, n (%) | ||||
| English | 74 (100.0) | 230 (98.7) | 117 (91.4) | 421 (96.8) |
| Spanish | 0 (0.0) | 0 (0.0) | 4 (3.1) | 4 (0.9) |
| Other | 0 (0.0) | 3 (1.3) | 7 (5.5) | 10 (2.3) |
| Deceased, n (%) | 4 (5.4) | 41 (17.6) | 36 (28.1) | 81 (18.6) |
Identifying Patients via Standardized ACP Documentation
At least one standardized ACP document was identified for 187 (42.9%) patients. Together, they had a total of 363 standardized ACP documents of which 148 (40.8%) of those documents were incorrect (e.g.,, document was not available in the EHR; the wrong document was listed; the document was incomplete or blank) or were a duplicate of another standardized document (Table 2). Thirty-one (16.6%) of the 187 patients with standardized ACP documents only had incorrect, incomplete, or duplicate(s) of incorrect or incomplete standardized documents. Of the 156 patients that had at least one correct standardized ACP document, 20 (12.8%) of those patients had a completed health care agent form as their only standardized ACP document.
Table 2.
Patients Identified via Standardized ACP Documentation
| Characteristic | Site 1 | Site 2 | Site 3 | Overall |
|---|---|---|---|---|
| Patient-Level Results | ||||
| Number of Patients Identified * , n (%) | 19 | 114 | 54 | 187 |
| Number of Documents per Patient, Mean, (SD) | 2.9, (3.4) | 1.5, (0.9) | 2.4, (2.0) | 1.9, (1.7) |
| Number of Patients with only HCP Documentation, n (%) | 2 (10.5) | 12 (10.5) | 6 (11.1) | 20 (10.7) |
| Number of Patients with only Incorrect, Incomplete, or Duplicates of Incorrect or Incomplete Documentation, n (%) | 3 (15.8) | 1 (0.9) | 27 (50.0) | 31 (16.6) |
| Document-Level Results | ||||
| Number of Correct Documents, n (%) | 27 (49.1) | 147 (83.6) | 41 (31.1) | 215 (59.2) |
| Advanced Directive/Description of EOL Wishes, n (%) | 14 (25.5) | 104 (59.1) | 1 (0.8) | 119 (32.8) |
| MOLST/Out of Hospital Code Status, n (%) | 0 (0.0) | 17 (9.7) | 7 (5.3) | 24 (6.6) |
| Post-Mortem Instructions, n (%) | 0 (0.0) | 4 (2.3) | 0 (0.0) | 4 (1.1) |
| HCP/DPOA for HC, n (%) | 13 (23.6) | 22 (12.5) | 33 (25.0) | 68 (18.7) |
| Number of Incorrect or Duplicate Documents, n (%) | 36 (50.9) | 29 (16.5) | 91 (68.9) | 148 (40.8) |
| Incomplete (including blank) Document | 0 (0.0) | 4 (2.3) | 2 (1.5) | 6 (1.7) |
| Reports as asked but not completed | 0 (0.0) | 0 (0.0) | 29 (22.0) | 29 (8.0) |
| Reports as available but document not available | 18 (32.7) | 1 (0.6) | 13 (9.8) | 32 (8.8) |
| Wrong Document (i.e. consent form, procedural safety checklist, HIPAA release) | 2 (3.6) | 11 (6.2) | 6 (4.5) | 19 (5.2) |
| Duplicate** (Identical to another form) | 8 (14.5) | 13 (7.4) | 41 (31.1) | 62 (17.1) |
HCP—Health care proxy; DOPA—Durable power of attorney; EOL—end of life; HIPAA—Health Insurance Portability and Accountability Act; MOLST—Medical Orders for Life Sustaining Treatment
To be ‘identified’ a patient must have at least one standardized ACP document in their EHR, regardless of whether it is a correct or incorrect document
This may be a duplicate of a correct or incorrect standardized ACP document
Identifying Patients via Free-Text ACP Documentation
Instances of free-text ACP documentation were evaluated for the same 435 patients from a total of 79,797 clinical notes using the ClinicalRegex software. In total, 238 (54.7%) patients had at least one clinical note with free-text ACP documentation (i.e., documentation containing evidence that at least one of the four evaluated domains was discussed). Eighty-eight (37.0%) patients had documentation regarding limitations on life-sustaining treatment, 108 (45.4%) had documentation regarding hospice, 132 (55.5%) had documentation regarding palliative care, and 203 (85.3%) patients had documentation related to goals of care conversations. Of the patients identified with free-text ACP documentation, 53 (22.3%) had free-text ACP documentation in all of the four evaluated domains: goals of care, hospice, palliative care, and limitations on life-sustaining treatments (Table 3).
Table 3.
Patients Identified via Free-Text ACP Documentation.
| Characteristic | Site 1 | Site 2 | Site 3 | Overall |
|---|---|---|---|---|
| Number of Patients Identified with Documentation in at Least One Domain, n | 32 | 95 | 111 | 238 |
| Number of Patients with Limitations on Life Sustaining Treatment Documentation, n (%) | 10 (31.2) | 26 (27.4) | 52 (46.8) | 88 (37.0) |
| Number of Patients with Hospice Documentation, n (%) | 16 (50.0) | 32 (33.7) | 60 (54.1) | 108 (45.4) |
| Number of Patients with Palliative Care Documentation, n (%) | 17 (53.1) | 43 (45.3) | 72 (64.9) | 132 (55.5) |
| Number of Patients with Goals of Care Documentation, n (%) | 20 (62.5) | 79 (83.2) | 104 (93.7) | 203 (85.3) |
| Number of Patients with Documentation in All Four Domains, n (%) | 3 (9.4) | 10 (10.5) | 40 (36.0) | 53 (22.3) |
Identifying Patients via Standardized and Free-Text ACP Documentation
When evaluating both standardized and free-text ACP documentation, patients can be identified to have the following documentation combinations: (1) free-text and standardized ACP documentation; (2) just free-text ACP documentation; (3) just standardized ACP documentation.
(1) Both Free-Text and Standardized ACP Documentation
Upon first pass, one hundred eight (24.8%) patients out of the total sample of 435 would have been identified as having instances of both standardized and free-text ACP documentation. Upon closer inspection, however, 24 (24/108, 22.2%) of those 108 patients identified with both free-text and standardized documentation were found to only have incorrect, incomplete, or duplicate(s) of incorrect or incomplete standardized documentation. Therefore, the true number of individuals with both free-text and standardized ACP documentation was 84 (19.3%) out of the total sample of 435.
(2) Only Free-Text ACP Documentation
One hundred thirty (29.9%) patients from the total sample of 435 were identified as having free-text ACP documentation yet lacking any standardized ACP documentation (Table 4). As outlined in the previous section, 24 patients ostensibly identified to have both free-text and standardized ACP documentation were found to have only incorrect, incomplete, or duplicate(s) of incorrect or incomplete standardized ACP documentation, rendering the total number of patients with only free-text ACP documentation to be 154 (35.4%).
Table 4.
Patients and ACP Documented Identified via Standardized and Free-Text ACP Documentation
| Type of ACP Documentation | Standardized and Free-Text | Standardized or Free-Text* |
|---|---|---|
| Patient-Level Results | ||
| Number of Patients Initially Identified, n | 108 | 317 |
| Number of Patients with Correct Documentation, n (%)** | 84 (77.8) | 286 (91.2) |
| Number of Patients with only HCP Documentation, n (%) | 8 (7.4) | 20 (6.3) |
| Document-Level Results | ||
| Number of Structured Documents Irrespective of Correctness, n | 231 | 363 |
| Number of Structured Documents that are Incorrect or Duplicate *** Documents, n (%) | 112 (48.4) | 148 (40.8) |
Patients identified by standardized or free-text ACP documentation includes those with only free-text documentation, only standardized documentation (irrespective of correctness), and both free-text and standardized documents
Number of patients with correct documentation represents the subset of initially identified patients that have at least one correct instance of standardized ACP documentation
This may be a duplicate of a correct or incorrect standardized ACP document
(3) Only Standardized ACP Documentation
A total of 79 patients had standardized ACP documentation but no free-text documentation. Seven (7/79, 8.7%) of the 79 patients identified with standardized ACP documentation only had instances of incorrect, incomplete, or duplicate(s) of incorrect or incomplete documentation. Therefore, only 72 (16.6%) patients of the 435 reviewed were identified as having true instances of standardized ACP documentation.
Figure 1 illustrates the identification of ACP documentation using both standardized and free-text ACP documentation.
Figure 1:
Patient Identification by Type of ACP Documentation
DISCUSSION
Main Findings
In this study, we examined how the type of documentation used to evaluate ACP impacted what can be considered as the rate, substance, and quality of ACP. Namely, we compared ACP documentation based on standardized documents and forms to documentation from the free-text of clinical notes. While standardized and free-text ACP documentation may represent different aspects of the ACP process, it is important to explore the prevalence of each within the same population to better understand the implications of choosing a specific type of documentation for ACP research. We found a substantially greater number of patients were classified as having ACP documentation when relying upon free-text ACP documentation (54.7%) than standardized ACP documentation (43.0%). Of the patients with standardized ACP documentation, 10% were identified as only having documentation of a health care proxy form which—while an important ACP document—in isolation may not fulfill all the goals of ACP.(1, 27) Moreover, our results indicate a substantial difference in the misclassification of ACP documentation, with 16.6% of the patients identified through standardized ACP documentation only having incorrect, incomplete, or duplicate(s) of incorrect or incomplete documents.
Unsurprisingly, relying on free-text and standardized ACP documentation in combination identified the most patients, 317 (72.8%), from our sample of 435. Only 108 of those 317 (108/317, 34.1%) patients identified as having standardized or free-text ACP documentation were identified as having both standardized and free-text ACP documentation. Additionally, of those 108 patients with both types of documentation, 24 (24/108, 22.2%) individuals had instances of only incorrect, incomplete, or duplicate(s) of incorrect or incomplete standardized documentation, revealing the true number of patients with both types of ACP documentation to be 84. Taken together, these results indicate a large error rate present when identifying patients using standardized ACP documentation.
The comparison between standardized and free-text ACP documentation reveals how different conclusions regarding the reach of ACP can be drawn from the same population of patients. For example, if one were to solely rely on standardized ACP documentation to evaluate ACP in the population, 43.0% of the patients would have been identified to have ACP documentation and, upon further examination, 16.6% of these patients would have been discarded for having only incorrect, incomplete, or duplicate(s) of incorrect or incomplete forms. Though the forms identified as standardized ACP documentation may have revealed the patient’s codified wishes for care, little evidence would have been provided as to whether this communication and support was ongoing, whether it was specific to this experience of serious illness, or if educational support was provided or comprehension was ascertained. Based on this evaluation, it would therefore be reasonable for researchers to conclude that ACP was not effective or for a policy maker to conclude that ACP was not occurring reliably. However, both of those assessments would be less a reflection of the reach of ACP, and more of the methods chosen to measure it. The result of such substantial measurement error has implications for pragmatic research done based upon standardized ACP documents; if vigorous validation practices are not employed, the accuracy of standardized ACP documents cannot be assumed. The prevalence of ACP as discerned by standardized documents, which are often misclassified and not representative of the total evidence of ACP in a patient’s chart, may therefore bias results towards the null, making it more difficult for ACP studies to be positive.
In contrast to the misclassification and errors associated with standardized ACP documentation, relying upon the free-text ACP documentation would have found evidence of ACP documentation in 54.7% of the patients, roughly 10% more than the standardized ACP documentation, without adjustment for structured document accuracy. Furthermore, free-text ACP documentation provides a more granular look into the type of ACP discussions the individual engaged in, accounting for the overarching topic present in each instance of ACP documentation with high face validity. Increasingly, NLP methods are being utilized to measure EHR documentation of the ACP process. (21, 28, 29)
Utilizing NLP methods for measuring free-text ACP documentation is not without its limitations. For example, rule-based NLP methods—like those relied upon in this manuscript—are only as accurate as the keyword library specified to guide them. Moreover, rule-based NLP methods can be impacted by human error and biases. (7) As a result, the use of NLP methods to measure ACP documentation must be carefully guided and validated to ensure that it is accurately capturing the desired concepts.
There are well-acknowledged difficulties in quantifying the complex concepts associated with ACP that have hindered researchers’ ability to study aspects of ACP through randomized clinical trials.(30) The labor intensity of incorporating note content into evaluations of ACP has made it difficult to operationalize the use of clinical notes (i.e., free-text ACP documentation) even though our study demonstrates that clinical notes can identify ACP documentation without the same concerns for correctness that is present when relying upon standardized ACP documentation.(7, 18) Improving the accuracy of ACP documentation does not imply anything about its effectiveness, however, using a more accurate metric for documenting ACP activity may allow us to better ask and answer questions about its effectiveness in the future.
There are a few possible explanations for why so few patients had both standardized and free-text ACP documentation. First, it is possible that some patients have pre-existing standardized ACP documentation and did not engage in any ACP conversations during the study period. Second, since some types of standardized ACP documentation, such as Advance Directive/Description of End-of-Life Wishes, may be completed without a patient and clinician signature, their completion may not infer a patient-clinician conversation and therefore may not yield documentation in the free-text of a clinical note. Third, patients may have engaged in ACP conversations but have not completed standardized forms to codify their wishes. Irrespective of the root cause, the discrepancy suggests that the process of engaging in ACP may not be well represented through the sole evaluation of standardized ACP documentation, especially given that even accurate standardized ACP documentation cannot capture the quality or nature patient-clinician communication like free-text ACP documentation.
Limitations
There are several limitations to this work. First, we relied upon a convenience sample of older adults with advanced cancer. The methods for engaging and documenting ACP for these patients may differ from different populations and disease types. Second, the standardized and free-text ACP documentation used in this work were collected prior to the COVID-19 pandemic. In the time since data collection, health systems may have created new methods for recording ACP in the EHR (e.g., an ACP EHR tab) that are not reflected in this study. Third, the analysis presented is limited to a comparison of standardized and free-text ACP documentation drawn in a population of patients. There was no comparison of the standardized and free-text ACP documentation for each patient, meaning that our methodology did not allow us to determine concordance, discordance, or the relationship between documents for a particular patient (e.g., if a patient has multiple instances of ACP documentation, are the wishes documented in each of those concordant with each other?). Furthermore, it was beyond the scope of this paper to evaluate the potential differential impact of each type of incorrect standardized documentation. Fourth, institutional policies, practices, and/or regulations may be in place to encourage clinicians to document ACP in a specific portion of the EHR. If this were the case, then the differences in documentation rates of standardized and free-text ACP may be due in part to it. To our knowledge, however, no such policies existed at the three sites included in this study. Lastly, though it allowed for a realistic look into ACP documentation in the clinical notes, evaluation of free-text ACP documentation is limited by the quality of that documentation and our evaluation of it, such as our exclusion of templated or quick-text pre-set phrases from qualifying as free-text ACP documentation. Future work is needed to capture real time ACP using NLP methodologies which allow for rapid conversion of speech to analyzable text.
Conclusion
Effective evaluation of ACP is predicated upon the ability to accurately measure its process. Patients may be faced with medical decisions that require in-the-moment decision making supported by ongoing ACP.(21) The findings presented in this article demonstrate that traditional methods of measuring ACP documentation via reliance on standardized forms may fall short in capturing all instances of ACP documentation in the EHR. The downstream impact of mismeasuring ACP may result in research and policy evaluations that are not reflective of actual clinical practice, thus may lead to practice changes that are detrimental to patients and caregivers. New approaches to measuring ACP, such as NLP capture from free-text of clinical notes and the NLP analysis of recorded clinical consultations, offer promising new methods for positively augmenting clinical practice and evaluating ACP’s occurrence and effectiveness.(31)
Supplementary Material
Key Message.
There has been little research comparing methods for measuring advance care planning (ACP), the complex process in which patients deliberate upon their preferences for medical care. This study demonstrates that written documents and forms scanned into the electronic health record may not fully capture ACP in a patient’s chart
Funding
This work was supported within the National Institutes of Health (NIH) Pragmatic Trials Collaboratory by cooperative agreement UH3AG060626 from the National Institute on Aging. This work also received logistical and technical support from the NIH Pragmatic Trials Collaboratory Coordinating Center through cooperative agreement U24AT009676 from the National Center for Complementary and Integrative Health (NCCIH), the National Institute of Allergy and Infectious Diseases (NIAID), the National Cancer Institute (NCI), the National Institute on Aging (NIA), the National Heart, Lung, and Blood Institute (NHLBI), the National Institute of Nursing Research (NINR), the National Institute of Minority Health and Health Disparities (NIMHD), the National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS), the NIH Office of Behavioral and Social Sciences Research (OBSSR), and the NIH Office of Disease Prevention (ODP). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Aging or the NCCIH, NIAID, NCI, NIA, NHLBI, NINR, NIMHD, NIAMS, OBSSR, or ODP, or the NIH.
Declaration of Competing Interest
Dr. Tulsky is a Founding Director of VitalTalk, a nonprofit organization focused on clinician communication skills training, from which he receives no compensation. Dr. Carney is a board member of Haven Behavioral Health. Dr. Volandes has a financial interest in the nonprofit foundation Nous Foundation (d/b/a ACP Decisions, 501c3). The nonprofit organization develops ACP video decision aids and support tools. Dr. Volandes’ interests were reviewed and are managed by Massachusetts General Hospital and Partners HealthCare in accordance with their conflict-of-interest policies. None of the other authors have any conflict of interests to disclose.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Research Ethics and Patient Consent
The analysis presented in this study is derived from ACP PEACE, and ongoing multi-site pragmatic trial with approval by the Institutional Review Board at Dana-Farber Cancer Institute (18–276, initial approval 8/27/2018). The trial registration number is NCT03609177. A wavier of consent was approved for the EHR review of primary study subjects who are not contacted by study staff unless a specific research declination is on file at that site.
Publishing Guidelines
The SQUIRE 2.0 guidelines were adapted for this manuscript.(25) SQUIRE 2.0 was appropriate for this work as it can be adapted to many studies that aim to report new knowledge about improving healthcare.
Data Statement
All data collected and analyzed as a part of this study was managed in accordance with ethical guidelines and regulations. The data presented in this manuscript will not be made public to best safeguard patient confidentiality.
References
- 1.Sudore RL, Lum HD, You JJ, Hanson LC, Meier DE, Pantilat SZ, et al. Defining Advance Care Planning for Adults: A Consensus Definition From a Multidisciplinary Delphi Panel. J Pain Symptom Manage. 2017;53(5):821–32 e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Pope TM. Legal Briefing: Medicare Coverage of Advance Care Planning. J Clin Ethics. 2015;26(4):361–7. [PubMed] [Google Scholar]
- 3.Communications N. HEDIS 2022: See What’s New, What’s Changed and What’s Retired: NCQA; 2021. [Available from: https://www.ncqa.org/blog/hedis-2022-see-whats-new-whats-changed-and-whats-retired/.
- 4.Sudore RL, Fried TR. Redefining the “planning” in advance care planning: preparing for end-of-life decision making. Ann Intern Med. 2010;153(4):256–61. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Hawkins NA, Ditto PH, Danks JH, Smucker WD. Micromanaging death: process preferences, values, and goals in end-of-life medical decision making. Gerontologist. 2005;45(1):107–17. [DOI] [PubMed] [Google Scholar]
- 6.McMahan RD, Knight SJ, Fried TR, Sudore RL. Advance care planning beyond advance directives: perspectives from patients and surrogates. J Pain Symptom Manage. 2013;46(3):355–65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Lindvall C, Deng CY, Moseley E, Agaronnik N, El-Jawahri A, Paasche-Orlow MK, et al. Natural Language Processing to Identify Advance Care Planning Documentation in a Multisite Pragmatic Clinical Trial. J Pain Symptom Manage. 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Lakin JR, Gundersen DA, Lindvall C, Paasche-Orlow MK, Tulsky JA, Brannen EN, et al. A Yet Unrealized Promise: Structured Advance Care Planning Elements in the Electronic Health Record. J Palliat Med. 2021;24(8):1221–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Fagerlin A, Schneider CE. Enough. The failure of the living will. Hastings Cent Rep. 2004;34(2):30–42. [PubMed] [Google Scholar]
- 10.Loewenstein G Hot-cold empathy gaps and medical decision making. Health Psychol. 2005;24(4S):S49–56. [DOI] [PubMed] [Google Scholar]
- 11.Billings JA. The need for safeguards in advance care planning. J Gen Intern Med. 2012;27(5):595–600. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Billings JA, Bernacki R. Strategic targeting of advance care planning interventions: the Goldilocks phenomenon. JAMA Intern Med. 2014;174(4):620–4. [DOI] [PubMed] [Google Scholar]
- 13.Lakin JR, Block SD, Billings JA, Koritsanszky LA, Cunningham R, Wichmann L, et al. Improving Communication About Serious Illness in Primary Care: A Review. JAMA Intern Med. 2016;176(9):1380–7. [DOI] [PubMed] [Google Scholar]
- 14.Winter L, Parks SM, Diamond JJ. Ask a different question, get a different answer: why living wills are poor guides to care preferences at the end of life. J Palliat Med. 2010;13(5):567–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Walker E, McMahan R, Barnes D, Katen M, Lamas D, Sudore R. Advance Care Planning Documentation Practices and Accessibility in the Electronic Health Record: Implications for Patient Safety. J Pain Symptom Manage. 2018;55(2):256–64. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Wilson E, Bernacki R, Lakin JR, Alexander C, Jackson V, Jacobsen J. Rapid Adoption of a Serious Illness Conversation Electronic Medical Record Template: Lessons Learned and Future Directions. J Palliat Med. 2020;23(2):159–61. [DOI] [PubMed] [Google Scholar]
- 17.Lamas D, Panariello N, Henrich N, Hammes B, Hanson LC, Meier DE, et al. Advance Care Planning Documentation in Electronic Health Records: Current Challenges and Recommendations for Change. J Palliat Med. 2018;21(4):522–8. [DOI] [PubMed] [Google Scholar]
- 18.Berkowitz CM, Ma J, Lowe J, Dolor RJ. Assessing Quality in Advance Care Planning Documentation: A Survey of Current Methods. Am J Hosp Palliat Care. 2021:10499091211060341. [DOI] [PubMed] [Google Scholar]
- 19.Lee KC, Udelsman BV, Streid J, Chang DC, Salim A, Livingston DH, et al. Natural Language Processing Accurately Measures Adherence to Best Practice Guidelines for Palliative Care in Trauma. J Pain Symptom Manage. 2020;59(2):225–32 e2. [DOI] [PubMed] [Google Scholar]
- 20.Udelsman BV, Lee KC, Lilley EJ, Chang DC, Lindvall C, Cooper Z. Variation in Serious Illness Communication among Surgical Patients Receiving Palliative Care. J Palliat Med. 2020;23(3):411–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Volandes AE, Zupanc SN, Paasche-Orlow MK, Lakin JR, Chang Y, Burns EA, et al. Association of an Advance Care Planning Video and Communication Intervention With Documentation of Advance Care Planning Among Older Adults: A Nonrandomized Controlled Trial. JAMA Netw Open. 2022;5(2):e220354. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Lakin JR, Brannen EN, Tulsky JA, Paasche-Orlow MK, Lindvall C, Chang Y, et al. Advance Care Planning: Promoting Effective and Aligned Communication in the Elderly (ACP-PEACE): the study protocol for a pragmatic stepped-wedge trial of older patients with cancer. BMJ Open. 2020;10(7):e040999. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Eneanya ND, Lakin JR, Paasche-Orlow MK, Lindvall C, Moseley ET, Henault L, et al. Video Images about Decisions for Ethical Outcomes in Kidney Disease (VIDEO-KD): the study protocol for a multi-centre randomised controlled trial. BMJ Open. 2022;12(4):e059313. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Lilley EJ, Lindvall C, Lillemoe KD, Tulsky JA, Wiener DC, Cooper Z. Measuring Processes of Care in Palliative Surgery: A Novel Approach Using Natural Language Processing. Ann Surg. 2018;267(5):823–5. [DOI] [PubMed] [Google Scholar]
- 25.Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf. 2016;25(12):986–92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Lindvall Lab: Computational Palliative Care Research [Available from: https://lindvalllab.dana-farber.org/.
- 27.McMahan RD, Tellez I, Sudore RL. Deconstructing the Complexities of Advance Care Planning Outcomes: What Do We Know and Where Do We Go? A Scoping Review. J Am Geriatr Soc. 2021;69(1):234–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Curtis JR, Lee RY, Brumback LC, Kross EK, Downey L, Torrence J, et al. Intervention to Promote Communication About Goals of Care for Hospitalized Patients With Serious Illness: A Randomized Clinical Trial. JAMA. 2023. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Lakin JR, Zupanc SN, Lindvall C, Moseley ET, Das S, Sciacca K, et al. Study protocol for Video Images about Decisions to Improve Ethical Outcomes with Palliative Care Educators (VIDEO-PCE): a pragmatic stepped wedge cluster randomised trial of older patients admitted to the hospital. BMJ Open. 2022;12(7):e065236. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Halpern SD. Goal-Concordant Care - Searching for the Holy Grail. N Engl J Med. 2019;381(17):1603–6. [DOI] [PubMed] [Google Scholar]
- 31.Ross L, Danforth CM, Eppstein MJ, Clarfeld LA, Durieux BN, Gramling CJ, et al. Story Arcs in Serious Illness: Natural Language Processing features of Palliative Care Conversations. Patient Educ Couns. 2020;103(4):826–32. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
All data collected and analyzed as a part of this study was managed in accordance with ethical guidelines and regulations. The data presented in this manuscript will not be made public to best safeguard patient confidentiality.