Implementation Challenges for a Multi-site Advance Care Planning Pragmatic Trial: Lessons Learned

Abstract

Background/Problem:

Advance care planning (ACP) pragmatic trials are needed.

Proposed Solution:

We determined key system-level activities to implement ACP interventions for a cluster-randomized pragmatic trial. We identified patients with serious illness from 50 primary care clinics across three University of California health systems using a validated algorithm. If patients lacked documented ACP within the last three years, they were eligible for an intervention: (Arm 1) an advance directive (AD); (Arm 2) AD + PREPAREforYourCare.org; (Arm 3) AD + PREPARE + lay health navigator outreach. Triggered by an appointment, we mailed and sent interventions through automated electronic health record (EHR) messaging. We collaborated with patients/caregivers, clinicians, payors, and national/health system leader advisors. We are currently finalizing 24-month follow-up data.

Outcomes/Methods:

We used the Consolidated Framework for Implementation Research (CFIR) and Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) frameworks to track secular trends and implementation efforts.

Key Message/Results:

Required multi-site, system-level activities: (1) obtaining leadership, legal/privacy, and EHR approvals; (2) standardizing ACP documentation; (3) providing clinician education; (3) validating an automated serious illness identification algorithm; (4) standardizing ACP messaging with input from over 100 key advisors; (5) monitoring secular trends (e.g., COVID); and (6) standardizing ACP workflows (e.g., scanned ADs). Of 8707 patients with serious illness, 6883 were eligible for an intervention. Across all arms, 99% received the mailed intervention, 78.3% had an active patient portal (64.2% opened intervention), and 90.5% of arm 3 patients (n=2243) received navigator outreach.

Lessons Learned:

Implementing a multi-site health system-wide ACP program and pragmatic trial, with automated EHR-based cohort identification and intervention delivery, requires a high level of multi-disciplinary key advisor engagement, standardization, and monitoring. These activities provide guidance for the implementation of other large-scale, population-based ACP efforts.

BACKGROUND

Advance care planning (ACP) has evolved over time from a focus on advance decisions about end-of-life procedures to a process of preparing patients and surrogate decision makers for communication and medical decision-making.^1,2 ACP has been shown to increase patient and surrogate satisfaction with communication and medical care and to decrease surrogate and clinician distress. In recent studies, ACP has also been associated with patient and caregiver reports of receiving goal concordant care.^3–5

However, the ACP process is complex, often involving multiple interventions targeted to patients (with varying disease and life trajectories) and/or surrogate and clinician behaviors that occur within and across complex health systems and communities with varying workflows, policies, and societal norms.

PROBLEM

As described in prior scoping reviews, many ACP studies have been small, of low quality, or focused on efficacy in standardized settings.^6,7 Pragmatic trials of ACP interventions in real-world settings are needed for translation, adoption, and dissemination of evidence-based ACP materials into clinical care.⁸

PROPOSED SOLUTION

We conducted a large, pragmatic, clustered randomized trial in three University of California (UC) health systems: UC, Los Angeles (UCLA), UC, San Francisco (UCSF), and UC, Irvine (UCI). The methods of this study have been previously described.⁹ Briefly, we included English and Spanish-speaking patients with serious illness from 50 primary care clinics across the three UC health systems. Patients were eligible to receive an ACP intervention if they lacked ACP documentation within three years. A community advisory board comprised of patients and caregivers from each UC site, each UC site’s Office of Population Health, and a larger study advisory group of clinicians, payors, and national/health system leaders collaborated throughout the project.

The ACP interventions were delivered in three arms requiring progressively more resources: Arm 1 included an advance directive (AD); Arm 2 included an AD plus the PREPAREforYourCare.org online program; and Arm 3 included an AD plus PREPARE plus outreach from a lay health navigator. Triggered by an upcoming primary care appointment, we sent a letter encouraging ACP and use of the study ACP materials through the mail to all patients (AD, PREPARE pamphlet) and automatically through the electronic health record (EHR) patient portal (URL links to ADs and PREPARE). We are currently finalizing our 24-month follow-up data.

Our trial’s primary and secondary outcomes have been described, and those findings are forthcoming.⁹ In this paper, we aim to describe the key system-level contextual factors and activities needed to implement ACP interventions for this cluster-randomized pragmatic trial by relying on implementation science theory. We used the Consolidated Framework for Implementation Research (CFIR) to consider the challenges faced in trial implementation. CFIR is one of the most well-established and referenced implementation science frameworks and was recently updated to a 2.0 version in 2022.¹⁰ CFIR consists of five, interrelated domains (i.e., Innovation, Outer Setting, Inner Setting, Individuals, Implementation Process) and several subdomains which are related to ACP implementation.¹¹ We also used the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework to track reach and adoption of our intervention.^12,13

OUTCOMES

In this paper, we describe: (a) the many steps required to launch and conduct this pragmatic trial; (b) the secular trends that occurred; (c) the specific contextual factors and challenges to trial implementation within the 5 CFIR domains; and (d) outcomes related to the RE-AIM framework.

KEY MESSAGE/RESULTS

Steps for Trial Implementation

Several steps were needed before, during, and after trial launch to conduct this complex, multi-site trial.¹⁴ The effort required for these tasks has been high due to coordination across each health system and clinic included in the trial (Table 1.) The length and depth of the list in Table 1 demonstrates both the complexity of pragmatic trials, and the often unfunded team effort required for trial development, such as obtaining leadership, clinician, and staff buy-in and creating, supporting, and maintaining patient and caregiver advisory boards. We describe the patient baseline characteristics in Table 2.

Table 1.

Characteristics of CNS tumor patients, ages 0–19 years, SEER Registries, 1990–2015 (N = 14,493)

1.	Worked with individual sites, Population Health, local EHR IT leaders to standardize ACP documentation in the EHR
2.	Leadership engagement, invited 5 UC sites and identified 3 UC site champions
3.	Obtained local Population Health buy-in at each site
4.	Obtained local primary care leadership buy-in
5.	Identified and included a patient and caregiver advisory group
6.	Identified and included a larger study advisory group (patients, caregivers, clinicians, insurers, national and state leaders, legal experts)
7.	Obtained central IRB (UCLA) approval for all 3 sites and obtained local IRB reliance
8.	Obtained individual clinic leadership, clinician, and staff buy in (50 clinics)
9.	Created an automated EHR algorithm to identify patients with serious illness
10.	Validated the automated EHR algorithm across the 3 UC sites
11.	Identified deceased patients through registries and chart review to manually exclude from the cohort
12.	Obtained approval from local Population Health and EHR leadership to develop and implement the algorithm within the local EHR
13.	Convened over 100 advisors from all sites, the patient and caregiver advisory group, and the external study advisory group to create appropriate messaging for ACP and tailor our ACP materials
14.	Created EHR code to deliver automated ACP messages to eligible patients that were identical across sites
15.	Determined with Population Health that all patients would get both patient portal messages and mailed messages
16.	Determined with Population Health whose signature would be included in the ACP messaging (e.g., Population health, clinic leadership, etc.)
17.	Identified standardized ACP clinical workflows, roles, and responsibilities at each site
18.	Conducted clinician and clinic staff training on ACP communication and standardized workflows
19.	Defined primary and secondary outcomes using EHR administrative data
20.	Developed a healthcare navigator toolkit (scripts, FAQs, EHR note templates)
21.	Conducted several in-person and virtual trainings for the healthcare navigators
	-----------------------------------------------------------------------------------------------------
	STUDY LAUNCH
22.	Used automated EMR algorithms to identify the population cohort and send the ACP intervention
23.	Monitored secular trends (e.g., accommodating COVID-19 telehealth visits, EHR infrastructure changes, and changing leadership individuals and priorities, etc.)
24.	Conducted semi-monthly Patient and Caregiver Advisory Group meetings
25.	Conducted bi-annual Study Advisory Group meetings
26.	Conducted ongoing meetings with Population Health, primary care leadership, clinic leadership and managers as necessary
27.	Conducted ongoing training for staff and healthcare navigators, including during staff turnover
28.	Updated patient messaging about ACP to address COVID-19 with Patient and Caregiver Advisory Group
29.	Updated messaging about ACP at 12 months to hone marketing with Patient and Caregiver Advisory Group

EHR = electronic health record, IT = Information Technology, IRB = institutional review board, ACP = advance care planning

This list does not include research components that would be a usual part of a clinical trial such as obtaining funding, hiring staff, implementing a Data Safety Monitoring Plan, convening Data Safety and Monitoring Board, and enrolling and randomizing participants and conducting baseline, 12- and 24-month follow up interviews for a research cohort subset.

Table 2.

Baseline Patient Characteristics

	N = 8,707
Site, n
UCI (%)	1354 (15.6%)
UCSF	746 (8.6%)
UCLA	6607 (75.9%)
Age in years, mean (range)	73 (20–90+)*
Gender, n (%)
Female	4330 (49.9%)
Male	4350 (50.1%)
Non-binary	4 (0.05%)
Race/Ethnicity n (%)
Hispanic/Latinx	1462 (16.8%)
White	4717 (54.2%)
Asian	1084 (12.4%)
Black	721 (8.3%)
Other	723 (8.3%)
Language spoken at home, n (%)
English	7360 (84/5%)
Spanish	748 (8.6%)
Other	599 (6.9%)
Social Vulnerability Index‡, mean (range)	0.37 (0.0–1.0)

^*HIPAA requirements precluded us from collecting or reporting age over 90

^‡Social Vulnerability Index ranges from 0 to 1 with 1 indicating maximal vulnerability

Secular Trends

Secular trends were expected in this longitudinal study (Table 3). A notable challenge was the COVID pandemic. COVID occurred shortly after the study launched leading to several local secular trends that affected our trial (e.g., a transition from in-person to tele-health visits in primary care). Some additional secular trends included a new or renewed focus on ACP at each study site, as well as at the UC Office of the President. This resulted in new ACP initiatives in Population Health and in other outpatient clinics and disciplines outside of primary care. In addition, each site, through various programs, has been involved in value-based reimbursement programs for ACP. These programs generate health system-wide interventions to stimulate and capture remunerated care processes (such as ACP discussions). In addition, other preventive care interventions can affect ACP. For example, at UCI, an EHR version of the Annual Wellness Visit (AWV) was launched, and the numbers of AWV increased at this site, which often include ACP. Because of this, it is possible that we may see a “rising tide phenomenon” or a pronounced increase in ACP over the course of our study due to widespread attention.¹⁵ Trial findings are forthcoming.

Table 3.

Secular Trends During This Multi-site Advance Care Planning Pragmatic Trial

Secular Trend	UCLA	UC Irvine	UCSF
External Changes
COVID pandemic, clinics shut down with shift to telehealth	3/2020	3/2020	3/2020
EHR Related Updates
EHR ACP Activity rolled out to all patients	Started 10/2018	Started 7/2020	Started 1/2019
Fillable AD pdf available for UCLA Health AD	Started 7/2020
AD’s added to health maintenance module	Started 10/2020	Started 7/2021	Started 6/2021
EHR Patient portal AD reminders instituted for the health system	Started 1/2021	Started 2/2022	Started 10/2021
Ability to upload AD through patient portal and reviewed by Health Information Services	Started 4/2021
Quality Improvement Programs
ACP intervention sent to primary care patients 65 and older without serious illness in study clinics	4/2020 (Arm 1 intervention only to health system quality improvement effort)		Started 10/19 (PCORI intervention to all patients 65 and older without serious illness)
ACP Quality Metrics provided to Oncologists as part of another QI effort	Started 10/2018
Population defined “Meaningful ACP” from HER elements and created EHR dashboard as part of another QI effort			Started 1/2019
Annual Wellness Visit Push for ACP	Started 4/2021 (nurse practitioners involved in preparing patients for the visits	Started 7/2022	Started 7/2019
Inpatient ACP intervention	Started 7/2020 (A “surprise” question added to documentation)		Started 7/2019 (education, dashboards, clinician training, incentives)
Separate QI projects use PCORI intervention and study materials in non-PCORI clinics	2/2021 (several primary care clinics)	10/2021 (primary care)	Started 10/2019 (primary care and surgery)
ACP Integrated into Cancer Center Materials	Started 6/2022		Started 1/2020
Clinician Education & Initiatives
Medicine resident ACP training	Annually	Annually	Annually
Clinician and staff outreach about ACP EMR changes and billing	2/2018	9/2021	Started 1/2019
Additional Health system ACP Clinician Training	4/2019 oncologists & social workers 7/2021 Bone Marrow Transplant nurse practitioners	7/2019 Primary care	4/2019 (4 hours offered to all primary care providers in study clinics)
Patient-Focused Interventions
ACP Group Meetings/Events for patients	Started 1/2019	Started 1/2019	Started 4/2019
Notaries in clinic			4/2019, shut down 3/2020 with COVID and staff transitions
Systemwide Healthcare Decisions Day Outreach	Started 4/2019		Started 4/2019
Law student home visits for ACP			Started 4/2019 UCSF/UC Hastings Consortium on Law, Science & Health Policy
Social workers allowed to complete ACP visits			Started 7/2019

EHR = electronic health record, AD = advance directive, ACP = advance care planning

CFIR Domains: Challenges to ACP Trial Implementation

The CFIR framework describes innovations as “the ‘thing’ being implemented” and the implementation process as “the activities and strategies to implement the innovation”.¹⁰ For this study, innovation and implementation were closely intertwined. Below we describe the CFIR domains of Innovation, Implementation Process, Outer Setting, Inner Setting, and Individuals.

Innovation

The innovations included the delivery (timed to a primary care appointment) of (a) an AD, (b) an AD plus PREPARE online program, and (c) for Aim 3 only, an AD plus the PREPARE online program plus outreach from a lay healthcare navigator.⁹ The PREPARE program was chosen because it has been shown in randomized trials to help historically marginalized English- and Spanish-speaking older patients engage in ACP and reduce disparities in ACP.^16–18 What enabled use of the PREPARE program and materials was that it was developed by our co-investigator and is copyrighted and licensed through the UC Regents for research and clinical purposes.

One challenge related to the delivery of ADs was standardizing forms across sites. For example, UCSF and UCLA had developed their own site-specific AD. Population Health and primary care leadership at UCSF and UCI agreed to move forward with the PREPARE AD for this trial as most primary care providers in the primary care study clinics were already using it because it is easy-to-read and available in over ten languages. In contrast, UCLA leadership preferred the use of the UCLA-specific form. Therefore, implementation of the ADs differed slightly between sites. In discussion with our advisory boards, this was a variation that was determined to be acceptable given the context of this real-world, pragmatic clinical trial.

Implementation Process

To implement these innovations required the following implementation processes: (a) identification of a serious illness population through an automated EHR-based algorithm; (b) development of EHR EPIC build to send ACP messages about the intervention to eligible patients 3 weeks prior to a primary care visit; (c) the development of ACP MyChart messages; (d) development of ACP mailed messages; (e) launch of the study across 50 clinics in stages; (f) clinician training; (g) development of EHR note templates to document ACP, particularly for healthcare navigators; and (h) consideration of patient’s ability to execute (i.e., sign, witness and/or notarize) AD forms.

(a). Cohort identification:

Identification of a serious illness population occurred through the development of an automated EHR-based algorithm (i.e., “EHR phenotype” for serious illness) using structured data elements that required iterative chart abstraction and validation across all three UCs.⁹ In this population-based trial many of the patients identified as seriously ill were in fact deceased. Date of death information is notoriously missing in the EHR, especially if deaths occurred outside of the hospital. Identification of deceased patients required retrospective chart review.

(b). EHR Build:

Each site also had different procedures for requesting, creating, and maintaining the EHR code that would allow us to target eligible patients for ACP messaging three weeks before a primary care visit. For example, UCSF created a new procedure that made this process automated, while UCLA and UCI developed processes that required weekly manual implementation. A benefit to the automated build is that it required less staff time. However, with yearly updates to EPIC, we learned that the EHR code also had to be updated for the study to proceed. Furthermore, another challenge of the EHR build was the transition from in-person appointments to telehealth across all sites during the COVID pandemic. After noticing our outreach numbers dropped substantially across sites, we realized that the build did not include tele-health visits and thus needed to be updated. This required a series of regulatory approvals at each site during a time when the health systems were stretched thin.

In addition, some sites included non-study populations in the ACP messaging roll out. Given pressure from value-based incentive programs, and in part due to ongoing engagement from our team, ACP became a priority for Population Health and primary care at our sites at different points in the study. At UCSF, at study launch, it was decided to also conduct a quality improvement initiative, outside of the trial, by including all patients aged 65 and older without serious illness to receive the ACP interventions by study clinic arm. Population Health at UCLA and UCI implemented ACP interventions with other patient populations later in the study period.

(c). ACP Messaging:

Furthermore, the content of the ACP messaging, both through MyChart and mailing, needed to be created with and for UC patients, caregivers, and key advisors. To do so required multiple iterations and buy-in from over 100 interested parties (patients, caregivers, Population Health leadership at each site, primary care leadership at each site, clinicians at each site, national health system leaders, and payors), with attention to normalizing the language, ensuring it was easy-to-read, culturally appropriate for a large audience, and would not scare patients. In addition, there was much debate about who should sign this letter with ACP messaging. Primary care clinicians were concerned about patients feeling coerced or worried by getting a letter signed by them. There were also logistical challenges of obtaining signatures for each provider. Some felt that if it were signed by Population Health or health system executive leadership, it may feel too impersonal. Therefore, each site decided to have either the leadership for primary care, or in some cases, the leadership for each clinic, sign the ACP message.

(d). Mailed messages:

We also needed to ensure we reached people who did not have access to the online EHR patient portal. Therefore, we mailed ACP information to all patients in addition to the patient portal messages. This ensured that most, if not all, of our patients would have access to ACP information.

During the COVID pandemic early in the study, our advisory group helped us create additional COVID specific messaging for the mailed ACP information as there was concern from our advisors and clinical partners that ACP during this time could cause anxiety and fear. Thus, we attempted to acknowledge the pandemic and normalize the ACP process. Later in the study, our advisory boards helped us to update our messaging and letter format to be more engaging. These updates and needed translations at each UC site that were not part of the original proposal, but necessary given the pragmatic nature of the trial.

In addition, some sites required additional mailing and messaging. For example, UCSF leadership decided to also include ACP letters in eight additional languages (Armenian, Chinese, English, Korean, Spanish, Russian, Tagalog, and Vietnamese).

(e). Staged study launch:

As part of the implementation process, due to the large number of clinics at UCLA, the roll out occurred in phases to ensure readiness of each clinic at study launch and to ensure the healthcare navigators would not be overburdened with too many calls for Arm 3 participants. This phased roll-out resulted in delays due to several clinics having to put the launch of the study on hold due to COVID and required us to request a no cost extension on the project.

(f). Clinician training:

We offered clinician training that was variably attended before study launch at each clinic. Given COVID and some of the delays, at UCLA some of the clinics were offered ACP training before and some after study launch. In addition, some clinics received concurrent ACP training from other sources (clinician or trainee projects) in primary care and other specialties (see secular trends, Table 3).

(g). EHR ACP note templates:

UCLA and UCSF had been working with health system leadership for years prior to this trial to implement an EPIC-based ACP activity and to create standardized ACP resources and templates within the EHR. This was a challenge for this trial as documentation practices varied across sites, including ACP note templates and charting practices. However, each site had a centralized EHR location for ACP data elements. In addition, for this study, we were able to create a standardized healthcare navigator note template and were able to obtain approval at each site to implement it in the EHR.

(h). Execution of ACP forms:

A particular challenge was, and continues to be, patient execution of AD legal forms. During the COVID pandemic most patients avoided in-person healthcare visits and were isolated from their friends and family. California requires that an AD have the signature of two witnesses (at least one unrelated) or a notary public. Although electronic signatures are allowed for ADs in CA, due to legal and privacy concerns, e-signatures were not approved at the UC sites. In addition, at the time, virtual notaries were not approved for use in CA. This may have reduced the AD completion rate during the trial, although our team is still blinded to this trial outcome.

Outer Setting

There were several enablers of ACP however, that we believe placed ACP as a priority for the UC health system. In the years prior to the trial, UCLA and UCSF were part of a statewide ACP quality metric initiative for ACP for public hospitals. In addition, primary care was attempting to standardize and bill for an extended ACP discussion during AWV visits and had launched clinician training initiatives. This allowed us to align with UC leadership priorities.

Outer setting challenges included the critical incident of COVID, which made ACP implementation challenging. Furthermore, we realized that given the CMS reimbursement structure, it was possible that some patients could be charged for ACP conversations if they occurred outside of an AWV visit. Over the course of the study, we heard of one complaint of such an occurrence. However, we believe it is a barrier to ongoing ACP conversations and may have resulted in decreased ACP discussions in primary care during the study period.

Inner Setting

Prior to this trial, the relationships we created with health system leaders and interdisciplinary clinicians in primary care and at the UC Office of the President took years to establish and were key to this trial’s implementation. In addition, some of the inner setting culture of the health system had begun to change. For example, each site has robust Population Health teams dedicated to the use of data for quality improvement and to reach their quality metrics. In addition, there has been a greater interest in health equity and providing patients with language-appropriate and literacy-appropriate health education materials, such as the PREPARE materials. Understanding these needs and trends allowed our team to align our project with leaderships’ priorities.

However, one of the biggest inner setting barriers was, and continues to be, dedicated staff time and resources for ACP. A goal of the study was to create an infrastructure to support ACP and improve the efficiency of workflows given the importance to ACP to patient-centered care and the lack of resources in primary care. During this study, we also realized that ACP was not “owned” by any one clinician, service, or clinic. This created challenges in attempting to identify the workflows for obtaining and scanning in documents, as well as training and standardizing ACP conversations and documentation (e.g., a favorite example was a form that was scanned into the EHR and ACP tab as an AD, but was actually a dermatology picture of a toe). To address these issues, our team had to identify the codes being used, who was using them, and conduct targeted education across clinics. Ultimately, we were able to support workflows for scanned forms with high accuracy.

COVID also resulted in additional inner setting challenges. The priorities of the health system changed during this time, personnel were pulled from clinics, and health care navigators were asked to focus on COVID vaccines and other metrics. As described above, tele-health also changed several ways that patients interacted with the health system and could execute AD forms.

Individuals

For the innovation recipients in this trial (patients), health literacy, digital literacy, and language barriers continue to be a challenge. However, the availability of easy-to-use and -read materials that are available in several languages helped us work toward equity for ACP reach. The biggest challenge under this CFIR domain related to the changes in Population Health leadership, primary care leadership, and clinicians, and new relationships and collaborations had to be built throughout the course of the study.

Future Implementation Processes

What is exciting for our study team is to see the subsequent adaptations and use of our study materials for other clinical programs and research. For example, the UCSF automated build that identifies eligible patients and sends out an ACP intervention has been re-developed to allow the platform to be used for other research studies and clinical programs that will not be affected by EHR updates. The code for this EHR build has been shared with and will be adopted by UCLA and UCI and has currently been adopted and implemented at UCSF for all primary care clinics and a subset of surgical patients. Furthermore, the serious illness algorithm and our ACP messaging has also been adopted and adapted for use in other disciplines such as palliative care, oncology, and surgery, as well as by other researchers. The UC Office of the President has also launched a project focused on cancer patients to catalog all ACP documentation so that it can be more easily be assessed in the UC data warehouse. These structural improvements will help with ongoing adoption and maintenance of our study innovations.

RE-AIM Framework:

We describe our RE-AIM framework dimensions, components, and data in Table 4. For reach, there were 8707 seriously ill eligible patients at baseline; 6883 were eligible for an ACP intervention (i.e., no AD within the last 3 years) (Table 5). Of those patients, 92.2% received at least one intervention (i.e., through mail or EHR) and each patient on average received 1.8 (SD 0.8) interventions during the study period. Among the 6348 patients who received an intervention, 78.3% had access to an active EHR patient portal, 64.2% opened at least one portal message, and 90.5% of patients randomized to Arm 3 received outreach from a healthcare navigator. For implementation and maintenance, our ACP programs, ACP messaging, ACP interventions, ACP healthcare navigator trainings, and ACP EHR patient portal builds to identify and automatically send ACP interventions through the EHR were standardized across sites and have since become part of routine primary care at all sites as well as other patient populations (e.g., oncology, surgery, etc.) (Table 4).

Table 4.

RE-AIM Framework for This This Multi-site Advance Care Planning Pragmatic Trial

Dimension	Components	Data
Reach	Number and representativeness of eligible patients -Is intervention reaching target population	See Table 5
Effectiveness	Intervention effects on targeted outcomes  a) Primary Outcome:   ACP EHR documentation  b) Secondary outcomes*:   1. New ACP EHR documentation for those w/o ACP in the past 3 years   2. Healthcare utilization among decedents	Forthcoming in trial publication
Adoption	Number and representativeness of participating settings and providers -Extent those targeted to deliver the intervention are participating	UCLA: 41 clinics eligible and Population Health and primary care leadership agreed that all could be included. Five clinics did not have care coordinator so only randomized to arm 1 or 2. UCSF: 11 clinics eligible, Population Health and primary care leadership allowed 3 clinics to be included due to other pressing quality metrics required at the time of study launch. UCI: 6 clinics eligible and Population Health and primary care leadership agree that all could be included.
Implementation	The extent to which the intervention was consistently implemented by staff members	Consistently Implemented  (a) Standardized, validated algorithm to identify eligible patients with serious illness (i.e., EHR phenotype)  (b) Standardized messaging across sites  (c) Standardized protocols and ACP documentation for healthcare navigators across sites  (d) Standardized tracking of ACP outcomes across sites Allowed Adaptations Between Sites  a) UCSF and UCI used the PREPARE AD and UCLA used their own AD  b) UCSF had one additional mailing of AD information per Population Health leadership request due to COVID  c) EHR build to identify and send automated ACP messages had coding variations (i.e., “genotype”), but the resulting processes the same (i.e., the same “phenotype”)
Maintenance	The extent to which an intervention becomes part of routine organizational practices, and maintains effectiveness	ACP messaging adopted by Population Health, oncology, surgery, and other research studies at the UC’s and the VA Algorithm and EHR build to identify and send automated EHR ACP messages adopted in primary care across sites Adopted EHR build and messaging by surgery at UCSF

ACP = advance care planning, EHR = electronic health record,

^*For a research subset we will also assess self-reported outcomes not listed here

Table 5:

Reach Metrics of the ACP Trial Interventions

Site (All Arms)	UCLA	UCSF	UCI	Total
Patients with serious illness, n (%)	6607	746	1354	8707
Number eligible for intervention (no AD w/in 3 years), n (%)	5156/6607 (78.0%)	551/746 (73.9%)	1176/1354 (86.9%)	6883/8707 (79.1%)
At least 1 intervention (mail or EHR) during the study period, n (%)	4680/5156 (90.8%)	539/551 (97.8%)	1129/1176 (96.0%)	6348/6883 (92.2%)
Number of interventions sent, mean (SD)	1.7 (0.8)	2.1 (0.7)	2.2 (0.8)	1.8 (0.8)
Have active EHR patient portal, n (%)	3690/4680 (78.8%)	448/539 (83.1%)	831/1129 (73.6%)	4969/6348 (78.3%)
Opened EHR ACP message, n (%)	3227/4680 (69.0%)	337/539 (62.5%)	512/1129 (45.3%)	4076/6348 (64.2%)
Healthcare navigator outreach, n (%)	1633/1839 (88.8%)	231/236 (97.9%)	165/168 (98.2%)	2029/2243 (90.5%)

LESSONS LEARNED

In this challenging and rewarding study, we learned that implementing a multi-site health system-wide ACP program and pragmatic trial in primary care, with automated EHR-based cohort identification and intervention delivery, required a high level of multi-disciplinary key advisor engagement, standardization, and monitoring. Key lessons learned over the course of the study (Table 6) include the importance of: (1) Understanding that ACP is not owned by any one clinician or clinical service requiring buy-in from multidisciplinary teams and leadership; (2) Fostering working relationships with health system leadership early in the process and aligning studies and clinical programs to their priorities; (3) Engaging patient and caregiver advisors throughout the entire project to ensure that the innovations, messaging, outcomes, and study materials meet their unique needs; (4) Allowing time to standardize the EHR infrastructure for documentation and data extraction on the topic of interest; (5) Creating robust algorithms to identify the seriously ill cohort of interest, including identifying patients who have died; (6) Monitoring secular trends and allocating time and resources to address needed modifications and/or additional requests from the health system; (7) Standardizing operational workflows within health systems, such as scanning in ADs to ensure they are available at the point of care and for outcome ascertainment; (8) Building new relationships as leadership and clinical champions may change over time; (9) Using both CFIR and RE-AIM implementation frameworks to plan and evaluate ACP innovations; and finally, (10) Choosing your team wisely. Conducting palliative care trials can be hard, intimidating, and time consuming. However, they are ultimately rewarding, especially as the innovations are adopted, implemented, and disseminated into clinical practice to help improve patient care. Because of the expected challenges in conducting pragmatic trials, it is important to assemble the right team of individuals with complementary skillsets and those individuals who can also serve as a source of mutual support.

Table 6:

Key Lessons Learned Over the Course of This Pragmatic ACP Trial

(1) Understanding that ACP is not owned by any one clinician or clinical service requiring buy-in from multidisciplinary teams and leadership

(2) Fostering working relationships with health system leadership early in the process and aligning studies and clinical programs to their priorities

(3) Engaging patient and caregiver advisors throughout the entire project to ensure that the innovations, messaging, outcomes, and study materials meet their unique needs

(4) Allowing time to standardize the EHR infrastructure for documentation and data extraction on the topic of interest

(5) Creating robust algorithms to identify the seriously ill cohort of interest, including identifying patients who have died

(6) Monitoring secular trends and allocating time and resources to address needed modifications and/or additional requests from the health system

(7) Standardizing operational workflows within health systems, such as scanning in ADs to ensure they are available at the point of care and for outcome ascertainment

(8) Building new relationships as leadership and clinical champions may change over time; (9) Using both CFIR and RE-AIM implementation frameworks to plan and evaluate ACP innovations

(10) Choosing your team wisely with individuals with complementary skillsets and those who can also serve as a source of mutual support