Population-Based Pragmatic Trial of Advance Care Planning in Primary Care in the University of California Health System

Abstract

Introduction: Varying intensity of advance care planning (ACP) interventions at the population level has not been compared among seriously ill patients in primary care. This project will implement, test, and disseminate real-world scalable ACP interventions among primary care clinics across three University of California Health systems. The three ACP interventions are (1) distribution of an advance directive (AD) with targeted ACP messaging, (2) the AD, messaging, plus prompting patients to engage with the Prepare For Your Care website (PREPARE), and (3) the AD, messaging, PREPARE, plus Care Coordinator engagement with patients and clinicians.

Methods: We will identify a population cohort of seriously ill primary care patients and implement the ACP interventions using electronic health record (EHR) patient portals and postal mailings. Forty-five clinics across the three health systems will be cluster randomized to one of the three ACP interventions. The primary outcome for the population cohort is AD or Physician Orders for Life-Sustaining Treatment documentation in the EHR. A subset of the population cohort will be surveyed to assess patient-centered outcomes, including care consistent with goals at baseline, 12 months, and 24 months. Caregivers will be interviewed if patients are unable to be surveyed or die. ACP documentation, goal concordant care, and among decedents, health care utilization will be compared among intervention arms.

Study Implementation: Challenges and Contributions: The project is guided by a Study Advisory Group and Community Advisory Groups at each site to ensure rigorous patient-centered methods and consistency of implementation. Intervention fidelity will be evaluated using the Reach, Efficacy, Adoption, Implementation, and Maintenance (RE-AIM) framework. Challenges to implementation of this three-site health system trial and to intervention fidelity stem from site/clinic/system cultures, increasing attention to end-of-life care from payers and regulators, and growing pressures by health systems to implement ACP interventions. Stakeholder engagement is required to ensure consistent interventions across sites.

Background

Seriously ill patients commonly receive treatments at the end of life that are not guided by their values and goals,^1,2 and their families often sustain financial³ and emotional injury.⁴ Advance care planning (ACP)—a process of understanding and sharing personal values, life goals, and preferences regarding future medical care⁵—is associated with earlier hospice referral, better end-of-life quality for patients, and better bereavement adjustment for families.¹

Experimental evidence shows that ACP interventions increase completion of advance directives (ADs), increase discussions about care preferences, and concordance between a patient's preferences and end-of-life care.⁶ Multicomponent interventions, many of which are not feasible in real-world settings, have been shown to increase the likelihood of AD completion,⁷ but much of this work has focused on hospitalized patients⁸ and those with cancer.⁹ A systematic review of ACP interventions found decreased use of life-sustaining treatment and increased hospice and palliative care but concluded that more experimental designs and community studies are needed.¹⁰

While studies have demonstrated efficacy of easy-to-use patient-facing interventions,^11,12 the comparative effectiveness of ACP interventions has not been tested in randomized trials at the health system level and population level and in primary care,¹³ where clinicians are already overburdened with the complex management of seriously ill patients. This is important because health systems need feasible evidence-based ACP interventions.

The conceptual framework of this study is based on the ACP model of Sudore and collegues,^14,15 which proposes that behavior change can be influenced through interventions that affect knowledge, self-efficacy, and readiness to engage in ACP. Easy-to-read ADs and the Prepare For Your Care (PREPARE) online program are based on this framework and have been shown to increase patients' readiness to engage in ACP.^11,12,16,17 One-on-one facilitation has also been shown to increase ACP documentation and engagement¹⁸ but is often costly and time consuming (Fig. 1). These interventions have not previously been compared.

FIG. 1.

Mediators and factors associated with ACP. ACP, advance care planning.

Objectives and hypotheses

The goal of this pragmatic trial is to implement, test, and disseminate real-world scalable ACP interventions among primary care clinics across three University of California (UC) Health systems with the following study aims.

Aim 1

Guided by stakeholders at each site, determine barriers, facilitators, and best workflow plans to implement the three ACP interventions.

Aim 2

Implement the three ACP interventions in primary care clinics among patients with serious illness using a cluster randomized design across the three UC Health systems: distribution of an AD (arm 1), distribution of the AD plus prompting the patient to engage with the PREPARE website (arm 2), and AD distribution plus PREPARE plus a nonclinician Care Coordinator (CC) intervention (arm 3).

Aim 3

Compare the effectiveness of the three interventions on formal ACP documentation (primary outcome) across the three UC Health systems and, among a subset of patients and caregivers, assess the patient-reported outcomes of goal-concordant care and ACP engagement.

We hypothesize that there will be additive improvement from arm 1 to arm 2 to arm 3 on formal ACP documentation (primary outcome) at the population level and receipt of goal-concordant treatment among the subset of surveyed patients and caregivers.

Methods

Study design

Study overview

In concert with patient and clinician stakeholder partners, this project will develop clinic level implementation strategies for the three evidence-based ACP intervention comparators. Using a cluster randomized design, we will implement real-world system-wide ACP interventions in primary care clinics for a seriously ill population cohort across the three UC Health systems. The effect of the interventions will be evaluated by formal ACP documentation (Fig. 2). To evaluate a more detailed set of outcomes including goal-concordant treatment and ACP engagement, a subset of patients in a research cohort will be surveyed and their medical records will be reviewed. Patients' caregivers will be interviewed if patients are unable or if the patient dies during the follow-up period (Fig. 3).

FIG. 2.

UC ACP study flow diagram: population cohort. ACP, advance care planning; AD, advance directive; UC, University of California.

FIG. 3.

UC ACP study flow diagram: research cohort.

Setting

Thirty-six UCLA primary care clinics, 6 UCI clinics, and 3 UCSF clinics. Five additional clinics at UCLA will be randomized to arm 1 or 2 only because these clinics do not have a CC.

Population

The population cohort (estimated N = 6400) includes patients aged 18 years or older who have serious illness and attended at least two primary care office visits during the prior 12 months. Patients are included regardless of prior ACP documentation. Serious illness was defined using administrative billing codes, encounter data, and clinical information available in the electronic health record (EHR) (Table 1). The definition of serious illness requires an at-risk medical diagnosis (cancer, heart failure, chronic obstructive pulmonary disease, end-stage liver disease, end-stage renal disease, or amyotrophic lateral sclerosis) linked with advanced age or a level of severity such that ACP would be a priority, defined as: (1) poor short-term survival prognosis or (2) developing incapacity or (3) worsening functional status or (4) high burden of disease (causing excessive suffering, which may be related to health care utilization). To validate the serious illness definition, chart abstraction of 306 patients across the three health systems confirmed that 301 (98%) met the ACP Priority criteria.

Table 1.

Advanced Illness Definitions and Identification Criteria

Advanced illness group	Advanced illness group definition	Identification criteria
Advanced cancer	Solid tumor or hematologic cancer that is incurable	(Problem List ICD code for advanced cancera AND oncology visit in the last 12 months) OR (Ambulatory encounter billing ICD code for advanced cancera AND chemotherapy in the last 2 Years)
Advanced HF	HF that substantially affects the patient's function ([{Shortness of breath or weakness or chest pain or ectopy with exertion or edema affecting function or cannot do activities} and not due to another cause] or class 3 or 4) AND last known LVEF <31%	(Problem List or Ambulatory encounter billing ICD code for HFb in the past year AND any left ventricular ejection fraction over the last 3 years <31%) OR (Problem List for HFb AND at least 1 hospital admission with an ICD code for HFb)
Advanced COPD	COPD that substantially affects the patient's function ([shortness of breath with exertion or cannot do activities and not due to another cause] or GOLD class 3 or 4) and FEV1 < 50% predicted OR O2-dependent at home (all the time or for exertion but not just at night)	Problem List ICD code for COPDc AND [(V or Z code for home oxygend) OR (At least 1 hospital admission with an ICD code for COPDc in the last year)]
Decompensated liver disease	Cirrhosis with evidence of decompensation represented by ascites, esophageal variceal bleeding, hepatorenal syndrome, or hepatic encephalopathy	Problem List ICD code for cirrhosise AND [hepatic decompensationf measured by: (Problem List or Ambulatory encounter billing ICD code) OR (MELD >18)]
End-stage renal disease	Chronic kidney disease on hemodialysis or hemodialysis and/or renal transplant being actively considered or history of renal transplant	(Problem List AND Ambulatory encounter billing ICD-10 code for end stage renal diseaseg) OR [(Problem List OR Ambulatory encounter billing ICD code for end stage renal diseaseg) AND (Nephrology visit in last year-inpatient or ambulatory)]
ALS	ALS with progressive symptoms impacting functional status	Problem List AND Ambulatory encounter billing ICD code for ALSh
Vulnerable elder with serious illness	Age 75 years or older with at least one serious illness	Age 75 years or older AND (Problem List ICD for advanced cancer,a heart failure,b COPD,c cirrhosis,e end stage renalg or ALSh)

ICD diagnoses:

^aCancer: ICD-9 liver (155, 155.1, 155.2), esophagus (150, 150.1-5, 150.8-9), stomach (151, 151.1-6, 151.8-9), pancreas (157, 157.1-3, 157.8-9), peritoneum (158, 158.8-9), brain (191, 191.1-9), secondary malignant neoplasms (197, 197.1-8, 198, 198.1-8, 198.81-2, 198.89), malignant pleural effusion (511.81), malignant ascites (789.51), disseminate malignant neoplasm (199), leptomeningeal carcinomatosis (349.2), heme malignancy (200.7, 200.71-8, 201, 201.4-7, 201.9, 202.8, 204.02, 204.12, 204.92, 205, 205.02, 205.12, 208.12). ICD-10 liver excluding HCC (C22.1-4), esophagus (C15.3-5, C158-9), stomach (C16.0-6, C16.8-9), pancreas (C25.0-3, C25.7-9), peritoneum (C48.0-2, C48.8), brain (C71, C71.0-9), secondary malignant neoplasms (C78.00, C78.1-2, C78.39, C78.4-7, C78.89, C79.00, C79.11, C79.19, C79.2, C79.31-2, C79.49, C79.51-2, C79.60, C79.70, C79.81, C79.82, C79.89, C79.9), malignant pleural effusion (J91.0), malignant ascites (R18.0), disseminated malignant neoplasm (C80.0), leptomeningeal carcinomatosis (G96.12), heme malignancy (C81.09, C81.19, C81.29, C81.39, C81.49, C81.79, C81.99, C85.19, C85.29, C85.89, C91.02, C91.12, C91.52, C91.62, C91.92, C91.A2, C92.02, C92.12, C95.12).

^bHF: ICD-9 (428, 428.1-4, 428.21-3, 428.31-3, 428.41-3, 428.9, 398.91, 402.01, 402.11, 402.91, 404.01, 404.03, 404.11, 404.13, 404.91, 404.93); ICD-10 (I09.81, I11.0, I13.0, I13.2, I50.1, I50.9, I50.20-23, I50.30-33, I50.40-43).

^cChronic obstructive lung disease: ICD-9 (491.2, 492.21-22, 493.2, 493.21-22, 496); ICD-10 (J43.9, J44.0, J44.1, J44.9).

^dOxygen: ICD-9 (V46.1-2, V46.11-14); ICD-10 (J95.850, Z99.1, Z99.11-12, Z99.81).

^eCirrhosis: ICD-9 (571.2, 571.5, 571.6); ICD-10 (K70.3, K70.30, K70.31, K70.4, K70.40, K70.41, K74.3-5, K74.60, K74.69).

^fHepatic decompensation: ICD-9 encephalopathy (572.2); ICD-10 (K72.90, K72.91); hepatorenal ICD-9 (572.4); ICD-10 (K76.7); peritonitis ICD-9 (567, 567.2, 567.21, 567.23, 567.29, 567.8, 567.89, 567.9); ICD-10 (K65.9, K67, K65.0, K65.2, K65.8); esophageal varices: ICD-9 (456, 456.2, 456.21); ICD-10 (I85.01, I85.10, I85.11); ascites: ICD-9 (789.5, 789.51, 789.59); ICD-10 (R18.8).

^gEnd-stage renal disease: ICD-9 (585.5, 585.6, V42.0, V45.1, V45.11, V45.12, V56, V56.0-3, V56.31-32, V56.8); ICD-10 (N18.5, N18.6, Z94.0, Z99.2, Z91.15, Z49.31, Z49.01, Z49.02, Z49.32).

^hALS: ICD-9 (335.2); ICD-10 (G12.21).

Cancer diagnoses from Anand et al.³⁸

ALS, amyotrophic lateral sclerosis; COPD, chronic obstructive pulmonary disease; FEV1, forced expiratory volume in 1 second; GOLD, global initiative for obstructive lung disease; HF, heart failure; ICD, International Classification of Diseases; LVEF, left ventricular ejection fraction.

The research cohort is a subset of the population cohort who will be surveyed and includes 900 patients (300 in each of the three study arms) who have no formal ACP documentation in the EHR in the past three years. During the consent process, individuals will be asked to designate and provide contact information for a caregiver.

Interventions

Interventions are conceptualized as system-based quality improvement efforts. Each intervention arm builds on the prior arm, so intervention arm 2 includes the components of intervention arm 1, and arm 3 includes the components of arm 2. The intervention will only be delivered to patients in the population cohort who have no formal ACP documentation in the EHR in the past three years. We chose a three-year time frame because surrogate specification and ACP preferences change over time, especially for individuals with serious illness.¹⁹

While these interventions are based on studies that implemented an intensive one-on-one introduction of an easy-to-read AD and the PREPARE website in an office setting, the proposed interventions aim to introduce these interventions using system-based mechanisms and reminders. The ACP intervention will use the Epic patient portal and postal service mailing three weeks before a nonurgent primary care appointment. Patients will receive an introductory letter from the health system, in English or Spanish, encouraging patients to engage in ACP. The messaging and letters were developed with more than 100 stakeholders across all sites. The message introduces ACP, the purpose of the AD and instructions for completing and returning it, and a prompt to discuss ACP at their primary care visit. ACP interventions are tied to primary care appointments. Patients without a scheduled appointment six months after the initiation of the ACP intervention will receive the intervention regardless of an upcoming appointment (i.e., bulk mailing). A reminder to bring the AD to clinic will be delivered before the scheduled clinic visit using automated patient portal messages or telephone calls, if available and according to local practice. We will resend the intervention to patients (e.g., a booster) if they do not have EHR ACP documentation six months after the initial intervention, and a reminder will be sent at six month intervals to those lacking ACP documentation.

The ACP intervention for patients in arm 1 includes a mailed AD or link to an AD accompanying the message. Arms 2 and 3 will include a referral to the PREPARE website (www.prepareforyourcare.org) via a link in the patient portal or the postal letter and a pamphlet. Website Uniform Resource Locators specific to each arm will track website use for implementation monitoring.

Patients in arm 3 will be contacted by a CC, who is a nonclinician health care navigator. Most eligible clinics across the sites have CCs providing health navigation to high-risk patients; this intervention will add ACP to their repertoire. CCs will be apprised of patients receiving the intervention and will: (1) evaluate AD completion; (2) introduce ACP to the patient and facilitate referral to the PREPARE website, access ADs, and answer basic ACP questions or refer to the patient's primary care provider (PCP); (3) facilitate completion and collection of ACP documents; (4) notify PCPs about obstacles to ACP, and (5) facilitate PCP consideration of Physician Orders for Life-Sustaining Treatment (POLST) in patients with a hospital discharge code status of do not resuscitate (DNR). CCs will be trained and provided scripts and checklists. The eight-hour training will introduce ACP and ACP documents and include role play scenarios with actors.

We will offer 20–30-minute ACP educational sessions for each primary care clinic, tailored to study arm, about ACP basics and ACP conversations, documentation, and billing.

Randomization

A statistician not involved in recruitment or data collection will randomize each clinic to an intervention arm. We will use constrained randomization to ensure that clinics with the same CC are not randomized to different arms and that each treatment arm has a similar number of patients, age of patients, and rates of AD completion overall and in the past three years.

Blinding

Since interventions are implemented in the context of a pragmatic quality improvement project at the population level, patients, clinicians, and research staff will not be blinded to intervention allocation.

Outcome measures

Primary outcomes

The primary outcome for the population cohort is formal ACP documentation (AD, POLST, power of attorney, or out-of-hospital DNR form) in the health system EHR using administrative data. Validation of administrative data for ACP documentation of 306 patients compared to medical record abstraction across the three UC sites demonstrated >95% accuracy.

The primary outcome for the research cohort is patient-reported receipt of goal-concordant care. There is currently no gold standard measure to assess goal concordant care; thus, we will use the three different approaches with survey and/or chart data including cross-sectional care preference questions from Study to Understand Prognosis and Preferences for Outcomes and Risks of Treatments (SUPPORT),²⁰ future care preference questions from the PREPARE AD using a method to match to chart abstraction data,²¹ and questions in the after death survey tapping into goal-concordant care^16,22 (Table 2).

Table 2.

Survey Measures

Survey items	Baseline	12-month patient	24-month patient	Caregiver after death
Patient items
Contact info	X
Demographicsa	X			X
Experience with ACP tools (intervention)	X	X	X
Patient engagement in ACP16	X	X	X	X
Medical care preference current (support)20	X	X	X	X
Care preference and future health states (PREPARE)	X	X	X
Confidence and trust in future care	X	X	X
Desired role in decisions24	X	Xb	Xb
CAHPS communication composite25	X	X	X
Self-rated health and QOL30	X	X	X	X
PROMIS global physical health (GPH2)30	X	X	X
Depression items from PROMIS 2931	X	Xb	Xb
Anxiety items from PROMIS 2931	X	Xb	Xb
Social support32	X
Religious/spiritual29	X
Bereaved family survey33				X
Goal-concordant care22				X
Decisional regret27				X

^aDemographics collected for first patient survey and, if needed, first caregiver survey.

^bCaregiver version excludes these items.

ACP, advance care planning; AD, advance directive; CAHPS, Consumer Assessment of Healthcare Providers and Systems; GPH2; PREPARE, Prepare For Your Care website with questions adapted from the easy-reading AD; PROMIS, Patient-Reported Outcomes Measurement Information System; QOL, quality of life.

Secondary outcomes

For the population cohort, we will evaluate new ACP documentation since the baseline intervention to 12 and 24 months. This analysis will include only patients with no formal ACP documentation in the past three years. In the population cohort, we will measure health care utilization among decedents including hospitalization, intensive care unit (ICU), and emergency department (ED) use.

Secondary outcomes for the research cohort include ACP engagement (e.g., self-reported discussions and documentation) using questionnaires²³ and medical record review. Measures include ACP documentation; readiness to engage in ACP,¹⁶ surrogate designation, discussions, and documentation; desired role in decisions²⁴; patient reports about physician communication²⁵; confidence and trust in future care (Totten AM et al.)²⁶; and caregiver reports of end-of-life care including decisional regret.²⁷

Recruitment and study procedures

Recruitment

Since the population cohort is part of a quality improvement effort that includes all seriously ill primary care patients, recruitment methods apply only to the research cohort. PCPs will be sent a secure e-mail to obtain permission to contact their eligible patients. PCPs will be asked to exclude patients with cognitive impairment, language other than English or Spanish, or for whom the survey might cause psychological harm. We will send two permission e-mails to physicians, after which we will assume assent to contact their patients.

Eligible patients will be mailed a letter introducing the study, a written consent form that also asks about a caregiver who we may contact if needed in the future, and the survey instrument. Patients may enroll in the study by signing the consent form, completing the survey, and mailing it back. Included in the introductory letter is an “opt out” telephone number.

Patients who do not return a survey by mail or opt out may be contacted two weeks after the survey mailing by telephone or e-mail to assess interest in participating. No response after three voice messages will be considered an opt out. For patients recruited by telephone, we will exclude individuals who cannot perform informed consent after three attempts via teach-back.²⁸ Surveys can be conducted by telephone or online, per patient preference; however, a signed consent form will be required to use the survey data.

Patients will receive a $30 gift card for each survey (baseline, 12-month, 24-month). If the patient loses capability before the 12- or 24-month survey, the designated caregiver, if provided by the patient, will be approached for follow-up surveys. For decedents, designated caregivers will be approached to complete the after death survey (administered three months after death). Caregivers will receive a $30 gift card for each completed survey.

Data collection and analysis plan

Data collection: administrative data

EHR data will be used to identify the population cohort, administer the intervention, and supply outcome information for the population cohort including formal ACP documentation, hospitalizations, ICU stays, ED use, and to the degree available hospice use, vital status and venue of death.

Data collection: questionnaires

For the research cohort, data collection will take place through written, telephone, and online questionnaires. Data will be entered into secure REDCap databases.

To characterize the research cohort, the survey will assess demographics, religiosity and spirituality,²⁹ self-rated health and quality of life,³⁰ depressive symptoms and anxiety,³¹ and social support.³² For implementation fidelity evaluation, the survey will ask about patient experience with ACP tools. The after death survey contains modified items from the Bereaved family survey,³³ caregiver self-rated health and quality of life,³⁰ and other items (Table 2).

Data collection: medical records

For the research cohort, medical records will be reviewed after the final survey or the patient's death. All notes will be reviewed including clinician, social work, and pastoral care. The abstraction will capture demographic and clinical information including comorbidities, content of ADs and POLST forms, and orders concerning initiating, withholding or withdrawing resuscitation, and other life-sustaining treatments. Documentation of ACP discussions, goals of care, surrogate specification, and decision making concerning life-sustaining treatment and end-of-life care will be abstracted to inform whether a patient received goal-concordant care. ED, hospital, and ICU episodes will be recorded as well as time spent in adverse health states. A 10% sample of records will be re-abstracted to evaluate reliability.

Data management and security

Administrative data for the population cohort will be maintained behind UC intuitional firewalls on secure servers. Survey data for patients in the research cohort will be collected on paper and then entered into REDCap using a unique study ID or entered into REDCap in real time by a research assistant during telephone or in person interview, or via an e-mail link. REDCap will be managed at the central site and stored behind the firewall. A unique, non-identifying patient study ID removed from any personal health information will be stored in a secure file. All paper surveys and consent forms will be stored in locked files in locked research offices. The study ID will be used to store medical record abstraction data behind the firewall.

Statistical analysis plan

Analyses will be performed in an intention-to-treat manner to compare the three study arms. For primary endpoints, Bonferroni correction will be applied to account for three pairwise comparisons, and a p-value of <0.017 will be considered statistically significant for each comparison. SAS software will be used.³⁴

Sample size and power

A simulation study was performed to evaluate the statistical power of the study using initial baseline data among 45 clinics (∼6400 serious illness patients) to be randomized to 3 arms. Prior studies suggest that similar interventions increase AD completion from 25% to 47%.³⁵ Consistent with these studies, the PREPARE trial demonstrated an effect size of 0.5; to account for the pragmatic nature of the proposed trial, a conservative estimate would be a 25% absolute increase in AD completion from baseline. With about 37% of serious illness patients having ADs at baseline, we expect that formal ACP documentation will increase to 47%, 62%, and 72% for arm 1, arm 2 and arm 3, respectively. Accounting for the multiple comparisons for the primary endpoint, we expect to have ≥80% power for all comparisons with intraclass correlations ≤0.02.

Evaluation of hypotheses

For the primary outcome, cumulative ACP documentation rates for the population cohort from baseline to 12 and 24 months (or death) will be compared as a difference of differences among the three study arms. The baseline date is defined as the clinic rollout date. Generalized estimating equation (GEE) logistic regression models will be used for this analysis to account for patients clustered within clinics. Covariates include, a priori, patient age, gender, race/ethnicity, serious illness category, clinic baseline AD completion rate, site, time (baseline, 12 months, or 24 months), study arm, and study arm–time interaction. We will also include any additional patient characteristics that differ by study arm. We hypothesize that ACP documentation will increase in arm 3 more than arm 2 more than arm 1. In the secondary analysis, new ACP documentation will be compared among the three arms at 12 and 24 months (or death) from baseline among patients with no formal ACP documentation in the past three years.

For decedents in the population cohort, we will assess hospital, ICU, and ED visits within 30 days of death between the three study arms using the GEE logistic model with patients nested in clinics using the covariates included above.

Ethics determination

This study was approved by the UCLA Institutional Review Board (IRB) (18-001612); the UCSF and UCI IRBs will rely on the UCLA IRB.

Study Implementation: Challenges and Contributions

Patient and key stakeholder engagement

The project is guided by a Study Advisory Group (SAG) that meets biannually with e-mail contact as needed. The SAG includes state and national leaders in ACP and palliative care, clinicians, patients and caregivers from each site, including the UC Office of the President, National POLST paradigm, Palliative Care Quality Network, Palliative Care Research Cooperative Group, Center to Advance Palliative Care, and National Palliative Care Research Center. Meetings are coordinated with assistance of the Coalition for Compassionate Care of California (CCCC). The SAG provides oversight to ensure that rigorous patient-centered methods are employed including consistency of intervention implementation across the clinics at the three sites.

Community Advisory Groups (CAGs) at each site include patients with serious illness, caregivers, and clinicians who work in participating clinics. CAGs meet two to three times each year with e-mail contact as needed. CAGs not only review and give feedback on the protocol and materials to ensure that they are patient-centered but also will review project progress, evaluate enrollment efforts, and review early study findings.

Fidelity monitoring in the context of complex interventions

Reach, Efficacy, Adoption, Implementation, and Maintenance (RE-AIM^36,37) of the interventions will be evaluated over time and across clinics and sites using the measures and sources listed in Table 3. The evaluation aims to facilitate implementation and afford a mechanism to evaluate heterogeneity of effect.

Table 3.

Framework for Monitoring Intervention Fidelity

RE-AIM construct	Measure	Data source
Reach: individuals from the target population who participated in the intervention	1. Number of ADs distributed	1. ADs accessed through portal link and mailed
	2. Number of patients who accessed the PREPARE website	2. PREPARE website login data provided through data dashboards
	3. Number of patients touched by CCs	3. CCs log of patients for whom they introduce ACP
Efficacy: intervention impact on outcomes	1. Population-level AD and POLST completion	1. Data from electronic health record
	2. Patient-level reports of goal-concordant care	2. Medical record and survey data
	3. Caregiver reports of end-of-life experience and goal-concordant care	3. Medical record and survey data
Adoption: those who adopted the intervention	Variation among clinics and sites in number of ADs distributed, number of patients who accessed the PREPARE website, and number of patients touched by clinical CCs	Collected as in “Reach” above
Implementation: extent intervention is implemented as intended	1. Receipt of ADs	1. Patient portal recipients that accessed message and AD link.
	2. Receipt of invitation to PREPARE website and accessibility	2. Patient portal recipients that accessed message and REPARE website link.
	3. Interaction with CCs and discussion with primary care physician	3. Follow-up survey items from the research cohort
Maintenance: extent to which intervention is sustained over time	Reassessment of reach, adoption, and implementation at 6, 12, 18, and 30 months. Reassessment of efficacy at 6, 12, and 30 months	Collected as in “Reach, Adoption, and Implementation” above

CC, care coordinator; POLST, Physician Orders for Life-Sustaining Treatment; RE-AIM, Reach, Efficacy, Adoption, Implementation, and Maintenance.

Response to challenges and barriers to study success

This project implements a coordinated system-level intervention across the three UC Health systems in an area of clinical practice that is receiving considerable attention from payers and regulators. This translates into pressure on health systems to implement interventions, even though evidence-based information about intervention effectiveness and efficiency is lacking. Furthermore, end-of-life health care costs are an increasing focus of health system intervention efforts. The project's main challenges relate to maintaining support for the proposed interventions at a time of change.

Health system pressure to modify interventions

Health systems may implement interventions in response to market forces or influence leaders rather than evidence from trials. The interventions proposed by this project are evidence based, but comparative effectiveness data at a population level are lacking. Keeping the three health systems focused on the proposed interventions rather than adding additional components has required substantial stakeholder engagement to maintain buy in. This tension translated into some clinics desiring to employ the arm 3 intervention, which would conflict with the proposed three-arm comparative effectiveness protocol. This required identification of a smaller number of protected clinics to participate in this trial. Redundancy in the project ensured adequate patient sample and clinics, preserving sample size and statistical power.

Secular trends

ACP is changing considerably including addition of new billing codes, increased monitoring, and greater pressure from insurers including direct-to-patient palliative care interventions. During the intervention period, all three health systems rolled out EHR ACP components separate from this project. This prompted additional efforts working with health system EHR leadership to preserve the proposed automated ACP interventions, which are built through the EHR. At each site, we are collecting environmental changes in ACP to be able to chronicle secular trends when data analysis is performed.

Ensuring fidelity to the intervention across sites

The proposed interventions depend on participation of clinics and a build within the EHR to implement consistent interventions across sites. The sites use the same EHR but run different versions, and there is a variable level of resource support for build efforts across sites. The patient cohort method is identical across sites with use of a consistent set of International Classification of Diseases (ICD) codes and administrative data elements that demonstrated valid patient identification on medical record review across sites. Concerning the build for automated ACP messaging and mailings, the EHR developers decided together that it was most feasible to build interventions that are “genotypically different, but phenotypically the same” meaning that the intervention as seen by the patient will be identical, although the inner working in the EHR will differ. The builders propose that this strategy is a methodological strength so that the best EHR build can be implemented across UC Health after study completion.

Plans for dissemination and spread

The SAG includes some of the nation's most influential organizations involved in improving ACP and palliative care. Their involvement will facilitate a variety of dissemination approaches, if the interventions are found to be effective. The UC Office of the President pledged to promulgate a successful intervention throughout UC Health. The Center to Advance Palliative Care, California Health Care Foundation, National POLST paradigm, and CCCC all have extensive relationships with community organizations that can be leveraged to ensure that study findings are implemented. We will work with the CAGs and SAG to design public forums for sharing study results.

Author Disclosure Statement

No competing financial interests exist.