Primary Palliative Care for Emergency Medicine (PRIM-ER): Protocol for a Pragmatic, Cluster-Randomised, Stepped Wedge Design to Test the Effectiveness of Primary Palliative Care Education, Training and Technical Support for Emergency Medicine

Abstract

Introduction

Emergency departments (ED) care for society’s most vulnerable older adults who present with exacerbations of chronic disease at the end of life, yet the clinical paradigm focuses on treatment of acute pathologies. Palliative care interventions in the ED capture high-risk patients at a time of crisis and can dramatically improve patient-centred outcomes. This study aims to implement and evaluate Primary Palliative Care for Emergency Medicine (PRIM-ER) on ED disposition, healthcare utilisation and survival in older adults with serious illness.

Methods and analysis

This is the protocol for a pragmatic, cluster-randomised stepped wedge trial to test the effectiveness of PRIM-ER in 35 EDs across the USA. The intervention includes four core components: (1) evidence-based, multidisciplinary primary palliative care education; (2) simulation-based workshops; (3) clinical decision support; and (4) audit and feedback. The study is divided into two phases: a pilot phase, to ensure feasibility in two sites, and an implementation and evaluation phase, where we implement the intervention and test the effectiveness in 33 EDs over 2 years. Using Centers for Medicare and Medicaid Services (CMS) data, we will assess the primary outcomes in approximately 300 000 patients: ED disposition to an acute care setting, healthcare utilisation in the 6 months following the ED visit and survival following the index ED visit. Analysis will also determine the site, provider and patient-level characteristics that are associated with variation in impact of PRIM-ER.

Ethics and dissemination

Institutional Review Board approval was obtained at New York University School of Medicine to evaluate the CMS data. Oversight will also be provided by the National Institutes of Health, an Independent Monitoring Committee and a Clinical Informatics Advisory Board. Trial results will be submitted for publication in a peer-reviewed journal.

Trial registration number

NCT03424109; Pre-results

Strengths and limitations of this study.

• Novel, pragmatic, cluster-randomised trial using stepped wedge design.

• Aims to shift the clinical practice paradigm for the emergency care of seriously ill older adults.

• Interventions tested in geographically and contextually diverse settings across 18 health systems nationwide.

• Potential limitations of the study design include provider cross-contamination and staff turnover.

Introduction

Rationale

The high intensity of end-of-life care in the USA is now considered an epic public health problem. Persons receiving many life-sustaining therapies do not appear to show a benefit of better health or longer life.¹ Emergency departments (EDs) care for society’s most vulnerable older adults who present with exacerbations of chronic disease at the end of life, yet the clinical paradigm continues to focus on treatment of acute illness and injury. Palliative care interventions in the ED capture high-risk patients at a time of crisis and can dramatically improve patient-centred outcomes.^{2 3}

Half of Americans 65 years and older are seen in the ED in the last month of life, and three-quarters visit the ED in the 6 months before their death.⁴ Emergency care has not fully adapted to the needs or goals of seriously ill patients who prefer to have care delivered at home.^{5 6} Palliative care teams are now present in over two-thirds of hospitals, as well as 98% of National Cancer Institute-designated cancer centres; however, consultation by palliative care teams is typically only available Monday–Friday during business hours, and palliative care teams are not routinely available to come to the ED when a patient is in crisis.⁷

An ED visit is often described as a sentinel event signifying a breakdown in care coordination for older adults.^{8 9} Since EDs sit at the crossroads of ambulatory and inpatient care, they can, and often play, a pivotal role in balancing the potential harms and benefits of hospitalisation for seriously ill, vulnerable older adults.^10–13Hospitalisation for older adults carries significant risks such as iatrogenic complications, functional and cognitive decline and loss of independence, but emergency providers may be unaware of safe alternatives.^14–19

Emergency medicine developed as a specialty to treat the acutely ill and injured, yet EDs increasingly care for older adults with multiple comorbid conditions who present for acute exacerbations of chronic illness. Visits to the ED by older adults are increasing both in frequency and as a proportion of all ED visits. In 2011, adults aged 65 years and older comprised 15% of total ED visits, had the highest severity of illness and represented 44% of all admissions from the ED.²⁰ The number and rate of admissions to the intensive care unit (ICU) by emergency providers have also increased, especially among older adults.²¹ The proportion of the US population 65 years and older will continue to grow, and EDs will see an increase in both the number of older adults and the complexity of care they are required to provide.²² The ED presents a key decision point at which providers set the subsequent care trajectory, including whether an older adult is hospitalised and to which setting. Emergency physicians can therefore play an integral role in transforming care for older adults through evidence-based models of care delivery that emphasise tradeoffs between potential benefits and harms.¹³ Until recently, little attention has been paid to the delivery of goal-concordant care in the ED for older adults with serious illness. The default treatment plan is to deliver treatment intensive care that favours life-sustaining therapies, many of which may contradict the wishes of older adult patients.

Objectives

This project seeks to implement and test a novel, highly efficient pragmatic intervention to shift the clinical practice paradigm of emergency medicine. Our generalisable approach will build on the effectiveness of specialty palliative care demonstrated in ED patients admitted to the hospital to increase primary palliative care skills and knowledge among emergency providers. This will improve quality of care and decrease healthcare utilisation for millions of older Americans.

The intervention itself is multifaceted and highly innovative, employing the latest technology and techniques in clinical informatics and innovations in medical education. Building on lessons learnt from prior studies, this intervention has been designed to employ the most effective pedagogical techniques, including the use of simulated patients and families, role-playing, teaching-to-teach exercises and small group learning with constructive feedback from master clinicians.^{23 24} Despite recognition of the importance of end-of-life care to individuals, families and society, many interventions have failed to capture its complexity. Interventions have focused on single-level solutions without considering more complex, multilevel approaches.

Our pragmatic, cluster-randomised stepped wedge design will test the effectiveness of Primary Palliative Care for Emergency Medicine (PRIM-ER) in 35 EDs across the US. PRIM-ER is an education, training, and technical support quality improvement (QI) intervention with four core components: (1) evidence-based multidisciplinary primary palliative care education, (2) simulation-based workshops on communication in serious illness, (3) clinical decision support (CDS) and (4) provider audit and feedback. These core components will be implemented in each participating health system to improve the care older adults with serious illness receive in the ED setting. In the pilot phase of the project, we will tailor PRIM-ER to the emergency provider workforce and a more diverse ED context using an agile implementation framework approach at two sites to test feasibility, fidelity and usability. In the implementation and evaluation (IE) phase, we will implement PRIM-ER in a cluster-randomised, stepped wedge design in the remaining 33 EDs and measure the effect of PRIM-ER on: (A) ED disposition to an acute setting; (B) healthcare utilisation in the 6 months following the ED visit; and (C) survival following the index ED visit.

Proposed study outcomes leverage existing experiences from prior research and empirical evidence, including but not limited to a Center for Medicare and Medicaid Innovation demonstration project whereby researchers addressed the gap in the emergency care of seriously ill older adults to test the traditional consultative model of palliative care in the ED via a randomised controlled trial.³ Findings demonstrated that early ED-based specialty palliative care consultation for patients with advanced cancer improved quality of life at 12 weeks.³ Results from this demonstration project also showed that ED-based primary palliative care reduced the percentage of geriatric ED admissions to the ICU from 2.3% to 0.9%. The overall geriatric admission rate declined by 9%, from 59% to 50%.³

While the primary goals of palliative care are to reduce burdensome symptoms and improve quality of life, it is often delivered too close to the end of life. For this reason, palliative care is often associated with end-of-life care, or even (wrongly) with hastening death. Despite this perception, there is no evidence that palliative care shortens life, and in fact, preliminary data show that it may lengthen survival in patients with metastatic cancer.^{25 26} While it is difficult to establish causality, palliative care may decrease psychosocial stress, which is associated with disease progression and shorter survival.²⁷

This protocol addresses an important gap in the emergency care of older adults at the end of life that is critically important to patients, families, providers and health systems. A substantial number of older adults continue to visit the ED during times of crisis and emergency providers set the trajectory for their care. Until recently, little emphasis has been placed on delivering goal-concordant emergency care. Palliative care in the ED improves quality of life and reduces healthcare use, and our four-pronged intervention will give emergency providers the knowledge and skills to align care plans with patient goals.

We hypothesise that it will be feasible to test PRIM-ER at two EDs with high level of fidelity and usability and implement the core components at all subsequent enrolled sites. We also hypothesise that older adult visitors with serious, life-limiting illness cared for by providers with primary palliative care skills will be less likely to be admitted to an inpatient setting, more likely to be discharged home or to a palliative care service, and will have higher home health and hospice use, fewer inpatient days and ICU admissions at 6 months and longer survival than those seen prior to implementation. By testing PRIM-ER in a diverse group of EDs in healthcare systems across the USA, we will be positioned to better understand how a system level intervention will impact healthcare utilisation and what contextual factors will impact the degree of change the occurs.

Methods and analysis

Trial design

This is a cluster-randomised QI programme that will implement and test the impact of PRIM-ER in various healthcare settings. We will first develop and pilot test the QI intervention at two sites in the pilot phase and then use a cluster-randomised, stepped wedge design to implement the education, training and technical support in our network of enrolled EDs over a 24-month period (May 2019–June 2021). Randomisation will occur at the ED level and be completed in advance by the New York University biostatistician to determine the order of the stepped-wedge implementation.

The overall approach involves ongoing asynchronous learning and technical support to bolster skills, conduct interdisciplinary case reviews and reinforce clinical pathways and protocols via provider audit and feedback. Electronic triggers for palliative care will be embedded in the electronic health record (EHR) to identify patients who may benefit from hospice or palliative care services. Using feedback and lessons learnt from the implementation of electronic triggers in the pilot phase sites, in the IE phase, we will collaborate and provide tailored technical assistance to each participating health system to ensure they adopt an EHR standard clinical workflow. Palliative care champions at each site will facilitate attendance at didactic and workshop sessions, disseminate information about local resources for outpatient palliative care, home care, and hospice, and work with the local informatics team to reinforce protocols and implement trigger criteria to identify older adults who may benefit from further needs assessment and follow-up. Physicians and nurses will receive audit and feedback reports to monitor their performance over time, and a learning monitoring system will track participation in educational activities.

In the pilot phase, members from the palliative care team, emergency nursing, social work/case management, informatics and ED operations from each of the 18 health systems will participate in workgroups to discuss how to best incorporate primary palliative care into the clinical workflow at each site. Pilot testing of PRIM-ER will also occur at two sites to optimise feasibility, fidelity and usability. Emergency providers at each pilot site will be invited to participate in usability testing of the CDS system.

In the IE phase, we will engage with eligible providers at the 33 sites based on the random sequential order in which the ED implementation occurs. Throughout the duration of the intervention, we will actively engage each health system by providing all ED staff with audit and feedback reports to monitor their performance over time. These reports will be provided routinely during the study period, as well as incorporated into ED-specific continuous QI processes. By providing this continuous and consistent feedback to ED personnel, we aim to encourage continued participation and active engagement throughout the QI interventions entirety.

In the evaluation portion of the IE phase, we will use the Centers for Medicare and Medicaid Services (CMS) claims of the beneficiaries in our patient cohort to measure outcomes, including ED disposition to an acute care setting, healthcare in the 6 months following the ED visit and survival following the index ED visit as a result of the intervention. The patient cohort will be extracted via CMS’s Research Data Assistance Center (ResDAC) using a two-step process to maximise diversity and minimise intentional or unintentional exclusions based on risk, age, health literacy, demographics or expected adherence. First, we will provide a comprehensive list of facility codes for the 35 participating sites to identify our cohort of patients 66 years and older who made a visit to any of the EDs between 2012 and 2022. Inpatient and ambulatory claims will be used to identify community-dwelling ED visitors 66 years and over who made a visit to any of the EDs from 2 years prior to study initiation until the last implementation day of PRIM-ER. ED claims will be identified via Revenue Center Code values of 0450–0459 (emergency room) or 0981 (professional fees-emergency room) according to ResDAC. We will then examine all inpatient, ambulatory, and carrier claims for the 12 months prior to each older adult’s index ED visit to calculate each beneficiary’s Gagne Index, a score developed to predict 1-year mortality in community-dwelling older adults.²⁸ The Gagne Index has been adapted from the Romano-Charlson Index and the Elixhauser system.^{29 30} It calculates a score based on the presence or absence of ICD-9s from inpatient and ambulatory claims in the prior year. Beneficiaries with a 1 year mortality of at least 30% (score >6) based on claims from the previous 12 months will be included in the analysis.

Prior to intervention implementation, all eligible full-time ED providers (including physicians, physician assistants (PAs), nurse practitioners (NPs) and nurses) will be asked to complete a baseline primary palliative care knowledge and skills survey, using a validated instrument. The survey measures self-rated knowledge about terminal and hospice care, provider’s support for hospice practice and philosophy and attitudes towards patient communication regarding prognosis and hospice.³¹

Description of the intervention

The selected components for our four-pronged intervention are patient-centred and have demonstrated efficacy in randomised controlled trials and observational research. Additionally, they are ready for comparative effectiveness research to test them as new models of care in real-world settings. Table 1 shows a description of each intervention component.

Table 1.

Description of intervention components

Intervention component	Actors	Target	Timing	Dose
Education in Palliative and End-of-Life Emergency Medicine	All full-time emergency physicians, nurse practitioners and physician assistants.	Primary palliative care knowledge and skills in needs assessment and referral.	Once during 3-week intervention period.	1-hour online didactic course.
EM Talk	All full-time emergency physicians, nurse practitioners and physician assistants.	Simulation workshop in end-of-life communication.	In-person during 3-week intervention period.	4-hour in-person simulation workshop.
End of Life Nursing Education Consortium Critical Care	All full-time emergency medicine nurses.	Primary palliative care knowledge and skills in needs assessment and referral.	Once during 3-week intervention period.	1-hour online didactic course.
CDS system	All emergency providers.	Alerts and new workflow to refer to palliative care, home care and hospice services.	Go live during intervention period.	Continued use throughout study period and beyond.
Audit and feedback	All emergency providers.	Individualised report of number of referrals to palliative care, home care and hospice services.	First report delivered in intervention period then tailored accordingly.	Weekly report during intervention period then incorporated into ED-specific continuous quality improvement process.

CDS, clinical decision support; ED, emergency department.

Emergency medicine and critical care-specific palliative care education programmes are well developed. A protocol has been developed to leverage existing programmes in an effort to standardise the core elements of PRIM-ER across all 35 sites. This includes primary palliative care education and simulation workshops on communication in serious illness. The educational components incorporate asynchronous learning and interdisciplinary case reviews.

Provider and nurse education

Education in Palliative and End-of-Life Care for Emergency Medicine (EPEC-EM) is a high-impact, conference-based education dissemination project that uses the train-the-trainer model developed in the original EPEC and has been shown to be effective when adapted for asynchronous learning.³²

To ensure intervention fidelity, in-person sessions (EM Talk) will be conducted by a national group of emergency medicine physicians with expertise in palliative care who have participated in a 6-day course in communications training around serious illness. The primary palliative care education will be taught through both online modules and didactic presentations. The detailed description of the EM Talk curricula can be found in previously cited literature.³³ Palliative care champions at each site will facilitate attendance at didactic and workshop sessions, disseminate information about local resources for outpatient palliative care, home care and hospice and work with the local informatics team to reinforce protocols and implement trigger criteria to identify older adults who may benefit from further needs assessment and follow-up. The End-of-Life Nursing Education Consortium (ELNEC) has developed modules for critical care that are also applicable to the ED setting. This standard set of modules will be delivered via an online platform to ensure intervention content fidelity to all 35 intervention sites.³⁴

CDS and audit and feedback

CDS will help reinforce clinical pathways and protocols and will enable providers to receive audit and feedback. Interruptive alerts in patients who are likely to benefit from palliative care will be embedded in the EHR to prompt providers to identify and refer patients with palliative care needs.³⁵ Physicians and nurses will receive individualised audit and feedback reports to monitor their performance over time, and a learning monitoring system will track participation in educational activities and encourage cross-fertilisation and learning among sites.³⁶

Patient and public involvement

This research was done without patient involvement and patients were not invited to comment on the study design.

Outcome measures

We have chosen the least intrusive method possible to report patient outcomes and therefore selected measures that could be tracked using administrative claims data rather than patient-reported outcomes that would require additional respondent burden. We will estimate the baseline rate of acute care admission, healthcare utilisation and survival following the index ED visit using Medicare claims data for visitors to each ED. We will use the master beneficiary summary file, inpatient, outpatient, home health, hospice and vital status files to monitor acute care admission, healthcare utilisation and survival monthly for up to 6 months after the index ED visit to evaluate whether there is a change before and after implementation. Table 2 outlines the primary and secondary dependent outcomes variables.

Table 2.

Dependent primary and secondary outcome variables

Variable	Instrument/coding	Source	Time
Primary outcome
Acute care admission	Yes/no (inpatient, non-palliative admission)	Inpatient and outpatient research identifiable files (RIF)	Index ED visit
Secondary outcomes
ED revisit	Count	Inpatient and outpatient RIF	Up to 6 months from index ED visit
Inpatient days	Count	Inpatient RIF	Up to 6 months from index ED visit
Hospice use	Yes/no	Hospice RIF	Up to 6 months from index ED visit
Home health use	Yes/no	Home Health RIF	Up to 6 months from index ED visit
Survival	Days (count)	Vital status RIF	Up to 6 months from index ED visit or death

*Primary and secondary outcomes to be measured as change in measures from baseline to 4 weeks postimplementation.

ED, emergency department.

The primary outcome is the proportion of eligible patients whose disposition is to an acute care setting (inpatient, non-palliative service). The secondary outcomes include healthcare service utilisation in the 6 months following the ED visit and survival times following the ED visit.

ED disposition will be measured on the index ED visit and will be a dichotomous variable for an acute care admission (yes/no). Acute care admission will be defined as admission to a non-palliative service, and non-acute care admission will include admission to a palliative care service or unit, discharge to home, observation (without a change to inpatient status) or transfer to inpatient or outpatient hospice. Healthcare utilisation will be measured as ED revisits (count), inpatient days (count), home health use (yes/no) and hospice use (yes/no) in the 6 months from the index ED visit. These will be identified through revenue codes in each site’s administrative data. We developed these measures of healthcare utilisation based on the Dartmouth Atlas Decedent Cohort Care Intensity Measures to monitor the quality of end-of-life care in Medicare patients with serious chronic illness.^37–39 Survival will be measured in days from the index ED visit to death or 6 months, whichever is sooner.

Measurement of what will be considered the baseline rate will continue until the month prior to implementation at each site, and postimplementation rates will be considered 1 month after implementation and continue until 6 months after the last site has undergone implementation. To reduce prevalence incidence bias, we will include a roll-in period of 12 months before we begin to include baseline rates of our outcomes in the analysis.⁴⁰ The index ED visit will be defined as the first ED visit to one of our 35 facilities during which the beneficiary has 12 months of prior inpatient, outpatient or carrier claims consistent with a Gagne Index >6 or >30% mortality. If a beneficiary’s index ED visit occurs during the roll-in period, they will be excluded from the baseline rate calculations if they return to one of our participating EDs and would otherwise meet our inclusion criteria.

To account for primary palliative care knowledge and skills on patient outcomes in the analysis, we will use survey data that assesses knowledge and attitudes of palliative and end-of-life care collected before PRIM-ER implementation from the emergency providers and nurses at all 35 participating EDs.³¹

Setting

The PRIM-ER QI intervention will be implemented in 35 diverse EDs ranging from academic medical centres to community hospitals across the US using a cluster-randomised stepped wedge study design (figure 1). All sites begin as part of the control condition and are block-randomised to receive the intervention approximately every 3 weeks over a 2-year period. All sites eventually receive the intervention. All four key core intervention components must be completed by each site during their randomised timeframe. Table 3 identifies the 35 PRIM-ER sites enrolled in the QI intervention and their respective geographical locations.

Figure 1.

Cluster-randomised stepped wedge study design. PRIM-ER, Primary Palliative Care for Emergency Medicine.

Table 3.

Enrolled sites and geographical location

Site	Location
NYU School of Medicine
Perelman Center for Emergency Care*	New York, NY
Bellevue Hospital Center	New York, NY
NYU Langone Hospital – Brooklyn	Brooklyn, NY
NYU Winthrop	Mineola, NY
Allegheny Health Network
Allegheny General Hospital	Pittsburgh, PA
Baystate Health
Baystate Medical Center	Springfield, MA
Baystate Franklin	Greenfield, MA
Beaumont Health System
Beaumont Royal Oak	Royal Oak, MI
Beaumont Troy	Troy, MI
Brigham and Women’s/Dana Farber Cancer Institute
Brigham and Women’s Hospital	Boston, MA
Brigham and Women’s Faulkner	Boston, MA
Christiana Care Health System
Christiana Hospital	Newark, DE
Henry Ford Health System
Henry Ford Hospital	Detroit, MI
Henry Ford Fairlane	Fairlane, MI
Henry Ford West Bloomfield	West Bloomfield, MI
Icahn School of Medicine at Mount Sinai
Mount Sinai Hospital	New York, NY
Mount Sinai Beth Israel	New York, NY
Mount Sinai West	New York, NY
Mayo Clinic Health System
Mayo Clinic, St. Mary’s	Rochester, MN
Mayo Clinic Austin-Albert Lea	Austin/Albert Lea, MN
Mayo Clinic Health Mankato	Mankato, MN
Ochsner Health System
Ochsner Medical Center	New Orleans, LA
The Ohio State University
Wexner Medical Center	Columbus, OH
Rutgers New Jersey Medical School
University Hospital Newark*	Newark, NJ
University of California, San Francisco
UCSF Medical Center	San Francisco, CA
Zuckerberg San Francisco General	San Francisco, CA
University of Florida Health
UF Health Kanapaha Hospital	Gainesville, FL
UF Health Shands Hospital	Gainesville, FL
UF Health Springhill Hospital	Gainesville, FL
University of Pennsylvania Health System
Hospital of the University of Pennsylvania	Philadelphia, PA
Pennsylvania Hospital	Philadelphia, PA
Penn Presbyterian Medical Center	Philadelphia, PA
University of Texas
MD Anderson	Houston, TX
University of Utah Health
University of Utah Hospital	Salt Lake City, UT
Yale New Haven Health System
Yale New Haven Hospital	New Haven, CT

*Denotes pilot site.

CA, California; CT, Connecticut; DE, Delaware; FL, Florida; LA, Louisiana; MI, Michigan; MN, Minnesota; NJ, New Jersey; NY, New York; OH, Ohio; PA, Pennsylvania; TX, Texas; UT, Utah.

The evaluation will include an analysis of CMS claims, which will be performed solely at the Ronald O. Perelman Department of Emergency Medicine at NYU Langone Health.

Eligibility criteria

Inclusion criteria

Eligible ED patients will be included in the CMS data analysis if they are 66 years or older with serious, life-limiting illness who visited any of our EDs during the implementation of PRIM-ER. Patients must demonstrate 1-year mortality of at least 30% (score >6) according to the Gagne Index, a validated instrument used to measure all cause 1-year mortality in community-dwelling older adults, calculated based on their prior 12 months before the index ED visit of CMS claims.⁴¹

All full-time ED providers including physicians, NPs, PAs and nurses will be invited to participate in the intervention and complete the baseline primary palliative care knowledge and skills survey prior to the start of the intervention.

Exclusion criteria

ED patients transferred from a nursing home on the index ED visit will be excluded from the analysis since prediction of mortality and disposition of such patients differs from community-dwelling adults. Patients currently receiving hospice at the time of the index ED visit will also be excluded since they have already received services.

Recruitment and informed consent

CMS claims of patients 66 years and older with serious, life-limiting illness who made a visit to any of our EDs during the study period will be used to measure outcomes in our patient cohort. We will seek a waiver of Health Insurance Portability and Accountability Act authorisation for ED patients as this study presents no more than minimal risk and cannot be practicably conducted without the waiver given the study’s geographic breadth and sheer number of participants (>300 000 eligible patients). Obtaining informed consent for participation and use of Medicare claims from all patients in this study is not feasible and will interfere with the conduct of this study.

Data analysis

Dependent variables

Table 2 outlines the dependent variables. ED disposition will be measured on the index ED visit and will be a dichotomous variable for an acute care admission (yes/no). Acute care admission will be defined as admission to a non-palliative service, and non-acute care admission will include admission to a palliative care service or unit, discharge to home, observation (without a change to inpatient status) or transfer to inpatient or outpatient hospice.

Healthcare utilisation will be measured as ED revisits (count), inpatient days (count), home health use (yes/no) and hospice use (yes/no) in the 6 months from the index ED visit. These will be identified through revenue codes in each site’s administrative data. We developed these measures of healthcare utilisation based on the Dartmouth Atlas Decedent Cohort Care Intensity Measures to monitor the quality of end-of-life care in Medicare patients with serious chronic illness.^37–39

Survival will be measured in days from the index ED visit to death or 6 months, whichever is sooner.

Independent variables

Table 4 outlines the independent variables. Independent variables will be assessed at the time of site implementation. Healthcare system and provider-level variables will be collected by a member of the New York University research team (programme manager) and via a provider survey at the level of each participating ED. Patient-level variables will be assessed using the CMS ResDAC Master Beneficiary Summary File, Base (A/B/D) Segment.

Table 4.

Independent outcome variables

Variable	Coding	Source
Implementation period	Weeks from time 0.	Programme manager
Implementation quality
Participation	Proportion of providers who complete all assigned training and workshops	Learning management system and programme manager
Fidelity	Proportion of sessions at each site that adhered to the course content and teaching methods.	Programme manager
Healthcare system/ED-level variables
Health system	Referred to in Table 3.	Programme manager
ED	1–35.	Programme manager
ED volume	30 000–49 999 visits, 50 000–69 999 visits, 70 000–89 999 visits and >90 000 visits.	Programme manager
Ownership	Non-profit, government and for profit.	Programme manager
Emergency medicine residency training site	Yes/no	Programme manager
Free-standing ED	Yes/no.	Programme manager
Dedicated ED social worker/care manager	Yes/no.	Programme manager
US region	Northeast, Midwest, Southeast, Southwest and West.	Programme manager
Metropolitan status+	Yes/no.	Programme manager
Outpatient palliative care	Yes/no.	Programme manager
EHR	Epic, Cerner and Pysis/Pulsecheck.	Programme manager
Trauma centre	Yes/no.	Programme manager
Provider variables
Provider age*	Average years (physicians and nurses).	Provider survey
Provider gender*	% Female (physicians and nurses).	Provider survey
Provider race/ethnicity*	% White (physicians and nurses).	Provider survey
Provider years in practice*	Average years (physicians and nurses).	Provider survey
Emergency provider advanced certification in hospice and palliative medicine*	Yes/no (physician and nurse).	Provider survey
Patient variables
Age	Years.	Master beneficiary summary file and base segment
Gender	Female, male and other.	Master beneficiary summary file and base segment
Race/ethnicity	Asian, black, Hispanic, white, North American Native, unknown and other.	Master beneficiary summary file and base segment
Gagne Index41	Count of conditions.	Inpatient and outpatient research identifiable files

*Physician and nurse provider variables will be considered separately; +population estimates by Metropolitan Statistical Area (MSA) are based on estimates of the civilian non-institutionalised population of the US as of 1 July 2013, from the 2013 National Health Interview Survey, National Center for Health Statistics, compiled according to the 2013 Office of Management and Budget definitions of core-based statistical areas. See http://www.census.gov/population/metro/ for more about metropolitan statistical area definitions.

ED, emergency department; EHR, electronic health record; ELNEC, End-of-Life Nursing Education Consortium; EPEC-EM, Education in Palliative and End-of-Life Care for Emergency Medicine.

Methods

The analytic plan accounts for the nested structure of the data and addresses issues related to missing data, study participation bias and baseline covariate balance. We address each of these in turn. All analyses will be conducted in SAS V.9.4 and R V.3.3.2 (R Foundation for Statistical Computing, Vienna).

Prior to conducting the outcome analyses, we will compare patients in each ED cluster with respect to patient, provider and facility characteristics. We will assess whether any adjustments will need to be made in the final statistical models based on whether the differences are clinically meaningful. To account for nesting in the data structure (patients nested in hospitals), we will use mixed effect multilevel models to estimate effect sizes. We anticipate two sources of variation

The primary outcome is the proportion of eligible patients whose disposition is to an acute care setting (inpatient, non-palliative service). The secondary outcomes include healthcare service utilisation in the 6 months following the ED visit and survival times following the ED visit. Exploratory outcomes may include discharge to skilled nursing facility and days spent at home if available in the claims data. The health utilisation outcomes include receipt of ED revisits (count), home health services (yes/no), inpatient days (count), admission to an ICU (yes/no) and admission to hospice (yes/no). The analysis of the effect of PRIM-ER on ED disposition in the context of a stepped-wedge design will be based on a Generalised Linear Mixed Model (GLMM). In particular, to assess the intervention effect, we will use a generalised linear binomial model with random site level effects. The model will be as follows:

$${\displaystyle \mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left[P\left(A_{ijt}=1\right)\right]={\beta }_0+b_{j0}+\left({\beta }_1+b_{j1}\right)\ast t+{\beta }_2{I}_{jt}}$$ (1)

where ${\displaystyle A_{ijt}}$ is an indicator variable, and ${\displaystyle A_{ijt}=1}$ if patient ${\displaystyle i}$ at site ${\displaystyle j}$ during month ${\displaystyle t,\left[t\in \left(0,1,\dots ,23\right)\right]}$ was admitted to an acute care setting from the ED, ${\displaystyle A_{ijt}=0}$ otherwise ${\displaystyle t=0}$ is the baseline quarter. ${\displaystyle I_{jt}}$ is an indicator variable, where ${\displaystyle I_{jt}=1}$ if site ${\displaystyle j}$ has received the primary palliative care training before month ${\displaystyle t}$, ${\displaystyle I_{jt}=0}$ otherwise. ${\displaystyle b_{j0}}$ is a random intercept for site ${\displaystyle j}$ with mean 0 and variance ${\displaystyle {\sigma }_{b0}^2}$; ${\displaystyle b_{j1}}$ is a random slope for site ${\displaystyle j}$ with mean 0 and variance ${\displaystyle {\sigma }_{b1}^2}$. We are making no assumptions about the variance–covariance structure of the random effects; the model will provide an estimate of the covariance between ${\displaystyle b_{j0}}$ and ${\displaystyle b_{j1}}$. In the primary outcome GLMM model, which is a conditional model, all within-site correlation will be accounted for using the random intercept and slopes.

Model 1 estimates the intervention effect with the within site difference between disposition rates pretraining and post-training, averaging across sites and accounting for possible secular trends that might confound with timing of the training. The primary outcome model makes no seasonality assumptions, since ED disposition decisions are made independently of time of year.

${\displaystyle {\beta }_0}$ is the (log-odds of the) average pretraining disposition rate for a site with both random effects equal to 0. ${\displaystyle {\beta }_1}$ represents the effect of the general time trend of the compliance rate without any training. ${\displaystyle b_{j1}}$ represents the intervention that is our primary outcome of interest. We will conduct tests with these null and alternative hypotheses: ${\displaystyle H_0:{\beta }_2=0}$ vs.${\displaystyle H_A:{\beta }_2\ne 0}$. The models will be fit using PROC GLIMMIX in SAS. The hypothesis will be tested using two-sided level of significance${\displaystyle \alpha =0.05}$.

The analysis of site, provider and patient-level characteristics and the quality of implementation associated with variation in impact after PRIM-ER is based on the RE-AIM Evaluation Framework for Implementation Research (table 5). We consider implementation outcomes separate from the healthcare utilisation and other patient outcomes, since implementation effectiveness is vital to treatment effectiveness.

Table 5.

Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) evaluation framework for implementation research by intervention component

Intervention component	RE-AIM implementation outcome affected
Education in Palliative and End-of-Life Emergency Medicine	Reach (#, % of eligible providers who participate).
EM Talk	Reach (#, % of eligible providers who participate).
End of Life Nursing Education Consortium Critical Care	Reach (#, % of eligible providers who participate).
CDS system	Maintenance (#, % of sites that continue to use CDS system beyond intervention period).
Audit and feedback	Adoption (#, % of eligible providers who initiate referrals to palliative care, home care, and hospice).

CDS, clinical decision support.

Models for the secondary analysis will all be analogous to the primary analysis. The secondary analysis will be based on extending models described in the analysis plan for primary outcome analysis to include independent variables related to the characteristics of interest. In particular, the models will be

$${\displaystyle \mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left[P\left(A_{ijt}=1\right)\right]={\beta }_0+b_{j0}+\left({\beta }_1+b_{j1}\right)\ast t+{\beta }_2{I}_{jt}+{\mathit{\gamma }}_1{\mathit{Z}}_{\mathit{i}}+{\mathit{\gamma }}_2{\mathit{Z}}_{\mathit{i}}{I}_{jt},}$$ (2)

where all the variables and parameters are the same as in model 1, except for ${\displaystyle {\mathit{Z}}_{\mathit{i}}}$, ${\displaystyle {\mathit{Y}}_{\mathit{i}}}$ and ${\displaystyle {\mathit{\gamma }}_2}$. ${\displaystyle {\mathit{Z}}_{\mathit{i}}}$ is a vector that represents all of the site, provider and patient characteristics that we will be exploring. ${\displaystyle {\mathit{\gamma }}_1}$ is a vector of parameters that represent the effects of these characteristics on the outcome prior to the intervention. ${\displaystyle {\mathit{\gamma }}_2}$ is a vector of parameters that represent the effect modification related to each of these characteristics. We will explore models that include site-specific treatment effects rather than site-specific time effects to see which best describes the observed patterns. These secondary models will provide insight into ways that the effect of the training might vary across institutions and individuals based on varying characteristics.

Ethics and dissemination

Ethical and safety considerations

The trial is registered at ClinicalTrials.gov. To minimise research-associated risk and protect the confidentiality of participant data, all investigators and staff involved in this project have completed extensive courses and passed certifying exams on the protection of human subjects in research through Collaborative Institutional Training Initiative training and Health Insurance Portability and Accountability Act certification.

Dissemination plan

In addition to the traditional form of dissemination via presentations at academic national meetings and peer-review publications, we will actively engage other patient advocacy, specialty group and healthcare system partners in disseminating the results. This will include our healthcare system partners that are collaborating on the project, patient and family stakeholder groups and specialty societies. We have purposefully designed our primary palliative care intervention to facilitate sustainability and scalability in real-world settings and will prioritise these activities in the last 6 months of the project.