Implementation of Complex Interventions

Background:

Despite the well-documented risks to patient safety associated with transitions from one care setting to another, health care organizations struggle to identify which interventions to implement. Multiple strategies are often needed, and studying the effectiveness of these complex interventions is challenging.

Objective:

The objective of this study was to present lessons learned in implementing and evaluating complex transitional care interventions in routine clinical care.

Research Design:

Nine transitional care study teams share important common lessons in designing complex interventions with stakeholder engagement, implementation, and evaluation under pragmatic conditions (ie, using only existing resources), and disseminating findings in outlets that reach policy makers and the people who could ultimately benefit from the research.

Results:

Lessons learned serve as a guide for future studies in 3 areas: (1) Delineating the function (intended purpose) versus form (prespecified modes of delivery of the intervention); (2) Evaluating both the processes supporting implementation and the impact of adaptations; and (3) Engaging stakeholders in the design and delivery of the intervention and dissemination of study results.

Conclusion:

These lessons can help guide future pragmatic studies of care transitions.

Patient safety is frequently at risk in transitions of care from hospital to outpatient settings, and current processes of care during these transitions are suboptimal.1–8 Despite growing evidence in support of interventions to improve transitional care,9,10 translation of effective strategies into routine care has not been realized for patients at high risk for fragmented care and poor outcomes.11–13 Care processes that aim to bridge acute and postacute care settings vary widely and are fraught with challenges, including different information technology systems, siloed networks, and misaligned reimbursement schemes. Health care professionals responsible for coordinating care transitions recognize the value of closing gaps in care but often are unaware of best practices or how to translate evidence-based strategies to local situations.

Frequent changes in institutional and health policies, patients’ needs, and local resources create a challenging environment for addressing care transitions and studying interventions in ways that are both valid and generalizable. This paper presents lessons learned from the portfolio of care transition studies funded by the Patient-Centered Outcomes Research Institute (PCORI). Three major themes are examined through the review of case studies: (1) Delineating function (intended purpose) versus form (prespecified modes of delivery of the intervention); (2) Evaluating both the processes supporting implementation and the impact of adaptations; and (3) Engaging stakeholders in the design and delivery of the intervention and dissemination of study results. We highlight the complexities of conducting such studies, offer lessons learned from our experiences, and provide exemplars to inform future research.

METHODS

Investigators of 9 comparative effectiveness studies of care transitions funded by PCORI describe their experiences implementing and evaluating transitional care interventions. Two studies were observational. Seven studies were randomized controlled trials of transitional care interventions, mostly of pragmatic design. Unlike a traditional explanatory trial, a pragmatic trial is designed to test an intervention in everyday clinical settings with heterogenous patient populations to maximize applicability and generalizability. Thus, a pragmatic trial asks whether an intervention works in real life versus a tightly controlled trial environment (ie, effectiveness versus efficacy).14 The 2 observational studies examined real-world practice. The 9 transitional care studies varied in their start dates, durations, target populations, clinical settings, and budgets (range: $1M−$16M), but all were studies of complex clinical interventions. The interventions were not funded by PCORI. Table 1 provides an overview of the case studies reviewed.

TABLE 1.

Overview of 9 Transitional Care Studies in Patient-Centered Outcomes Research Institute’s Transitional Care Portfolio15–20

Study	Institution	Study Design (Unit of Randomization)	Sample Size	Core Function (Purpose) (Example)	Form (Activities) (Example)	Adaptation (Example)	Results for Primary Research Question	Outcomes for the System	Outcomes for Patients
Early Supported Discharge for Improving Functional Outcomes after Stroke (COMPASS) 15	Wake Forest University Health Sciences	Pragmatic cluster randomized RCT (hospital)	6024 adult stroke patients 40 hospitals in North Carolina	Comprehensive assessment (neurological, medical, cognitive, social determinants of health) of patient and caregiver needs, which informs a tailored care plan that is handed to the patient	Timing of assessment and care planning ideally within 14 d of discharge for billing purposes, but can be completed with 30 d to be considered delivered	In addition to care plan, several hospitals provided patients with a blood pressure cuff as most care plans indicated patient needed to control blood pressure. Adaptation not assessed	The primary outcome was measured for 59% of patients and was not significantly influenced by the intervention. Mean (±SD) Stroke Impact Scale-16 was 80.6±21.1 in TC vs. 79.9±21.4 in usual care	90 d all-cause readmission; 30 d and readmission, 1 y mortality; 30, 90 d, and 1 y recurrent stroke readmission	Physical function mortality, disability, medication adherence, depression, cognition, self-rated health, fatigue, care satisfaction, home blood pressure monitoring, and falls
Comparative Effectiveness of Rehabilitation Services for Survivors of an Acute Ischemic Stroke16	Duke University	Observational (NA)	147,716 adults treated for acute ischemic stroke at 1192 US hospitals	Transition from hospital to continued inpatient care of different rehabilitation intensities (rehabilitation or skilled nursing facility) or to home with different levels of service provision (home health, outpatient therapy, or no further services except clinic-based medical care)	Continued inpatient care required to have a 3 d stay but type and dose of therapy in any of the comparison groups could vary	No adaptations assessed	Stroke patients experience 3+ transitions from acute hospital care and 3 and 12 mo outcomes were most favorable for patients who received inpatient rehabilitation care	Days at home and health care utilization (days living at home and no longer receiving stroke-related services; timeliness of community-based care; 3 and 12 mo rehospitalizations; nursing home institutionalization)	Disability, quality of life, mortality
Improving Transition from Acute to Post-Acute Care following Traumatic Brain Injury—the BRITE Study	University of Washington	Pragmatic RCT (patient)	Enrollment goal: 900 individuals with moderate to severe traumatic brain injury (TBI) discharged from inpatient rehabilitation and 540 caregivers across 6 sites across the United States (WA, IN, OH, NY, PA, TX)	Patient and caregiver participants have contact with a TBI Care Manager across the first 6 mo postdischarge to identify and address needs, provide care coordination, education and support on TBI, and transition to appropriate support at 6 mo	TBI Care Manager contacts the patient and caregiver within 7 d postdischarge, weekly for 4 wk, then bimonthly, then monthly for up to 12 contacts	Contacts number and timing is flexible depending on patient and caregiver needs	Study is still in-progress, results not yet available	Completing discharge recommendations and health care utilization across first year postdischarge	Patient: participation, health related quality of life, planned outpatient visits, urgent care visits/unplanned hospitalizations. Caregiver: burden, health-related quality of life, satisfaction with roles, time spent in caregiving
Emergency Medicine Palliative Care Access (EMPallA)	NYU Grossman School of Medicine	Pragmatic RCT (patient)	1350 emergency department (ED) patients 50+ years with serious illness at 16 health systems across the United States	Patient receives either nurse-led telephonic case management to facilitate sustainability and scalability in real-world settings or outpatient specialty palliative care; These interventions may improve patient-centered outcomes including quality of life, loneliness, and social isolation, and may reduce strain in informal caregivers. These interventions may also decrease future health care utilization (ie, ED visits and hospital admissions) and increase hospice use in patient participants; This study will help align discussions for patients and providers regarding goals of care, end-of-life wishes, advance directives, pain and symptom management, and community resources	Enrollment ideally will occur in-person for patients who are scheduled for ED discharge or observation status, but can occur within 48 h of discharge	Due to the coronavirus disease of 2019 pandemic, research coordinators were unable to recruit in the ED. Study protocol was adapted to allow for telephonic recruitment. Preliminary adaptation data demonstrates that despite recruitment method (in-person or telephonic) intervention engagement is the same	Study is still in-progress, results not yet available	A product of the dissemination and implementation plan will include a business plan that will accurately lay out the 1, 3, and 5 y return on investment for implementing a nurse-led telephonic case management program for seriously ill patients within an integrated health system and/or health plan. The business plan will include a toolkit that we will make openly accessible to payers and health care systems interested in adapting it for their local health care market	Compare nurse-led telephonic case management to facilitated, outpatient specialty palliative care on: quality of life; health care use in the 6 mo following enrollment; loneliness; and caregiver strain
PATient Navigator to rEduce Readmissions (PArTNER)17	University of Illinois at Chicago	Pragmatic RCT (patient)	1029 adults hospitalized with heart failure, pneumonia, chronic obstructive pulmonary disease, myocardial infarction sickle cell disease crisis	Standardized comprehensive assessment of self-management skills, socioeconomic resources, and access to postacute care. Information used to develop personalized plan (navigator intervention) to support safe hospital-to-home transitions	In-hospital and home visits by community health worker. Peer-to-peer coaching delivered by community-based patient advocacy organizations via phone	Patient preference and feasibility used to define extent and timing of in-hospital and home-based interventions, and number of peer coaching sessions	Results pending	30 and 60 d hospital readmissions	30 and 60 d PROMIS measures of emotional, social, and physical health
The Sickle Cell Trevor Thompson Transition Project (ST3P-UP) Study	Atrium Health	Pragmatic RCT (practice)	537 emerging adults (16−25 y old) with sickle cell disease	Publication in development	Publication in development	No adaptations assessed	Study is in progress; results not yet available	Standardized care for individuals with sickle cell disease across pediatric and adult practices using the 6 core elements of transition	Patient reported outcomes (social support, health related quality of life, patient satisfaction, quality of care received; transition readiness
Relative Patient Benefits of a Hospital-PCMH Collaboration within an ACO to Improve Care Transitions18	Brigham and Women’s Hospital; Massachusetts General Hospital; Partners Health care	Pragmatic RCT (patient)	In press	Inpatient discharge advocate: nurse who communicates with outpatient team to ensure a safe discharge plan and prepares patients for discharge	Hospital 1: 1 NP played role, high degree of quality control, but not enough availability to see all patients; Hospital 2: Attending nurses played role, but restricted to communication with responsible outpatient clinician, less quality control	No adaptations assessed	In press	In press	In press
Project ACHIEVE (Achieving Patient-Centered Care and Optimized Health in Care Transitions by Evaluating the Value of Evidence) 19	University of Kentucky	Observational (NA)	7939 patients at 42 hospitals across nation	22 transitional care strategies	22 transitional care strategies	No adaptations assessed	One group of transitional care strategies—Hospital-based trust, plain language, and coordination—is significantly associated with reduced utilizations and positive patient-reported outcomes	Health care utilization (30 d readmissions, 7 d ED visits)	Patient-reported outcomes (physical health, mental health, daily activities) patient experience
An Emergency Department-to-Home Intervention to Improve Quality of Life and Reduce Hospital Use20	University of Florida	RCT (patient)	1101 from 2 hospitals; Intervention group (IG): 557; Usual care (UC): 544	Core functions—IG: patient received DC instructions in ED; 3 follow-up phone calls and 1 home visit by trained health coach within 30 d. UC: patient received DC instructions in ED	Follow-up visits and phone calls from health coach following Coleman TCI protocol+transportation and home delivered meals as needed (IG) vs. usual postdischarge follow-up (UC)	No adaptations	NS differences return ED visits, hospital admissions, outpatient visits; IG had reduced hospital admission in those with return ED visits (OR=0.64)	Return ED visits, hospital admissions, outpatient visits	Return ED visits hospital admission outpatient visits informational support anxiety physical function

ACO indicates accountable care organization; NA, not applicable; NP, nurse practitioner; NS, not significant; NYU, New York University; OR, odds ratio; PCMH, patient-centered medical home; PROMIS, Patient-Reported Outcomes Measurement Information System; RCT, randomized controlled trial; TCI, transitional care intervention.

RESULTS

Three major themes emerged when study investigators compared and contrasted their experiences.

1. Delineating the function versus form of the intervention and allowing for adaptation to the form while preserving fidelity to function. The complexity of care transitions interventions can make it difficult to compare findings or translate models from one context to another. PCORI’s Methodology Committee uses 2 descriptive terms that help with this dilemma.21,22 Function refers to the core purposes of an intervention applied to meet an intended goal. Form refers to the strategies and activities needed to carry out the functions of the intervention, including the menu of intended modes of delivery, providers involved, materials or tools required, and intervention dose.22 Too much, reactive, or the wrong kind of adaptation can weaken an intervention’s effectiveness and make it difficult to understand what is being evaluated. Thus, it is critical to differentiate functions from forms, allow adaptations to the form while keeping core functions intact.

2. Evaluating both the processes supporting implementation and the impact of adaptations. The case studies describe planned and unplanned adaptations to transitional care interventions. They highlight the utility of analyzing process data to better understand adaptations as well as the difficulty in assessing the actual (intended and unintended) effects of adaptations. Findings highlighted the need to recognize and systematically measure adaptations in implementation; guided by the research design, existing data, and established frameworks; and systematically evaluated with quantitative and qualitative data collected during implementation.

3. PCORI’s approach to engaging stakeholders in the design and delivery of the intervention and dissemination of study results appears to work. The philosophy of stakeholder engagement is now established in the scientific literature; however, investigators and funders often agree that the value of engagement it is not always realized due to the effort and resources required to implement and sustain it properly. Engagement can also cause problems in the community if it is poorly conceived, poorly conducted, or implemented without long-term commitment. The expectations PCORI placed on study teams to detail engagement plans as part of their contracts and to engage meaningfully with stakeholders, coupled with the flexibility afforded to teams to match engagement to their particular study needs, likely mitigated issues related to the possible negative implications of engagement.

These themes are further discussed below.

DISCUSSION

Delineating the Function Versus Form of the Intervention

Hawe and colleagues called for designers of pragmatic complex interventions to standardize functions, while allowing the form of the intervention to be tailored to the participant or local conditions.21,23,24 Similarly, the PRagmatic Explanatory Continuum Indicator Summary-225 framework recommends designing trials with the desired level of pragmatism in mind, and allowing flexible options for delivery, while maintaining appropriate fidelity to the intervention.

An intervention’s effectiveness is driven by its core functions. Thus, functions should not be adapted, and fidelity to core functions should be assessed. Functions can take on multiple forms, and these forms may be adapted as long as they fulfill the core function. The substitution of a particular form does not compromise the integrity of the intervention if a different form could be used to achieve the core function.24

The concepts of forms and functions were developed by Hawe and colleagues and were recently elaborated on by Perez Jolles and colleagues, and Kirk and colleagues who recommend the use of these terms over previously used terms. Often used interchangeably are the terms: core components, core elements, core principles, active ingredients, key features, the theory of change process or logic model.21,24,26–29

Despite the shared understanding of their importance, intervention developers infrequently specify core functions and forms. Most protocols only specify forms (who does what, when, where). For example, a protocol might state “Nurse A asks patients who meet X, Y, Z inclusion criteria a specific screener question to determine if they are eligible for the transitional care program. If eligible based on the screener, Nurse A places note for physician in chart to initiate referral.” The methodological concern is that most intervention protocols do not specify core functions. Thus, measuring fidelity to the functions is generally not possible using only the protocol. Core functions need to be specified first, describing why and how—why does what they are doing work and how does it work (mechanisms of action).

Historically, the focus has been on fidelity to the intervention protocol, which means we have only measured fidelity to forms. In the example above, a study might be criticized for not having fidelity if Nurse A changed the wording of the question or if someone other than Nurse A asked the question. Instead, such adaptations need to be measured and evaluated in terms of their potential impact on core functions. To illustrate further, a nurse might spend 5 minutes teaching a patient about a medication at discharge, or the patient might watch a 5-minute video at home. However, these may lead to different levels of effectiveness despite serving the same purpose of patient education (function). Also, patient preferences for video or nurse-directed education may produce different levels of knowledge. Therefore, it is important to evaluate whether adaptations to form are associated with different implementation outcomes (eg, acceptability) and produce different patient outcomes.

Historically, adaptations were considered a potential threat to implementation and the validity of studies that evaluated them. Yet adaptations may be necessary to address the needs of a particular patient population due to heterogeneity of disease severity or complications, other patient factors, or the local social context or environment—or because funding is insufficient to implement an intervention as originally designed. The goal should be to balance fidelity (to function) and context-aligned adaptation (to form) to maximize effectiveness, translation into routine clinical care, and sustainability.

New work by Hawe, Perez Jolles and colleagues, and Kirk and colleagues suggests we should rethink how we consider fidelity entirely. Instead of focusing on protocol fidelity (which, given the way protocols are currently written, means we are measuring form fidelity only), we should instead focus on function fidelity. The advantages of disentangling functions from forms can be applied to interventions that include “just” clinical components as well as to interventions that are packages of clinical components and implementation strategies.

Ideally, developers would specify how each intervention function is expected to generate an effect on the prespecified outcome in each patient population (or in specified patient subgroups). Core functions and forms can also be identified post hoc. 28 The idea of making causal pathways explicit is also reflected in PCORI’s methodology standards for studies of complex interventions.22 In the case studies below, we discuss the importance of delineating function versus form.

Case Study: “Improving Transition from Acute to Post-Acute Care Following Traumatic Brain Injury—The BRITE Study”30 is an ongoing multisite trial whose participants have highly variable ability to participate depending on the severity of their traumatic brain injury and their postdischarge community resources. The study compares the effectiveness of 2 approaches of transition from the hospital to the community. Flexibility in how the intervention is implemented depends on the individual’s needs and the community resources available to meet those needs.21,24 The research team intended to focus their assessment of fidelity on monitoring function by specifically assessing which forms were used in the intervention (assessment of all needs, provision of specific resources, etc.), through listening to recorded contacts between interventionists and participants and utilizing a fidelity checklist. Ultimately, this approach to fidelity was not pragmatic on the PRagmatic Explanatory Continuum Indicator Summary-2 scale,31 and it was more closely aligned with the idea of fidelity to form (ie, documentation of particular kinds of patient contact) rather than function. As the study evolved, the research team clarified its definition of function and emphasized fidelity to function while allowing flexibility in form. The process was further documented in a manual to embed flexibility in the intervention, and fidelity to function (eg, were needs assessed, were resources given, were contacts with participants made) is now tracked in a clinical database. Lesson Learned: Complex transitional care interventions should be flexible in form to maximize their effect on different patient subgroups and contexts.

Case Study: The “Emergency Medicine Palliative Care Access” study is an ongoing pragmatic, multicenter randomized controlled trial with emergency department patients comparing 2 established models of palliative care: nurse-led telephonic case management versus specialty outpatient palliative care. The interventions were both complex,32 with multiple core functions completed by a nurse or another provider. The form of delivery (telephonic versus face-to-face) was not originally designed to be adaptable, but the coronavirus disease of 2019 pandemic closed outpatient palliative care clinics temporarily, and all visits transitioned to a telehealth platform. However, the core functions of palliative care, such as assessing symptom burden and performing a goal of care conversation, remained. The research team will compare effectiveness of the mode of delivery as an additional exploratory analysis. Lesson Learned: Flexibility regarding an intervention’s form allows necessary adaptation in response to external, real-world changes. Complex interventions may require adaptation under real-world conditions and will likely be adapted by future adopters.

Case Study: In the PATient Navigator to rEduce Readmissions (PArTNER, Evaluating a Program to Improve Patient Experiences after Discharge from the Hospital) pragmatic trial, patients hospitalized at a minority-serving institution with a diagnosis of heart failure, pneumonia, chronic obstructive pulmonary disease, myocardial infarction, or sickle cell disease were randomly allocated to a Navigator intervention delivered by community health workers and peer coaches or to usual care. The Navigator intervention included (1) Assessment of barriers to safely transition from hospital to home during community health worker-led in-person visits in the hospital before discharge and at a home visit; and (2) Follow-up supports provided via phone by peer coaches from patient advocacy organizations over a 60-day period. In the Navigator group, completion rates of interventions varied across components (eg, >90% had assessments of barriers to care, ~80% had postdischarge home visits, but only 60% had 1 or more peer coaching calls).33 Lesson Learned: Fidelity to form can vary greatly across various components of a complex intervention. The need for adaptations to form may therefore vary among intervention components and should be considered in the design phase of the study, evaluated, and specified for future adopters.

Evaluating Both the Processes Supporting Implementation and the Impact of Adaptations

Process evaluations inform researchers about expected effects and associated mechanisms of an intervention and can help explain differences between expected and observed outcomes.34,35 PCORI methodology standards for studies of complex interventions also emphasize the value of planning and conducting a full process evaluation. This information can also help elucidate how an intervention’s effectiveness may be altered by adaptations to different settings, populations, changing contextual factors, or the external environment. Some events may be anticipated and adjustments made during trial implementation can be planned; others cannot be anticipated.

The case studies below describe unplanned adaptations in 2 pragmatic trials. They highlight the utility of analyzing process data but did not evaluate the impact of these adaptations on implementation or outcomes.

Case Study: “Relative Patient Benefits of a Hospital-Patient Centered Medical Home Collaboration within an Affordable Care Organization to Improve Care Transitions,” was a cluster-randomized stepped wedge trial of a multicomponent intervention to improve care transitions among general medical and surgical adult patients. One component of the intervention, an inpatient discharge advocate, was delivered in different ways in the 2 participating hospitals, an unplanned adaptation in response to local contextual differences. In 1 hospital, a dedicated nurse practitioner served in this position, while in the other, this position was filled by expanding the roles of multiple existing “attending nurses.” While the former had a high degree of quality control (fidelity to function) but limited ability to see all patients, the latter had more variable fidelity to function but greater ability to see more patients and a clearer path to sustainability. Lesson Learned: Mixed methods program evaluation can provide valuable lessons learned when circumstances lead to unplanned adaptations to the form of an intervention.

Case Study: The “Comprehensive Post-Acute Stroke Services” study tested a comprehensive model of postacute stroke care in a cluster-randomized design including 40 hospitals and over 6000 participants.15,36 The protocol specified that a 2-person clinical team (a nurse and either a physician or a stroke-trained advanced practice provider) conduct a follow-up visit after hospital discharge and comprehensively assesses patient and caregiver needs. These data then informed an individualized care plan including secondary prevention, rehabilitation, recovery, and referral to community services. However, only 35% of eligible patients at intervention hospitals actually received the follow-up visit and individualized care plan within 30 days of discharge. As is common in pragmatic trials, fidelity to function was low. In the intention-to-treat analysis, no benefit on the primary outcome (patient-reported physical function) was detected. Self-reported blood pressure monitoring at 90 days was more prevalent in the intervention arm compared with usual care (72% vs. 64%) and was the only significant secondary outcome in the intention-to-treat analysis. The intervention produced a clinically meaningful improvement in home blood pressure monitoring which is critical for a population at high risk of future stoke. Clinical teams were trained to emphasize to patients at discharge the importance of home blood pressure monitoring. During implementation, hospital staff deemed it essential to provide their patients with blood pressure cuffs if telling them to monitor their blood pressure at home. This sentiment was common, and 12 (63%) intervention hospitals used their own resources to provide blood pressure cuffs to patients at discharge. This unplanned adaptation was not assessed. Lesson Learned: Evaluating processes that support implementation can elucidate the hypothesized causal pathway and can inform future attempts to adapt the model to other settings and populations. Evaluating adaptations generates evidence to support what should and should not be incorporated into the manualized intervention.

Kirk and colleagues27 developed a model and decision aid for understanding intended and unintended impacts of adaptations. Rather than just describing adaptations, they recommend a systematic assessment of relationships between constructs. Their model for adaptation design and impact recommends researchers consider potential causal pathways of adaptations (eg, mediators and moderators) and adaptations’ intended and unintended impacts on outcomes. Model for adaptation design and impact recommends practitioners to design adaptations in a way that anticipates intended and unintended impacts and leverages best practice from research.

Deciding which adaptations are permissible for an intervention should be guided by an established framework. Escoffrey et al37,38 published a review of frameworks for adapting evidence-based interventions. The Substance Abuse and Mental Health Services Administration of the US Department of Health and Human Services also has a quick guide for how to adapt interventions, and Wiltsey Stirman et al33 proposed a framework for reporting adaptations. If evidence is available, that too should guide adaptations. Naylor et al39 has published a review of known adaptations of the evidence-based Transitional Care Model in the United States.

Currently, there are very few rigorously conducted studies evaluating implementation factors that are associated with higher fidelity to function. For future research, especially pragmatic trials, we recommend an approach to implementation in which adaptation is encouraged, recognized, respected; guided explicitly by the research design, existing data, and established frameworks; and systematically evaluated with quantitative and qualitative data collected during implementation. We recommend specifying which adaptations to the intervention’s form are allowable and monitoring for potential intended and unintended impacts of these adaptations. We also recommend developers provide manuals describing the core functions and the menu of forms available for each function and, ideally, guidance on which forms to select for which situations; with decisions supported by data.

Engagement of Stakeholders: First Do No Harm

While the benefits of stakeholder engagement are often discussed,40,41 few studies describe the challenges that accompany engagement,42 particularly as they relate to the communities being served. “Engagement” is a term often used by researchers to describe interactions with the community. However, the quality of those interactions often hover on the lower ends of the community engagement continuum (eg, outreach and consultation), lack true leadership and bidirectional learning from communities (which occur at higher levels such as collaboration/shared leadership), and are rarely robust.

In addition, there are few patient-centered, ecologically focused definitions of “community.”43 Therefore, researchers and community members often enter into partnerships not understanding the operationalized definition of community which lends the work of the partnership to a unilateral focus and further exclusion within the community.44

The benefits of engagement begin early. Early initiation of high-quality, clearly defined community/patient engagement can mitigate mistrust, especially among marginalized communities, enhance participant enrollment and implementation of findings, and eliminate some of the difficulties with recruitment.45 Therefore, we must engage the target patient population and health system stakeholders providers early in the intervention and study design phases to understand their needs and determine the implications of not addressing those needs within the study. Flexibility and the ability to adjust strategies are essential to engage community partners most effectively.

While investigators and funders often recognize the value of engagement, it is not always well-implemented or sustained due to the effort and resources required. Engagement that is poorly designed, poorly implemented, or not sustained can increase rather than decrease mistrust in the community. The standards and milestones that PCORI requires of funded projects—from study design and recruitment to the dissemination of results—help to ensure that study teams engage meaningfully with stakeholders throughout the life of the study. Furthermore, PCORI recognizes the importance of allowing for flexibility of form so long as fidelity of function is preserved, enabling teams to match engagement to their particular study and stakeholder needs and to modify engagement form, as needed, to best cultivate trust in the stakeholder community.

Engagement is essential in transitions of care given the complexity of different levels of care and heterogeneity of the populations of interest who are in transition. Particularly challenging, care transitions research involves the study of processes that encompass different providers, settings, and systems with the patient as the only consistent factor. Furthermore, it frequently involves an intervention that is itself complex and involves multiple layers (functions). In addition, while the professionals asked to execute the intervention may recognize its importance, they may lack awareness or agency to ensure the delivery of what is needed in the next step in the care chain. Factors such as these may explain the low fidelity to function observed in some studies.

The case studies presented here represent a broad range of community stakeholders. All provide examples of partnerships that exemplify characteristics of higher levels of the community engagement continuum. In the case study below, the researcher-stakeholder partnerships developed a common definition of “community,” and each demonstrated ways to account for the complexities of community in their implementation and evaluation processes. This case study illustrates the importance of understanding stakeholders’ capacity for engagement and how best to achieve meaningful, bidirectional engagement.

Case Study: “Sickle Cell Trevor Thompson Transition Project study: The Comparative Effectiveness of Peer Mentoring vs. Structured Education-Based Transition Programming for the Management of Care Transitions in Emerging Adults with Sickle Cell Disease” includes 14 sites in 7 states. The study examines the effectiveness of peer support in reducing acute care visits and improving quality of life among youth with sickle cell disease as they transition to adult care. Each site is represented by a “triad”: a pediatric and adult sickle cell disease provider, a community-based organization, and the patient. Early on, the team noted that community relationships and capacity for community engagement varied significantly among sites. The study team worked iteratively with patient stakeholders and community partners to operationally define engagement and a purpose statement. These tools helped sites explore the concept of engagement within their own unique contexts. Creating a unique engagement goal also fostered a sense of ownership within each site and provided a foundation for sustainability of efforts beyond the study. Engagement pitfalls abound when researchers’ values (eg, publications) misalign with community values (eg, quality of life). Therefore, co-developing the engagement goals fostered greater involvement in the process as well as the outcomes of the research. Lesson Learned: Developing a common definition of community allows partners to collectively recognize the heterogeneity of the community and to eliminate division that may inadvertently occur within the community due to the mistrust of the partnership among members of the community not directly involved in the partnership. Effective engagement begins with approaching all introductions without hierarchy or positions of power assists collaborators in dissolving pre-established power structures that may exist. Allowing sites to customize engagement approaches within the context and capacity of each site can enhance communication among stakeholders and support both implementation and sustainability.

Often, the outcomes that matter most to patients, such as engagement in the discharge process, ability to carry out the discharge plan, physical and cognitive function, and caregiver burden, are harder to measure, lack validated instruments, and/or are deemed less rigorous by the larger scientific community.46 Obtaining stakeholder input allows researchers to assess benefits and harms of interventions to enhance informed decision-making, account for heterogeneity in participants and settings, and clearly articulate all possible outcomes to stakeholders.47 Patient-centered outcomes can often be proxy measures for important downstream variables including health maintenance and reduction of high-cost service use (eg, emergency department visits). In the case study below, we discuss how the target population can influence expectations for intervention success and therefore the outcomes most important to them after discharge.

Case Study: In “Relative Patient Benefits of a Hospital-Patient Centered Medical Home Collaboration within an Affordable Care Organization to Improve Care Transitions,” the team chose postdischarge adverse events as the primary outcome. Their reasons were: (1) these were felt to be important to the patients and caregivers in the study’s Patient and Family Advisory Council and to other stakeholders; and (2) unlike readmissions, postdischarge adverse events are more directly related to quality of transitional care via a clear causal pathway. Lesson Learned: Postdischarge adverse events could only be measured via intensive postdischarge follow-up with patients, followed by physician adjudication. Yet the outcome was important to patients and the team found this outcome much more amenable to change than readmission rates.18

Broadening Dissemination Venues

Investigators have an ethical obligation to disseminate research findings. The first path for dissemination is often peer-reviewed abstracts for conferences and followed by journal publication. Yet, this strategy has well-known disadvantages: journal reviewers may be biased about the value of efficacy research over effectiveness studies, journals may not publish studies that have not demonstrated benefit despite the rigor of the design or significance of the findings as a whole, the process of reviews and revisions is lengthy, and paywalls (ie, for journal access) can prevent findings from being widely accessible.48

We found that pragmatic trials of transitions of care interventions were difficult to publish in top-tier scientific journals, with the most common issue being that reviewers appeared to use traditional randomized controlled trial design criteria to evaluate them. This may change as more studies specify methodology standards used, are recognized for their methodologic rigor, and as the scientific community becomes accustomed to pragmatic trials. We recommend further educating reviewers to recognize the rigor and value of pragmatic, patient-centered studies of complex interventions (see Reeves36 as an example).

Stakeholders in each PCORI-funded study can help identify audiences who would benefit from being informed of the findings. Those who help develop each dissemination plan can also facilitate entrée into their own networks as vehicles to share findings with audiences who would directly benefit from and use that information.

Patients and Caregivers as Dissemination Partners

Dissemination of study results should begin with choosing stakeholders to codevelop and support plans for messaging, channels for dissemination, reach, and intent. There is now a trend to include stakeholders as coauthors on papers and presentations, as in community-based participatory research. However, educating stakeholders to take on this role typically requires intensive coaching and time. We recommend budgeting for stakeholders to help disseminate findings; they bring the research to life for decision makers. In the case studies below, we illustrate how nontraditional research partners can enhance dissemination of study results

Case Study: In “An Emergency Department-to-Home Intervention to Improve Quality of Life and Reduce Hospital Use,” involvement of a broad stakeholder group was critical to shaping every step of the research study, including dissemination. Patient and caregiver stakeholders created videos, presented at an annual PCORI meeting and on the PCORI website, outlining their contributions, why patient participation is critical to research, and what the contribution meant to them. Lesson Learned: Patients, clinicians, and other stakeholders engaged in research can provide powerful voices for change and strengthen research dissemination.

Case Study: The development and publication of interim findings emerging from patient and caregiver focus groups of Project ACHIEVE offered an opportunity to pilot test principles and steps in their dissemination plan. This included leveraging Project ACHIEVE’s extensive network of research partners and stakeholders and customizing messages for targeted end users—press releases, drafted tweets and 1-page summaries—for targeted end users (eg, patients and family caregivers, clinical providers and policy makers). Importantly, health literacy experts and patient/family caregiver representatives reviewed the patient-focused materials to ensure they were understandable and conveyed meaningful, actionable messaging. In addition, major hospital and advocacy organizations including the American Hospital Association, America’s Essential Hospitals and Joint Commission Resources reviewed materials with incorporation of their suggested modifications for clinical providers, hospitals and policy makers. These efforts resulted in the peer-reviewed publication obtaining an Altmetric score of 226, placing it among the top 5% of research articles rated by the platform. Over 40 national news outlets covered the findings. One-page summaries remain readily available for download on the Project ACHIEVE website (https://achieve.med.uky.edu/achieve-news). Lessons Learned: Stakeholders are valuable active partners for dissemination of study results to users and policy makers; the research team needs to ensure that the message is compelling, direct, and succinct for end users. Importantly, dissemination as an interactive, iterative process, not a one-time activity.

Timing of Dissemination

When policy reforms are proposed, transitional care studies are not always completed or teams are not always able to share findings. Dissemination can be challenging to align with policy reform and professional societies as demonstrated in this next case study.

Case Study: The “Comparative Effectiveness Study of Post-Acute Care Options for Survivors of Stroke” was a retrospective study using clinical registry, survey, and claims data. Preliminary findings comparing outcomes for stroke patients who received postacute care in inpatient rehabilitation versus skilled nursing facilities were disseminated to the Medicare Payment Advisory Commission and key stakeholders because changes to payment were being considered that would impact delivery of inpatient rehabilitation care. The study’s results illustrated how clinical data and patient-reported outcomes enhanced information in Medicare claims and improved the ability to assess effectiveness of post-acute care versus claims data alone.49,50 The Commission concluded that site-neutral payments for inpatient rehabilitation and skilled nursing facilities should exclude stroke patients, because clinical and patient-reported outcomes showed the benefits of inpatient rehabilitation. Although disseminating directly to decision makers to influence policy reduced significance for peer-reviewed publications, findings were incorporated in the American Heart Association clinical guidelines that continue to recommend stroke patients go to an inpatient rehabilitation (versus skilled nursing) facility after hospital discharge.51 Lesson Learned: Engaging stakeholders in dissemination before publication can lead to important advancements measured outside of peer-reviewed publications. This engagement approach also stands in contrast to press releases and publishing on websites like www.kff.org or https://evidencebasedprograms.org/ that enable policy officials and others to quickly evaluate the evidence underlying health and social programs.

There is scant literature on how different stakeholders can be directly engaged with one another to drive translation of research into practice. In particular, while there evidence on ways that researchers can deliver study results to policy makers, little research has been conducted on how to engage patients and caregivers in this process. Heckert et al52 acknowledge the inherent tension between the time it takes to meaningfully involve stakeholders and the time pressures to share timely results to inform care. The evidence-based framework designed by Lavis et al53 may provide 1 way for researchers to more effectively and efficiently engage patients and caregivers in the transfer of knowledge to decision makers. This framework is rooted in 5 questions: (1) What should be transferred to decision makers? (2) To whom should research knowledge be transferred? (3) By whom? (4) How? And (5) With what effect? Table 2 illustrates stakeholder-driven strategies that researchers can use to successfully convey their study results to policy makers.

TABLE 2.

Examples From Lavis and colleagues’ Framework for Conveying Scientific Findings to Decision Makers

Questions research team should ask	Evidence-based strategies for effectively conveying scientific findings which can be done with or by stakeholders
What should be transferred to decision makers?	Transfer actionable messages from a body of research knowledge, not simply a single research report or the results of a single study to influence decision making
To whom should research knowledge be transferred?	Use multiple audience-specific messages. Identify the audience and tailor the message to the types of decisions they face and the types of decision-making environments in which they live or work (which can be time-consuming)
By whom?	Using credible messengers or trusted intermediaries (the messenger becomes the message)
How?	Interactive engagement (2-way exchanges) is more effective than passive processes (1-way push efforts by the researchers)
With what effect?	Knowledge transfer might inform debate, change awareness, or change a course of action to have an impact on decision-making around health

CONCLUSIONS

Interventions that engage stakeholders, comprehensively address gaps in care, and improve the stakeholder experience and outcomes are complex. These complexities are magnified when evaluating novel transitional care interventions under real-world conditions.