Using Hybrid Effectiveness Studies to Facilitate Implementation in Community-Based Settings:Three Case Studies in Dementia Care Research

Abstract

The pipeline from discovery to testing and then implementing evidence-based innovations in real-world contexts may take two decades or more to achieve. Implementation science innovations, such as hybrid studies that combine effectiveness and implementation research questions, may help to bridge the chasm between intervention testing and implementation in dementia care. This paper describes hybrid effectiveness studies and presents three examples of dementia care interventions conducted in various community-based settings. Studies that focus on outcomes and implementation processes simultaneously may result in a truncated and more efficient implementation pipeline, thereby providing older persons, their families, healthcare providers, and communities with the best evidence to improve quality of life and care more rapidly. We offer post-acute and long-term care researchers considerations related to study design, sampling, data collection, and analysis that they can apply to their own dementia and other chronic disease care investigations.

Summary:

We present three examples of hybrid effectiveness studies, with the goal of providing an overview of methodological approaches that can expedite the implementation of dementia care innovations.

Introduction

With the exponential increase in the number of people living with dementia, there is a need for more systematic approaches to bring evidence-based interventions to real-world settings.¹ Although there is a large body of tested dementia care interventions,^2–4 few have been implemented widely and sustained in health and community-based care settings. Numerous factors contribute to this research-practice gap, including use of a traditional, linear pathway from intervention development to efficacy to effectiveness testing; conduct of highly controlled evaluations of dementia care interventions in settings that maximize internal rather than external validity; highly diverse dementia care contexts; and consideration of implementation concerns relatively late in the translational science pipeline.

One approach to address this gap is using hybrid effectiveness studies, which examine effectiveness and implementation type research questions concurrently. Hybrid studies offer important methodological advancements to the traditional stepped, elongated pipeline. These studies balance tradeoffs between internal and external validity in intervention testing, thereby generating evidence for both effectiveness and implementation that may shorten the pipeline from development to scaling interventions.^5–7

As hybrid studies have only recently been considered in dementia care research, this paper examines their characteristics and provides three examples to illustrate distinct types and their use. We offer considerations within and across each example to guide researchers who aspire to adopt hybrid effectiveness studies in their own intervention work.

What are Hybrid Effectiveness Studies?

Hybrid effectiveness (HE) studies reflect a continuum based on their prioritization of effectiveness and implementation-type questions. As Table 1 outlines and described below, there are three primary types.

Table 1.

Overview of Three Hybrid Effectiveness Designs

Design Type	Primary Aim(s)	Secondary Aim	Key Features
Type I	Effectiveness of intervention on clinical outcomes	Understanding implementation challenges	-A range of controlled designs to examine clinical outcomes; -Use of theory/framework to guide implementation evaluation; -Implementation data collected using qualitative approaches to understand barriers and facilitators of acceptability and how the intervention is deployed.
Type II	Effectiveness and implementation simultaneously considered	--	- A range of controlled designs to examine clinical outcomes; -Use of theory/framework to guide implementation evaluation; -Formal analysis of implementation aspects such as acceptability of the intervention by stakeholders and interventionists using quantitative, qualitative, and or mixed methods strategies
Type III	Implementation outcomes	Effectiveness outcomes	- A range of controlled designs to examine effectiveness of different implementation strategies; -Evaluation of clinical outcomes to assure intervention effectiveness and as a test of implementation strategies

Type I studies evaluate a clinical intervention for its effectiveness as well as its potential for implementation in a real-world situation. The primary focus is on whether the intervention exerts a statistically and/or clinically significant effect with fidelity on a measured health outcome of participants. A secondary focus is the evaluation of acceptability and feasibility of implementing the intervention and identifying potential barriers and facilitators.⁸ Implementation information may be qualitative or quantitative and gathered prior to, during, and/or following analysis of intervention effectiveness. Type I studies are appropriate when an intervention has demonstrated limited clinical effectiveness⁵ and the goal is to lay the groundwork for its implementation and inform future implementation strategies (specifically, the intervention implemented is the “what” in HE studies, while implementation strategies are the “how”).

Type II studies simultaneously evaluate the effectiveness of an intervention and implementation strategy/strategies. Type II studies place equal emphasis on both clinical and implementation outcomes. These studies differ from Type I in that they explicitly use one or more identified implementation strategies (e.g., training, technical assistance, stakeholder involvement) and measure an implementation outcome (e.g., adoption, fidelity, acceptability by interventionists).^9,10 Type II studies are often used to understand how to implement an innovation when there is “implementation momentum” such as demand due to a system, advocacy, policy, or government mandate coupled with previous evidence of efficacy or effectiveness.⁵

At the other end of the continuum are Type III studies in which the primary focus is on implementation outcomes, whereas the secondary focus is on intervention effectiveness. These designs test an a priori specified implementation strategy or set of strategies to examine processes and outcomes such as adoption, acceptability, feasibility, implementation appropriateness, fidelity, and/or sustainability.¹¹ Strategies tested are examined to determine successful implementation of evidence-based practices and can include approaches to staff training, education, coaching, or computerized reminders to encourage intervention use.

Although HE studies are distinct as to their relative emphasis on effectiveness and implementation, they share common features. Hybrid types are flexible and can be incorporated into various evaluation designs.⁷ For experimental trials, randomization may occur at the individual or setting level with the latter requiring a cluster design. Another common feature is their incorporation of pragmatic elements (i.e., the extent to which a trial mirrors real-world conditions). For example, various HE studies use limited inclusion/exclusion criteria to obtain study samples. Whereas Type III studies typically include all who would otherwise receive standard care/services in that setting, Type I studies may include a subsample to examine implementation processes.⁵ The liberal inclusion criteria better simulate real-world participants and conditions and are aligned with their emphasis on external validity.

Each hybrid type also lends itself to the use of mixed methods to analyze implementation processes.^12,13 Mixed methods approaches integrate qualitative and quantitative data collection and can inform potential adaptations needed for implementation and identify barriers and facilitators of implementation at individual, staff, and organizational levels. Finally, regardless of HE type, similar implementation theories/frameworks, strategies, and outcomes may be deployed.^14,15

Examples

To understand the benefits and nuances of hybrid types, we describe three studies and highlight their unique and shared characteristics as well as challenges in Table 2.

Table 2.

Summary of Programs, Hybrid Designs, and Pragmatic and Implementation Elements

Program & Delivery context	Hybrid Design & Outcomes	Pragmatic Design Elements			Implementation Design Elements
		On-site staff provide program	On-site staff identify participants	Select data elements collected from program	Adaptations needed to Intervention	Theories/frameworks Guiding Implementation	Measures	Implementation strategies	Challenges
ADS Plus Adult day services	Design: Cluster randomized trial at site level and qualitative interviewing Type I (Test of effectiveness and secondarily, understanding implementation feasibility and acceptability) Outcomes: Caregiver depression; stress; days using ADS; perceived benefits of intervention by staff and caregivers	Yes	Yes (At intake or by appointed staff research coordinator)	Census data	Materials translated into Spanish; Creation of videos for recruitment which speak to caregiver cultural values; Specialized training of select sites	Facilitators and barriers to implementation	Qualitative individual interviews with select staff providing ADS Plus and caregivers in the intervention	-Stakeholder involvement -Manuals -Staff training -Staff member as recruitment coordinator -Coaching	-Fluctuations in organizational readiness due to census, funding, staff turnovers -Limited staff time for training and variable dementia knowledge -COVID-related disruptions in operations and intervention delivery -Selecting a framework post hoc to interpret implementation data -Modifications made due to COVID-19, but a formal modification framework was not used to document
COPE-Connecticut Home and community-based Medicaid Waiver Programs	Design: Randomized controlled trial at individual level and focus groups Type II (test of effectiveness and implementation acceptability using a theory base and focus groups Outcomes: PLWD behavioral symptoms, functional dependence, and activity engagement; caregiver wellbeing and distress about PLWD behavioral symptoms; care manager and interventionist acceptability of intervention congruence	No	Yes (Care managers based on caseload clients identified by EMR)	Evidence of dementia; identification of families for provisional eligibility	No	NPT	Focus groups with care managers and interventionists using NPT domains to guide interviews	-Stakeholder involvement -Manuals -Staff training -Coaching	-Inconsistent communications between care managers and interventionists; Inconsistent follow-up with medical care providers
COPE-Pace PACE	Design: Cluster randomized trial at site level; pre-post staff competency in program delivery Type III (test of two different staff training approaches (self-paced online training vs. in-person intensive training) on staff ability to implement intervention and family outcomes Outcomes: Caregiver upset and wellbeing; PLWD behaviors and functional dependence; staff confidence in implementation	Yes	Yes (Caseloads of participating OTs, nurses, or other health professionals)	Assessment scores and intervention progress recorded in electronic records	Addition of social worker to collaborate with caregivers who are depressed and/or need bereavement support; integration of case presentations at ongoing staff meetings	NPT; PARIHS	-Organizational Readiness -NoMAD -ERIC checklist of implementation strategies used	-Stakeholder involvement -Manuals -Staff training -Early adopters to serve as site champions -Coaching	-Staff overwhelmed -Limited staff time for training -COVID-related disruptions in operations and intervention delivery

Note: PLWD = Person living with dementia; EMR = electronic medical record; OT = occupational therapy; NPT=Normalization Process Theory; PARIHS=Promoting Action on Research Implementation in Health Services (PARIHS); ERIC=Expert Recommendations for Implementing Change; NoMAD= Normalization MeAsure Development

Type I Example: Adult Day Services Plus (ADS Plus)

ADS Plus is a program in which on-site staff provide family caregivers of adult day services (ADS) clients systematic support by meeting with them over 12-months to provide disease education, skills training, strategies for taking care of themselves and managing complex care challenges, and referrals and linkages. Prior quasi-experimental evaluations of ADS Plus indicated positive benefits on caregiver outcomes and reduced nursing home admission for ADS clients.^16,17

To evaluate this approach, the ADS Plus evaluation relied on a Type I study to test its effectiveness on caregiver well-being and client service utilization at six and 12-months using a cluster randomized trial design with 34 sites and 203 caregivers.¹⁸ Randomization occurred at the site level with 18 ADS serving as controls (usual ADS only) and 16 as intervention (usual ADS and ADS Plus). The study’s secondary aim was to understand how and the extent that ADS Plus was embedded within routine ADS delivery. Using qualitative and quantitative data, the study evaluated whether staff interventionists and caregivers found the intervention feasible, acceptable and helpful, and one that identifies key implementation concerns. Study participants (caregivers) were interviewed by telephone at baseline, six, and 12 months to capture intervention benefits. Additionally, at six months ADS Plus caregivers completed a brief survey to assess intervention utility, acceptability, and feasibility. At 12 months, semi-structured interviews were conducted with a subsample of 24 ADS Plus caregivers to obtain in-depth insights as to intervention benefits, barriers and facilitators of using prescribed care strategies. Additionally, 15 ADS staff interventionists completed semi-structured interviews to understand their perspectives on ADS Plus implementation. As such, implementation data/information were collected alongside an evaluation of effectiveness, reflecting the goals of a HE Type 1 study.

The ADS Plus study featured several pragmatic elements: ADS sites identified candidate family caregivers from their census; study inclusion criteria were broad reflecting characteristics of ADS users; the intervention was delivered by staff onsite, by telephone or a combination; and intervention delivery was integrated within routine workflows of ADS staff who met with caregivers when they dropped off or picked up family members or during times of mutual convenience. Built into the documentation of intervention delivery were brief measures to assess caregiver progress (changes in upset/stress, level of difficulty, and confidence managing care challenges). Also, routine program data were collected from sites’ electronic records (number of days clients attended ADS programs; caregiver attendance in sessions).

Key implementation challenges included: fluctuating organizational readiness to implement change;¹⁹ staff turnover and shortages affecting intervention training and delivery; disruptions in census and funding levels; COVID-related closures; and the need for cultural and linguistic adaptations to maximize ADS Plus applicability and reach.²⁰ Given the large number of available frameworks to select from when interpreting the range of implementation data available in the ADS Plus Type I HE study (see Discussion section), selecting an appropriate framework to guide this effort was a challenge. Also, although modifications in intervention delivery mode and other elements were made in response to COVID-19, a formal intervention modification framework (see below) was not utilized to record the process of adaptations.

Type II Example: Care of Persons with Dementia in their Environments (COPE-Connecticut)

The COPE intervention provides disease education and skills to address self-identified care challenges with the goal of maximizing physical function and quality-of-life in people living with dementia in the community. The intervention works by realigning their capabilities with environmental demands and supporting family caregivers. Family care challenges may involve managing behavioral symptoms and functional declines, coordinating care, juggling family needs, employment or their own health conditions. COPE includes up to 10 home visits over four months by an occupational therapist and one home visit and follow-up phone call by a nurse who provides education about common medical concerns (hydration, pain management, nutrition, advance care directives), along with ruling out underlying medical disorders and inappropriate medication-taking that place the person with dementia at risk. Prior research found COPE efficacious in improving well-being of persons with dementia and their caregivers as well as service use outcomes in three randomized controlled trials with community volunteers.^21–23

The research team used a HE Type II study (2014–2020) to evaluate effectiveness and implementation outcomes of COPE when introduced into Connecticut’s Home and Community Based Services (HCBS) for older adults. Type II was selected to simultaneously consider statewide effectiveness of COPE as well as evaluate implementation success within the HCBS program across Connecticut. HCBS are available to older adults who are medically eligible for nursing home admission and financially eligible for services funded by Connecticut’s Medicaid program and state revenues. Because most states have similar HCBS (e.g., “waiver” programs), lessons learned from this trial could stimulate efforts to disseminate COPE throughout Connecticut and other states. The HE Type II study was appropriate because an implementation theory/framework was utilized to guide strategies to embed COPE in the Connecticut HCBS program and implementation outcomes were utilized to ascertain the success of these efforts. Simultaneously, the COPE evaluation was designed to evaluate the effectiveness of the intervention on health, well-being, and service use outcomes for people living with dementia and their caregivers.

The study used a randomized trial at the individual level, during which families in the HCBS were identified by care managers from their caseloads and then randomly assigned to receive HCBS services alone or HCBS services plus COPE which was delivered by trained research staff. Broad inclusion criteria reflected the waiver dementia population. Also, COPE families were referred back to their care managers for additional services as needed, extending the potential impact of HCBS and COPE for families.^22,24,25

The implementation component of the HE Type II trial was guided by Normalization Process Theory.²⁶ This framework provides a systematic approach when accounting for both individual participants (e.g., COPE interventionists) and organizational factors (e.g., daily practice routines) that influence the implementation potential of sustaining COPE within HCBS. Criteria defining feasibility and acceptability to staff are: coherence (does the intervention make sense and fit practice goals and activities); cognitive participation (are staff willing to invest time and energy to learn and practice the intervention); collective action (activities/operations staff engage in to facilitate the intervention); and reflexive monitoring (how staff perceive an intervention after it is delivered for some time). These four criteria guided the conduct of focus groups which were convened annually for three consecutive years with COPE interventionists and HCBS care managers. COPE was perceived as acceptable and appropriate by both care managers and interventionists, with staff strongly supporting its full integration and continuation.²⁷

As COPE was delivered in homes and by research staff and the approach is person and family-centric, the research team did not find any programmatic adaptations necessary. They identified two primary implementation challenges, however: (1) inconsistent communications between care managers and interventionists about family needs and recommended strategies, and (2) inconsistent follow-up with medical care providers by family caregivers and COPE research staff. These challenges will inform the integration of COPE within HCBS moving forward.

As a result of positive outcomes including COPE’s cost savings, impact on families, and perceived benefits by care managers/HCBS staff, Connecticut executive and legislative branch efforts are currently underway to implement COPE as an available service statewide under authority of the Medicaid program and their informal caregivers. The Type II HE study provided evidence simultaneously of effectiveness and implementation to facilitate its subsequent dissemination.

Type III Example: COPE in Programs of All-Inclusive Care for the Elderly (COPE-PACE)

Researchers are also partnering with Trinity Health, the largest Program of All-Inclusive Care for the Elderly (PACE) provider in the United States, to implement COPE in ten facilities selected for their organizational readiness to participate as well as their diversity in location, size, and populations served. The trial uses a Type III HE study to test the effectiveness of two different implementation approaches to training PACE staff in delivering COPE: a labor-intensive, face-to-face 3–4-day training versus a self-paced online program involving 10 engaging modules of up to 50 minutes each. The purpose of the study is to evaluate which staff training approach (i.e., implementation strategy) is most effective for learning the COPE program and implementing it with PACE families. Of ten sites, five are randomized to have occupational therapy and nursing staff trained face-to-face and five are randomized to have staff trained via online. Implementation outcomes include staff confidence implementing COPE, and effectiveness outcomes of the program for families include reduced behavioral symptoms and functional dependence as well as improved caregiver well-being.

Two primary implementation theories/frameworks inform this study: Normalization Process Theory (NPT) and Promoting Action on Research Implementation in Health Services (PARIHS). As described above, NPT informs an understanding of individual determinants of staff behavior in PACE that may influence client outcomes.²⁶ The PARIHS framework includes a comprehensive, multi-level taxonomy of factors influencing implementation (e.g., inner and outer settings of the organization, implementation processes).²⁸ Both NPT and PARIHS are synergistic in driving implementation success and guided our consideration of what implementation strategies were important to target and measure.

As a first step, Implementation Mapping (IM) was used to help organize conceptual frameworks in order to systematically pre-plan implementation strategies and the measurement of implementation processes and outcomes.²⁹ Measures of implementation process and outcomes include the following: 1) the Normalization MeAsure Development (NoMAD) instrument,³⁰ derived from NPT, that captures the ways in which work must be reconfigured both individually and collectively by those involved in implementation; 2) the Organizational Readiness to Change Assessment (ORCA),³¹ derived from the PARIHS model, which tracks organizational readiness on the part of interventionists as well as PACE representatives at the middle and upper management levels; 3) the Framework for Modification and Adaptation (FRAME) tool to report adaptations and modifications prior to, during and after program implementation;³² and 4) an investigator-developed checklist based on the Expert Recommendations for Implementing Change (ERIC) to classify implementation strategies.⁹ The ERIC tool includes thematic clusters of specific strategies mapped to each cluster that were selected based on relevance to the dementia setting, alignment with conceptual frameworks, and prior dementia research demonstrating earlier success. The strategies applied in the PACE COPE Type III HE study include building relationships with partners, engaging participants in the development of the dynamic training process, development of clinician reminder systems, and consideration of PACE payment structure for occupational therapy/registered nurse services.

The ERIC checklist and identified adaptations will be gathered by a designated study team member in collaboration with a site designee, documented in a database, and discussed in on-going team meetings of research and site administrators. NoMAD and ORCA will be administered prior to implementation and four months post-implementation.

Discussion

The three examples highlight several similarities, including the common use of randomization (individual or site level); reliance on organizational staff to collect data on intervention fidelity/implementation and, in some instances, on client outcomes; and use of various implementation strategies. The three example studies also systematically trained staff; involved stakeholders in research meetings to assure shared understandings; provided systematic and consistent training to interventionists with follow-up coaching; and relied on manuals to support fidelity with the aim of facilitating later implementation efforts.

One theme that consistently emerged across the three examples was organizational readiness. Organizational readiness is defined as “organizational members’ psychological and behavioral preparedness to implement change.”^{19, p. 217} Organizational readiness was identified as a significant challenge in the ADS Plus Type I HE study as it was not considered nor measured prior to implementing the intervention in adult day programs, which caused issues in the study as many programs that originally agreed to participate (via a letter of support from program directors) did not proceed further with staff training or other protocol elements due to the time investment required or similar issues. This is contrast to the other two examples; both the COPE HBCS Type II and the COPE PACE Type III HE studies incorporated assessments of organizational readiness in their designs, facilitating intervention implementation across their community-based settings.

Yet another shared challenge concerns staff and staff communication. Turnover, communication challenges, availability, disease knowledge, training needs, and acceptance of the intervention were challenges initially, during and after implementation across all three HE studies. Additionally, as institutional review board and other research oversight requirements are extensive, staffing burden and retention also posed challenges to participating agencies and may cause delays in implementation (e.g., in the ADS Plus Type I study). Across all three case examples, even if a percentage of staff time was budgeted in parent grants to support integration of evidence-based interventions into community-based programs, staff in these programs were still expected to perform their regular responsibilities. These time pressures heighten the need to align training (in terms of length, mode, and delivery), intervention delivery, fidelity assessment, and resources so that they are incorporated into program workflows effectively. Otherwise, as noted in the examples noted here, challenges related to regular and efficient communication, integration of a dementia care intervention as intended, and similar concerns may be exacerbated throughout the process of implementation.

The three examples also highlight the shared challenges encountered when evaluating effectiveness and implementation in real-world settings. One challenge is choosing from among the considerable number of theories, models, and frameworks that are now available to guide implementation efforts. Whereas theories in implementation science often include predictions of implementation outcomes, models can be used to describe implementation processes and frameworks can inform identification of barriers impacting outcomes. As shown, each hybrid type relied upon different theories, frameworks, and/or models to guide their respective implementation approaches.¹⁴

Another common challenge concerns the level of pragmatism that can be achieved, if so desired. Each study had to balance the need to rely on data ascertained during the course of usual clinical care to maximize efficiency and minimize burden³³ with the need to examine aspects of implementation that range from intervention effectiveness (Type I) to implementation strategies (HE design Types II-III).¹¹ Pragmatic data sources range from administrative claims to other clinical data routinely documented in patients’ electronic health records. Yet for HE studies that are more pragmatic in design, modifications to providers’ documentation may need to occur to capture relevant data to understand implementation processes. Furthermore, when conducting pragmatic HE studies that include people with dementia or those with limited capacity, researchers must take into account the regulatory and ethical implications associated when using secondary data sources in order to mirror real-world conditions that are the goal of more pragmatic trials.^11,33

Implications for Practice, Policy, and/or Research

Although randomized controlled trials of dementia care interventions have been available for decades, a consistent refrain across meta-analyses and systematic reviews are the high proportion of studies characterized by small samples, inadequate reporting, lack of conceptualization, and high attrition which makes conclusions concerning their efficacy tenuous. Hybrid effectiveness studies may thus be favored given that the need to establish the effectiveness of dementia care interventions in real-world care settings remains paramount. Similarly, dementia care scientists can take advantage of the full HE continuum and not only evaluate effectiveness, but implementation potential as well. However, research in dementia or other areas relevant to post-acute and long-term care must move beyond the inordinate value placed on traditional clinical trials and situate evidence “quality” on a continuum of efficacy, effectiveness, and implementation as offered by HE studies. In doing so, the field will progress towards more viable, scalable, and acceptable intervention approaches that more quickly benefit people living with dementia and those who care for them.