The design of embedded pragmatic clinical trials: methodological developments and statistical lessons learned from the first cycle of the NIA IMPACT collaboratory

Abstract

INTRODUCTION

The National Institute on Aging‐funded IMbedded Pragmatic Alzheimer's disease and AD‐Related Dementias Clinical Trials (IMPACT) Collaboratory was established to build capacity for conducting embedded pragmatic clinical trials (ePCTs) within healthcare systems for people living with dementia and their care partners. Here we present methodology and describe lessons learned from the first five years of IMPACT's Design and Statistics Core (DSC).

METHODS

The DSC assembled a multidisciplinary team focused on advancing the design, analysis, and implementation of ePCTs.

RESULTS

The DSC developed and disseminated methods for design and sample size of cluster randomized designs with complex correlation structures; guidance for pilot ePCTs; approaches to patient–care partner ePCTs; approaches to testing health‐equity‐relevant hypotheses; and guidance for training quantitative and clinical scientists developing ePCTs.

DISCUSSION

Key gaps remain in applying dyadic designs and when studying heterogeneity of treatment effects, which will be major priorities for the next funding cycle.

Highlights

• We share lessons learned from the first five years of the NIA‐funded IMPACT Collaboratory funded to build the nation's capacity to conduct pragmatic clinical trials of non‐pharmacological interventions embedded within healthcare systems (i.e., ePCTs), among PLWD and their CPs.

• The goal of ePCT designs is to provide generalizable evidence directly applicable to real‐world clinical or care practice or to inform policy decisions.

• Implications for pilot trials are considered.

• Recommendations for health equity and use of dyadic designs are provided.

• Resources for power and sample size computations are described.

1. INTRODUCTION

The findings of conventional randomized controlled trials (RCTs) often lack “real‐world” salience for patients living with Alzheimer's disease (AD) and related dementias (AD/ADRD), owing to the understudied complexity of delivering interventions to this population in the context of routine clinical care.

The IMbedded Pragmatic Alzheimer's disease and AD‐Related Dementias Clinical Trials (IMPACT) Collaboratory ¹ , ² was funded in 2019 by the National Institute on Aging to build the nation's capacity to conduct pragmatic clinical trials of non‐pharmacological interventions embedded within healthcare systems (i.e., embedded pragmatic clinical trials [ePCTs]) among people living with dementia (PLWD) and their care partners (CPs).

IMPACT is organized into cores, teams, and working groups. The Design and Statistics Core (DSC) is charged with developing methods for the experimental design and analysis of ePCTs. To date, the DSC has developed and disseminated novel biostatistical approaches and training materials that include guidance to investigators on design, sample size, and analytic methods, emphasizing contemporary standards of rigor and reproducibility in application.

Here we describe DSC activities during IMPACT's first five years that can inform investigators planning ePCTs among PLWD and CPs. We describe created resources and areas for further development.

1.1. Randomized clinical trials need modification for the ePCT setting

The goal of ePCTs is to provide generalizable evidence directly applicable to real‐world clinical or care practice or to inform policy decisions. Thus, ePCTs contrast sharply with conventional RCTs by prioritizing broad eligibility criteria using minimal overt recruitment, focusing on effectiveness in real‐world settings rather than efficacy under ideal conditions.

Most self‐described pragmatic trials are not fully pragmatic in all respects; investigators may leverage tools, such as the PRagmatic Explanatory Continuum Indicator Summary‐2 (PRECIS‐2), ³ to assist in constructing designs that are appropriate and practicable. Investigators should assess the strength of evidence supporting the conduct of an ePCT – that is, the intervention's “readiness” to be assessed within a pragmatic framework, using tools such as the Readiness Assessment for Pragmatic Trials framework. ⁴

ePCTs must acknowledge the “embedded” delivery of interventions within structures imposed by healthcare systems, which may require modifications to interventions appropriate to specific settings. Some modifications may be fundamental in nature; for instance, we observed that one of the more common adaptations proposed in IMPACT's first cycle entailed testing efficacious interventions that had not previously been studied among PLWD. Another common adaptation is the attempt to formally assess the dependent roles of PLWDs and CPs; in what follows, we describe dyadic designs and how investigators may avoid the confusion between these and designs involving proxy measurement.

Investigators may seek to convert an intervention delivered at the level of the patient to one delivered at the level of the healthcare practice or system, necessitating clustered designs. Investigators may propose changes in setting, such as from inpatient to long‐term care; increases or decreases in the number of components of an intervention; or the frequency, duration, and intensity with which an intervention is delivered.

Many of these adaptations have implications for statistical power that are understudied. Below we present a summary of novel methods developed by the DSC in this area and describe publicly available resources developed as a part of this effort. Resources that are relevant to ePCT design, developed by both IMPACT and by external investigators, are described in Table 1.

TABLE 1.

Resources relevant for the design of ePCTs, from IMPACT and other sources.

Clustered and embedded designs
	https://douyang.shinyapps.io/adrdicc/	A web‐based interactive tool that lets users view intra‐cluster correlation coefficient (ICC) estimates on a map of the USA and retrieve regional statistics to support design and sample size planning for cluster randomized trials.
	https://kendra‐davis‐plourde.shinyapps.io/SWCRT_3Level_DesignEffect/	A web‐based interactive tool for generating sample size estimates for designing stepped‐wedge cluster randomized trials with subclusters.
	https://kendra‐davis‐plourde.shinyapps.io/SWCRTbaseline/	A web‐based interactive tool for generating sample size estimates for designing stepped‐wedge cluster randomized trials with a baseline measurement of the outcome.
	https://cluster‐hte.shinyapps.io/shinyapp/	An interactive web tool that helps researchers perform sample size and power calculations for detecting treatment effect heterogeneity in cluster randomized trials across various designs using linear mixed models.
	Giraudeau B, Weijer C, Eldridge SM, Hemming K, Taljaard M. Why and when should we cluster randomize?, Journal of Epidemiology and Population Health 2024;72(1)	Considerations for whether to conduct and how to design clustered randomized trials
	Hemming K, Taljaard M. Key considerations for designing, conducting and analysing a cluster randomized trial. Int J Epidemiol. 2023;52(5):1648–1658.
	Hemming K, Taljaard M. Reflection on modern methods: when is a stepped‐wedge cluster randomized trial a good study design choice? Int J Epidemiol. 2020;49(3):1043–1052	Considerations as to the appropriateness of design and analysis schemes within the stepped‐wedge framework
	Li F, Hughes JP, Hemming K, Taljaard M, Melnick ER, Heagerty PJ. Mixed‐effects models for the design and analysis of stepped wedge cluster randomized trials: An overview. Stat Methods Med Res. 2021 Feb;30(2):612–639.
Dyadic Designs
	Lyons KS, Russell LT, Bonds Johnson K, Brewster GS, Carter JH, Miller LM. Evaluating the Dyadic Benefits of Early‐Phase Behavioral Interventions: An Exemplar Using Data From Couples Living With Parkinson's Disease. Gerontologist. 2024;64(7):gnad172.	Example of statistical approaches to analyzing small‐sample dyadic data
	Vranceanu AM, Bannon S, Mace R, Lester E, Meyers E, Gates M, Popok P, Lin A, Salgueiro D, Tehan T, Macklin E, Rosand J. Feasibility and Efficacy of a Resiliency Intervention for the Prevention of Chronic Emotional Distress Among Survivor‐Caregiver Dyads Admitted to the Neuroscience Intensive Care Unit: A Randomized Clinical Trial. JAMA Netw Open. 2020;3(10):e2020807. doi: 10.1001/jamanetworkopen.2020.20807.	Example of reporting feasibility outcomes in dyadic research
	Bannon SM, Cornelius T, Gates MV, Lester E, Mace RA, Popok P, Macklin EA, Rosand J, Vranceanu AM. Emotional distress in neuro‐ICU survivor‐caregiver dyads: The recovering together randomized clinical trial. Health Psychol. 2022;41(4):268–277. doi: 10.1037/hea0001102.	Example of using APIM in dyadic data analyses
	https://mghconfide.org	MassaChusetts General Hospital Roybal CeNter For BehavIoral Dyadic ResEarch in Alzheimer's Disease and Related Dementias (CONFIDE‐ADRD); includes workshops and works in progress on dyadic approaches to dementia trials

1.2. Cluster randomization has implications for ePCTs among PLWD and CPs

RESEARCH IN CONTEXT

1. Systematic review: The IMPACT Collaboratory was funded in 2019 by the National Institute on Aging to build the nation's capacity to conduct pragmatic clinical trials of non‐pharmacological interventions embedded within health care systems (i.e., ePCTs) among PLWD and their CPs.

2. Interpretation: The IMPACT Collaboratory's first 5 years produced substantial methodological advances and practical resources addressing critical gaps in ePCT design for PLWD and their CPs. Key contributions include empirical ICC estimates with geographic visualization tools, novel sample size methodologies for complex clustered designs, and equity‐centered trial frameworks that address treatment effect heterogeneity. Dissemination efforts include methodological tutorials, certification program for building health equity into ePCTs, and publicly available software and calculators that enable trialists to operationalize best practices. Together, these efforts cultivate a community of practice, expand the pipeline of trialists equipped to lead high‐quality ePCTs in dementia care, and ensure rapid dissemination of emerging methods and lessons learned.

3. Future directions: Continued development of flexible dyadic intervention designs and analytic models, which clarify distinctions among proxy, patient‐focused, and dyadic approaches while accommodating variable cognitive capacity and CP availability in real‐world settings. Future designs should be planned to explore the heterogeneity of treatment effects that are likely to occur in pragmatic trials using real‐world data.

Cluster randomization is often indispensable for ePCTs among PLWD and their CPs because non‐pharmacological interventions in this population typically target providers, care environments, or system‐level healthcare delivery processes. ⁵ Cluster randomization can additionally prevent contamination that would be unavoidable if individuals within the same grouping unit were randomized to different conditions. ⁶ Hence, cluster randomized trials (CRTs) may be the preferred design for maintaining intervention fidelity, while respecting the operational complexity of dementia care systems.

In CRTs, outcomes measured on individuals within the same cluster may exhibit similarities due to the influence of shared care environment and organizational culture. This redundancy, quantified by the intracluster correlation coefficient (ICC), ⁷ imposes inefficiency, inflating the sample size required to achieve the statistical power obtained by individually randomized trials. The design effect for a standard two‐arm parallel cluster design, expressed as 1 + (average cluster size − 1) × ICC, quantifies this inflation; a seemingly small ICC will yield substantial inflation even when clusters are of the moderate size typical in ePCTs. For instance, when the ICC is 0.02 and the average cluster size 30 participants, the design effect is nearly 1.6.

The DSC, in collaboration with IMPACT's Technical Data Core, supported methodological work to generate empirical ICC estimates using national Medicare data. Ouyang et al. analyzed over 1.8 million beneficiaries with ADRD across 3436 hospital service areas ⁸ to estimate unadjusted and covariate‐adjusted ICCs for several outcomes relevant to future ePCTs, including death, hospitalization, and emergency department visits. Their findings demonstrated that, although overall ICCs were small, region‐specific ICCs varied substantially across the United States, with design effects ranging from modest to extremely large depending on geography and cluster size. To facilitate the use of these empirical ICC estimates, we developed an interactive application ⁹ that allows the visualization of ICCs on a national map with retrievable region‐specific values for trial planning. This resource is among the first efforts to generate a data‐driven foundation for planning ePCTs among PLWD and CPs.

1.3. Design of pilot ePCTs requires careful consideration

Investigators should conduct pilot and feasibility ePCTs to assess the viability of a larger effectiveness‐focused ePCT. Guidance for pilot and feasibility ePCTs builds on that for conventional randomized trials, ¹⁰ augmented by pragmatic features. Implementing all aspects of the protocol intended for the larger ePCT is likely to be impractical within the pilot, where resources will be limited. We recommend that investigators carefully tailor pilot and feasibility ePCTs and consider whether certain aspects of design can be deferred, for example, to consider carefully whether randomization is necessary in the pilot and whether recruitment of one cluster (as opposed to several clusters) may be sufficient to adequately demonstrate the feasibility of specific processes. A critical consideration is the degree to which the intervention must be refined to permit embedding it within the context of a healthcare delivery system. The IMPACT Collaboratory process involves consultation with the DSC and allied teams to support trials in making these decisions.

Where a pilot is concerned with processes that can be quantified, for example, in participant enrollment or rates of data capture, predefined feasibility targets can aid decisions about progression to a full‐scale ePCT, ¹¹ with pilot/feasibility sample sizes chosen to ensure adequate precision in estimating these targets. ¹² , ¹³ , ¹⁴ Investigators should establish clear criteria for progression to full‐scale ePCT. ¹¹ Key design elements of the subsequent trial, such as outcome assessments via electronic medical records or identification of CPs, should be piloted as appropriate.

1.4. Power and sample size computations can take advantage of a growing ecology of resources

Because ePCTs frequently use cluster randomization, IMPACT's DSC has reviewed ¹⁵ and developed methods and tools for sample size calculation across several commonly used clustered designs, including stepped‐wedge and cluster‐crossover trials. These designs require careful specification of multiple sources of correlation: the correlation among individuals within the same cluster and study period (within‐period ICC), the correlation among individuals within the same cluster but across different periods (between‐period ICC), and, for closed‐cohort designs (where the same individuals are measured over multiple periods), the correlation between these repeated measurements (intra‐subject ICC).

DSC members identified substantial gaps in existing sample size methodology for these designs and created new tools to support investigators. These include methods for powering longitudinal CRTs with additional hierarchical structure ¹⁶ (e.g., patients nested within providers nested within hospitals), approaches for trials with multiple correlated primary outcomes ¹⁷ (e.g., two quality‐of‐life subscales), strategies for incorporating baseline measures ¹⁸ to improve precision, and guidance on powering subgroup analyses ¹⁹ and treatment effect heterogeneity. ¹⁹ , ²⁰ , ²¹ DSC members also developed an application ²² to facilitate exploration of the influence of design features on statistical power and sample size requirements. Investigators should leverage these and other resources in designing ePCTs so that they are adequately powered and estimation made as efficient as possible.

1.5. Health equity should be built into the design of ePCTs

ePCTs are recognized as essential tools for advancing health equity by evaluating whether interventions address the needs of diverse populations under real‐world conditions. ²³ , ²⁴ An equity‐oriented perspective necessitates consideration of the way in which intervention effects may vary across patient subgroups differing in social, demographic, or clinical characteristics. ²⁵ As highlighted in a recent methodological review ²⁶ of pragmatic trials in ADRD populations, equity considerations are often under‐addressed in trial objectives, design, and reporting, with few trials explicitly defining equity‐relevant objectives, engaging marginalized communities, or presenting disaggregated subgroup analyses. Even when baseline characteristics are described, subgroup evaluations are seldom powered or carefully pre‐planned, limiting the ability of trials to inform equity‐relevant decisions regarding who benefits most from an intervention.

Achieving equity in interventions with clustered designs across populations requires careful attention to trial planning. ²⁷  To adequately power confirmatory heterogeneity of treatment effect analyses that can inform equity‐oriented healthcare decisions, investigators must account for two critical design parameters ²⁸ , ²⁹ , ³⁰ : the association between participant outcomes within clusters and the degree of similarity of any effect modifiers within observations obtained within a cluster. In this context, the concept of the covariate‐ICC is critical: When effect modifiers demonstrate substantial within‐cluster homogeneity, the variance of interaction effect estimators increases, necessitating larger sample sizes to detect meaningful subgroup differences that may reveal disparities in treatment response. These methodological considerations directly address the IMPACT Collaboratory's best practices ³¹ for pragmatic trials in AD/ADRD, to “clearly state health‐equity‐relevant aims and hypotheses” and “be explicit in sample size justification with regard to the health equity objective.” To support investigators in implementing these complex requirements, the DSC developed methodology regarding the sample size calculation for powering subgroup‐specific treatment effect in CRTs ¹⁹ and for powering the confirmatory heterogeneity of treatment effect analyses in cluster randomized crossover trials. ²¹ These developments were supplemented by the empirical covariate‐ICC estimates on age, race, and sex in Ouyang et al., as well as an online calculator on sample size estimation for detecting treatment effect heterogeneity across various CRT designs. ⁸ Echoing the Collaboratory's guidance, we recommend that ePCTs, as a matter of emerging practice, pre‐specify and report disaggregated subgroup analyses by key sociodemographic characteristics, even when health equity is not a primary aim. When equity‐relevant hypotheses are of primary interest, investigators should further ensure adequate power for the confirmatory heterogeneity of treatment effect analyses, accounting for the covariate‐ICC in sample size planning.

1.6. Designs considering PLWD and CPs should make formal use of dyadic methods where practicable

An intervention may assume the presence of a supportive CP. In such cases, we recommend that the ePCT adopt a dyadic health research framework, ³² , ³³ under which it is crucial that ePCTs be able to identify the care partner, that eligibility of both PLWD and CP for enrollment can be established, that both individuals are assessed in data collection, and that preference is given to each individual's report of their own health status (i.e., that direct assessments are favored over proxy reports) where practicable. ³⁴ We recommend that ePCTs adequately characterize CPs in a way that allows for the assessment of CPs’ influence on interventions’ effectiveness.

In addition, ePCTs may test interventions meant to improve not only the health status of the PLWD but that of the CP as well. Here it is critical that interventionists acknowledge the potential for the application of the intervention to the dyad ³⁵ rather than to each member individually, and that ePCTs acknowledge in their designs these aspects of the intended intervention.

1.7. Patients and CPs will benefit from continued training of healthcare professionals and dissemination of results through IMPACT and allied professionals

Training and dissemination are central to strengthening the field's capacity to design and implement ePCTs for PLWD and their CPs. IMPACT's efforts are directed toward a broad but defined audience including early‐stage investigators, experienced trialists, clinicians, health system partners, and biostatisticians engaged in pragmatic trials. To meet the needs of these groups, training is delivered through complementary modalities.

Asynchronous online learning modules and certificate programs provide foundational instruction in ePCT design, statistical considerations, ethical challenges, dyadic and proxy‐respondent methods, and the complexities of real‐world research. These are complemented by live workshops and webinars that emphasize application through case studies and design scenarios. Podcasts and written guidance offer accessible entry points and ongoing learning, particularly for clinicians and stakeholders less likely to engage in formal ePCT training. ²

While methodological development often targets the biostatistical community, the DSC emphasizes translation for a broader multidisciplinary audience. Training materials and consultations are tailored to participant roles and audience levels, with more technical content for statisticians and more practice‐oriented guidance for investigators and implementation partners. Across all formats, the DSC promotes reproducible analysis, transparent reporting, and tools that can be directly applied in trial planning and conduct.

Dissemination efforts also include methodological tutorials, publicly available software, and calculators to support implementation of best practices (Table 1). IMPACT is currently conducting an internal evaluation to assess the reach and usability of these resources and to guide ongoing improvements.

2. CONCLUSIONS

The IMPACT Collaboratory's first five years produced substantial methodological advances and practical resources addressing critical gaps in ePCT design for PLWD and their CPs. Key contributions include empirical ICC estimates with geographic visualization tools, novel sample size methodologies for complex clustered designs, and equity‐centered trial frameworks that address treatment effect heterogeneity.

Particularly impactful going forward will be the continued development of flexible dyadic intervention models to clarify distinction among proxy, patient‐focused, and dyadic approaches while accommodating variable cognitive capacity and CP availability in real‐world settings. Together with comprehensive training and dissemination efforts, these innovations provide a strong foundation for rigorous ePCTs applicable to routine dementia care.

Continued methodological refinement, expanded empirical resources, and sustained investment will be essential to fully realize the promise of ePCTs for PLWD and their CPs.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest. Author disclosures are available in the Supporting Information.

Supporting information

Supporting Information