Statistical Considerations for Embedded Pragmatic Clinical Trials in People Living with Dementia

Abstract

There is overwhelming need for nonpharmacological interventions to improve the health and well-being of people living with dementia (PLWD). The National Institute on Aging Imbedded Pragmatic Alzheimer’s Disease (AD) and AD-Related Dementias Clinical Trials (IMPACT) Collaboratory supports clinical trials of such interventions embedded in healthcare systems. The embedded pragmatic clinical trial (ePCT) is ideally suited to testing the effectiveness of complex interventions in vulnerable populations at the point of care. These trials, however, are complex to conduct and interpret, and face challenges in efficiency (i.e., statistical power) and reproducibility. In addition, trials conducted among PLWD present specific statistical challenges, including difficulty in outcomes ascertainment from PLWD, necessitating reliance on reports by caregivers, and heterogeneity in measurements across different settings or populations. These and other challenges undercut the reliability of measurement, the feasibility of capturing outcomes using pragmatic designs, and the ability to validly estimate interventions’ effectiveness in real-world settings. To address these challenges, the IMPACT Collaboratory has convened a Design and Statistics Core, the goals of which are: to support the design and conduct of ePCTs directed toward PLWD and their caregivers; to develop guidance for conducting embedded trials in this population; and to educate quantitative and clinical scientists in the design, conduct, and analysis of these trials. In this article, we discuss some of the contemporary methodological challenges in this area and develop a set of research priorities the Design and Statistics Core will undertake to meet these goals.

INTRODUCTION

To advance the general design and conduct of embedded pragmatic clinical trials (ePCTs),¹ the National Institutes of Health Healthcare Systems Collaboratory has generated a substantial knowledge base, noting the technical difficulties in embedding interventions in healthcare systems.^2,3 Complementary literature addresses some relevant challenges in the context of interventions tailored to specific morbidities⁴; incorporation of multiple healthcare systems and stakeholders⁵; measurement via administrative and billing data^6–8; and joint measurement of participant and caregiver health outcomes.⁹

Despite these advances, important gaps remain in the methodology supporting ePCT trials. For example, novel designs, such as the stepped wedge^10,11 and other multiple-period cluster trials, are increasingly common, but lack a complete methodological foundation or consistent ethical conduct and reporting.¹² Designs testing multicomponent interventions and formally accounting for treatment heterogeneity are likewise understudied in this context, and there exists only limited guidance for developing pilot embedded trials.

Embedded pragmatic trials conducted among people living with dementia (PLWD) present additional statistical challenges meriting special focus. For example, ascertaining outcomes directly from PLWD may be challenging, necessitating reliance on reports by caregivers,^13,14 and robust methods accounting for the dyadic nature of these data are needed. Trials among PLWD also typically encounter the more general difficulties attending studies in aging,¹⁵ including missing data¹⁶; irregular observation periods; interference from “competing” events, such as death or hospitalization; heterogeneity in measurements across different settings or populations; and the need for proxy responses for participants with limited communicative capacity.¹⁷ These persistent challenges undercut the reliability of measurement and the feasibility of capturing outcomes using pragmatic designs. Methods to enhance the validity and ease of use of the electronic health record (EHR) are hence a persistent need.

The overarching goals of the Design and Statistics Core of the National Institute on Aging Imbedded Pragmatic Alzheimer’s Disease (AD) and AD-Related Dementias (ADRDs) Clinical Trials (IMPACT) Collaboratory are: to be a national resource supporting the design and conduct of ePCTs in healthcare systems testing interventions directed toward PLWD and their caregivers; to develop and disseminate guidance for conducting ePCTs in this vulnerable population; and to educate a cadre of quantitative and clinical scientists to design, conduct, and analyze these trials.

We elucidate and illustrate some of the contemporary methodological challenges in this area using the Collaboratory Demonstration Project Pragmatic Trial of Video Education in Nursing Homes (PROVEN).^19,20 PROVEN is a large parallel arm pragmatic cluster-randomized trial of advance care planning (ACP) video intervention among nursing home residents with advanced illness previously described. In brief, 359 facilities were randomized in approximately 1:2 ratio with intervention facilities trained to offer a video education program designed to address common ACP decisions to nursing home residents and their caregivers. PROVEN hypothesized that the ACP video program would reduce hospital transfer rates in long-stay nursing home patients with advanced illness.

INTERVENTIONS DELIVERED TO PLWD AND THEIR CAREGIVERS

Trials in AD/ADRD often target both members of the PLWD/caregiver dyad, but it may be difficult to determine whether an intervention will obtain greater effectiveness by targeting the PLWD, the caregiver, or both. Studies in multiple illness contexts demonstrate that the measures of health recorded for both participants and caregivers are highly interrelated and influence each other over time.^18,19 In PROVEN, such measures could include evidence of advance directives for patients and caregivers. When these measures are the same for both dyad members, approaches, such as the Actor Partner Interdependence Model, explicitly consider the influence of the participant’s and caregiver’s health status on future outcomes.²⁰ This model can be used to estimate intervention effects,²¹ but few, if any, ePCTs have exploited that capability in dementia, perhaps because of challenges in obtaining equivalent or complementary measurement in participants and caregivers. To enhance the utility of dyadic methods in ePCTs, some outstanding priorities are: to develop and refine such complementary measures; to develop algorithms leveraging EHR and administrative data that include linked dyads; and to develop methods for analyzing dyadic data in which measurements on participants are increasingly difficult to obtain with time, because of accelerated cognitive decline or other intercurrent complexities.

MULTICOMPONENT INTERVENTIONS

ePCTs conducted in PLWD are commonly evaluating the effectiveness of a multicomponent behavioral intervention. Participants randomized to clusters in the experimental arm may receive all components, or only those suggested by their current health status, in a “standardly tailored” fashion.²² For example, the intervention in PROVEN was composed of five ACP decisions, and clinicians could choose which video(s) to show and the administration tool (local tablet or online web link). This approach is attractive for use in an ePCT because it mimics clinical practice in its “pragmatic” delivery of intervention components to match clinical care needs, but portends some analytic challenges. For instance, tailoring of interventions may evolve with time. In addition, there is likely to be heterogeneity²³ in delivery of specific intervention components, and some patients may be exposed to more videos than others. Moreover, in other trials, participants may be indirectly exposed to unassigned components via interaction with the other dyad member. Outstanding methodological problems in this area include consideration of the influence of participant/caregiver interactions on treatment assignment and effect estimation; estimating the heterogeneity of treatment effect with multicomponent interventions; and developing machinery to estimate sample sizes for tailored interventions (Table 1).

Table 1.

Selected Research Priorities to Address Contemporary Challenges in Designing and Analyzing Embedded Pragmatic Trials Among People with Dementia and Their Caregivers

Challenge	Research priorities
Evaluating multicomponent interventions	Methods that consider the influence of participant/caregiver interactions on treatment assignment and effect estimation and clarify population heterogeneity in treatment effects Methods or simulations informing sample size computations
Testing interventions on PLWD/caregiver dyads	Measures capturing complementary information from PLWD and caregivers Methods to clarify the influence of PLWD/caregiver interactions on treatment assignment and intervention effects Algorithms to identify caregivers in the electronic health record and link PLWD and caregiver data Models to estimate intervention effects in the context of secular declines in measurement quality
Developing methods and tools for contemporary cluster designs	Reliable, responsive, and reproducible measurement protocols leveraging the electronic health record and administrative data Sample size tools for noncontinuous outcomes and complex correlation structures Valid and efficient designs that reduce participant burden
Capturing protocol adherence in estimating effectiveness	Measurement for, and models incorporating the effects of, adherence for participants and caregivers Methods to address heterogeneity of adherence and follow-up at the participant and facility level Methods or simulations supporting sensitivity analyses, capturing the effects of adherence and attrition
Creating robust standards for embedded trials	Guidelines for design and reporting of embedded pilot trials

Abbreviation: PLWD, people living with dementia.

CLUSTER-RANDOMIZED DESIGNS

Because of the importance of providers and care delivery to the health of PLWD, we often seek to evaluate the effectiveness of interventions on PLWD by assigning these interventions at the level of the facility or healthcare system.² Cluster-randomized trials, which assign intact groups—such as all PLWD being treated in a hospital, living in a nursing home, or seen by a particular primary care provider—are well suited to addressing such questions.^24–26

In cluster trials, if we seek to analyze data at the level of the participant or caregiver—rather than aggregated by cluster—the analysis can become methodologically complex. This is because members of the same cluster share geographical, social, or other connections and their outcomes behavior may be similar, and this so-called intracluster correlation must be acknowledged in both design and analysis. In PROVEN, nursing homes have varying transfer rates, which may relate to different nursing homes’ cultures. This feature of cluster randomization decreases the effective sample size (i.e., the amount of independent information obtained from the trial), sometimes to a dramatic degree. Thus, cluster trials require larger samples—sometimes much larger—than individually randomized trials.²⁷ They also require methods of analysis that account for the intracluster correlation.

With the increasing emphasis on ePCTs in recent years, novel cluster-randomized designs, such as the stepped wedge, have becoming increasingly popular.¹⁰ An attractive feature of this design is that all clusters eventually receive the intervention: they start in the control condition, but then gradually cross over to the intervention in randomized order (Figure 1). This characteristic can help facilitate cluster recruitment,²⁸ and allows the researcher to directly observe within-cluster, and sometimes within-participant, effects of the intervention. By this mechanism, these designs tend to be more efficient than traditional parallel-arm cluster trials, obtaining greater power at a fixed sample size.²⁹

Figure 1.

Depiction of a stepped wedge cluster-randomized trial, with eight clusters and five time periods. White cells (lower left) denote time periods for which the control condition is administered, whereas blue cells (upper right) denote periods for which the intervention is delivered. Clusters, which might be collections of people living with dementia in specific settings or affiliated with specific care providers, range from two to four periods in their duration of exposure to intervention.

A major methodological consideration in a stepped wedge trial is that the intervention effect is confounded with time by design—the control protocol always precedes the intervention protocol (Figure 1)—and failure to account for any secular trends in the analysis may therefore bias intervention effect estimates.^11,30 For example, if PROVEN had been designed as a stepped wedge, natural changes in the transfer rates over time could be confounded with the intervention effect. In addition, there remain substantial methodological challenges to the deployment of this design among people living with AD/ADRD. Methods for determining sample size requirements are not fully known for noncontinuous outcomes in certain settings, especially in the presence of complex correlation structures.^31–34 Furthermore, robust methods of analysis for stepped wedge trials with small numbers of clusters have not been fully developed.

Because ePCTs in AD/ADRD will often consider assessments from not only the PLWD but also their caregiver, it is critical to further develop models that accommodate measurements on both (and their association with one another) in the context of the stepped wedge or other clustered design. A related area in need of development is the so-called “incomplete” stepped wedge, in which participants’ outcomes are captured in only a subset of all possible data collection periods.^35,36 In principle, this approach reduces participant data collection burden, but optimal design features for maintenance of validity and statistical power are yet to be investigated.

ADHERENCE TO STUDY PROTOCOL

In randomized trials, the intention-to-treat strategy, requiring that all participants are analyzed as randomized, is commonly advocated in estimating the effectiveness of interventions.^37–39 This approach is relevant for embedded pragmatic trials because it mimics imperfect, “real-world” implementation.⁴⁰

When trials experience nonadherence or differential loss to follow-up, this may produce attenuated intervention effect estimates.³⁸ This can lead researchers to conclude that an intervention is ineffective for all populations, whereas it may be effective for some, in particular, those participants who have sufficient reserve to complete it. Nonadherence with the assigned treatment arm and differential loss to follow-up (including that due to death) may be especially prevalent among PLWD, making adherence and trial participation challenging.^41,42 Moreover, in many ePCTs, the healthcare providers administer the interventions. However, these frontline providers that care for PLWD have limited time in each encounter and may not have the capacity to offer and apply additional experimental intervention, resulting in increasing nonadherence.⁴³ In PROVEN, nonadherence at the provider level corresponds to the rate at which nursing homes failed to offer the videos to all eligible residents. At the individual level, nonadherence corresponds to eligible residents and caregivers who decline watching the videos. Thus, ePCTs conducted on PLWD should emphasize methods that increase collection of outcomes on all participants and consider postrandomization auxiliary variables and proxy measures of outcomes at both the PLWD/caregiver and the provider levels.^44,45

The potential outcomes framework of causal inference⁴⁶ is useful in conceptualizing the potential influences of noncompliance and loss to follow-up.⁴⁷ Under this framework, one estimable quantity of interest is the average effect among individuals who would adhere to both the intervention and control.⁴⁸ This idea has been extended to incorporate information on posttreatment variables (e.g., levels of adherence) to refine comparisons between treatment groups and to different randomization settings.^49,50 Generally, maximizing adherence through study design is the preferred approach. However, when adherence is low, specialized analysis and data collection are required.⁵¹

Important areas of further methodological research (Table 1) include development of adherence measures for PLWD and the integration of such measures in trials; addressing heterogeneity of adherence and follow-up at the participant and cluster level; development of methods that adjust for both participants’ and caregivers’ adherence; and development of postintervention measures and strategies for multicomponent interventions. Because attrition and nonadherence occur postrandomization, statistical and practical assumptions are necessary to develop proper inferences. Thus, the development of methods to examine the sensitivity and robustness of results derived from ePCTs is an important area of further research.

RELIABLE AND RESPONSIVE OUTCOMES

Clinical trials increasingly emphasize person-centered outcomes, such as quality of life, functionality, mood, and caregiver burden.⁵² EHR outcomes capture is not structured to solicit outcomes directly from a participant, but rather relies on secondary data sources; as a result, the outcomes are more focused on process measures, like hospitalizations, emergency department visits, or location of death.

Outcomes for trials in dementia should be both reliable and responsive.^53,54 Reliability reflects consistency of measurement across different raters in the same period or consistency of measurement across limited time for the same participant or caregiver. Responsiveness indicates sensitivity to amelioration or exacerbation in the underlying construct, such as depression or stress. PROVEN used a process measure as the outcome that meets these definitions. However, an outcome that has floor or ceiling effects, such as the Decision Making Involvement Scale,⁵⁵ could be non-responsive for the portion of the cohort at the relevant end of that construct’s distribution.

In ePCTs, outcomes are likely to be derived from the EHR. When the EHR uses abbreviated versions or subsets of validated scales, reliability and responsiveness cannot be assumed to be inherited from the parent scales. Researchers may need to conduct independent assessments of these modified instruments to ensure that the measured outcomes are appropriate goals, and/or develop transformations that allow estimation of the full scale from the measured portion. Important outstanding problems in this area include integrating person-centered outcomes into the EHR while minimizing participant burden and developing methods to evaluate the reliability and responsiveness of these measures so that they can be used confidently in ePCTs.

DESIGNING AND PILOTING PRAGMATIC INTERVENTIONS

In its mission to transform the delivery, quality, and outcomes of care provided to PLWD and their caregivers by accelerating the testing and adoption of evidence-based interventions within healthcare systems, the IMPACT Collaboratory funds 1-year pilot studies of ePCTs. The Design and Statistics Core will play a major role in supporting applicants in the design and statistical analysis of their pilot applications and funded studies, although much more work needs to be done to understand how to optimally design pilots to ensure the successful progression to a full-scale ePCT.

Designing pilot studies for pragmatic interventions can be challenging, especially when applying multicomponent interventions, and the future trial will use a complex design, such as cluster randomization. The U.K. Medical Research Council⁵⁶ provides guidelines for the development and evaluation of complex interventions. The intervention development phase focuses on developing the intervention to the point where it can be reasonably expected to have a meaningful effect. This phase often includes systematic reviews of the existing evidence base and development of a theoretical basis for the mechanism by which the intervention operates, possibly using interviews with key stakeholders, such as healthcare system managers, care providers, PLWD, and their caregivers. In the pilot testing phase, the focus is on conducting a smaller-scale study in preparation for embarking on a full-scale study (i.e., fully powered trial) of the intervention. This may not necessitate randomization. The main goal is to determine whether and how to proceed with the full-scale ePCT, and not to estimate the intervention’s effectiveness.^57–60 As such, a pilot trial for an ePCT should consider areas of uncertainty specific to the full-scale evaluation of the intervention in conditions that resemble usual clinical care. For example, in PROVEN, there may be uncertainty regarding: the acceptability of the intervention in the skilled nursing facilities; the success of planned PLWD and caregiver identification and recruitment procedures to ensure representativeness; the sites’ ability to deliver the video intervention as planned; PLWD and caregivers’ anticipated adherence to the intended intervention in usual care conditions; availability and completeness of routinely collected data for outcome assessment; and the extent to which the trial can be conducted without interfering in real-world care.

Pilot designs should identify primary and secondary feasibility objectives. Qualitative, quantitative, and mixed methods should be considered when assessing feasibility. Outcomes may or may not include those to be used in the larger trial; however, hypothesis testing for effectiveness or efficacy is not recommended. The treatment difference observed in a pilot should not dictate the effect size that the future trial is powered to detect, although it may provide limited evidence of whether obtaining clinically important thresholds in a larger trial is conceivable. A pilot can usefully provide an estimated standard deviation, mean, or proportion in the control arm to inform the sample size calculation for the future trial, although there could be substantial uncertainty about these estimates if the pilot sample size is small. Confidence intervals around these parameters can be helpful to inform the choice of more conservative values to avoid underpowering the main study.

Although pilot ePCTs will have modest sample sizes, the sample size should nevertheless be justified, and this may be driven by the primary feasibility objective. Pilot trials are seldom of sufficient size to reliably estimate parameters required for the design of cluster-randomized trials, including variances and the intracluster correlations of outcomes measurement. When routinely collected EHR or administrative data are available on a larger number of sites and participants, researchers should consider leveraging these resources to calculate intracluster correlation coefficients to inform the design of the larger trial.

A final methodological consideration is prespecified criteria to determine whether or how to proceed with a full-scale ePCT. Analysis of pilot trials should set these criteria at design, and their interpretation should address whether progression criteria were met. For example, a pilot trial in advance of PROVEN may have specified a minimum level of adherence to be achieved in delivering the Advance Care Planning Video Program to residents, such that the intervention can plausibly achieve a difference that would affect decision-making.

The manner in which findings from the pilot will influence the design of the larger trial should likewise be considered during planning of the pilot trial.

Investigators conducting a pilot ePCT should register the trial and consider publishing its protocol. Robust guidelines for designing pilot embedded pragmatic trials are a critical need in this area and are currently in development (Table 1).

CONCLUSION

There is overwhelming need for nonpharmacological interventions to improve the health and well-being of PLWD and their caregivers. The ePCT is ideally suited to testing the effectiveness of complex interventions in vulnerable populations, but this applicability comes at nontrivial cost to operability, efficiency, interpretability, and reproducibility. The IMPACT Collaboratory Design and Statistics Core is focused on biostatistical methods to design, conduct, and analyze embedded pragmatic trials among PLWD; to develop and promulgate best practices in this area; and to provide guidance to researchers planning or conducting such trials. The research priorities identified above are tailored to supporting the IMPACT Collaboratory’s mandate to build the nation’s capacity to conduct ePCTs within healthcare systems for PLWD and their caregivers.