Selecting Outcomes to Ensure Pragmatic Trials Are Relevant to People Living with Dementia

Abstract

Outcome measures for embedded pragmatic clinical trials (ePCTs) should reflect the lived experience of people living with dementia (PLWD) and their caregivers, yet patient- and caregiver-reported outcomes (PCROs) are rarely available in large clinical and administrative data sources. Although pragmatic methods may lead to use of existing administrative data rather than new data collected directly from PLWD, interventions are truly impactful only when they change outcomes prioritized by PLWD and their caregivers. The Patient- and Caregiver-Reported Outcomes Core (PCRO Core) of the IMbedded Pragmatic Alzheimer’s Disease (AD) and AD-Related Dementias Clinical Trials (IMPACT) Collaboratory aims to promote optimal use of outcomes relevant to PLWD and their caregivers in pragmatic trials. The PCRO Core will address key scientific challenges limiting outcome measurement, such as gaps in existing measures, methodologic constraints, and burdensome data capture. PCRO Core investigators will create a searchable library of AD/AD-related dementias (ADRD) clinical outcome measures, including measures in existing data sources with potential for AD/ADRD ePCTs, and will support best practices in measure development, including pragmatic adaptation of PCROs. Working together with other Cores and Teams within the IMPACT Collaboratory, the PCRO Core will support investigators to select from existing outcome measures, and to innovate in methods for measurement and data capture. In the future, the work of the IMPACT Collaboratory may galvanize broader embedded use of outcomes that matter to PLWD and their care partners in large health systems.

Imagine that in a single site efficacy trial, investigators find that a transitional care intervention tailored for people living with Alzheimer’s disease (AD) and AD-related ds (ADRDs) is associated with improved patient quality of life and neuropsychiatric symptoms and reduced caregiver burden, with fewer subsequent hospitalizations. Their results are compelling, but evidence remains insufficient to drive nationwide implementation of their transitional care intervention. The investigators therefore propose a logical scientific next step—a multisite pragmatic clinical trial with a more diverse patient population. Consistent with pragmatic trial design, they elect to focus on hospitalizations as the primary outcome and forego outcome measures requiring questionnaires or interviews. Grant reviewers then raise concerns that fewer hospitalizations could be counter to priorities and preferences of persons in minority race or ethnic groups, and urge consideration of more person-centered outcomes. Investigators appreciate the relevance and scientific significance of outcomes reported by people living with dementia (PLWD) and their caregivers, yet they are unsure how to incorporate these outcomes in a pragmatic clinical trial.

INTRODUCTION AND RATIONALE

AD and ADRDs affect 5.8 million Americans and their caregivers, and their prevalence is increasing.^1–3 Dementia care in the United States costs more than $200 billion annually, and is financially toxic to families who cover 70% of these costs.^4–7 Meaningful treatment or cure is many years away, and there is an urgent need to improve outcomes that are relevant to PLWD and their caregivers.

Efficacy and effectiveness trials inform clinical strategies to improve outcomes. Although efficacy trials test clinical innovations under controlled circumstances, embedded pragmatic clinical trials (ePCTs)—a type of effectiveness trial embedded in real-world healthcare settings with pragmatic data collection on outcomes—provide evidence regarding whether behavioral and nonpharmacological interventions are effective in real-world settings, and with diverse populations.⁸ Given the extraordinary public health impact of AD/ADRD, this level of evidence is critical to change care and outcomes.

To catalyze the scientific response to this compelling public health need, the National Institute on Aging (NIA) created the IMbedded Pragmatic Alzheimer’s Disease and AD-Related Dementias Clinical Trials (IMPACT) Collaboratory. As an essential component of IMPACT, the Patient- and Caregiver-Reported Outcomes Core (PCRO Core) aims to promote optimal use of outcomes relevant to PLWD and their caregivers in pragmatic trials.

Outcome measures for ePCTs should reflect the lived experience of AD/ADRD, yet PCROs are rarely available in large clinical and administrative data sources. PCROs are collected by asking patients or caregivers to report their health outcomes using surveys or interviews. This data collection method is rarely utilized in usual clinical practice, and thus poorly matched to pragmatic trial design. Although pragmatic methods may strongly recommend use of existing administrative data rather than new data collected directly from PLWD, interventions are truly impactful only when they change outcomes prioritized by PLWD and their caregivers. It is conceivable that outcome measures for pragmatic trials could still be relevant to the lived experience of AD/ADRD, even when measurement does not come from PCROs. For example, if strong evidence establishes a link between frequent hospital transfers and high caregiver burden, an investigator may argue for hospital transfers as a patient and caregiver relevant outcome measure.

Further, the list of important outcomes is extensive. For example, research using patient-, caregiver-, and clinician-reported outcomes shows that distressing symptoms are prevalent in this disease, including shortness of breath, pain, neuropsychiatric symptoms, feeding problems, and problems with personal cleanliness.^9–16 Also, people living with late-stage AD/ADRD experience frequent burdensome transitions between care settings, resulting in poorly coordinated and fragmented care.^17–19 Caregivers report high levels of strain, depression, and physical illness and loss of employment linked to their caregiving role.^20,21 These and other important person- and caregiver-centered outcomes are largely measured using questionnaires or interviewing PLWD and their caregivers, and are not reflected in administrative data. Indeed, the need for development and use of PCROs has been highlighted by the Alzheimer’s Association, and identified as a priority in the 2017 NIA National Research Summit on Care, Services, and Supports for Persons with Dementia and Their Caregivers.²² However, as the hypothetical example above illustrates, use of PCROs in ePCTs presents unique challenges. The objectives of this article are to outline critical scientific challenges in outcome measurement for pragmatic trials enrolling people with AD/ADRD and their caregivers and to describe research strategies to address and overcome these challenges.

SCIENTIFIC CHALLENGES IN AD/ADRD OUTCOME MEASUREMENT FOR ePCTs

There are three key scientific challenges in ADRD outcome measurement for ePCTs. They relate to gaps in existing measures, methodologic considerations, and existing data collection systems.

Gaps Remain in Development of Outcome Measurements Relevant to PLWD and Their Caregivers

At every stage of AD/ADRD, at least some measures based on written questionnaires or interviews are available to capture the lived experience of the disease as PCROs.^23–27 Investigators have developed PCROs in many domains, including function, cognition, satisfaction with care, quality of life, physical symptoms, neuropsychiatric symptoms, advance care planning, and family caregiver experience. Despite this progress, AD/ADRD research is still constrained by a lack of measure development and testing for some outcomes prioritized by PLWD and caregivers.²⁸ Critical gaps in outcome measurement for this population include (1) a deficiency-focused approach, with focus on impairment as opposed to measures capturing strengths, adaptations, resilience, and well-being; (2) insufficient stakeholder input to define and prioritize outcome domains across the stages of AD/ADRD; and (3) limited validation and language translation for use with minority racial or ethnic populations who may have a different lived experience of AD/ADRD.^29–31

Methodologic Challenges Limit the Use of Existing AD/ADRD PCROs in Pragmatic Trials

As described in the introductory scenario, investigators face unique methodologic challenges as they seek to use existing PCROs in ePCTs. Design features of many instruments used to capture PCROs impede pragmatic use. Written questionnaires or interviews typically impose high respondent burden, and are rarely tested in real-world clinical settings for wide-scale application. Examples of methodologic limitations include (1) psychometric gaps for some existing outcome measures, particularly lack of evidence for responsiveness and sensitivity to change; and (2) lack of scientifically sound approaches for use and analysis of PCRO self-report versus proxy report. PCROs are rarely tested for performance in the diverse populations and settings relevant to AD/ADRD ePCTs. Further, early in the disease trajectory, use of PCROs is feasible, but over time the PLWD loses the capacity to self-report outcomes—yet no best practices exist for transition to proxy report or for crosswalk and analysis of self-report plus proxy report.^32–34 These methodologic challenges limit the design and interpretation of outcome measures for ePCTs.

Methodologic Challenges Affect Data Capture of PCROs in ePCTs

Investigators who design an AD/ADRD ePCT must not only select an outcome that matters to PLWD, they must design outcome data capture that is both feasible and consistently implemented. Barriers include (1) lack of pragmatic and robust data collection systems for data capture of AD/ADRD outcomes, cutting across use of patient-, caregiver-, and observer- or clinician-reported outcomes; (2) research ethics standards for written consent when PCROs are collected; (3) lack of efficient and consistent methods to include the voice of the PLWD in early and moderate stage disease, and to use proxy reporting in later stages; and (4) the “denominator problem” of underdiagnosis, inconsistent International Classification of Diseases (ICD) coding of AD/ADRD, and lack of staging variables. These challenges impede feasible and consistent data capture for outcomes in the design of AD/ADRD ePCTs.

RESEARCH TO PROMOTE PERSON- AND CAREGIVER-CENTERED OUTCOMES IN ePCTs

At this early phase of the IMPACT Collaboratory, five activities can promote PCROs in ePCTs: use of a related conceptual framework, creation of a searchable library of potential measures, evaluation of existing measures for their suitability, promotion of future opportunities to fill measurement gaps, and promotion of best practices to use PCROs in ePCTs.

Utilize a Conceptual Framework for AD/ADRD Outcome Measurement

The IMPACT PCRO Core work plan uses a conceptual framework of person- and caregiver-centered outcomes for AD/ADRD. Given the critical role of the Alzheimer’s Association in setting standards and recommendations for person-centered care, the PCRO Core seeks to align person- and caregiver-centered outcome measures with the domains of care used in the 2018 Dementia Care Practice Recommendations (Figure 1). Outcomes can be mapped to the nine care domains included in their guideline: Detection and Diagnosis; Assessment and Care Planning; Medical Management; Information, Education, and Support; Dementia-Related Behaviors; Activities of Daily Living; Workforce; Supportive and Therapeutic Environment; and Transitions and Coordination of Services.³⁵ Further, the process of mapping existing outcome measures will clarify important gaps in measure development. Thus, the work of the IMPACT PCRO Core will be coordinated with that of the Alzheimer’s Association new Leveraging an Interdisciplinary Consortium to Improve Care and Outcomes for Persons Living with Alzheimer’s and Dementia Network, which is critiquing existing measures and promoting the development of new ones, as well as their dissemination.

Figure 1.

Mapping outcome domains to the Alzheimer’s Association Care Practice Recommendations.⁵⁶

Create a Searchable Library of AD/ADRD Clinical Outcome Measures

Existing AD/ADRD PCROs and other outcome measures provide a starting point for outcomes methodology for pragmatic trials.^36,37 Therefore, the PCRO Core investigators will create a searchable web-based library of AD/ADRD clinical outcome measures (Table 1). The library will include patient-reported outcomes, caregiver-reported outcomes, clinician-reported outcomes, performance/observation, and health service and other utilization outcomes. As in the examples provided in Table 1, each outcome measure will be linked to one of the nine domains in the Alzheimer’s Association’s conceptual framework for ADRD measurement (Figure 1). Outcome measures will be prioritized for inclusion if they (1) address an AD/ADRD person- or caregiver-centered outcome domain; (2) are acceptable to PLWD or their care partners; (3) have demonstrated importance to other key stakeholders, such as health system leadership; (4) meet psychometric standards for validity, reliability, and responsiveness/sensitivity to change; and (5) demonstrate pragmatic properties, such as feasibility and low respondent burden.³⁸ Relevant outcome measures will be identified in a variety of ways, including review of peer-reviewed scientific literature and technical reports. In addition, they will build on and reference existing repositories, similar in structure to that compiled by one of the European Union Joint Programme-Neurodegenerative Disease Research working groups.³¹ Further, as evidence for feasibility, we will seek clinical outcome measures that have been used in “real-world” efficacy (stage 3), effectiveness, and pragmatic trials (stage 4).^39–44 Finally, we will work with the Stakeholder Engagement Team within the IMPACT Collaboratory to ensure that included measures—whether PCROs or collected in other ways—do address outcomes that are relevant to the lived experience of AD/ADRD. PCRO Core investigators will provide descriptions of included outcome measures so that investigators can explore the library based on conceptual domains, psychometric properties, pragmatic properties, type of respondent, use across AD/ADRD stages, and other characteristics.

Table 1.

Examples of AD/ADRD Clinical Outcome Measures Suited for ePCTs

Outcome domain	Clinical outcome tool	Measure type	Methods for data capture
Detection and diagnosis (cognitive function)	Brief Interview for Mental Status	Person-reported outcome	Embedded in Minimum Data Set
Assessment and care planning	Preference Assessment Tool	Person-reported outcome	Embedded in Minimum Data Set
Medical management	Pain Assessment in Advanced Dementia57	Clinician-reported outcome	Brief clinician observational tool with five items for pain behaviors; suitable for embedding in EHR
Information, education, and support	Short-form Zarit Caregiver Burden Interview58	Caregiver-reported outcome	Brief survey in formats ranging from one to six items, suitable for embedding in EHR
Dementia-related behaviors	Confusion Assessment Method	Clinician-reported outcome	Embedded in Minimum Data Set49; suitable for embedding in EHR
Activities of daily living	Short Functional Survey	Clinician-reported outcome	Embedded in Minimum Data Set; suitable for embedding in EHR
Workforce	Staff hours in direct caregiving	Utilization outcome	Administrative data sources
supportive and therapeutic environment	Caregiver report of quality of hospice care	Caregiver-reported outcome	CAHPS Hospice Survey
Transition and coordination of services	Hospital transfers	Utilization outcome	Administrative data sources or EHR
Person centered	Dementia Quality of Life–Care Home59	Clinician-reported outcome	Staff survey with items suitable for embedding in EHR; subsets of item capture engagement, function, positive emotion, or negative emotion

Abbreviations: AD, Alzheimer’s disease; ADRD, AD-related disorder; CAHPS, Consumer Assessment of Healthcare Providers and Systems; EHR, electronic health record; ePCT, embedded pragmatic clinical trial.

Evaluate Outcome Measurement Tools in Existing Data Sources with Potential for AD/ADRD ePCTs

Historically, AD/ADRD PCROs are absent from large-scale administrative data sets, such as Medicare. Other large-scale data sources, including the nursing home Minimum Data Set (MDS) 3.0, Consumer Assessment of Healthcare Providers and Systems (CAHPS) Surveys, electronic health records (EHRs), and clinical registries, have only recently begun to include a limited number of PCROs. National cohorts (e.g., Health and Retirement Study and National Health and Aging Trends Study) capture some PCROs, but these items require further adaptation and testing to make them feasible in ePCTs. Some large-scale data sets have embedded PCROs or other items with the potential for use as outcome measures. Evidence for psychometric properties may be found separately, in published instrument validation studies, or may be presented in technical reports available on the Centers for Medicare and Medicaid Services (CMS) website. Although embedded measures are clearly pragmatic, investigators will also have to address limitations. For example, people living with AD/ADRD frequently transition between acute, subacute, and home settings, yet measures in these administrative data sources are typically setting specific. In addition, the timing of data collection, such as the quarterly MDS assessment in nursing homes, may be inadequate to make the trajectory of outcomes expected for a chosen intervention. Nonetheless, several existing data sources include potential outcome measures.

• Standardized Assessment Tools in Post-Acute Care and Long-Term Care: Since 1991, the MDS has been used to generate standardized data on resident characteristics and outcomes in U.S. nursing homes. MDS 3.0 introduced resident and proxy reported items.^45–48 As part of CMS efforts to implement the mandates of the Improving Medicare Post-Acute Care Transformation Act of 2014 (IMPACT Act), some of these items are being expanded in MDS and included as standardized data elements in the Home Health Outcome and Assessment Information Set, the Long-Term Care Hospital Continuity Assessment Record and Data Set, and the Inpatient Rehabilitation Facility Patient Assessment Instrument. Other items, some of which are also being standardized across assessment tools, address utilization, diagnoses, and observer reported outcomes. Because of the comprehensive scope of these tools and their intended use in busy clinical settings and completion by available clinical staff without specialized training, most of the items aim to screen for potential problem areas and then trigger more in-depth clinical review and assessment. As a result, items might not address all relevant dimensions of a topic. For example, the performance-based Brief Interview for Mental Status, although highly correlated with cognitive screening instruments, only assesses three domains of cognition and does not assess executive function, safety judgment, or decisional capacity.⁴⁹ In addition, some items are reliant on the accuracy of routine staff report and documentation, where competing priorities or incentives may exist.⁵⁰ Despite these limitations, some of the standardized items might serve as useful outcome tools in AD/ADRD ePCTs.

• CAHPS surveys: The CAHPS is a set of standardized surveys about patient experience with health care. Of this set of instruments, only two rely on informal caregivers (i.e., CAHPS Hospice Survey) or allow proxy respondents (Home Health CAHPS). CAHPS Hospice, administered after the death of a hospice patient, potentially includes caregiver-reported outcomes, such as unmet needs for palliation of symptoms, concerns with communication, and whether the caregiver received the right amount of support for his/her religious or spiritual beliefs. However, there are challenges to their use in ePCTs. Unlike the MDS, personal identifiers are not recorded on the CAHPS Hospice Survey. Thus, the unit of analysis is by necessity the hospice provider rather than an individual patient or family. A potential work around is that CMS allows the hospice program to add a small number of supplemental items to the CAHPS survey. With a Data Use Agreement, individual-level data can potentially be obtained from the hospice program and supplemental items added to the CAHPS Hospice Survey, allowing for PCROs to be collected at a fraction of the cost of independent survey administration.⁵¹

• Medicare claims: Medicare claims data potentially can be used to create outcome measures that are indicative of potentially burdensome treatment or burdensome healthcare transitions.^52,53 For example, healthcare transitions in the last 3 days of life among all decedents have been validated against bereaved family member perceptions of quality of care.⁵⁴ Other measures of burdensome treatment are based on expert opinion. For example, the use of invasive mechanical ventilation in a hospitalized person with AD/ADRD does not improve survival and often results in functional decline if the hospitalization is survived. An important limitation of these measures is that these administrative data do not have information on preference for care. With the exception of futility, a person or his/her proxy decision maker may request invasive mechanical ventilation or other treatments that clinicians or investigators perceive as burdensome.⁵⁵

• EHR: Some clinical EHRs now provide a platform into which brief PCROs can be embedded, and many have system-wide embedded PCROs, such as depression screening tools. In addition, EHRs permit clinicians or practice groups to customize clinical encounter templates, and these pathways have the potential to facilitate real-world clinical data capture of brief or pragmatic PCROs. As patient portals are expanded, they may also expand the potential for PLWD or their caregivers to self-report key outcomes.

Promote Future Opportunities to Fill PCRO Measurement Gaps in AD/ADRD Research for ePCTs

Investigators who design ePCTs may need additional study preparation or other resources to design and implement novel AD/ADRD clinical outcome measures. For example, an investigator may require stakeholder engagement to design a novel pragmatic PCRO. Existing brief PCROs may need adaptation or language translation to ensure acceptability in culturally diverse populations. Investigators who use existing PCROs will need to create shortened versions to make data collection feasible and reliable. Although these contributions to measurement science may take the form of additional pilot work or secondary aims in an efficacy trial, they will augment the significance and innovation of an intervention made ready for ePCT testing.

Promote Best Practices to Adapt PCROs for Use in AD/ADRD ePCTs

For many PCROs, data capture methods are poorly adapted to ePCTs. Some measurement methods impose respondent burden, and require intensive interpersonal support. Thus, an important future opportunity in AD/ADRD research is development and promotion of best practices for pragmatic data collection.

Emerging research and new technologies can be leveraged to define best practices in measurement science and pragmatic data capture. First, methods used to embed PCROs for data capture in large data sets, such as the MDS, may be replicated for data capture of carefully selected PCROs in EHRs or clinical registries. Second, emerging practices for novel data capture may facilitate AD/ADRD ePCTs. Examples include computer adaptive testing to reduce the item burden for self-report by PLWD, smart phone applications that facilitate PCRO reporting, use of automated interactive voice response telephone calls to collect data from PLWD who do not have internet access, and wearable devices that capture data on activity or function. Third, AD/ADRD investigators may establish new evidence to ensure that a more pragmatic outcome correlates with a PCRO. For example, in preparation for a future ePCT, investigators may need to validate a widely available measure of pain medication administration as strongly correlated with a PCRO measure of the pain experience. These and other innovations in data capture for outcome measures will be an essential component of expanding AD/ADRD ePCTs, while ensuring these trials prioritize what matters to PLWD and their families.

CONCLUSION

Continuing the case scenario, investigators design a pragmatic multisite clinical trial of their promising transitional care intervention for persons with AD/ADRD. Working closely with hospitals that will be trial sites, they seek to embed brief measures of caregiver burden and dementia-specific quality of life into the hospitals’ routine telephone follow-up for patients at high risk of readmission. During the implementation evaluation of their trial, investigators plan evaluation of the response rate and psychometric properties of outcome measures collected in this novel way. Hospital leadership endorses this workflow, and plans to sustain the practice based on evidence from the pragmatic trial.

Use of person- and caregiver-centered outcomes in AD/ADRD ePCTs is critically important. Working together with other Cores and Teams within the IMPACT Collaboratory, the PCRO Core will support investigators to select from existing outcome tools, and to innovate in methods for measurement and data capture. Advised by members of the PCRO Core, investigators can promote the development of new brief PCROs, or pragmatic adaptation of existing PCROs. The PCRO Core will also advise on novel approaches to data capture, including methods to embed PCROs in EHRs or use of technology to facilitate self-report. Individual investigators generally lack the leverage to embed AD/ADRD outcome measurement tools, particularly PCROs, in existing large-scale data sources, but the PCRO Core will analyze and publicize those that do exist and support their use in ePCTs. Data collection and database management and analysis are costly and time-consuming, and the burdens on health systems and clinicians are real. In the future, the work of the IMPACT Collaboratory may galvanize broader embedded use of outcomes that matter to PLWD and their care partners in large health systems.