Evaluating the Pragmatic Characteristics of Advance Care Planning Outcome Measures in Dementia Clinical Trials: A Scoping Review

Abstract

Background:

Advance care planning (ACP) is a process that involves discussing a person’s goals, values, and preferences; it is particularly important for persons living with dementia (PLWD) given that dementia is incurable and progressive. To ensure results that will impact real-world practices, ACP outcome measures must be psychometrically strong, meaningful to key partners, and pragmatic to collect. Therefore, we conducted a scoping review of outcome measures utilized in ACP randomized controlled clinical trials (RCTs) enrolling PLWD or their care partners and evaluated their pragmatic characteristics.

Methods:

We searched MEDLINE/PubMed, EMBASE, CINAHL, PsycINFO, and Web of Science for peer-reviewed ACP RCTs enrolling PLWD or their care partners from 2011–2021. We abstracted characteristics of primary and secondary outcome measures, including pragmatic characteristics using an adapted Psychometric and Pragmatic Evidence Rating Scale and ACP outcome domains using the standardized ACP Outcome Framework (i.e. process, action, healthcare, or quality of care).

Results:

We included 21 ACP RCTs. Trials included 103 outcome measures (39 primary and 64 secondary), of which 11% measured process, 14% measured action, 49% measured healthcare, and 26% measured quality of care. Twenty-four (23%) outcome measures were highly pragmatic, the majority of which (67%) reflected healthcare outcome measures. Sixty-one (59%) outcomes were assessed as highly relevant to PLWD or their care partners. Only 20% (n=21) of outcome measures were embedded into clinical practice. Most (62%) RCTs were conducted in nursing homes, and 33% were focused PLWD with advanced stage disease.

Conclusions

In RCTs testing ACP interventions to support PLWD, only 23% of outcome measures were highly pragmatic, and most of these measured healthcare utilization. Outcome assessments were rarely integrated into the EHR during routine clinical care. New outcome measures that address the lived experience of PLWD and their care partners plus have high pragmatic characteristics are needed for embedded pragmatic clinical trials.

Introduction:

Alzheimer’s disease and related dementias (AD/ADRD) are progressive and incurable and are the seventh leading cause of death in the United States.¹ Persons living with dementia (PLWD) suffer from progressive decline in cognition and function, resulting in compromised ability to make decisions for themselves and express their goals, values, and preferences for their healthcare.^2–6 As such, it is important for PLWD to discuss their healthcare goals and values and future care preferences as their disease progresses.⁷ For PLWD, this is best done early in the illness trajectory when they can express what matters most to them, personal definitions of quality of life, and treatment preferences, commonly described as advance care planning (ACP).^8,9 As dementia progresses and PLWD lose decisional capacity and communication ability, their care partners must take a large role in helping make medical decisions for them.^9,10

Research examining ACP interventions have revealed significant variability in the use of standardized vs non-standardized outcome measures, making evaluation of ACP effectiveness challenging.^7,11–20 Most recently, a scoping review of ACP randomized controlled trials (RCTs) mapped out outcome measures using the standardized ACP Outcome Framework²¹; which categorizes outcome measures as either process (e.g., readiness), action (e.g., communication), healthcare (e.g. healthcare utilization), or quality of care (e.g., satisfaction). This review showed that most high-quality ACP intervention trials had a positive impact on process and action outcome measures with mixed results on quality of care (such as goal concordance) and healthcare (such as hospitalization) outcome measures.¹⁹ Since then, there has been a call for both the use of pragmatic and standardized outcome measures across ACP studies.^22,23

Given that the ultimate goal of ACP is to support patient-centered care along with shared and surrogate decision-making, prioritization in the use of patient and caregiver-reported outcome measures have been recommended to better account for the interpersonal, social, and cultural variability that will impact ACP efficacy.²⁰ In addition, the insufficient emphasis on pragmatic strategies along with the failure to use patient and caregiver-reported outcome measures that are pragmatic in nature has been attributed to the slow and unreliable translation of ACP into clinical practice.^7,24,25 Key criteria for pragmatic outcome measures were described by Glasgow and Riley²⁶ as those that are important to key stakeholders (which may be defined as patients or care partners, clinicians, leaders, and/or policymakers), actionable, sensitive to change, and have a low burden of data collection when used in routine clinical care. Many outcome measures utilized in research are lengthy and burdensome to clinicians and patients for data collection; thus limiting their ability to be adopted and collected in real-world settings.^27,28 The main goal of pragmatic approaches is to bridge the gap between research and practice by generating generalizable results that are relevant to key stakeholders as well as rigorous in nature with a focus on patient-centered outcomes.²⁹ For ACP interventions to impact clinical care, integration of the intervention into routine clinical practice is essential, but so is the integration of relevant outcome measures into the EHR.^29–32 Furthermore, ACP intervention may also need to be designed to occur in the home and community settings where PLWD spend most of their time. These settings may require site-specific pragmatic data collection strategies such as the use of electronic collection devices (e.g. smartphones/tablets) or specialized software (e.g. OpenClinica) along with community partnerships.

Although there is common agreement on the importance of using pragmatic outcome measures, there still remains some ambiguity and uncertainty surrounding what constitutes a pragmatic outcome measure.^33,34 Stanick et al. developed a pragmatic rating scale called “Psychometric and Pragmatic Evidence Rating Scale (PAPERS)” to help researchers identify the degree to which an outcome is pragmatic (i.e. can be feasibly implemented into a real-world setting) or explanatory (i.e. more suitable for use in efficacy research settings).³⁵ The PAPERS scale, which has four categories and eleven properties of pragmatic measurement (e.g. easy to complete, appropriate, etc.), can help researchers evaluate the pragmatic characteristics of outcome measures and guide them in selecting measures that are more likely to be implemented successfully.

To address the broader goals of the NIA’s IMbedded Pragmatic Alzheimer’s Disease and AD-Related Dementias Clinical Trials (IMPACT) Collaboratory, we recognized a need for best practices in pragmatic outcome measurement for embedded pragmatic trials enrolling PLWD or their care partners.²⁹ Since ACP interventions are common and clinically significant for this population, we conducted a scoping review of ACP RCTs enrolling PLWD or their care partners in order to evaluate the pragmatic characteristics of primary and secondary outcome measures utilized in those studies. The purpose of this report is to 1) summarize the results and implications of this scoping review and 2) provide recommendations for future research and practice.

Methods:

Study Design

We conducted a scoping review using the standardized guidelines introduced by Arksey and O’Malley and refined in the Preferred Reporting Items for Systematic Review and Meta-analysis – Scoping Review (PRISMA-ScR) framework.³⁶

Search Strategies

With a health sciences librarian, we developed a search strategy across MEDLINE/PubMed, EMBASE, Web of Science, PsycINFO, and CINAHL. (Supplementary Appendix S1)

Selection of Studies

We restricted publications to reports of the results of RCTs testing interventions meeting the published consensus definition for ACP, which defines ACP as “a process that supports adults at any age and stage of health in understanding and sharing their personal values, life goals, and preferences regarding future medicare”.⁹ We included RCTs that enrolled adults 18 years of age and older with any cause or stage of progressive cognitive impairment from mild cognitive impairment to advanced AD/ADRD, or their proxy decision-makers published in English between January 1^st, 2011 to December 31^st, 2021. We excluded pediatric populations, pilot studies, non-randomized studies, non-randomized pre-posttest design studies, studies whose primary or secondary outcomes were not related to ACP, subgroup/secondarily analysis unrelated to intervention efficacy, multicomponent interventions in which ACP was not the primary component, intervention protocol papers, literature reviews, conference abstracts, guidelines or reports, and gray literature. One Author (M.K.) reviewed titles and removed duplicates. Two Authors (J.G. and L.H.) independently assessed the titles and abstracts of each article to determine if the study met eligibility criteria, then in those who were felt to meet eligibility, a full manuscript review occurred with both authors agreeing on the final list of included articles; any discrepancies were resolved by consensus.

Data Extraction and Analysis

We used a data extraction tool to record: article characteristics (author, journal, and year published), population, stage of cognitive impairment (early stage defined as either mild cognitive impairment or mild AD/ADRD, moderate stage, advanced stage, mixed, or did not specify), setting, intervention, and measures used for primary and secondary outcomes as defined by the authors. We categorized the outcome measures into domains of the ACP Outcome Framework: process (e.g., readiness), actions (e.g., documentation), healthcare (e.g., hospitalizations), or quality of care (e.g., satisfaction).²¹ We also noted whether outcome measures were reported by the patient, care partner, or “other” respondents such as healthcare providers/staff. For each outcome measure, we assessed: the number of items/length of the measure, completion time, literacy, cost/copyright, data capture burden, relevancy to key stakeholders, and overall pragmatic characteristics. We defined and categorized data capture burden as either low (simple to score, simple to interpret, and easy to embed and extract from EHR), moderate (requires moderate effort to score and interpret and may be challenging to embed and extract from the EHR), or high (a high burden to interpret, not routinely collected data in EHR and would be hard to embed in EHR). The Psychometric and Pragmatic Evidence Rating Scale (PAPERS) was used but adapted to categorize pragmatic characteristics of outcome measures as either high (having several high pragmatic characteristics), moderate (having several moderate pragmatic characteristics), or low (having several low pragmatic characteristics) across 14 metrics.³⁵(Supplementary Appendix S1). Relevance to stakeholders, defined as PLWD or their care partner for this project, was rated using the Readiness Assessment for Pragmatic Trials (RAPT) model, which is a model used to assess an intervention’s readiness to proceed to a pragmatic clinical trial.³⁷ Relevance to stakeholders was scored as either low (stakeholders are unlikely to believe that the outcome is useful), medium (some stakeholders are likely to believe the outcomes are useful), or high (most stakeholders are likely to believe the outcomes are useful).²⁹

Results

Article Selection

Figure 1 outlines the search results and article inclusion; 64% were published in or after 2018. (Supplementary Appendix S1)The search identified 608 articles, which included 256 duplicate articles leaving 352 articles for screening of titles and abstracts for relevance to the study aim. Screening of titles and abstracts resulted in the exclusion of 310 articles, leaving 42 relevant articles for full-length review, of which 25 met eligibility criteria. These 25 publications addressed 21 distinct RCTs, with 4 additional articles included since they cited additional effectiveness outcome measures.

Figure 1: PRISMA Flowchart of study selection.

Legend: ACP=Advance Care Planning, RCT= Randomized controlled Trial, PRISMA=Preferred Reporting items for Systematic Review and Meta-Analyses Diagram, CP=Care Partner

Setting, Population, and Intervention Modalities

Of the 21 RCTs reviewed (Supplementary Appendix S1), all but two^38,39 were efficacy or effectiveness trials, 15 (71%) used a cluster randomized controlled design, and 7 (33%) were done within the United States. 14 (67%) were implemented in English only with only one study being implemented across 2 or more languages. The majority were conducted in the nursing home setting (N=13, 62%). The remaining trials were done in the home (n=3, 14%), hospital (n=1, 5%), and outpatient/community settings (n=4, 19%). Among PLWD, 7 (33%) of the studies focused on those with advanced stage AD/ADRD, 1 (5%) on moderate stage, 3 (14%) on early stage, 9 (43%) on mixed stages, and 1 (5%) did not specify. Ten (47%) RCTs either did not provide race or ethnicity characteristics of participants, or only provided whether they were white vs other; only 3 (14%) trials enrolled ≥ 20% racial/ethnic minority groups. Interventional modalities included: facilitated discussions (52%), video (14%) or print (14%) decision aids, and educational programs (38%). Most educational programs were workshops with 28% targeted towards only surrogates/care partners. Almost half of the interventions were delivered by the research team (42%); one also involved a psychologist and another involved a nurse ambassador.

Overview of Outcome Measures:

Overall, in this scoping review, we found 103 outcome measures (39 primary and 64 secondary) across 21 RCTs enrolling PLWD or their care partners, of which 11 (11%) measured process, 14 (14%) measured action, 51 (49%) measured healthcare, and 27 (26%) measured quality of care outcome measures. Only 20% (n=21) of outcome measures were embedded into clinical practice.

Pragmatic Characteristics of Primary Outcome Measures

The 21 RCTs reported 39 primary outcome measures; some measures were utilized in more than one study. (Supplementary Appendix S1)There were 5 (13%) process, 4 (10%) action, 20 (51%) healthcare, and 10 (26%) quality of care primary outcome measures. Many measures required multiple response items; the average number of items for primary outcome measures was 9.5 (with a large Standard Deviation of 6.5). Measure completion time and health literacy level of primary outcome measures were rarely reported, making it difficult to judge these aspects of pragmatic measurement.

Of the outcome measures that had documentation regarding whether there was a copyright, 9 had a copyright to use with only one having a cost associated with its use. 75% (n=24/32) of outcomes were validated (if validation was appropriate, for example, hospitalization rates and hospital length of stay are not appropriate for validation). Regarding data capture burden, overall, 8 (20%) were considered low data capture burden. Finally, using the RAPT scoring for relevance to stakeholders (defined as PLWD or their care partners), overall, 27 (69%) primary outcome measures were rated as highly relevant. In evaluating overall pragmatic characteristics, only 9 (23%) primary outcome measures were highly pragmatic in nature; the majority of these measured healthcare use (n=7). (Table 1) An additional 9 (23%) primary outcome measures were rated moderately pragmatic in nature (Table 2).

Table 1.

Outcome Measures with High Pragmatic Characteristics

Primary Outcomes Measures with High Pragmatic Characteristics (n=9)
Outcome	ACP Domain	Key Pragmatic Characteristics	Reference
ACP conversation & documentation	Action	Variable, potentially highly pragmatic if a standardized EHR embedded tool is utilized (e.g., ACPWise tool); low data extract burden, can be completed with ease, no associated cost, and <50 items.	Tilburgs (2020), Gabbard (2021), Sævareid (2019), Sampson (2011)
Social engagement and withdrawal	Healthcare	High, since routinely measured and collected in Minimum Data Set, low data capture burden, and only 6 items.	Hilgeman (2014)
Burdensome Treatment	Healthcare	High, since using existing EHR data.	Moyo (2021)
Depression-Patient Health Questionnaire (PHQ-9) scale	Healthcare	High, since routinely collected and embedded in most EHRs along with only 9 items.	Reinhardt (2014)
ED visits and hospitalizations	Healthcare	High, since using existing EHR data.	Martin (2019)
Hospital-Length of stay	Healthcare	High, since using existing EHR data.	Lamppu (2021)
Hospital transfer	Healthcare	High, since using existing EHR data.	Moyo (2021) Loomer (2021) Mitchell (2020)
Late transitions	Healthcare	High, since using existing EHR data.	Moyo (2021)
Satisfaction with care	Quality of Care	High, since simple to score & interpret, commonly collected by healthcare systems.	Reinhardt (2014)
Secondary Outcome Measures with High Pragmatic Characteristics (n=15)
Documentation of Surrogate Decision Maker	Action	High, since routinely collected and embedded in most EHRs.	Gabbard (2021) Overbeek (2018) Saevareid (2019)
Documentation of Preferences for Life Sustaining treatments	Action	Variable, potentially highly pragmatic if a standardized EHR embedded tool is utilized	Reinhardt (2014) Sævareid (2019) Loizeau (2019)
Documentation of Preferences for Comfort	Action	Variable, potentially highly pragmatic if a standardized EHR embedded tool is utilized	Mitchell (2018)
ACP conversation & documentation	Action	Variable, potentially highly pragmatic if a standardized EHR embedded tool is utilized	Mitchell (2018) Saevareid (2019)
Use of ACP billing codes	Action	High, since using existing EHR data.	Gabbard (2021)
% of patients who reported having feeding tube discussions	Action	Variable, potentially highly pragmatic if a standardized EHR embedded tool is utilized	Hanson (2011)
Rates of burdensome treatment	Healthcare	High, since using existing EHR data.	Mitchell (2018,2020) Loomer (2021) Saevareid (2019)
Rates of Hospital Transfers	Healthcare	High, since using existing EHR data.	Mitchell (2020) Hanson (2017)
Rates of Hospice Enrollment	Healthcare	Variable, embedded in some EHR data, though some only have referral rates.	Mitchell (2020) Hanson (2017)
Rates of Hospitalization and ED visits	Healthcare	High, since using existing EHR data.	Loomer (2021) Martin (2019) Brazil (2018) Lamppu (2021) Saevareid (2019)
Hospital-Length of stay	Healthcare	High, since using existing EHR data.	Martin (2019)
DNR Orders	Healthcare	High, since using existing EHR data.	Brazil (2018)
Feeding tube placement	Healthcare	High, since routinely embedded in most EHRs.	Hanson (2011)
Medical Care Use (hospital care, procedures, medical interventions)	Healthcare	High, since using existing EHR data.	Overbeek (2018)
Mortality Rate (in-hospital and nursing home)	Healthcare	High, since using existing EHR data.	Martin (2019)

Table 2.

Outcome Measures with Moderate Pragmatic Characteristics

Primary Outcomes Measures with Moderate Pragmatic Characteristics (n=9)
Outcome	ACP Domain	Key Pragmatic Characteristics	Reference
Intention to write a care plan	Process	3 item, moderate data capture burden, has potential to be embedded into EHR	Bonner et al (2021)
Written care plan about CPR, MV, and TF	Action	Dichotomous (yes/no) 3-item questionnaire, has potential to be embedded into EHR	Bonner et al (2021)
Quality of life–visual analog scale (EQ-VAS)	Healthcare	Low completion time (<5 minutes), easy to score/interpret, has potential to be embedded into EHR	Husebø et al (2019)
Cornell Scale for Depression in Dementia	Healthcare	19 items, scored 0–2 scale, moderate data capture burden, has potential to be embedded into EHR	Hilgeman et al (2014)
Do-Not-Rehospitalize (DNH) Directives	Healthcare	Single item, has potential to be embedded if a standardized EHR tool is utilized.	Mitchell et al (2018)
Clinical Global Impression of Change (CGIC)	Quality of Care	Single item, easy to score/interpret, has potential to be embedded into EHR	Aasmul et al (2018)
Healthcare provider-surrogate goal concordance	Quality of Care	Single item, moderate data capture burden, may be challenging to embed and extract from EHR	Hanson et al (2017)
Quality of Communication (QOC) scores	Quality of Care	13 item, scored 0–10 scale, moderate data capture burden, has potential to be embedded into EHR	Hanson et al (2017) Gabbard et al (2021)
Treatment consistent with wishes (ACP problem score)	Quality of Care	3 item, moderate data capture burden, has potential to be embedded into EHR	Hanson et al (2017)
Secondary Outcomes Measures with Moderate Pragmatic Characteristics (n=8)
ACP Forms (AD/Living will, MOLST/POLST) Completion Rates	Action	Moderate data capture burden, though most EHRs have embedded ways to extract upload rates	Gabbard et al (2021) Overbeek et al (2018) Mitchell et al (2018)
Healthcare Cost	Healthcare	Routinely collected in the EHR, no survey delivery burden, but can be labor intensive to analysis data	Lamppu et al (2021)
Mobilization-Observation-Behavior-Intensity-Dementia–2 pain scale	Healthcare	10 item scale, easy to score/interrupt but requires proxy assessment which would make challenging to embed	Husebø et al (2019)
Pain and distress	Healthcare	2 items, low data capture burden, has potential to be embedded into EHR	Sampson et al (2011)
Physical self-maintenance scale for activities of daily living (ADL)	Healthcare	6 items, easy to score/interpret, has potential to be embedded into EHR	Husebø et al (2019)
Place of death	Healthcare	Single item, often EHR data if dies within hospital/network but could be challenging if other locations	Brazil et al (2018)
Patient Satisfaction Questionnaire (PSQ-18)	Quality of Care	18 item, low completion time (<5 minutes), moderate data capture burden, has potential to be embedded into EHR	Overbeek et al (2018),
Clinical Global Impression of Change (CGIC)	Quality of Care	Single item, easy to score/interpret, has potential to be embedded into EHR	Husebø et al (2019)

Pragmatic Characteristics of Secondary Outcome Measures

The 21 RCTs reported 64 secondary outcome measures; some measures were utilized in more than one study. (Supplementary Appendix S1)There were 6 (9%) process, 10 (16%) action, 31 (48%) healthcare, and 17 (26%) quality of care secondary outcome measures. Many measures required multiple response items; the average number of items for secondary outcome measures was 12.6 (with a large Standard Deviation of 14.7). Similarly, as seen with primary outcome measures, measure completion time and health literacy level of secondary outcome measures were rarely reported.

Of the secondary outcome measures that had documentation regarding whether there was a copyright, 11 had a copyright to use with only two having a cost associated with its use. The majority (67%) of secondary outcome measures were validated (if validation was appropriate). Regarding data capture burden, overall, 12 (19%) were a low capture burden. Overall, 35 (54%) were rated as highly relevant to PLWD or their care partners, which was slightly less than seen with primary outcome measures. In evaluating overall pragmatic characteristics, only 15 (23%) secondary outcome measures were highly pragmatic in nature if standardized embedded EHR tools were utilized; with the majority being measures of healthcare use (n=9). (Table 1). An additional 8 (12%) secondary outcome measures were rated moderately pragmatic in nature (Table 2).

Discussion

Pragmatic clinical trials are essential to accelerate the translation of ADRD interventions into real-world clinical practice.⁴⁰ In order to generate generalizable results, embedded pragmatic clinical trials commonly take place where patients receive their routine care (e.g. hospitals, clinics, nursing homes, etc.) or in the home and community setting with the goal of embedding directly into clinical workflow without the need for specialized trained research staff for data collection, which may be more challenging than others depending on the setting, intervention complexity, and targeted population.^37,41 Therefore, to conduct these trials, researchers require the use of pragmatic outcome measures that are relevant to PLWD, their care partners, providers, and other decision-makers. Our scoping review showed that a minority (23%) of primary and secondary outcome measures utilized in ACP RCTs enrolling PLWD or their care partners were highly pragmatic; yet, 59% of these outcome measures were rated as highly relevant to PLWD or their care partners. Among highly pragmatic outcome measures, the majority (67%) were healthcare utilization measures. This is likely due to the fact that healthcare utilization is commonly measured using data in existing administrative sources or EHRs, while other outcome measures often require direct patient or care partner reports that are rarely routinely collected or embedded in EHRs or routine clinical care. Healthcare utilization outcome measures are highly relevant to some stakeholders, such as healthcare administrators, yet relevance to PLWD may vary depending on the patient’s healthcare goals.⁴² Further, healthcare utilization is affected by a multitude of factors unrelated to ACP such as healthcare access, support networks, and illness characteristics along with systemic and structural inequities and injustices.⁷

Capturing data on important aspects of the lived experience of PLWD or their care partners may require embedding brief patient and caregiver-relevant outcomes (PCROs) measures or other novel measures in the EHR or other electronic platforms if possible.^29,37,41,43 This gap between relevance to PLWD or their care partners and inadequate integration of PCROs is one of the major challenges of using EHR systems to design and conduct ADRD pragmatic clinical trials.³⁰ Very few PCROs are brief and efficiently collected.⁴⁴ This was highlighted in our results with only 19% of outcome measures having a low data capture burden. Some brief PCROs have been broadly incorporated into clinical practice (e.g., PHQ-9), demonstrating the potential for expansion of this approach.⁴⁵ Importantly, in our review, we also found six action outcome measures that were highly pragmatic if an embedded EHR tool was utilized. These included documentation surrounding goals, preferences, and surrogate decision-makers. The use of embedded EHR tools is a more pragmatic and patient-centered alternative to retrospective chart review^7,46,47 In addition, we found 17 moderately pragmatic outcome measures; thus, highlighting their potential to be adapted, integrated, and embedded into existing EHRs for use in future pragmatic trials.

To adapt PCROs for use in routine clinical care, some preparatory work may be required. To ensure PCROs are relevant to diverse populations, some of these measures may require cultural and literacy tailoring along with linguistic translation.^29,48 In this review, we found that only one ACP RCT adapted their intervention along with outcome assessment for people with diverse primary languages. Pragmatic trials need to be designed to mimic the broad accessibility of clinical services in diverse populations, and if future embedded pragmatic trials use brief PCROs, linguistic accessibility for outcome data capture should be considered a priority. This approach has already been successful in PCROs such as the PHQ-9 and Brief Interview for Mental Status (BIMS) nationwide in the nursing home Minimum Dataset 3.0 (MDS).⁴⁹ Also, to make sure that the capture of PCROs is equitable and inclusive, addressing the digital divide, multiple modes of delivery must be considered (e.g. providing a device, voice response systems, in-person completion in clinic, etc.).⁴⁸ Additionally, ACP is one of the quality measures included in the Dementia Management Quality Measurement set⁵⁰ and has been included in many value-based payment schemes; thus healthcare systems are incentivized to invest in informatics support to measure the occurrence of ACP and to reward higher-performing providers.⁵¹ Furthermore, healthcare systems along with EHR vendors need to invest and support in embedding PCROs into EHRs and clinical workflows.³⁰ An example of this being done successfully is the integration of several short PROMIS^® measures including computer adaptive tests (CATS) and Item response theory (IRT) approaches into EHR Vendors like Cerner and EPIC.^52,53 Experts have published recommendations on how to successfully accomplish this using a staged approach but implementation strategies will need to be considered to overcome common implementation barriers (e.g. time and resource requirements for training and administration, staffing, technical support, etc.).^31,54–57 Some strategies recommended include developing standard operating procedures around the collection of PCROs, clinical triggers, provider champions, limiting the number of questions asked, user-friendly collection software, and point-of-care PCRO results.^57,58

Our results have important implications for pragmatic clinical trial research. In the short term, our results provide examples of highly pragmatic, highly relevant outcome measures that are ready for use in trials. Outcome measures that are highly relevant but moderately pragmatic are promising options for adaptation and innovation in data capture methods, thus expanding the range of potential outcome measures for evaluating ACP interventions. In the future to ensure that outcomes highly relevant to PLWD and their care partners can be used in pragmatic clinical trials, investigators need to expand the array of potential outcome measures and involve end-users, especially participants from underserved groups, in the design and delivery of PCROs to mitigate barriers to implementation.^33,48 The IMPACT iLibrary provides information on potentially pragmatic outcome measures in a searchable format along with guidance documents on outcome priorities for PLWD and their care partners.⁵⁹ As seen with the PAPERS criteria, outcome measures traditionally are not defined dichotomously as either being pragmatic or not but rather having pragmatic characteristics which may be more important than others depending on the stakeholders involved (i.e. patients, providers, policymakers, etc.) and what the researcher is trying to measure. In addition, it is often nearly impossible for an outcome measure to have all the key pragmatic characteristics. Thus, in Figure 2, we categorized core pragmatic characteristics, ideal pragmatic characteristics, and then hallmark pragmatic characteristics of outcome measures that hopefully can help researchers conceptualize this spectrum of pragmatism when accessing and selecting outcome measures. Pragmatic characteristics listed in Figure 2 are based on prior consensus literature and experts in the field and are features that will enhance a measure’s adoption and use in real-world clinical practice.^{24,28,35,60,61} We recommend that when researchers are choosing trial primary outcome measures that they try to include measures that have at least the core pragmatic characteristics; though not all of the usual psychometric criteria may be relevant depending on what the researcher is trying to measure. Furthermore, some level of consistency or standardization is needed in the use of outcome measures to enable the ability to compare across studies to more accurately measure the impact of ACP interventions in PLWD or their care partners; though it is important to consider the impact of the intervention setting when interpreting study findings as results can vary in different settings.^62–64

Figure 2: Pragmatic Outcome Measure Characteristics.

Novel outcome measures also need to be developed using patient, care partner, or clinician reports of patient experiences, provided these measures are brief, meaningful to key stakeholders (i.e., providers, patients, leadership), and could be readily embedded into EHRs and routine clinical practice. One example is the two new PCROs developed by the American Academy of Hospice and Palliative Medicine which measures how much a patient feels heard and understood and whether a patient received desired help for their pain.⁶⁵ These measures are brief, highly relevant, psychometrically strong, and could easily be embedded into EHRs.⁶⁵ The use of proxies to help measure patient-reported outcome measures also may hold some promise for PLWD if viewed as complementary rather than substitutive.^66,67 Emerging data capture methods such as natural language processing (NLP) to measure ACP documentation, patient portals to measure electronic PCROs along with wearable devices, smartphones, and artificial intelligence to help measure prognosis and patient-related parameters may also yield future highly pragmatic outcome measures.^51,68–72 In ADRD pragmatic trials conducted in the home or community setting, a multi-level approach will likely need to be considered for the collection of PCROs in the absence of healthcare records or administrative data collection methods.⁷³

Strengths and Limitations

To our knowledge, this is the first literature review to examine the pragmatic nature of outcome measures utilized in ACP RCTs enrolling PLWD or their care partners, and it has several strengths. Our methodology was robust, following published research guidelines for scoping reviews.^36,74 We used standardized criteria when defining ACP outcome domains, relevancy to stakeholders, and pragmatic characteristics of outcome measures using the PAPERS criteria; though some items did require some subjectivity.^21,35,37 In addition, we specifically focused on the feasibility of these outcome measures for integration into the EHR. Nevertheless, our search strategy may not have captured all relevant RCTs, and we did not account for research that was in progress, feasibility trials, non-randomized pre-posttest trials, nor grey literature. We also did not focus on the psychometric characteristics of outcome measures. Furthermore, because our focus was on outcome measures rather than the interventions themselves, we did not exclude studies based on quality per the PRISMA guidelines.^36,75

Conclusions

In this scoping review of ACP RCTs enrolling PLWD or their care partners, we found that the majority of outcome measures utilized were not highly pragmatic in nature and had high data capture burden; but over half were likely highly relevant to PLWD or their care partners. The use of PCROs in ACP dementia-related RCTs is critically important in being able to accurately evaluate the effectiveness of ACP and promote patient-centered care. Future research is needed to develop novel brief and highly relevant outcome measures that can be readily embedded into clinical practice and routine EHRs. In addition, to address health equity, these measures need to be validated and accessible across different languages, literacy levels, and cultures. Emerging data-capturing methods (e.g., CAT and IRT approaches) likely will also start playing a larger role in better measuring patient and care partner experiences.

Key Points.

• Embedded pragmatic clinical trials for persons living with dementia (PLWD) depend on the use of outcome measures that are relevant to PLWD, their care partners, and other key partners and are actionable, sensitive to change, and have a low burden for data collection in routine clinical care.

• Few outcome measures selected for advance care planning (ACP) randomized controlled trials that focused on PLWD or their care partners are highly pragmatic; though many are highly relevant to PLWD and their care partners, current outcome measures may not meet criteria for pragmatic trials.

• To fill the critical gap in pragmatic outcome measures that matter to PLWD and their care partners, new patient and care partner experience outcome measures need to be embedded into clinical practice that have high pragmatic characteristics.

Why Does this Paper Matter?

This scoping review highlights the importance of developing and embedding pragmatic outcome measures into clinical practice that are relevant to PLWD and their care partners to promote patient-centered care.