Using Healthcare Data in Embedded Pragmatic Clinical Trials among People Living with Dementia and the Caregivers: State-of-the-Art

Abstract

Embedded pragmatic trials (ePCTs) are embedded in health care systems as well as their data environments. For people living with dementia (PLWD), settings of care can be different from the general population and involve additional people whose information is also important. ePCT designs have the opportunity to leverage data that becomes available through the normal delivery of care which may be particularly valuable in Alzheimer’s Disease and Alzheimer’s Disease-Related Dementia given the complexity of case identification and diversity of settings of care. Grounded in the objectives of the Data and Technical Core of the newly established National Institute on Aging (NIA) Imbedded Pragmatic Alzheimer’s Disease and AD-Related Dementias Clinical Trials Collaboratory (IMPACT Collaboratory), this paper summarizes the state-of-the-art in using existing data sources (e.g. Medicare Claims, electronic health records) in AD/ADRD ePCTs and approaches to integrating them in real world settings.

Introduction

Health-system embedded pragmatic clinical trials (ePCTs) that address the needs of people living with dementia (PLWD) and their caregivers are critical for developing and disseminating evidence-based, non-pharmacological interventions. These trials create the opportunity to design interventions that will work in real-world patients and settings, but they require investigators to embrace new methods and partnerships as described by Mitchell et al.¹ A key element of the ePCT approach is to leverage data derived from and integrated with the healthcare system workflow into the trial’s design, conduct, and dissemination. This approach allows cost-effective identification of participants and outcome data ascertainment. With the rise of electronic health records (EHRs) and focused attention on ePCTs stimulated by the National Institute of Health (NIH) Collaboratory, capabilities for using healthcare-generated data have been advancing. Yet unique issues facing PLWD necessitate innovative strategies in using healthcare-generated data across the multiple healthcare settings targeted in Alzheimer’s Disease and Related Dementias (AD/ADRD) ePCTs.

The unique challenges of using existing data sources to conduct ePCTs in AD/ADRD fall into several categories: 1) AD/ADRD are under-diagnosed² and stigmatized diseases,^3–5 2) caregivers often need to be identified,⁶ 3) data must be accessed from settings outside the traditional acute care medical system (e.g. primary care, nursing homes and assisted living), 4) patient- and caregiver-reported outcomes must be ascertained, and 5) measures are needed that span multiple settings (e.g. care transitions). Despite these challenges, opportunities exist to strengthen our ability to identify PLWD and their caregivers and to measure outcomes by leveraging data sources available through administrative data or electronic health records. These data are useful at multiple points in the ePCT process, including the design phase, conduct of the pilot and full trial, and subsequent dissemination. But careful consideration of the “fitness for use” of a particular data strategy is critical at all stages. This report will provide an overview of using healthcare data in ePCTs on which the objectives of the Data and Technical Core of the newly established National Institute on Aging (NIA) Imbedded Pragmatic Alzheimer’s Disease and AD-Related Dementias Clinical Trials Collaboratory (IMPACT Collaboratory) are based and serve as the groundwork our future work in addressing the unique data challenges in ePCTs among PLWD.

Overview of Types of Data and Sources

Clinical trialists are well-versed in data collection strategies for studies that directly recruit individual participants into intervention and control groups. These approaches use validated instruments for assessing participants and their outcomes, and employ research staff to meticulously collect data either in-person, by phone, or through electronic media. The challenges of scaling this traditional approach to data collection can limit a trial’s size and the settings in which it can be conducted. When conducting an ePCT, investigators have an opportunity to reduce costs and burden by using existing data that have been ascertained in the course of usual clinical care (Table 1). These data include “administrative data,” which are generated for billing or regulatory purposes, and EHR data, which include structured elements (e.g. laboratory results, diagnostic codes, medications) and unstructured or text fields (e.g. clinical notes and imaging reports).

Table 1:

Healthcare Generated Data Types, their Content, Examples of Potential Uses and the Sources Employed

Data Type	Information contained in data type	Examples of Possible Use	Example Sources for Data Type
Claims Data	Inpatient, clinic, home health, hospice, medication data used for billing	Identify participants diagnosed with dementia; Measure outcomes (e.g. readmissions, hospital transfers, antipsychotic use)	CMS Virtual Research Data Center (FFS Medicare), NIH Collaboratory Distributed Research Network (Medicare Advantage and commercial)
Enrollment Data	Demographic, geographic, and plan type for enrollees in insurance (Fee-for-service/ Medicare Advantage/ commercial)	Estimate available eligible sample Identify settings for further dissemination	CMS Virtual Research Data Center (FFS Medicare), NIH Collaboratory Distributed Research Network (Medicare Advantage and commercial), OPTUM
Assessment Files	Clinical data for quality reporting & payment in nursing homes and home care	Identify home health agencies or nursing homes with high proportion PLWD	Minimum Dataset (nursing home), OASIS (home health) from CMS or directly from healthcare setting.
Electronic Health Records	Structured data (labs, problem lists), text fields, billing data, patient reported outcomes	Cognitive screens & clinical notes to identify undiagnosed PLWD	Directly from participating healthcare setting, federated data intermediaries (e.g. PCORnet, ACT)
Provider Files	Type, size, location, ownership	Find physician practices serving high ethnic minority populations	PECOS, Provider of Service Files, Medicare Provider Practice & Specialty

Key: VRDC = Virtual Research Data Center; DRN = Distributed Research Network, PECOS = Medicare Provider Enrollment, Chain, & Ownership System; POS= Medicare Place of Service File;

These existing data can be accessed from federal sources, private payers, directly from specific health systems, or in some cases through intermediaries who facilitate collaboration with multiple healthcare systems and payers, such as the Distributed Research Network established NIH HCS Collaboratory (https://rethinkingclinicaltrials.org/nih-collaboratory-drn/). The Distributed Research Network implements a common data model that facilitates use of data from both Medicare Advantage and commercially insured individuals across multiple payers. PCORnet is another distributed research network funded by the Patient Centered Outcomes Research Institute (PCORI) that has implemented a related common data model based on EHR data (www.pcornet.org).

As the age of onset of dementia is most commonly over 65 years, data from the Centers for Medicare and Medicaid Services (CMS) are a particularly valuable asset to ePCTs for PLWD. Medicare claims from CMS include hospital, post-acute care, clinic, hospice and skilled nursing facility billing data. Medicare claims data have the advantage of being complete, as they include all Medicare beneficiaries in the healthcare setting whether the participants complete the trial or not, and are uniform across various healthcare systems. While these data have historically been available only for people enrolled in fee-for-service Medicare, data from Medicare Advantage are increasingly becoming available. In addition to claims, CMS also has “assessment data” from nursing homes (NHs) captured in the Minimum DataSet (MDS), and from home health agencies captured in the Outcome and Assessment Information Set (OASIS). CMS requires regular collection of these data to calculate payments and monitor quality. MDS and OASIS also include clinical information such as cognitive and functional status and behavioral issues. For example, MDS data capture standardized assessment of all residents in over 15,000 NHs in the US. Assessments are administered at minimum quarterly making the MDS a rich source of resident status over time. In addition, although the NH setting lags behind hospitals, 60 percent of NHs also have EHRs.⁷

Uses of Healthcare Generated Data in ePCTs

Healthcare-generated data can be used at multiple stages in the ePCT process, from the design phase, to conduct of the pilot study and full trial, to subsequent dissemination (Figure 1). In the design phase, which is the main focus of the IMPACT Collaboratory, investigators can use existing data to identify potential participants to calculate power and sample size estimates. Existing data can also help in the design phase by enabling the identification and characterization of eligible healthcare settings, including considerations of representation of diverse populations. Aggregated data from healthcare systems are available from EHR data infrastructures; and publicly available data sources such as provider files available through CMS (e.g. PECOS,⁸ Certification and Survey Provider Enhanced Reports (CASPER)⁹); or through websites (e.g. OHSDI Atlas,¹⁰ ACT,¹¹ Nursing HomeCompare (medicare.gov/nursing HomeCompare),¹² LTFocus (ltcfocus.org),¹³ Dartmouth Atlas¹⁴). These aggregated data can also be used to assure balance on key measures between clusters in each trial arm during the randomization process.

Figure 1:

Overview of Uses of Administrative or EHR data in Embedded Pragmatic Clinical Trials

In the process of study execution, both claims and EHRs can be used to identify specific participants, evaluate adherence to protocol, and measure outcomes. Using administrative data to measure longer term outcomes, including utilization and spending, is critical because it can be done long after the trial is complete without the need for direct participant contact. Finally, administrative data are useful to identify new sites for the next phases of implementation.

Identification of PLWD from Healthcare-Generated Data

One of the greatest potential advantages of using healthcare data in AD/ADRD ePCTs is the ability to identify eligible participants without directly assessing individual participant’s cognitive status. Instead, diagnoses in administrative data required for clinician billing can be used to identify a PLWD. Similarly, EHRs contain structured data elements with diagnoses populating problem and medical history lists. These diagnoses can be used to identify participants for ePCTs, with the major caveat that under-diagnosis limits this approach. A recent meta-analysis estimated 60% of AD/ADRD cases are undiagnosed in the community.² Studies evaluating the accuracy of claims diagnoses have shown good performance, but with under-ascertainment of mild disease in particular.^15–19 Moreover, none of the algorithms for the identification of AD/ADRD in Medicare claims has yet been validated using ICD-10 diagnostic coding that began in 2015.

Access to EHRs that can be searched for symptoms, clinician comments, results of annual wellness screening exams, and other data elements presents an opportunity to rely less on confirmed diagnoses from purely administrative sources. The ability to use documented clinical data may be particularly important for identification of racial/ethnic minorities in whom differences in stigma attached to AD/ADRD and beliefs about cognitive loss as a normal part of aging may contribute to lower rates of formal diagnosis.^{4, 20–23} Despite major gaps in consistent assessment of AD/ADRD, significant advancements have been made by changing data collection strategies from reliance on one data source to using combinations of EHR, claims, survey, and other data.

By combining different data types and sources, validated data combinations, called “computable phenotypes” in informatics, can be created for more sensitive and reliable assessments of AD/ADRD status. Barnes et al²⁴ describe eRADAR, a high-performing algorithm (that uses common EHR data to identify patients with undiagnosed dementia. The eMERGE consortium has a public computable phenotype to identify people visits for dementia-related diagnosis or prescriptions for dementia-related drugs.²⁵ In other recent work, McCoy et al applied a validated natural language processing (NLP) tool to examine the association of cognitive symptoms with incident dementia diagnosis using longitudinal EHRs,²⁶ and Beltrami et al²⁷ used NLP to identify early linguistic signs of cognitive decline, not necessarily dementia itself, in a population of older adults.²⁷ In addition, a large database of multimedia interactions and transcripts, DementiaBank,²⁸ is available for the study of communication in dementia patients, and has been used to study natural language processing techniques to classify and analyze the linguistic characteristics of AD patients.^29–31

Despite these exciting developments in use of EHR data for identifying PLWD, there are a number of critical issues left to address. It is imperative that investigators using their own “computable phenotype” to identify PLWD in a healthcare system validate and share their approaches. Investigators who opt to use an existing validated computable phenotype can both shorten development time and provide measures of accuracy for further validation in this growing field. Several online resources are available, including the Phenotype Knowledge Base (PheKB, phekb.org); via PhenX, a curated resource for research-specific definitions; and via the NIH Clinical Data Elements database, among others. Sharing the definitions used in ePCTs via open source, online resources can help continue to improve the quality and transferability of future trials.

Several cautionary issues pertaining to use of EHR data in conducting ePCTs in PLWD are worth noting. It is important for investigators to be aware of poor or uneven data quality that continues to exist in EHRs. Even previously validated definitions and algorithms for identifying AD/ADRD populations and assessing their outcomes must be validated locally in each healthcare system to account for variations across settings and purposes. When data quality issues are discovered, it may be possible to mitigate them by combining with other data, using advanced statistical approaches such as imputation, or by performing sensitivity analyses. In addition, there are tradeoffs to consider when choosing which healthcare generated data source to use. While EHR data are timely, their completeness and accuracy are variable. In addition, algorithms identifying PLWD suffer from potential biases, such as lower accuracy in minority populations and those with lower healthcare access. On the other hand, Medicare claims data may be less timely and have lower sensitivity for early disease, but are simpler to implement, have a reasonable evidence-base in terms of validation, and minimal missing data.

Case studies – METRIcAL & PROVEN Trials

We provide two case studies of ePCTs in nursing homes for patients with advanced dementia that use administrative data and EHR data to highlight the advantages and potential limits of using the pragmatic healthcare data approach. While using similar data sources, the differing aims of each study highlight how the degree of pragmatism that can be achieved varies and the importance of piloting the planned data strategy to assess its fitness for the intended use in the trial.

Music & MEmory: A Pragmatic TRIal for Nursing Home Residents with ALzheimer’s Disease (METRIcAL)

METRIcAL is an ePCT of a personalized music intervention for nursing home residents with dementia. Personalized music is one of several sensory and reminiscence therapies being explored as low-risk alternatives to pharmaceutical approaches in managing behavioral and psychological symptoms of dementia.³² In METRIcAL, nursing home staff identify music a resident preferred when s/he was younger and deliver the music at early signs of agitation. The primary aim of METRIcAL is to determine whether or not personalized music reduces agitation among residents with advanced dementia compared to usual care. The pilot phase of METRIcAL was completed in 2018. The ePCT is currently underway; 81 NHs from four corporations are enrolled, 27 NHs receive the intervention in each study year (2019, 2020, 2021).

PRagmatic trial of Video Education in NHs (PROVEN)

PRagmatic trial Of Video Education in NHs (PROVEN) was an ePCT of a video to assist with advance care planning for nursing home residents with advanced dementia or advanced cardiopulmonary disease.³³ This population was chosen because it is likely to experience unnecessary and non-beneficial care at the end of life, including multiple hospital transfers. The primary outcome of interest was hospital transfers per person days alive. 360 NHs (intervention arm n=119; control arm n=241) within two NH healthcare systems were enrolled in the trial. Early results suggest the videos helped residents and their surrogates think differently about their medical choices and prompted conversations with a provider.³⁴

Data Use and Lessons Learned from METRIcAL and PROVEN Trials

Both ePCTs benefitted from the routine collection of MDS assessments which contain diagnosis and cognitive and physical function for all nursing home residents. MDS was used to identify residents who had been in the nursing home at least 90 of the last 100 days and who had a dementia diagnosis. The PROVEN intervention was delivered as a quality improvement intervention to everyone in the nursing home during the study period, so nursing home staff did not need to be aware of which residents were targeted for outcome analyses. This highly pragmatic approach was not possible in METRIcAL in which a subset of eligible residents were targeted to receive the intervention because of its resource intensive nature, requiring equipment (mp3 players, headphones, etc.) and staff effort in personalizing music selection. An onsite formal process for selecting study targets from potentially eligible residents identified in MDS was necessary in treatment and control nursing homes.

A second important use of healthcare data in both ePCTs was to measure the main outcome, hospital transfers using Medicare claims data for PROVEN and occurrence of agitated behaviors assessed in MDS for METRIcAL. The PROVEN trial approach was straightforward; claims data allowed for complete case ascertainment because participants were all in Medicare and hospitals uniformly submit bills for admission. For METRIcAL, the main outcome of agitated behavior collected through MDS is dependent on recognition and documentation of the behaviors by nursing staff. During the pilot phase of METRIcAL, investigators discovered that agitated behaviors were under-detected in MDS, likely due to staff normalization of those behaviors over time.³⁵ The measurement strategy for the full-scale ePCT was adjusted to include on-site data collection for a randomly selected subset of participants.³⁶

In the design and pilot phase, both trials used what they observed in administrative data about their main outcome measures to address imbalances across sites that could be addressed by altering their randomization protocols. In METRIcAL, based on observed variation in documentation of agitated behaviors across nursing homes and the process for selecting eligible residents, study arms were balanced on behaviors and number of potentially eligible residents prior to randomization. Similarly, in PROVEN, trial arms were purposefully balanced at baseline on their historical rate of hospital transfers (primary outcome) to address the high underlying variation in the rate of hospitalization across nursing homes.

Finally, both PROVEN and the pilot phase of METRIcAL inserted new fields into the EHR to capture implementation adherence. The customized report integrated into the EHR to capture implementation adherence with the ePCT was underused and disliked by front-line providers, for whom the report had no relevance to clinical care.³⁷ When planning an ePCT, researchers should use caution when inserting new elements into a workflow that do not serve a clinical purpose evident to front-line providers.

These two recent ePCTs conducted in nursing homes illustrate how routinely collected administrative data can be leveraged to promote balanced randomized clusters, streamline NH recruitment, facilitate patient selection, and enable an efficient, pragmatic approach to outcome ascertainment. However, the data strategy can introduce new challenges and like many aspects of conducting a clinical trial benefit from pilot testing.

Access, Protection and Sharing of Data

Investigators need to initiate plans for accessing healthcare data early and plan for a lengthy process of gaining approvals and developing partnerships. Obtaining administrative data from federal sources, such as CMS, has a well-defined but lengthy process managed by ResDAC (https://www.resdac.org). Obtaining data directly from health systems is attractive, but healthcare systems may not be familiar with Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule as it pertains to research and may find the regulatory process daunting. Even when willing, it can take months or years to enter into the appropriate arrangements if healthcare systems are unfamiliar with the process.

The use of EHR and other administrative data within ePCT designs, coupled with a focus on PLWD, require researchers to consider their data privacy and sharing options very carefully. Traditional efficacy trials typically obtain informed consent that includes explicit assurances to protect privacy and also authorizes plans for data-sharing covering future use. In fact, the NIH has proposed an expansion of data sharing rules for funded research.³⁸ When ePCTs use data generated through the delivery of healthcare that is in the possession of providers, delivery systems, and payers, there are additional considerations: 1) data obtained with waiver of consent, especially for system-level interventions or cluster randomized trials, precludes specific consent for data sharing; 2) data volume and content include a large number of data points making deidentification of individuals difficult, perhaps impossible;³⁹ and 3) providers, delivery systems, and payers may not agree to participate if data about their organization can be used for unspecified secondary purposes (Simon et al, 2015).³⁹ In many cases, the ability to share individual data may ultimately be limited and researchers should be able to provide a detailed description of the steps they took in obtaining their data that provides a practical guide for researchers looking to do similar research in the future.

Summary

Data available through normal delivery of care to PLWD within healthcare system present tremendous opportunities to strengthen the design and conduct of AD/ADRD ePCTs. However, as this field is both complex and relatively nascent, novel methodologies and approaches must proceed thoughtfully and rigorously. Under the leadership of the Technical Data Core, the IMPACT Collaboratory will help advance our ability to conduct successful ePCTs that can improve care for PLWD by supporting investigators’ efforts to use healthcare-generated data. The Core will help devise approaches to overcoming some of the barriers associated with using data obtained in the course of care by: 1) connecting investigators to validated algorithms for identifying PLWD and contributing to creating them where they do not exist; 2) finding and developing measures for outcomes important to stakeholders, including PLWD, caregivers and health systems; and 3) generating information to help investigators find settings and healthcare system partners whose characteristics and populations served are well aligned with the study’s aims.