Regulatory considerations for developing remote measurement technologies for Alzheimer’s disease research

Abstract

The Remote Assessment of Disease and Relapse – Alzheimer’s Disease (RADAR-AD) consortium evaluated remote measurement technologies (RMTs) for assessing functional status in AD. The consortium engaged with the European Medicines Agency (EMA) to obtain feedback on identification of meaningful functional domains, selection of RMTs and clinical study design to assess the feasibility of using RMTs in AD clinical studies. We summarized the feedback and the lessons learned to guide future projects.

Introduction

Alzheimer’s disease (AD) is a progressive neurodegenerative disorder, characterized by cognitive decline impairing daily function^1,2. Cognitive and functional decline in AD clinical studies is traditionally measured through standard neuropsychological assessments, including questionnaires and interviews with patients and caregivers. The use of remote measurement technologies (RMTs), such as apps and wearables, can provide a broader, more objective, more frequent, or even continuous assessment of function and has the potential to reduce assessment burden on study participants^3,4. An increasing number of clinical research programs are exploring the use of RMTs in clinical studies in other disease areas⁵. Their potential has been recognized by regulators^6,7 and insights into regulatory recommendations may stimulate the use of novel RMTs⁸. The recent qualification of the stride velocity 95th centile as a primary endpoint by the European Medicines Agency (EMA) for ambulatory Duchenne Muscular Dystrophy studies, demonstrates the role that RMT-based assessments can play in drug development and approval^9,10.

Regulatory endorsement of novel assessments is essential for their use in clinical studies. However, this process is long and requires leveraging results from multiple studies^10,11. Public-private partnerships offer excellent opportunities to support this procedure, bringing together pharmaceutical companies, academic and technical experts, along with input from Patient Advisory Boards^12–15.

The project aimed to identify and evaluate RMTs for remote assessment of functional impairment in participants within the complete AD spectrum^16–20.

One of the objectives of the RADAR-AD consortium was to discuss the approach taken to identify RMT-based assessments with regulators, to obtain guidance on how to develop a path for formal qualification for their use in AD studies. The consortium met with the EMA through the Innovation Task Force and Qualification Advice procedures and discussed the Concept of Interest (CoI), the Context of Use (CoU), the RMT selection process, RADAR-AD clinical study design, and the results of the interim analysis. Here, we outline the identification of CoI(s), selection of RMTs and feasibility of use assessment in a clinical study, and discuss the feedback received from the Agency and the lessons learned to guide future projects aiming to qualify RMT-based outcome assessments for use in AD.

Identification of concept of interest (CoI)

In AD, the meaningful aspects of health are activities of daily living (ADLs) that are strongly associated with declining quality of life, rate of disease progression, and loss of independence²¹. In RADAR-AD, the meaningful aspects of health were defined with the support from patients and caregivers within the framework of patient and caregiver involvement work with the patient advisory board²². The meaningful aspects of health were then narrowed down to measurable health concepts (functional domains) that correspond to conversion from MCI to AD dementia, early impairment in AD, predictability of decline, and relevance to patients and caregivers. These can ultimately be translated to a final CoI(s), a practically measurable element of an aspect of a disease that is important to a patient (Fig. 1). Thereafter, different digital measures were selected for quantification of each functional domain and included in the study to assess the feasibility of their use in clinical study settings^16,23.

Fig. 1. Concept of Interest Derived from a Meaningful Aspect of Health (Activities of Daily Living).

Health experiences (a) reported by the Patient Advisory Board (PAB) were mapped onto the functional domains (b) that were then operationalized (c). For example, the functional domain ‘Managing finances’ was assessed with an App mimicking the withdrawal of money. Operationalizations of this functional domain are then subdivided into measurable elements such as the total duration of an attempt to withdraw money, the duration to enter the correct PIN code, or the duration of entering the correct amount of money to be withdrawn. These ultimately translate into a Concept of Interest (CoI).

Context of use (CoU) of RMT-based assessments

The proposed CoU for RMT-based assessments was to serve as secondary endpoints in clinical studies to complement standard clinical scales for the assessment of ADLs in individuals with preclinical to moderate AD in the home setting to support drug marketing applications. The RMT-based assessments could be used on their own or in combination, as a battery of performance tests, to support efficacy assessments in AD clinical studies.

Selection of RMTs and feasibility assessment

The evaluation of available RMTs was conducted by technical and clinical experts with input from the patient advisory board as described previously^16,22–24. A final selection of RMTs was made by mapping them to functional domains identified as relevant by the end users and included three smartphone applications (passive RMT app, the Mezurio app, and Altoida Neuro Motor Index application sofware), a wearable camera (Vicon Autographer) and two wrist-worn activity trackers (Axivity AX3 and Fitbit Charge 3) for at home assessments. The Gait Up device (now Mindmaze) Physilog sensors (worn on shoes and hip), the Banking app (developed by The Center for Research & Technology, Hellas, CERTH), and Altoida were administered in-clinic (see Supplementary Table 1)^16,23. Data collection and exchange were realized through the RADAR-base open-source platform (https://radar-base.org).

The main RADAR-AD study was an 8-week, cross-sectional, observational study comparing digital information from selected RMTs with established clinical measures in 237 participants (over 50 years of age) with preclinical AD (PreAD), prodromal AD (ProAD), mild-to-moderate AD dementia (MildAD), and age-matched healthy controls (HC) at 13 sites in 12 European countries^16–20. Participants and their study partners gave written informed consent before study start and the study was conducted in accordance with the Declaration of Helsinki. Each local ethics committee approved the study separately: The Netherlands: Medisch Ethische Toetsingscommissie VUmc (2019.518); Spain: Drug Research Ethics Committee (CEIm) of Universitat International de Catalunya (MED-FACE-2020-07); Italy: Comitato Etico IRCCS Centro San Giovanni di Dio Fatebenefratelli di Brescia; Switzerland: Commission cantonale d’éthique de la recherché (2022-00002); Portugal: Comissão de Ética do Centro Académico de Medicina de Lisboa (388/19); United Kingdom: London – West London & GTAC (Gene Therapy Advisory Committee) Research Ethics Committee (20/LO/0183); Germany: Ethics Committee II of the Ruprecht-Karls-University of Heidelberg (Medical Faculty Mannheim) (2020-508 N); Norway: Regionale komiteer for medisinsk og helsefaglig orskningsetikk (98842); Sweden: Swedish Ethical Review Authority (2020-03497); Greece: Ethics Committee of Medical Faculty of Aristotle University of Thessaloniki and Ethics Committee of Alzheimer Hellas (198/2018 AI).

All participants in the AD spectrum were defined by positive positron emission tomography (PET) and/or cerebrospinal fluid amyloid status, and further subdivided by mini-mental state examination (MMSE) and global clinical dementia rating (CDR) scores consistent with the National Institute on Aging and the Alzheimer’s Association (NIA-AA) criteria defining Alzheimer’s disease based on the biology²⁰. ProAD is synonymous to Mild Cognitive Impairment (MCI), a syndromic diagnosis, indicating deficits in a single cognitive domain which does not interfere with daily life independence. Such a syndrome can be caused by other disorders than AD as well. In the context of the RADAR-AD study, however, all participants that functioned at MCI level also had biomarker confirmation of AD pathology. Hence, a definition of ProAD was considered more suited.

The study addressed and examined three research objectives: (1) Compare features from each RMT between the groups (known-groups validity), (2) Identify if there is an association between the features extracted from each RMT and the relevant functional domain scores from standard clinical questionnaires (convergent validity), and (3) the feasibility of RMT use in this setting¹⁶.

Interim analysis

An interim analysis of data from 175 participants was performed to explore (1) known-groups validity and (2) convergent validity of features of 6 RMTs. Due to timelines associated with the scientific advice procedure at EMA, the qualification advice meeting took place while the clinical study was still ongoing. EMA’s advice was hence based on interim results (see Supplementary Information). The interim results showed a trend of deteriorating function across the AD spectrum (preclinical to mild AD) across various features of several RMTs. Earliest functional impairment in the PreAD group was captured by the complex cognitive domains as assessed with the Altoida application which has been confirmed in the full dataset²⁵. Evidence of convergent validity for various features across several RMTs was also shown through correlation with established clinical measures where appropriate. The interim analysis results for all RMTs including the table of established clinical measures are shown in Supplementary Tables 2–15 and Figs. 1–15. The consortium concluded that the selected measures were ecologically valid and appropriate to identify clinically meaningful outcomes to detect functional changes even in very early stages of AD. The consortium also proposed using machine learning algorithms to combine several features from each RMT to improve on known group by using stratified cross-validation techniques across multiple devices^26,27.

Health authority consultations

The consortium had an EMA Innovation Task Force meeting in 2020 for an informal discussion to obtain feedback on the project. The consortium was advised to use CE marked devices for medical use and to compare information from RMTs with established standard measures in prospective studies in the CoU. This advice was included in the selection of technologies when possible. The consortium included research grade technologies when CE-marked alternatives were not available. The RADAR-base open-source platform, used for data collection and data exchange, was considered acceptable, if it complied with regulatory principles and standards, such as Good Clinical Practice.

The consortium also initiated a Qualification Advice meeting from the EMA to obtain advice on identification of functional domains, selection of suitable RMTs, their clinical evaluation, data quality and user experience. Additionally, a preliminary protocol proposal was presented, describing a follow-up longitudinal clinical study. The protocol aimed to evaluate the sensitivity of RMT-based assessments to disease progression. For this, participants with amyloid- and tau-confirmed prodromal AD, who are most likely to show disease progression within the study duration of 18 months, were considered as the study population. The Qualification Advice procedure and timelines are outlined in Fig. 2 with the EMA feedback for the current study and recommendations for a future study summarized in Box 1.

Fig. 2. Qualification advice process and timelines.

Overall, the advice period lasted 7 months, which was preceded by the compilation of the briefing book (5 months). Validation comments from the European Medicines Agency (EMA) were received 3 months after submission. After submitting a revised version of the briefing book, the consortium received a list of issues that were addressed in a written response as well as an online meeting. The final Qualification Advice was received 7 months after the briefing book submission. Submissions by the consortium are shown in blue, CHMP responses in brown and meetings in green.

The Committee for Medicinal Products for Human Use (CHMP) agreed with the consortium’s proposed process for item selection and identification of functional domains reflecting meaningful aspects of AD at face value. However, whether these functional domains could distinguish different stages of AD, specifically at the very early stages, predict, or monitor disease progression (e.g., MCI to AD dementia conversion) would need to be studied further using a longitudinal approach. In addition, CHMP agreed to the limitation in assessing convergent validity of certain functional domains such as gait due to the lack of a standard clinical instrument or ground truth. Depending on the AD subpopulation, the combinations of relevant features could differ and functional domains (e.g., difficulties at work) might carry different weight. People’s capabilities to use the RMTs is also anticipated to change over time as the disease progresses. The CHMP recommended to continue the development work for domains relevant for individuals who are still in the workforce and domains that were identified by the patient advisory board, such as interpersonal interactions, motivation, and apathy, but could not be included in the study as suitable RMTs were not available.

The consortium’s approach to select the RMTs based on literature reviews, expert and patient feedback, and available technical information, was considered reasonable. The CHMP expressed concerns regarding the risk of changes in the RMTs introduced by the manufacturers, the use of consumer devices for which the access to the full dataset, including raw data and algorithms may not be possible. The CHMP also highlighted the importance of General Data Protection Regulation (GDPR) compliance, particularly when using consumer devices.

The CHMP emphasized that the convergent and divergent validity of the RMT-based assessments with conventional measures would be key. The selection on the frequency of use of some RMTs (daily, weekly) should be thoroughly justified to decrease patient burden. Furthermore, the learning curve of repeated use of the active RMTs (i.e., RMTs that require user interaction) is important. In the study, participants performed learning exercises during on-site training sessions of the RMTs at the baseline visit, but these were not repeated as it was beyond the scope of this exploratory study. The CHMP recommended that the learning effect of the active RMTs be assessed throughout a future study to understand its impact on the assessment of functional domains. The CHMP also commented that the proposed prospective, longitudinal, observational study with 3 in-clinic visits over a period of 18 months may be too short to establish the relationship between a change in the RMT-based assessments and function outcome as measured by Alzheimer’s Disease Cooperative Study – Activities of Daily Living for Mild Cognitive Impairment (ADCS-ADL-MCI) scale, especially in a prodromal AD population. A future, longitudinal study should be of sufficient duration to allow for conclusions on disease progression or changes in cognition, given the slow progression in AD. Additionally, although most participants would tolerate and be willing to use the devices, particularly wearables, this willingness is likely to decrease over time in longer studies, leading to missing data.

The consortium’s proposed approach to quantify floor and ceiling effects by comparing distributions of the whole population and compare these to distributions known to have floor and ceiling effects, such as ADCS-ADL, was agreed upon. However, it was highlighted that accuracy is only one aspect of validation and the convergent and divergent validity of specific RMTs (or combinations thereof) for the functional domains of interest is also important.

The use of artificial intelligence and machine learning techniques to evaluate RMT-based outcome assessments is accepted as an exploratory exercise, identifying potential promising concepts and tools as alternative assessments of functional domains. Many machine learning algorithms are entirely data driven, generating theoretical constructs. The CHMP agreed with the consortium’s plan to identify different AD stages by combining results from specific RMTs for the functional domains of interest. The Agency also indicated that, if a composite of RMTs features allows an overarching construct, this concept should be made plausible and replicable. However, its content and its convergent and divergent validity will remain to be established against existing scales that measure the same or partially overlapping/related concepts.

The consortium’s approach to investigate known group validity by investigating possible differences between the metrics extracted from each RMT-based assessment and the functional domain scores derived from the conventional questionnaires (convergent validity) was accepted. However, the large overlap in scores between the groups in the RMTs (indicating limited discriminative power of the tests) was noted along with cases of weak correlations (even if significant). For this, the sensitivity to change of a specific (combination of) RMT(s), minimal important difference and sensitivity to show a treatment effect should be part of future validation efforts.

The EU medical device regulation conformity for the technologies not CE marked as medical devices was briefly discussed. The CHMP stated that the qualification of medical devices and algorithms is not within EMA’s remit but falls within the Notified Bodies (organizations authorized by the EU member states to evaluate CE conformity of products prior to marketing). The Scientific Advice Working Party did not agree with the consortium’s position that the proposed CoU of RMTs (only data collection in clinical studies) should not classify them as medical devices. The study participants frequently receive information from the sensors, which could modify their health behavior, and potentially affect the study results. The CHMP also added that when the medical devices are used outside their intended use, setting up a research protocol to test the new CoU is advised. Furthermore, if RMTs investigated were not CE certified, this could impact their long-term scientific reliability, as manufacturers could seek to upgrade features in response to market forces, which could potentially impact the data collection.

In conclusion, in AD drug development, RMT-based assessments have the potential to offer important advantages over conventional methods by allowing continuous and objective assessment of daily function, and can complement current practices, provide new ways to capture existing measures or enable novel measures. Successful implementation of RMT-based assessments in clinical studies is a complex matter and Health Authority consultations early in the process are essential for regulatory acceptance. Herein, we focused on implementation strategies and considerations for longitudinal follow-up studies to validate RMT-based measures in clinical trials. Additionally, it is noted that the received feedback presented aims to provide an overall roadmap for interacting with Health Authorities rather than to address specific RMTs.

Key learnings from the RADAR-AD project (see Box 2) include the identification of a CoI that is relevant and meaningful to the target population. The definition of the CoU, along with a detailed description of how the outcome measure is to be used in the target study population is imperative for regulatory assessment. To increase the probability of success, iterative approaches may be considered, e.g., initial qualification for secondary endpoint followed by considerations for use as primary endpoint¹⁰. In addition, a process on how to expand to additional CoUs or target diseases should be agreed upon. The RADAR-AD data calls for a longitudinal follow-up study to further investigate and validate the most promising tools. Designing a larger and longer validation study at the time of EMA qualification advice procedure, however, was beyond the scope of the RADAR-AD project and the available resources. Designing such a study could be a follow-up activity (post consortium) for which qualification advice from EMA and other Health Authorities can be obtained and incorporated in the design prior to implementation.

Considering the complexity of the qualification process^28,29, it is advisable to engage with Health Authorities utilizing available advice procedures (e.g., Innovation Task Force and EMA Qualification Advice meetings), as appropriate³⁰, and plan for multiple Health Authority engagements to obtain feedback prior to initiation of the studies. In this way, implementation of the feedback received into the study design is possible, along with follow-up meetings to discuss results and next steps. The interactions with Health Authorities are resource intensive and require full engagement of all partners relevant to the discussion topics. Hence, resource allocations and timelines for these interactions should be built into the program plans and objectives. Considering that these technologies will support global projects, engagement with other major Health Authorities (e.g., US Food and Drug Administration, UK Medicines and Healthcare Products Regulatory Agency, or Japan Pharmaceuticals and Medical Devices Agency) to obtain broader acceptance is essential.

Box 1 EMA feedback and recommendations.

EMA feedback and recommendations

RADAR-AD Exploratory Study

• The process for identifying functional domains and RMTs was acceptable

• The use of CE-marked RMTs is desirable

• Quantification of floor and ceiling effects of RMT parameters is desirable

Future Study

• Additional prospective longitudinal study of identifying functional domains indicative of different AD stages is needed

• A multi-year study would be needed to adequately measure disease changes

• Ensure that learning curves for RMT use are assessed

• Include relevant assessments that capture changes from very early stages of AD

Box 2 Guidance for researchers seeking advice from health authorities.

Guidance for Researchers Seeking Advice from Health Authorities

• Establish a regulatory strategy, including planning for Health Authority meetings aligned with the project goals

• Identification of CoI that is meaningful to the patients

• CoU is an essential component of Health Authority review

• Prioritize multiple CoU cases and adopt a staged approach for qualification

• The preparation for successful Health Authority meetings is time and resource intensive

Author contributions

G.E. provided regulatory oversight to the project, was the main contact for the EMA Qualification Advice meeting and a major contributor in writing the manuscript. MG Performed data analysis, EMA Qualification Advice Briefing package, and was a major contributor in writing the manuscript. B.W. contributed to the EMA Qualification Advice Briefing package and was a major contributor in writing the manuscript. S.V. performed data and statistical analyses, contributor to EMA Qualification Advice Briefing package, and reviewed the manuscript. J.C. performed data analysis, contributed to the EMA Qualification Advice Briefing package, and reviewed the manuscript. D.A. designed the study, advised on the data analyses, and reviewed the manuscript. G.W. reviewed the manuscript. S.N. designed the study, advised on the data analyses, and reviewed the manuscript. M.M. recruited participants and collected data, performed the data analyses, and reviewed the manuscript. H.F. advised on the data analysis and reviewed the manuscript. C.d.B. designed the study, advised on the data analyses, and reviewed the manuscript. N.M.S. contributed to the EMA Qualification Advice Briefing package and reviewed the manuscript. V.J.M.N. Foundation Lygature acted as formal requester of the Qualification Advice on behalf of the Consortium; Project management and reviewed the manuscript. N.C. reviewed the statistical methodology and analysis and reviewed the manuscript. D.G. contributed to the EMA Qualification Advice Briefing package, coordinated the work of the RADAR-AD Patient Advisory Board, and reviewed the manuscript. A.D. contributed to the EMA Qualification Advice Briefing package, coordinated the work of the RADAR-AD Patient Advisory Board, and reviewed the manuscript. S.F. contributed to the EMA Qualification Advice Briefing package and reviewed the manuscript. W.D. contributed to the EMA Qualification Advice Briefing package and reviewed the manuscript. A.K.B. designed the study, advised on the analyses, contributed to the EMA Qualification Advice Briefing package, and was a major contributor in writing the manuscript. All authors read and approved the final manuscript.

Competing interests

G.E., N.C., J.C., and W.D. are employees of Novartis and may hold shares in the company but have no non-financial competing interests. B.W., S.V., G.W., and S.F. are employees of Janssen Research & Development, LLC, a Johnson & Johnson company, and may hold stock options or shares in the company but have no non-financial competing interests. N.M.S. is an employee of Takeda and holds shares in the company but has no non-financial competing interests. D.A. has received research support and/or honoraria from Astra-Zeneca, H. Lundbeck, Novartis Pharmaceuticals, Biogen, and GE Health, and served as a paid consultant for H. Lundbeck, Eisai, Heptares, Mentis Cura, and Roche Diagnostics, but declares no non-financial competing interests. H.F. has received research support from UCB Pharma and AbbVie GmbH & Co KG. All other authors declare no financial or non-financial competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41746-024-01211-8.