Feasibility and acceptability of remote smartphone cognitive testing in frontotemporal dementia research

Abstract

Introduction

Remote smartphone assessments of cognition, speech/language, and motor functioning in frontotemporal dementia (FTD) could enable decentralized clinical trials and improve access to research. We studied the feasibility and acceptability of remote smartphone data collection in FTD research using the ALLFTD Mobile App (ALLFTD‐mApp).

Methods

A diagnostically mixed sample of 214 participants with FTD or from familial FTD kindreds (asymptomatic: CDR®+NACC‐FTLD = 0 [N = 101]; prodromal: 0.5 [N = 49]; symptomatic ≥1 [N = 51]; not measured [N = 13]) were asked to complete ALLFTD‐mApp tests on their smartphone three times within 12 days. They completed smartphone familiarity and participation experience surveys.

Results

It was feasible for participants to complete the ALLFTD‐mApp on their own smartphones. Participants reported high smartphone familiarity, completed ∼ 70% of tasks, and considered the time commitment acceptable (98% of respondents). Greater disease severity was associated with poorer performance across several tests.

Discussion

These findings suggest that the ALLFTD‐mApp study protocol is feasible and acceptable for remote FTD research.

HIGHLIGHTS

• The ALLFTD Mobile App is a smartphone‐based platform for remote, self‐administered data collection.

• The ALLFTD Mobile App consists of a comprehensive battery of surveys and tests of executive functioning, memory, speech and language, and motor abilities.

• Remote digital data collection using the ALLFTD Mobile App was feasible in a multicenter research consortium that studies FTD. Data was collected in healthy controls and participants with a range of diagnoses, particularly FTD spectrum disorders.

• Remote digital data collection was well accepted by participants with a variety of diagnoses.

1. BACKGROUND

Frontotemporal dementia (FTD) is a heterogenous collection of clinical syndromes characterized by behavioral, cognitive, language, and motor deficits. ¹ FTD syndromes include behavioral variant frontotemporal dementia (bvFTD), primary progressive aphasias (PPA), corticobasal syndrome (CBS), and progressive supranuclear palsy (PSP). ² , ³ , ⁴ , ⁵ , ⁶ FTD is a common form of early‐onset dementia. Although most cases are sporadic, roughly 20%‐30% of cases are caused by dominantly inherited mutations (familial or f‐FTD). ⁷ , ⁸

There are currently no approved treatments for FTD, but clinical trials are underway. ⁹ Successfully identifying effective therapies, however, is contingent on well‐defined trial endpoints that capture the clinical heterogeneity of FTD. ¹⁰ Moreover, FTD is relatively rare and affected individuals are geographically dispersed. Recent estimates suggest that f‐FTD clinical trials will require global recruitment to achieve sufficient power. ¹¹ Decentralized trials may be necessary to overcome recruitment barriers and will require new tools to enable remote data collection.

Smartphones, which are becoming ubiquitous, ¹² , ¹³ may provide a vehicle for addressing recruitment challenges in FTD research and potentially improving sensitivity to early symptoms. We previously found that digital tests of executive functioning, which is affected across FTD syndromes, detected deficits in asymptomatic f‐FTD mutation carriers. ¹⁴ Building upon these findings, ALLFTD investigators (https://www.allftd.org) developed a comprehensive battery of smartphone assessments to facilitate remote assessment of FTD's diverse clinical features. The ALLFTD Mobile App (ALLFTD‐mApp) battery is designed for FTD observational research and clinical trials, and includes surveys and measures of motor, memory, and executive functioning, including gamified versions of classic tests (e.g., Flanker, 2‐back) that are sensitive to FTD. ¹⁵ , ¹⁶ The app also features a flexible infrastructure for designing speech/language tests and capturing speech samples. Here, we describe the ALLFTD‐mApp battery and study design, and we report on the feasibility and acceptability of remote smartphone‐based data collection in sporadic and f‐FTD cohorts.

2. METHODS

2.1. Participants

Participants underwent annual standardized evaluations that included neurological assessment, caregiver or companion interview, neuropsychological testing (UDS v3.0) ¹⁷ , brain MRI, and biofluid collection. Diagnoses were made based on multidisciplinary case conferences according to published diagnostic criteria. ³ , ⁴ , ⁵ , ⁶ , ¹⁸ Disease severity was defined using the Clinical Dementia Rating Scale plus National Alzheimer's Coordinating Center FTLD module (CDR®+NACC‐FTLD). This clinician‐administered interview of participants and their study partners is the gold standard for quantifying FTD disease severity. Eight subdomain scores were summarized to create a five‐point Global score ¹⁹ , ²⁰ and grouped for analyses as 0 (asymptomatic), 0.5 (prodromal), and 1‐3 (symptomatic). The asymptomatic group contained both healthy controls and asymptomatic f‐FTD mutation carriers. ²¹

RESEARCH IN CONTEXT

1. Systematic Review: The authors reviewed the literature using traditional sources (e.g., Pubmed), meeting abstracts, and presentations. Smartphone measures of cognition, speech, and motor function are increasingly studied among individuals from across the lifespan with a range of clinical conditions. No studies were found, however, that explored the suitability of such measures in the context of frontotemporal dementia (FTD) research.

2. Interpretation: Our findings suggest that the collection of a self‐administered comprehensive battery of relevant smartphone measures is both feasible and well‐tolerated by individuals with FTD and healthy controls, opening new avenues for efficient data collection in FTD research.

3. Future Directions: An evaluation of the reliability and validity of these smartphone tests will provide crucial information about the utility of these tests in FTD research and clinical care. Further exploration of factors that influence initial engagement and sustained adherence will support future implementation efforts in both longitudinal observational research and clinical trials.

Data collection occurred in two successive studies. Study 1 was a pilot study conducted at UCSF from January to March 2020 that enrolled 20 participants with CDR®+NACC‐FTLD scores ≤1 who were selected to represent symptomatic FTD, asymptomatic f‐FTD, healthy adults, and study partners. Study 2 was the main observational study of FTD patients and asymptomatic individuals from f‐FTD families with a known C9orf72 expansion or pathogenic/likely pathogenic mutation in GRN or MAPT. Mutation carriers and non‐carrier family controls were included. Recruitment began in July 2020.

Participants were enrolled through 18 ARTFL/LEFFTDS Longitudinal Frontotemporal Lobar Degeneration (ALLFTD; NCT04363684) centers. Participants were also recruited through UCSF studies of FTLD (Advancing Research and Treatment for Frontotemporal Lobar Degeneration (ARTFL; NCT02365922), Longitudinal Evaluation of Familial Frontotemporal Dementia Subjects (LEFFTDS; NCT02372773), ²² The 4‐Repeat Tauopathy Neuroimaging [4RTNI; NCT02966145A], and studies of functionally intact older adults. ²³ Procedures were approved by the UCSF Institutional Review Board (IRB) and the Johns Hopkins Central IRB. Consent was obtained from all participants, legally authorized representatives, and study partners.

Inclusion criteria for the pilot and main studies were: (1) 18 years or older; (2) sufficient fluency in English to engage with study procedures; and (3) access to a smartphone (no smartphones were provisioned in this study). Participants were asked to complete the tests on their own smartphones and encouraged to enroll with a study partner who had at least weekly contact with the participant. Study partners were required for symptomatic participants. ²¹ Recruitment primarily targeted those with CDR®+NACC‐FTLD Global ≤ 1 (mild dementia), but sites had discretion to enroll more severely impaired participants who could participate. Exclusion criteria were consistent with the parent ALLFTD study.

2.2. ALLFTD Mobile App Overview

ALLFTD investigators partnered with Datacubed Health (www.datacubed.com) to develop the ALLFTD‐mApp on Datacubed Health's Linkt platform, which includes (1) a backend interface for enrollment and participation tracking, and (2) a smartphone app for participants and study partners available for Android operating system Version 6+ and iOS 11+. Data are stored securely on an Amazon Web Service server that is HIPAA, GDPR, and FDA CFR 21 Part 11 compliant.

The ALLFTD‐mApp incorporates core reward principles of neuroeconomics to maximize participant engagement and retention (Figure 1A‐1C). ²⁴ Participants received remuneration upon completion of each set of tasks. The full list of surveys, tasks, and passive measures is displayed in Table 1 and tasks are described in Supplemental Appendix A.

FIGURE 1.

ALLFTD Mobile App overview and task schedule Note. Figure 1A: Participants progress through a virtual world as they complete tasks. The circle with the “+9″ indicates the number of remaining tasks. Participants press this circle to access their tasks. Figure 1B: Participants can create a personalized avatar and gems can be used to purchase accessories. Gems are received after completing each task. Figure 1C: This is an example of one of the cognitive tasks, a gamified version of the classic Flanker paradigm. Participants are asked to indicate the direction of the center duck by pressing the left or right arrow. Figure 1D: This schematic illustrates the pilot and observational study participation schedules. For an example participant starting the observational study on a Monday, Session 1 tasks are available for 3 days (Monday‐Wed) followed by a washout day during which no tasks are available (Thursday). A new set of tasks become available in Session 2 (Friday – Sunday), followed by a second washout day (Monday). The third set of tasks become available in Session 3 (Tuesday – Thursday). Note that tasks were only available for 2‐day windows in the pilot study. Pilot participants recommended a longer window to complete tasks, citing the scenario in which the 2‐day window fell on the weekend as a particular barrier to adherence.

TABLE 1.

ALLFTD Mobile App battery.

ALLFTD Mobile App Battery
Surveys
Guidance/Feedback		Participant Characteristics
Navigating the App Mobile App Experience		Technology Familiarity
Cognitive Tests
Executive Functioning	Speech & Language	Memory	Motor Functioning
Card Sort	Dysdiadochokinesia	Adaptive Memory Test	Gait
Stroop	Automatic Speech	Picture Recall	Balance
Flanker	Sustained Phonation		Finger Tapping
2‐Back	Spontaneous Speech
Go/No‐go	Picture Description
	Volume Control
	Passage Reading
Passive Data
Screen Time		GPS Location
Step Count		Battery Life
App Usage

Note. The current battery of tasks and passive data options available in the ALLFTD Mobile App battery are displayed here. The battery underwent minor changes throughout the study as additional tests were developed. Descriptions of each task are found in Supplemental Appendix A.

2.3. ALLFTD Mobile App Measures

2.3.1. Cognitive tests

Participants completed 3‐ to 5‐min gamified versions of classic executive functioning tasks developed by Datacubed Health: Stroop, ²⁵ Flanker, ²⁶ 2‐back, ²⁷ Go/No‐Go, ²⁸ and Wisconsin Card Sort Test ²⁹ paradigms. ¹⁴ , ³⁰ Participants also completed a 4‐min adaptive memory test. Executive functioning and memory are commonly affected by FTD, including at prodromal stages. ¹⁶ , ³¹ , ³² , ³³

2.3.2. Speech and language tests

Deficits in speech and language are common in FTD. ³⁴ , ³⁵ , ³⁶ Speech samples were recorded while participants completed a ∼7‐minute speech and language battery.

2.3.3. Motor tests

Participants with FTD can experience a variety of motor changes, including parkinsonism, dystonia, and motor neuron disease. ³⁷ The ALLFTD‐mApp includes a 3‐ to 5‐minute motor battery including tests of gait, balance, and finger tapping modeled after previously validated smartphone measures. ³⁸ , ³⁹

2.3.4. Surveys

1. Technology Familiarity: asked about frequency of and confidence in smartphone use, ownership and use of a tablet or computer, use of a screen protector, and phone condition.

2. Navigating the App: immediately following enrollment, participants were asked questions about the ALLFTD‐mApp procedures and could choose to review information.

3. Mobile App Experience: administered at the end of each testing session, this survey assessed the acceptability of study design. The first available response from each participant was included in this report.

The Navigating the App and Mobile App Experience surveys were designed using principles of the Capability, Opportunity, and Motivation Behavior Change Theory. ⁴⁰

2.4. Procedures

2.4.1. ALLFTD Mobile App procedures

Participants were asked to enroll in the ALLFTD‐mApp study within 90 days of their annual ALLFTD study visits. Site research coordinators assisted participants with ALLFTD‐mApp download, setup, and orientation.

Participants were asked to complete the first task with the coordinator present, either in person or through video call. All other tasks were self‐administered in a predefined order. Participants were given detailed instructions to minimize distractions when taking tests at home. Study partners of symptomatic participants were asked to remain present during participation to help navigate the ALLFTD‐mApp but not to assist with task completion. Written take‐home instructions were provided.

The ALLFTD‐mApp visit was divided into three assessment periods, referred to as “Session,” which were spread over the course of 8‐12 days (see Figure 1D for schedule). Each Session consisted of a 25‐ to 35‐min task battery, a reduction in time relative to the ≥1 hour cognitive testing sessions that are part of standard ALLFTD procedures. Most measures were repeated in every Session to potentially improve task reliability ⁴¹ and to assess test‐retest reliability and practice effects. This triplicate of Sessions is being administered every 6 months for the duration of the ALLFTD study, but only the baseline triplicate was analyzed in the current study.

Participants received push notifications at the arrival of tasks in each Session and 24‐hours prior to the end of the Session if any tasks remained incomplete. All task data were collected remotely and unsupervised except for some participants in the pilot study (detailed below). Some UCSF participants were incentivized with $25 at 6 months and 1 year for ≥ 50% adherence. Starting in March 2021 under a separate IRB approval, ALLFTD participants were incentivized with $20 Amazon gift codes, automatically sent using the in‐app inbox, after completing all tasks within a Session for a maximum payment of $60 for each triplicate.

Study partners also downloaded a version of the ALLFTD‐mApp on their phone and were asked to complete several surveys: (1) Revised Self‐Monitoring Scale, ⁴² (2) Corticobasal Functional Scale, ⁴³ (3) Zarit Burden Inventory, ⁴⁴ (4) Technology Familiarity, and (5) a study partner experience survey. Study partners received notifications when participant tasks were available. Study partner data are not included in this report.

2.4.2. Additional pilot study procedures

Pilot participants completed all tasks in the first Session with a coordinator present (in‐person or virtually), and feedback was gathered from a subset (N = 16) in which participants discussed aloud their experience with the ALLFTD‐mApp interface, instructions, and tasks. ⁴⁵ , ⁴⁶ Participants were asked to read all task instructions aloud and rank (6‐point Likert) the comprehensibility of the instructions (1 = very difficult to understand, 6 = very easy) and cognitive test difficulty (1 = very difficult to complete, 6 = very easy). Participants and study partners were then interviewed over the phone after the third Session to invite feedback.

2.5. Statistical analysis

Pilot data and main study data were analyzed separately. Differences in continuous demographic variables (e.g., age, education) and cognitive task scores by disease severity were explored using linear regression and pairwise t‐tests with Tukey adjustment when the omnibus test was significant. Age, sex, and education were entered as covariates in models comparing cognitive test scores. For frequency data (e.g., sex, race), chi‐squared difference tests were used. Adherence was defined as the proportion of tasks completed of the total tasks available. Seven main study participants who did not complete any tasks (all CDR®+NACC‐FTLD global > 0) were dropped from analyses. Six additional participants were excluded from adherence analyses due to a software error that impacted task completion, including four task crashes and two cases of tasks not arriving on schedule. Responses to the two technological familiarity questions were summarized into a single binary variable, such that cases endorsing both daily smartphone use and highest confidence level were denoted as a “high” familiarity score and any other responses were considered “low.” To explore whether familiarity was affected by disease severity, logistic regression models were fit with binary smartphone familiarity as the outcome and CDR®+NACC‐FTLD global score as the predictor. A logistic regression model was fit in asymptomatic participants using age, sex, and education as predictors to understand the relationship between demographics and smartphone familiarity. Adherence was compared between levels of disease severity using linear regression with age, sex, and education as covariates. Adherence was further described in a subsample who completed at least 25% of all tasks. This cutoff suggests adequate understanding and engagement with the study, with many of the participants completing at least one full testing session. Linear regression was used to explore differences in adherence among asymptomatic participants by age, sex, and education. Adherence stability across app Sessions was explored using linear mixed effects models with random slopes and intercepts with adherence as the outcome and Session as the predictor. This model was fit first in the entire sample and then separately in asymptomatic individuals adjusting for age, sex, education, and interaction terms between Session number and demographics. All analyses were performed with R software version 4.2.0.

3. RESULTS

3.1. Pilot study

3.1.1. Participant characteristics

The pilot study enrolled 20 participants, including 10 clinically normal older adults, 6 asymptomatic participants from f‐FTD families, and 4 symptomatic participants. Participant characteristics are summarized in Table 2. Participants completed 88.5% of all available tasks and 81.3% of the remote, self‐administered tasks (Sessions 2 and 3).

TABLE 2.

Sample description, baseline demographics, and clinical characteristics

Characteristic	Pilot study	Main study
	All	All	CDR®+NACC FTLD global scores
			Score = 0 (Asx)	Score = 0.5 (Pd)	Score = 1+ (Sx)	p‐Value b	Post hoc comparison c
Sample size	20	194	89	45	48	–	–
Age—yr (mean(SD))	64.5 (16)	55.2 (15)	47.0 (16.0)	60.1 (11.0)	65.4 (8.4)	<0.001	Sx > (Pd = Asx)
Female—n (%)	12 (60.0%)	103 (53.4%)	56 (62.9)	21 (46.7)	21 (43.8)	0.042	Asx = Pd = Sx
Education—year (mean(SD))	17.5 (1.9)	16.5 (2.2)	16.3 (2.1)	16.7 (2.2)	16.3 (2.5)	0.417	Asx = Pd = Sx
Race—n (%)
White	19 (95.0%)	180 (92.8%)	81 (91.0%)	43 (95.6%)	47 (97.9%)	0.175	Asx = Pd = Sx
Non‐White	1 (5.0%)	8 (4.1%)	4 (4.5%)	0 (0.0%)	1 (2.1%)	–	–
Not available	0 (0.0%)	6 (3.1%)	4 (4.5%)	2 (4.4%)	0 (0.0%)	–	–
Genetic status (N)
C9orf72	1	24	16	3	1	–	–
GRN	0	7	5	1	1	–	–
MAPT	1	15	6	2	0	–	–
Non‐carrier a	0	64	24	5	17	–	–
Not available	18	84	18	19	19	–	–
Diagnoses—n (%)
Clinically normal	15 (75.0%)	96 (49.5%)	85 (95.6%)	5 (11.1%)	0 (0.0%)	–	–
MCI/MBI	0 (0.0%)	19 (9.8%)	1 (1.1%)	14 (31.1%)	2 (4.2%)	–	–
Depressive symptoms	1 (5.0%)	0 (0.0%)	0 (0.0%)	0 (0.0%)	0 (0.0%)	–	–
Psychiatric/personality disorder	0 (0.0%)	1 (0.5%)	1 (1.1%)	0 (0.0%)	0 (0.0%)	–	–
bvFTD	2 (10.0%)	32 (16.5%)	0 (0.0%)	7 (15.5%)	24 (50.0%)	–	–
Corticobasal syndrome	1 (5.0%)	10 (5.2%)	0 (0.0%)	6 (13.3%)	4 (8.3%)	–	–
FTD/ALS	0 (0.0%)	3 (1.6%)	0 (0.0%)	1 (2.2%)	2 (4.2%)	–	–
lvPPA	1 (5.0%)	1 (0.5%)	0 (0.0%)	0 (0.0%)	1 (2.1%)	–	–
nfvPPA	0 (0.0%)	11 (5.7%)	0 (0.0%)	7 (15.5%)	4 (8.3%)	–	–
svPPA	0 (0.0%)	8 (4.1%)	0 (0.0%)	2 (4.4%)	6 (12.5%)	–	–
PSP	0 (0.0%)	8 (4.1%)	0 (0.0%)	3 (6.7%)	5 (10.4%)	–	–
Other	0 (0.0%)	2 (1.0%)	1 (1.1%)	0 (0.0%)	0 (0.0%)	–	–
Not available	0 (0.0%)	3 (1.5%)	1 (1.1%)	0 (0.0%)	0 (0.0%)	–	–
MoCA score	19.8 (12.5)	26.5 (11.1)	27.9 (2.3)	25.6 (3.7)	22.7 (23.6)	0.004	(Asx = Pd), (Pd = Sx), Asx > Sx
Device type — n (%)
Android	4 (20.0%)	55 (28.4%)	30 (33.7%)	14 (31.1%)	8 (16.7%)	0.084	Asx = Pd = Sx
iPhone	16 (80.0%)	139 (71.7%)	59 (66.3%)	31 (68.9%)	40 (83.3%)	–	–
Total adherence (%)	88.5%	70.3%	71.4%	78.4%	59.0%	0.005	(Asx = Pd) > Sx
Session 1 adherence	98.0%	80.2%	83.0%	80.4%	71.8%	0.014	(Asx = Pd), (Pd = Sx), Asx > Sx
Session 2 adherence	80.0%	69.4%	69.8%	82.5%	56.1%	0.011	(Asx = Pd) > Sx
Session 3 adherence	82.5%	60.0%	60.5%	72.2%	46.4%	0.026	(Asx = Pd), (Asx = Sx), Pd > Sx
ALLFTD‐mApp task scores
Card Sort — # correct (SD)	NA	32.7 (8.4)	36.0 (5.1)	31.5 (8.1)	26.4 (10.4)	<0.001	Asx > Pd > Sx
Stroop — # correct (SD)	45.2 (18.0)	50.6 (17.7)	57.7 (15.8)	39.4 (13.2)	–	<0.001	Asx > Pd
Flanker — seconds to complete (SD)	135.8 (20.7)	128.3 (27.1)	113.1 (15.3)	135.8 (21.9)	151.6 (29.3)	<0.001	Asx > Pd > Sx
Adaptive memory — # correct/trial (SD)	3.7 (1.2)	3.9 (1.1)	4.6 (0.8)	3.5 (0.8)	3.0 (0.9)	<0.001	Asx > Pd > Sx
2‐Back — d‐prime (SD)	1.9 (0.8)	2.0 (1.2)	2.4 (1.1)	1.2 (0.9)	–	<0.001	Asx > Pd
Go/No‐go — # correct ‐# incorrect (SD)	70.6 (20.0)	68.4 (26.3)	77.0 (4.4)	70.4 (16.4)	48.0 (45.6)	<0.001	Asx > Pd > Sx

Note. Total adherence refers to the number of tasks completed out of the total number of tasks available. CDR®+NACC FTLD Global Scores was not measured for 1 participant in the pilot study and 12 participants in the main study. In the majority of participants, results of genetic testing for pathogenic FTD mutations were not available at the time of this report.

Abbreviations: Asx, asymptomatic; bvFTD, behavioral variant frontotemporal dementia; FTD/ALS, frontotemporal dementia/ amyotrophic lateral sclerosis; MCI/MBI, mild cognitive/behavioral impairment; Pd, prodromal; PPA, primary progressive aphasia, including lvPPA, logopenic variant PPA, nfvPPA, nonfluent variant PPA, and svPPA, semantic variant PPA; PSP, progressive supranuclear palsy; Sx, symptomatic.

^a Non‐carrier refers to individuals from a family with known pathogenic mutations who tested negative for that mutation.

^b The reported p‐values are from the omnibus test.

^c For cognitive tests, the post hoc comparisons present the group with scores representing least impairment on the left of the inequality sign. All other post hoc comparisons display the group with the highest value on the left of the inequality sign.

3.1.2. Cognitive pretesting

Participants ranked the clarity of cognitive task instructions and task difficulty as 4.94/6 (6 = very clear) and 4.23/6 (6 = very easy), respectively. Participant feedback prompted several modifications to the ALLFTD‐mApp's design and schedule configuration prior to initiating the main study, including: (1) increased font size, (2) introduction of in‐app payments with Amazon gift codes, (3) removal of difficult tasks for participants with CDR®+NACC‐FTLD > 0.5, (4) modifications to standard operating procedures and take‐home instructional materials, and (5) increased Session duration to 3 days to provide more time to complete tasks. Table S1 contains individual task feedback.

3.2. Main study

3.2.1. Participant characteristics

The main study enrolled 194 participants. Characteristics are displayed in Table 2. The sample was 53.4% female, average education was 16.5 years, and the average age was 55.2 years with a wide range (20 – 84). Of the 182 participants with known CDR®+NACC‐FTLD scores, nearly half (48.9%) were asymptomatic. The remaining participants were prodromal (24.7%) or symptomatic (26.4%). Most diagnosed cases had bvFTD (33.7%), followed by PPA (21.1%) and mild cognitive or behavioral impairment (20.0%). Genetic data were available from 87 participants; of those, most f‐FTD mutations were in C9orf72 (N = 22) followed by MAPT (N = 13) and GRN (N = 5). Most participants used iPhones (71.7%).

3.2.2. Technology familiarity

Frequency of smartphone use was high among the 171 participants who completed the Technology Familiarity Survey; 91.8% reported using their smartphone more than once per day (> daily). Most participants (91.8%) expressed either consistent confidence in using their smartphone (66.7%) or confidence depending on the task (25.1%). The percentages of individuals reporting > daily smartphone use among asymptomatic (N = 78), prodromal (N = 39), and symptomatic individuals (N = 42) were 100.0%, 92.3%, and 76.2%, respectively (Figure 2). Notably, 24/27 bvFTD cases reported > daily smartphone use. Table S2 provides a summary of all survey responses by diagnostic group. Compared to asymptomatic participants, the odds of high “smartphone familiarity” were lower among prodromal (odds ratio [OR] = 0.32, 95% confidence interval [CI] [0.13, 0.77], p = 0.01) and symptomatic (OR = 0.10, 95% CI [0.04, 0.23], p < 0.01) participants. Among asymptomatic individuals, each decade increase in age was associated with lower smartphone familiarity (OR = 0.47, 95% CI [0.27, 0.76], p < 0.01). While not reaching statistical significance, beta coefficients and confidence intervals suggest that smartphone familiarity may have been higher among asymptomatic males (OR = 3.38, 95% CI [0.80, 18.99], p = 0.16) compared to females. Years of education was not significantly associated with smartphone familiarity (OR = 1.13, 95% CI [0.80, 1.59], p = 0.55).

FIGURE 2.

Smartphone Familiarity Survey. Note. Technology Familiarity survey responses from 154 participants with a known CDR®+NACC‐FTLD score were plotted. The number of participant responses is counted on the y‐axis, and the bars are color coded by disease severity (CDR®+NACC‐FTLD). The percent of all participants who selected the response option, regardless of CDR®+NACC‐FTLD, is displayed at the top of each bar. Most participants indicated they use their smartphones multiple times per day (A) and are either confident in using their smartphone or it depends on the task (B). All participants who were not daily users or not usually confident in using their device were prodromal or symptomatic. CDR®+NACC‐FTLD = Clinical Dementia Rating Scale plus National Alzheimer's Coordinating Center Frontotemporal Lobar Degeneration module.

3.3. Feasibility

3.3.1. Navigating the App

On average, participants (N = 148) elected to view more information for 2.25/10 questions related to ALLFTD‐mApp procedures. Asymptomatic (N = 75), prodromal (N = 33), and symptomatic (N = 34) participants viewed additional information from 1.9, 2.8, and 2.7 questions, respectively (Figure S1). Most viewed were descriptions of where to locate Amazon gift codes (38.1% of respondents viewed) and notification schedule (30.4%), while least‐viewed items were how to log in (4.8%) and earn gift codes (14.4%).

3.3.2. Adherence (proportion of tasks completed)

Participants completed 70.3% of all available tasks. Asymptomatic, prodromal, and symptomatic participants completed 71.4%, 78.4%, and 59.0% of available tasks, respectively. Adherence in asymptomatic participants did not differ statistically from prodromal participants (beta: 7.8%, 95% CI [−5.4%, 20.9%], p = 0.25) nor symptomatic participants (beta: −12.3%, 95% CI [−26.5%, 2.0%], p = 0.09), though adherence in symptomatic participants was significantly lower than prodromal participants (beta: −20.1%, 95% CI [−34.6%, −5.5%], p = 0.007). Average adherence among participants who completed ≥ 25% of available tasks (N = 154) was 81.0% (asymptomatic = 83.0% (N = 75), prodromal = 84.0% (N = 38), symptomatic = 74.3% (N = 30)). Participants who initiated batteries tended to complete all available measures (72.9% of batteries fully completed), further explored by disease severity in Table S3. Among asymptomatic participants, adherence was 14.6% higher among females compared to males (95% CI [6.5%, 22.7%], p < 0.01). Neither age (beta = 0.2%, 95% CI [−0.1%, 0.4%], p = 0.23) nor education (beta = 0.4%, 95% CI [−1.6%, 2.4%], p = 0.70) were significantly associated with adherence.

Average adherence decreased across successive Sessions by 10.0% (95% CI [−13.3%, −6.7%], p < 0.01), though this was not statistically significant among asymptomatic participants (beta = −18.5%, 95% CI [−21.4%, 58.5%], p = 0.37).

3.4. Acceptability

3.4.1. Mobile App experience

This survey was administered to 123 participants who completed at least one Session. Figure 3 displays survey results for 113 respondents with known CDR®+NACC‐FTLD. Using a 5‐point scale, most respondents (93.5%) ranked the clarity of task instructions to be 3 (Neutral) or better. Seven of eight participants who rated the instructions as unclear/somewhat unclear were prodromal or symptomatic, and the two who rated the instructions as unclear were diagnosed with semantic variant PPA (Table S2). Participants reported a range of perceived difficulty in completing the tasks, though only 4.1% considered the tasks very difficult (all CDR®+NACC‐FTLD global > 0). Nearly all participants reported that the text size was comfortable to read always (89.4%) or most of the time (9.8%). Most participants (97.6%) reported that the time required to complete the tasks was acceptable, and 15.8% reported they would consider committing more time. Close to half (43.9%) reported that they would attempt to complete all tasks in future Sessions in one sitting, whereas 44.8% planned to spread their participation across multiple days.

FIGURE 3.

Participant ratings of their experience with the ALLFTD Mobile App. Note. Upon completion of each Session, participants were asked about their experience with the ALLFTD‐mApp. This figure displays results from the first available survey from 113 participants with a known CDR®+NACC‐FTLD score. Each bar displays the number of participant responses for each answer choice, color coded by disease severity (CDR®+NACC‐FTLD). The percentage of all participants with a known CDR®+NACC‐FTLD score who selected the response option is displayed at the top of each bar. CDR®+NACC‐FTLD = Clinical Dementia Rating Scale plus National Alzheimer's Coordinating Center Frontotemporal Lobar Degeneration module.

3.5. Cognitive test scores

Summary metrics for the executive functioning and memory tasks are displayed in Table 2. Greater disease severity (CDR®+NACC‐FTLD) was associated with stepwise declines in performance across all smartphone tests (p‐values < 0.001). In contrast, asymptomatic participants performed significantly better than symptomatic on the MoCA, but neither group differed from prodromal participants.

4. DISCUSSION

Remote data collection using the ALLFTD Mobile App (ALLFTD‐mApp), a comprehensive battery of cognitive, speech/language, and motor tasks, is both feasible and generally well accepted in the context of multicenter FTD research. Although some differences in familiarity were observed between age and diagnostic groups, smartphone familiarity was generally high among all participants, including older adults and symptomatic participants, mirroring recent findings showing high acceptability and adherence in older adult populations. ⁴⁷ , ⁴⁸ , ⁴⁹ Participants considered the time commitment to be acceptable, and many endorsed a willingness to complete additional tasks. Task completion rates were acceptable and consistent with studies in similar populations ⁴⁷ , ⁴⁸ . Preliminary group comparisons suggest that the ALLFTD‐mApp cognitive tests are sensitive to increasing disease severity, even in the mildest stages (CDR®+NACC‐FTLD Global = 0.5 vs. 0), which was not observed for the MoCA. Together, these findings suggest that the ALLFTD‐mApp may help address recruitment challenges in FTD research and facilitate more frequent longitudinal assessments.

Participants completed most tasks (70%), although a dichotomy in adherence rates (proportion of tasks completed) was observed, such that in the ∼ 80% of the sample who completed at least 25% of tasks, adherence was high (81.0% of total tasks), whereas adherence was < 10% in the remainder of the sample. Furthermore, completion rates declined across testing sessions, particularly in symptomatic cases, suggesting sustained engagement in this group may require adjustments to the study protocol. Adherence and retention is a well‐known hurdle to conducting remote and unsupervised longitudinal research ⁵⁰ , ⁵¹ and has ranged widely (60%‐100%) in prior studies of smartphone testing batteries. ⁴⁷ , ⁵² , ⁵³ , ⁵⁴ Our adherence rates are similar to other studies of smartphone cognitive measures among non‐FTD cohorts with similar testing volume, and notably, our study population included individuals with dementia (CDR®+NACC‐FTLD > 0.5) ⁴⁷ , ⁴⁸ . Additionally, greater perceived benefit of participation, such as when smartphone testing is integrating into a clinical trial, may improve adherence. ⁵⁵ , ⁵⁶ We plan to conduct participant interviews to identify strategies for improving adherence and longitudinal retention in this study population.

Participants reported the task instructions were clear, the difficulty level was variable, and the burden (∼30 min, three replicate batteries, over 2 weeks) acceptable, with many participants willing to commit additional time. Although adherence, technology familiarity, and acceptability responses were generally high among symptomatic individuals, this group also reported most of the lowest levels of responses in each category, indicating usability was affected by clinical stage. Further work is warranted to improve feasibility and acceptability among participants with dementia. Incorporating the voice of the patient and their family is critical for digital tools to be effective in clinical studies ⁵⁷ and to align with participants’ preferences regarding the use of digital biomarkers in FTD research and clinical care.

There are several limitations to this work. First, we remained blinded to mutation status; there may be informative differences between asymptomatic f‐FTD mutation carriers and non‐carriers. Second, although we recruited a geographically diverse sample, all participants resided in the United States and represented limited ethnocultural, socioeconomic, and language diversity. At the time of this study, only participants who spoke English with smartphone access were able to participate, limiting the generalizability of these findings. Given that clinical trials in FTD, particularly familial forms, will likely require global recruitment, we are preparing for global studies by translating the app into different languages, considering cross‐cultural threats to validity, and developing the workflow for provisioning smartphones to participants who need them. Finally, detailed reasons that participants were not enrolled was unavailable at the time of this study, but these data are being collected prospectively.

In summary, this study demonstrated the feasibility and acceptability of the ALLFTD‐mApp within a multicenter observational study of FTD. Remote smartphone batteries such as the ALLFTD‐mApp may become valuable tools for increasing enrollment in FTD research by enabling participation of those traditionally unable to complete in‐person visits due to transportation challenges or features of the disease (e.g., apathy, motor impairment). Future studies will investigate the reliability and validity of these measures to understand their utility for early detection and monitoring of the clinical features of FTD.

CONFLICT OF INTEREST STATEMENT

Appleby, BS – receives research support from CDC, NIH, Ionis, Alector, and the CJD Foundation. He has provided consultation to Acadia, Ionis, and Sangamo.

Boeve, BF – has served as an investigator for clinical trials sponsored by Alector, Biogen and Transposon. He receives royalties from the publication of a book entitled Behavioral Neurology Of Dementia (Cambridge Medicine, 2009, 2017). He serves on the Scientific Advisory Board of the Tau Consortium. He receives research support from NIH, the Mayo Clinic Dorothy and Harry T. Mangurian Jr. Lewy Body Dementia Program, and the Little Family Foundation.

Boxer, AL – receives research support from NIH (U19AG063911, R01AG038791, R01AG073482), the Tau Research Consortium, the Association for Frontotemporal Degeneration, Bluefield Project to Cure Frontotemporal Dementia, Corticobasal Degeneration Solutions, the Alzheimer's Drug Discovery Foundation, and the Alzheimer's Association. He has served as a consultant for Aeovian, AGTC, Alector, Arkuda, Arvinas, AviadoBio, Boehringer Ingelheim, Denali, GSK, Life Edit, Humana, Oligomerix, Oscotec, Roche, Transposon, TrueBinding and Wave, and received research support from Biogen, Eisai, and Regeneron. As a co‐inventor of ALLFTD‐mApp tasks, Dr. Boxer has received licensing fees.

Brushaber, D – nothing to disclose.

Clark, AL – nothing to disclose.

Dickerson, BC – Dr Dickerson is a consultant for Acadia, Alector, Arkuda, Biogen, Denali, Eisai, Genentech, Lilly, Merck, Novartis, Takeda, Wave Lifesciences. Dr Dickerson receives royalties from Cambridge University Press, Elsevier, Oxford University Press. Dr Dickerson receives grant funding from the NIA, NINDS, NIMH, and the Bluefield Foundation.

Domoto‐Reilly, K – receives research support from NIH, and serves as an investigator for a clinical trial sponsored by Lawson Health Research Institute.

Fields, JA – receives research support from NIH.

Forsberg, L – receives research support from NIH.

Ghoshal, N – has participated or is currently participating in clinical trials of anti‐dementia drugs sponsored by the following companies: Bristol Myers Squibb, Eli Lilly/Avid Radiopharmaceuticals, Janssen Immunotherapy, Novartis, Pfizer, Wyeth, SNIFF (The Study of Nasal Insulin to Fight Forgetfulness) study, and A4 (The Anti‐Amyloid Treatment in Asymptomatic Alzheimer's Disease) trial. She receives research support from Tau Consortium and Association for Frontotemporal Dementia and is funded by the NIH.

Gorno‐Tempini, ML – receives research support from NIH. As a co‐inventor of one of the ALLFTD mApp tasks, Dr Gorno‐Tempini's lab receives licensing fees , consistent with UCSF institutional policy.

Graff‐Radford, N – receives royalties from UpToDate, has participated in multicenter therapy studies by sponsored by Biogen, TauRx, AbbVie, Novartis, and Lilly. He receives research support from NIH.

Grossman, M – receives grant support from NIH, Avid, and Piramal; participates in clinical trials sponsored by Biogen, TauRx, and Alector; serves as a consultant to Bracco and UCB; and serves on the Editorial Board of Neurology.

Hall, MGH – nothing to disclose.

Heuer, HW – nothing to disclose.

Huey, ED – receives research support from NIH.

Irwin, D – receives support from NIH, Brightfocus Foundation, and Penn Institute on Aging.

Kornak, J – has provided expert witness testimony for Teva Pharmaceuticals in Forest Laboratories Inc. et al. v. Teva Pharmaceuticals USA, Inc., Case Nos. 1:14‐cv‐00121 and 1:14‐cv‐00686 (D. Del. filed Jan. 31, 2014, and May 30, 2014) regarding the drug Memantine; for Apotex/HEC/Ezra in Novartis AG et al. v. Apotex Inc., No. 1:15‐cv‐975 (D. Del. filed Oct. 26, 2015, regarding the drug Fingolimod. He has also given testimony on behalf of Puma Biotechnology in Hsingching Hsu et al, vs. Puma Biotechnology, INC., et al. 2018 regarding the drug Neratinib. He receives research support from the NIH.

Kramer, J – receives research support from NIH and royalties from Pearson Inc.

Kremers, W—receives research funding from AstraZeneca, Biogen, Roche, DOD, and NIH.

Lapid, MI – receives research support from the NIH.

Litvan, I – receives research support from the National Institutes of Health grants: 2R01AG038791‐06A, U01NS100610, U01NS80818, R25NS098999; U19 AG063911‐1 and 1R21NS114764‐01A1; the Michael J Fox Foundation, Parkinson Foundation, Lewy Body Association, CurePSP, Roche, Abbvie, Biogen, Centogene. EIP‐Pharma, Biohaven Pharmaceuticals, Novartis, and United Biopharma SRL‐UCB. She is a Scientific advisor for Amydis (Gratis) and Rossy Center for Progressive Supranuclear Palsy University of Toronto. She receives her salary from the University of California San Diego and as Chief Editor of Frontiers in Neurology.

Mackenzie, IR – receives research funding from Canadian Institutes of Health Research, Alzheimer's Association US, NIH, Weston Brain Institute.

Manoochehri, M – nothing to disclose.

Masdeu, JC – is a consultant and received research funding from Eli Lilly, parent co. of Avid Radiopharmaceuticals, manufacturer of 18F‐flortaucipir, receives personal fees from GE Healthcare, grants and personal fees from Eli Lilly, grants from Acadia, Avanir, Biogen, Eisai, Janssen, NIH, Novartis, with no relation to the submitted work.

Mendez, MF – receives research support from NIH.

Mester, C – nothing to disclose.

Nevler, N – receives research funding from the NIH and Department of Defense.

Onyike, C – receives research funding from the NIH, Lawton Health Research Institute, National Ataxia Foundation, Alector Inc., and Transposon, Inc. He is also supported by the Robert and Nancy Hall Brain Research Fund, the Jane Tanger Black Fund for Young‐Onset Dementias, and the gift from Joseph Trovato. He is a consultant with Alector, Inc. and Acadia Pharmaceuticals.

Pascual, B – receives research support from NIH.

Pressman, PS – receives research support from NIH.

Rankin, KP – receives research support from NIH and NSF, and serves on a Medical Advisory Board for Eli Lilly.

Rao, M – nothing to disclose.

Rojas, JC – receives research support from NIH and is a site PI for clinical trials sponsored by Eli‐Lilly and Eisai.

Rosen, HJ – has received research support from Biogen Pharmaceuticals, has consulting agreements with Wave Neuroscience and Ionis Pharmaceuticals, and receives research support from NIH.

Ratnasiri, B – nothing to disclose.

Staffaroni, AM – received research support from the NIA/NIH, Bluefield Project to Cure FTD, and the Larry L. Hillblom Foundation, and has provided consultation to Alector, Lilly/Prevail, Passage Bio, and Takeda. Dr Staffaroni is a co‐inventor of four ALLFTD mApp tasks and receives licensing fees, consistent with UCSF institutional policy.

Tartaglia, M – has served as an investigator for clinical trials sponsored by Biogen, Avanex, Green Valley, and Roche / Genentech, Bristol Myers Squibb, Eli Lilly/Avid Radiopharmaceuticals, Janssen. She receives research support from Canadian Institutes of Health Research. Author disclosures are available in the supporting information.

Taylor, JC – nothing to disclose.

Wise, AB – nothing to disclose.

Welch, AE – nothing to disclose.

Wong, B – receives research support from the NIH.

ALLFTD Investigators

Liana Apostolova MD¹, Brian Appleby MD², Sami Barmada MD, PhD³, Ece Bayram MD, PhD⁴, Bradley F. Boeve MD⁵, Hugo Botha MD⁵, Adam L. Boxer MD PhD⁶, Andrea Bozoki MD⁷, Danielle Brushaber BS⁸, David Clark MD¹, R. Ryan Darby MD⁹, Gregg S. Day MD, MSc¹⁰, Bradford C. Dickerson MD¹¹, Dennis Dickson MD¹⁰, Kimiko Domoto‐Reilly MD MS¹², Kelly Faber MS¹³, Anne Fagan MS¹⁴, Julie A. Fields PhD¹⁵, Tatiana Foroud PhD¹⁶, Leah Forsberg PhD⁵, Douglas Galasko MD⁴, Ralitsa Gavrilova MD⁵, Tania Gendron PhD¹⁰, Daniel Geschwind MD, PhD¹⁷, Nupur Ghoshal MD PhD¹⁴, Jill Goldman MS, Mphil¹⁸, Neill Graff‐Radford MD¹⁰, Jonathan Graff‐Radford MD⁵, Ian M. Grant MD¹⁹, Murray Grossman MD EdD²⁰, Matthew GH Hall MS⁶, Hilary W. Heuer PhD⁶, Ging‐Yuek (Robin) Hsiung MD²¹, Eric Huang PhD⁶, Edward D. Huey MD²², David Irwin MD²⁰, Noah Johnson BA⁸, David T. Jones MD⁸, Kejal Kantarci MD⁵, David Knopman MD⁵, Tyler Kolander BA⁵, John Kornak PhD²³, Walter Kremers PhD⁸, Argentina Lario Lago PhD⁶, Maria I. Lapid MD¹⁵, Gabriel C. Léger MD⁴, Irene Litvan MD²⁴, Peter Ljubenkov MD⁶, Diane Lucente MS²⁵, Ian R. Mackenzie MD²⁶, Masood Manoochehri BA²², Joseph C. Masdeu MD PhD²⁷, Lauren Massimo PhD²⁰, Scott McGinnis MD²⁵, Mario F. Mendez MD PhD²⁸, Carly Mester BA⁸, Bruce L. Miller MD⁶, Chiadi U. Onyike MBBS MHS²⁹, Belen Pascual PhD²⁷, Henry Paulson MD, PhD³, Leonard Petrucelli PhD¹⁰, Peter Pressman MD³⁰, Rosa Rademakers PhD³¹, Vijay Ramanan MD, PhD⁵, Eliana Marisa Ramos PhD³², Katherine P. Rankin PhD⁶, Katya Rascovsky PhD²⁰, Aaron Ritter MD³³, Erik D. Roberson MD, PhD³⁴, Emily Rogalski PhD¹⁹, Howard J. Rosen MD⁶, Rodolfo Savica MD, PhD⁵, William Seeley MD⁶, Adam M. Staffaroni PhD⁶, Jeremy M. Syrjanen MS⁵, Maria Carmela Tartaglia MD FRCPC³⁵, Jack Carson Taylor MAS⁶, Philip W. Tipton MD³⁶, Marijne Vandebergh PhD³¹, Lawren VandeVrede MD, PhD⁶, Sandra Weintraub PhD¹⁹, Dylan Wint MD³³, Bonnie Wong PhD¹¹

¹Department of Neurology, Indiana University, Indianapolis, IN, 46202, USA

²Department of Neurology, Case Western Reserve University, Cleveland, OH, 44106, USA

³Department of Neurology, University of Michigan, Ann Arbor, MI, 48109, USA

⁴Department of Neurosciences, University of California, San Diego, La Jolla, CA, 92037, USA

⁵Department of Neurology, Mayo Clinic, Rochester, MN, 55905, USA

⁶Department of Neurology, Memory and Aging Center, University of California, San Francisco, Weill Institute for Neurosciences, San Francisco, CA, 94158, USA

⁷Department of Radiology, University of North Carolina, Chapel Hill, NC, 27599, USA

⁸Department of Quantitative Health Sciences, Division of Biomedical Statistics and Informatics, Mayo Clinic, Rochester, MN, 55905, USA

⁹Department of Neurology, Vanderbilt University, Nashville, TN, 37212, USA

¹⁰Department of Neuroscience, Mayo Clinic, Jacksonville, FL, 32224, USA

¹¹Department of Neurology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, 02114, USA

¹²Department of Neurology, University of Washington, Seattle, WA, 98195, USA

¹³National Centralized Repository for Alzheimer's, Indiana University, Indianapolis, IN, 46202, USA

¹⁴Department of Neurology, Center for Advanced Medicine Memory Diagnostic Center, Washington University, Saint Louis, MO, 63110, USA

¹⁵Department of Psychiatry and Psychology, Mayo Clinic, Rochester, MN, 55905, USA

¹⁶Department of Medical and Molecular Genetics, Indiana University, Indianapolis, IN, 46202, USA

¹⁷Department of Neurology, Institute for Precision Health, University of California, Los Angeles, Los Angeles, CA, 90095, USA

¹⁸Department of Neurology, Taub Institute for Research on Alzheimer's Disease and the Aging Brain, New York, NY, 10033, USA

¹⁹Department of Psychiatry and Behavioral Sciences, Mesulam Center for Cognitive Neurology and Alzheimer's Disease, Northwestern Feinberg School of Medicine, Chicago, IL, 60610, USA

²⁰Department of Neurology, University of Pennsylvania, Philadelphia, PA, 19104, USA

²¹Division of Neurology, University of British Columbia, Musqueam, Squamish & Tsleil‐Waututh Traditional Territory, Vancouver, BC, V6T 2B5, Canada

²²Department of Neurology, Columbia University, New York, NY, 10033, USA

²³Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, CA, 94158, USA

²⁴Department of Neurosciences, University of California, San Diego, La Jolla, CA, 92093, USA

²⁵Department of Neurology, MGH Frontotemporal Disorders Unit, Massachusetts General Hospital, Charlestown, MA, 02129, USA

²⁶Department of Pathology, University of British Columbia, Vancouver, BC, V6T 1Z7, Canada

²⁷Department of Neurology, Houston Methodist, Houston, TX, 77030, USA

²⁸Department of Neurology, University of California, Los Angeles, Los Angeles, CA, 90095, USA

²⁹Department of Psychiatry and Behavioral Sciences, Johns Hopkins University, Baltimore, MD, 21205, USA

³⁰Department of Neurology, University of Colorado, Aurora, CO, 80045, USA

³¹VIB Center for Molecular Neurology, University of Antwerp, Antwerpen, Belgium

³²Department of Neurology, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, 90024, USA

³³Cleveland Clinic Lou Ruvo Center for Brain Health, Cleveland Clinic Las Vegas, Las Vegas, NV, 89106, USA

³⁴Department of Neurology, University of Alabama, Birmingham, AL, 35233, USA

³⁵Tanz Centre for Research in Neurodegenerative Diseases, Division of Neurology, University of Toronto, Toronto, Ontario, M5T 2S8, Canada

³⁶Department of Neurology, Mayo Clinic, Jacksonville, FL, 32224, USA

Supporting information

Supplementary Information

Supplementary Information

Taylor JC, Heuer HW, Clark AL, et al. Feasibility and acceptability of remote smartphone cognitive testing in frontotemporal dementia research. Alzheimer's Dement. 2023;15:e12423. 10.1002/dad2.12423