Abstract
Frequent, brief and repeated self‐administered mobile assessments of cognitive function, conducted in everyday life settings, are a promising complementary tool to traditional assessment approaches. Mobile cognitive assessments promote patient‐centered care and might enhance capacity to inform individual‐level outcomes over time (i.e. detecting subtle declines in cognitive function), as well as in assessing cognition and its correlates in the naturalistic environment. The goal of this systematic review was to assess the feasibility and psychometric properties of mobile cognitive assessments. Through a comprehensive search, we identified 12 articles using self‐administered, mobile phone‐based cognitive assessments. Studies sampled participants between 1 and 6 times per day for 1–14 days. Samples ranged in age from 14 to 83 years old and were generally healthy populations without cognitive impairment. Working memory was the most frequently‐assessed cognitive domain (n = 7), followed by attention/reaction time (n = 4). Seven studies reported adherence, with mean adherence rates of 79.2%. In addition to positive evidence of feasibility, there was general support for high levels of between‐ and within‐person reliability and construct validity. While research has only begun to explore the utility of mobile cognitive assessments, studies to‐date indicate they may be a promising complementary tool to traditional assessment methods with potential to improve clinical care and research.
Keywords: cognition, ecological momentary assessment, patient‐centered care, repeated sampling, technology
1. INTRODUCTION
Ecological momentary assessment (EMA), a method involving frequent, repeated assessment of thoughts, feelings, experiences and behaviors in the naturalistic environment, has contributed to the understanding of many psychiatric disorders (Cain, Depp, & Jeste, 2009; Ebner‐Priemer, Eid, Kleindienst, Stabenow, & Trull, 2009; Granholm, Loh, & Swendsen, 2008; Shiffman, Stone, & Hufford, 2008). Depending on the research question, these assessments may be event‐based, time‐based, or randomly prompted. EMA possesses a number of advantages over typical point‐in‐time assessments, which include: (1) the reduction of retrospective recall biases, (2) the ability to model within‐person trends over time, (3) the assessment of temporal sequences among theoretically‐linked constructs (Depp et al., 2016a; Depp et al., 2016b), (4) the generation of more stable estimates of constructs for outcomes research (Moore, Depp, Wetherell, & Lenze, 2016), (5) the capacity to monitor treatment response or progress in real‐time, and (6) the assessment of phenomena as they occur in real‐world settings. The ability to assess fluctuation and change in affect, personality, and behavior over time, known as dynamic processes, represents one of the greatest strengths of EMA. For the interested reader, Trull, Lane, Koval, and Ebner‐Priemer (2015) provide an excellent review of assessing affective dynamics via EMA. The constructs most frequently assessed in EMA studies include self‐reports (e.g. social context, symptoms and behavior) and such subjective information has increasingly been linked to objective data, such as those derived from activity sensors (e.g. actigraphy; see Carpenter, Wycoff, and Trull [2016] for a review of new data collection methods). However, only a handful of studies have used an EMA paradigm to objectively assess cognitive functioning, despite that many of the same advantages would be potentially applicable in comparison to traditional in‐person cognitive testing. In this systematic review, we summarize research that has employed EMA paradigms in assessing real‐world cognitive functioning.
There are several reasons why EMA could be particularly beneficial in assessing cognition. First, many research questions concern change over time in cognitive ability, both in the short term and the long term, and tools to enable longitudinal monitoring of cognitive ability would facilitate better understanding of the dynamics of cognition and their determinants. Understanding the short‐term dynamics of cognition in the context of critical periods, such as after injuries or onset of illness, or in response to treatments, could provide novel insights that are difficult to capture with panel‐type studies of cognitive ability. Second, there is a growing body of literature indicating that intra‐individual variability in cognitive functioning is linked with brain illnesses and change over time in a way that is somewhat independent from levels of performance (Hultsch, Strauss, Hunter, & MacDonald, 2008). Third, traditional cognitive testing occurs in controlled, artificial environments, and for many psychiatric phenomena there is interest in understanding the temporal (e.g. diurnal rhythms) or social‐contextual influences on cognitive functioning as individuals interact with the real‐world environment. Several studies have developed and tested repeatable cognitive tests on a computerized or mobile platform that are delivered in a fixed environment, such as a person's home. While cognitive assessments administered in such environments can have advantages over laboratory‐based assessments, limitations include being administered at a predetermined time and in a quiet, unsupervised environment. Mobile cognitive tests, however, can be delivered at any time during a person's daily life. Given these tests occur while a person is contending with all of the distractions and constraints of daily life, the data may better characterize “typical” cognitive functioning as opposed to cognitive functioning under optimized conditions. Moreover, given the ubiquity of mobile devices in daily life, it is now possible to administer measures throughout the day rather than only when respondents are restricted to the home environment. Additionally, other data that is gathered by the mobile device (accelerometry, location) can be used in concert with the cognitive data in mobile cognitive testing. Finally, even though standard cognitive testing occurs in controlled settings, there are sources of error (e.g. waxing/waning motivation, mood symptoms) that contribute to diminished test–retest reliability. Aggregated within‐person estimates of cognitive functioning may produce more stable estimates of these variables by collecting data on the same test over time, as seems likely in symptom measures (Moore et al., 2016).
Concerning computerized cognitive assessment, the translation of clinical instruments to computer‐based platforms has demonstrated certain limitations (Bauer et al., 2012). Unsupervised cognitive testing does not easily enable standardized observations of effort and attention, and there is greater possibility for task distraction, which may diminish performance or lead to artificial inflation of performance through “cheating”. Measures ported to mobile testing environments may alter the nature of the test and therefore make cross‐mode validation challenging. It is also unclear whether certain cognitive domains are more amenable to EMA‐based cognitive tests than are others. In addition, the patient‐ and test‐related determinants of adherence to protocols and participant acceptance of repeated cognitive testing across clinical and non‐clinical populations is poorly understood.
To better understand the promise and potential pitfalls of mobile cognitive testing, we conducted a systematic review of research studies that employed repeated mobile device‐based cognitive testing in clinical and non‐clinical populations. We examined the feasibility of mobile cognitive assessments, as indicated by the rate and documented predictors of adherence, as well as on summarizing the specific applications, research design, cognitive domains assessed, and measured reliability and construct validity in these studies. Our objective was to summarize the current literature on this approach, in order to identify gaps and opportunities for future development in this area.
2. METHODS
2.1. Article selection and search strategy
In order to identify articles for the review, we first searched the PubMed database using the search terms mobile, cognitive, assessment, ecological momentary assessment, smartphone, and personal digital assistant. The search was completed on August 30, 2016. Second, we reviewed the reference lists of retrieved articles to find earlier relevant studies (“ancestry approach”; Cooper, 1998). We restricted our searches to studies published in peer‐reviewed English‐language journals. No age restriction was placed on the samples, nor were restrictions placed on the minimum number of participants. All relevant and accessible journal articles that assessed the use of mobile cognitive assessments were included. Studies selected needed to contain repeated mobile cognitive assessments that were self‐administered on a mobile device. Studies that used cognitive assessments delivered by research staff on mobile devices (e.g. the NIH Toolbox; Weintraub et al., 2013), regardless of the setting, or administered on a tablet (e.g. iPad) instead of an ambulatory mobile device, were excluded.
2.2. Data extraction and synthesis
We developed a database to organize information on: (a) author names; (b) publication year; (c) sample characteristics (sample size, study population, and demographic characteristics); (d) cognitive domains assessed; (e) duration of the study; (f) timing and completion time of the mobile tests; (g) device used; (h) compliance; (i) study type; (j) major findings. Age and adherence data were weighted by sample size: for each study, sample size was multiplied by mean age or mean study adherences ratings then divided by the total sample size across all studies. Mean and standard deviation (SD) values were calculated from the weighted scores.
3. RESULTS
3.1. Studies selected
A total of 87 unique studies were identified. After reviewing abstracts and subsequently full‐text articles and their reference sections, 12 articles were selected for this review (see Figure 1).
Figure 1.

Flowchart of detailed search strategy
3.2. Sample characteristics
Characteristics of the 12 studies included in this review are presented in Table 1. The number of participants in these studies ranged from 20 to 219 (mean = 73.8, SD = 66.5, median = 50.5). Weighted by sample size, mean age of participants was 46.9 years old. For studies that reported the participants' age range, the range was from 14 to 83 years. Eight studies were conducted in European countries, and four in the United States. A majority of studies targeted healthy, community‐dwelling volunteers; only four studies selected patients based on illness or behavioral foci (adults with epilepsy; pre‐surgical patients; adults who consumed >5 units of alcohol per week; adult smokers).
Table 1.
Characteristics of the 12 studies
| Author/year | N | Sample population | Study location | Mean age (years) (SD; range) | Percentage female | Mobile test(s) constructs |
Timing |
Number of days | Completion time | Device | Compliance (%) | Study type |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Allard et al., 2014 | 60 | Non‐demented elderly agricultural workers | Rural France |
75 (SD: 4.7; range: not provided) |
45.0 | Semantic memory | 2 times/day | 7 | ≤2 minutes | PDA | 78.6 | Observational |
| Brouillette et al., 2013 | 57a | Non‐demented older volunteers | United States | 67 (SD: 1; range: not provided) | 70.2 | Processing speed | 1 time/day | 1 | 2 minutes | Smartphone | N/A | Observational |
| Frings et al., 2008 | 20 | Adults with epilepsy | Germany | 42 (SD: 16.5; range: not provided) | Not provided | Working memory | 3 times/day | 6 | Not provided | PDA | Not provided | Observational |
| Kennedy et al., 2011 | 198 | Healthy, employed males | United Kingdom | 39 (SD: 6.8; range: not provided) | 0.0 |
(1) reaction time (2) working memory |
2 times/day | 5b | Not provided | Mobile phone | Not provided | Randomized controlled trial |
| Riediger et al., 2014 | 89 | Community‐dwelling adolescent and adult volunteers | Germany | 42 (SD: 19.0; range: 14–83) | 55.0 | Working memory | 6 times/day | 2 | Not provided | Mobile phone | 58.3 | Observational |
| Schuster et al., 2015 | 39 | Adolescent volunteers at high‐risk for smoking | United States | 21 (0.7; range: 19–22) | 48.7 | Working memory (visual) | Variable: structured and context dependent | 7 | 40 seconds | PDA | 87 | Observational |
|
Schweitzer et al., 2016 |
75 | Non‐demented elderly agricultural workers | Rural France | 76.9 (SD: 1.3; range: not provided) | 57 |
(1) semantic memory (2) episodic memory – immediate free recall, recognition, and delayed recall (3) executive functions |
5 times/test/week | 7 | ≤2 minutes | Smartphone | 82 | Observational |
| Sliwinski et al., 2016 | 219 | Healthy volunteers | United States | 47 (SD: 10.7; range: 25–65) | 66 |
(1) processing speed (2) working memory (2 tasks; 1 visual) |
5 times/day | 14 | <3 minutes | Mobile phone | 85 | Observational |
| Timmers et al., 2014 | 26c | Community‐dwelling volunteers | The Netherlands | 23 (SD: 3.0; range: 19–29) | 50 | Working memory | 4 times/day | 1 | Not provided | Smartphone | Not provided | Observational |
| Tiplady et al., 2009 | 38 | Adult volunteers with >5 units/week alcohol use | United Kingdom | 22.8 (SD not provided; range: 18–54) | 47.4 |
(1) attention (2) working memory |
2 times/day | 14 | Not provided | Mobile phone | 80.3 | Observational |
| Thomson et al., 2009 | 20 | Pre‐surgical patients | United Kingdom | 46 (10; range: 28–64) | 46 | Visual reaction time | Variable; conducted in laboratoryd | 1 | 2 minutes | Mobile phone | Not provided | Observational |
| Waters & Li, 2008 | 44 | Smoker and non‐smoker adults | United States | 36.4 (SD not provided; range: 18–65) | 56.8 | Reaction time | 4 times/day plus when reported feeling anxious | 7 | 4 minutes | PDA | 81.2e | Observational |
Note: SD, standard deviation; PDA, personal digital assistant; N/A, not available.
All 57 participants completed the mobile cognitive assessments in the laboratory and a subset of 11 participants also completed them in the real‐world to establish test–retest reliability.
Mobile cognitive assessments administered once per week for five weeks.
Thirteen participants completed mobile cognitive assessments in the laboratory and 13 participants completed the assessments in the real‐world.
Mobile cognitive assessments completed as part of an in‐laboratory sedation drug trial.
Participants responded to 81.2% of random prompts.
Five studies excluded for a diagnosis of dementia, cognitive impairment, or a central nervous system compromising disorder/neurological condition (e.g. cerebrovascular disease, Parkinson's disease). No other common exclusionary criteria in these reports were observed.
3.3. Data capture methods, data capture schedules, and timing and duration
In general, studies conducted prior to 2009 used personal digital assistants (PDAs; n = 4) and studies after 2009 used a mobile phone (n = 8). Four studies specified if the mobile phone was a smartphone (Brouillette et al., 2013; Schweitzer et al., 2016; Sliwinski et al., 2016; Timmers et al., 2014), while the other studies using mobile phones either did not use smartphones or did not make this specification. In all but one study (Brouillette et al., 2013) the mobile devices were provided to the participants. Regarding the technical implementation aspects of the assessments, the investigative teams completed custom programming for all of the mobile cognitive tests. Four studies reported that the tests were pre‐installed on the device, whereas the other studies did not provide this information. Three studies specified having disabled phones with no internet or data plans; the other studies did not specify whether data plans were necessary for test administration. Across studies, data was either stored locally on the devices or uploaded to a server (e.g. SQL server).
In terms of data capture schedules, a majority of the studies used a fixed‐schedule design (n = 9), in which participants responded to an alarm at predetermined and consistent intervals. Two studies used a combination of the fixed‐schedule and event‐based design. Schuster, Mermelstein, and Hedeker (2015) used a combination of random prompts (5–7 times per day), subject‐initiated events (immediately after participant smoked a cigarette and when participant wanted to smoke a cigarette but were unable to do so due to situational restrictions). Waters and Li (2008) had participants complete reaction time tests on a random, four‐times per day schedule, as well as subject‐initiated tests (when feeling anxious). Lastly, one study used a purely event‐based design, in which pre‐surgical patients completed a mobile visual reaction time test after increases in sedation via propofol, so the number of assessments completed across patients varied (Thomson, Nimmo, Tiplady, & Glen, 2009).
Timing of the studies varied from one to 14 days, with an average study duration of six days (SD = 4.5, median = 6.5 days). Of the studies that reported completion times of the mobile cognitive tests, the longest completion time was four minutes (range = 40 seconds to 4 minutes). We did not observe a pattern between duration of the study and frequency of mobile cognitive tests. For example, there were two studies with a 14‐day duration. In one of these studies, participants completed cognitive tests five times per day (Sliwinski et al., 2016), while in the other study participants only completed cognitive tests twice daily (Tiplady, Oshinowo, Thomson, & Drummond, 2009).
3.4. Cognitive domains assessed
A summary of research questions and neuropsychological domains assessed are presented in Tables 1 and 2. Eight studies administered a single mobile cognitive task for the duration of the study, while the other four studies administered more than one task. Tasks were unique in each study; while similar tasks were observed across some studies, no two studies used the same version of a given task. Cognitive domains assessed included: working memory (n = 7), attention/reaction time (n = 4), processing speed (n = 2), semantic memory (n = 2), short‐term memory (n = 1), delayed memory (n = 1), and executive functions (n = 1). Each study reported developing these instruments “in house”, and none of the tests are publically available in app stores or online test catalogs such as Pearson Assessments or Psychological Assessment Resources, Inc. (PAR, Lutz, FL, USA). Investigators discussed the importance of adapting the tests to the limited size of screens and displaying only enough characters to be easily legible (e.g. see Tiplady et al., 2009). For most studies, response selections were made with touch screens (or stylus for PDA studies) and automated scoring was conducted; one variation was in the Schweitzer et al. (2016), in which participants' responded orally and responses were recorded on the smartphones for later scoring.
Table 2.
Major findings from studies
| Major findings | ||
|---|---|---|
| Non‐clinical populations | ||
| Allard et al., 2014 | Non‐demented elderly agricultural workers |
1) Validation of mobile semantic memory with two laboratory semantic memory tests (Wechsler similarities: γ = 0.32, t = 2.10, p < 0.05; Isaacs set test at 15 seconds: γ = 0.17, t = 4.92, p < 0.001; Isaacs set test at 30 seconds: γ = 0.12, t = 5.37, p < 0.001; and Isaacs set test at 60 seconds: γ = 0.07, t = 4.74, p < 0.001). 2) Mobile, but not laboratory, semantic memory, related to hippocampal volume (left hippocampus: γ = 0.00, t = 2.95, p < 0.01; right hippocampus: γ = 0.00, t = 2.94, p < 0.01). 3) EMA reports of engagement in intellectually stimulating activities prospectively associated with increases in subsequent mobile semantic memory performance (γ = 0.89, t = 2.43, p < 0.05). |
| Brouillette et al., 2013 | Non‐demented older volunteers |
1) Good test–retest reliability for laboratory‐based and real‐world performance of mobile processing speed test (r = 0.73, p = 0.02). 2) Convergent validity of mobile processing speed test with standard laboratory‐assessed global cognition (mini mental status examination; r = 0.52, p < 0.001) and measures of processing speed (digit symbol: r = 0.51, p < 0.001; trails making test A: r = ˗0.65, p < 0.001) and attention (digit span forward: r = 0.43, p < 0.001; digit span backward: r = 0.43, p < 0.001). 3) Discriminant validity of mobile processing speed test unrelated to laboratory tests naming (Boston naming test: r = 0.024, p > 0.05), verbal fluency (animal list generation: r = 0.15, p > 0.05; vegetable list generation: r = 0.02, p > 0.05), and memory (logical memory I: r = 0.22, p > 0.05; logical memory II: r = 0.15, p > 0.05). |
| Kennedy et al., 2011 | Healthy, employed males | 1) Study had two groups: placebo control or multi‐vitamin mineral supplement. No group differences in mobile reaction time or working memory tests (test statistics not reported), but did find group differences on EMA items of subjective cognition (assessment × treatment; participants in the multi‐vitamin/mineral group rated better ability to concentrate: F = 6.17, p = 0.01). |
| Riediger et al., 2014 | Community‐dwelling adolescent and adult volunteers |
1) EMA‐reported feeling nervous related to worse concurrent mobile working memory performance among middle‐aged and older adults, but not among younger adults (estimate = ˗0.074, SE = 0.038, p < 0.01). 2) Increased momentary heart rate also related to worse concurrent mobile working memory performance among middle‐aged and older adults, but not among younger adults (estimate = ˗0.009, SE = 0.005, p < 0.05). |
| Schuster et al., 2015 | Healthy, adolescent volunteers at high‐risk for smoking |
1) Demonstrated feasibility of mobile visual working memory test among adolescents/young adults. 2) Convergent validity of mobile visual working memory test with laboratory measures of auditory and visual working memory (WAIS‐IV digit span: r = 0.42, p < 0.01; WAIS‐IV letter‐number sequencing: r = 0.46, p < 0.01; WAIS‐IV working memory index: r = 0.44, p < 0.01; WMS‐IV spatial addition: r = 0.53, p < 0.001; WMS‐IV symbol span; r = 0.60, p < 0.001; WMS‐IV visual working memory index: r = 0.64, p < 0.001). 3) Discriminate validity of mobile visual working memory test unrelated to laboratory measures of verbal abilities or processing speed (WAIS‐IV similarities: r = 0.28, p > 0.09; WAIS‐IV symbol search: r = 0.23, p > 0.09; WAIS‐IV coding: r = 0.25, p > 0.09; WAIS‐IV processing speed index: r = 0.27, p > 0.09). |
|
Schweitzer et al., 2016 |
Non‐demented elderly agricultural workers |
1) Demonstrated feasibility of mobile cognitive tests among older adults. 2) No fatigue effects associated with study duration. 3) Practice effects observed for semantic memory, immediate recognition, and delayed free‐recall scores. 4) Convergent validity of mobile semantic memory (Coeff =0.09, T ratio = 5.67, p < 0.001), immediate free recall (Coeff =0.06, T ratio = 2.52, p = 0.01), recognition (Coeff =0.08, T ratio = 2.28, p = 0.03), delayed recall (Coeff =0.23, T ratio = 2.32, p = 0.02), and executive functions (Coeff =0.18, T ratio = 4.83, p < 0.001) tests with laboratory measures of the same constructs. Controlled for practice effects, age and sex on mobile tests. |
| Sliwinski et al., 2016 | Healthy volunteers |
1) Excellent between‐person variance for brief battery of mobile working memory and perceptual speed tests (symbol search: 0.44; dot memory: 0.77; n‐back: 002). 2) Reliable within‐person variance for brief battery of mobile working memory and perceptual speed tests (symbol search: 0.36; dot memory: 1.19; n‐back: 0.01). 3) Convergent validity for mobile working memory and perceptual speed tests with laboratory measures of the same constructs (in‐laboratory speed tasks and mobile dot memory: r = 0.61–0.74, p < 0.01; in‐laboratory working memory tasks and mobile dot memory: r = ˗0.39 to ˗0.45, p < 0.01) in‐laboratory working memory tasks and n‐back: r = 0.24–0.36, p < 0.01). |
| Timmers et al., 2014 | Community‐dwelling volunteers |
1) Study had two groups: participants who repeatedly performed a short‐term memory test in the laboratory (controlled environment) and participants who repeatedly performed the short‐term memory test in their everyday life (uncontrolled environment). No group differences in task performance (F = 0.28, p > 0.60). 2) Test–retest reliability in uncontrolled environment condition (ICC = 0.76) better than in controlled condition (ICC = 0.52). |
| Clinical populations | ||
| Frings et al., 2008 | Adults with epilepsy |
1) Study had two epilepsy groups: patients being titrated with levetiracetam (LEV) and a control group of patients on stable treatment with antiepileptic drugs. Demonstrated feasibility of mobile assessment of working memory during titration of anti‐epileptic drugs. 2) No group differences in overall mobile working memory test (test statistics not reported). 3) Group × day interaction with control participants performed better than LEV group on mobile working memory task during the morning assessment (test statistics not reported). |
| Tiplady et al., 2009 | Adult volunteers with >5 units/week alcohol use |
1) Both mobile (memory scanning n incorrect: t = 3.37, p < 0.01; sustained attention to response task [SART] false negatives: t = 2.39, p = 0.02; number pairs n incorrect: t = 2.87, p < 0.01) and laboratory‐based (memory scanning n incorrect: t = 6.05, p = 0.02; SART false positives: t = 9.88, p < 0.01; number pairs n incorrect: t = 10.97, p < 0.01) attention and working memory tests related to alcohol use. 2) Expected diurnal changes in performance on the attention tests (memory scanning reaction time: F(37,4) = 2.58, p = 0.04; number pairs: F(37,4) = 4.75, p < 0.01). |
| Thomson et al., 2009 | Pre‐surgical patients |
1) Time to complete mobile visual reaction time test steadily increased as participants became more sedated via propofol (no effect of propofol on visual reaction time until propofol at or above 0.9 μg ml˗1; p < 0.05). 2) Reaction times on the mobile visual reaction time test more sensitive at higher levels of sedation than patient‐reported sedation scores (p < 0.001); this mobile test may be useful for identifying imminent over‐sedation. |
| Waters & Li, 2008 | Smoker and non‐smoker adults |
1) Demonstrated feasibility of mobile reaction time tests (classic‐Stroop, emotional‐Stroop, & smoking‐Stroop tasks) among smokers and non‐smokers. 2) Classic‐Stroop effect associated with age (SE = 1.86, p < 0.05), emotional‐Stroop effect associated with state anxiety (SE = 8.06, p < 0.01), and smoking‐Stroop effect associated with the Fagerstrom test for nicotine dependence scores (SE = 7.45, p < 0.01). 3) Adequate internal reliability of mobile reaction time tests (large classic‐Stroop effects for smokers (mean estimate [ME] = 156.3 ms, SE = 12.8, p < 0.001) and non‐smokers (ME =115.8 ms, SE = 11.6, p < 0.001; smaller emotional‐Stroop effects for smokers (ME =12.6 ms, SE = 5.74, p < 0.05) and non‐smokers (ME =21.0 ms, SE = 6.36, p < 0.01; no significant smoking‐Stroop effects for smokers or non‐smokers [p > 0.01]). |
Note: ICC, intraclass correlation coefficient; SE, standard error; WAIS‐IV, Wechsler Adult Intelligence Scale – Fourth Edition; WMS‐IV, Wechsler Memory Scale – Fourth Edition.
Working memory paradigms included variations of the n‐back working memory task (Frings et al., 2008; Kennedy et al., 2011; Sliwinski et al., 2016), a variation of Salthouse, Babcock, and Shaw's (1991) Numerical‐Memory Updating Task (Riediger et al., 2014), variations of Ickikawa's (1983) Dot Memory Task and Milner's (1971) Corsi Blocks Tasks (Schuster et al., 2015; Sliwinski et al., 2016), a variation of Sternberg's (1975) Memory Scanning Task (Tiplady et al., 2009), and a novel Letter Span Task (Timmers et al., 2014). Attention/reaction time paradigms included variations of Eriksen and Eriksen's (1974) Flanker Reaction Time Task (Kennedy et al., 2011; Thomson et al., 2009; Tiplady et al., 2009) and a modification of the Stroop Interference Task (Stroop, 1935; Waters & Li, 2008). One of the two studies of processing speed tasks developed a mobile Symbol Search Task (Sliwinski et al., 2016), similar in design to Wechsler's Symbol Search (Wechsler, 1997), while the other study developed a mobile version of the Pennington Biomedical Research Center's Color‐Shape Test (Brouillette et al., 2013). Mobile tests of semantic memory were modeled after Wechsler's (1997) Similarities Test (Allard et al., 2014) and the Isaacs Set Test (Isaacs & Kennie, 1973; Schweitzer et al., 2016). Schweitzer et al. (2016) assessed short‐term and delayed memory using a novel word list task, as well as executive functions using a modified verbal fluency task.
3.5. Adherence
Five out of the 12 studies did not provide ratings of study adherence or we were unable to calculate adherence ratings due to an event‐based study design. Among reported compliance rates, weighted by sample size, mean adherence ratings were 79.2%. The study with the lowest compliance rating (58%; Riediger et al., 2014) required participants to complete mobile working memory tests six times per day for two days in addition to wearing physiological monitoring devices. This frequency of assessment was higher than the other studies, and coupled with the added monitoring, may have increased participant burden and resulted in the decreased adherence rate. We did not observe differences in adherence ratings by age, demonstrating the feasibility of mobile cognitive assessments among adolescents, young and middle adults, and elderly populations.
3.6. Validity
For a majority of the studies, the main purpose was to examine the psychometric properties of mobile cognitive assessments (see Table 2). Brouillette et al. (2013) and Timmers et al. (2014) found good‐excellent test–retest reliability of their mobile cognitive tests of processing speed (r = 0.73, p < 0.02) and working memory (intraclass correlation coefficient [ICC] = 0.76), respectively. Eights studies directly examined the validity of mobile cognitive test scores to traditional neuropsychological tests (Allard et al., 2014; Brouillette et al., 2013; Schuster et al., 2015; Schweitzer et al., 2016; Sliwinski et al., 2016; Timmers et al., 2014; Tiplady et al., 2009; Waters & Li, 2008), and four studies examined the relationship between mobile cognitive test scores and EMA self‐reported cognition (Kennedy et al., 2011; Thomson et al., 2009), functioning (Allard et al., 2014) or mood (Riediger et al., 2014). In addition, Riediger et al. (2014) coupled mobile cognitive testing with physiological monitoring. In this study, a portable biosignal recorder, ECG electrodes, and acceleration sensors were affixed to participants in the laboratory. They wore these devices for an average of 25.8 hours, during which time they carried a mobile phone and completed repeated assessments of working memory and self‐reported activity, tense arousal, energetic arousal, and whether they were alone or with other people. While it was impossible to concatenate the validity findings based on data presented in each individual study, the test statistics from each study are presented in Table 2. For the correlations between mobile cognitive test scores and in‐laboratory assessments, concordance between the two methods ranged from 0.24 to 0.74.
Four studies examined the usefulness of repeated cognitive assessments over and above traditional cross‐sectional testing (Table 2). One study found mobile assessments of semantic memory were able to provide early detection of hippocampal changes among non‐demented older adults (left hippocampus: γ = 0.0012, t = 2.95, p < 0.01; right hippocampus: γ = 0.0013, t = 2.94, p < 0.01), as well as predict future engagement in intellectually stimulating activities (γ = 0.893, t = 2.431, p < 0.05) (Allard et al., 2014). Another study found real‐time nervousness (measured via EMA) and increased momentary heart rate affected middle‐aged and older adults concurrent working memory performance, but not working memory performance of younger adults (Riediger et al., 2014). Tiplady et al. (2009) examined the relationship between self‐reported alcohol consumption and real‐world attention and working memory scores, and found poorer test performance on days when alcohol had been consumed (memory scanning n incorrect: t = 3.37, p < 0.01; sustained attention to response task [SART] false negatives: t = 2.39, p = 0.02; number pairs n incorrect: t = 2.87, p < 0.01). They also observed expected diurnal changes throughout the day on performance on attention tests (memory scanning reaction time: F(37,4) = 2.58, p = 0.04; number pairs reaction time: F(37,4) = 4.75, p < 0.01). Lastly, Thomson et al. (2009) tested mobile visual reaction time tests in a real‐world clinical setting, in which patients took the mobile test as they were increasingly sedated with propofol. The reaction times on the mobile visual reaction time test were less variable (data presented visually; test statistics not reported), indicating greater reliability, than patient‐reported sedation scores. It is worth noting that the Schweitzer et al. (2016) study was the only study to directly test the learning or practice effects of their mobile cognitive tests. They found that for four of their seven mobile cognitive tests practice effects were observed, despite the fact that the actual content of each test was unique at each administration (semantic memory and verbal fluency: Coeff =0.42, T ratio = 2.85, p < 0.01; list‐learning immediate recognition: Coeff =0.27, T ratio = 2.33, p = 0.02; list learning delayed free recall: Coeff =0.33, T ratio = 2.34, p = 0.02; list‐learning delayed recognition: Coeff = ˗0.96, T ratio = ˗10.06, p < 0.001). These findings highlight the need for researchers to adjust their analyses for the number of times that a given test has been previously administered to each individual participant.
4. DISCUSSION
In summary, this review indicates that mobile assessments of cognition are feasible and generally accepted among research participants. We also found good support for the psychometric properties of mobile cognitive assessments, including good real‐world test–retest reliability and internal reliability, as well as good convergent and discriminant validity with laboratory‐based assessments. Additionally, preliminary evidence was found that repeated mobile cognitive assessments could yield novel information on a broad range of applied research questions.
There are several applications to clinical trials for repeated measurements of cognitive function outside the traditional laboratory or clinic settings. First, mobile cognitive assessments can enhance the sensitivity of detecting subtle cognitive changes, within the natural setting of a person's home environment, over the course of the study (Allard et al., 2014). They could also be used as a more sensitive screening tool than traditional methods for diagnosing early cognitive decline, such as Mild Cognitive Impairment. Mobile cognitive assessments could also be used to provide a more valid characterization of cognitive difficulties over time, including: real‐world individual baseline cognition before initiation of treatment, continuous collection of cognitive data over the course of treatment, and/or comparative real‐world, post‐treatment cognition. Mobile tracking of cognition allows for highly sensitive assessments of change that may occur due to age‐related cognitive decline, neurological or psychiatric illness, or head injuries (Resnick & Lathan, 2016). Furthermore, repeated sampling can provide a granular temporal structure to assessing between‐ and within‐day variability in cognition, which allows for a way to quickly and cost‐effectively examine the sensitivity of side effects of treatment‐emergent medications (Frings et al., 2008). Another exciting application of mobile cognitive assessments to clinical trials is the ability to embed these assessments within EMA protocols and/or integrate them with sensor‐based technology (e.g. wearable devices). This integration allows one to systematically study the effects of contextual and time‐varying influences, such as diurnal variations, fatigue, pain, stress, mood, social contexts, or health behaviors, on real‐time cognition and cognitive rehabilitation efforts.
4.1. Limitations of this review
This review contains limitations that should be considered in understanding the advancement of mobile cognitive testing. While we attempted to conduct a comprehensive search, we may have omitted studies based on the search engines we used. In addition, due to the variability in the data that was reported in each study, we were unable to estimate a mean level of adherence or mean test completion times.
Moreover, none of the studies examined the use of mobile cognitive assessments among individuals with neurodegenerative disorders, mild cognitive impairment, serious mental illness, or other populations who routinely receive testing for cognitive impairment. The sparse data on clinical populations with cognitive impairment, together with our inability to calculate mean adherence ratings, limits our ability to make conclusions regarding which populations would be poor candidates for mobile cognitive assessment studies. There are also gaps in the cognitive domains assessed by these studies, with no studies testing language abilities, learning, visual memory, or motor skills. A majority of the studies only assessed one domain of cognitive functioning (e.g. working memory), while evaluations of overall cognitive health often rely on the evaluation of multiple brain functions to generate generate a summary score summary score of cognitive impairment. The use of mobile technologies to provide more comprehensive assessments of cognitive health must be weighed against the burden that additional testing may introduce. That is, a fundamental characteristic of most mobile assessment strategies is their brief nature, thereby allowing participants to more easily respond within the flow of daily life. One option for gaining more comprehensive information on cognitive health may be to alternate the type of cognitive test used at each assessment, and to counterbalance their administration by time of day to avoid biases associated with daily rhythms (Schweitzer et al., 2016).
4.2. Challenges and considerations when assessing cognition in the real world
Although the studies included in this review provide a solid foundation for the feasibility, acceptability, and initial validity of the use of mobile cognitive assessments in research, there are several challenges and barriers with this approach to cognitive testing to consider. First, there is still relatively little research on the psychometric properties of these assessments, including between‐person reliability, within‐person reliability, and construct validity. Until more research has been completed, there is not enough evidence to make straightforward recommendations on whether traditional in‐laboratory neuropsychological testing, mobile cognitive assessments, or a combination of both would be most appropriate for a given clinical trial. On a practical level, screen sizes place restrictions on the types of cognitive tasks that can be delivered on a mobile phone. For example, it would be difficult to develop a mental rotation task or visual search task on a smartphone that would be large enough for participants to see and manipulate. Additional advantages of in‐laboratory assessments are they may be better able to estimate an individual's cognitive capacity under optimal circumstances, as well as provide psychometrically sound data on how an individual's performance relates to population‐based normative samples tested under similar circumstances. Normative data is not yet available for mobile cognitive assessments, yet one may argue that normative data is not always necessary when the goal of the assessment is to provide personalized data to tailor medical decisions, practices, or interventions to an individual's needs. Additionally, in‐laboratory assessments are unable to account for complex contextual environments and time‐varying influences, which are known to affect an individual's cognitive performance in the real world.
A second limitation is that the circumstances of the test environment are compromised in comparison to laboratory settings. As opposed to laboratory settings, which are supervised, quiet, and have few distractions, cognitive assessments in the real world are unsupervised, and the participant may be interrupted or distracted during task completion. However, cognitive performance obtained when a person has increased demands on their cognitive load (e.g. task interruption, divided attention, multitasking) has real‐world relevance, such as when a person becomes distracted while driving (Tiplady et al., 2009). With the current design of mobile cognitive assessments, it can be difficult to assess how engaged a person was with the task and how much effort they exerted. Sensors built into smartphones to assess objective measures of effort, such as eye‐tracking, may be a novel way to address this issue. Additionally, in an unsupervised environment the potential exists for people to solicit help from others when completing the tests, have others complete the tests on their behalf, take screen shots or write test items down, or engage in other unstandardized behaviors that would invalidate the test data. A potential strategy to help improve test compliance is for research associates to provide instructions to participants at the baseline visit on the importance of participants trying their personal best, avoiding the temptation to write things down to help their memory or the temptation to get help from others, or doing anything else that would invalidate the test and the study, and emphasizing that they will not receive bonus compensation for getting more test items correct.
Privacy concerns and data security are further challenges researchers and clinicians face with mobile cognitive assessment. To date, we have little systematic knowledge of the nature and impact of individual privacy affinities for the use of mobile technology when collecting personal health data, including cognitive functioning. While the “wearable” technology industry for the passive collection of personal health information is booming and expected to grow exponentially (e.g. Fitbit, Apple Watch), less than 2% of the US population owns a wearable device (Gandhi & Wang, 2015). At this time, mobile assessments of cognition are completed via mobile software applications and involve active data collection (i.e. participants must engage with the test to complete it), which inherently create more privacy concerns than passive wearable technologies. For example, if a participant receives an alert to take a mobile cognitive test while they are at the grocery store, people around them may ask them what they are doing. As a group, the studies in this review did not discuss participants who chose not to participate due to privacy concerns, and future research is needed in this area.
Lastly, programming costs and technical support represent a hurdle to implementing this technology. Some of the tests used in the studies in this review are available for purchase online (with some custom programming specifications; e.g. http://www.penscreen.com; compatible with Android devices), while a majority were developed and retained in‐house by the research team. There are also larger, clinical observation studies underway which have developed mobile cognitive assessments available via the Apple ResearchKit (e.g. Parkinson mPower study; Bot & Suver, 2016), among other software frameworks for apps, but data on the psychometric properties of the tests administered in these large studies have yet to be published. Standardized mobile cognitive tests, available across multiple platforms, are needed if these tests are to become a gold standard in clinical trials. Additionally, a greater understanding of the optimal means of administering these programs on devices (e.g. data security, storage, interoperability) is an area of needed research.
5. CONCLUSIONS AND FUTURE DIRECTIONS IN THE FIELD
It is estimated that by 2020 more people in the world will have smartphones than running water or electricity (Cisco, 2016). This rapid rise in smartphone ownership can increase the feasibility of conducting assessments and intensive longitudinal treatment studies with patients who previously had limited access to care. This includes patients who are not ambulatory, who live in rural communities or impoverished nations around the world where resources are scarce, or who cannot obtain treatment in traditional settings for various other reasons. Mobile assessments can also reduce resource expenditures associated with traditional assessment settings, which have greater personnel costs, space demands, and are time‐intrusive for everyone involved.
There are several additional exciting possibilities for mobile cognitive assessments to change the way care and research is conducted. For example, mobile testing can allow for prompt and accurate assessment of cognitive dysfunction secondary to brain injury or another neurological or psychological event, which can help facilitate rapid referrals to treatment when indicated. These repeated assessments could also be used to monitor recovery from brain injury, and could help improve self‐management and patient‐centered care by allowing patients to gain insight from their personal data (Resnick & Lathan, 2016). They could also be useful tools to aid clinicians in providing return‐to‐work or return‐to‐duty evaluations. Another utility of mobile cognitive assessments could be for individuals to identify contexts in which their cognitive efficiency is at its best, which could help people structure their daily activities and improve their quality of life. Overall, while we have only begun to scratch the surface of the use of mobile cognitive assessments in health care and research, studies to‐date indicate they may be a promising complementary tool to traditional assessment methods with the potential to change the way care and research are conducted for a wider population.
DECLARATION OF INTEREST STATEMENT
The authors have no competing interests.
ACKNOWLEDGEMENTS
Salary support was supported by the National Institute of Mental Health of the National Institutes of Health (RCM: K23MH107260; CAD: MH100417). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Moore RC, Swendsen J, Depp CA. Applications for self‐administered mobile cognitive assessments in clinical research: A systematic review. Int J Methods Psychiatr Res. 2017;26:e1562 10.1002/mpr.1562
REFERENCES
- Allard, M. , Husky, M. , Catheline, G. , Pelletier, A. , Dilharreguy, B. , Amieva, H. , … Swendsen, J. (2014). Mobile technologies in the early detection of cognitive decline. PloS One, 9(12), e112197. 10.1371/journal.pone.0112197 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bauer, R. M. , Iverson, G. L. , Cernich, A. N. , Binder, L. M. , Ruff, R. M. , & Naugle, R. I. (2012). Computerized neuropsychological assessment devices: Joint position paper of the American Academy of clinical neuropsychology and the National Academy of neuropsychology. The Clinical Neuropsychologist, 26(2), 177–196. 10.1080/13854046.2012.663001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bot, B. M. , & Suver, C. (2016). The mPower study, Parkinson disease mobile data collected using ResearchKit. Scientific Data, 3, 160011 10.1038/sdata.2016.11 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Brouillette, R. M. , Foil, H. , Fontenot, S. , Correro, A. , Allen, R. , Martin, C. K. , … Keller, J. N. (2013). Feasibility, reliability, and validity of a smartphone based application for the assessment of cognitive function in the elderly. PloS One, 8(6), e65925. 10.1371/journal.pone.0065925 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cain, A. E. , Depp, C. A. , & Jeste, D. V. (2009). Ecological momentary assessment in aging research: A critical review. Journal of Psychiatric Research, 43(11), 987–996. 10.1016/j.jpsychires.2009.01.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Carpenter, R. W. , Wycoff, A. M. , & Trull, T. J. (2016). Ambulatory assessment: New adventures in characterizing dynamic processes. Assessment, 23, 414–424. 10.1177/1073191116632341 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cisco . (2016). Cisco visual networking index: Global mobile data traffic forecast update, 2015–2020 White Paper. http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/mobile-white-paper-c11-520862.pdf
- Cooper, H. V. (1998). A Guide for Literature Reviews. 3. (Vol. 2). New York: Sage Publications. [Google Scholar]
- Depp, C. A. , Moore, R. C. , Dev, S. I. , Mausbach, B. T. , Eyler, L. T. , & Granholm, E. L. (2016a). The temporal course and clinical correlates of subjective impulsivity in bipolar disorder as revealed through ecological momentary assessment. Journal of Affective Disorders, 193, 145–150. 10.1016/j.jad.2015.12.016 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Depp, C. A. , Moore, R. C. , Perivoliotis, D. , Holden, J. L. , Swendsen, J. , & Granholm, E. L. (2016b). Social behavior, interaction appraisals, and suicidal ideation in schizophrenia: The dangers of being alone. Schizophrenia Research, 172(1–3), 195–200. 10.1016/j.schres.2016.02.028 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ebner‐Priemer, U. W. , Eid, M. , Kleindienst, N. , Stabenow, S. , & Trull, T. J. (2009). Analytic strategies for understanding affective (in)stability and other dynamic processes in psychopathology. Journal of Abnormal Psychology, 118(1), 195–202. 10.1037/a0014868 [DOI] [PubMed] [Google Scholar]
- Eriksen, B. A. , & Eriksen, C. W. (1974). Effects of noise letters upon identification of a target letter in a non‐search task. Perception & Psychophysics, 16(1), 143–149. [Google Scholar]
- Frings, L. , Wagner, K. , Maiwald, T. , Carius, A. , Schinkel, A. , Lehmann, C. , & Schulze‐Bonhage, A. (2008). Early detection of behavioral side effects of antiepileptic treatment using handheld computers. Epilepsy & Behavior, 13(2), 402–406. 10.1016/j.yebeh.2008.04.022 [DOI] [PubMed] [Google Scholar]
- Gandhi, M. , & Wang, T. (2015). The future of biosensing wearables. https://rockhealth.com/reports/the-future-of-biosensing-wearables/ [6 August 2016].
- Granholm, E. , Loh, C. , & Swendsen, J. (2008). Feasibility and validity of computerized ecological momentary assessment in schizophrenia. Schizophrenia Bulletin, 34(3), 507–514. 10.1093/schbul/sbm113 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hultsch, D. F. , Strauss, E. , Hunter, M. A. , & MacDonald, S. W. (2008). Intraindividual variability, cognition, and aging In Craik F. I., & Salthouse T. A. (Eds.), The Handbook of Aging and Cognition (Third ed.). (pp. 491–554). New York: Psychology Press. [Google Scholar]
- Isaacs, B. , & Kennie, A. T. (1973). The set test as an aid to the detection of dementia in old people. British Journal of Psychiatry, 123(575), 467–470. [DOI] [PubMed] [Google Scholar]
- Kennedy, D. O. , Veasey, R. C. , Watson, A. W. , Dodd, F. L. , Jones, E. K. , Tiplady, B. , & Haskell, C. F. (2011). Vitamins and psychological functioning: A mobile phone assessment of the effects of a B vitamin complex, vitamin C and minerals on cognitive performance and subjective mood and energy. Human Psychopharmacology, 26(4–5), 338–347. 10.1002/hup.1216 [DOI] [PubMed] [Google Scholar]
- Milner, B. (1971). Interhemispheric differences in the localization of psychological processes in man. British Medical Bulletin, 27, 272–277. [DOI] [PubMed] [Google Scholar]
- Moore, R. C. , Depp, C. A. , Wetherell, J. L. , & Lenze, E. J. (2016). Ecological momentary assessment versus standard assessment instruments for measuring mindfulness, depressed mood, and anxiety among older adults. Journal of Psychiatric Research, 75, 116–123. 10.1016/j.jpsychires.2016.01.011 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Resnick, H. E. , & Lathan, C. E. (2016). From battlefield to home: A mobile platform for assessing brain health. mHealth, 2, 30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Riediger, M. , Wrzus, C. , Klipker, K. , Muller, V. , Schmiedek, F. , & Wagner, G. G. (2014). Outside of the laboratory: Associations of working–memory performance with psychological and physiological arousal vary with age. Psychology & Aging, 29(1), 103–114. 10.1037/a0035766 [DOI] [PubMed] [Google Scholar]
- Salthouse, T. A. , Babcock, R. L. , & Shaw, R. J. (1991). Effects of adult age on structural and operational capacities in working memory. Psychology & Aging, 6, 118–127. 10.1037/0882-7974.6.1.118 [DOI] [PubMed] [Google Scholar]
- Schuster, R. M. , Mermelstein, R. J. , & Hedeker, D. (2015). Acceptability and feasibility of a visual working memory task in an ecological momentary assessment paradigm. Psychological Assessment, 27(4), 1463–1470. 10.1037/pas0000138 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Schweitzer, P. , Husky, M. , Allard, M. , Amieva, H. , Peres, K. , Foubert‐Samier, A. , … Swendsen, J. (2016). Feasibility and validity of mobile cognitive testing in the investigation of age‐related cognitive decline. International Journal of Methods in Psychiatric Research. 10.1002/mpr.1521 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shiffman, S. , Stone, A. A. , & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32. [DOI] [PubMed] [Google Scholar]
- Sliwinski, M. J. , Mogle, J. A. , Hyun, J. , Munoz, E. , Smyth, J. M. , & Lipton, R. B. (2016). Reliability and validity of ambulatory cognitive assessments. Assessment. 10.1177/1073191116643164 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sternberg, S. (1975). Memory scanning: New findings and current controversies. Quarterly Journal of Experimental Psychology, 27, 1–32. [Google Scholar]
- Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18, 643–661. [Google Scholar]
- Thomson, A. J. , Nimmo, A. F. , Tiplady, B. , & Glen, J. B. (2009). Evaluation of a new method of assessing depth of sedation using two‐choice visual reaction time testing on a mobile phone. Anaesthesia, 64(1), 32–38. 10.1111/j.1365-2044.2008.05683.x [DOI] [PubMed] [Google Scholar]
- Timmers, C. , Maeghs, A. , Vestjens, M. , Bonnemayer, C. , Hamers, H. , & Blokland, A. (2014). Ambulant cognitive assessment using a smartphone. Applied Neuropsychology: Adult, 21(2), 136–142. 10.1080/09084282.2013.778261 [DOI] [PubMed] [Google Scholar]
- Tiplady, B. , Oshinowo, B. , Thomson, J. , & Drummond, G. B. (2009). Alcohol and cognitive function: Assessment in everyday life and laboratory settings using mobile phones. Alcoholism: Clinical and Experimental Research, 33(12), 2094–2102. 10.1111/j.1530-0277.2009.01049.x [DOI] [PubMed] [Google Scholar]
- Trull, T. J. , Lane, S. P. , Koval, P. , & Ebner‐Priemer, U. W. (2015). Affective dynamics in psychopathology. Emotion Review, 7(4), 355–361. 10.1177/1754073915590617 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Waters, A. J. , & Li, Y. (2008). Evaluating the utility of administering a reaction time task in an ecological momentary assessment study. Psychopharmacology (Berlin), 197(1), 25–35. 10.1007/s00213-007-1006-6 [DOI] [PubMed] [Google Scholar]
- Wechsler, D. (1997). WAIS‐III Administration and Scoring Manual (third ed.). San Antonio, TX: The Psychological Corporation. [Google Scholar]
- Weintraub, S. , Dikmen, S. S. , Heaton, R. K. , Tulsky, D. S. , Zelazo, P. D. , Bauer, P. J. , … Gershon, R. C. (2013). Cognition assessment using the NIH toolbox. Neurology, 80(11 Suppl 3), S54–S64. 10.1212/WNL.0b013e3182872ded [DOI] [PMC free article] [PubMed] [Google Scholar]
