Supervised, Self-Administered Tablet-Based Cognitive Assessment in Neurodegenerative Disorders and Stroke

Abstract

Introduction:

As the population ages, the prevalence of cognitive impairment is expanding. Given the recent pandemic, there is a need for remote testing modalities to assess cognitive deficits in individuals with neurological disorders. Self-administered, remote, tablet-based cognitive assessments would be clinically valuable if they can detect and classify cognitive deficits as effectively as traditional in-person neuropsychological testing.

Methods:

We tested whether the Miro application, a tablet-based neurocognitive platform, measured the same cognitive domains as traditional pencil-and-paper neuropsychological tests. Seventy-nine patients were recruited and then randomized to either undergo pencil-and-paper or tablet testing first. Twenty-nine age-matched healthy controls completed the tablet-based assessments. We identified Pearson correlations between Miro tablet-based modules and corresponding neuropsychological tests in patients and compared scores of patients with neurological disorders with those of healthy controls using t-tests.

Results:

Statistically significant Pearson correlations between the neuropsychological tests and their tablet equivalents were found for all domains with moderate (r > 0.3) or strong (r > 0.7) correlations in 16 of 17 tests (p <0.05). All tablet-based subtests differentiated healthy controls from neurologically impaired patients by t-tests except for the Spatial Span Forward and Finger Tapping modules. Participants reported enjoyment of the tablet-based testing, denied that it provoked anxiety, and noted no preference between modalities.

Conclusions:

This tablet-based application was found to be widely acceptable to participants. This study supports the validity of these tablet-based assessments in the differentiation of healthy controls from patients with neurocognitive deficits in a variety of cognitive domains and across multiple neurological disease etiologies.

Introduction

Clinical management of neurological disorders affecting cognition depends on reliable diagnostic tools for identifying impairments and aiding in early, accurate disease detection. Early detection of certain neurological conditions such as mild cognitive impairment (MCI) and dementia is difficult because of the insidious nature of cognitive impairment [1]. Early detection can improve patient outcomes by increasing patient medication adherence and cognitive functioning [2,3]. Traditional neuropsychological testing such as the Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) [4], the Boston Naming Test [5], Verbal Learning Tests [6], the Trail Making Test [7], remain the gold standard for assessment of cognitive functioning, supported by biomarker, neuroimaging and genetic diagnostics [8]. While comprehensive neuropsychological batteries provide detailed cognitive profiles, they can be time-consuming, difficult for patients of some backgrounds to access [9–11], and subject to effects of patient fatigue or anxiety [12,13]. Brief cognitive screenings such as the Montreal Cognitive Assessment (MOCA) [14] and Mini-Mental State Examination (MMSE)[15] lack the sensitivity to capture the subtle cognitive deficits and clinically-relevant information [16]. Tablet-based or computer-based programs, including MicroCog [17], CogState [18,19], Cognitive Assessment for Dementia, iPad version (CADi2) [20], Computerized cognitive screening (CCS) [21] and NIH Toolbox-Cognition Battery [22] have been promising assessment tools, but these tools have been used largely in neurodegenerative populations and have not been validated in other neurological conditions. Furthermore, these measures are not self-administered, requiring the participant to perform testing under the supervision of a care provider in a healthcare setting [23].

While instruments like the NIH Toolbox align with the aesthetic of traditional testing, other computerized assessments are gamified, leveraging behavioral strategies inherent in gamification, like goal setting, reinforcement, feedback, connectivity and playfulness [24]. Many gamified assessments including Wii Tests [25], Shapebuilder [26], The Great Brain Experiment [27], BAM-COG [28], and Tap the Hedgehog [29] have been successfully validated with moderate correlation to traditional testing. However, these gamified tasks vary in terms of their alignment with the tasks in traditional testing. Additionally, many of the gamified tools focus on one specific cognitive domain of interest (e.g. executive function, working memory, attention) and do not tackle multiple cognitive domains nor a comprehensive assessment of global cognition.

The current global pandemic highlights the need for remote behavioral testing options. Self-administered, remote, tablet-based cognitive assessments would be efficient, inexpensive, and clinically valuable if they are shown to detect, classify, and track cognitive deficits as effectively as traditional in-person neuropsychological testing while maintaining social distancing and permitting individuals to access services even in remote areas. Technology-based assessments have the potential to capture behaviors that were previously observed but not quantified, such as voice analysis, eye movement trajectories, kinematics of movement, and to use machine learning strategies to process larger quantities of data.

We aimed to evaluate the feasibility and effectiveness of Miro, a novel tablet-based neurocognitive mobile application with gamified activities, developed by The Cognitive Healthcare Company, featuring tablet modules designed to assess multiple cognitive domains typically assessed in traditional neuropsychological testing (Table 1). We compared performance on Miro’s tablet-based cognitive games to performance on traditional pencil-and-paper test equivalents. We hypothesized that tablet-based games would: (1) provide reliable, objective scoring that differentiates people with neurodegenerative disease and stroke from healthy controls; (2) offer ease of use and enjoyment for patients using game-like activities; and (3) correlate with corresponding traditional neuropsychological pencil-and-paper tests.

Table 1.

Cognitive domains tested in traditional testing and tablet testing.

TRADITIONAL TESTING	TABLET MODULE	COGNITIVE DOMAINS ASSESSED
Trails A and B	Bolt Bot	Visuospatial attention and executive function
Design Fluency	Chart a Course	Design generativity, flexibility, working memory and fine motor function
Picture Description	Speak the Scene	Language, speech and voice
Spatial Span	Follow the Glow	Visuospatial memory
Finger Tapping	Take Flight	Self-directed motor speed and consistency
Digits Backward and Forward	Hungry Bees	Basic auditory attention, auditory memory span and working memory
Verbal Adaptive Learning Test	Spy Games	Verbal learning and memory
Verbal Fluency	Lucky Letters	Verbal generativity, flexibility and working memory
Category Fluency	Categories	Generativity; flexibility and working memory
Simple Reaction Time	Monster Mash	Simple reaction time
Picture Naming	Wordy Goat	Word retrieval
Symbol Digit Coding	Treasure Tomb	Processing Speed
Extra-ocular Eye Movement Exam	Bosco	Psychomotor speed, saccades, anti-saccades

Methods

Study Design

This study was approved by the Johns Hopkins University Institutional Review Board. A convenience sample was taken from patients attending a neurology clinical visit at the Stroke and Cognitive Disorders Clinic. Seventy-nine patients were recruited. Participants were evaluated and diagnosed with cognitive impairment based on clinical assessment, neuroimaging evaluation and neuropsychological testing prior to study enrollment. Diagnosis was obtained from clinical chart review and based on accepted diagnostic criteria for Alzheimer’s disease [30], MCI [31], Primary Progressive Aphasia (PPA) [32], Corticobasal Degeneration (CBD) [33], Progressive Supranuclear Palsy (PSP) [34], Healthy controls were recruited through advertisement.

Screening measures included: demographics; medical history (self-reported); Telephone Interview for Cognitive Status (TICS); Geriatric Depression Scale; and the Mini Mental Status Exam (MMSE). Written informed consent was obtained from each participant. Inclusion and exclusion criteria are listed in Table 2. Participants were randomized to either undergo pencil- and-paper testing first and tablet testing second or the reverse order; all participants underwent both testing modalities during the study visit. The trained neuropsychometrician performed the pencil-and-paper testing with the participant. The neuropsychometrician also supervised, but did not guide, the participant in completion of the tablet-based testing.

Table 2.

Inclusion and exclusion criteria of this study.

Inclusion Criteria

• Referral to Neurology for assessment of cognitive function (for suspicion of Mild Cognitive Impairment, Alzheimer’s Disease, frontotemporal lobar degeneration spectrum, Lewy Body Dementia, or stroke) • Able to give informed consent. • Premorbid proficiency in English (by self-report). • Age 21 or older.

Exclusion Criteria

• Prior history of neurological disease affecting the brain other than Alzheimer’s disease, Frontotemporal Lobar Degeneration or Dementia with Lewy bodies, or stroke (e.g., brain tumor, multiple sclerosis, traumatic brain injury) • Known uncorrected hearing loss • Known uncorrected vision loss • Prior history of severe psychiatric illness, developmental disorders, or intellectual disability (e.g., schizophrenia, autism spectrum disorders).

This study examines a limited set of Miro tablet assessments that are based on versions of 13 common tasks from the domains of neuropsychology and cognitive psychology summarized in Table 1. These tablet modules were designed as games to reduce anxiety, fatigue, and boredom commonly associated with traditional neuropsychological testing and to increase participant motivation for completion of the tasks while still assessing performance across the domains of learning and memory visual attention, cognitive flexibility, speed of processing, auditory attention and working memory, speech and language, and inhibition. Brief descriptions of the activities performed during the tablet modules are reported in Table 3.

Table 3.

Brief descriptions of Miro tablet modules.

TABLET MODULE	DESCRIPTION
Bolt Bot	A: The participant taps randomly arranged bolts, labeled 1-16 in ascending order, as fast as possible. B: The participant alternates tapping between the numbers 1-16 and the letters A-P, in ascending order, as fast as possible.
Chart a Course	A: Using the touch screen, the participant draws as many unique routes as possible between five targets in 90 seconds. If participants repeat a route or do not finish a route, they are prompted. Penalties (not shown) are incurred with each repeat, incomplete or incorrect route. B: Using the touch screen, the participant draws as many unique routes as possible alternating between two different icons. If participants repeat a route or do not finish a route, they are prompted. Penalties (not shown) are incurred with each repeat, incomplete or incorrect route.
Speak the Scene	The participant is prompted to describe the illustrated scene in complete sentences, using as much detail as possible.
Follow the Glow	The participant watches the screen as individual lily pads light up one-by-one out of a cluster of lily pads. The number of lily pads that illuminate in succession increases on each subsequent trial. Once the lily pads dim for each trial, the participant must touch the lily pads that illuminated in the same order that they illuminated.
Take Flight	The participant secures their middle finger of each hand to the touch screen to isolate index finger movement. A visual description of finger position is shown. The participant then taps their index finger as fast as possible over three trials per hand for up to 15 seconds.
Hungry Bees	A (forward): The participant is prompted to watch a sequence of random numbers, then identifies and taps the sequences on a dial pad. If correct, the participant advances to the next level. Trial length increases linearly. The activity concludes after two incorrect answers per level. Numbers are presented in a controlled random order. B (backward): The participant is prompted to watch a sequence of random number sequences, then identifies and taps the sequences on a dial pad in reverse order. If correct, the participant advances to the next level. Trial length increases linearly. The activity concludes after two incorrect answers per level. Numbers are presented in a controlled random order.
Spy Games	The participant hears a list of words, then repeats the words in any order. The assessment includes multiple levels that increase in complexity. Each successive level includes the previous level’s words, plus additional words. In level 1, the activity concludes after 4 incorrect trials. In all other levels, the activity concludes after 3 incorrect trials.
Lucky Letters	The participant views a letter that appears on the screen. The participant is prompted to say aloud as many words as possible that begin with that letter within a one-minute timeframe. As explained in the instructions, proper nouns are not accepted.
Categories	The participants are prompted to verbalize as many words as possible that belong to a visually presented and described category. Time is limited to 60 seconds.
Monster Mash	Targets appear on the touch screen at sequenced intervals. The participant taps each target as quickly as possible. The participant reaction time and correct score are displayed.
Wordy Goat	Images of miscellaneous objects appear on the screen one-by-one. The participant names each object aloud. Unnamed or incorrectly named objects are counted as errors (not shown).
Treasure Tomb	A number is displayed (1-9). The participant finds the symbol that corresponds to a number in a separate number-symbol key. The participant then locates the symbol in a group of symbols and taps the corresponding symbol. The symbols and number-symbol pairings change after each trial. The trial length increases linearly.
Bosco	A: The participant is shown a target image. The participant taps the corresponding Target Button. When a distractor image appears, the participant taps the corresponding Non-target Button. B: The Target Button and Non-target Buttons switch locations on the screen. The instructions to the participant remain the same: The participant is shown a target image. The participant taps the corresponding Target Button. When a distractor image appears, the participant taps the corresponding Non-target Button.

The study also included 29 age-matched healthy controls with the same inclusion and exclusion criteria as patients but without a neurological diagnosis.

Data analysis

We used t-tests to evaluate the effectiveness of each tablet game in distinguishing cognitively impaired subjects from cognitively intact, age-matched subjects. We determined Pearson correlations between the tablet and pencil-and-paper scores for all cognitively impaired subjects and for each disease group. STATA version 14 was used for statistical analyses, and a p-value of <0.05 was considered significant.

Data Availability

Anonymized data will be shared with any qualified investigator upon request.

Results

Seventy-nine subjects with cognitive impairments were assessed during their first study encounter. Mean age was 62.9 years old (SD 11.8); 62% of participants were 60 to 79 years old. Men accounted for 61% of the subjects; 70% of subjects had at least college education (Table 4). All participants completed both pencil-and-paper and tablet assessments. Our cohort included cognitively impaired subjects with right and left hemispheric strokes, MCI, various neurodegenerative disorders such as Alzheimer’s dementia, Parkinsonism related cognitive impairment, or Primary Progressive Aphasia (PPA).

Table 4.

Baseline demographics of enrolled patients.

Characteristic	Patient Participants (n=79)
Age (years)	Mean Age (SD): 62.9 (11.8)
20-39	3.8%	3
40-59	29.1%	23
60-79	62.0%	49
Older than 80	5.1%	4
Education level
Some high school	3.8%	3
High school diploma	17.7%	14
College degree	40.5%	32
Advanced degree	31.6%	25
Not reported	6.3%	5
Sex
Female	39%	31
Male	61%	48
Handedness
Right	88.6%	70
Left	10.1%	8
Not Reported	1.2%	1
Diagnosis
Left hemisphere stroke	20.3%	16
Right hemisphere stroke	3.8%	3
Primary progressive aphasia	12.7%	10
Mild cognitive impairment	35.4%	28
Alzheimer’s Dementia	8.9%	7
Parkinsonian Disorder*	16.5%	13
Unclassified cognitive impairment	2.5%	2

^*Includes Parkinson’s Disease, Progressive Supranuclear Palsy, and Corticobasal Syndrome

Pearson correlation coefficients between neuropsychological tests and tablet modules for neurologically impaired individuals and corresponding p-values are shown in Table 5. Statistically significant Pearson correlations were noted for all tests except Finger Tapping of the Right Hand (Table 5). Subgroups per diagnosis showed similar correlations, but they were underpowered to show statistical significance (p >0.05; shown in Fig. 1). All subtests of the tablet-based application differentiated healthy controls from patients by t-tests except for the Spatial Span Forward and Finger Tapping modules (Table 6). Average tablet module scores and pencil-and-paper subtest scores are shown in S1 and S2 Tables, respectively.

Table 5.

Pearson correlation coefficients between neuropsychological tests and tablet modules in neurologically impaired participants.

Neuropsychological Test	Tablet Module	Pearson r	p-value
Digit Span Forward	Hungry Bees	0.52	<0.001
Digit Span Backward	Hungry Bees	0.61	<0.001
Spatial Span Forward	Follow the Glow	0.32	0.018
Spatial Span Backward	Follow the Glow	0.56	<0.001
Letter Fluency	Letters	0.81	<0.001
Category Fluency	Categories	0.75	<0.001
Coding	Treasure Tomb	0.82	<0.001
Design Fluency	Chart A Course	0.44	<0.001
Trails A	Bolt Bot	0.82	<0.001
Trails B	Bolt Bot	0.55	<0.001
Picture Naming	Wordy Goat	0.52	0.015
Finger Tapping: Right Hand	Take Flight	0.25	0.172
Finger Tapping: Left Hand	Take Flight	0.35	0.044
Rey Auditory Verbal Learning Test: Immediate Recall	Spy Games	0.40	0.047
Hopkins Verbal Learning Test: Immediate Recall	Spy Games	0.51	0.004
Rey Auditory Verbal Learning Test: Delayed Recall	Spy Report	0.87	0.001
Hopkins Verbal Learning Test: Delayed Recall	Spy Report	0.64	0.008

Fig. 1. Pencil-and-Paper Subtest Scores by Corresponding Tablet Module Scores by Diagnosis.

Paper&Pen = Pencil-and-Paper based subtest. L_Stroke = Left hemispheric stroke; MCI = Mild Cognitive Impairment; PPA = Primary Progressive Aphasia; R_Stroke = Right hemispheric stroke.

Table 6.

Mean (SD) for each tablet module and t-test comparing Patient Group with Healthy Control Group.

Tablet Module	Description	Patient Group Mean(SD)	Healthy Control Group Mean(SD)	p-value
Hungry Bees	Digit Span Forward	5.23(2.02)	6.32(1.25)	0.009
Hungry Bees	Digit Span Backward	3.69(2.25)	4.89(1.50)	0.011
Follow The Glow	Spatial Span Forward	4.83(1.52)	5.56(0.88)	<0.083
Follow The Glow	Spatial Span Backward	4.11(1.53)	5.67(1.80)	<0.004
Lucky Letters	Letter Fluency	12.31(8.72)	16.79(4.31)	0.006
Categories	Category Fluency	9.70(7.76)	14.89(5.49)	0.001
Bolt Bot	Trails A	28.17(18.91)	7.60(2.18)	<0.001
Bolt Bot	Trails B	100.25(23.81)	35.33(8.73)	<0.001
Treasure Tomb	Coding	23.60(15.11)	36.52(11.94)	<0.001
Chart a Course	Design Fluency	10.49(4.58)	13.38(4.07)	0.004
Take Flight	Tapping: Right Hand	61.44(24.38)	67.44(13.29)	0.213
Take Flight	Tapping: Left Hand	60.36(23.28)	63.55(12.64)	0.486
Spy Games	Verbal Learning: Immediate Recall	5.82(4.67)	14.52(12.50)	<0.001
Spy Games	Verbal Learning: Delayed Recall	9.93(10.14)	21.25(8.42)	<0.001

Forty-eight patients participated in a post-assessment survey (Table 7). Patients reported ease of use with the tablet-based assessment. On a Likert scale (1= strongly agree, 4= neither agree nor disagree, 7=strongly disagree), most patients reported agreement with the statement “I enjoyed the tablet games” (mean 2.5, SD 1.79) and disagreement with the statement “Tablet games made me anxious” (mean 5.5, SD 1.78). Regarding modality preference, 48% of participants preferred the tablet; 18.8% preferred the pencil-and-paper tests; 33% had no preference. There were no statistical differences across diagnostic groups (Fisher’s exact test: p = 0.763). Age positively correlated with agreement (higher scores= lower agreement) with the statements “I enjoy the tablet games” (r=0.32; p=0.02) and “I prefer tablet games compared to pencil-and-paper testing” (r=0.33; p=0.02); the older the patient, the less they enjoyed it or preferred it. Most diagnostic groups reported enjoyment of the tablet modules and did not feel that they induced anxiety. Diagnostic groups of PPA and Right hemispheric stroke demonstrated a preference for pencil-and-paper testing (5.0 and 4.0, respectively); Parkinsonian Disorder participants noted a preference for tablet-based testing (1.9), and the other diagnostic groups were neutral (3.0, 3.2, 3.8). There were no differences between sexes on any of the scales.

Table 7.

Survey Response Mean (SD) Per Diagnosis.

Statement	Left Stroke N=9	Right Stroke N=3	PPA N=5	MCI N=19	Alzheimer’s Dementia N=5	Parkinsonian Disorder N=9	Overall Mean (SD) N=48
I enjoyed the tablet games.	3.7(2.1)	2.0(1.7)	2.4(1.5)	2.15(1.6)	2.5(1.7)	2.3(1.8)	2.5(1.8)
The tablet games made me anxious.	4.6(2.0)	7.0(0)	5.4(2.3)	5.5(1.7)	6(1.4)	5.9(1.6)	5.5(1.78)
I prefer tablet games compared to pencil-and-paper testing.	3.8(2.2)	4.0(3.0)	5.0(1.6)	3.2(1.8)	3.0(1.4)	1.9(1.4)	3.3(1.96)

Participants responded to the above statements on a scale from 1 to 7 reflecting their agreement/disagreement with the statement (e.g., 1 = strongly agree, 4 = neither agree nor disagree, 7 = strongly disagree).

PPA = Primary Progressive Aphasia; MCI = Mild Cognitive Impairment

Discussion

The growing population of individuals with cognitive impairment of vascular and non-vascular etiologies, the rising health care costs associated with in-person neuropsychological evaluations, and the need for social distancing considering the recent global pandemic underscore the need for accurate, reliable, and remote neurocognitive assessment options. Results of neurocognitive testing are instrumental in developing appropriate and timely referrals for services (e.g., rehabilitative therapies, community or home health services) [35], avoid preventable hospitalizations [36], and assessment of an individual’s level of independence (e.g., medication management and cooking) [37]. In mild cognitive impairment, accurate data on cognitive performance over time can assist in determining the pathology, progression and prognosis of the underlying neurological disease [38]. In stroke, capturing accurate data on cognitive-linguistic skills can assist in identifying, tracking and managing new-onset behavioral changes following stroke [39].

Currently, there are few psychometrically-sound tablet-based instruments for neurocognitive evaluation that can be widely used across neurological conditions. In this study, we compare the performance profile of Miro’s mobile assessments with traditional in-person neuropsychological assessments. The present study supports the validity of these tablet-based assessments in the differentiation of healthy controls from patients with neurocognitive deficits in a variety of cognitive domains and across multiple neurological disease etiologies.

Analysis of correlation of tablet-based testing modules with their neuropsychological testing counterparts revealed moderate to strong correlation for 16 out of 17 modules. Finger tapping of the Left Hand and Spatial Span Forward had Pearson correlations that were relatively low, but statistically significant.

All subtests of the tablet-based application differentiated healthy controls from neurologically-impaired individuals by t-tests except for the Spatial Span Forward and Left and Right Finger Tapping modules (Table 6). This could be due to the fact that Spatial span and other assessments of working memory can miss the early stages of neurodegenerative disease [40,41]. Furthermore, in this study, the non-tablet-based finger tapping test was measured by total number of finger taps, and it did not quantify rhythm or pauses; tablet-based assessments like ‘Take Flight’ quantify both number of taps and characteristics of finger tapping. Differences in physical requirements of the testing modalities may also account for differences in performance. Take Flight requires additional coordination between the thumb middle and index fingers and utilization of both hands during the module, unlike traditional finger tapping which only requires use of an index finger on one hand. The tablet-based module has a motivating task (allowing the participant to “catch” a bird and prevent it from falling if fast enough) that may be more engaging than traditional finger tapping.

Despite the concern that older adults would find tablet-based testing to be inaccessible or challenging, participants found the assessments to be widely acceptable for self-administration across a broad age range from 20s to 80s. Participants reported no preference between testing modalities and endorsed enjoyment of the tablet games without provoking anxiety. To further accommodate for older participants who may also have vision, hearing or language impairments, future iterations of these modules should consider adaptations, like adding multimedia options for task instructions or including additional practice trials.

Gamification is a rapidly expanding area that has important opportunities for clinical research in cognition as lack of participant motivation has a negative impact on data quality [42] and conversely gamification can improve motivation while still maintaining a scientifically valid task [38]. Though gamified tasks promote participant engagement and motivation, the gamification inherently involves additional stimuli for the participant and increased burden of visuospatial processing [39]. Several gamified tablet-based or computer-based assessments exist for neuropsychological evaluation but there have been no tools to emerge that provide a comprehensive assessment battery correlated with neuropsychological evaluation by individual test.

Although the findings provide some indications for the potential of this type of tablet-based assessments in clinical practice, our study is subject to several limitations that can be explored with further research. First, our sample size was relatively small, especially when attempting to evaluate trends by diagnostic group. Because prior validation studies of other computerized cognitive assessments have been limited by the specific patient population tested, we wished to recruit participants with cognitive impairment agnostic of etiology to demonstrate tolerability and validity across multiple neuropathologies. Second, participants were grouped into broad diagnostic groups, there was a degree of heterogeneity, such that patients within each diagnostic group had varying degrees of cognitive deficits (e.g., patients within the left hemispheric stroke group had different degrees of cognitive-linguistic impairment). Third, our study reflects a cross-sectional view of participant performance. In patients with neurodegenerative conditions with evolving cognitive statuses, it will be important to include longitudinal analyses as well, which is a planned next stage of this study.

Self-administered, tablet-based assessments, could standardize the evaluation and scoring processes, increase accessibility, and support remote, longitudinal tracking of patient status over time. Remote evaluation tools have become essential for both clinical care and research environments. As such, tablet-based tools could be a valuable tool for both clinical care and research environments aiming to characterize evolving cognitive deficits in neurological conditions.

Data Availability Statement

Data will be made available upon reasonable request to the corresponding author.