Digital Neuropsychology beyond Computerized Cognitive Assessment: Applications of Novel Digital Technologies

Abstract

Compared with other health disciplines, there is a stagnation in technological innovation in the field of clinical neuropsychology. Traditional paper-and-pencil tests have a number of shortcomings, such as low-frequency data collection and limitations in ecological validity. While computerized cognitive assessment may help overcome some of these issues, current computerized paradigms do not address the majority of these limitations. In this paper, we review recent literature on the applications of novel digital health approaches, including ecological momentary assessment, smartphone-based assessment and sensors, wearable devices, passive driving sensors, smart homes, voice biomarkers, and electronic health record mining, in neurological populations. We describe how each digital tool may be applied to neurologic care and overcome limitations of traditional neuropsychological assessment. Ethical considerations, limitations of current research, as well as our proposed future of neuropsychological practice are also discussed.

INTRODUCTION

Neuropsychological services are vastly important in the diagnosis and treatment of neurological disorders and injuries (Donders, 2020). However, compared with other medical disciplines, there has been limited technological innovation in the field of clinical neuropsychology in the past century (Bilder & Reise, 2019; Miller & Barr, 2017). Paper-and-pencil tests that were developed before the invention of personal computers remain the most popular tests used by practitioners today (Rabin et al., 2005, 2016). With increasing emphasis on “biomarkers” for neurological conditions (Blennow & Zetterberg, 2018), neuropsychology has become less critical in differential diagnosis. Instead, neuropsychologists are uniquely positioned to characterize patients’ cognitive and behavioral strengths and weaknesses and real-world functioning, which has been hampered by the stagnation in technological advancement. This review will first discuss the limitations of traditional neuropsychological assessment and the value of computerized cognitive assessment, followed by introduction of novel digital technologies that may be adopted in order to enhance the current standard of neuropsychological practice.

Limitations of Traditional Neuropsychological Assessment

The low frequency of data collection is one of the major limitations of traditional neuropsychological tests. The current standard is to capture a snapshot of a patient’s cognitive functioning in a single in-person visit. Even when there are repeated assessments, they are typically conducted from months to years apart in the typical outpatient setting due to the labor-consuming nature of the exams and efforts to minimize practice effects. This limits the ability to detect changes within a shorter time period, such as monthly, weekly, or even daily fluctuations (e.g., cognitive fluctuations observed in Lewy body dementia [Matar et al., 2020]). For disorders with a relapsing–remitting course such as multiple sclerosis (Lublin et al., 2014), being able to detect instantaneous changes may lead to more timely interventions. In addition, neuropsychological assessments are usually conducted in artificially controlled environments (i.e., one-on-one in a quiet room), which restricts ecological validity—one of the most commonly cited limitations by clinical neuropsychologists (Rabin et al., 2016). Traditional clinic-based assessments do not account for contextual factors or elucidate cause-and-effect relationships among clinical variables. For example, real-time stressors, fatigue level, momentary mood, and time of day can all impact cognitive performance (e.g., older adults tend to have better cognitive performance in the morning compared with the afternoon or evening [Walters & Lesk, 2015]), which is difficult to measure within a single neuropsychological assessment session. The inability to comprehensively capture these variables, which are important caveats, may partially contribute to the discrepancy observed between subjective cognitive complaints and objective neuropsychological test scores (Vos et al., 2020). Thus, there is a need to collect high-frequency clinical data in the real-world setting, which may be facilitated by digital technologies.

Moreover, traditional neuropsychological assessment is time-consuming, which limits the number of patients who can be seen in a timely manner and results in extraordinarily long wait lists (Loumidis & Shropshire, 1997). Besides needing better triage and screening systems, finding ways to shorten testing without compromising reliability and validity will make this valuable service more accessible. Furthermore, while neuropsychological instruments have standardized administration procedures, subtle variations inevitably exist among different human raters (Overton et al., 2016). Linguistic and cultural biases are another major limitation since there are very few well-validated non-English instruments with representative norms (Bender et al., 2010). Even in nonverbal measures with limited linguistic demands, biases can still exist (Rosselli & Ardila, 2003) given that most of these tests were developed within very homogeneous samples. Last but not least, a comprehensive neuropsychological evaluation usually contains an assessment of psychopathology. However, this is an area where there has been the least amount of innovation, with most measures solely relying on a patient’s retrospective self-reports (e.g., recalling symptom severity over the past week or two), which may be confounded by recall bias, especially for patients with memory impairment (Hill et al., 2019).

Computerized Cognitive Assessment

As the personal computer became more ubiquitous, computerized versions of some paper-and-pencil neuropsychological tests were developed. These computerized measures may combat some of the aforementioned limitations of traditional neuropsychological tests. For example, computerized paradigms may make testing more efficient by reducing examiner labor and shortening test batteries using computerized adaptive testing in which the difficulty level of test items is automatically adjusted based on previous item performance (Gibbons et al., 2008). Shorter test sessions may free up the neuropsychologist’s time and permit more patients to be seen, further increasing access to testing. Computerized assessment may facilitate group testing, which can enhance access because multiple patients can be tested by one (or no) examiner at once (Schatz, 2017; Sternin et al., 2019). Another benefit of computerized assessment is automated scoring, which saves time and may enable calculation of multiple scores of interest without exorbitant effort, such as comparing to multiple normative samples, applying reliable change (Stein et al., 2010) and regression-based formulas (Duff, 2012), and calculating various process scores to understand patients’ approaches (Milberg et al., 2009). Because the data are already stored in a digital format, they can be easily aggregated into large normative databases (e.g., National Neuropsychology Network [Loring et al., 2022]). Furthermore, computerized paradigms have the potential to be more precise in their presentation of stimuli and measurement of response times, down to the millisecond, compared with human raters (Bauer et al., 2012; Sternin et al., 2019), depending on latency and response time of the hardware and software being used as well as speed and quality of Internet connectivity if required (Nicosia, Wang, et al., 2023; Passell et al., 2021). Unlike immutable physical stimulus booklets, computerized tests may enable creation of almost unlimited numbers of alternative forms (Zygouris & Tsolaki, 2015) using randomly selected items from a large stimuli bank, which can minimize practice effects. Finally, because computerized testing is automated, test instructions can be translated into different languages (Zygouris & Tsolaki, 2015), allowing testing of non-English speakers without requiring examiners who speak those languages.

Despite these potential advantages, there is limited uptake of computerized testing among neuropsychologists (Rabin et al., 2014). One major concern involves psychometric properties; existing studies show only moderate levels of reliability and low concurrent validity with their paper-and-pencil counterparts or clinician diagnoses (Feenstra et al., 2017; Schmand, 2019). Lack of appropriate norms or validation in the clinical populations in question also preclude consistent usage of computerized measures (Rabin et al., 2014). More importantly, most computerized paradigms validated for clinical use are essentially computerized adaptations of the original paper-and-pencil tests and do not address many limitations of traditional neuropsychological assessment as noted above. It is time for neuropsychologists to consider more radical innovations. In this review paper, we will outline novel digital tools and their potential applications to neurologic care. Specifically, we will discuss ecological momentary assessment (EMA), smartphone-based assessment and sensors, wearable devices, passive driving sensors, smart homes, voice biomarkers, and electronic health record (EHR) mining.

OVERVIEW OF NOVEL DIGITAL TECHNOLOGIES

Ecological Momentary Assessment

EMA is a method of repeatedly sampling an individual’s state, behavior, and context in real-time within the individual’s real-world environment (Shiffman et al., 2008). While EMA is not a new method, research using EMA has proliferated in the past several years, as it has become more easily conducted using mobile technology (e.g., smartphone-based administration). Using an EMA framework, information about a participant can be collected actively (e.g., surveys, smartphone-based tasks) and/or passively (using sensors from mobile or wearable devices). EMA enables the collection of longitudinal data (i.e., multiple time points throughout the day or week using active measures, continuous data streams from passive sensors), which can be used to establish real-time temporal associations among symptom self-reports, performances on cognitive and motor tasks, and contextual factors (e.g., surveys about location and activity of participants, time of day based on timestamp, location data from global positioning system [GPS]; Chen, Cherian, et al., 2022; Torous et al., 2018; Weizenbaum et al., 2022). The instantaneous nature of EMA responses reduces recall bias that is often associated with retrospective self-report inventories (Colombo et al., 2019) because patients are asked to report their states and behaviors in real time. In order to gather multiple samples within a relatively short time frame (e.g., within a day, a week, or a month), active EMA surveys/tasks are usually brief (taking only a few minutes at a time). Studies have found convergence between brief EMA symptom reports/task performances and longer gold standard measures administered at baseline in neurologic and psychiatric populations (Harvey et al., 2021; Moore et al., 2022; Sliwinski et al., 2018), validating the use of EMA to study these processes.

EMA complements traditional gold standard assessment in its ability to gather data in higher frequency and within the participant’s naturalistic setting (Shiffman et al., 2008). Such longitudinal and real-world data can help clinicians and patients better understand real-world outcomes, symptom fluctuations, and trajectories over time (Hufford et al., 2001; Schmitter-Edgecombe et al., 2020). Indeed, emerging EMA research has shown significant within-person variability in mood, fatigue, and cognitive and motor functioning that are not captured by traditional single-time-point assessments, among various populations such as heathy aging, multiple sclerosis, and Parkinson’s disease (Chen, Cherian, et al., 2022; Schmitter-Edgecombe et al., 2020; Weizenbaum et al., 2022). It has been proposed that intraindividual variability may be a more sensitive marker for cognitive decline than absolute levels of performance (Anderson et al., 2018; Halliday et al., 2019). Hackett and Giovannetti (2022) illustrated how this intraindividual variability and everyday functioning may be operationalized using digital technologies across the cognitive aging spectrum in their Variability in Everyday Behavior (VIBE) model (see Fig. 1; Hackett & Giovannetti, 2022). According to their model, cognitive abilities decrease subtly in early stages of decline, which may not be detectable by traditional clinical tools but may be captured by high dimensional digital technologies (e.g., decreased social contact or movement via EMA or passive smartphone sensors). During these early stages, intraindividual variability increases, which may be captured by repeated EMA (e.g., surveys, smartphone-based cognitive assessment). Because EMA is conducted in the individual’s real-world environment, external influences (e.g., location, stressors, mood, social context) on clinical outcomes such as cognitive performance, psychotic symptoms, and physical fatigue can be directly examined (Harvey et al., 2021; Powell et al., 2017; Shvetz et al., 2021) instead of qualitatively hypothesized. Studies investigating compliance rates of elderly individuals using EMA suggest good compliance in this population, averaging above 80% (Cain et al., 2009; Rullier et al., 2014; Schweitzer et al., 2017; Yao et al., 2023). It is also possible that these rates may be improved through the use of passive EMA methods, which are outlined below in the discussion of passive sensors, that do not require active engagement from the patient.

Fig. 1.

The Variability in Everyday Behavior (VIBE) model proposed by Hackett and Giovannetti (2022). According to this model, cognitive abilities decrease gradually in early stages of decline and more rapidly in later stages. Intraindividual variability follows an inverse U-shape pattern and peaks during the MCI stage. As it is often difficult to detect subtle cognitive changes during the prodromal and MCI stages, the authors described the advantage of high dimensional data from digital sources in characterizing cognitive functioning and variability in the real world through repeated data points (e.g., EMA, passive sensors). Figure reprinted from open-access article published by (Hackett & Giovannetti, 2022), distributed under the creative commons attribution license, which permits unrestricted use, distribution, and reproduction in any medium.

Smartphone-Based Cognitive Assessments

Given their increasing ubiquity (Pew Resarch Center, 2021), smartphones have become the ideal vehicle with which to examine real-world health outcomes. Using an EMA framework, brief cognitive tasks can now be self-administered through an individual’s personal smartphone, providing multiple brief snapshots of real-world cognitive functioning. There are a variety of mobile adaptations of gold standard neuropsychological tests such as the Stroop Color and Word Test (Moore et al., 2020), the Trail-Making Test (Fellows et al., 2017), and the Symbol Digit Modalities Test (Lam, Van Oirschot, et al., 2022), which have shown moderate to strong correlations with their paper-and-pencil counterparts. While these tests suffer from the same limits in innovation as existing computerized programs, the brief and mobile nature of smartphone-based tests allow more frequent sampling of the individual’s cognitive functioning in more diverse settings. However, unlike many more mature computerized assessments, research in smartphone-based cognitive assessment is still in its infancy and these tests cannot yet to be used for diagnostic purposes. There are ongoing efforts to validate smartphone-based assessments in large normative cohorts. One example is the National Institute on Aging (NIA)-funded Mobile Toolbox (Gershon et al., 2022), which is a smartphone adaptation of the well-established NIH Toolbox (Weintraub et al., 2013).

The biggest advantage of smartphone-based assessment is the ability to gather high frequency, ecologically valid data, which enables the characterization of individual-level changes over time (Liu et al., 2019; Nicosia, Aschenbrenner, Balota, et al., 2022). The high-frequency data may establish a stronger baseline (e.g., mean of five or ten measurements across multiple days is less influenced by confounders such as sleep deprivation and high stress compared with single-time-point measurements) to which changes may be more reliably measured, especially in cases where there are limited normative comparator samples. That being said, due to the high sampling frequency, there can be significant practice and ceiling effects (Liu et al., 2019), which may be attenuated by incorporating mass practice during the pre-baseline period until performance plateaus or utilizing many alternate forms/stimuli (Goldberg et al., 2015). With any self-administered remote data collection, compliance may be of concern. Recent studies utilizing smartphone-based cognitive assessments have shown compliance rates of greater than 70% (Cerino et al., 2021; Lancaster et al., 2019; Moore et al., 2020; Thompson et al., 2022), demonstrating its feasibility. However, of note, individuals with lower baseline cognitive scores and greater symptom burden (e.g., pain, fatigue) tend to have lower response rates (Weizenbaum et al., 2022).

Because smartphone-based assessments are typically administered in unsupervised settings, task performance may be influenced by various contextual factors, including location, motivation, ambient temperature, and recent physical and social activities (Weizenbaum et al., 2020). Unsurprisingly, stronger cognitive performances are observed at home compared with being in unfamiliar locations where there may be more distractions in the environment (Weizenbaum et al., 2022). Ambient temperature has been shown to negatively impact cognitive performance among populations with known problems with heat sensitivity such as multiple sclerosis (Pratap et al., 2020). Attention and working memory may be especially vulnerable to contextual factors (Öhman et al., 2022). While these external effects may be considered confounds, they may also provide valuable clinical information. For example, for patients with cognitive complaints who score within the normal range on traditional neuropsychological tests, the feedback session can be greatly enhanced if the clinician can incorporate real-world data, identifying situations where cognitive performance may be suboptimal. Not only can this validate the patient’s concerns, a personalized recommendation plan can be developed to minimize negative impact from acute stressors on cognition and maximize productivity by taking advantage of periods of peak cognitive performance.

Passive Smartphone Sensors

Besides being a platform to deliver active assessments, smartphones contain many passive sensors that may used to characterize real-world functioning. Smartphone studies have used their inertial measurement units (IMUs; measuring movement; consisting of accelerometer, gyroscopes, and magnetometer), GPS, microphones, cameras, keyboard typing features, and general phone usage metrics such as frequency of calls and texts to predict motor, cognitive, and mood symptoms (Chen, Leow, et al., 2022; Dagum, 2018; Jacobson et al., 2020; Lipsmeier et al., 2018; Omberg et al., 2022; Torous et al., 2018). Because smartphone sensor data are multi-dimensional, there is potential to develop highly accurate predictive algorithms to detect the presence of a disease state from healthy controls, such as multiple sclerosis, Parkinson’s disease, Alzheimer’s disease, human immunodeficiency virus (HIV), amyotrophic lateral sclerosis (ALS), schizophrenia, and mild cognitive impairment (MCI; Hackett & Giovannetti, 2022; Kourtis et al., 2019; Lam, Twose, Lissenberg-Witte et al., 2022; Lipsmeier et al., 2018; McKinney et al., 2020).

Smartphone sensor metrics have been coined as digital phenotypes or digital biomarkers, both of which refer to real-time quantification of states and behavior using digital devices including, but not limited to smartphones (Coravos et al., 2019; Prakash et al., 2021). To facilitate medical discoveries using smartphone sensors, there are open-source frameworks such as Apple’s ResearchKit and CareKit for iOS devices (Apple, 2023) and ResearchStack for Android devices (ResearchStack.org) available for app development. For those who do not have the expertise or resources to develop their own apps, there are open-source solutions available for collecting active and passive smartphone data, including mindLAMP which was developed by the Division of Digital Psychiatry at Beth Israel Deaconess Medical Center (Vaidyam et al., 2022) and mCerebrum from the Center of Excellence for Mobile Sensor Data-to-Knowledge (MD2K; Kumar et al., 2017), with options for integrating with wearable devices. The Research Electronic Data Capture (REDCap) now has a new participant-facing mobile app called MyCap (Harris et al., 2022) that includes both survey capability and performance-based tasks through Apple’s ResearchKit (e.g., finger tapping).

The biggest advantage of smartphone sensors over active smartphone-based tasks is the unobtrusive nature of data collection which does not rely on participant adherence. Since users are not required to engage with active tasks, they may be more willing to be monitored over longer periods of time, which may be more conducive to early detection of disease (Kourtis et al., 2019) compared with traditional clinic visits which do not typically take place until more apparent symptoms manifest. One example is the use of keyboard dynamics and smartphone IMU’s to passively collect data which can indicate the presence of symptoms such as essential tremor and multiple sclerosis (Brindha et al., 2022; Lam, Twose, McConchie et al., 2022). Another example is the potential to continuously assess aspects of cognitive functioning via speech dynamics using the smartphone microphone. These methods allow for continuous (or near-continuous) data collection, which provides a richer picture of daily functioning, compared with discrete time points from active smartphone-based tasks or single time points from clinic-based assessment. The highly longitudinal data can provide a more reliable picture of the individual’s functioning. Outliers due to unique circumstances (e.g., higher-than-usual stress levels, sleep deprivation) will have less influence on the overall mean. When outliers truly represent transient disease activity and processes (e.g., relapses), they will be more easily noticeable as they stand out from the average range of data points. Furthermore, the process of actively taking an assessment alters the environment and may thus alter a person’s performance as a result. Passive smartphone sensors make the act of “measuring” less obvious, and the individual is more likely to exhibit their usual behavior. Moreover, smartphone sensors may provide a more objective measure of constructs traditionally reliant on self-report, such as mood symptoms and neuropsychiatric behavioral disruptions (Barnett et al., 2018; Jacobson et al., 2019) or nonobservable physiological symptoms such as pain and fatigue (Jacobson & O'Cleirigh, 2021; Qiao et al., 2016). Metrics such as frequency of calls and messaging, time spent on calls and social phone applications, and movement in and out of the home based on GPS location can help infer mood and behavior (Barnett et al., 2018; Jacobson et al., 2019). Changes in facial expressions based on video recordings and reduction in physical activity may elucidate pain and fatigue (Jacobson & O'Cleirigh, 2021; Qiao et al., 2016). Since self-report regarding activity level and changes in social contacts can vary among individuals and within persons over time, developing objective metrics based on smartphone sensor data may minimize such bias (Harari et al., 2016; Mohr et al., 2017).

For smartphone studies that examine activity and location, one should be cognizant that smartphone sensor data may underestimate activity level as users may not always have their phones with them. Moreover, in order to observe participants’ natural behavior and budgetary considerations, many smartphone studies utilize participants’ own devices (instead of providing study devices). While this allows results to be more generalizable across platforms, there can be additional noise from differences in hardware and software which may need to be corrected. For example, older devices or operating systems may be slower and their sensors may be less accurate (Nicosia, Wang, et al., 2023). Furthermore, the impact of the immediate context has on phone sensor quality must be considered, such as high ambient temperature causing potential overheating and slowed processing. Studies, which establish norms for various devices and operating systems, will be vital to ensuring that assessments using these sensors are valid and reliable.

Wearable Devices

To gather more accurate activity and location data or physiological signals that are unavailable through a smartphone, wearable devices may be more suitable. There is a myriad of research/medical-grade and commercially available wearable devices that contain sensors such as IMUs, portable electrocardiograms or photoplethysmography to detect heart rate, portable electroencephalograms, skin temperature sensors, pedometers, pressure sensors, GPS, and cameras (Iqbal et al., 2021). These sensors can be placed on various parts of the body depending on the signal source of interest, such as the wrist, ankles, chest, head, lower back, or neck (Iqbal et al., 2021). Wearable devices have been used to study a variety of neurologic conditions including the MCI and dementia spectrum, movement disorders such as Parkinson’s disease and essential tremor, and mixed neurologic samples such as stroke, multiple sclerosis, and epilepsy (Bowman et al., 2021; Brindha et al., 2022; Khosroazad et al., 2023; Silva et al., 2017; Zhou et al., 2020). Wearable devices that can be used to monitor physiological states (e.g., heart rate, skin conductance) can further augment smartphone sensors in objectively measuring mood and psychopathology. For example, heart rate variability and skin temperature, in addition to activity levels, may help distinguish between depressive and manic episodes among individuals with bipolar disorder (Maatoug et al., 2022). For those interested in examining wearable device data, the National Institutes of Health (NIH)-funded All of Us research program contains publicly available Fitbit data, in addition to surveys, physical measurements, and EHRs (All of Us Research Program Investigators, 2019).

Like smartphone sensors, wearable devices yield high dimensional, continuous data that can be used as inputs for machine learning models with high accuracy. For example, movement and respiratory sensor-derived measures have been used to classify MCI from individuals with normal cognition (Khosroazad et al., 2023). Abnormal motor outcomes such as tremors and falls can be detected using IMU (movement)-based wearable devices (Brindha et al., 2022; Zhou et al., 2020). Many of these wearable devices and algorithms are ready for prime time, including hardware and software currently authorized by the Food and Drug Administration (FDA) to assess for sleep disorders, seizures, and Parkinson’s disease symptoms (U.S. Food and Drug Administration, 2023a, 2023b, 2023c). Besides assessment, wearable devices may be integrated into interventions. Wearable cameras have been used as memory aids, which have led to patient reports of reduced depression, higher levels of functioning in daily activities, and improved autobiographical memory (Dassing et al., 2020; Silva et al., 2017). Wearable devices may augment biofeedback interventions, most commonly explored during exercise by providing participants with real-time feedback about their gait and balance (Bowman et al., 2021). While passive sensing does not face the same compliance issue that active assessment does, it is still worth considering the role of the user in data collection and quality. For example, users will still have to remember to wear the devices, charge the devices if necessary, and ensure that the devices are worn and used correctly.

Passive Sensing of Real-World Driving

An emerging application of passive sensing technology involves monitoring of real-world driving abilities. GPS devices, cameras, radar, accelerometers, yaw rate sensors, and custom devices that made to capture vehicle operation metrics can be placed in vehicles to collect information related to driving performance every time the vehicle is operated. This includes the average/total distance traveled, travel settings (e.g., highway v. local roads), destinations, number of trips, trip timestamps, hand brake usage, sudden acceleration or deceleration, and over/under speeding (Babulal et al., 2016, 2021; Bayat et al., 2021; Eby et al., 2012; Marshall et al., 2013). Alterations in real-world driving performance have been found to be associated with symptomatic Alzheimer’s disease, such as driving shorter distances, visiting fewer unique destinations, and having a smaller driving space (Bayat et al., 2021; Davis et al., 2012; Kostyniuk & Molnar, 2008). Due to their high-frequency nature, sensor-derived driving data may elucidate subtle cognitive decline in preclinical stages, during which time the patient may not exhibit clinical impairment on traditional neuropsychological tests (Allison et al., 2016; Bayat et al., 2021; Coughlan et al., 2018). Studies have found alterations in real-world driving behavior and increased driving errors in people with preclinical Alzheimer’s disease when compared with healthy controls, demonstrating the feasibility of using driving performance as a neurobehavioral marker for early stages of decline (Babulal et al., 2021; Roe et al., 2017, 2019).

Detecting driving behavior using sensors is advantageous over traditional methods of assessing driving performance, such as on-the-road tests, driving simulators, and self-report inventories and diaries (Babulal et al., 2016; Bayat et al., 2021), which suffer from limitations in objectivity, cost-effectiveness, availability, reliability, and generalizability (Bayat et al., 2021; Eby et al., 2012). By repeatedly collecting data points on metrics such as speed, acceleration, and location throughout operation of the vehicle, driving behaviors and performance can be assessed continuously in a naturalistic environment, with high ecological validity and more precise characterization of intraindividual performance over time (Babulal et al., 2016). Beyond assessment, real-time monitoring of driving may help protect older or impaired drivers. Abnormal driving behaviors can be flagged in real-time and alert caregivers, and GPS allows location of drivers who have become disoriented and lost (Babulal et al., 2021). Furthermore, as some car manufacturers have started to do so, sensor-derived driving metrics may inform real-time feedback and assistance features (e.g., warning drivers of incoming obstacles, directing driver during difficult lane changes).

While promising, there are still many methodological issues that need to be worked out in this research. Depending on budget and feasibility, sensors and analytic approaches vary significantly across studies. There is no consensus regarding common data elements for assessing real-world driving. Another major limitation is the fact that older drivers and drivers in beginning stages of decline often compensate (driving less frequently, avoiding more difficult driving situations [e.g., highways, nighttime] Crizzle & Myers, 2013; Seelye et al., 2017), subtle changes in driving ability (e.g., slower reaction time when braking, not seeing oncoming traffic when turning or yielding) require a closer examination of specific driving behaviors and situations, which has not been well explored by extant literature.

Smart Homes

Smart homes refer to residences which integrate various automated electronic systems into the home (Robles & Kim, 2010). This can range from a home which contains intelligent objects and systems that an individual can interact with using an electronic home controller (Robles & Kim, 2010) to homes with sensors built into any number of appliances or items, ultimately allowing for unobtrusive gathering of data (Martin et al., 2008). Examples include smart carpets and pressure mats used to passively record aspects of a person’s gait, automated pill boxes tracking medication adherence, sensors such as accelerometers built into the bed monitoring sleep quality, motion detectors monitoring movement between rooms and in/out of the house, power usage, and ambient temperature (Ahamed et al., 2020; Gali et al., 2020; Khosroazad et al., 2023; Kim et al., 2020; Santos et al., 2022). As a result, smart homes can simultaneously capture multiple streams of information about the environment, the people within it, and how people interact with the environment and each other. Not only do the capabilities of smart homes allow information to be passively collected, but they also allow for active interaction with tenants for interventional purposes (Martin et al., 2008). Automated processes can activate smart light bulbs, audio from smart speakers, or other types of notification systems to alert and interact with those in the home (Ault et al., 2020; Boumpa et al., 2019; Gali et al., 2020) and robotic systems can physically assist people in daily activities (Wilson et al., 2019).

Studies using smart home technology have demonstrated utility in detecting MCI, dementia, and sleep-related disorders in elderly individuals (Ahamed et al., 2020; Khosroazad et al., 2023; Kim et al., 2020). Technologies such as smart carpets/mats, smart lights, smart speakers, and robots have shown to be effective tools in rehabilitation, assistance in activities of daily living, and emergency intervention for individuals with gait or walking related issues and dementias (Ault et al., 2020; Boumpa et al., 2019; Gali et al., 2020; Santos et al., 2022; Wilson et al., 2019). Additionally, smart home technologies have the ability to improve caregivers’ quality of life (Ault et al., 2020), through robotics that can perform caregiving tasks in place of the caregiver and automated alert systems that can reduce the amount of active oversight of patients required from the caregiver (Ault et al., 2020; Boumpa et al., 2019; Gali et al., 2020; Santos et al., 2022; Wilson et al., 2019). For the interested reader, the Oregon Center for Aging and Technology at the Oregon Health & Science University has made their smart home data available for external researchers (Lyons et al., 2015). Of note, the comprehensiveness of smart homes in capturing an individual’s daily life can also be a disadvantage, as it can feel invasive for the user. Privacy-protecting measures, such as blurring faces in video data, having clear indicators to individuals in the household that video and/or audio recording is occurring, the ability for users to turn recordings on or off, and password protection, should be considered. The large quantities of data from smart homes may introduce analytic challenges. Future research on how to best synthesize and interpret these large amounts of multimodal data will be important for the practical use of smart homes in continuous patient assessment and monitoring.

Voice Biomarkers

Voice biomarkers refer to the features captured from audio signals of human voices (Fagherazzi et al., 2021). Voice biomarkers can elucidate speech alterations (i.e., changes in duration of unvoiced segments, speech rate, duration of syllables), which may be one of the earliest indicators of cognitive decline (Ambrosini et al., 2019; Calzà et al., 2021). This is meaningful because studies suggest that neuropathological changes could begin many years before the onset of apparent clinical symptoms (Hajjar et al., 2023; Mahon & Lachman, 2022) and may be difficult to detect by the patients themselves or through traditional neuropsychological assessment. For example, using its rich longitudinal data, researchers from the Framingham heart study (Lin et al., 2020) found that early speech alterations (e.g., more hesitations and pauses) during the asymptomatic phase were predictive of the later development of dementia. Since voice biomarkers can easily be recorded, they can be used to monitor patients’ cognitive status in a simple, rapid, and reproducible way and assist with clinical diagnosis (Alhanai et al., 2017). Speech and language analyses usually require large corpuses, or collections of written or spoken words. Some of the most commonly used publicly available corpuses come from the TalkBank project (MacWhinney, 2007), which includes speech recordings and transcripts from a myriad of clinical populations, including dementia, aphasia, and traumatic brain injury.

There is a variety of methods to collect voice biomarker data, ranging from recording spontaneous speech to structured language tasks (Robin et al., 2020), which may be analyzed using speech and language algorithms (e.g., acoustic analysis of speech, automatic speech recognition, natural language processing [NLP]). Regarding spontaneous speech, an experienced clinician may be able to qualitatively detect subtle speech abnormalities that reflect cognitive deficits (e.g., paraphasias, word-finding difficulties, abnormalities in pitch, clarity, and intensity) during the clinical interview, which may be quantified using automatic speech analysis techniques (Martínez-Nicolás et al., 2021). Even with existing language tasks (e.g., verbal fluency, confrontation naming), speech algorithms may provide additional process information (e.g., speed or pattern of word retrieval, error types) which can greatly complement the total summed scores (Putcha et al., 2020). In terms of limitations, cross-language studies are scarce with the majority of research in this area conducted with English samples, although some studies demonstrated the feasibility of using automatic speech analysis across different languages (Chien et al., 2018; Espinoza-Cuadros et al., 2014; Mirzaei et al., 2018; Satt et al., 2013). For example, a multilingual Parkinson’s disease study identified certain voice features that were considered language independent, at least among English, Italian, Spanish, German, and Czech speakers in the study. These language independent features included pitch variation, speech rhythm, pause time, silence duration, length of speech, and number of nouns and auxiliaries (Favaro et al., 2023).

Due to the early stage of this type of research, there are currently no standard definitions of voice biomarker features, but some of the most common features can be classified as lexical, syntactic, acoustic, and paralinguistic (Beltrami et al., 2018). Lexical features are derived from the words and meanings that people use when they speak. Some common lexical features include vocabulary size, content density (i.e., the ratio of open-class words [nouns, verbs, and descriptive terms] to closed-class [grammatical function] words) (Land Jr et al., 2020), and lexical content (i.e., proportions of pronouns and verbs) (Ahmed et al., 2013). People with cognitive impairment may experience word-finding difficulties, which may manifest as less frequent use of high-frequency words and word modifiers (i.e., adjectives) (Yeung et al., 2021). Syntactic features investigate grammatical correctness, coherence, and syntactic complexity (i.e., mean length of utterance, number of embedded clauses, syntactic errors, nouns preceded by determiners and verbs with inflections). Patients with cognitive deficits may show more syntactic errors and general simplification (e.g., fewer complex phrases and embedded structures) in their speech compared with people with normal cognition (Zanini et al., 2010). Acoustic features are the characteristics of speech sounds, including language fluency (i.e., production rate, the number of pauses, hesitation), response time, length of segments, and the false start of the sentences (Alhanai et al., 2017; Ammar & Ayed, 2021). More unique features such as relative variation in energy and vocal pitch can also be extracted from sound files (Rapcan et al., 2010). Extant research indicates that compared with acoustic features, more impairments are observed in lexical and syntactic features in early stages of cognitive decline (Ahmed et al., 2013; Alhanai et al., 2017).

Electronic Healthcare Record (EHR) Mining

As of 2021, 88% of the hospitals in the United States have adopted EHRs (Office of the National Coordinator for Health Information Technology, 2022). EHRs have enabled the aggregation of complex patient data into easily accessible databases that better inform diagnosis and treatment compared with paper-based medical record systems (Payne et al., 2015). These databases, including structured fields about patients’ demographics, diagnoses, procedures, medications, and test results, have allowed researchers to study large samples of clinical populations without the expense of collecting their own data (Casey et al., 2016). Besides quality improvement within a particular healthcare system, researchers can use the EHR for public health inquiries, identifying risk factors for disease, determining the effects of treatments through natural experiments, and evaluating risk for various clinical/functional outcomes (Casey et al., 2016). For example, EHR data have helped uncover clinical outcomes associated with the coronavirus disease (COVID-19) infection, including the risk of developing neurologic and psychiatric conditions (Taquet et al., 2021). There are publicly available EHR data that can be leveraged by external researchers, including the Medical Information Mart for Intensive Care III (MIMIC-III; Johnson et al., 2016), the All of Us research program (All of Us Research Program Investigators, 2019), and the National COVID Cohort Collaborative (N3C; Haendel et al., 2021). In recent years, researchers also started exploring the possibility of integrating patient-generated data from digital systems with real-time monitoring function into EHRs. For example, sleep data from smartphone apps can be imported into EHR for clinicians to review, which could be more reliable than patients’ retrospective self-report data (Zulueta et al., 2020).

In addition to the structured fields that can be relatively easy to extract and analyze, there is a large proportion of unstructured clinician notes that require more advanced analytic methods such as NLP (Koleck et al., 2019). The most common approach is to identify frequencies of particular words and phrases that are related to the population or construct of interest within the free text notes. These features may then be used as inputs for algorithms to predict a clinical outcome of interest (e.g., diagnosis). Studies have shown the feasibility of predicting risk for neurologic conditions such as dementia, multiple sclerosis, and delirium by searching for condition-related terms (e.g., symptoms) within unstructured notes (Chase et al., 2017; McCoy Jr. et al., 2019; Pichon et al., 2021; Wong et al., 2018). EHR algorithms have the potential to identify onset of disease before formal diagnosis, which may lead to more timely interventions (Chase et al., 2017; McCoy Jr. et al., 2019; Wong et al., 2018). Besides utilizing symptom keywords, EHR data have also been used to identify lifestyle risk factors for cognitive decline, which can further refine risk algorithms (Zhou et al., 2019).

Because EHR data are collected for clinical, and not systematic research purposes, there are several limitations in using them to answer questions for which the data were not collected. There can be large variances across providers and health systems in terms of data completeness, accuracy, and consistency (Kataria & Ravindran, 2020; Sarwar et al., 2022). There are natural biases in the EHR based on the patient’s specific medical circumstances. For example, sicker patients tend to have more health records compared with healthy patients (Graham et al., 2020; Weiskopf et al., 2013). These factors may be potential confounds that need to be controlled.

DISCUSSION

The digital tools reviewed in this paper have tremendous promise in bringing neuropsychology into the 21st century. Singh and Germine (2021)’s hybrid neuropsychology model provides a useful road map for integrating digital tools into neuropsychological practice, including gradually incorporating digital methods of data collection, leveraging data science in the collection, storage, and retrieval of multimodal sources of data, and engaging in interdisciplinary collaborations (see Table 1) (Singh & Germine, 2021). However, there is still much work to be done before these applications are ready for standard clinical use. In the following sections, we describe ethical considerations, limitations of current research, and our vision of the future of neuropsychology.

Table 1.

Hybrid neuropsychology model proposed by Singh and Germine (2021).

	Action item 1	Action item 2	Action item 3
Action Item	Develop a technology-based practice	Integrate data science	Engage with innovators in other fields
Goal	Incorporate multiple modalities when collecting patient data	Aggregate data collected across patients and multiple modalities	Collaborate with innovators to develop, share, and implement new digital tools
Benefits	• Timely delivery • Increased precision and accuracy • Ability to monitor changes in cognition • Access to state- and context-dependent functioning • Greater scalability of the field • Improved health equity	• Improved methods for accessing, storing, and sharing patient data • Greater consistency and integration of data points collected across clinics • More straightforward development of a national data repository and new tests	• Better integration with technology-based fields • Ability to fill gaps in evaluations by measuring constructs not currently assessed using traditional tests • Greater exposure to medical and scientific community
Barriers	• Validity of novel digital tools • Resistance to adopt innovative technologies	• Less familiarity with data science • Reaching a consensus regarding types of data management tools to use	• Risk aversion
How to address barriers	• Cross-validate new tools with traditional tests • Ensure validity through empirical studies and strengthen communication within the field	• Collaborate with individuals in information technology and data science • Increase communication within the field regarding best practices	• Recognize that neuropsychology can be innovative without eliminating traditional, trusted measures

Note: Table from (Singh & Germine, 2021), with permission to reprint from the source journal.

Ethical Considerations/Limitations of Digital Technologies

Patient privacy and confidentiality are major concerns for any remote monitoring approach. To prevent unauthorized access of patient data, proper encryption infrastructure for both storage and transmission of data (e.g., uploading from device to cloud-based storage) should be in place. Collection of identifying information through these technologies should be avoided when possible. If a participant has already enrolled in a study or clinic, an anonymized research or clinic ID may be used to log into the apps as opposed to their names to minimize the impact of breach of confidentiality. If age is required for the algorithm, age instead of date of birth should be used if it does not interfere with the research or clinical protocol. Collection of information that is not essential for the research or clinical question should be avoided to protect privacy. For example, in most of the keystroke dynamics studies (Alfalahi et al., 2022), the content of what is typed (i.e., exact words and sentences) is not typically gathered, since researchers are primarily interested in the pattern of typing (e.g., typing speed, rate of typing errors) and not the content. Meanwhile, it is worthwhile to be cautious when using third-party software and hardware to collect data which may lead to personal identification in unexpected ways. For example, sensor data may reveal IP addresses (Martinez-Martin, Greely, et al., 2021; Martinez-Martin, Luo, et al., 2021). Furthermore, it is important to verify the identity of the user when they log into the apps, both to protect any demographic or clinical information that can be accessed through the app dashboards and to ensure that any intervention triggered by information collected through the devices is from the actual user (not someone else using their device). User accounts should be password-protected at a minimum, and more advanced biometric verification (e.g., fingerprint, facial features) may be considered as well. For remote monitoring methods that may involve people other than the patient in question (e.g., other household members in a smart home, bystanders captured by wearable cameras), consent from these external individuals may need to be obtained. If obtaining consent is not feasible, measures need to be taken to ensure privacy of these individuals (e.g., giving participants the option of turning off a wearable camera as needed, blurring the faces of bystanders automatically during preprocessing of videos before analysis).

Technological/digital literacy is another important consideration. Some of the aforementioned technologies require more active engagement from the user (e.g., smartphone-based assessments), which may not be appropriate for individuals with limited familiarity with these devices and software, illiterate or linguistically diverse populations, or cognitively impaired clinical groups. As a result, it will be important to account for factors such as comfortability and familiarity with these devices and whether variance in these factors will contribute to reduced reliability in measurements. Of note, research has shown that lower levels of technological literacy do not inherently lead to reduced adherence (Nicosia, Aschenbrenner, Adams, et al., 2022), highlighting the importance of considering more than technological literacy when assessing whether a particular technology is appropriate for a person. This way, steps can then be taken to reduce these confounding factors, such as making use of more passive monitoring approaches (e.g., smart homes) with some individuals. While smartphones and many wearable devices are becoming increasingly ubiquitous in the United States even among lower resourced populations (Pew Resarch Center, 2021), this may not be true for all parts of the world, which may have limited access to these technologies. Even among groups who own their own devices such as smartphones and wearables, research has demonstrated differences in quality (e.g., lag times) among devices of varying prices; unsurprisingly, less expensive smartphones have longer display and touch latencies than more expensive smartphones (Nicosia, Wang, et al., 2023). This is an important caveat especially when response time is a primary outcome. Without appropriate adjustments for hardware/software, individuals with low socioeconomic status may be unfairly penalized.

Limitations of Current Research/Future Directions

Before these technologies can be used in standard neurological/neuropsychological practice, it is important to address their current limitations. Further psychometric studies are needed to demonstrate the reliability and validity of these new metrics (Bauer et al., 2012). However, simply validating these novel technologies with traditional paper-and-pencil tests is not enough. Since the goal of the novel technologies is to address the limitations of traditional neuropsychological measures, it is critical to utilize ground truths that more accurately represent the patient’s real-world functioning, such as subjective (especially from informants) and objective measures of activities of daily living and clinician diagnoses. Also, one of the largest limitations present in all the novel digital tools reviewed is the restricted generalizability of existing research due to small and homogenous samples. Thus, it will be important for future research to verify the validity of these technologies with larger and more diverse samples. In portable active assessment technologies such as smartphone-based cognitive assessment or symptom questionnaires via EMA, potential confounders in the environment can continuously arise, such as distracting environments and device/connectivity failure. Thus, it will be important to gather contextual information through device metadata or additional sensors. This will help elucidate unintentional confounders but also for intentional cheating, which may be difficult to monitor without human supervision (Nicosia, Wang, et al., 2023).

Furthermore, some studies demonstrated limitations in specificity when distinguishing among different diseases and syndromes (Hajjar et al., 2023; Lin et al., 2020), necessitating further studies using modern psychometrics and machine learning methods to identify more specific phenotypes. This is an important step to overcoming the specificity issues already present in traditional neuropsychological assessment through the use of higher dimensional data that allows for more accurate predictive models. Another potential limitation is the between-person reliability of various hardware and software. Due to the rapid rate at which manufacturers update devices and varying quality levels of devices, between-person differences in factors such as the processing speed of devices or quality of recorded information pose a limitation to the reliability of measurements performed by these devices (Germine et al., 2019). Thus, despite potential improvements in precision over human raters, variables such as differences in device display and latencies (Nicosia, Wang, et al., 2023; Passell et al., 2021) must be controlled. Hardware- and software-specific norms will also be helpful. Practice effects are another limitation that will have to be accounted for in these continuous or high dimensional data streams (Hackett & Giovannetti, 2022; McKinney et al., 2020). Digital stimuli are uniquely equipped to address this limitation by creating almost unlimited numbers of alternate forms.

Future of Neuropsychology

We envision future neuropsychological practice to incorporate some if not most of these novel digital technologies. Unobtrusive monitoring technologies (e.g., smart home and wearable sensors) may be easily integrated into the lives of at-risk populations (e.g., older adults) to monitor for possible decline. This addresses a common problem encountered in neuropsychological evaluations, where it is often difficult to estimate the patient’s premorbid functioning. Diagnostic algorithms embedded within the EHR may serve as another early detection system and alert providers when their patient may be at risk of cognitive decline and appropriate referrals for neurological or neuropsychological services can be made. Risk scores may also be developed using these EHR algorithms to help with triage decisions.

For the actual neuropsychological evaluation, it is not implausible for clinicians to tap into the patient’s existing devices (e.g., smartphones, wearables) to get a sense of their real-life functioning with appropriate informed consent, which may complement their in-clinic assessment. Computerized assessment may make testing more efficient (and keep more precise response times) and reduce burden for both the patient and the clinician. For those concerned about not having human observation of important qualitative information during computerized assessment, many behavioral indicators can be easily programmed into the testing software. For example, time spent on each screen can be recorded, which may help detect inattentiveness to task instructions and/or impulsivity. Additional sensors (e.g., cameras, eye-tracking glasses, activity, and physiological monitoring) may be deployed concurrently to monitor for distractibility, hyperactivity, and anxious mood states that may negatively affect cognitive performance. Voice biomarkers derived from the clinical interview and other language-based tasks may help detect more subtle changes in speech. If the clinician is concerned for a specific instrumental activity of daily living, they may order remote monitoring of that particular activity in the patient’s real life (e.g., driving [Seelye et al., 2017]); no longer do clinicians have to rely primarily on self- or informant-reports. For individuals diagnosed with dementia, it will be crucial to monitor for functional decline and safety (e.g., risk of falls). Smart home sensors may help keep these patients out of institutions for as long as possible, especially for those without 24/7 caregivers.

Besides detection, smart homes may consist of interventional elements, such as voice-based reminders for medications and appointments and contacting emergency services if a possible fall is detected. Wearable and smartphone apps may be used for just-in-time adaptive interventions, which are interventions that are delivered at the right time based on the person’s internal state or context (Nahum-Shani et al., 2018), such as an app that prompts the user to exercise after long sedentary periods (based on activity monitoring) and delivers a mindfulness exercise when heightened anxiety is detected (through heartrate monitoring). We posit that the field of neuropsychology should embrace these technological advancements. Due to financial incentives, these technologies are already being (and will continue to be) applied to neuropsychological care by industry. It is imperative for neuropsychologists to have a seat at the table during the development and validation of these digital tools, in order to maintain scientific rigor, maximize impact on patient care, and minimize potential harm to patients.

FUNDING

This article was partially supported by the National Academy of Neuropsychology and the Eunice Kennedy Shriver National Institute of Child Health and Human Development (K23HD104855).

CONFLICT OF INTEREST

None declared.