Clinical classification of memory and cognitive impairment with multimodal digital biomarkers

Abstract

INTRODUCTION

Early detection of Alzheimer's disease and cognitive impairment is critical to improving the healthcare trajectories of aging adults, enabling early intervention and potential prevention of decline.

METHODS

To evaluate multi‐modal feature sets for assessing memory and cognitive impairment, feature selection and subsequent logistic regressions were used to identify the most salient features in classifying Rey Auditory Verbal Learning Test‐determined memory impairment.

RESULTS

Multimodal models incorporating graphomotor, memory, and speech and voice features provided the stronger classification performance (area under the curve = 0.83; sensitivity = 0.81, specificity = 0.80). Multimodal models were superior to all other single modality and demographics models.

DISCUSSION

The current research contributes to the prevailing multimodal profile of those with cognitive impairment, suggesting that it is associated with slower speech with a particular effect on the duration, frequency, and percentage of pauses compared to normal healthy speech.

1. BACKGROUND

Dementia causes substantial loss of health and quality of life and represents a significant health burden on individuals, their families, caregivers, and society at large. ¹ Dementia is currently the most costly health condition in the United States. ² Globally, the cost of caring for those with dementia was $1.3 trillion in 2019 and is expected to reach $3 trillion in 10 years’ time. ³ , ⁴ While pharmaceutical interventions like Lecanemab and Donanemab have recently demonstrated measurable slowing of disease progression, there are currently no widely available disease‐modifying treatments for Alzheimer's disease (AD). ⁵ , ⁶ , ⁷ , ⁸ When these treatments eventually do become widely available, their risk profile is such that assessment of cognitive impairment is warranted before prescribing. ⁹ In addition to recent pharmaceutical successes, there is also strong scientific evidence which indicates that early engagement in lifestyle modification strategies like physical activity and social interaction amongst others can reduce the risk of developing dementia by as much as 40%. ¹⁰ , ¹¹ Thus, the early detection of cognitive impairment is critical to improving the healthcare trajectories of aging adults, enabling early intervention and management, as well as potentially preventing or mitigating disease progression.

Traditional cognitive screening tools (e.g., Mini Mental State Exam (MMSE), Montreal Cognitive Assessment (MoCA), or MiniCog) lack sensitivity in populations at risk of or at mild stages of cognitive decline. ¹² , ¹³ , ¹⁴ , ¹⁵ , ¹⁶ Patients often present with cognitive complaints long after subtle symptoms began, thus these traditional tests are often administered after the onset of overt symptoms. Traditional assessments also do not easily fit the time constraints of primary‐care settings‐which are frequently at the forefront of cognitive screening‐without significant workflow modifications. Previous research has indicated that a reliable, objective, digital assessment may significantly increase the accuracy in identifying early cognitive impairment and those likely to convert to AD. ¹⁷

Recently, the Digital Clock and Recall (DCR) has emerged as a novel analysis of graphomotor and delayed recall memory performance. The DCR, a medical device registered in the United States, European Union (EU), and Canada, combines a digital clock drawing test (DCTclock), with a three‐word immediate and delayed verbal recall test. ¹⁸ , ¹⁹ , ²⁰ Importantly, unlike the MMSE, MoCA, or MiniCog which are administrator‐scored and demonstrate low inter‐rater agreement; the DCR enables multimodal assessment of the process through which an individual performed these cognitive tasks, as opposed to only observing the final outcome. Cognitive domains assessed by the DCTclock include drawing efficiency, simple and complex motor function, information processing, visuospatial reasoning, along with the explicit assessment of verbal memory. The DCTclock can detect early signs of cognitive impairment by capturing a wide variety of cognitive and motor metrics during the clock‐drawing process using machine learning (ML) algorithms. ²⁰ , ²¹

With the inclusion of a verbal recall task, during which speech utterances are digitally recorded and analyzed, speech timing and acoustic voice features may be included in the analysis of cognitive status. Speech and voice feature analysis is feasible, scalable, and cost‐effective compared to comprehensive neuropsychological batteries which can take several hours to complete. Speech and voice metrics have shown promise in classifying individuals with cognitive impairment from their cognitively unimpaired (CU) counterparts in clinical and research settings. ²² , ²³ , ²⁴ Further, high accuracy has been reported in differentiating CU individuals and those with AD (88%–97% accuracy) using various ML models that incorporate predominantly speech and voice features. ²⁵

The current investigation evaluated the clinical utility of combining multiple modalities including speech and voice, graphomotor features, and verbal recall accuracy into a brief, objective cognitive assessment. We propose a novel multimodal machine learning model, known as the DCRP (DCR plus voice and speech features) for classifying memory impairment using the Rey Auditory Verbal Learning Test (RAVLT) long delay score. We also identified the set of features most important for classifying memory and global cognitive impairment.

RESEARCH IN CONTEXT

1. Systematic review: Current literature was reviewed using traditional search engines (e.g., PubMed, Embase, CIELO). This literature review revealed limited but strong evidence that dementia and cognitive impairment may be associated with speech timing and acoustic voice patterns. However, certain inconsistencies were seen in the manner in which analysis was performed and very few results were available on the analysis of short, verbal recall tasks using multimodal modeling approaches.

2. Interpretation: The current research suggests that accurate predictions of memory impairment can be made from a 3 minute digital cognitive assessment. It also suggests that voice and speech features extracted from as few as three words can aid in the prediction of cognitive impairment.

3. Future directions: Future work in this field should investigate the additional predictive accuracy of multimodal analyses including speech and voice features in the prediction of blood‐ and brain‐based biomarkers of amyloid and tau burden. It should also investigate these features longitudinally, as well as in conjunction with behavioral and pharmaceutical interventions.

2. METHODS

A sample of 922 older adults (537 female, aged 72.0 ± 6.7 years) completed a battery of cognitive and motor function tests as part of the Bio‐Hermes study on brain health (Table 1). The Bio‐Hermes study, managed by the Global Alzheimer's Platform (GAP), investigates state‐of‐the‐art methods in brain health research. ²⁶ Ethical approval was granted by each institution participating in the GAP consortium. All participants provided written informed consent to participate in the study. Inclusion criteria were adults 60–85 years of age, willing to undergo an amyloid PET scan within 60 days of signing informed consent; have a study partner who, in the investigator's judgment, has sufficient and frequent contact with the participant and is able to provide accurate information regarding the participant's cognitive and functional abilities; willing to comply with all study procedures as outlined in the informed consent, including blood sampling; willing to be contacted for possible participation in clinical research trials once their participation in this study ends, fluent in the language of the tests used and the test site, and with a MMSE score of 20–30 at screening. Exclusion criteria were extensive and based on underlying conditions. In short, exclusion criteria included participants who, in the opinion of the Site Principal Investigator, have serious or unstable medical conditions that would prohibit their completion of all study procedures and data collection; serious or unstable medical conditions that would likely preclude their participation in an interventional research trial; are unable to undergo amyloid PET; have reported or have a known negative amyloid PET scan in past 12 months; participants with self‐reported, untreated conditions such as vitamin B₁₂ or folate deficiency or bladder infections; history of stroke or seizures within 1 year of screening visit; participants with history of certain cancers within the past 5 years; known or suspected alcohol or drug abuse or dependence; a Geriatric Depression Scale (GDS) score greater than or equal to 8; and participants who are unable to complete cognitive testing due to inadequate visual or auditory acuity. More information can be obtained by contacting the GAP consortium and in recently published material. ²⁶ , ²⁷ , ²⁸

TABLE 1.

Age, gender, ethnicity, handedness, education, and race demographics of the 942 study participants.

		Gender		Ethnicity		Handedness			Education			Race
Cohort	Age Mean ± SD	Male	Female	Hispanic and Latino	Non‐Hispanic	Right	Left	Ambi.	Less than HS	HS degree	Some college	White	Black	Asian	Other
All subjects	72 ± 7	405 (43%)	537 (57%)	112 (12%)	830 (88%)	843 (89%)	85 (9%)	14 (2%)	24 (2%)	144 (15%)	258 (27%)	807 (86%)	108 (11%)	17 (2%)	10 (1%)
RAVLT impaired	73 ± 6	235 (54%)	198 (45%)	53 (12%)	380 (87%)	382 (88%)	42 (9%)	9 (2%)	15 (3%)	81 (18%)	114 (26%)	366 (84%)	55 (12%)	8 (1%)	4 (1%)
RAVLT unimpaired	72 ± 7	170 (33%)	339 (66%)	59 (11%)	450 (88%)	461 (90%)	43 (8%)	5 (1%)	9 (1%)	63 (12%)	144 (28%)	441 (86%)	53 (10%)	9 (1%)	6 (1%)
Significance RAVLT impaired vs unimpaired	‐2.4 (p = 0.01)	40.7 (p < 0.001)		NS					6.7 (p = 0.009)		8.3 (p = 0.004)	NS
MMSE < 27	74 ± 6	161 (43%)	210 (56%)	54 (14%)	317 (85%)	337 (90%)	30 (8%)	4 (1%)	20 (5%)	72 (19%)	107 (28%)	292 (78%)	67 (18%)	9 (2%)	3 (1%)
MMSE ≥ 27	71 ± 7	244 (42%)	327 (57%)	58 (10%)	513 (89%)	506 (88%)	55 (9%)	10 (1%)	4 (1%)	72 (12%)	151 (26%)	515 (90%)	41 (7%)	8 (1%)	7 (1%)

	Gender		Ethnicity		Handedness			Education			Race
Age Mean ± SD	Male	Female	Hispanic and Latino	Non‐Hispanic	Right	Left	Ambi.	Less than HS	HS degree	Some college	White/non‐Hispanic	Black/African American	Asian	Other
72 ± 7	405 (43%)	537 (57%)	112 (12%)	830 (88%)	843 (89%)	85 (9%)	14 (2%)	24 (2%)	144 (15%)	258 (27%)	807 (86%)	108 (11%)	17 (2%)	10 (1%)

Notes: Percentages indicate the proportion of the total number of participants. T‐test statistic results for “Age” and Chi2 statistics (p values) for “Gender” and “Education” are provided for significant differences between RAVLT determined cognitive impairment groups. NS indicates non‐significant results between groups.

Abbreviations: Ambi., ambidextrous; HS, high school; MMSE, Mini‐Mental State Examination; RAVLT, Rey Auditory Verbal Learning Test.

Each participant received a clinical examination which produced a cohort status of Cognitively Normal, Mild cognitive Impairment (MCI), or Alzheimer's Disease Related Dementia (ADRD) based on physician clinical judgment, objective evaluation of functional impairment on the Functional Activities Questionnaire, and results of the RAVLT ²⁹ , ³⁰ , ³¹ and the MMSE. ³² Participant scores on the RAVLT (age‐adjusted long delay) and MMSE (total score) are detailed in Table 2. The age‐adjusted RAVLT long delay ranges from 0 to 15 and is scored as the number of words recalled after a roughly 20‐minute delay from the first five immediate recall trials. For the purposes of this study, the sample was classified into cognitively “impaired” and “unimpaired” based on the RAVLT long delay scores. Impairment on the RAVLT long delay was defined as at least one standard deviation below the age‐adjusted norms. ³³ , ³⁴ , ³⁵ , ³⁶ Included in the examination was the DCR (three‐word verbal recall and clock drawing). The participants’ voices were recorded and analyzed for acoustic and speech features during both immediate and delayed three‐word recall tasks. Cognitive testing was administered via an iPad Pro (11″, 4th Generation, Apple, CA, USA), and data including voice recordings were encrypted on the tablet and uploaded to a secure cloud server.

TABLE 2.

RAVLT age‐adjusted delayed recall scores alongside mean and standard deviation delayed recall score.

	Cognitive status		Score (Mean ± SD)
Target	Impaired	Unimpaired	Impaired	Unimpaired
RAVLT	433 (54%)	509 (46%)	2.8 ± 1.8	8.5 ± 2.3

Note: Percentages indicate the proportion of the total number of participants.

Abbreviation: RAVLT, Rey Auditory Verbal Learning Test.

T‐test statistic results for “Age” and Chi2 statistics (p values) for “Gender” and “Education” are provided for significant differences between RAVLT‐determined cognitive impairment groups (Table 1). These results are consistent with previous literature and RAVLT norms established in the field. ³³ , ³⁷

2.1. Temporal and acoustic analysis

Audio was time‐segmented and transcribed using Amazon Web Services (AWS) Automatic Speech Recognition (ASR) platform “Amazon Transcribe” (www.aws.amazon.com/transcribe/). Word‐level time‐stamped transcriptions are then used to determine verbal recall accuracy and dynamics of speech rate and timing. The Librosa package (www.librosa.org/) was applied to three‐word recall audio to extract acoustic features of fundamental frequency (e.g., mean, standard deviation, minimum, maximum, and jitter), loudness (e.g., mean, minimum, maximum, standard deviation, and shimmer). Metrics of speech timing (word and syllable rates, overall time to complete speech segments, etc.) were calculated using the AWS ASR output. ²⁵ , ³⁸ , ³⁹ , ⁴⁰ The Librosa analysis package allows for the detection of speech obtained from decibel (dB) and frequency (Hertz; Hz) bands where speech segments are most likely found (30–80 dB and 70–500 Hz threshold bands were used in this analysis; Figure 1). Speech timing features included initiation time, speech rate per word (total audio time/# of spoken words), speech rate per syllable (total audio time/no. of spoken syllables), and short and long silent speech pauses. Importantly, long silent speech pauses (sections of audio that did not reach the 30 dB threshold) in the current research were considered pauses of 2 seconds or longer, occurring between the start and end of the speech. These pauses were included in the analysis only if they occurred during speech and did not include pauses before the speech began or after the final word was produced. This ensures a “true” long silent speech pause metric. Short silent speech pauses were those shorter than 2 seconds long. The resultant output of the semi‐automated analyses was manually reviewed for accuracy. A complete list of the temporal and acoustic features extracted from speech are listed in Tables S1 and S2. Speech timing and acoustic voice features obtained from the audio were subsequently included as candidate features for selection to the DCRP model, to classify cognitive impairment status.

FIGURE 1.

Above depicts the waveform over time and the estimated speech segments (in green) based on dB level analysis (y axis) over time (x axis; in seconds). The red trace follows fluctuations in dB above the speech threshold. A sample of DCTclock drawing (below left) is provided and scored in terms of drawing efficiency, simple and complex motor, information processing, and spatial reasoning (below right). This individual was classified as having cognitive impairment. DCTclock, digital clock test.

2.2. Clock drawing analysis

DCTclock derived features contributed significantly to DCTclock, DCR, and DCRP models. Those found in each model's feature set in Table 3 were selected from 252 drawing position, timing, and drawing rate features originally produced by the DCTclock algorithm. This algorithm utilizes a hierarchy of low‐level features which are aggregated in a series of eight composite scales: four for the Command Clock (i.e., drawing a clock with a particular time without a visual cue; COM) and four for the Copy Clock (i.e., redrawing a clock on the basis of a visual example; COP). These composite features are then themselves aggregated into a single DCTclock score (0–100), which is used to classify a participant's overall cognitive status. Some of these composite features were recurrently selected in the three aforementioned models, including COM Component Placement (i.e., a feature measuring the spatial relationship between command clock drawing attributes), COP and COM Information Processing (i.e., copy and command clock features used to assess non‐motor cognitive function, reliant on low‐level features such as stroke latency and the rate of time spent thinking rather than drawing), and COM Spatial Reasoning (i.e., a feature assessing the geometric properties of the command clock drawing, including the circularity of the clock face as well as the overall size and position of the clock on the tablet canvas). A complete list of the drawing features extracted from the DCTclock are listed in Table S3.

TABLE 3.

Ranked feature importance sets for DCRP, DCR, and DCT models in predicting age‐adjusted RAVLT scores.

Model	DCRP	DCR	DCT
Features	DCR score Correct Words Count Recall COM Component Placement COM Spatial Reasoning DCT score (0‐100) COP Log Stdev. Latency COM Longest Latency COP Information Processing COM Information Processing Gender COP Log Relative Long Latency Jitter PPQ5 F0 Mean Jitter COM Stroke Count Conformity Long Silent Speech Pause Count Jitter Absolute	DCR score Correct Words Count Recall COM Component Placement COM Spatial Reasoning DCT score (0‐100) COM Log Stdev. Latency COM Information Processing COM Average Latency COP Information Processing Gender COP Log Longest Latency COP Log Count Long Latencies COP Log Relative Long Latency COM Percent Think Time One‐Side COM Percent Think Time COP Spatial Reasoning COP Percent Think Time One‐Side COP Percent Ink Time	COM Component Placement COM Spatial Reasoning DCT score (0‐100) COM Log Stdev. Latency COM Information Processing COM Average Latency COP Information Processing Gender COP Log Longest Latency COP Log Count Long Latencies COP Log Relative Long Latency COP Percent Think Time One‐Side

Note: Italics indicate the speech timing and acoustic voice features found to contribute to the model predictive accuracy.

Abbreviations: Abs., absolute; COM, command clock; COP, copy clock; DCRP, DCR plus voice and speech; DCT, DCTclock™; RAVLT, Rey Auditory Verbal Learning Test.

2.3. Preprocessing and modeling

To ensure a balanced composition of impaired and unimpaired participants in the test set the dataset was split 80–20 (N = 754, N = 188) into training and test sets. To normalize the data, the training set was fit and transformed using a min‐max scaler for a range of 0–1. The test set was then transformed using the same scaler previously fit to the training set. To handle missing data via imputation, a “missForest” iterative imputation method was used, whereby a random forest regressor was employed to impute all missing values for each feature. This was done iteratively to support imputation accuracy. ⁴¹ As with normalization, imputation was fit to the training set, and then used to transform both the training and test sets.

Logistic regression with feature selection and regularization was used for each classifier model in the study. DCRP, DCR, and DCTclock models were subject to hyperparameter tuning, determined using the mean results of a 10‐fold cross‐validation (10 CV) grid search method on the training set. For smaller feature set models (i.e., Age, Gender, Age and Gender, RAVLT), an L2 regularization penalty (Ridge) was applied, whereas for larger feature set models (i.e., DCRP, DCR, DCTclock), an elastic net regularization penalty was used with varying L1 ratio values. A complete set of the hyperparameters chosen for each model are listed in Table S4. ¹⁷

2.4. Statistical feature selection

Features for DCRP (DCR plus speech and voice features), DCR, and DCTclock models were pruned first on the basis of relevance and redundancy, then further pruned using a 10 CV recursive feature elimination method on the training set. For all models, feature selection was conducted within the innermost loop of cross‐validation to ensure unbiased model selection.

First, features with below 0.15 Pearson correlation with the target were removed from the feature set. Passing features were then assessed for any high correlation across all features in the set; any features which had correlation above 0.99 with another feature of higher correlation with the target were subsequently removed as well. The remaining features were then assessed for their rate of missing values; any with a rate above 0.05 were then individually assessed for high correlation with features outside of the set, specifically those with a lower rate of missing values. If other features were discovered to have a correlation above 0.75 with the high missing value feature, said features were then reincorporated into the feature set. If no other features were found meeting these requirements, and the assessed feature had a rate of missing values in excess of 0.10, then this feature was removed from the final set.

The final set of features was then pruned through a process of recursive feature elimination on the training set, where features were individually dropped from the set if their inclusion did not demonstrably affect the model's area under the curve (AUC) score when predicting the target, that is, age‐adjusted RAVLT long delay score. ³³ , ⁴² DCRP, DCR, and DCTclock models had a subset of all features which were first composed using RAVLT long delay score as a target (Table 3.).

2.5. Missing data

While there were no missing values in the outcome variable, 58 individuals, particularly those with cognitive impairment, were not able to recall any of the words in the delayed recall task. Timing or acoustic values were not calculated in these cases; thus, values were imputed based on observed covariates using the missing forest algorithm in Python v3.9. ⁴¹ The acoustic voice timing features selected are robust to environment noise and recording equipment. ⁴³ , ⁴⁴

3. RESULTS

Results of logistic regression classifier models using missing forest imputation demonstrated that incorporating features across multiple modalities increased cognitive classification accuracy with each subsequent model. The model including the DCTclock graphomotor metrics, DCR verbal recall accuracy, and DCRP provided the most accurate means of classifying cognitive impairment. Age and gender models were also evaluated for reference and benchmarking purposes (Table 4).

TABLE 4.

Classifier performance for DCRP, DCR, DCT, and age + gender models in predicting memory impairment determined by age‐adjusted RAVLT scores.

Model	Acc.	AUC	Sen.	Spec.	PPV	NPV	Thresh.
DCRP	0.80 [0.75, 0.86]	0.83 [0.77, 0.89]	0.81 [0.73, 0.90]	0.80 [0.73, 0.87]	0.85 [0.79, 0.92]	0.75 [0.65, 0.84]	0.53
DCR	0.79 [0.74, 0.86]	0.82 [0.76, 0.88]	0.80 [0.71, 0.89]	0.79 [0.71, 0.87]	0.84 [0.77, 0.91]	0.74 [0.63, 0.83]	0.55
DCT	0.72 [0.65, 0.79]	0.77 [0.70, 0.84]	0.75 [0.65, 0.85]	0.70 [0.61, 0.78]	0.79 [0.71, 0.88]	0.65 [0.54, 0.75]	0.45
Age + Gender	0.62 [0.55, 0.69]	0.67 [0.60, 0.75]	0.75 [0.65, 0.84]	0.52 [0.43, 0.62]	0.74 [0.64, 0.83]	0.54 [0.44, 0.63]	0.40
Age	0.51 [0.44, 0.58]	0.59 [0.50, 0.67]	0.82 [0.73, 0.90]	0.28 [0.19, 0.36]	0.68 [0.54, 0.82]	0.46 [0.38, 0.54]	0.44
Gender	0.63 [0.57, 0.70]	0.63 [0.56, 0.70]	0.59 [0.47, 0.69]	0.67 [0.58, 0.76]	0.69 [0.60, 0.78]	0.57 [0.46, 0.68]	0.39

Abbreviations: ACC, accuracy; DCR, digital clock and recall; DCRP, DCR plus voice and speech; DCT, DCTclock™; NPV, negative predictive value; PPV, positive predictive value.

The speech timing and acoustic voice features were shown in the final model to be most strongly associated with cognitive impairment as determined by the RAVLT using logistic regression are displayed in Table 3. Long silent speech pause count was the most important speech timing feature in classifying cognitive impairment. Acoustic voice features related to jitter (peak‐to‐peak frequency variations) and average fundamental frequency (perceptually equivalent to vocal pitch) met the criteria for inclusion in the DCRP model for classifying cognitive impairment as defined by RAVLT long delay (Table 3).

Figure 2 details the results of logistic regression models used to predict RAVLT determined cognitive impairment (Table 4). AUC of the receiver operating characteristic (ROC) curve, classification accuracy (Acc), sensitivity (Sens), and specificity (Spec) were used to evaluate the model performance. Sens provides a percentage of impaired individuals correctly identified where Spec represents the percentage of individuals correctly identified as unimpaired. Finally, positive and negative predictive values (PPV and NPV) indicate the proportions of correct positive and negative memory impairment classification (Table 4).

FIGURE 2.

ROC curves for logistic regression models, evaluated binary classification of cognitive impairment as determined by the RAVLT long delay. AUC, area under the curve; DCR, digital clock and recall; DCRP, Digital Clock and Recall Plus Voice and Speech; DCT, digital clock test (DCTclock); RAVLT, Rey Auditory Verbal Learning Test; ROC, receiver operating characteristic.

Results indicate that features included in the DCRP speech and voice model provided the highest classification value (AUC = 0.83; Sens = 0.81, Spec = 0.80). This is an improvement over the DCR (AUC = 0.82; Sens = 0.80, Spec = 0.79), and DCTclock of only (AUC = 0.77; Sens = 0.75, Spec = 0.70) model feature sets. The DCRP outperforms all single domain and demographics models as predictors of RAVLT‐determined memory impairment (Table 4).

4. DISCUSSION

The novel multimodal approach incorporating analysis of graphomotor, verbal recall, speech, and voice features results in higher Sens and Spec than single modality and demographic statistical models in classifying cognitive impairment as determined by the RAVLT. This novel multimodal approach results in incrementally higher accuracy in classifying cognitive impairment. A comparison of classification accuracy between models based on demographics, DCTclock features alone (graphomotor), DCR metrics (DCTclock and verbal recall), and DCRP (DCR, speech timing, and acoustic analysis of verbal recall) revealed the highest Sens, Spec, PPV, and NPV values for DCRP models in impairment on the RAVLT long delay. Thus, the addition of several advanced speech and voice features extracted from a short verbal recall task to memory and global cognitive assessment should be considered in impairment classification models.

Acoustic features, largely language independent, can be altered in individuals with cognitive impairment and can be referenced longitudinally to show decline in performance. ⁴⁵ This study, however, is one of a very small number of studies to attempt to classify memory and cognitive impairment using a speech task of a maximum of only three words. The acoustic features are more commonly extracted from longer speech samples such as picture description or spontaneous speech tasks, the acoustic features identified in the short verbal recall task of the DCRP are consistent with those noted in other studies. ¹⁷ , ²⁵ , ³⁰ , ³⁶ Acoustic voice timing features included in this analysis have been shown to be reliable to changes in environment noise and recording equipment, making them ideal for in‐clinic and mobile screening (feature sets and mean (SD) values detailed in Table S5). ⁴³ , ⁴⁴ In particular, five speech timing and acoustic voice features extracted from the DCRP verbal recall carried substantial importance in predicting memory impairment on the RAVLT. The acoustic features that provided the most contribution to the multimodal cognitive impairment classification model were the average fundamental frequency, jitter, jitter Period Perturbation Quotient (PPQ5), and absolute‐valued jitter (Table 4).

Fundamental frequency, a correlate of voice pitch, has been shown in a number of studies to predict MCI and ADRD. ⁴⁶ , ⁴⁷ , ⁴⁸ These studies have investigated fundamental frequency (F0) and cycle‐to‐cycle characteristics of frequency revealing that individuals with cognitive impairment tend to have lower mean of F0 and higher cycle‐to‐cycle variations. From a real‐world perspective, these acoustic features result in lower sounding vocal pitch and a flat, monotonous affect, respectively. This research has shown consistently lower fundamental frequency (and range of frequencies) in individuals with varying levels of cognitive impairment. Clinically, this results in a flat affect, with fewer fluctuations in the tone of the voice.

Jitter quantifies the cycle to cycle (peak to peak) variations of speech frequencies. ⁴⁰ , ⁴⁴ Disordered jitter output is commonly seen in dysphonic voices where the vocal folds vibrate in an abnormal way (or are very breathy). Jitter PPQ5 and absolute‐valued jitter compare individual voice signal periods with the five speech cycles that surround them. Generally, higher jitter values correlate with rougher sounding, breathy voice quality that sounds less natural and more disordered than traditional voice. ⁴⁹ , ⁵⁰ Several researchers suggest that mean, absolute, and PPQ5 jitter tend to be higher in individuals with cognitive impairment. ²⁵ , ⁵¹ , ⁵² Perceptually, an increase in these jitter measures is associated with breathy, strained voices that lack stability. ⁵⁰ Increased vocal instability also affects the overall perception of voice quality.

The speech timing and acoustic voice features shown to be most associated with cognitive impairment in this study are consistent with previous literature related to predicting MCI and ADRD. ²⁵ Research indicates that the speech of individuals with cognitive impairment is slower, with more frequent silent pauses both long and short. ²³ , ²⁵ , ⁵³ , ⁵⁴ Most consistently, there is a higher percentage, count, length, and rate of silent pauses especially compared to speech content. The number of long pauses has evidence of being negatively impacted by cognitive impairment in a number of different speech contexts: reading, sentence repetition, picture description, and spontaneous speech. ²⁵ , ⁵³ , ⁵⁵ , ⁵⁶ The current research demonstrates increased number and length of silent pauses in a relatively short three‐word recall context for individuals with cognitive impairment.

The ROC curves in Figure 2 demonstrate the ability of each statistical model in classifying memory impairment as determined by the age‐adjusted RAVLT. ³³ , ⁴² These findings are of increasing clinical importance due to recent research indicating a wide range of Sens and Spec for commonly used tests like the MMSE, MiniCog, and MoCA, particularly in the early identification of MCI. ¹⁵ Arevalo‐Rodrigez et al. published a systematic review of the MMSE's utility in early detection of cognitive impairment in AD, stating: “Although the MMSE […] covers multiple cognitive domains at once, it is necessary to remark that this test was not developed to identify the early stages of dementia or even to predict the development of dementia in the long term. In our review, […] the descriptive data provided by the studies showed that neither the sensitivity nor specificity exceeded 80% at the same time.” ¹⁵ They then called for cautioned use of the commonly used cognitive assessments and recommended a more specialized cognitive screening tool.

Besides the statistical model based on demographics, the DCTclock only model demonstrated the lowest performance of the cognitive assessments included in this analysis. However, the DCTclock graphomotor features proved crucial in cognitive impairment classification both when recall and speech and voice analysis were included in the models (Table 4 and Figure 2). In particular, DCTclock latency values and spatial reasoning features are important when paired with recall accuracy and result in a more accurate cognitive impairment DCRP classification model.

At an average time to completion of 3 min, the DCRP is quicker to administer than most other commonly used cognitive assessments, can be easily integrated into electronic health records, and avoids many false positives that might otherwise be cumbersome to primary care and clinical workflows. Including voice and speech to enhance cognitive impairment analytics models, the DCRP has increased classification accuracy over graphomotor analysis during digital clock drawing and verbal recall accuracy alone, thereby increasing its value in a clinical setting. By removing the subjectivity in scoring and the necessity for manual data entry required by paper and pencil cognitive testing, the DCRP is a brief, scalable, multimodal, alternative cognitive screening option.

Limitations of this work include the cross‐sectional nature of the data collected. While research indicates that the acoustic features used in this study present good to excellent reliability in normal healthy individuals (ICC = 0.76 to 0.99), ⁵⁷ , ⁵⁸ there is evidence to suggest small but statistically significant variability of these features over time. ⁵⁹ , ⁶⁰ , ⁶¹ , ⁶² Future investigations should include a longitudinal analysis of speech timing and acoustic voice features from a number of speaking contexts in memory and cognitively impaired populations. Further longitudinal analysis should include validation of findings. Finally, the cohorting used to classify memory (RAVLT long delay) did so into cognitively impaired and unimpaired groups. Future research should be enhanced with inclusion of neuroimaging or fluid biomarkers of MCI or mild dementia due to AD such as beta‐amyloid and/or tau to understand the ability of the DCRP to classify individuals who exhibit both cognitive impairment and AD biomarker positivity.

The current research contributes to the prevailing speech profile of individuals with cognitive impairment, suggesting that it is associated with slower speech, with a particular effect on the duration, frequency, and percentage of pauses compared to speech. Novel contributions include increased classification accuracy of memory impairment using speech timing and acoustic features extracted from a limited number of words compared to traditional, longer duration speech tasks. Consistent with the recent consensus work group requirements for brief cognitive assessments, the DCRP can be completed under 5 min, available in multiple languages, with the capability of integrating with electronic health records. ¹⁷ The current study shows that the DCRP also meets the rigorous requirements of the consensus work group that clinically relevant brief cognitive assessments are recommended to show 80% sensitivity and specificity.

CONFLICT OF INTEREST STATEMENT

Alvaro Pascual‐Leone is a co‐founder and Chief Medical Officer of Linus Health and declares ownership of shares or share options in the company. Alvaro Pascual‐Leone serves as a paid member of the scientific advisory boards for Neuroelectrics, Magstim Inc., TetraNeuron, Skin2Neuron, MedRhythms, and Hearts Radiant. David Bates is a co‐founder and Chief Executive Officer of Linus Health, and declares ownership of shares or share options in the company. John Showalter is the Chief Product Officer of Linus Health and declares ownership of shares or share options in the company. All other authors are employees of Linus Health and declare ownership of shares or share options in the company. Author disclosures are available in the supporting information

CONSENT STATEMENT

Written informed consent was obtained from all participants, or in the case of cognitively impaired persons, from proxies.

Supporting information

Supporting Information

Supporting Information

Supporting Information

Supporting Information

Supporting Information

Supporting Information

Supporting Information

Supporting Information

Banks R, Higgins C, Greene BR, et al. Clinical classification of memory and cognitive impairment with multimodal digital biomarkers. Alzheimer's Dement. 2024;16:e12557. 10.1002/dad2.12557

DATA AVAILABILITY STATEMENT

The data that support the findings of this study were collected as part of the Bio‐Hermes study (ClinicalTrials.gov Identifier: NCT04733989) and are governed by the Global Alzheimer's Platform (GAP) consortium agreement. Data will be made available to the public via the Alzheimer's Disease Data Initiative (ADDI) Work bench in January 2025. All requests for data access should be made directly to GAP. The code used to calculate the reported results are available from Linus Health, Inc. upon reasonable request and with the permission of Linus Health, Inc. Usage restrictions apply to the availability of this code, which is not immediately publicly available.