From fingers to brain: virtual reality-based test capturing fine hand movements predicts cognitive function in older adults

Abstract

Background and Objectives

Early detection of mild cognitive impairment (MCI) is vital for managing cognitive decline in older adults. Hand movements are closely linked to cognitive function, prompting this study to develop a virtual reality (VR)-based wearable system to capture detailed hand movements. The main goal was to assess the system’s potential in predicting cognitive health and aiding MCI diagnosis.

Research Design and Methods

The study involved 607 participants aged 60–84 (mean age 67.41 ± 4.71 years). Each completed four VR tasks while wearing the system, which recorded fine hand movement data. Cognitive function was assessed using the Beijing version of the Montreal Cognitive Assessment (MoCA-BJ). Statistical analyses were conducted to correlate hand movement metrics with cognitive performance.

Results

Participants with cognitive impairments performed worse on VR-based fine motor tasks. Metrics from tests like the Pegboard, Block Placement—Flipping, and Tapping Tests were predictive of cognitive abilities. Indicators related to finer movements and non-dominant (left) hand use showed superior predictive power, achieving an AUC of 0.687 for predicting MCI, comparable to machine learning models such as Random Forest (0.762) and SVM (0.644).

Discussion and Implications

Hand movement data can provide valuable insights into cognitive function in older adults, highlighting the importance of fine motor skills in early MCI detection. This VR-based system could serve as a useful clinical tool for assessing cognitive health and supporting MCI diagnosis, enabling timely intervention strategies for cognitive decline.

Translational Significance:

This study shows the feasibility of a virtual reality (VR)-based system for early detection of mild cognitive impairment in older adults. Using affordable VR and precise hand movement tracking, it offers a new and scalable, accessible alternative to the traditional methods of cognitive assessment. The prototype proves the concept and aims to reduce reliance on specialized personnel and clinical spaces. Future developments include a portable, home-based version to ease healthcare access and enable large-scale monitoring. This approach can improve older adults’ quality of life and reduce the societal burden of age-related cognitive disorders through timely interventions and proactive management.

The rapid growth of the global aging population has led to a significant rise in the prevalence of age-related cognitive impairments. According to the World Health Organization (WHO), dementia affected an estimated 57.4 million people worldwide in 2019, and this figure is projected to rise to 152.8 million by 2050 (GBD 2019 Dementia Forecasting Collaborators, 2022). The economic and caregiving burden imposed by dementia is substantial, with the global cost estimated at $345 billion in 2023 alone. Cognitive decline is a gradual process, and there is a preclinical stage known as mild cognitive impairment (MCI) preceding the onset of dementia. MCI is characterized by noticeable declines in cognitive abilities, such as memory, attention, and executive function, which are more severe than typical age-related changes but do not interfere with daily life activities (Rafii & Aisen, 2023). A recent systematic review indicated that the prevalence of MCI ranges widely, from 18.3 to 21.1% (Song et al., 2023). Early detection and intervention for MCI are critical as they can delay the onset of more severe conditions and, in a broader sense, impact the health and quality of life of older adults.

Currently, the identification of early cognitive decline relies heavily on the detection of biomarkers through structural magnetic resonance imaging (MRI), blood tests, and cerebrospinal fluid analysis(Whelan et al., 2022). However, these methods have limitations, such as high costs and invasiveness, which hinder their widespread implementation. Traditional cognitive assessments, such as the Mini-Mental State Examination (MMSE) and the Montreal Cognitive Assessment (MoCA), although valuable (Siqueira et al., 2019), often fall short in detecting subtle changes in cognitive function, especially during the early stages of MCI. Additionally, these assessments rely on professional clinicians and are subject to scoring biases, which can limit their effectiveness (Jia et al., 2021). In recent years, researchers have turned to alternative methods that can provide more nuanced insights into cognitive health(Lee et al., 2024; Tanaka et al., 2024). One promising area of investigation is the examination of fine motor skills, specifically fine hand movements, as a marker of cognitive function (Zhang et al., 2024).

There is a well-established link between fine hand movements and brain function (Lundborg, 2014). Fine motor skills, which involve the coordinated movement of small muscle groups, are closely tied to the integrity of neural pathways in the brain, particularly those related to the prefrontal cortex, basal ganglia, and cerebellum (Dhawale et al., 2021; Fine & Hayden, 2021; Prati et al., 2024). Particularly, fine-tapping metrics of the non-dominant hand have also been found to correlate with the severity of medial temporal lobe atrophy in patients with Alzheimer’s disease (Sugioka et al., 2022). For children, the development of fine motor skills often serves as a marker of cognitive development (Capio et al., 2024). On the other hand, the loss of fine motor hand function is associated with age-related degeneration of gray and white matter in the brain (Holtrop et al., 2014; Hoogendam et al., 2014; Zapparoli et al., 2022).Notably, changes in these structures can manifest as alterations in the ability to perform precise hand movements, even before more overt cognitive symptoms become apparent(Hesseberg et al., 2020; Lariviere et al., 2019), leading researchers to advocate for integrating the assessment of hand movements with the identification of the prodromal stage of dementia in clinical practice (Ilardi et al., 2022).

Traditional hand function assessment tools are cumbersome and poorly integrated with cognitive function assessments; they depend on complex instructions or props and encounter challenges in quantifying function; observations and scoring based on two-dimensional planes are limited in their ability to comprehensively evaluate fine motor and are not well-suited for detailed output (Logue et al., 2021). To address these limitations, research utilizes advanced multi-source fusion of hand motion capture technology, achieving high precision in motion acquisition. Based on MEMS (Micro-Electro-Mechanical Systems) inertial sensors, motion capture technology employs wireless inertial sensors attached to the fingers and back of the hand to measure the rotation of each bone segment in real time (Cerveri et al., 2007). Utilizing forward and inverse kinematics, the system computes the motion parameters of the hand, facilitating continuous and precise capture of fine hand movements (Huang et al., 2023).

Furthermore, in head-mounted virtual reality (VR) devices, inside-out optical tracking technology achieves wrist position tracking by mounting light-emitting devices on the target wrist, capturing light signals, and converting these into spatial position data via specialized visual algorithms; VR simulation technology integrates the virtual and real worlds, enabling natural user interaction through diverse forms of feedback, sensing, and motion tracking (Buckingham, 2021). Indeed, there are numerous precedents for utilizing VR-related devices to assist with or conduct MCI assessments. For instance, Jang et al. (2022) developed a VR-based cognitive assessment program simulating daily life scenarios, demonstrating its effectiveness in classifying MCI as comparable to the traditional MoCA test. Furthermore, a meta-analysis by Liu et al. (2023) concluded that VR-based tests exhibit considerable detection performance for MCI, with high sensitivity and specificity. Therefore, by organically combining VR with fine hand movement capture technologies, a comprehensive and precise system for capturing and analyzing fine hand movements in virtual environments and facilitating cognitive screening can be created.

In addition to technical updates, it is noted that previous studies have frequently concentrated on a single aspect of fine motor function, which makes it difficult to fully comprehend the relationship between fine motor and cognitive functions (Ilardi et al., 2022). Comprehensive evaluations of fine motor movements, encompassing multiple dimensions like dexterity, coordination, and stability, could offer a deeper insight into their connection with cognitive performance (Zhang et al., 2024).

In response to this gap, the objective of this study was to develop a predictive tool for the early identification of cognitive impairment. This was achieved by proposing the use of Virtual Reality-based Micro-Electro-Mechanical Systems to capture fine motor characteristics and explore their correlation with cognitive functions. The integration of MEMS sensors with VR presents a non-invasive, cost-effective, and scalable solution for evaluating fine motor movements, which holds potential for application in large-scale screening programs in developing countries. Moreover, AI-driven automated diagnostics have become a significant trend (Kale et al., 2024). Fine hand movement data provide a wealth of specific features that are conducive to the automated classification and diagnosis of MCI. When combined with a database of real-world hand movement characteristics from older individuals, machine learning models and associated classification prototypes (e.g., support vector machine, SVM, and Random Forest model) hold promise for enhancing clinical applications. We hypothesized that performance on tasks demanding fine hand movements, particularly those involving the non-dominant hand, can serve as a sensitive indicator of cognitive status and predict the presence of MCI. By harnessing the precision and interactive capabilities of VR, our goal was to advance the development of more accurate and accessible methods for the early detection of cognitive decline. This advancement aimed to improve outcomes for at-risk individuals by enabling earlier interventions and personalized care plans.

Method

Participants

Participants were recruited from the community using convenience sampling methods. Recruitment efforts included advertisements in local senior centers, community newspapers, and online platforms targeting older adults in the Beijing metropolitan area. Potential participants were initially screened via telephone to assess basic eligibility based on age and self-reported medical history. Eligible individuals were then invited to Beijing Hospital for an in-person screening, which included a brief cognitive assessment (Beijing version of the Montreal Cognitive Assessment; MoCA-BJ) and a review of their medical records to confirm inclusion/exclusion criteria. Inclusion criteria included: (1) Age ≥ 60 years; (2) Ability to complete cognitive and hand fine motor assessments based on VR; (3) Voluntary participation with informed consent. Exclusion criteria included: (1) Suspected progression to dementia, indicated by Montreal Cognitive Assessment (MoCA-BJ) score ≤ 17; 2) Any diagnosed neuro-psychiatric diseases (including depression, anxiety, bipolar affective disorder, schizophrenia, and intellectual disabilities); (2) Neurological injuries affecting fine motor function (e.g., stroke, Parkinson’s disease); (3) Skeletal muscle injuries impacting fine motor function (e.g., tendon injuries, hand trauma); (4) Significant health events within the past six months (e.g., acute coronary events, severe infections, major surgery); (5) Participation in other interventional clinical trials.

Out of 637 individuals recruited, 607 were ultimately included in the study after excluding those who did not meet the inclusion criteria. The research protocol was approved by the Ethics Committee of Beijing Hospital (Approval No.: 2021BJYYEC-291-01).

MCI diagnostic

Cognitive function was assessed using MoCA-BJ. MoCA-BJ, a widely used screening tool covering domains such as memory, language, attention, abstract thinking, orientation, visuospatial structural skills, and executive functioning (Yu et al., 2012). A total score of 30 was possible. MCI was defined as a MoCA-BJ score less than 24, while a score of 24 or higher indicated healthy controls (HCs). The internal consistency reliability of MoCA-BJ in the current data, as measured by Cronbach’s alpha, was 0.83.

MCI diagnosis follows Petersen et al. (2014)’s criteria: (1) Self-reported memory decline or memory impairment reported by informants. (2) Overall cognitive function is normal. (3) Objective evidence of cognitive impairments, with cognitive function scores <1.50–2.00 standard deviations from age and education-adjusted mean, in our data (ie, 17 < MoCA-BJ < 24). (4) Normal activities of daily living. (6) Does not meet criteria for dementia diagnosis.

Measurements for daily motor function

Physical Self-Maintenance Skills

Physical Self-Maintenance Skills (PSMS) refers to the basic self-care activities necessary for daily living. This assessment evaluates six basic Activities of Daily Living (ADL): bathing, dressing, toileting, transferring, continence, and feeding. Each activity is scored as 1 (independent) or 0 (dependent). The total PSMS score ranges from 0 to 6, with a higher score indicating greater independence (Lawton & Brody, 1969).

Instrumental Activities of Daily Living

Instrumental Activities of Daily Living (IADL) refers to the more complex tasks that are essential for independent living, such as managing finances, preparing meals, doing housework, and using transportation. This assessment evaluates eight IADLs: using the telephone, shopping, food preparation, housekeeping, laundry, mode of transportation, responsibility for own medications, and handling finances. Each activity is scored as 1 (independent) or 0 (dependent). The total IADL score ranges from 0 to 8, with a higher score indicating greater independence (Hopkins et al., 2017).

Short Physical Performance Battery

The Short Physical Performance Battery (SPPB) assesses lower extremity function and includes three components (de Fátima Ribeiro Silva et al., 2021): (1) Standing Balance Tests: Evaluates standing balance at varying difficulty levels (side-by-side, semi-tandem, tandem), each held for a maximum of 10 seconds. Scoring ranges from 0 to 4 points based on hold time; inability to complete earns 0 points. (2) Walking Speed: Measures the time to walk 4 meters at a comfortable pace. Scoring ranges from 0 to 4 points based on time, with faster speeds receiving higher scores. (3) Chair Stand Test: Measures the time to stand up from a chair five times. Scoring ranges from 0 to 4 points based on time, with shorter times receiving higher scores. The total SPPB score is calculated by summing the scores from the three test sets.

VR-based fine motor movement assessment

The fine motor skills assessment system uses a high-fidelity VR environment powered by the HTC VIVE Focus 3 standalone VR headset. This headset offers a dual-eye resolution of 4896 × 2448, a 120-degree field of view, and adjustable interpupillary distance (57mm to 72mm) for a comfortable and immersive experience. Hand interactions are captured using Hi5 2.0 VR gloves, which feature wireless sensors with a gyroscope range of ±2000 dps and an accelerometer range of ±8g, providing precise motion tracking with a resolution of 0.02° and a frame rate of 500Hz. Wrist tracking is enhanced with HTC VIVE Wrist Trackers for accurate positional data. Fine motor movements are captured using the Perception Neuron^® system (Ma et al., 2022), which includes MEMS inertial sensors, a data router, and a computer terminal. Participants wear six inertial sensors per hand (five on the second knuckle of each finger and one on the back of the hand) to collect motion data, transmitted wirelessly via TDMA to avoid interference. The system is managed through dedicated software on a PC (CPU i7 or higher, 8GB RAM, 1920x1080 display), built with ­Microsoft Visual C++ 2015 and compatible with Windows 10 or later. This setup ensures spatial accuracy within 10mm, interactive software response latency ≤500ms, and data communication latency ≤1s, providing a responsive and accurate assessment and training experience. The tasks included in the assessment are:

3D Painting-Connect-the-Dots Test

The 3D Painting-Connect-the-Dots Test is designed to assess the participant’s ability to perform precise hand movements within a VR environment. In this test, participants are required to trace over target lines that are presented in a 3D space using their hand (no specific finger differentiation is necessary). The goal is to accurately connect the dots by tracing the lines as completely as possible. During the test, the following metrics are recorded: (1) Coverage: The percentage of the target line that has been successfully traced over by the participant. The target line is considered to be “traced” if the participant’s trajectory is within a defined threshold distance of the line. This threshold was set at 10 mm based on pilot testing to ensure accurate and reliable measurement. Coverage is then calculated as the percentage of sampling points along the target line that fall within this threshold distance of the participant’s trajectory. (2) Test Duration: The total time taken by the participant to complete the tracing of all target lines. Participants are instructed to start the test and proceed at their own pace, ensuring they maintain focus and accuracy throughout the task. The test is concluded once all the target lines have been traced, and the aforementioned metrics are used to evaluate the participant’s performance in terms of the precision of their hand movements.

Block Placement and Flipping Test (adapted from Minnesota Manual Dexterity Test)

The Minnesota Manual Dexterity Test is designed to assess an individual’s hand coordination and dexterity(Desrosiers et al., 1997). Test takers are required to complete a series of hand manipulation tasks within a specified time frame, such as moving blocks or arranging pins. In this version of the test, participants are required to perform specific hand manipulation tasks in a VR environment. The tasks involve flipping and placing 6*2 corks both front-to-back and side-to-side. These actions include single-handed flips and placements with both the left and right hands, as well as dual-handed operations that combine flipping and placement. For each task, the system records and outputs the following metrics: (1) Completion time (single-handed flipping, single-handed placing per hand, double-handed flipping and placing); (2) Detailed motion parameters, including completion times for each of the six hand sensors and the time intervals between sensor activations during the movement sequence

Pegboard Test (adapted from Purdue Pegboard Test)

The Purdue Pegboard Test is used to evaluate an individual’s hand coordination and finger dexterity (Irie et al., 2020). This test typically comprises several distinct sections, with the most common being “placement” and “assembly” tasks. The “placement” task requires participants to rapidly insert standard-sized pins into a board with holes arranged in a specific pattern, assessing finger dexterity, speed, and accuracy. The “assembly” task involves constructing a specific structure using pins and additional components, evaluating hand-eye coordination, finger dexterity, and problem-solving skills through more complex operations like assembly or disassembly. In this VR adaptation of the Test, participants wear motion capture gloves and are tasked with placing six sets of pins using both their left and right hands. The system records and outputs the completion time for both placement and assembly tasks, along with detailed motion parameters from 12 sensors (six on each hand). This includes the completion time for each pin placement and the interval times between sensor activations during the sequence of movements, providing a comprehensive analysis of both the speed and accuracy of the participant’s performance.

Tapping Test

The Tapping Test is used to assess an individual’s finger dexterity, speed, and motor control. In this test, participants are required to tap a button or device as quickly as possible using each finger. In the VR adaptation of the Tapping Test, participants wear motion capture gloves and perform single-finger tapping with all ten fingers, alternating between the left and right hands. Each finger taps the button six times. The system records the completion time for each finger and provides detailed motion parameters from 12 sensors, offering a comprehensive analysis of finger dexterity and motor control.

Tasks were optimized for VR by adapting them to a 3D environment for natural hand movements, providing real-time visual feedback, integrating motion capture for detailed hand function assessment, and adjusting parameters like target size and placement for optimal difficulty and sensitivity.

Participants were given instructions and 10 minutes to familiarize themselves with the assessment system before the evaluation began. Each participant had one attempt to complete the test, and the entire fine motor assessment took approximately 15 minutes. The presentation of the task scenario can be seen in Figure 1.

Figure 1.

Test scenario and subjective view.

The images in the four corners show screenshots of the four tasks being performed (top left: Pegboard Test; top right: Tapping Test; bottom left: 3D Painting-Connect-the-Dots Test; bottom right: Block Placement and Flipping Test) and the subjective visual perspective during the Block Placement and Flipping Test (center). For a more intuitive presentation, please refer to Supplementary material for the test videos.

Confounding variables

Potential confounding variables were selected based on previous studies and included demographics (age, sex, marital status, education, living situation, family income, occupation), health status and lifestyle factors (smoking, alcohol consumption, body mass index, grip strength), and chronic diseases (diabetes, hypertension, COPD, atrial fibrillation, chronic heart failure, coronary heart disease).

Statistical analyses

Data were analyzed using SPSS Statistics Version 27.0 for Mac. Continuous variables were described as mean ± standard deviation, and count data were described by frequency. T-tests were used to compare continuous variables between the HCs and MCI groups, and chi-squared tests were used for categorical variables. Univariate linear regressions were conducted to analyze the relationship between fine motor indexes and MoCA-BJ scores. Principal component analysis was used for dimensionality reduction of the fine hand movement data. To further assess the diagnostic performance of the identified fine motor test predictors for cognitive decline, receiver operating characteristic (ROC) curves were generated for individual or integrated metrics to calculate the area under the curve (AUC) for detecting mild cognitive decline (MoCA-BJ score < 24 vs. the rest). We validated Random Forest and SVM models to differentiate MCI from healthy controls using 58 refined hand movement features. Preprocessing involved removing missing values and standardizing features. For the Random Forest, we conducted a random search for hyperparameters (300 iterations) using 10-fold cross-validated AUC. For the SVM, we performed a random search for linear kernel hyperparameters (500 iterations). The best parameters were selected, and sklearn was used to evaluate parameter grids. The set with the highest cross-validation accuracy was chosen, and stratified 10-fold cross-validation was applied. AUC was calculated to plot the ROC curve, and feature importance was visualized.

Results

Participant characteristics

The characteristics of the study participants are presented in Table 1. The age range of the participants was from 60 to 84 years, with a mean age of 67.41 ± 4.71 years. The gender distribution in the sample is relatively balanced, with females comprising a slightly higher percentage at 58.2%.

Table 1.

Characteristics of participants.

Characteristics	All	Healthy Controls	MCI Group	p
	(N = 607)	(n = 347)	(n = 260)
Age, years	67.41 ± 4.71	67.24 ± 4.65	67.63 ± 4.80	.311
Gender (female), n (%)	353 (58.2)	213 (61.4)	140 (53.8)	.063
Marital Status (unmarried, widowed or divorced), n (%)	59 (9.7)	32 (9.2)	27 (10.4)	.589
Living Statue (alone), n (%)	47 (7.7)	28 (8.1)	19 (7.3)	.426
Education, n (%)				<.001
Lower	164 (27.0)	71 (20.5)	93 (35.8)
Secondary	266 (43.8)	157 (45.2)	109 (41.9)
High	177 (29.2)	119 (34.3)	58 (22.3)
Pre-retirement occupation type (mental labor), n (%)	385 (63.4)	238 (68.6)	147 (56.5)	<.001
High family income (≥10000RMB/m), n (%)	297 (48.9)	172 (49.6)	125 (45.4)	.031
Smoking, n (%)	76 (12.5)	36 (10.4)	40 (15.4)	.116
Drinking alcohol, n (%)	80 (13.2)	38 (10.9)	42 (16.2)	.145
BMI, kg/m²	24.44 ± 3.07	24.28 ± 2.86	24.65 ± 3.32	.137
Left-handedness, n (%)	21 (3.4)	10 (2.9)	11 (3.8)	.366
Grip strength, kg
Dominant hand	27.56 ± 8.60	27.85 ± 8.53	27.16 ± 8.70	.329
Non-dominant hand	26.66 ± 9.60	26.83 ± 0.50	26.42 ± 8.25	.601
Diabetes, n (%)	122 (20.1)	64 (18.4)	58 (22.3)	.142
Hypertension, n (%)	250 (41.2)	131 (37.8)	119 (45.8)	.066
Hyperlipidemia, n (%)	125 (21.0)	78 (22.8)	47 (18.5)	.116
Atrial fibrillation, n (%)	7 (1.2)	5 (1.4)	2 (0.8)	.359
Cardiovascular disease, n (%)	42 (6.9)	17 (4.9)	25 (9.6)	.019

Note. BMI = body mass index; MCI = mild cognitive impairment. ­Education = Lower: middle school or below; Secondary: high school; High: college or above. The p value in bold indicates a significant difference.

The average MoCA-BJ score was 24.85 ± 2.80, and 42.8% of the participants were classified as having MCI. It can be observed that there is no significant difference in age between the two groups, but the MCI group shows lower levels of education, fewer mental labor occupations, and typically comes from families with lower incomes. Additionally, the prevalence of cardiovascular diseases is higher in this group. Specific details and statistics can be found in Table 1.

Motor performance in the HCs and MCI

We conducted a comparison between the MCI group and the HCs in terms of daily function and general motor function, utilizing tests with PSMS, IADL, and SPPB. The inter-group comparison of specific indicators within these tests revealed that for the MCI group, the score of IADL was significantly lower than that of the healthy group(t₍₆₀₅₎ = 2.26, p = .024). In the general motor test, the score of the balance test demonstrated differences between the groups, with the MCI group scoring lower than the healthy group(t₍₆₀₅₎ = −3.15, p = .002). However, other indicators did not show significant disparities.

In VR-based fine hand motion measurement tasks, a significant number of indicators showed inter-group differences, with the MCI group performing worse compared to healthy participants. Specifically, out of a total of 36 indicators, 31 (86.1%) showed significant inter-group differences. The breakdown by task is as follows: 3D Painting-Connect-the-Dots Test (0/2), Block Placement and Flipping Test (7/7), Pegboard Test (5/6), and Tapping Test (19/21). Please refer to Table 2 for specific descriptive statistics.

Table 2.

General and fine motor performance in the HCs and MCI groups.

	Healthy Controls		MCI Group
Indicators	Mean	SD	Mean	SD	t	p
General motor function
ADL-PSMS	0.02	0.15	0.05	0.36	1.35	.179
ADL-IADL	0.20	0.52	0.34	0.96	2.26	.024
ADL total	0.22	0.56	0.39	1.04	2.54	.011
Balance (score)	3.98	0.19	3.91	0.32	−3.15	.002
Gait speed (score)	3.72	0.65	3.66	0.69	−1.13	.258
5-Times sit-to-stand (score)	3.64	0.75	3.62	0.81	−0.34	.735
SPPB total	11.27	1.41	12.74	24.89	1.10	.27
4-Meter walk time (s)	3.90	1.49	4.09	1.52	1.49	.137
5-Times sit-to-stand time (s)	9.80	2.72	9.96	2.97	0.69	.488
VR-based fine motor test
3D Painting-Connect-the-Dots Test
Coverage rate	0.73	0.24	0.71	0.24	−1.21	.227
Completion time	17.34	7.85	18.37	8.62	1.53	.128
Block placement and flipping test
Single-hand flipping completion time (left)	25.92	18.02	29.41	20.36	2.22	.027
Single-hand flipping completion time (right)	21.71	11.16	25.26	14.48	3.38	<.001
Single-hand flipping total time	48.60	22.22	54.69	24.95	3.15	.002
Placement test completion time (left)	16.85	9.58	19.80	14.10	3.06	.002
Placement test completion time (right)	14.74	6.36	16.62	9.20	2.97	.003
Placement test total time	32.32	12.03	36.81	16.69	3.83	<.001
Two-hand flipping completion time	39.13	16.79	43.05	18.34	2.66	.008
Pegboard test
Placement test completion time (left)	19.55	11.25	25.11	18.33	4.59	<.001
Placement test completion time (right)	19.16	10.00	19.96	9.80	0.98	.326
Placement total time	41.69	16.02	47.84	20.19	4.16	<.001
Assembly test interaction distance (left)	12.19	3.26	12.90	3.29	2.63	<.001
Assembly test interaction distance (right)	12.07	3.28	13.09	3.26	3.79	<.001
Assembly test completion time	65.19	23.81	75.46	24.99	3.79	<.001
Tapping test
Thumb variability rate (left)	0.75	0.24	0.78	0.26	1.43	.153
Thumb completion time (left)	12.53	6.71	13.94	7.30	2.46	.014
Index finger variability rate (left)	0.49	0.25	0.55	0.26	2.91	.004
Index finger completion time (left)	6.08	3.51	6.69	3.84	2.02	.043
Middle finger variability rate (left)	0.46	0.22	0.48	0.24	1.36	.174
Middle finger completion time (left)	5.35	3.08	5.71	3.11	1.40	.161
Ring finger variability rate (left)	0.44	0.22	0.51	0.25	3.45	<.001
Ring finger completion time (left)	5.46	3.86	6.34	5.06	2.41	.016
Pinkie variability rate (left)	0.45	0.22	0.49	0.23	2.63	.009
Pinkie completion time (left)	5.38	3.85	6.13	4.17	2.27	.023
Thumb variability rate (right)	0.56	0.25	0.62	0.26	2.73	.007
Thumb completion time (right)	6.90	3.93	7.56	4.35	1.95	.051
Index finger variability rate (right)	0.42	0.24	0.46	0.25	2.10	.037
Index finger completion time (right)	4.80	2.19	5.60	4.43	2.92	.004
Middle finger variability rate (right)	0.41	0.23	0.46	0.26	2.06	.04
Middle finger completion time (right)	4.80	2.36	5.52	3.97	2.75	.006
Ring finger variability rate (right)	0.45	0.23	0.50	0.26	2.20	.028
Ring finger completion time (right)	5.11	3.21	6.39	8.61	2.53	.012
Pinkie variability rate (right)	0.45	0.22	0.49	0.25	2.02	.043
Pinkie completion time (right)	5.28	3.28	5.93	4.73	2.00	.046
Tapping test total time	57.23	22.48	64.55	29.64	3.44	<.001

Note. ADL = activities of daily living; IADL = instrumental ADL; MCI = mild cognitive impairment; PSMS = Physical Self-Maintenance Skills; SD = standard deviation; SPPB = Short Physical Performance Battery; VR = virtual reality.

Regression analysis of fine motor skills as predictors of cognitive function

In simple linear regression analysis, we found that in the 3D Painting-Connect-the-Dots Test, the coverage was positively associated with MoCA-BJ scores (r = 0.087, p = .020), while the task completion time did not show a significant association. For the Block Placement and Flipping Test, the left and right single-hand placement times, flip completion times, and bilateral flip completion times independently predicted MoCA-BJ scores (r = −0.147 to −0.110, p_s < .007). The shorter the completion time, the higher the MoCA score, indicating stronger cognitive abilities. In the Pegboard task, the assembly test left-hand interaction distance, right-hand interaction distance, assembly test completion time, and left single-hand placement test completion time also independently negatively predicted MoCA-BJ scores (r = −0.243 to −0.152, p_s <.001); however, the right-hand placement test time did not reach statistical significance for predicting MoCA-BJ scores (r = −0.076, p = .063). In the tapping test, the completion times and variability rates of both hands using all ten fingers independently predicted MoCA-BJ scores (r = −0.163 to −0.091, p_s <.025). The detailed correlation matrix is provided in Figure 2. Additionally, we performed stepwise logistic regression to further identify the most significant indicators. The relevant analysis can be found in Supplementary Material.

Figure 2.

Correlation matrix of fine hand motion indicators and cognitive functions.

PCA of tasks and ROC analysis of individual and composite metrics

To better understand the predictive power of each individual task for MCI, we initially conducted Principal Component Analysis (PCA) on all metrics from each task. Based on the criterion that eigenvalues should be greater than one, we derived a composite wiring score from the 3D Wiring Task, explaining a total variance of 54.26%. From the Block Placement and Flipping Test, a single composite score was extracted, accounting for 37.01% of the variance. The Pegboard Test yielded two components: the Pegboard Assembly Test score, explaining 45.49% of the variance; and the Pegboard Placement Test score, explaining a total of 18.69% of the variance. Additionally, the Tapping Test provided two independent scores: the Left Hand Tapping score (31.96%) and the Right Hand Tapping score (30.09%).

We performed ROC analyses on these derived scores. The AUC for the 3D Wiring Task was 0.520, consistent with the aforementioned regression results, indicating it has the weakest predictive capability. The single component score from the Block Placement and Flipping Test had an AUC of 0.618. The AUC for the Pegboard Assembly score was 0.602, while the AUC for the Pegboard Placement score was 0.624, suggesting that the placement test has stronger predictive capabilities. The AUC for the Left Hand Tapping score was 0.613, and for the Right Hand Tapping score, it was 0.591, indicating that left-hand related measures have stronger predictive power.

By integrating the individual metrics from above regression Analysis—specifically, the Pegboard Placement (left hand) and the total completion time of the Pegboard Assembly Test, along with the variability measure of the left ring finger from the Tapping Test and the right completion metric from the same test—we developed a fine-grained composite predictive index. ROC analysis indicated an AUC of 0.687, which is significantly higher than the predictive performance of any single task or individual metric(p_s <.05). For detailed information, refer to Figure 3.

Figure 3.

ROC curve of individual and composite metrics.

The upper left part of the figure shows the ROC curve, and the table below lists the AUC, confidence intervals, and significance levels for each individual metric and the combined metrics. Composite indicators: regression score of Pegboard assembly completion time, the left-hand placement test time in the Pegboard task, the variability rate of the left ring finger in the tapping test, and the completion time of the right middle finger in the tapping test.

Machine learning enables data-driven classification

The optimal configuration identified through parameter optimization (with a maximum tree depth of 39) was used to train a Random Forest classifier. After evaluation using 10-fold cross-validation, the best AUC value achieved was 0.762 with a standard deviation of 0.042. For the SVM model, the best-performing model during 10-fold cross-validation reached an AUC of 0.644. The ROC curves and feature importance for both models are shown in Figure 4. The common top five features with the highest importance from Random Forest and SVM were: (1) completion time of the Pegboard Placement Test (Left), (2) the completion time of the Pegboard Assembly Test, (3) Pegboard—Assembly Test Interaction Distance, (4) Block Placement Test Completion Time (Left), and (5) Tapping test-Ring finger Variability Rate (Left). Our analyses suggest that a more concise test battery may be feasible. The regression analysis, PCA, and machine learning results consistently highlighted the predictive power of the Pegboard Test and the Tapping Test, with specific indicators such as the Pegboard assembly completion time, the left-hand placement test time, and the variability rate of the left ring finger showing strong associations with cognitive function.

Figure 4.

Visualization of the results from automated machine learning classification of fine hand movement complex parameters.

Panel (a) shows the feature importance distribution for the Random Forest, a classification visualization based on the top three feature weights, and the average ROC curve from validation. Panel (b) illustrates the feature importance distribution for the SVM model, a classification visualization based on the top three features along with the hyperplane, and the average ROC curve from validation.

Supplemental analysis considering handedness

Our initial findings suggested that left-hand-related indicators from the Placement and Tapping Tests are informative for predicting MCI, highlighting the value of non-dominant hand performance. Our sample was mostly right-handed, with only 21 (3.4%) left-handed participants. There was no significant difference in MCI and HC proportions between handedness groups (χ² = 0.81, p = .366). To explore handedness, we repeated our core analyses (difference tests, regression, PCA, and machine learning), excluding left-handed participants. The results were largely consistent with the full sample. In the right-handed sample, the logistic regression beta coefficient for the Left-hand Placement Test Completion Time increased slightly from 0.024 to 0.026, and the AUC for the Tapping Test Left-hand Score principal component increased marginally from 0.613 to 0.614, supporting the importance of non-­dominant hand indicators. The small number of left-handed participants limited a detailed investigation. However, a combined index predicted MCI with an AUC of 0.908 in the 21 left-handed participants (11 with MCI), and there was no clear difference in predictive power between left- and right-hand indicators in this group.

Discussion

Our study aimed to develop a VR-based fine hand motion assessment system, enabling the collection of detailed hand movement data as older adults perform four specific tasks. The goal was to determine if these fine hand movement data could predict cognitive function and contribute to the auxiliary diagnosis of MCI. Results show that hand movement metrics can generally predict cognitive abilities in older adults, particularly those involving fine motor skills of the fingers, such as the Block Placement and Flipping Test, Pegboard Test, and individual finger function tapping tests. Participants with cognitive impairments typically performed worse on these tasks. Completion times and other metrics from these tests positively predict neurocognitive deficits in older adults. We observed that the predictive power of different tasks and metrics varied. ­Metrics related to finer movements, finger dexterity, and non-dominant (left) hand function exhibited superior predictive efficacy. Additionally, the highest performing indicators in our composite logistic regression model achieved an Area Under the Curve (AUC) of 0.687 for predicting MCI. Data-driven feature selection and SVM machine learning classification provide a slightly superior classification accuracy, and the highly weighted features are consistent with those identified through manual selection. These findings suggest that it is possible to use hand movement data to gain insights into cognitive brain function in older adults, emphasizing the importance of fine motor actions and non-dominant hand performance.

A strong relationship exists between hand movements and brain function, rooted in neurological and motor systems. Research consistently links coordinated hand movements to cognitive processes, particularly attention, memory, and executive function (Capio et al., 2024; Donoghue et al., 2012; Ilardi et al., 2022). Hand motor skill deficits are common in individuals with cognitive impairments, potentially modulated by BDNF and dopamine pathways (Martins et al., 2024), making this association relevant for early MCI screening (Curreri et al., 2018). While previous non-VR studies, such as the review by Ilardi et al. (2022), have explored this link, our study offers a more detailed analysis of fine motor movements using VR and constructs a predictive model based on a large dataset. Additionally, aligning with Sugioka et al.’s (2022) findings on non-dominant hand movements in Alzheimer’s disease, we also identified several particularly noteworthy features within our rich feature matrix, such as finer movements and left hand advantages (which will be discussed in detail below).

On the other hand, in terms of the expansion of VR-based assessments, numerous VR tools have demonstrated high specificity and accuracy in the screening and diagnosis of MCI, primarily through the use of various simulated daily tasks or memory tasks (Liu et al., 2023). It is hypothesized that incorporating a broader range of more realistic cognitive tasks in VR environments could further enhance test performance (Jang et al., 2022). Here, our study offers a different and innovative direction: hand motor assessments alone can provide valuable insights into cognitive function and achieve comparable predictive accuracy. This approach, which focuses on the detailed analysis of hand motor skills, may complement and enhance existing VR-based methods, offering a novel and effective way to detect MCI.

It is noteworthy that the differences between gross motor tasks and fine motor tasks in predicting brain function have been highlighted, with evidence suggesting that baseline measurements of general motor abilities, such as speed and functionality of gross motor movements, are less sensitive in predicting MCI or distinguishing cognitive impairments (Donoghue et al., 2018). Conversely, metrics based on fine motor skills of the hands have shown significantly greater predictive power when compared directly with the other gross movement metrics (Liu et al., 2021). Our findings align with this body of research, indicating that while general motor measurements like the speed and functionality of gross motor actions do not provide a strong predictive value for MCI or cognitive impairment, most measures derived from fine motor skills of the hands do demonstrate predictive power. This suggests that the assessment of fine motor control may serve as a more sensitive marker for the early detection of cognitive decline.

The significance of fine and unusual small movements in cognitive prediction lies in their ability to reflect subtle changes in cognitive functions that may not be apparent through more generalized assessments. Fine motor tasks, which involve intricate and precise movements, can be more sensitive indicators of cognitive decline or impairment because they require high levels of coordination and often engage multiple cognitive processes, including attention, memory, and executive function (Williams & Werner, 1985). In our assessments, the complexity of tasks varies. The 3D drawing task, which requires only whole-hand movements, represents the coarsest task and thus has the least predictive power. On the other hand, finger tapping tests, which involve very small and precise movements, have a higher overall predictive capability. Similarly, the Pegboard Test, which demands high levels of finger dexterity, demonstrates better overall predictive efficacy.

Non-dominant hand function in particular can offer unique insights into cognitive status, as the less frequently used hand may not benefit from the same level of skill enhancement through daily use (Papadatou-Pastou, 2018), thus providing a purer reflection of cognitive aging effects. Interestingly, in simpler tasks such as tapping and placement, the non-dominant hand (typically the left hand for right-handed individuals) exhibits a better predictive value for cognitive function. This phenomenon is akin to the difference between crystallized and fluid abilities. While crystallized abilities improve with experience, fluid abilities do not, and both are interdependent in the aging process (Tucker-Drob et al., 2022). The dominant hand, through frequent daily use, accumulates a wealth of experience and skill refinement, which can mask the effects of cognitive aging. In contrast, the non-dominant hand is less influenced by routine labor and daily activities, thereby offering a clearer indication of cognitive function changes due to aging (Sebastjan et al., 2017). It’s worth noting that the non-dominant hand demonstrates advantages primarily in relatively simple tasks. For more complex tasks, the performance of the dominant and non-dominant hands shows little difference in predictive efficacy, or the dominant hand may even outperform the non-dominant hand. This suggests that task difficulty influences the impact of handedness, which is consistent with the environment-experience theory (Birkett, 1987). In any way, these findings underscore the importance of considering handiness in movement-cognitive assessments, as the non-dominant hand can provide a more direct measure of cognitive health, free from the confounding effects of habitual use.

The integration of multiple effective measures into a composite index offers several advantages over relying solely on individual metrics or single-task assessments. Our research indicates that a composite measure combining two efficacious tasks—the Pegboard Test and the Tapping Test, specifically focusing on the left-hand placement and the ring finger tapping—demonstrates superior predictive power compared to isolated task-­specific metrics. This enhanced predictive capability arises from the fact that a composite index captures a broader spectrum of cognitive and motor functions, thereby providing a more holistic view of the individual’s cognitive status (Reischies & Hellweg, 2000). By aggregating the strengths of distinct tasks, a composite ­indicator reduces the noise inherent in individual assessments and amplifies the signal of underlying cognitive decline (Barclay et al., 2019). This approach leverages the complementary nature of different motor skills, allowing for a more robust and reliable evaluation of cognitive health (Ilardi et al., 2022). The Pegboard Test, which evaluates fine motor dexterity and coordination, combined with the Tapping Test, which assesses rapid repetitive movements, creates a synergistic effect that enhances the sensitivity of the assessment to subtle changes in cognitive function.

In our SVM data-driven model, different tasks’ features are assigned varying weights, which indirectly underscores the importance of having rich task data. Despite the marginal advantage of data-driven automated methods over manually theory-screened features, this observation suggests the potential for using interpretable test indicators—such as those grounded in theoretical foundations—to more effectively guide clinical practice. Of course, current early diagnosis of MCI emphasizes the benefits of machine assistance and automation (Kale et al., 2024). Whether through formulaic algorithms or black-box classification models, these approaches can achieve clinical significance. In clinical settings, the use of an automated multi-feature classification system can streamline diagnostic procedures by reducing the need for multiple separate assessments, thereby saving time and resources and contributing to more reliable conclusions about the progression and treatment of cognitive conditions.

Nevertheless, critical consideration in the development of cognitive assessment tools is balancing data quantity, predictive accuracy, and patient experience. While more tasks and data can improve the quantification of the relationship between hand movements and brain function, they can also increase test duration and cognitive load, potentially affecting patient compliance and data quality (Wang et al., 2020). Our findings suggest that a concise test battery focusing on the Pegboard Test and Tapping Test may be sufficient for accurate MCI prediction. Future research should explore the optimal combination of tasks and indicators to maximize predictive power while minimizing patient burden, which could also include individualized adaptive testing strategies (Gibbons et al., 2016).

In our design, the advent of multi-sensor technology has made the collection of data on fine motor skills increasingly feasible. Wearable sensors capable of capturing subtle movements can provide continuous and precise measurements of hand dexterity and coordination, which are critical for understanding the interplay between motor and cognitive functions (Huang et al., 2023). These technological advancements support the notion that hand–brain functional coordination can be objectively measured and potentially used as a biomarker for cognitive health. Our data support the concept of hand–brain functional coordination, providing empirical evidence that hand motor function can be utilized to predict brain function, thus paving the way for practical applications in the field of cognitive health assessment.

It is worth noting that VR environments provide a unique platform for hand motion capture assessments by leveraging their immersive qualities and the ability to simulate real-world scenarios within a controlled digital space (Tortora et al., 2024). However, VR introduces an additional layer of complexity compared to traditional methods, particularly regarding depth perception (Vienne et al., 2020). In VR, the perception of depth can differ from that in the physical world, which means users must undergo an extra learning process to adapt to the virtual environment (Yi et al., 2023). Beyond depth perception challenges, the limited field of view (FOV) in VR headsets can restrict peripheral vision and affect spatial awareness, which could be mitigated by using headsets with wider FOVs (Sanchez-Garcia et al., 2020). The lack of tactile, sound, and force feedback can significantly impact the interactive experience, reduce task realism, and affect performance, especially in fine hand manipulation tasks (Shi & Shen, 2024). To minimize these limitations, proper calibration and training are essential. Participants should have sufficient time to familiarize themselves with the VR environment and tasks before data collection. Individual differences in adapting to VR should also be considered when interpreting results.

Interestingly, the adaptation challenges inherent in this novel modality also offer an opportunity to enhance or adjust the assessment’s difficulty, potentially revealing differences in cognitive abilities. For older participants, the need to learn and adapt to the nuances of VR can be both a challenge and a form of cognitive stimulation (Micarelli et al., 2019). The additional effort required to navigate and interact within a virtual space could highlight variations in cognitive adaptation capacities among individuals. While this may increase the complexity of the assessment, it can also provide insights into how well a person can adapt to new situations and cope with unfamiliar stimuli (Yarossi et al., 2021), which are crucial aspects of cognitive flexibility and resilience. It can be assumed that the introduction of depth perception discrepancies and the associated learning curve inherent in VR can serve as a tool to differentiate between varying levels of cognitive agility and adaptability. Therefore, VR-based assessments can offer a more nuanced understanding of an individual’s cognitive profile, including their capacity to manage and integrate new sensory inputs. It’s worth noting that VR-based cognitive training has become a widely studied important intervention for improving dementia, with one of its mechanisms involving extensive new visual inputs and adaptive learning (Kokorelias et al., 2024).

This VR-based system offers a dual benefit: early detection and targeted intervention, representing an innovative fusion of technology and healthcare. Integrating it into routine geriatric assessments allows clinicians to identify individuals at risk for MCI based on their fine motor task performance. This early identification enables personalized intervention strategies, such as engaging VR-based hand exercise programs designed to enhance fine motor skills and cognitive function. This approach holds significant potential for proactive cognitive healthcare in older adults, as finger movement exercises have been demonstrated to contribute to cognitive enhancement in MCI individuals (Wang et al., 2022) or become important protective factors against the progression of dementia (Chen & Kim, 2024). The current system’s virtual environment provides a cost-effective training space, and its objective data facilitates continuous monitoring and adaptation of the intervention plan, maximizing its effectiveness in slowing cognitive decline and improving the quality of life for individuals with MCI. Furthermore, the system’s components have the potential to be developed into more portable and accessible products, reducing the clinical distance for older adults, streamlining geriatric cognitive management, and ultimately promoting widespread adoption and impact.

Despite the significant potential of our technology—­combining VR and wearable motion capture systems—in predicting cognitive function and aiding in the diagnosis of MCI in older adults, several limitations warrant acknowledgment. First, the sample used in our research may not fully represent the broader population, as it primarily consisted of a specific demographic (older individuals with higher education levels living in first-tier cities), which limits the generalizability of our findings. While our sample size of 607 participants is relatively large, it is still a finite number and may not fully capture the heterogeneity of the older adult population.

For instance, while our results suggest that non-dominant hand (left hand for right-handed individuals) performance may be particularly valuable in predicting cognitive decline, our limited data on left-handed participants prevented a systematic investigation of the relationship between handedness, fine motor skills, and cognitive health. Future research should explore this issue further with a more balanced sample of right- and left-handed individuals. Moreover, the current study did not account for long-term adherence to the assessment protocols, which is crucial for longitudinal studies aiming to track cognitive decline over time. Future research should focus on strategies to maintain participant engagement and ensure sustained compliance with the testing regimen.

In terms of the limitations of the application, the possibility of VR-induced motion sickness could significantly limit the system’s applicability. VR can cause motion sickness in some users due to the conflict between visual and vestibular (inner ear) sensory information (Simon-Vicente et al., 2024). Although we took several steps to minimize the impact, including giving participants ample time to familiarize themselves with the VR environment and tasks, designing tasks with simple and controlled hand movements, encouraging breaks as needed, and closely monitoring for signs of discomfort. Future research should explicitly assess motion sickness using validated questionnaires to quantify its prevalence and severity (Kim et al., 2018). Additionally, future studies could explore strategies to mitigate motion sickness, such as using anti-motion sickness techniques (e.g., providing a stable visual reference point) or adapting the VR environment to minimize visual-vestibular conflict.

Currently, this study only provides a conceptual prototype, and validation research is still ongoing in a laboratory setting, which incurs certain costs. Further refinement of these technologies to make them more intuitive and user-friendly is necessary to reduce the burden on both participants and healthcare providers. Ultimately, at-home measurement and data tracking by patients could reduce healthcare costs and improve accuracy.

In conclusion, our study has developed and validated an innovative approach using advanced VR and fine hand motion capture technologies. By creating a composite index that integrates the most effective elements from various tasks of the Pegboard and Tapping Tests, while also taking into account the performance of the non-dominant hand, we offer a powerful tool for the detection of cognitive decline in older adults. This integrated method not only enhances the accuracy and reliability of cognitive assessments but also provides a more engaging and less stressful experience for older participants, potentially leading to earlier and more precise identification of cognitive impairments.

Supplementary material

Supplementary material is available at Innovation in Aging online (https://academic.oup.com/innovateage).

Data availability

Upon approval, researchers may request access to the data and software used in this study by contacting the corresponding author. We aim to provide the necessary resources to support further research. The study was preregistered internally at Beijing Hospital with the project number Z221100003522015.

Funding

This research project is supported by Interdisciplinary Research Program on Frontiers of Medical Health, Chinese Academy of Medical Sciences (2023-I2M-QJ-020); National High Level Hospital Clinical Research Funding (BJ-2023-074); National High Level Hospital Clinical Research Funding (BJ-2024-196); Beijing Municipal Science & Technology Commission “AI + Health Collaborative Innovation Cultivation” Project (Z221100003522015); Non-profit Central Research Institute Fund of Chinese Academy of Medical Sciences (2021-JKCS-024).

Conflict of interest

We declare that there are no conflicts of interest in relation to this research project.