Simultaneous Dataset of Brain, Eye and Hand during Visuomotor Tasks

Abstract

Visuomotor integration is a complex skill set encompassing many fundamental abilities, such as visual search, attention monitoring, and motor control. To explore the dynamic interplay between visual inputs and motor outputs, it is necessary to simultaneously record multiple brain activities with high temporal and spatial resolution, as well as to record implicit and explicit behaviors. However, there is a lack of public datasets that provide simultaneous multiple modalities during a visual-motor task. Functional near-infrared spectroscopy and electroencephalography to record brain activity simultaneously facilitate more precise capture of the complex visuomotor of brain mechanisms. Additionally, by employing a combined eye movement and manual response, it is possible to fully evaluate the effects of visuomotor outputs from implicit and explicit dimensions. We recorded whole-brain EEG (34 electrodes) and fNIRS (44 channels) covering the frontal and parietal cortex along with eye movements, behavior sampling, and operant behavior. The dataset underwent rigorous synchronization, quality control to highlight the effectiveness of our experiments and to demonstrate the high quality of our multimodal data framework.

Background & Summary

Visuomotor processing and coordination are as critical for the successful operation of a task as noticing and performing planned actions in the appropriate order. For example, seeing a falling volleyball and coordinating your hand to hit it. This complex process requires visual spatial attention, working memory, belief formation, motor control, and more. However, in traditional research, the constraints of simplified experiments and single-modal data collection not only fail to fully capture the highly complex and variable visuomotor processes but also limit the ecological validity of the conclusions. This problem is being addressed with new approaches driven by advances in AI-driven modeling, human-computer interaction, and complex systems. Ideally, by collecting brain-eye-hand data or other physiological data under natural behavior, multimodal datasets can enhance the robustness of machine learning or artificial intelligence-powered models against noise and data corruption. We can extract patterns of functional interactions to advance our understanding of the mechanism of the visuomotor system in complex cognitive tasks.

For brain activity in the visuomotor research field, the main challenge is the lack of open-access EEG-fNIRS data and protocols. Electroencephalography (EEG) often employed in attention-related studies, captures neural activity with millisecond-range temporal precision. Functional near-infrared spectroscopy (fNIRS) shows greater resilience than EEG against motion and noise artifacts. The emerging EEG-NIRS hybrid can enhance the understanding of brain activity by merging the spatial strengths of fNIRS and the temporal strengths of EEG. However, public datasets that were simultaneously recorded using EEG and fNIRS only available for the N-back task¹, Stroop tasks², and visual working memory³.

Monitoring eye movements plays a crucial role in visuomotor research, serving as a vital link between visual perception and motor control⁴. This process allows researchers to understand how implicit behavior guides explicit motor responses. Compared with implicit behavior, manual response is linked to explicit processing, which is in freely moving scenarios. The natural movements enable further inferences from movement trajectories, accuracy, belief formation, and reaction time⁵. The movement trajectories of joysticks contain rich spatiotemporal information about the user’s actions and provide unique insights into how cognitive processes unfold over time^6,7. Moreover, electrocardiograms (EKGs) play a unique and complementary role during visuomotor tasks, primarily by offering insights into autonomic nervous system responses, which can be used to measure physiological stress, emotional arousal, and engagement levels⁸. Using EEG in combination with EKG and fNIRS, Ahn et al.⁹ investigated mental fatigue in drivers.

Advances in the assessment of visuomotor processes are evident, especially in concurrent EEG-fNIRS recording and in the coupling of implicit and explicit behavior such as eye movement, EKG, and manual performance. Exact synchronization and rigorous quality control across all data modalities are challenging, as a multimodal research approach requires unifying various methodologies specific to corresponding processes. To date, there is a lack of public datasets that integrate brain, heart, eye and hand data to simultaneously record activity for studying visuomotor systems.

We adopted a visuomotor paradigm to detect attention and motor processes simultaneously with the use of multiple modalities (Fig. 1). The stimuli of studies were utilized in a target aircraft identification scenario to increase the level of ecological validity. We collected EEG, fNIRS, EKG, eye movement, and manual response from 50 healthy participants. Then, we extracted the prominent features of each modality to confirm the quality of the data, such as event-related potentials (ERP), EEG topography; oxygenated hemoglobin (HbO) concentration, deoxygenated hemoglobin (HbR) concentration, total hemoglobin (HbT) concentration and spatial activation (fNIRS); fixations and saccades (eye movement); heart rate (EKG); and operating precision and temporal structure (joystick). Compared with prior datasets, this analysis can provide data to develop algorithms to explore neurophysiological correlates of visual motor processes and further establish a more comprehensive theory and model for predicting an individual’s performance.

Fig. 1.

Overview of the apparatus, data collection and potential analysis.

Methods

Participants

Fifty healthy participants (32 males and 18 females, 48 right-handed and 2 left-handed, average age 23.6 ± 2.8 years) without any neurological, psychiatric, or brain-related disorders participated in this study. Please note that the data is unbalanced. Despite random recruitment, there are more male than female participants, and the proportion of left-handed individuals is low. Further demographic details are available in Table 1. This study was approved by the Beijing Normal University Institutional Review Board (approval number: IRB_B_0016_2022002). All participants were informed about the experimental procedures, provided informed consent before the experiment, and were financially compensated afterward.

Table 1.

Demographic data including age, gender, and dominant hand.

Subject	Gender	Age	Handness	Subject	Gender	Age	Handness
1	Female	24	Right	26	Female	32	Right
2	Female	21	Right	27	Male	22	Right
3	Female	19	Right	28	Male	20	Right
4	Female	20	Right	29	Female	24	Right
5	Male	24	Right	30	Female	19	Right
6	Male	24	Right	31	Male	20	Right
7	Male	23	Right	32	Male	24	Right
8	Female	25	Right	33	Male	26	Right
9	Male	24	Right	34	Male	25	Right
10	Female	23	Right	35	Female	24	Right
11	Male	23	Right	36	Male	25	Lift
12	Female	29	Right	37	Male	23	Right
13	Female	26	Right	38	Male	24	Right
14	Female	24	Right	39	Female	21	Right
15	Female	21	Right	40	Female	25	Left
16	Female	25	Right	41	Female	22	Right
17	Female	23	Right	42	Female	26	Right
18	Female	23	Right	43	Male	24	Right
19	Female	20	Right	44	Male	22	Right
20	Female	20	Right	45	Female	26	Right
21	Female	23	Right	46	Female	22	Right
22	Female	25	Right	47	Female	21	Right
23	Male	24	Right	48	Female	30	Right
24	Female	23	Right	49	Male	29	Right
25	Female	27	Right	50	Female	23	Right

The average age at the end of this table is presented as mean ± standard deviation.

Procedures

The stimuli were displayed on a 21.5-inch LCD monitor with a resolution of 1920 × 1080 pixels at a refresh rate of 60 Hz using PsychToolbox software (http://psychtoolbox.org/) in the MATLAB environment (The MathWorks Inc., Natick, MA), and the viewing distance was 63 cm. The experiment consisted of 6 blocks, each preceded by a 120 s rest. The instructions were displayed on the screen to signal the start of the blocks. Feedback on the mean performance was provided after each block. Each block began with an easy task, where the target remained in a fixed position in the block (Fig. 2a, left), followed by a hard task with random target locations (Fig. 2a, right). There were 43 trials in the easy task and 129 trials in the hard task, resulting in a total of 258 trials and 774 trials, respectively.

Fig. 2.

Experimental procedures. (a) Simple task (left) and search task (right) in the visuomotor paradigm. (b) Six possible positions of targets. (c) Motor response to the joystick. (d) Behavior sampling of attention motor duration disturbed between 2.2–5.2 s sampled by 10 Hz.

To perform the easy and hard tasks, the participants had to complete three stages: visual search, motor response and attention monitoring. For the visual search, the participants were instructed to fixate their gaze on the central cross. The trial started with a visual search display for 0.2 s. The visual search display consisted of a yellow plane (target) and 5 green planes (distractor). The numbers labeled “Tag” refer to the target positions rather than the trial numbers. Target 1 to Target 6 corresponds to angles of 30°, 90°, 150°, 210°, 270°, and 330° from the 12 o’clock position, each at a visual angle of 5° (Fig. 2b). For motor response, the participants were instructed to steer the joystick from its central position toward the target position by using the right hand (Fig. 2c). As soon as they confirmed the direction and location of the target, they pressed the trigger on the joystick within 2 s and then released the joystick to recover it to its initial state and waited for the next trial. Reaction time was defined as the interval from the onset of the stimulus until the joystick moved beyond a 20% radius of its neutral position, while movement time referred to the period of movement until the trigger was pressed. The response time was the total reaction time and movement time. The stimulus onset asynchrony (SOA) between the current target and the next target was modified from the task used by Fiebelkorn et al.¹⁰, which is called an attention sampling design. Specifically, the variable SOA denoted attention monitoring duration ranging from 2.3 to 4.7 s in steps of 0.1 s corresponding to a 10 Hz sampling frequency and balanced randomly across trials to prevent anticipatory responses (Fig. 2d).

Synchronous recording

All equipment was used to synchronously collect the data in the entire experiment. In the online recording, the fNIRS signals were acquired at 10 Hz, the EEG signals were acquired at 500 Hz, the joystick signals were acquired at 50000 Hz, and the eye-tracking signals were acquired at 1000 Hz. Throughout the entire experiment, upon the presentation of task cues, synchronized trigger signals were generated by PsychToolbox-3 within MATLAB. These trigger signals were then simultaneously transmitted to the EEG/EKG, fNIRS, eye-tracking, and joystick devices via a parallel port (TTL), serial port (RS-232), ethernet port (TCP/IP), and USB 2.0 port, respectively.

Behavioral recording and preprocessing

A TCA joystick (Airbus Edition, Guillemot Corporation S.A., France) was used to record movement trajectories. The joystick handle could move nearly freely and was equipped with 3D (Hall effect) magnetic sensors. The joystick had a resolution of more than 268 million values over the X and Y axes (65,536 × 65,536). A custom function (joystick2beh.m) was implemented in the PsychToolbox program in the MATLAB environment to record each participant’s joystick position and response time. A response time of less than 0.2 s or more than 2.2 s was considered an operation timeout. If the joystick moved away from the center position by less than 500 pixels, the operation was considered invalid. Deviation angles were calculated relative to the participant’s mean across six directions, with threshold angles for general hits and accurate hits set at ±10° and ±0.5°, respectively.

EEG recording and preprocessing

The EEG data were recorded using a SynAmps2 amplifier (Compumedics Neuroscan Inc., USA) with a data acquisition package (Curry 8.0, Compumedics Ltd., Australia). As shown in Fig. 3, EEG signal acquisition utilized a standard international 10–20 system EEG cap comprising 32 Ag/AgCl electrodes and one reference electrode (A1). To ensure optimal electrode-to-scalp contact, impedance was maintained below 5 kΩ, minimizing noise interference. The signals were digitized at a sampling rate of 500 Hz. The raw *.cdt files were imported into EEGLAB¹¹ v2023.1 using the loadcurry v3.2.3 plugin. Due to artifacts caused by the trigger channel, manual removal was performed to ensure data integrity. Bad channels were detected and corrected via interpolation. We applied the EEGLAB function pop_eegfiltnew(), using a Hamming windowed sinc FIR bandpass filter (0.1–30 Hz, zero-phase, non-causal) with a 0.1 Hz transition bandwidth. Independent component analysis (ICA) was performed using ICLabel¹² v1.6 to identify and reject artifact components (up to 4 components) based on thresholds of [0.95 1] and [0.85 1] for muscle and eye movement artifacts, respectively.

Fig. 3.

Spatial layout of EEG-fNIRS measurements. (a) 2D distribution of electrodes, light sources (pink circles), detectors (blue circles), and channels (red lines). (b) Mapping of channels onto a 3D model based on brain MNI space.

fNIRS recording and preprocessing

In this study, we employed a continuous wave instrument (ETG-4000, Hitachi Medical Corp., Japan) to measure relative changes in the tissue concentrations of HbO and HbR at a sampling rate of 10 Hz. A total of 16 light sources with wavelengths of 695 nm and 830 nm, along with 14 detectors, constituted 44 channels. The sources and detectors were alternately positioned in space using two-piece flexible rubber holders, maintaining an interoptode distance of 30 mm. The light sources and detectors of the fNIRS system were fixed on two 3 × 5 probe arrays covering the frontal and parietal cortex (Fig. 3a). For the coregistration of fNIRS channels on participants’ cortical regions, all measurement channels were labeled with vitamin E capsules, which appear as bright spheres on structural magnetic resonance imaging (MRI). Then, the cortical regions beneath the different channels (Fig. 3b) were determined based on the coregistration of fNIRS channels to the Montreal Neurological Institute (MNI) brain space and anatomical label (see Table 2). Before each experimental session, the channels were systematically inspected, and signal quality was verified. The original *.csv file was converted to *.nirs format, followed by the examination of abnormal light intensity data, which included negative values, infinite values, invalid data, or standard deviations exceeding relatively lenient thresholds (0.05, 1). Interpolation using neighboring channels was attempted for data repair. Using the Homer3¹³ v1.80.2 toolbox, fNIRS data were transformed into HbO, HbR, and HbT concentrations. A bandpass filter of 0.01–0.2 Hz was applied, followed by the correlation-based signal improvement (CBSI)¹⁴ method to correct head motion.

Table 2.

MNI coordinates of fNIRS Channel.

Channel	Anatomical label, Percentage of Overlap	MNI
		X	Y	Z
1	Precentral_R, 0.63229	60	2	35
2	Frontal_Sup_R, 0.66791	28	7	67
3	Supp_Motor_Area_R, 0.69481	−29	7	67
4	Frontal_Sup_L, 0.64207	−59	3	35
5	Precentral_R, 0.87029	61	13	18
6	Frontal_Mid_R, 0.904	39	19	54
7	Frontal_Sup_R, 0.70242	0	22	69
8	Supp_Motor_Area_R, 0.9758	−38	19	54
9	Frontal_Mid_L, 0.73208	−60	13	18
10	Frontal_Mid_R, 0.98745	53	28	27
11	Frontal_Sup_R, 0.5754	26	34	56
12	Frontal_Mid_L, 0.6614	−24	34	55
13	Frontal_Sup_L, 0.63386	−51	28	28
14	Frontal_Inf_Tri_R, 0.52899	53	36	11
15	Frontal_Mid_R, 1	33	44	40
16	Frontal_Sup_R, 0.836	0	47	53
17	Frontal_Sup_L, 0.71461	−32	44	41
18	Frontal_Mid_L, 0.78539	−51	37	13
19	Frontal_Mid_R, 0.94372	42	49	17
20	Frontal_Sup_R, 0.6758	21	55	37
21	Frontal_Mid_L, 0.8692	−19	56	37
22	Frontal_Sup_L, 0.81308	−40	50	17
23	Parietal_Inf_L, 0.82787	−56	−51	41
24	Parietal_Inf_L, 0.46502	−27	−51	70
25	Parietal_Sup_R, 0.535	27	−51	70
26	Parietal_Inf_R, 0.56934	55	−51	40
27	SupraMarginal_L, 0.90698	−55	−62	26
28	Parietal_Inf_L, 0.80488	−35	−64	57
29	Parietal_Sup_L, 1	−1	−64	68
30	Parietal_Sup_R, 1	34	−63	57
31	Parietal_Inf_R, 0.71774	53	−63	26
32	Angular_L, 0.75972	−44	−73	33
33	Angular_L, 0.67111	−21	−74	53
34	Angular_R, 0.68161	20	−74	54
35	Angular_R, 0.82143	42	−73	33
36	Temporal_Mid_L, 0.78451	−43	−80	14
37	Angular_L, 0.67323	−24	−82	38
38	Parietal_Sup_L, 0.3252	−1	−82	44
39	Occipital_Sup_R, 0.41365	22	−82	38
40	Angular_R, 0.53629	41	−80	14
41	Occipital_Mid_L, 0.98819	−30	−87	18
42	Occipital_Mid_L, 0.81522	−12	−88	30
43	Occipital_Mid_R, 0.53817	10	−89	29
44	Occipital_Mid_R, 0.86166	28	−87	18

Eye movement recording and preprocessing

An eye-tracking device (EyeLink 1000 Plus, SR Research Ltd., Canada) was used for monocular eye tracking at 1000 Hz. To ensure accuracy and reliability in the data collected, a standard 9-point calibration and validation procedure was performed at the beginning of each block. This calibration process is crucial for aligning the eye tracker with the participant’s gaze, thereby enhancing the precision of the eye movement data recorded throughout the experiment. Using EYE-EEG v1.0^15,16, the raw files were transformed into a MATLAB structure. The positions, amplitudes, velocities, and durations of saccades; the positions, durations, and average pupil sizes of fixations; and the blink durations were extracted.

EKG recording and preprocessing

Two gel electrodes were applied to both sides of the chest and connected to the bipolar channels of the amplifier. For the EKG recordings, the same amplifier used for the EEG signals was used, and electrode impedance was maintained below 5 kΩ. The EKG data were extracted from the EEG files and filtered using a 0.01–40 Hz bandpass filter. Employing the HRVTool¹⁷ v1.07 toolbox, heart rates were computed with a threshold set at [50 220] beats per minute. The trimmed moving average filter was set at a length of 0.2 s, and the windowed extrema was set at 167. Annotations were converted into RR sequences with artifact removal.

Data Records

All the raw and preprocessed behavioral, EEG/EKG, eye movement, and fNIRS data used in this work have been stored as zip files in a public database¹⁸. A summary of the information, such as the format and file size, is shown in Tables 3, 4. For detailed descriptions of the data fields, please refer Table 5. Specific steps of the workflow are detailed in Table 6.

Table 3.

Raw data file information.

Type	File	Format	Variable	Value	Size
Behavior	sub(X).mat	MATLAB data format	fidnew	1 × 1 struct	4.59 MB
Eye movement	sub(X).edf	Eyelink data format			1.46 GB
EEG	sub(X).cdt	CURRY raw float format			20.8 GB
fNIRS	sub(X)_P1.csv sub(X)_P2.csv	Comma-separated values			1.01 GB
EKG	sub(X)_rst.mat sub(X)_tsk.mat	MATLAB data format	EKG_RST EKG_TSK	1 × 6 struct 1 × 6 struct	424 MB
Total					23.7 GB

Table 4.

Preprocessed data file information.

Type	File	Format	Variable	Value	Size
Behavior	sub(X).mat	MATLAB data format	BEH	172 × 6 struct	217 MB
Eye movement	sub(X).edf	MATLAB data format	EYE	172 × 6 struct	4.65 MB
EEG	sub(X)_rst.mat sub(X)_tsk.mat	MATLAB data format	EEG_RST EEG_TSK	1 × 6 struct 1 × 6 struct	17.4 GB
fNIRS	sub(X)_rst.mat sub(X)_tsk.mat	MATLAB data format	NIR_RST NIR_TSK	1 × 6 struct 1 × 6 struct	2.48 GB
EKG	sub(X)_rst.mat sub(X)_tsk.mat	MATLAB data format	EKG_RST EKG_TSK	1 × 6 struct 1 × 6 struct	476 MB
Total					20.6 GB

Table 5.

Data structure description.

Type	Field	Value	Description
Behavior	Tag	double	Target position
	Rt1	double	Reaction time
	Rt2	double	Movement time
	Rt3	double	Response time
	MvX	double	X coordinate
	MvY	double	Y coordinate
	Dst	double	Distance of movement
	The	double	Orientation angle
	tTh	double	Mean orientation angle
	dTh	double	Deviation from mean orientation angle
	Acc	double	Accuracy
Eye movement	Fix	* × 6 double	Fixations: start time, end time, duration, averaged gaze X, averaged gaze Y, averaged pupil size
	Sac	* × 9 double	Saccades: start time, end time, duration, start position X, start position Y, end position X, end position Y, amplitude, peak velocity
	Bli	* × 3 double	Blinks: start time, end time, duration
	Tag	double	Target position
	* indicate the number of detected eye movement events. If a cell is empty, it signifies that no valid eye movement events were identified
EEG	—	—	EEGLAB struct
fNIRS	HbO	* × 44 double	Oxygenated hemoglobin
	HbR	* × 44 double	Deoxygenated hemoglobin
	HbT	* × 44 double	Total hemoglobin
	Evt	n × 1 double	Event type
	* represent the sampling points
EKG	REST	1 × * single	Resting
	SAME	1 × * single	Easy task
	DIFF	1 × * single	Hard task

Table 6.

Steps of format conversion, synchronization and preprocessing.

Step	Type	Comments
Format Conversion	Behavioral	Extract required variables from the raw behavioral data and renames them according to the study specifications.
	Eye movement	Convert eye-tracking data from.asc format to.mat format and extracts eye movement events (e.g., fixations, saccades, blinks).
	EEG	Convert EEG data from.cdt format to EEGLAB’s.set format for further processing in EEGLAB.
	fNIRS	Convert near-infrared spectroscopy (NIRS) data from.csv format to.nirs format.
Synchronization	All	Synchronize the sampling times of all modalities based on the event markers from the EEG data.
Preprocessing	Behavioral	Calculate the joystick’s angle at the time of button press based on XY coordinates. It then computes the deviation of each trial’s angle from the mean angle.
	Eye movement	Remove duplicate data samples in the eye-tracking data.
		Extract eye-tracking events (fixations, saccades, blinks) within specific time windows.
	EEG	Detect bad channels based on predefined thresholds and attempts to repair them using interpolation.
		Apply bandpass filter to the EEG data, with cutoff frequencies of 0.1 Hz and 30 Hz.
		Remove irrelevant channels, and performs ICA to decompose the EEG signal.
		Use ICLabel to classify ICA components and sorts them by weight.
		Remove electromyographic (EMG) and electrooculographic (EOG) components based on a threshold value.
	fNIRS	Check and flag bad NIRS channels based on criteria such as negative values, infinities, NaNs, and abnormal standard deviations.
		Perform preprocessing steps including light signal conversion, band-pass filtering, and motion correction.
		Attempt to repair bad channels in the NIRS data using interpolation from nearby channels.
	EKG	Extract the EKG channel from the raw EEG data.
		Apply high-pass and low-pass filters to the EKG data, with cutoff frequencies of 0.01 Hz and 40 Hz, respectively.
		Segment the EKG data into rest state and two task states based on event markers.

Raw dataset

Behavioral data were stored in.mat files, primarily in a structure named “fidnew,” containing joystick positions, timestamps, and corresponding event information. Eye-tracking data were stored in.edf files in a binary format, which could be converted to ASCII using the EDF2ASC tool (https://www.sr-research.com/support/thread-7674.html). The raw EEG data were stored in.cdt files, from which the raw EKG data were extracted and saved as.mat files. The raw near-infrared measurement data were stored in.csv files, with each subject having two probes.

Preprocessed dataset

All preprocessed data underwent temporal alignment and formatting to.mat files. Preprocessed behavioral data were stored in a 172 (trials) × 6 (blocks) structure, including corrected reaction times, joystick movement distances, and angles. The preprocessed eye-tracking data of each subject, including fixations, saccades, blinks, and target identifiers, were also stored in a 172 × 6 structure. EEG and fNIRS data were segmented into resting-state and task-state signals, with 6 blocks per subject. EEG data were formatted into EEGLAB structures, while fNIRS data were converted into HbO, HbR, HbT, and event sequences. The EKG data also comprised 6 structures, each containing rest, easy and hard task conditions.

Technical Validation

Synchronization testing and correction

Due to different data recording systems using their own local clocks for sampling (with local clocks of different devices being asynchronous), there is an inevitable issue of data sampling time inconsistency in multimodal data. To ensure temporal alignment across multimodal data, event markers from the EEG data were employed as a reference (see Fig. 4). We extracted the event marker sequences from the raw EEG data as a reference. By performing a linear fit on the time series of the same event in multimodal data, we calculated the slope and intercept representing the time drift between modalities, as shown in Eq. (1). We then corrected the timestamps of the other data types accordingly, and applied linear interpolation. This simple time-axis correction method is effective for quickly diagnosing synchronization issues and improving data usability.

$$T_{\mathit{corrected}}=\frac{T_{\mathit{original}}-\mathit{Offset}}{\mathit{Slope}}$$ 1

Fig. 4.

Comparison of time alignment before and after synchronization based on identical event markers. (a) Before calibration. (b) After calibration.

However, real-world factors such as network transmission delays or operating system driver latencies introduce randomness, which presents certain limitations to a purely linear model approach. Each device in the setup uses its own independent internal clock for data sampling, typically derived from a crystal oscillator or similar source¹⁹. Because of the inherent inaccuracies of electronic components, data from different formats are not strictly time-synchronized. Additionally, clock drift accumulates over longer periods, affecting the synchronization of multimodal data²⁰. Varying degrees of offset and drift were observed in the behavioral, eye-tracking, and fNIRS data (Fig. 4a). Behavioral and eye-tracking data exhibited drift rates of +0.05 ms/s and -0.05 ms/s, respectively. Linear correction was applied to constrain relative time errors within ±0.1 ms (Fig. 4b). The data synchronization improved after calibration, making it suitable for long-term analysis.

Behavioral data validation

We manipulated the difficulty of the visuomotor task by setting easy and hard conditions. The comparison between two conditions allowed us to assess visuomotor processing. We hypothesize that our behavioral data recording and analysis can distinguish explicit behavioral indicators resulting from variations in task difficulty, thereby validating the quality of the behavioral data.

Figure 5a–d displays the differences in reaction time, movement time, response time, and deviation angle across the two task difficulties. Paired t tests were conducted to compare reaction time, movement time, response time, deviation angles, and response variabilities between the two tasks. In terms of speed, the reaction time of the different target groups was significantly greater than that of the same target group (t₄₇ = 18.900, p < 0.001, d = 5.514), while there was no significant difference in movement time (t₄₇ = 1.498, p = 0.141, d = 0.437). Since the response time equals the sum of the reaction time and movement time, it is not surprising that response time also exhibited a significant difference (t₄₇ = 9.780, p < 0.001, d = 2.853). Similarly, the deviation angle of the hard task was significantly greater than that of the easy task (t₄₇ = 4.960, p < 0.001, d = 1.447), and the distributions of the mean probability at different deviation angles in both tasks are shown in Fig. 5e. The response error distribution was fit with the mixture model²¹ using the Analog Report Toolbox in MATLAB. Using mixture mode, the concentration parameter K of the von Mises distribution was calculated to describe the response variability. The modeling results also showed that the response variability of the easy task was significantly greater than that of the hard task (t₄₇ = 6.387, p < 0.001, d = 1.863). Additionally, we also examined the time and deviation angle for different target directions, but no significant differences were found. These behavioral results are consistent with those of previous research²².

Fig. 5.

Statistics of the behavioral data. (a) Mean reaction time of subjects across easy and hard tasks. (b) Mean movement time of subjects across easy and hard tasks. (c) Mean response time of subjects across easy and hard tasks. (d) Mean deviation angle of each subject across easy and hard tasks. (e) Probability of response deviation in easy and hard tasks. (f) Response variability K of subjects in easy and hard tasks.

Behavioral temporal structure validation

Attention moves rhythmically due to changes in arousal or alertness, which contributes to performance variation over time. This is traditionally calculated by sampling attention. Our main goal was to examine the temporal structure of attention monitoring. To achieve a dense temporal assessment of behavioral performance, we used a time-resolved measure, allowing the target to appear at one of 48 temporal intervals (see Fig. 2c), in steps of 100 ms, from 2.3 s to 4.7 s after the recovery state, corresponding to a sampling frequency of 10 Hz. The SOA of each trial was pseudorandom from 2.3 s to 4.7 s. RTs that were 3 SDs across all trials were excluded from further analysis. The remaining RTs were then normalized within each subject individually to remove the large variance between individuals in overall motor responses.

We adopted the method proposed by Brookshire²³. Surrogate datasets were generated using an autoregressive (AR) model. Specifically, an AR model with a single positive coefficient (AR-1) was used to model the time series data of the target detection performance of each participant utilizing exact maximum likelihood estimation through a Kalman filter. This AR-1 model effectively captured the lag-1 autocorrelation and the nonperiodic characteristics of the original series. The model was then employed to create 10,000 surrogate time series. Each of these series was detrended by removing the quadratic polynomial trend. Following detrending, the series underwent transformation into the frequency domain via a fast Fourier transform (FFT). The mean power spectrum of the surrogate data was then calculated for all participants. This mean was compared to the mean power spectrum of the actual data on a frequency-by-frequency basis. Frequencies where the actual power spectrum exceeded the 95th percentile of the surrogate mean were identified as statistically significant. Multiple frequency comparisons were corrected using the Bonferroni method. Our results showed that behavioral performance fluctuated periodically at 1-2 Hz in reaction time for the hard task (Fig. 6b, p < 0.05, corrected for multiple comparisons). There was no significant difference in the rhythms of the other behavioral indices (Fig. 6, ps > 0.351).

Fig. 6.

The temporal structure of attention monitoring. (a) Reaction time for the easy task, (b) reaction time for the hard task, (c) movement time for the easy task, and (d) movement time for the hard task.

Eye movement data validation

Previous research suggests that microsaccades may occur during visual search stages and are related to task difficulty²⁴. We hypothesize that microsaccades will occur 200–300 ms after the target appears, even if participants are instructed to fixate on the central cross. The probability of saccade occurrence over time was assessed for two tasks after the stimulus presentation. A t test was conducted within a 100 ms window following the stimulus to examine differences in saccade duration. In the process of handling eye-tracking data, one of the steps was data cleaning, which involved removing abnormal data caused by device errors, external disturbances, or atypical participant behavior. This process included excluding periods when the eyes were closed during recording and eliminating data points that fell outside normal physiological ranges.

The saccadic probability and fixation heatmaps are displayed in Fig. 7. During the stimulus presentation (0–200 ms), eye movement remained relatively constant. In the easy task, the fixation points were mainly concentrated in the central area, while in the hard task, the fixation points were relatively dispersed. The probability of saccades occurring within the 200–300 ms window showed an initial increase followed by a decreasing trend, with significant differences between easy and hard tasks (Fig. 7a, t₄₃ = 5.818, p < 0.0001, d = 1.774). The duration of saccades in the hard task between 200–300 ms was also greater than that in the easy task (Fig. 7b, t₄₃ = 3.188, p = 0.002, d = 0.972). The probability of fixating on the target direction increased in the easy task in both the 0–200 ms (Fig. 7c) and 200–300 ms (Fig. 7d) periods. Subsequently, within the 350–500 ms range, the probability of saccades gradually increased. Fig. 7a shows that the probability of saccades occurring is lower than fixations within 0–300 ms of stimulus presentation, while Fig. 7c,d further demonstrate that the probability density of fixation differences is concentrated mainly near the center of the screen, with a smaller difference than in other regions. This indicates that participants largely maintained their gaze on the fixation point during the first 300 ms of stimulus presentation, consistent with the experimental instructions. These findings confirm the validity of the eye movement data, as they are consistent with previous research²⁵. Additionally, we conducted trial-by-trial statistical tests on the average duration of saccades within this interval, revealing that the duration of saccades in the hard task was significantly greater than that in the easy task.

Fig. 7.

Comparison of eye movement. (a) The probability of saccades over time from stimulus presentation (gray bar indicates stimulus presentation). (b) Statistics of the duration of saccades within 200–300 ms. (c) Heatmap of the difference in fixation probability between easy and hard tasks in the 0–200 ms period. (d) Heatmap of the difference in fixation probability between the easy and hard tasks in the 200–300 ms period.

EEG data validation

The N2pc is a negative deflection in the ERP component observed approximately 200–300 milliseconds after the presentation of a visual stimulus. The N2pc is usually recorded over posterior scalp sites. This component is commonly induced in visual search tasks, where participants are required to find a target among distractors²⁶. The presence of the N2pc can provide insights into the efficiency and dynamics of attentional selection. Therefore, we hypothesized that the N2pc will typically be induced by a target aircraft identification scenario in our visuomotor paradigm. EEG segments were extracted from −200 ms to 400 ms relative to visual search onset and set to a baseline between –200 ms and 0 ms before the onset of the visual search in each trial. According to the appearance of the target in the left or right visual field, averaged contralateral and ipsilateral waveforms of posterior site electrodes (P3/4 and P7/8) were computed. Then, based on the differences between them, the N2pc component was derived.

As shown in Fig. 8, we utilized ERPs to validate the quality of the EEG data. Even across different tasks, the contralateral and ipsilateral ERP waveforms exhibited considerable consistency. The first component observed was the N100 component, which showed no significant differences across conditions. Subsequently, within the 200–300 ms window, the difference wave obtained by subtracting the ipsilateral from the contralateral wave showed a negative deflection (Fig. 8c), with a significantly greater amplitude observed in the hard task than in the easy task (Fig. 8d, t₄₇ = 4.091, p < 0.001, d = 1.193). A topographic map of the N2pc amplitude difference between easy and hard tasks is shown in Fig. 8e. The EEG results for N2pc were corroborated by previous research²⁷.

Fig. 8.

Contralateral and ipsilateral ERPs of easy (a) and hard (b) tasks at P3/P4 and P7/P8. (c) The waveforms of N2pc in the easy and hard tasks. (d) The mean amplitude of N2pc for each subject averaged between 200–300 ms in the easy and hard tasks. (e) A topographic map of the difference in the N2pc amplitude between the hard and easy tasks. The white dots represent the 2 pairs of electrodes, P3/P4 and P7/P8, that were used to calculate the N2pc.

fNIRS data validation

Both the prefrontal and parietal cortices are involved in the integration of visual input to motor output²⁸. We analyzed the fractional amplitude of low-frequency fluctuations (fALFF)²⁹ index within the prefrontal and parietal cortices to assess neuronal spontaneous activity and hemodynamic changes during visuomotor tasks. To quantify the proportion of low-frequency fluctuations in overall brain activity, the fALFF was calculated by dividing the amplitude in the low-frequency band (0.01–0.1 Hz) by the total amplitude across the entire frequency range (0.01–0.2 Hz). Subsequently, these fALFF values were standardized into z scores to facilitate comparisons across different conditions.

Hemodynamic activity, reflecting blood flow and oxygenation changes linked to neuronal activity, increases in regions with heightened activation during cognitively demanding tasks. These hemodynamic changes are captured by the fALFF z-score differences between the easy and hard tasks. To visually represent the spatial activation of hemodynamic activity, we mapped the relevant results onto a 2D channel layout (Fig. 9a) and a 3D brain model (Fig. 9b). For the two tasks, the z scores of the fALFF showed a similar distribution pattern (p < 0.050, FDR corrected), with lower frequency fluctuations in the frontal cortex than in the parietal cortex. Compared to the easy task, the hard task showed positive activation in the bilateral superior occipital regions and bilateral prefrontal regions, while the middle frontal region exhibited negative activation. These spatially specific activation patterns are consistent with previous studies³⁰. The fusion of NIRS data and EEG data in a rapid event-related design can be achieved through a GLM-based EEG-informed analysis^3,26,27.

Fig. 9.

Spatial activation of hemodynamic activity (a) 2D interpolated images of the averaged z scores of fALFF. (b) Hemodynamic activity of the hard v.s. easy tasks from different perspectives (p < 0.05, FDR).

EKG data validation

As shown in Table 7 the MeanRR is slightly lower in the hard task (795 ± 15 ms) compared to the easy task (800 ± 14 ms). SDNN shows 98 ± 13 ms for the easy task and 101 ± 15 ms for the hard task. The rMSSD values are similar between tasks, with 65 ± 6 ms for the easy task and 66 ± 8 ms for the hard task. Finally, the pNN50 percentages are almost identical, with 28 ± 3% for the easy task and 27 ± 3% for the hard task. All values were consistent with previous research³¹, which validates the effectiveness of the EKG data. Four commonly used cardiac indices, including the mean RR interval (MeanRR), standard deviation of the R-R intervals (SDNN), root mean square value of the difference between adjacent RR intervals (rMSSD), and percentage of successive RR intervals >50 ms (pNN50), were computed.

Table 7.

Parameter of heart rate variability under two tasks.

HRV	Easy task	Hard task
MeanRR (ms)	800 ± 14	795 ± 15
SDNN (ms)	98 ± 13	101 ± 15
rMSSD (ms)	65 ± 6*	66 ± 8*
pNN50 (%)	28 ± 3*	27 ± 3*

(a) MeanRR, (b) SDNN, (c) rMSSD, (d) pNN50 (Results are expressed as mean ± s.e.m. *indicates significance of comparison with resting state)

Although no statistically significant differences were found between the different conditions, as shown in Table 7, the task-related rMSSD and pNN50 exhibited significant differences compared to those in the resting state, while there were no significant differences between the easy and hard tasks. All measures of heart rate variability were consistent with previous research³¹, which validates the effectiveness of the EKG data. We observed a significant increase in the rMSSD during the easy task (t₄₄ = 4.172, p < 0.001) and the hard task (t₄₄ = 2.684, p = 0.010) compared to the resting state. Similarly, for pNN50, there was also a significant increase during the easy task (t₄₄ = 3.603, p < 0.001) and the hard task (t₄₄ = 3.191, p = 0.002).

Usage Notes

The raw data and preprocessed data are stored in separate directories based on their modalities: behavior, EEG/EKG, eye tracking, and fNIRS. It is strongly recommended that users utilize preprocessed data for the following reasons. Due to communication line faults or external interferences, individual event marker information may be lost, which can be repaired using redundant information from other modalities. After time calibration, the time drift and jitter of the different modalities’ data throughout the entire experiment are within ±0.1 seconds, which is crucial for continuous and long-term analysis. Aligning various computations and markers is a very time-consuming operation, and using preprocessed data can greatly enhance the efficiency of subsequent research.

Author contributions

Hao Zhang: Data processing and analysis, writing-original draft. Yiqing Hu: Data collection and visualization, writing-original draft. Yang Li: Provided advice on manuscript revision. Shuangyu Zhang: Performed experimental sessions and manuscript editing. XiaoLi Li: Supervised the study and edited the manuscript. Chenguang Zhao: Funding acquisition, conceptualized the study, experimental design, supervision. All authors reviewed the manuscript and approved the final manuscript.

Code availability

All data preprocessing code are available at https://github.com/Chenguang918/visuomotor. Users only need to configure the initial data path (input and output) and install the relevant toolboxes to run directly in MATLAB. The code prefixes step_(X) indicate the order of execution steps.

Competing interests

The authors declare no competing interests.