ADHD detection based on human action recognition

Abstract

Attention Deficit Hyperactivity Disorder (ADHD) is a highly prevalent human neurobehavioral and neurodevelopmental disorder worldwide. Recently, deep learning-based techniques have been exploited in ADHD detection and diagnosis due to their outstanding performance. However, the majority of these methods relying on fMRI and EEG data suffer from the limitation of requiring expensive equipment and incurring high operational costs. Therefore, inspired by the fact that the symptoms of ADHD may manifest in actions and daily behaviors (as stated in the Medical Statistical Manual of Mental Disorders, Fifth Edition (DSM-V)), we introduce a novel ADHD detection system based on human action recognition. We design a novel hyperactivity test for capturing ADHD features and record a real multimodal ADHD dataset (M-ADHD) for the first time. The proposed system detects ADHD symptoms based on acquired action characters from raw RGB videos from the M-ADHD. Our system outperforms conventional competitors in terms of accuracy and AUC on the real multimodal ADHD dataset. Our proposed method, based on simple, non-wearable sensors, has the advantages of being cost-efficient and easy to operate. It is widely applicable for remote ADHD screening and further applies to understanding, treating, and preventing brain disorders.

1. Introduction

Attention deficit hyperactivity disorder (ADHD) is a highly prevalent neurodevelopmental disease worldwide. A systematic review investigating the adult prevalence of ADHD globally found the following prevalence rates. In adults with persistent ADHD (a childhood onset), the rate was 2.58%; in adults with symptomatic adult ADHD (regardless of childhood onset), it was 6.76%. This translates to 139.84 million and 366.33 million affected adults in 2022 worldwide (Pawley et al., 2024; Nash et al., 2022; Ricart et al., 2024). According to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-V), the traditional ADHD diagnostic process relies on clinical observation and self-reporting (Americana, 2013; Ricart et al., 2024). It highly depends on the qualifications and judgment of the psychological clinicians. Due to the time-consuming process of conventional diagnosis and the shortage of experienced clinical consultants, it has been reported that the waiting period for ADHD diagnosis and treatment can extend to several years (Nash et al., 2023; Serra-Añó et al., 2019).

Recently, machine learning methods for ADHD detection and diagnosis have attracted significant attention. Detection and diagnosis methods based on Magnetic Resonance Imaging (MRI) (Pawley et al., 2024; Bellec et al., 2017; Li et al., 2023a) and Electroencephalography (EEG) (Tang et al., 2020; Palagini et al., 2024; Li et al., 2023b) have achieved high accuracy of over 95% on related datasets, i.e., ADHD-200. As shown in Fig. 1, these researches mainly focus on the neurobiological symptoms with objective brain changes, limited by their expensive equipment and high operational costs (Loh et al., 2022).

Fig. 1.

A typical MRI Scanner (left) and an example of fMRI scan output from ADHD-200 (right). MRI scanners are expensive equipment, so they are constantly used at hospitals and operated by trained professional staff.

Supportive evidence shows that nodal efficiency is profoundly affected in prefrontal, temporal, and occipital cortice regions in patient brains with ADHD, resulting in persistent attention deficit disorder and behavioral hyperactivity symptoms (Chen et al., 2020). However, ADHD-related actions such as fidgeting and restlessness in clinical notes are typically generalized rather than characterized by repetitive, stereotyped movements (Americana, 2013). In the traditional diagnostic process, it is challenging to accurately count and extract these action-related behavior characteristics solely through clinical observation (Tian et al., 2023; Hasan et al., 2024; Tang et al., 2022). Hence, machine learning methods based on action recognition have the feasibility and application prospects to identify and analyze ADHD symptoms (Calesella et al., 2024). Basic sensor-generated videos are readily available and can be used for extensive foundational screening. Compared with neurobiological methods, action recognition-based detection and diagnostic approaches are well-suited for application with simple raw videos (Li et al., 2023c, 2023d).

Human action recognition (HAR) has recently been crucial for telemedicine and telehealth applications(Calesella et al., 2024; Sun et al., 2022). By employing machine learning-based HAR methods, manual data limitations have been overcome by extracting action information from raw videos, such as skeleton-joint and optical flow. These methods can mitigate the impact of privacy-sensitive information in the environment or capture and predict changes in actions, demonstrating significant robustness in dynamic and complex settings (Palagini et al., 2024; Duan et al., 2022).

This paper proposes a novel ADHD detection system that utilizes action recognition from human skeleton-joint modality and raw RGB videos. Our work summarizes ADHD detection based on action features and employs video-based detection for the first time, in contrast to previous methods primarily relying on fMRI or EEG, which incur high equipment costs. The proposed method incorporates innovative evaluation criterion based on action feature statistics throughout entire videos. Moreover, our work significantly extends ADHD detection techniques and provides a relevant real multimodal ADHD dataset. Based on our previous work, this paper expands to a dual-network diagnostic system and includes an ablation study validating the effectiveness of different modules. Additionally, we introduce trajectory tracking and angle calculation for key joint points to supplement action-related ADHD features. By performing late fusion of outputs from different modules and analyzing 2D joint-skeletal information from action-dense segments, the proposed method offers significant advancements over the state-of-the-art, demonstrating robust performance and comprehensive ADHD detection.

Our main contributions include:

• •In alignment with ADHD characteristics, a trial assessing ADHD action and reaction abilities has been designed. This trial is the foundation for recording and compiling video, audio, tactile, and questionnaire data. It also provides the first real multimodal ADHD dataset.
• •A novel automatic ADHD detection system is proposed based on human action recognition, exploiting both the raw RGB videos and extracted skeleton grid sequences from the above-mentioned real multimodal ADHD dataset. Action recognition networks are applied to ADHD detection for the first time. The performance of the system is also validated by the medical approved body.
• •New performance measurements are proposed to classify the detection results and analyze the ADHD-related action characteristics. Furthermore, we also verify the efficiency and feasibility of the proposed method with detailed analysis, including the ablation studies and cross-validation results.

2. Proposed methodology

Various physiological and psychological researchers have verified a series of typical action-related symptoms that occur in the ADHD group. Thus, we propose an ADHD detection system based on action recognition to extract action-related characteristics from raw videos. The system contains five main parts: data recording and preparation, action recognition, performance measurements, and action analysis. Our method uses two novel performance measurements to evaluate the action and frequency features, then classify and detect ADHD automatically through comprehensive scoring results. Moreover, based on the results of ADHD detection, we selected segments containing concentrated actions from the original videos for further analysis. Fig. 2 illustrates the proposed framework of our ADHD detection method. We will present the details of each part in the following subsections.

Fig. 2.

Illustration of the proposed framework for ADHD detection and analysis approach based on our real multimodal ADHD dataset.

2.1. Data recording and preparation

To ensure the robustness and accuracy of the system, we only apply our novel multimodal ADHD dataset at the testing stage of the proposed ADHD detection system. The multimodal ADHD dataset is independent and remains completely unseen during the training phase of the network. After taking the dataset as input in the testing stage, the proposed system automatically labels the actions and classifies the ADHD subjects and neurotypical controls. The real multimodal ADHD dataset contains 10 adult ADHD subjects diagnosed by medical bodies/consultants under DSM-V criteria and 12 neurotypical controls. The dataset includes five male and five female ADHD subjects and eight male and four female neurotypical controls. Since our research is focused on adult ADHD detection and analysis, the age distribution of all the subjects and the controls is between 18 and 45 years. All subjects are provided by the CNTW-NHS Foundation Trust, while the controls are volunteers from Newcastle University, UK.

Fig. 3 shows the Intelligent Sensing Lab layout and the recording environment of the multimodal ADHD dataset. We fully recorded videos from three field-of-views (FoV) with three GoPro cameras, i.e., the front, left, and right sides of each participant. In addition, the questionnaire, audio, and the keypad's touch signal were also recorded during each test. Since our tests focus on the attention ability of participants, a screen with distraction posters is placed within the participants' line of sight, and small objects like pens and spinners are placed on the desk.

Fig. 3.

Recording room layout for the Intelligent Sensing-based real multimodal (audio-video-torch/keypad sensors) ADHD dataset. Camera 1 records the facial information and upper torso information. Cameras 2&3 record information on the left and right body.

The consecutive tests focus on the ability of attention and responsiveness, which is required for all participants. We prepare four continuous dialogue tasks/tests: 1) a brief conversation between a subject/control during the interview takes approximately 10–20 min. The interview task consists of 21 questions selected from Diagnostic Interview for ADHD in Adults (DIVA), i.e., used by NHS medical consultants and professionals; 2) performing Cambridge Neuropsychological Test Automated Battery (CANTAB) test, including Cambridge Gambling Task (CGT), Stop Signal Task (SST), Rapid Visual Information Processing (RVP), and Spatial Working Memory (SWM). This test takes about 40–50 min; 3) the beep reaction task requires a participant to respond to different beeps generated randomly to capture the attention response. This task takes 6 min; 4) watching videos, including a math video labelled ‘boring’ and a rally video labelled ‘exciting’. This task takes 10 min. The entire testing process takes about 1–1.5 h. Videos are recorded by three GoPro cameras, which contain a front-faced Camera 1 to record facial information. Meanwhile, Cameras 2&3 record information on the left and right sides of the torso and limbs, respectively. The resolution of the three cameras is the same, i.e., 3840 × 2160 pixels.

For the training of the proposed action recognition network for ADHD detection. In addition, we also recorded approximately 5 h of action data. This action data was recorded independently and unseen in the same recording environment with Cameras 2 and 3. During the recording, volunteers were asked to perform still-position (Action 1), limb-fidgets (Action 2), and torso-movements (Action 3). The collected data is manually processed, labelled as a separate dataset named as Action-3. Again, the Action-3 data, only for training only is completely independent of the real multimodal ADHD dataset. It is worth mentioning the proposed approach mitigates the scarcity of healthcare data, e.g., recruiting subjects formally diagnosed by medical bodies is not an easy task.

2.2. Action recognition networks in the proposed system

As discussed above, we focus on the action-related characteristics of ADHD. The general action recognition networks could not apply to typical ADHD symptoms and action recognition, e.g., slight limbs and body fidgeting. We design and present a two-stage action recognition network based on raw RGB video and human skeleton grid sequences and apply it to our ADHD detection system.

Based on the raw video, we choose a 3D-CNN structure (C3D) as the main core network (Tran et al., 2015). The 3D-CNN focuses on the temporal and spatial continuity in the video, which has a wide range of applications in action recognition. The 3D-CNN extending the convolution kernel applies to 3D structures. It is widely used in video processing to extract temporal information (LeCun et al., 2015). The frames are stacked to form a 3D structure for post-convolution calculation. The weights in the 3D-CNN convolution kernels are shared, and multiple convolution kernels can be used to extract multi-dimensional information (Tran et al., 2015).

There are 8 convolution layers that have 3 × 3×3 kernels with 1 stride. The kernel size of the first pooling layer is 1 × 2×2, and the rest of the layers are 2 × 2×2. Different from the original C3D structure, we add a fully connected layer to fit the size of the input data. The probabilities of each action are obtained with three fully connected layers with 8192 units and a Softmax activation.

The loss function of the training process is to minimize the cross entropy of the outputs and true label results:

$$H(P_l|P_o)=-\sum _i^m{P}_l(i)\log P_o(i)$$ (1)

where ith means the set of labels with length m. The P_l and P_o are the distributions of true labels and classification output, respectively.

In skeleton extraction and action recognition tasks, pose information, including skeleton-joint grid sequences, are captured with the detector and estimator from raw RGB video signals. The skeleton extraction task records action-related information without contextual nuisances, such as background variation and unrelated personnel interference (Sabater et al., 2021).

In general, 2D poses are of better quality and higher accuracy than 3D poses (Liu et al., 2019), which is crucial for the real application of medical detection and diagnosis. In this part, a ResNet50-based Faster-RCNN network is used as the detector (Sun et al., 2019); the pose estimator is a pre-trained HRNet because they achieve the state-of-the-art results on the commonly used human posture estimation dataset: MPII (top) and COCO (bottom) datasets (Sun et al., 2019).

As shown in Fig. 4, we use this combination of detector and estimator to capture standard benchmarks such as COCO-keypoints of each subject and control in a sitting position(Duan et al., 2022). To be consistent with the previous step, we also applied a 3D-CNN structure (PoseConv3D) as the main core network in this step (Duan et al., 2022). Compared with grid-based GCN methods, the interoperability of PoseConv3D makes it easier to process human skeletons in multi-feature fusion, potentially used in ADHD detection and diagnosis. Meanwhile, the PoseConv3D performs better on most existing action detection datasets, such as UCF101, NTURGB-D, FineGYM, etc. (Cheng et al., 2019).

Fig. 4.

Skeleton extraction results from our dataset's left and right side cameras and the COCO-Keypoints information.

Different from the original implementation (Duan et al., 2022), the first convolution layer of our PoseConv3D network is changed to 17 × 25 × 56 × 56 kernels with 1 × 1 stride to fit the size of our input data format.

Based on this ADHD detection framework, we adapt two popular 3D structure networks, i.e., R2Plus1D and R3D, to the action recognition task based on raw videos. We also compare the proposed method with the other two commonly used skeleton-based action recognition networks (ST-GCN, MS-G3D). To ensure the fairness of the experiment, we use the same configurations, i.e., estimator and detector, in the skeleton extraction for ST-GCN and MS-G3D as the proposed PoseConv3D. Recognizing the substantial impact of action recognition results on ADHD detection results. We select training parameters by referring to the benchmarks of the UCF-101 dataset in the literature and subsequently make adjustments. Each hyperparameter is adjusted through grid search, and the hyperparameters yielding the highest accuracy on our action recognition dataset Action-3 are chosen for the parameter settings applied in the system.

2.3. Proposed performance measurements

Regarding typical ADHD symptoms, the actions and performances of subjects and controls in our proposed method mainly contain three categories: still position, small ranges of limb fidgets, and large rotations of torso movements. The existing action recognition evaluation criterion can hardly be applied to ADHD-specific detection. We propose a novel Hyperactivity Score (H) and two novel measurements, i.e., Attention Deficit Ratio (R_AD) and Stationary Ratio (R_S), as the evaluation criterion for action recognition results and apply to ADHD detection. They respectively focus on the action change frequency and the percentage of stationary periods of the subjects and controls during the test, which is also defined as the participant's ability to focus on keeping the posture.

The Hyperactivity Score (H) is calculated as :

$$H_n=\left\{\begin{array}{ll}{H}_{n-1}+1,& \text{ if }{l}_n=l_{n-1}\\ H_{n-1}-1,& \text{ otherwise }\end{array}\right)$$ (2)

where n denotes the number of labels, H_n denotes the score of nth label, l_n denotes the nth label in the label sequence. H increases if the action is consistent in the continuous time frame. Otherwise, it is reduced.

According to the effect of video length on H, we normalize the results by the ratio of H to the n labels and denote it as R_AD which is calculated as:

$$R_{AD}=\frac{H_n\cdot 100}{n}$$ (3)

where (⋅) is dot multiplication. The R_S is defined as:

$$R_S={\alpha }_1/({\alpha }_1+{\alpha }_2+{\alpha }_3)$$ (4)

where α₁ denotes the number of the samples of predicted still position, α₂ is the number of samples of small ranges (less than 30°) of limb fidgets, and α₃ is the number of the samples of large rotations (more than 30°) of torso movements.

The R_ADL and R_SL are calculated for the left side camera and similarly R_ADR and R_SR are calculated for the right side camera. The final R_ADavg value is the average of R_ADL and R_ADR. The same applies to the R_Savg value.

The detection result (R) is obtained by binary classification as:

$$R=\left\{\begin{array}{ll}0,& \text{ if }{R}_{\mathit{ADavg}}\text{or}{R}_{\mathit{Savg}}<T\\ 1,& \text{ otherwise }\end{array}\right)$$ (5)

where T is the threshold. The 0 and 1 denote subject and control, respectively.

In addition, we perform late fusion on the action recognition results of the stages of action recognition from raw RGB video and skeleton grids to combine the frequency and stationary characteristics of action for classification.

The Fusion Ratio (R_F) of each participant is calculated as follows:

$$R_F=(w_1\cdot R_{\mathit{ADavg}}+w_2\cdot R_{\mathit{Savg}})/2N$$ (6)

where (⋅) is dot multiplication, w₁ denotes the accuracy of classification with R_ADavg, and w₂ denotes the accuracy of classification with R_Savg. The accuracy is calculated by a number of correctly classified values with each ratio and dividing by the total number of values. And N denotes the number of action recognition networks.

The performance of the proposed method is also evaluated by accuracy, sensitivity, recall, F1 score, and the area under curve (AUC) (Yang et al., 2022).

2.4. Action analysis

We propose an action analysis based on action recognition and the ADHD detection results. Through this module, we analyze the action characteristics of subjects and controls. In this part, a human knee fidgeting analysis algorithm based on the AlphaPose is proposed to recognize the knee rotation angle of participants.

As shown in Fig. 5, the four tasks focus on the participants' different attention directions. We selected videos containing ‘Task 1: Questions’ and ‘Task 4: Watching videos’ as the primary test video segments for action analysis. This decision is based in the consideration that Task 1 marks the initiation of the entire testing procedure, while Task 4 is at the end of the whole test. Notably, a pronounced variability in participants' attention levels is observed between these tasks, as evidenced by discernible distinctions in their actions. Moreover, in Task 1 and Task 4, participants are not required to perform additional actions, facilitating the identification and analysis of subtle actions related to hyperactivity. In contrast, Task 2 and Task 3 involve additional action requirements. Consequently, the ADHD detection thresholds for Task 2 and Task 3 are deemed unsuitable for cross-validation experiments involving entire videos and other tasks. The experimental component analyzes the trajectory waveforms of key points of each participant's left and right knees while in the sitting position. Two reasons underlie the choice of knee analysis: 1) leg fidgeting is a common symptom among ADHD patients, and 2) changes in body posture, such as lateral rotation, can impact the knee bending angle as captured by the left and right side cameras, making it easier to identify and quantify.

Fig. 5.

The multimodal ADHD real dataset recording tasks.

The trajectory of the key points of the joints is extracted from synchronized video sequences using pre-trained AI models: Object detector YOLOv3, and Pose estimator AlphaPose. The concepts are stored in a JSON file. Key points represent the coordinates of a joint [x, y], forming a list of 17 key points. The data comprises a list of m poses, where m corresponds to the number of frames in the video.

As illustrated in Fig. 6, once the trajectory of the key points has been collected, the raw angle kinematics are employed to analyze the action characteristics.

Fig. 6.

The process of skeleton-joint changes on the action of torso rotation and stationary state. The knee opening angle changes as the action occurs on the participants' torso and legs.

3. Experimental results

3.1. Dataset and experimental setup

We design a two-stage action-recognition system based on raw videos and skeleton-joint grid sequences. As mentioned in Section 2.1, to ensure the reliability and robustness of the system, we employ completely independent action data (which are from our separately recorded and manually labelled Action-3 dataset) in the training-related tasks of our networks. We only utilize ADHD data from our real multimodal ADHD dataset to test the whole system. The system could automatically annotate actions in the video and output ADHD detection results. A demo video is available at our YouTube channel .1

We apply the separate three-class action recognition dataset (Action-3), which is independent of our multimodal ADHD, for the training of action recognition networks. In Action-3, the three actions: still position contains 88 video clips; limb fidgeting contains 110 clips; and torso movements contains 101 clips. Each clip is between 10 and 15 s long. As we mentioned in Section 2.1, all video clips are manually labelled. For the training, validation, and testing of the action recognition network, the data in Action-3 is divided into 6/2/2, respectively. The accuracy of this action recognition network before being applied to multimodal ADHD detection is over 90%.

The training epoch for raw video-based action recognition network is 80, and the learning rate is empirically set to 1 × 10⁻⁹. For the skeleton-based network, the training epoch for the action detection is 30, and the learning rate is 4 × 10⁻³. All experiments are run on a workstation with four Nvidia GTX 1080Ti GPUs.

We applied the data collected by Cameras 2&3 on the right and left sides from our real multimodal ADHD dataset for ADHD detection. This testing ADHD dataset is unseen in the network training with 44 videos, including the whole process videos from the left and right sides of all 10 subjects and 12 controls. The length of each video is 60–90 min. In the whole ADHD detection system, all videos are reduced to 16 frames per second (FPS). Since the raw frame size is too large to feed into the action recognition module, the resolution of frames is also reduced to 228 × 128. After processing the frame segmentation and patch extraction, a square patch containing the samples’ torso and limb information in the center of the image is fed to the network. The patch size is 120 × 120. In the skeleton-based action recognition module, the resolution of the input frame is reduced to 1280 ×720. Action-related information is extracted through skeleton extraction tasks without contextual nuisances, such as background variation and unrelated personnel interference (Sabater et al., 2021). Actions are labelled per three frames in all detection steps.

3.2. Time-action-based detection decision results

As mentioned in Section 1, our system addresses this by fully capturing and visualizing actions from raw videos and the skeleton grid sequence of each participant. Fig. 7 presents the two-stage action recognition results on the testing real multimodal ADHD dataset from a randomly selected subject and control in a timeline bar chart.

Fig. 7.

Action change timeline chart of a randomly selected subject and control. The blue section represents the outcome of the skeleton-based action recognition stage, while the green section represents the raw video-based action recognition stage. Action 1 is still-position, Action 2 is limb-fidgets, and Action 3 is torso movements.

Through Fig. 7, it can be easily observed that the action change frequency for the ADHD subject is significantly higher than the control. Meanwhile, the proportion of the control remaining stationary (gray part) is much higher than that of the subject.

We further provide the R_AD and R_S performance of all 10 subjects and 12 controls, as shown in Table 1. The average for all participants R_ADavg and R_Savg based on RGB raw videos and skeleton grid sequences are denoted as R_{AD_RGB}, R_{AD_SKE}, R_{S_RGB} and R_{S_SKE}, respectively.

Table 1.

The Attention Deficit Ratio (R_AD), the Stationary Ratio (R_S) and the Final Ratio (FR) comparisons for the overall subjects and controls.‘ R_{AD_RGB}’, ‘R_{S_RGB}’, ‘R_{AD_SKE}’, ‘R_{S_SKE}’ indicate R_ADavg and R_Savg output from action recognition results-based RGB raw videos and skeleton grid sequence, respectively. ‘S’, ‘C’, ‘F’, ‘M’ indicate subject, controls, female, and male, respectively. While ‘Avr_S’ and ‘Avr_C’ indicate average output from subjects and controls, respectively.

Sample	S1 (M)	S2 (M)	S3 (M)	S4 (F)	S5 (F)	S6 (M)	S7 (F)	S8 (F)	S9 (F)	S10 (M)	C1 (M)	Avr_S
	C2 (M)	C3 (M)	C4 (M)	C5 (M)	C6 (M)	C7 (M)	C8 (F)	C9 (M)	C10 (F)	C11 (F)	C12 (F)	Avr_C
RAD_RGB	81.1	66.3	71.7	68.2	78.5	68.7	52.2	87.2	70.4	69.2	89.9	71.4
	96.4	98.9	79.2	95.3	95.7	96.1	72.0	84.1	90.6	95.8	98.1	91.0
RS_RGB	39.4	24.1	71.8	63.9	36.0	75.2	66.1	50.8	47.2	46.5	86.1	52.1
	91.0	96.8	73.3	89.9	94.7	87.3	73.2	77.0	89.4	86.5	93.1	86.5
RAD_SKE	48.9	75.7	67.8	71.9	72.6	72.1	76.1	68.3	69.8	58.4	80.8	68.2
	79.9	83.9	81.7	84.0	81.3	86.6	65.6	86.5	86.1	82.7	85.5	82.1
RS_SKE	56.1	69.4	46.8	55.5	78.2	70.8	71.2	64.2	48.7	24.3	79.9	58.5
	71.2	81.5	89.8	98.3	90.1	83.2	68.2	74.8	81.3	78.9	87.2	82.0
RF	49.7	52.0	57.3	57.6	58.5	63.6	59.0	59.8	52.3	44.0	74.6	55.4
	75.0	80.0	71.7	81.3	80.1	78.2	61.8	71.5	77.0	76.2	80.6	75.7

Table 1 shows a noticeable difference between the subjects and controls with different networks in the average R_{AD_RGB}, R_{AD_SKE}, R_{S_RGB} and R_{S_SKE}. Therefore, the average values in R_{AD_RGB}, R_{AD_SKE}, R_{S_RGB} and R_{S_SKE} for all 22 participants (i.e., 82.1%, 70.9%, 75.8%, and 71.3%) are adapted as the threshold T for ADHD detection.

3.3. Decision results

Table 2 shows the F1 score, precision, recall, accuracy, and AUC results with R_F for a set of networks applied to the ADHD detection system and our proposed two-stage ADHD detection networks.

Table 2.

ADHD detection performance with different neural networks.

	F1 Score (%)	Precision(%)	Recall (%)	Accuracy(%)	AUC(%)
R3D (Tran et al., 2018)	69.6	72.7	66.7	68.2	78.1
R2Plus1D (Targ et al., 2016)	15.4	100.0	7.4	50.0	56.0
ST-GCN (Cheng et al., 2019)	88.0	84.6	91.7	86.3	85.4
MS-G3D (Liu et al., 2020)	78.2	71.8	85.7	88.2	72.5
C3D + PoseConv3D	95.7	100.0	91.2	95.5	99.2

From Table 2, the proposed two-stage network (C3D + PoseConv3D) is significantly higher than the single-stage network in R3D, ST-GCN, and MS-G3D in precision, recall, accuracy, F1 Score, and AUC.

The proposed C3D network concentrates on the features from both the spatial and the temporal dimensions, thereby capturing the action information encoded in multiple adjacent frames, which plays an important role in ADHD typical human action recognition (Ji et al., 2012). The PoseConv3D network takes advantage of combining the skeleton grid and heatmap, which leads to improved performance on action recognition tasks, indicates a clear differentiation between ADHD subjects and controls in diagnosis outcomes, and improves detection accuracy. Therefore, the proposed two-stage network shows high sensitivity in the recognition results of the small range of limb fidgets and improves the performance of ADHD detection results. Furthermore, when combined with the strengths of the two networks based on raw RGB videos and skeleton grid sequences, the AUC of our proposed two-stage system shows improvement. Through late fusion, it can leverage the attributes from both networks, effectively mitigating background interference with sensitivity to action variations. As a result, the capability of our proposed system for action-based ADHD detection has been further enhanced.

3.4. Ablation study and comparison with state-of-the-art

In the ablation study experiment, we randomly collected approximately 250 video clips (each clip was between 15 and 20 s) from the real multimodal ADHD dataset and reorganized them into a binary ADHD classification dataset. The C3D-1 is a diagnostic discriminant network trained on this binary ADHD classification dataset. Aside from the binary ADHD classification dataset, PoseConv3D introduces the module of skeleton information extraction. The PoseConv3D-2 and C3D-2 are action recognition networks with and without skeleton extraction modules trained on the Action-3 dataset, as mentioned in Section 2.1, respectively. It is highlighted that the action recognition module should be related to R_AD and R_S measurements and cannot be separated. The AUC results are shown in Table 3:

Table 3.

Ablation study of four contributions in the proposed method. Bold indicates the best results.

Ablation Settings				Backbone	AUC (%)
Raw RGB	Skeleton	Action + RAD	Action + RS
✓	✗	✗	✗	C3D-1	51.7
✗	✓	✗	✗	PoseConv3D-1	59.1
✓	✗	✓	✗	C3D-2	95.1
✓	✗	✗	✓	C3D-2	93.0
✗	✓	✓	✗	PoseConv3D-2	95.3
✗	✓	✗	✓	PoseConv3D-2	97.0
✓	✗	✓	✓	C3D-2	95.3
✗	✓	✓	✓	PoseConv3D-2	98.1
✓	✓	✓	✓	C3D-2+PoseConv3D-2	99.2

According to the results of the ablation study, firstly, the action recognition significantly improves accuracy (e.g., 51.7% → 95.1% and 59.1% → 95.3%). Secondly, the combination of physical stationarity in action (R_S) and frequency of action (R_AD) also improves the distinction ability of the single-branch networks. Thirdly, the performance of the skeleton grid outperforms raw RGB. Skeleton features can mitigate privacy and provide more informed input to the networks related to human action recognition. Hence, skeleton features can yield explainable AI, and obtain better performance than raw RGB videos, with limited participants, when recruiting subjects formally diagnosed by medical bodies is challenging.

Finally, the late fusion networks, i.e., C3D-2+PoseConv3D-2 further enhance the performance. On one hand, there is information loss during skeleton extraction, while raw RGB preserves more comprehensive features from the original frames. On the other hand, the limited number of participants results in minor improvement when using raw RGB to the skeleton grid. This limitation will be addressed by including more participants in future studies. Therefore, in the late fusion networks, these two modalities are weighted and combined to address the limitations of a single modality.

Table 4 shows the performance of the state-of-the-art ADHD detection systems on other real ADHD datasets containing EEG and trajectory signals collected by wearable sensors. Due to the particularity of medical-related data, the datasets used in most ADHD studies are not open source, so we present the experimental results from the references. Additionally, we have calculated the CO2 emissions for each method to highlight the environmental impact of different diagnostic approaches.

Table 4.

ADHD detection performance of state-of-the-art methods and proposed method, where ‘S’ means ADHD subjects, and ‘C’ means controls. ‘CO2 Emiss’ means CO2 emissions in the whole process.

Author	S/C	Data Input	Accuracy	Cost ($)	CO2 Emiss (kg)
Luo et al. (Luo et al., 2020)	36/36	MRI & DTI	76.6%	15,000–100,000	19.0
Dubreuil et al. (Dubreuil-Vall et al., 2020)	20/30	EEG	88.0%	1000–25,000	1.25
Muñoz et al. (Muñoz-Organero et al., 2018)	11/11	Trajectory	93.8%	599	0.9
Proposed Method	10/12	Videos	95.5%	450	0.65

From Table 4, the proposed method outperforms the state-of-the-art ADHD diagnosis system. Compared with other machine learning methods for ADHD diagnosis, our proposed action-based system can more intuitively observe ADHD-related action characteristics. Simultaneously, our system is cost-efficient and environment-friendly, and it is also easy to apply. Therefore, the proposed method exhibits improved generalization and applicability, with the application potential for widespread screening and remote diagnosis.

3.5. Action analysis results

In connection with the ADHD detection and action recognition results in Section 3.4, we analyze the ADHD-related action characteristics of the specific tasks in terms of frequency and amplitude.

Fig. 8 shows trajectory plots of knee points for a group of randomly selected subjects and controls on Task 1 and Task 4, respectively.

Fig. 8.

Trajectory plots of knee points from two randomly selected control and subject in Task 1 and Task 4. The red and blue lines indicate the left and right knee points.

As shown in Fig. 8, there is a noticeable disparity in the knee trajectories between the subject and the control. The subject's knee exhibits a considerably wider range of angle changes compared to the control.

Table 5 provides the R_AD value of the subjects and the controls within 10 min and the knee joint's average opening angle change angle (calculated every 10 s) in Task 1 and Task 4, respectively.

Table 5.

The Attention Deficit Ratio (R_AD), the Knee Opening Angle (KOA) comparisons for the overall subjects and controls in Task 1 and Task 4. ’S’, ’C’, ’F’, ’M’ indicate subject, controls, female, and male, respectively. While ‘Avr_S’ and ‘Avr_C’ indicate average output from subjects and controls, respectively. ’-’ indicated that the knee opening angle cannot be calculated due to the absence of some joints.

Sample	S1 (M)	S2 (M)	S3 (M)	S4 (F)	S5 (F)	S6 (M)	S7 (F)	S8 (F)	S9 (F)	S10 (M)	C1 (M)	Avr_S
	C2 (M)	C3 (M)	C4 (M)	C5 (M)	C6 (M)	C7 (M)	C8 (F)	C9 (M)	C10 (F)	C11 (F)	C12 (F)	Avr_C
RAD-1(%)	48.7	67.5	82.1	75.3	65.0	83.8	79.0	46.7	68.2	39.0	76.3	65.5
	89.1	82.3	76.4	88.1	81.7	98.0	78.3	76.3	78.6	74.5	87.7	82.3
KOA-1(°)	34	13	23	–	12	27	2	23	6	31	10	17
	7	3	18	8	7	20	7	0	16	–	10	8.8
RAD-4(%)	62.8	86.4	73.3	73.6	77.6	89.5	83.3	57.2	85.9	48.4	86.0	70.8
	95.7	87.6	85.7	94.1	100.0	99.7	78.8	96.4	91.3	84.6	92.0	91.0
KOA-4(°)	28	8	18	13	4	16	–	33	–	27	14	15
	4	12	13	7	–	24	7	2	6	–	11	8

During the calculation of open angles, the absence of certain joints may impact the accuracy of the opening angle calculation. This suggests the absence of significant knee joint movement and conspicuous torso position actions in the test video clips, resulting in a lack of joint information throughout the entire duration. Therefore, we record them as 0.

As shown in Table 5, the variation in the knee opening angles of the controls is significantly smaller than that of the subjects in both tasks. Meanwhile, the subjects have a higher action frequency. At the same time, the action range of all participants on Task 4 is larger than the action range on Task 1, and the performance of subjects is more obvious.

Fig. 9 shows the boxplot of average knee open angles with gender.

Fig. 9.

Boxplot of average knee open angles with gender in Task 1 and Task 4. Control_M represents the male control group, Control_F represents the female control group, Subject_M represents the male subject group, and Subject_F represents the female subject group.

As shown in Fig. 9, the results indicate that the range of action in the knee joint of male subjects is more strenuous than that of female subjects, indicating a greater range of action. Consequently, this suggests that females with ADHD are more susceptible to minor torso fidgeting or slight hand movements, aligning with our observations. And the red dot in Fig. 9 (b), indicates that Subject 8 (female) shows more serve actions than other female subjects during the test.

To verify the robustness of the designed detection results, we calculate the R_F results for Task 1 and Task 4, respectively, and classify them using different thresholds. The experimental results indicate that when using the mean value of the Task 1 experimental results as the threshold for classification, the accuracy is 75.0%. Similarly, when using the mean value of the Task 4 experimental results as the threshold for classification, the accuracy is 82.4%. Employing the mean value of the experimental results from the entire video as the threshold, the classification accuracies for Task 1 and Task 4 are 75.0% and 77.0%, respectively. These results suggest that, even with part of the tasks, the system still exhibits the ability to detect and classify ADHD, demonstrating a certain level of robustness and anti-interference capability. However, the system's classification performance is more effective on videos of longer duration. This observation may be attributed to the fact that the attention deficit in ADHD patients is influenced by the duration of the test, making their action-related characteristics more pronounced over an extended period.

Compared with traditional machine learning ADHD detection methods, our system exhibits higher accuracy at a lower cost and broader applicability. However, it is important to note that in the detection results of ADHD in females, the accuracy is relatively lower. This may be attributed to the behavioral differences between males and females, with females manifesting prolonged small actions that are easy to ignore. Moreover, the performance of our system on videos of shorter duration is not as good as that of long-lasting videos. Our future work will be more focused on solving these issues.

4. Conclusion and future work

In this paper, we propose a two-stage action analysis system for identifying and diagnosing ADHD from raw videos. Our ADHD detection method introduces novel measurements based on action recognition results in a real multimodal dataset. The final ADHD decision is derived from the late fusion results. In the ADHD analysis stage, we summarize and analyze gender- and time-related action characteristics of ADHD through joint-level calculations. Experimental results demonstrate that our system outperforms state-of-the-art methods in terms of F1 score, accuracy, and AUC. Compared to conventional EEG and fMRI-based techniques, our system is cost-effective, highlighting its potential for clinical practice. The collected data and results can be shared with doctors to support their diagnosis and follow-up procedures. For future work, we plan to expand the dataset to better represent real-world patient distributions and record more multimodal data, such as EEG and fMRI, for comprehensive fusion and evaluation. Additionally, we aim to develop new attention test tasks focusing on relevant features, including combined audio-visual inputs.

Declaration of competing interest

The authors disclosed no relevant relationships.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.