A Hybrid Approach to Attention Deficit Hyperactivity Disorder Detection Leveraging Transformer and XGBoost Models Using XSparseFormerNet

Abstract

Early detection of neurodevelopmental disorders such as attention-deficit/hyperactivity disorder (ADHD) is crucial for improved outcomes and prompt intervention. However, traditional detection methods often suffer from challenges due to subjectivity and misinterpretation, lack of resources, and diagnostic biases that can lead to underdiagnosis or overdiagnosis. Early detection of these neurodevelopmental disorders not only helps individuals obtain proper ministrations but can also improve their social, cognitive, and mental development. This study proposes an ensemble model—XSparseFormerNet—that leverages, a custom encoder-decoder Transformer enhanced with various attention mechanisms alongside an XGBoost model to enhance diagnostic accuracy. The proposed model aims to increase the accuracy and efficiency of diagnosis via a custom transformer architecture with the gradient boosting algorithm XGBoost. By using a preprocessed EEG dataset and a customized ensemble model, XSparseFormerNet achieves 85% accuracy, outperforming traditional methods across evaluation metrics. Moreover, this research contributes to future development in the field by offering methodologies that can be useful for studies on disorder detection.

Introduction

Attention-deficit hyperactivity disorder (ADHD), recognized as a neurobehavioral condition, manifests during the early stages of human development. ADHD comprises a collection of symptoms, which are characterized by common behavioral traits associated with a range of mental, intellectual, and learning disabilities. A thorough clinical assessment examining symptoms and their impact on daily functioning is essential for diagnosing ADHD. This complexity affects research on its neurological characteristics and origins. Furthermore, identifying ADHD becomes even more challenging as more than 60% of patients with ADHD concurrently exhibit at least one other mental disease, including disruptive behavior disorders, anxiety, or despair¹. ADHD may arise from various factors, including both environmental and genetic influences. ADHD is among the most heritable psychiatric disorders owing to its significant genetic component. Research reveals that specific relatives exhibit a markedly elevated likelihood of having this disorder, as the condition tends to be familial.

The impact of ADHD spans across age groups and multiple aspects of life. To comprehensively diagnose the disorder, four supplementary criteria must be satisfied: Symptoms must have commenced before the age of 7². ADHD-related challenges differ by age; for example, children are at an elevated risk of subpar academic performance, diminished educational attainment, retention, and disciplinary actions such as suspensions or expulsions. ADHD in adolescents may result in detrimental peer and familial connections, school dropout, hostility, behavioral issues, delinquency, and premature substance use. In adults, ADHD is associated with vehicular accidents, interpersonal conflicts, marital challenges, and occupational obstacles. The influence of ADHD permeates multiple facets of life, including domestic environments, professional settings, interpersonal connections, financial matters, and leisure pursuits. Given that the majority of ADHD referrals pertain to school-age children, early identification is crucial². Detecting ADHD in a timely manner can provide prompt intervention, thereby markedly improving quality of life. Symptom control can be achieved effectively through pharmacotherapy and psychotherapy³.

Electroencephalography (EEG) has been used for ADHD research for more than 75 years because of its affordability and ability to measure brain activity. However, its diagnostic value remains debated due to methodological limitations. EEG recordings use electrode placements that follow the 10-20 system to capture brain activity patterns associated with ADHD. Traditional models such as Support Vector Machines (SVM), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Transformer-based architectures can detect subtle differences in brain activity. CNNs process neuroimaging data effectively, whereas RNNs and LSTMs capture temporal dependencies efficiently. Time-series models such as transformer models extract meaningful features from large datasets because they are sequence-based models, improving accuracy and generalizability. The transformer can work through the patterns and relations between the EEG signals and recognize the similarity between each subpart. These AI-driven approaches present a scalable and objective alternative to traditional diagnostic methods, enhancing early detection and treatment of ADHD.

We selected Transformer-based models, specifically the encoder-decoder architecture, for ADHD detection from EEG signals due to the model’s ability to capture long-range temporal dependencies and complex spatial-temporal interactions compared to traditional sequence models such as RNNs and LSTMs. RNNs and LSTMs, though effective in handling sequential data, process inputs step by step, which limits their efficiency and makes them prone to vanishing gradients, thereby constraining their ability to capture dependencies across long time spans. EEG signals are inherently sequential, high-dimensional, and often contain patterns distributed across extended time windows and multiple frequency bands. Transformer architectures employ self-attention mechanisms that allow the model to attend to all time steps simultaneously, enabling the extraction of both local and global relationships in the EEG sequence. The encoder component efficiently learns compact temporal-spatial representations, while the decoder utilizes these embeddings to enhance classification performance, offering robustness against noise and variability in EEG signals. This parallel processing capability, combined with the ability to focus adaptively on relevant signal segments, makes Transformers particularly well-suited for ADHD detection where subtle neural markers may emerge over longer temporal ranges and across different brain regions.

Recent works⁴ in EEG-based analysis demonstrate how Transformer models outperform RNNs and LSTMs in capturing long-range dependencies and improving classification accuracy in neurological detection tasks. Transformers work well on long-range dependencies by using self-attention methods. By incorporating different attention modules, the transformer architecture can be more efficient and work better on the possible sequence between the subparts. The integration of XGBoost with the Transformer in XSparseFormerNet enhances performance by combining two complementary modes of learning. XGBoost excels at handling structured, tabular features and applies boosting with strong regularization, which stabilizes learning and reduces overfitting. It follows a supervised learning paradigm that builds an ensemble of decision trees through gradient boosting, adding stability by focusing on clear and important features. In contrast, Transformers are specialized for sequential and high-dimensional data, enabling the discovery of hidden dependencies and long-range temporal–spatial patterns within EEG signals. It act as powerful sequence models, capable of capturing complex relationships and long-range dependencies, thereby adding flexibility and representational depth.

By ensembling their outputs, XSparseFormerNet balances the strengths of both approaches. Errors made by the Transformer in noisy sequential data can be corrected by XGBoost’s structured focus, while limitations of XGBoost in modeling subtle sequential dependencies are complemented by the Transformer. This synergy improves generalization, produces more calibrated outputs, and reduces variance.

Our research aims to understand the role of brain signals in identifying children with ADHD and develop an efficient technique for correctly detecting neurobehavioral conditions. The objective is to provide a more precise and effective technique for detecting and identifying these illnesses.

1. The proposed model, XSparseFormerNet, ensembles XGBoost with a customized encoder-decoder Transformer, combining gradient-boosted trees and advanced attention mechanisms for robust and accurate ADHD classification.

2. A preprocessed EEG dataset⁵ was used, ensuring clean and structured data for model training and analysis.

3. The Firefly Algorithm was applied to select the most relevant features, optimizing the input data and enhancing model performance.

4. Statistical and signal-based feature extraction methods were performed on the selected features to construct a comprehensive feature vector for training the XGBoost model.

5. The dataset was also processed through the customized encoder-decoder Transformer, incorporating Sparse Attention and CBAM modules to better capture complex relationships within the data.

6. Predictions from XGBoost and the Transformer were ensembled, integrating their strengths to improve overall accuracy and robustness.

The rest of this paper covers related work, the proposed models and attention modules, the experimental setup, the results and discussion, and the conclusions.

Related works

There are several related works on the detection of ADHD via various machine learning and deep learning models to obtain an understanding of these areas-

The author of this paper⁶ examines the potential of utilizing the resting state of Functional Magnetic Resonance Imaging (fMRI) for the automated classification of ADHD. Training a PCA-LDA classifier with feature vectors. The use of a brain mask and 3-cycle map features yields a classification accuracy of 69.59%.

The ADHD/CD-NET architecture relies on EEG signals and is capable of performing objective classification of three groups–CD, ADHD + CD, and ADHD. The model achieved a classification accuracy of 93.70%, specificity of 95.35%, and model sensitivity and precision of 90.83% and 91.85%, respectively⁷.

Paper⁸ presents SSRepL-ADHD, a novel model hat combines self-supervised representation learning and transfer learning with LSTM and GRU layers for ADHD classification using EEG data. Three models were tested: RF, LSSRepL-DNN, and SSRepL-ADHD. SSRepL-ADHD achieved the highest accuracy (81.11%), outperforming RF (78.01%) and LSSRepL-DNN (73.67%). The results highlight the model’s effectiveness in capturing temporal EEG patterns for ADHD detection.

This paper⁹ utilized EEG recordings for ADHD detection with CNN and RNN architectures. The models achieved promising results, with classification accuracy reaching up to 98.3%, further confirming EEG as a robust modality for ADHD diagnosis.

This study¹⁰ applied EEG data for ADHD classification using deep learning models such as CNN and LSTM. The proposed approach achieved a classification accuracy of around 96.64%, demonstrating the strong potential of EEG signals for reliable ADHD detection.

This paper¹¹ explores ADHD detection using social media text (Reddit: 12,070 ADHD + 12,070 controls; Twitter: 1000 ADHD + 1000 controls) with ML, BiLSTM, and transformer models.The proposed BiLSTM and MentalRoBERTa achieved the best in-domain performance (F1 = 0.77), while Gradient Boosting generalized best out-of-domain (F1 = 0.70).The study highlights the trade-off between accuracy and interpretability, identifying linguistic features as potential digital biomarkers for ADHD.

This systematic review analyzes 342 studies on AI in ADHD. Results show the U.S., China, and England lead research, with hotspots in diagnosis, neural networks, and classification models using MRI, EEG, and behavioral data. AI demonstrates high diagnostic accuracy (>90%) but requires larger clinical validation and standardized methods.¹²

This paper¹³ proposes a multimodal ADHD detection system combining EEG and diagnostic data with NeuroDCT-ICA preprocessing, Rhinofish Optimization for feature selection, and ADHD-AttentionNet for classification. It achieved 98.52% accuracy on diagnostic data and 97.89% on EEG, outperforming existing models with higher precision, lower false positives, and faster, efficient processing.

This author¹⁴ introduces a video-based ADHD detection system using human action recognition instead of costly fMRI or EEG. By analyzing behaviors like fidgeting from RGB and skeletal data, it achieved 95.5% accuracy and 99.2% AUC, showing a scalable, cost-effective, and non-invasive tool for remote ADHD screening.

Table 1 shows an overview of related studies that have been employed for the detection of ADHD.

Table 1.

Overview of Models Utilized in Reference Papers.

Related researches	Dataset used	Model used	Accuracy / performance
ADHD detection by resting-state fMRI6	Resting-state fMRI	PCA-LDA (brain mask + 3-cycle map features)	69.59%
ADHD/CD-NET for EEG classification7	EEG	ADHD/CD-NET	93.70% (ACC), 95.35% (SPEC), 90.83% (SEN), 91.85% (PREC)
SSRepL-ADHD with self-supervised learning8	EEG	RF, LSSRepL-DNN, SSRepL-ADHD	73.67% (LSSRepL-DNN), 78.01% (RF), 81.11% (SSRepL)
EEG with CNN and RNN9	EEG	CNN, RNN	98.3%
EEG with deep learning10	EEG	CNN, LSTM	96.64%
ADHD detection via social media text11	Reddit (24,140 users), Twitter (2000 users)	ML, BiLSTM, transformer (MentalRoBERTa)	F1 = 0.77 (BiLSTM, MentalRoBERTa in-domain), F1 = 0.70 (GB out-of-domain)
Systematic review of AI in ADHD12	MRI, EEG, behavioral data (342 studies)	Various AI/ML and DL models	Accuracy > 90% (reported across studies)
Multimodal ADHD detection with AttentionNet13	EEG + diagnostic data	NeuroDCT-ICA + Rhinofish optimization + ADHD-AttentionNet	98.52% (diagnostic), 97.89% (EEG)
Video-based ADHD detection14	RGB + skeletal video data	Human action recognition	95.5% (ACC), 99.2% (AUC)

As shown here, most of the related studies for the detection of ADHD have focused on machine learning models or deep learning models. It can be detected well through brain signal data. Recent advancements in neuroimaging and electrophysiological techniques, such as EEG (electroencephalogram), have provided insights into the neural mechanisms of ADHD.

Limitations of the related works

Many studies on ADHD detection have limitations that impact their reliability and generalizability. One of the common issues is the use of small datasets, which increases the risk of overfitting and reduces the model’s ability to perform well on unseen data. EEG signals are highly variable and often noisy; however, many studies do not fully address preprocessing techniques to manage these inconsistencies. While deep learning models have always had high accuracy, their decision-making process is often not explainable.

Another major challenge is the computational complexity of certain models, which reduces their efficiency for real-time ADHD detection. Some studies also indicate that increasing model complexity, such as using deeper networks such as DeepConvNet, does not always improve performance, suggesting that more sophisticated architectures do not necessarily lead to better results. Certain models focus exclusively on functional brain connectivity without considering structural connectivity, which could provide deeper insights.

Therefore, we have integrated a hybrid model that combines a custom-made Transformer model and XGBoost named “XSparseFormerNet,” which has shown promising results. By integrating different models, the strength of each can be leveraged, leading to more robust and accurate classification performance.

Preliminary studies of applied models

In this section, we discuss the architecture applied in our study: Transformer and XGBoost. These models were selected on the basis of their effectiveness in handling complex patterns in EEG data for ADHD classification. Here, we provide an overview of both models and their relevance to our study.

Transformer architecture

The Transformer, introduced by Vaswani et al. in 2017¹⁵, revolutionized NLP by using self-attention for parallel data processing, fully connected layers in the encoder, and layered self-attention in the decoder.

• Attention function: maps queries and key-value pairs to outputs by weighting values according to query–key similarity.

• Scaled dot-product attention: computes query–key dot products, scales by , and applies Softmax:

  
• Multi-head attention: runs multiple attention functions in parallel, concatenates results, and projects:

  

  where .
• Feed-forward networks: position-wise fully connected layers with ReLU:

  

• Embeddings: map tokens into -dimensional vectors.

• Positional encoding: sinusoidal functions encode token positions, enabling generalization to unseen sequence lengths.

XGBoost algorithm

XGBoost¹⁶ is a gradient boosting method that builds decision trees sequentially, each correcting the errors of the previous one. It optimizes predictions using gradients and Hessians and accelerates training with parallel GPU processing.

Overview of attention module

In this section, we discuss the Attention Module, also known as the attention mechanism, which is used to determine the relative importance of each component in a sequence. For our proposed model, we have incorporated two different attention modules: CBAM (Convolutional Block Attention Module) and Sparse Attention.

Convolutional block attention module (CBAM)

The Convolutional Block Attention Module (CBAM)¹⁷ produces a 1D channel attention map and a 2D spatial attention map from a feature map. It consists of two components: the Channel Attention Module (CAM) and the Spatial Attention Module (SAM).

Channel attention module (CAM) highlights important feature channels using inter-channel interactions and average pooling:

1

where is the input feature map, is the channel attention map.

Spatial attention module (SAM) generates spatial context descriptors from :

2

where is the channel-refined feature map, is the spatial attention map.

Sparse attention module

Sparse attention¹⁸ refines the transformer by focusing on the most relevant elements, improving efficiency on large datasets.

• Factorized attention heads: clusters attention heads to enhance pattern recognition and computation.

• Scaling to many layers: uses specialized embeddings for various data types (text, audio, images).

• Handling different data types: optimizes memory by recomputing intermediate steps instead of storing all.

• Effective sparse attention: focuses on key input regions via block-sparse methods.

• Saving memory: employs mixed-precision training to reduce memory and computation while maintaining accuracy.

Methodology

We propose a model named “XSparseFormerNet” which is a customized transformer incorporating multi-head self-attention, sparse attention, cross-attention, and CBAM with fusion with the Gradient Boosting Algorithm: XGBoost. Owning to the transformer’s ability to recognize the sequence and pattern within input features and XGBoost’s role as a regularizer, by incorporating both output probabilities together, the performance of the model is enhanced.

The EEG dataset undergoes preprocessing and feature selection via the Firefly Algorithm, which reduces the dimensionality to 12 relevant electrodes. The 12 features are applied to extract features through statistical and signal features. After the features are extracted, a feature vector is formed, which trains the XGBoost model via the XGBClassifier. Batch processing prevents memory overhead during prediction.

For SparseFormerNet, the EEG dataset is applied, and the model embeds input data for pattern learning. The embedded layer helps balance computational efficiency. Positional encoding ensures the input sequence order and improves the model’s comprehension. The Convolutional Block Attention Module (CBAM) uses Channel Attention and Spatial Attention to enhance the model’s functionality. CBAM and Sparse Attention enhance the model by going beyond standard self-attention, which treats all tokens more uniformly. Sparse Attention improves efficiency and robustness by selectively focusing only on the most informative temporal and spatial regions in EEG sequences, reducing noise and computational overhead. CBAM complements this by applying channel attention and spatial attention. Together, these mechanisms allow the model to filter out irrelevant or noisy features and amplify discriminative signal patterns.

The encoder layer consists of six layers, each with a position-wise feed-forward network and multi-head attention. The decoder layer processes the output from the encoder and outputs prediction, using multi-head self-attention, sparse attention, and cross-attention mechanisms. The output tokens are transformed into vectors, and a Softmax activation function is applied to convert them into probability distributions.

The top overview of our proposed model XSparseFormerNet presented in Fig. 1 below.

Fig. 1.

Top overview of proposed model: XSparseFormerNet.

Ensembling XGBoost and Transformer models improves accuracy by combining XGBoost’s structured data efficiency with the Transformer’s ability to capture complex patterns. The outputs are averaged post-sigmoid transformation for binary classification.

Overview of the dataset

Our research used an open access dataset⁵, and 61 children with ADHD (48 boys and 13 girls, with an average age of 9.62 ± 1.75 years) and 60 healthy children (50 boys and 10 girls, average age of 9.85 ± 1.77 years) were included. Children with ADHD were diagnosed by a seasoned child and adolescent psychiatrist in accordance with the DSM-IV criteria¹⁹.

The EEG recording procedure was structured around a visual attention test for young individuals. The children were asked to count the characters in 20 photos, each containing between five to sixteen randomly chosen characters that were large enough to be easily viewed²⁰. For EEG recording, 19 electrodes (Fz, Cz, Pz, C3, T3, C4, T4, Fp1, Fp2, F3, F4, F7, F8, P3, P4, T5, T6, O1, O2) were placed on the scalp according to the 10–20 system. The placement locations of the electrodes are given in Table 2.

Table 2.

Electrode placement locations for EEG recording.

Electrode	Placement location
Fz	Frontal midline
F7	Left frontal midline
F8	Right frontal midline
Fp1	Left frontal pole
Fp2	Right frontal pole
F3	Left frontal region
F4	Right frontal region
Cz	Central midline
C4	Right central midline
C3	Left central midline
T3	Left temporal lobe
T4	Right temporal lobe
T6	Right posterior temporal region
T5	Left posterior temporal region
P3	Left parietal lobe
Pz	Parietal midline
P4	Right parietal lobe
O1	Left occipital region
O2	Right occipital region

To maintain consistent stimulation, each image appeared immediately after the child responded during the EEG recording. Since no rewards were given and the responses were not checked for accuracy, the duration of the EEG session was determined by the child’s response time²⁰.

In Fig. 2, most features appear to have higher concentration values around 0 (Normal), which indicates a highly skewed or centered distribution. There are also some features, such as Cz and C3, that have bipolar variation in the signals.

Fig. 2.

Distribution of features in the EEG dataset.

Data pre-processing

The EEG data comprised approximately 20,000-30,000 data points for each participant. To create a sustainable and representative dataset, we have selected all 120 participants, which included 61 children diagnosed with ADHD and 60 control children. After selecting all of the participants from the ADHD and Control groups, we merged the data points while ensuring that the ADHD and Control groups had equal numbers of data points.

Following the merging of two different datasets, a new target column named “Target” was created to distinguish between ADHD participants and control participants, with a value of 1 indicating the ADHD group and 0 indicating control children. The data points of both the ADHD children and the control children were combined into a single dataset. The final dataset contains approximately 3,973,303 rows and 20 columns, where 19 columns present the EEG electrode points and the “Target” column is represents binary classification.

Figure 3 3D visualization of feature distribution based on principal component analysis (PCA) for ADHD and control groups. The plot highlights the separation between the two classes, indicating potential discriminative power of the extracted features.

Fig. 3.

Feature distribution in 3D space for ADHD classification.

Data augmentation

We scale the pre-processed dataset to ensure that all features can equally contribute to the models. We utilized standard scaling to obtain a mean of zero and a variance of one, resulting in a balanced dataset.

We flipped the scaled data to normalize the data distribution and reduce any biases.

Due to being preprocessed, our dataset did not contain any null or zero values.

Figure 4 shows that some features have strong correlations, whereas a few have weak or almost no correlation between features in the dataset.

Fig. 4.

Correlation heatmap visualization for ADHD classification.

Feature selection process: firefly algorithm

The Firefly Algorithm²¹, inspired by fireflies’ flashing behavior, optimizes solutions by evaluating and moving toward brighter fireflies. Inspired by the natural flashing behavior of fireflies, the Firefly Algorithm greatly improves the feature selection process. Each firefly in this algorithm represents a subset of possible features, and the flash’s brightness indicates the subset’s quality or fitness. Others are drawn to brighter fireflies, which directs the search for more potent answers. This procedure guarantees that the algorithm systematically finds the most informative brain signal features while organically eliminating noisy or irrelevant ones that might impair model performance when used for EEG-based ADHD detection.

Algorithm 1.

Firefly algorithm for feature selection.

Applied to the pre-processed EEG dataset, the Firefly Algorithm selected 12 relevant features: Fz, F7, Fp1, Fp2, Cz, T4, T6, T5, P4, Pz, O1, and O2.

Figure 5 shows O2, T6, and Pz as the most important features.

Fig. 5.

Distribution of selected feature importance.

Feature extraction: statistical feature extraction & signal feature extraction

The process of data analysis requires feature extraction. To select features that are more suited for modeling, raw data must be chosen and the original features must be transformed into a new set of features. Feature extraction is crucial for several reasons, such as reduced computational cost, improved model performance, better insights, and overfitting prevention.

We have applied two feature extraction methods, statistical feature extraction and signal feature extraction.

Statistical Feature Extraction measures elucidate the statistical properties of EEG signals, revealing their distribution and variability. The mean, variance, skewness, and kurtosis are descriptive statistics that provide insights into the central tendency, dispersion, and distribution structure of the EEG signal.

Signal Feature Extraction is an important step in analyzing EEG data for ADHD classification. It involves transforming raw EEG signals into meaningful numerical representations. The mean frequency, peak frequency, and power spectral density are used to obtain representations of the EEG signals.

Figure 6 reveals strong positive correlations between variance and standard deviation, power density, peak frequency, mean, and other features, suggesting that these factors may contribute independently to classification tasks. However, weaker correlations indicate that these features may contribute unique information to the model.

Fig. 6.

Correlation matrix of extracted features.

Gradient boosting with XGBoost

The extracted features were merged to create a feature vector, which was applied to the XGBoost algorithm.

The XGBoost model is implemented via the XGBClassifier from the xgboost library. It is useful because it supports parallelism, handles missing values, and can prevent overfitting by using regularization.

After the model was trained, predictions were made via a custom batch processing function. The function handles large datasets by breaking them into smaller batches for prediction to avoid memory overhead and overfitting.

Customized transformer model: SparseFormerNet

Our customized model, “SparseFormerNet” contains an encoder- decoder transformer architecture incorporating various attention mechanisms/modules, including multi-head self-attention, sparse attention, cross-attention, and Convolutional Block Attention Module (CBAM). The input file consisting of the ADHD/Control children dataset is applied to our model, and binary classification is used to predict ADHD/Control children.

The workflow of SparseFormerNet is shown in Fig. 7 which illustrates the overall structure of the custom model.

Fig. 7.

SparseFormerNet workflow.

The pre-processed dataset was again applied to the transformer architecture.

The input data are transformed via an embedded layer. By converting the raw input data into a dense vector, the embedding dimension balances our model’s computational efficiency. By using embedding dimensions, it allows our model to transform raw features into a format that helps to learn the patterns better and to use the complex features without overwhelming the computational resources.

Our model’s comprehension of the input sequence order is ensured via positional encoding. To ensure the input sequence and to inject information about the feature’s position within the sequence, it is added to the embedding layer. The encoding is based on sinusoidal functions that have the same dimension as the embedding dimension, and the dropout rate is 0.1 to avoid overfitting.

Convolutional block attention module (CBAM): the CBAM module uses both channel attention and spatial attention. First, it is applied to the channel gate and then to the spatial gate.

The Channel Gate includes global average pooling, a dense layer followed by ReLU activation, and sigmoid activation for scaling between and to enhance the channel features. The input feature comprises batch, channel, and sequence length dimensions, and the average pooled output is passed through the MLP, which results in channel attention that forms the output feature. The Spatial Gate employs a Conv1D layer to capture the channel information; it performs a forward pass to eliminate the sequence dimension, and then channel aggregation returns two vectors. The model subsequently applies a Conv1D layer to the combination of these two vectors, and by using the Sigmoid function and attention maps, enhances its functionality to manage complex inputs effectively.

Encoder layer: the Encoder is incorporated with self-attention and feedforward components. The proposed model consists of six encoder layers, and each of the six encoder layers in the suggested model incorporates a position-wise feed-forward network and multi-head attention.

The encoder layers rely on a Multi-Head Self-Attention technique to capture dependencies among various components of the input sequence or tokens.

Sparse attention is used to reduce the computational error. It reduces the error without affecting the performance of the model. Sparse attention focuses only on the subset of the relevant parts of the input sequence and uses it to handle long sequences such as EEG data.

In the encoder layer, CBAM is applied to channel the attention to specific features to improve the numbers of tokens. Channel attention focuses on the representation of each of these tokens, whereas spatial attention focuses on the sequence during processing. CBAM extracts important features to enhance the performance of the model.

In addition to the attention mechanism, the encoder layer incorporates a position-wise feed-forward network featuring a hidden dimension of 1024. This has two linear transformations interspersed with a ReLU activation function and a 0.3 dropout rate to prevent overfitting and to introduce non-linearity.

Decoder layer: the decoder, similar to the Encoder, comprises six layers. Each layer processes the output from the encoder in conjunction with the output prediction.

The decoder uses the multi-head self-attention mechanism, which helps to consider different aspects of the output sequence while focusing on the previous token and prediction (the Target column). The decoder uses a masked multi-head self-attention mechanism to prevent seeing future tokens and inaccurate predictions.

The decoder is enabled to utilize the encoded representation or encoder output of the input sequence by using a cross-attention mechanism. It receives information from the encoder layer and integrates it with the intricacy of the Multi-Head Self-Attention. This mechanism thus enables the decoder to focus on the relevant components of the input sequence present in the target sequences in order to enhance accuracy. Channel attention is a mechanism that helps to pick out the needed features while blocking the remaining channels from affecting the process. Spatial attention is used to orient towards certain areas of the sequence, ensuring that the correct information is extracted from the appropriate areas; thus, better results are achieved.

Each decoder layer contains a Point-Wise Feed-Forward Network that is applied independently to each point within the sequence. This has two lines later with a ReLU activation, which helps to add nonlinearity to the transformer model. The feed-forward network has 32 dimensions, and it ensures that the model has the ability to capture the complex transformations.

After the decoder layer, the output is produced as a set of vectors, which represent a token in each output sequence. The output tokens from the decoder layers are transformed into a linear layer, which projects the tokens into a target space. After the linear layer, a softmax activation function is applied to take the output tokens and convert them to a probability distribution over all the possible tokens for each position in the output sequence.

Fusion mechanism

The performance of our model is enhanced through the ensembling of both XGBoost and transformer models. First, the sigmoid function is applied to convert logits into probabilities, and then the predictions of both outputs are averaged. The averaged prediction is then transformed into a binary output. As XGBoost is effective at handling structured data and the Transformer model works better to at capturing complex patterns between data, ensembling both outputs yields better results.

Experimental setup

In this section, the hardware and software configurations are shown in tabular form.

The experiments were conducted on a high-performance computing setup, detailed in Table 3.

Table 3.

Hardware configuration.

Component	Specification
Motherboard	MSI MAG X570S TOMAHAWK MAX WIFI
Processor	AMD RYZEN 9 5950X
GPU	RTX 3080 Ti, 12GB
RAM	64 GB
Storage	1TB SSD, 1TB HDD

We have trained other traditional models like Multi-layer Perceptron (MLP), Recurrent Neural Network (RNN), Deep Belief Network (DBN), and Long Short-Term Memory (LSTM) to compare the performance of our model XSparseFormerNet.

The experiments were performed via Python with libraries such as TensorFlow, Keras, PyTorch, Scikit-learn, Pandas, and Numpy. The dataset was preprocessed, standardized where necessary, and divided into training and test sets.

The experimental setup details for each model are given in Table 4.

Table 4.

Experimental setup for reference models.

Model	Layers/neurons	Optimizer	Loss function	Epochs
MLP	1 hidden (30 neurons)	SGD	Cross-entropy	20
RNN	RNN layer (100 units)	SGD	Cross-entropy	20
DBN	Dense (512, 256, 128, 64)	SGD	–	20
LSTM	LSTM (100 units)	Adam	Binary cross-entropy	20

Each model was trained for 20 epochs, and the validation accuracy was checked at each epoch. The MLP and RNN models use stochastic gradient descent (SGD) as the optimizer, whereas the LSTM model employed the Adam optimizer. The DBN was trained via a layer-wise training approach with multiple dense layers. After training, the models were evaluated on a separate test set, and their predictions were analyzed via classification reports and confusion matrices.

The configurations of the customized model SparseFormerNet are shown in Table 5.

Table 5.

SparseFormerNet experimental setup.

Component	Configuration
Embedding dimension	Adaptive to input size
Dropout rate	0.3
Encoder and decoder layers	6 layers
Multi-head self-attention heads	8
Feed-forward network (encoder)	Hidden dimension: 1024, ReLU activation
Feed-forward network (decoder)	Hidden dimension: 32, ReLU activation
Epochs	30
Gate channels	128

The model was trained in 30 epochs, and the validation accuracy was checked at each epoch. The model was trained for more epochs because of its higher architectural complexity and capacity to learn deeper representations from the EEG data. The training and testing results were evaluated and analyzed, and their predictions were analyzed via classification reports and confusion matrices.

Result analysis

In this section, we discuss the performance of all the applied models and compare and analyze it with the performance of XSparseFormerNet.

Our research focuses on the early detection of ADHD, using a fusion model combining a custom Transformer with various attention mechanisms and an XGBoost model.

To compare the performance of our model, we applied the dataset to traditional machine learning and Deep Learning Models- Multilayer Perceptrons (MLP), Recurrent Neural Network (RNN), Deep Belief Network (DBN), Long Short-Term Memory (LSTM).

Below is a comparison of the applied model along with its performance metrics in table 6.

Table 6.

Performance matrices comparison of different models.

Model	Accuracy (%)	Precision	Recall	F1-score	Training time (s)	Testing time (s)
MLP22	76	0.77	0.76	0.76	2671.71	14.70
RNN23	78	0.78	0.77	0.77	1729.79	15.196
DBN24	81	0.82	0.82	0.81	3960.02	57.64
LSTM25	80	0.80	0.80	0.80	2477.62	53.56
XSparseFormerNet	85	0.85	0.85	0.85	14753.26	39.58

The barchart in Fig. 8 shows the differences among all the model performance matrices.

Fig. 8.

Performance matrices comparison using barchart.

Figure 8 presents a comparative analysis of the all applied models across the four evaluation metrics. Among the models, XSparseFormerNet outperforms the others in all the metrics, by achieving the highest recall and F1-score, suggesting its effectiveness in handling complex patterns of the data.

Figure 9 shows the confusion matrices of the applied models.

Fig. 9.

Comparison of model performance through confusion matrices.

A confusion matrix shows four different predictions: correct predictions (true positives and true negatives) and incorrect predictions (false positives and false negatives). With respect to ADHD detection, DBN performs better than the other methods do (highest true positives).

RNNs have more false negatives but do the greatest job of correctly identifying control instances. The MLP outperforms the DBN and RNN in terms of total right predictions, but it has the fewest wrong classifications overall.

Although LSTM performs admirably, it misclassifies more often than do the DBN and RNN. The Ensemble Model, XSparseFormerNet, appears to perform well in accurately identifying class 1 lesions on the basis of its large number of true positives. To assess the trade-offs, it is necessary to compare the false positives and false negatives with those of other models.

The prediction probability of the ensemble model as follows (Fig. 10)

Fig. 10.

Prediction probability of the ensemble model.

The distribution of the predicted probabilities indicates that the model produces a calibrated and interpretable output. As shown in Figure 10, the probabilities are distributed across the 0–1 range which allows better flexibility in tuning. Therefore, this model was chosen for its balance of accuracy, generalization ability, and probabilistic interpretability.

The precision of the model is illustrated in Fig. 11. The model demonstrates high precision, correctly identifying most of the positive predictions. Misclassifications are primarily due to [False Positives/False Negatives, whichever is higher], indicating that when the model predicts a positive, it is usually correct. This highlights the model’s reliability in making confident positive predictions.

Fig. 11.

Precision of the ADHD classification model.

The Matthews Correlation Coefficient (MCC), shown in Fig. 12, further quantifies the model’s overall performance by capturing the correlation between predicted and actual classes. With an MCC of 0.57, the model demonstrates balanced and robust classification even under class imbalance, unlike accuracy which could be misleading.

Fig. 12.

Matthews correlation coefficient (MCC) for the ADHD classification model.

Figure 13 shows the accuracy curves of the applied models. The accuracy curve provides insight into the learning process of models and helps to analyze the model performance better.

Fig. 13.

Comparison of model performance through the accuracy curve.

Both recurrent neural network (RNN) and long short-term memory (LSTM) experience overfitting, and the performance gain is smaller. The multilayer perceptrons (MLP) and deep belief network (DBN) achieve better learning, which suggests better generalizability. Recurrent neural network (RNN) and long short-term memory (LSTM) experience fluctuations in validation accuracy, indicating potential overfitting or instability. Recurrent neural network (RNN) and long short-term memory (LSTM) might struggle with sequence learning because of vanishing gradients or improper regularization.

The training accuracy of the XSparseFormerNet curve increases significantly, indicating a smooth and stable learning process. The validation accuracy starts high and remains consistently above the training accuracy, indicating good generalizability. The gap between the validation accuracy and training accuracy is small, suggesting that the model is not overfitting significantly and is learning well.

Ablation studies

To evaluate the performance of individual components within our custom transformer architecture, SparseFormerNet, we trained a series of ablation studies. The model comprises two key modules: the Sparse Attention Module and the Convolutional Block Attention Module (CBAM), which are integrated into both the encoder and decoder layers.

Our model, SparseFormerNet, contains transformer architecture with Sparse Attention Module and Convolutional Block Attention Module (CBAM). These components work differently to optimize the performance of the model. The sparse module reduces computational efficiency by working on certain subsets of the input features, and the Convolutional Block Attention Module works on the complex inputs and effectively manages and enhances the functionality.

To assess the contribution of each component, we trained four model variants by selectively disabling or repositioning CBAM:

• Figure 14a: Full SparseFormerNet, with both Sparse Attention and CBAM in the encoder and decoder.

• Figure 14b: CBAM removed from both the encoder and decoder layers.

• Figure 14c: CBAM retained only in the encoder.

• Figure 14d: CBAM retained only in the decoder.

Fig. 14.

Model performance comparisons via ablation of CBAM in SparseFormerNet.

As illustrated in Fig. 14, all the configurations demonstrate comparable training accuracies, indicating similar learning capacitities across architectures. However, the validation accuracy varies:

• The full model (Fig. 14a) yields the most stable and highest validation accuracy, suggesting robust generalizability.

• Removing CBAM entirely (Fig. 14b) leads to increased fluctuations and a slight drop in validation accuracy.

• When CBAM is applied only in the encoder (Fig. 14c), the performance is better than that in Fig. 14b, but still less stable than that of the full model.

• Retaining CBAM only in the decoder (Fig. 14d) results in performance close to that of the full model, with high validation accuracy and stability.

These results confirm that both Sparse Attention and CBAM contribute to the performance of SparseFormerNet, with CBAM in the decoder having a slightly greater impact than in the encoder. Nevertheless, the full integration of both components leads to the best generalizability and stability, underscoring the effectiveness of our proposed architecture.

Discussion

Early detection is necessary for neurodevelopmental disorders such as ADHD, which have a substantial impact on individuals’ cognitive function. Owing to being a neurological disorder, the best way to detect ADHD is through brain waves such as EEG signals. Many studies have investigated the differences in patterns between ADHD patients and non-ADHD patients. Most of these patterns have been applied to machine learning or deep learning models to differentiate between both patient groups, and most of these results are satisfactory; however, there are also some limitations in previous related works, such as model efficiency, overfitting, and smaller datasets. Using sequence-based models such as transformers helps to recognize the patterns between each raw signal because of their ability to handle long-range dependencies in time-series data.

To achieve better performance and efficiency, we applied two different attention modules: Sparse Attention Module and Convolutional Block Attention Module (CBAM) onto our model SparseFormerNet. Both help to recognize the patterns within the sequence while reducing computational cost, allowing the transformer architecture to work more effectively. By combining XGBoost with SparseFormerNet, we further increased model performance and improved generalizability. Feature selection methods, such as the Firefly algorithm, and feature extraction methods, including statistical and signal feature extraction, were used to obtain informative features effectively. These features were applied to the gradient boosting method XGBoost to act as a regularizer by learning from the extracted features. By ensembling the results from both SparseFormerNet and XGBoost, feature learning and structured decision-making were balanced.

The model’s ability to fully capture variability in larger and more diverse populations is limited by the dataset’s relatively small size, which consists of 60 healthy controls and 61 children with ADHD. Although XSparseFormerNet performed well in this dataset, more research is needed to confirm its scalability and suitability for diverse cohorts. The accuracy of classification may be impacted by the potential noise and signal variability introduced in larger datasets. However, with proper retraining techniques, the ensemble design of XSparseFormerNet–which combines the Transformer’s ability to model intricate temporal dependencies with XGBoost’s efficiency in handling structured features–positions it well for adaptation.

The multi-modal ensemble improves performance due to the combination of two complementary learning methods. XGBoost handles structured features such as numerical EEG-derived features, and its use of boosting and regularization prevents overfitting. In contrast, the Transformer identifies hidden patterns and relationships in sequential data, such as EEG time-series signals. When combined, these approaches complement each other:

1. XGBoost follows a supervised learning paradigm that utilizes gradient boosting to create an ensemble of multiple decision trees, adding stability by focusing on clear, important features.

2. Transformers are neural network architectures that transform an input sequence into an output sequence, adding flexibility by learning complex patterns and long-range dependencies.

By averaging their outputs, errors made by one model are often corrected by the other, reducing overfitting, improving generalization to new data, and producing more balanced predictions. In our results, the ensemble reached 85%, which is higher and more stable than using only XGBoost or only the Transformer.

The incorporation of multiple attention mechanisms in XSparseFormerNet is necessary to enhance feature representation at different levels. Multi-head self-attention provides the base by capturing long-range dependencies across tokens, allowing the model to learn diverse contextual relationships in parallel. However, it alone can attend indiscriminately, introducing noise. Sparse attention selectively focuses on the most informative tokens, improving efficiency while filtering irrelevant information. Cross-attention enables effective fusion when multiple feature sets are involved, aligning representations from different sources to retain meaningful interactions. CBAM refines these representations by applying channel- and spatial-level attention, emphasizing features that may be overlooked by self-attention. Finally, integrating these enriched features with XGBoost leverages decision tree ensembles, which excel at handling nonlinear boundaries and improving robustness, particularly in small or imbalanced datasets.

The ablation study provides valuable insight into the contribution of individual components within the architecture. Its strength lies in isolating the effects of mechanisms such as sparse attention and CBAM, offering direct evidence of their impact on accuracy, precision, recall, and F1-score. However, ablation studies also have limitations, as they may oversimplify interactions between components and their findings can be dataset-dependent. The results showed that CBAM in the decoder layer had the greatest impact on overall performance, underscoring the importance of fine-grained spatial and channel-level feature refinement at the final stage of representation learning. While sparse attention in the encoder plays a critical role in improving efficiency and filtering irrelevant information to produce compact and meaningful features, CBAM in the decoder enhances discriminative power by re-weighting and emphasizing the most relevant features before classification. Together, these findings highlight the complementary nature of the mechanisms: sparse attention strengthens upstream representation learning, while CBAM maximizes downstream decision-making, making their placement within the architecture as important as their inclusion.

Future work

Despite good results, several limitations remain. ADHD is a neurodevelopmental disorder that relies on behavioral assessment, making large-scale dataset collection challenging. Most datasets used in related studies are small, and ethical and privacy concerns restrict public availability. In our study, we used an open-sourced dataset from a published author¹⁹.

Future work could extend XSparseFormerNet beyond ADHD by adapting it for multi-class and multi-label classification, enabling analysis of diverse neurodevelopmental disorders such as Autism Spectrum Disorder (ASD) or dyslexia. Weighted cross-entropy or focal loss could address class imbalance, while hierarchical heads could support subtype prediction.

For multimodal data, EEG features could be combined with fMRI tokens (derived from ROI time-series or connectivity graphs) and behavioral assessments encoded as tabular or sequential tokens. Fusion strategies may include late fusion for stability or cross-attention for richer interactions, with ModDrop ensuring robustness to missing modalities. Domain-adversarial training and harmonization methods could mitigate site and protocol variability, while self-supervised pretraining and few-shot adaptation would improve data efficiency. Evaluation should emphasize macro-level metrics, leave-site-out validation, and interpretability through attention maps or saliency on ROIs and behavioral features, supporting both clinical insight and generalization across populations.

Recent work in dyslexia detection²⁶ highlights the importance of multimodal integration, using MRI, EEG, and handwriting data with hybrid transformer blocks and attention-based fusion to achieve near-perfect accuracy. Their results suggest that extending XSparseFormerNet to ASD or dyslexia will require similar cross-modality encoders (e.g., graph transformers for EEG, vision transformers for MRI, lightweight models for behavioral or handwriting data) and an adaptive fusion mechanism to combine neural and behavioral features. Furthermore, incorporating interpretability methods such as Grad-CAM or LIME will be crucial to ensure clinical usability across disorders.

Conclusion

In essence, the purpose of our research is to enable early detection of ADHD, as individuals with ADHD require specialized care and management. Early diagnosis can significantly benefit these patients by providing timely interventions, potentially mitigating the long-term cognitive, social, and academic challenges associated with the disorder. Traditional detection methods often struggle with early identification, highlighting the need for alternative, more objective approaches. By utilizing EEG data from both ADHD patients and control patients, we aimed to improve the accuracy and reliability of neurodevelopmental disorder detection. Transformers work better in identifying complex patterns within sequential data, such as EEG signals, owing to their ability to analyze patterns and relations efficiently. The adaptability of these methods allows them to handle various types of data while offering reliable, precise, and reproducible analysis for ADHD detection.

Additionally, XGBoost, a powerful gradient boosting technique, performs efficiently on large datasets and excels at structured feature learning. By integrating the transformer model with XGBoost, our approach achieves an accuracy of 85%, demonstrating the value of combining sequential pattern recognition with structured feature-based learning. This research contributes to the advancement of neurodevelopmental disorder detection, ultimately supporting earlier interventions and improving the quality of life of affected individuals. The incorporation of advanced ensemble methods, such as combining XSparseFormerNet with machine learning or deep learning models, may further enhance generalizability, robustness, and stability across diverse populations. Moreover, exploring different feature selection techniques and leveraging larger, more diverse datasets could lead to further performance improvements, paving the way for more accurate, clinically applicable, and scalable neurodevelopmental disorder detection systems.

Author contributions

All authors (S.R.S., S.M., M.M.P., R.R., F.U.I., and M.G.R.A.) contributed equally to the conception, methodology, implementation, data analysis, and writing of the manuscript. All authors reviewed and approved the final version of the manuscript.

Data availability

The EEG dataset used in this study is publicly available from IEEE DataPort and can be accessed via the following DOI: https://dx.doi.org/10.21227/rzfh-zn36 (Motie Nasrabadi et al., 2020).

Declarations

Competing interests

The authors declare no competing interests.