Generalization or mirage? Data leakage and reported performance in neonatal EEG seizure detection models: a systematic review

Abstract

The reliability of deep learning models in neonatal seizure detection depends on rigorous experimental design and transparent reporting of performance. In this systematic review, 48 peer-reviewed studies and 145 distinct study observations of EEG-based neonatal seizure detection were examined to assess how data partitioning (validation) strategies, open data (code), and data leakage influence reported performance. We analyzed the extracted data using Fisher’s exact test, chi-square test, and partial least squares structural equation modeling (PLS-SEM). Of the 48 studies included, public datasets were used in 87.69% of papers, but only 22.9% reported code availability. While studies with low risk of data leakage are common, we found that 16.7% of papers or 31.7% of study observations exhibited a high risk of data leakage. The PLS-SEM results and chi-square test align well, supporting that rigorous data partitioning reduces leakage and that leakage adversely affects performance. Data Partitioning (Validation) strategy emerged as a strong and statistically significant predictor of data leakage (p < 0.001, R² = 0.598), explaining 59.8% of its variance. Data leakage has a statistically significant effect on reported model performance (p = 0.005, R² = 0.202). Still, they account for only about 20.2% of the variation in performance, indicating other factors might play a substantial role, such as dataset size, class balance, model architecture, and preprocessing. Overall, the evidence suggests a generalizable relationship between leakage and performance, though caution is warranted due to variability in accuracy outcomes across studies. These findings highlight the importance of methodological standards to reduce data leakage and promote transparency in neonatal EEG seizure detection models.

Supplementary information

The online version contains supplementary material available at 10.1186/s13040-025-00516-y.

Introduction

Neonatal seizures are “a clinical manifestation of a sudden and abnormal electrical discharge in the immature brain”  [1]. They are often subclinical, characterized by sudden, paroxysmal, and abnormal changes in brain activity  [2–4]. Early detection is crucial to initiate prompt treatment and prevent brain injury or dysfunction. However, identifying neonatal seizures remains a considerable challenge because they are frequently subclinical and difficult to detect reliably  [2–4]. Although continuous electroencephalography (EEG) monitoring is the gold standard  [5], manual interpretation of EEG recordings is time-consuming, inherently subjective, susceptible to human error, relies on specialized expertise, and is often limited in neonatal intensive care units  [6–8].

To address these challenges, recent advancements in artificial intelligence (AI) techniques, especially machine learning (ML) and deep learning (DL), have demonstrated significant progress in automating EEG-based neonatal seizure detection  [9]. These advancements offer transformative solutions, including: (I) Automatic feature extraction that reduces manual feature engineering efforts, minimizes human errors and interobserver variability, limits information loss, and improves pattern recognition  [10–14]; (II) AI applications for detecting and removing artifacts- such as high-amplitude electrical signals from ocular or muscular activity, electromagnetic interference, or inadequate electrode contact- thereby enhancing signal quality and reducing false alarms to increase diagnostic reliability  [12, 15–18]; (III) Improved diagnostic accuracy and efficiency, providing more reliable diagnoses, lowering misdiagnosis rates, and demonstrating superior sensitivity, specificity, and fewer false positives compared to traditional methods- transforming EEG seizure detection from manual to automated systems and thus improving diagnostic precision  [10, 11, 13, 14, 18–21]; (IV) Real-time capabilities that enable low-latency seizure detection, allowing timely clinical interventions in critical situations and preventing severe neurological outcomes; AI supports end-to-end processes with lightweight computational operations suitable for clinical settings requiring immediate processing  [20, 22]; and (V) Addressing variability in patient data and ensuring adaptability to complex data scenarios- including patient-specific differences, data scarcity, imbalance, and heterogeneity- through methods such as transfer learning, few-shot learning, domain adaptation, and data augmentation  [23, 23–27]. Notably, overfitting issues in seizure detection models often arise due to small datasets, where data augmentation helps expand limited training samples  [25, 28]. Nonetheless, the performance of neonatal seizure detection algorithms depends heavily on dataset characteristics, such as sample size, diversity of seizure types, and demographic representation  [10, 29].

In summary, despite the significant progress made with automated seizure detection models in automating labor-intensive tasks  [10, 14, 18], reducing inter-observer variability and misdiagnosis rates to ensure standardized and reliable diagnoses  [10, 11, 16, 17], and utilizing advanced feature extraction techniques with deep learning  [13, 19, 30], which detect and eliminate artifacts from EEG signals to lower false alarms while enhancing the specificity and trustworthiness of seizure detection models  [15, 21], along with capabilities for real-time diagnosis  [22], adaptability to heterogeneity and limited datasets  [10, 23, 24, 26, 27], and overall improved performance  [13, 14, 18, 21, 31–33], there remain key challenges that hinder the broader adoption and integration of AI in epilepsy management. These include issues related to model interpretability  [10, 31], generalizability and robustness in real-world settings  [10, 34], data availability and quality, as well as the integration into clinical workflows and ethical considerations  [10, 35].

Furthermore, on one hand, deep learning has shown state-of-the-art performance in seizure detection, yet it remains under-deployed due to poor real-world performance and generalizability, leaving a persistent gap between algorithmic innovation and clinical implementation. Prior studies showed that misalignment between model training and clinical priorities occurs when models are trained on idealized datasets and fail to reflect the complexity and variability of real-world clinical settings  [10], making practical application of such tools difficult due to infrastructural and staffing limitations  [36]. This may result from overfitting or inappropriate training, where high-performing algorithms do not generalize well to clinical environments, limiting their diagnostic reliability and interpretability. Bridging this gap requires not only rigorous methodology but also a reevaluation of data practices, validation frameworks, and model design choices that emphasize clinical realism and usability. On the other hand, the current neonatal seizure detection model faces challenges with reproducibility, often worsened by subtle methodological choices such as data partitioning, evaluation methods, or transparency in reporting  [37]. Overall, an increasing amount of evidence indicates that problems like data leakage, overfitting, and validation bias may be more widespread than previously recognized. Recent research shows that these performance improvements often conceal deeper issues  [37] like data leakage, where accidental overlap between training and testing data results in overly optimistic outcomes, significantly inflating performance metrics and undermining real-world reliability.

Despite growing awareness of data leakage, poor generalization, and inflated model performance (”mirage metrics”), systematic frameworks to quantify and model the pathways by which data leakage emerges and its impact on model performance remain underdeveloped. Unlike many other domains, the properties of EEG signals, high temporal autocorrelation and significant inter-subject heterogeneity, create a high risk for data leakage, demanding rigorous validation protocols. Without strict subject-level independence in data splits, models can achieve artificially high performance by exploiting non-generalizable features, leading to a severe overestimation of true generalization error. The field of EEG diagnostics thus serves as an exemplary testbed for demonstrating how improper data partitioning inflates performance metrics and undermines the development of reliable machine learning systems. This study aimed to systematically evaluate the relationships between validation strategies (data partitioning), open code (data), data leakage, and reported model performance in EEG-based neonatal seizure detection literature. Specifically, to examine the association between data leakage and reported performance in Neonatal EEG Seizure Detection Models. First, we used the chi-square / Fisher’s exact test to assess the categorical association between validation strategies, open code (data), data leakage, and reported model performance.

Finally, to capture the complex interplay and directional dependencies among these latent constructs, we employed partial least squares structural equation modeling (PLS-SEM). This approach allowed us to simultaneously model Validation Strategies as a formative construct shaped by validation strategies (data partitioning), and to estimate the direct and indirect effects between validation strategy, open data (code), data leakage, and reported performance. Each causal path is supported by literature. For instance, data partitioning (validation) strategies, such as subject-level splitting or leave-one-out validation, prevent data leakage risks by avoiding patient overlap between training and testing sets, which can inflate performance estimates and threaten generalization  [38–41]. Transparent reporting, including open data (code) and validation protocols linked to reproducibility, is crucial; the absence of these elements correlates with undetected leakage and compromised reliability  [42, 43]. Meanwhile, data leakage violates independence assumptions and leads to misleading metrics, causing discrepancies between reported performance and actual generalization  [41, 44]. Through this layered methodology, we aimed to quantify the extent to which upstream design choices influence downstream performance, both statistically and structurally. Overall, as depicted in Fig. 1, the study proposed the following hypotheses to examine the interrelationships between validation strategy, open data (code), data leakage and reported performance: (I) Hypothesis (H) 1: Data leakage negatively impacts reported model performance, (II) H2: Open code (data) negatively affects data leakage, (III) H3: Validation strategy positively affects Open code (data), and (IV) H4: Validation Strategy negatively affects data leakage.

Fig. 1.

Hypothesized relationships in neonatal seizure detection model: SEM analysis of validation strategies, data practices, and performance outcomes

Methods

Approach and study selection

This systematic review followed PRISMA 2020 guidelines  [45] to systematically evaluate the literature on deep learning, machine learning, and transformer models for neonatal EEG seizure detection. We developed a search strategy and conducted comprehensive searches across major electronic databases, including PubMed, IEEE Xplore, Scopus, and Web of Science, from 2015 to 2025, publishing only in English. Details of our search strategy are provided in Appendix A. Following automated deduplication using Covidence  [46], two independent reviewers screened all titles and abstracts against predefined eligibility criteria, resolving conflicts by consensus. Full-text articles of potentially relevant studies were then retrieved and independently assessed for final inclusion, with reasons for exclusion meticulously documented (see Supplementary Information for detailed criteria and PRISMA flow diagram in Appendix B). Excluded studies include: (I) Those that didn’t use EEG as input modality or relied on manual detection; (II) Those focused exclusively on feature engineering or signal denoising without classification; (III) Those that used adult, non-neonatal, and animal EEG; and (IV) Studies published before 2015.

Data extraction and quality assessment

From each included study, two reviewers independently and systematically extracted data based on predefined criteria. Detailed study characteristics were recorded, including: cross-validation strategy (e.g., leave-one-patient-out, leave-one-subject-out, 10-fold, or train–test split), patient-independent splitting (i.e., whether patient-level independence was ensured between training and test sets), data split timing (whether the data split occurred before or after feature extraction or signal processing steps such as segmentation or spectrogram generation), risk of data leakage (classified as low, medium, or high based on splitting strategy and timing), reported performance metrics (e.g., accuracy (ACC) or area under the curve (AUC), and the validation scheme under which they were obtained), code availability (whether the source code was publicly accessible), and additional notes (including relevant observations such as dataset origin, evidence of patient leakage, or validation rigor). Any discrepancies in data extraction were resolved through discussion and consensus between the reviewers.

Risk of data leakage was categorized as high, medium, or low based on predefined criteria (see Appendix C). A study was classified as high risk if there was no explicit patient-wise data splitting, if subject overlap was possible or likely, or if data splitting occurred after feature extraction, particularly when using spectrograms or time–frequency images. Studies applying sliding window methods globally before splitting were also considered high risk, as this could cause information leakage across time or between patients. Additionally, cross-validation performed solely on samples or windows, without accounting for subject identity, contributed to this classification. A medium risk was assigned when patient-independent cross-validation was suggested but not clearly described or enforced. This included cases where data splitting occurred after feature or image generation, introducing the risk of shared patterns between training and testing sets. The use of sliding windows or overlapping segments may have introduced unintentional temporal dependencies across folds. Random splitting of instances or images without subject-level stratification also contributed to a medium risk rating. A low risk was attributed to studies that explicitly employed patient-independent cross-validation strategies, such as leave-one-patient-out (LOPO), leave-one-subject-out (LOSO), or patient-wise k-fold approaches. In these studies, data splitting was performed before feature extraction, ensuring independence at the raw EEG level. No patient overlap was observed between training, validation, and test sets, and any data augmentation or windowing techniques were confined strictly within the training folds. In all, data extraction was performed in two stages: a high-level overview and a detailed analysis. At the detailed level, some studies conducted multiple experiments with different data splitting strategies. In such cases, we treated certain experiments individually, in addition to the primary proposed approach. Moreover, paper ID, study title, cross-validation strategy (Yes/No), patient-independent splitting (Yes/No), LOSO validation strategy (Yes/No), K-fold cross-validation strategy (Yes/No), train-test splitting validation strategy (Yes/No), data splitting time (after or before pre-processing), risk of data leakage, accuracy reported (Yes/No), AUC reported (Yes/No), reported AUC value, reported accuracy value, other reported metrics, data availability (Yes/No), dataset name, dataset category (Public, Proprietary, Mixed), code availability (Yes/No), and notes and summary about the study were extracted and ready for analysis.

Statistical analysis

To investigate the relationship between methodological practices (i.e., data splitting/validation strategy, data and code availability), data leakage, and reported performance, extracted quantitative data were subjected to statistical analysis. Our analysis was conducted in two stages. Initially, we reviewed 48 studies as individual data points. In the second stage, the dataset was expanded to include a total of 145 samples by treating each validation experiment within these studies as a separate observation. This approach accounts for studies reporting multiple experiments using different sampling strategies, thereby increasing the overall sample size for more granular analysis.

Initially, the distribution and characteristics of methodological practices across studies were computed. Frequencies and percentages were calculated for all categorical variables. For example, the proportion of studies reporting patient-independent data splitting, the use of specific cross-validation strategies, or the timing of data splitting. In addition, we quantified the prevalence of methodological practices such as code availability, data availability, and so on. Continuous performance metrics such as reported ACC and AUC were categorized into five ordinal bins: Excellent, Very Good, Good, Fair, and Low/Very Low (see Appendix D). Discretizing continuous performance scores into ordinal categories enhances the analysis’s robustness and interpretability. This approach establishes a standardized framework for comparing heterogeneous evaluation metrics (e.g., AUC, ROC) while reducing sensitivity to the spurious precision of reported scores.

Following this, to assess associations between categorical variables, contingency tables were first constructed, and category frequencies were summarized using cross-tabulation. e.g., data leakage [Yes/No] vs. reported acc/AUC [High/Medium/Low], or open-source code [Yes/No] vs. patient independent splitting [Yes/No], Validation Strategy (Train/Test vs K-Fold vs LOOCV) vs. Data Leakage (Yes/No), and so on. Statistical significance was then evaluated using the chi-square test, or Fisher’s exact test in the case of 2 × 2 tables. Corresponding p-values were reported, with statistical significance defined at p < 0.05 unless otherwise specified. The strength of association was quantified using Cramér’s V, interpreted as follows: 0 indicates no association; approximately 0.1, weak; 0.3, moderate; and 0.5 or higher, strong association [47].

We then analyzed the causal relationships and indirect effects using Partial Least Squares Structural Equation Modeling (PLS-SEM)  [48, 49]. Given the conceptual structure of the constructs, Validation Strategy, Open Code (Data), Data Leakage, and Reported Performance were modeled as formative constructs. The Validation Strategy latent variable was formed by four distinct indicators: Patient Independent Splitting, K-fold Cross Validation (KCV), Leave-One-Out Cross Validation (LOOCV), and Train-Test Split, representing common validation techniques in deep learning. These methods are not interchangeable and collectively define the operational design of the training pipeline, justifying a formative specification. Similarly, Reported Performance was constructed from two related but separate performance metrics (Reported Accuracy and AUC), making Reported Performance a formative construct as well. Because Accuracy and AUC measure conceptually distinct aspects of performance (overall correctness versus threshold-independent discrimination), they do not necessarily co-vary. Therefore, modeling them as formative indicators creates a composite performance construct that represents these multi-faceted aspects of a model’s utility without assuming they are redundant.

Finally, the interrelationships between these variables were hypothesized in the proposed model (Fig. 1). Furthermore, all constructs were modeled as formative composites and validated for multicollinearity prior to estimation. To ensure construct validity in the formative measurement model, we examined outer weights and used bootstrapping to evaluate their significance. Additionally, collinearity diagnostics (VIF) were performed to confirm that indicators within each construct did not show problematic multicollinearity (VIF < 3.3). Path coefficients (β) between latent constructs were interpreted based on their sign, magnitude, and significance (p-values), with special attention to potentially inflated estimates caused by weak or suppressive indicators.

At the end, to assess whether reported performance metrics reflect generalizable findings or mirage metrics (inflated results due to methodological flaws like data leakage), we triangulated evidence from both classical statistical tests and PLS-SEM.

Results

A total of 48 studies were included. From these, we extracted 145 distinct study observations, as many studies reported multiple experiments, validation strategies, or data splitting approaches. Eighty-six percent of the studies used public datasets, but only 22% reported their code as openly available. We found that LOSO and Patient Independent Splitting (PIS) exhibited multicollinearity based on the variance inflation factor (VIF), and therefore, include only LOSO in the analysis. We also observed that, across all the studies, data splitting was performed after pre-processing.

Prevalence and characteristics of methodological practices

Summary of dataset use and accessibility

Across 48 studies, datasets from Helsinki University Hospital  [50], the Cork University Maternity Hospital (CUMH) Datasets, and CHB-MIT  [51] were used in 58.46%, 23.08%, and 6.15% of the papers, respectively (Fig. 2). While other studies employed Fetal Sheep Models Dataset (Fetal), Parma University Hospital Dataset, Duke University Medical Center Dataset, Sophia Children’s Hospital (SCH) Dataset, Temple University Hospital EEG Corpus (TUEG) Dataset, ANSeR Clinical Dataset, and 6NICUs  [52] proprietary datasets, which were used once, except the Fetal dataset.

Fig. 2.

Datasets usage and availability

Summary of code availability and openness

Out of 48 papers and 145 study observations, only 11 papers (22.9%) and 26 studies (distinct study observations) (17.9%) explicitly provided access to their code (Fig. 3).

Fig. 3.

Summary of code availability and openness

Summary of validation strategy of study observations

Out of 145 study observations, 22.8% (33/145) used LOSO and PIS as the most common validation strategy (Fig. 4). Followed by Train-Test Split (TTS) and a combination of LOSO, KCV, and PIS with 21.4% (32/245) and 15.2% (22/145), respectively. However, 14.5% (21/145) did not explicitly explain their validation strategies, while three studies used all four validation strategies.

Fig. 4.

Summary of validation strategy of study observations

Summary of reported performance metrics of study observations

Out of 145 study observations, 47.6% (69/145) reported full metrics, while 20.7% (30/145) didn’t report AUC, and 17.9% (26/145) reported ACC (Fig. 5a). Moreover, 11.0% (16/145) and 1.4% (2/145) only reported AUC and ACC, respectively. We also observed that 22.1% (32/145) and 28.9% (42/145) were not reporting AUC and ACC, respectively (Fig. 5b, c). In both ACC and AUC rank reports, “Excellent” and “Very Good” were the top reported, while “Very Low” was the least.

Fig. 5.

Summary of reported performance metrics of study observations

Summary of data leakage risk

We observed that 16.7% (8/48) of the papers, or 31.7% (30/145) of the study observations, presented a high risk of data leakage (Fig. 6). However, a low data leakage risk was observed in 54.2% (26/48) of the papers as well as 47.6% (69/145) of the study observations.

Fig. 6.

Summary of data leakage risk

Methodological factors associated with reported performance metrics

Validation strategies and their association with reported metrics

Table 1 presents the chi-square test results for associations between validation strategies and reported metrics using papers and distinct study observations. In the 48-paper study, the chi-square test of independence showed a statistically significant association between the Leave One Patient Out strategy and the reported accuracy (, p = 0.0006) (Table 1). The strength of association was moderate, as indicated by Cramér’s . It also revealed a statistically significant association between the Patient Independent Splitting strategy and the reported accuracy ((, p = 0.0005). The strength of association was moderate, as indicated by Cramér’s . A chi-square test of independence using 145 distinct study observations showed that: (I) A statistically significant association between the Leave One Patient Out strategy and the reported accuracy (, p < 0.0001). The strength of association was strong, as indicated by Cramér’s ; (II) A significant association between Leave One Patient Out strategy and the reported AUC(, p = 0.0076). The strength of association was weak, as indicated by Cramér’s V = 0.2218; (III) A statistically significant association between Patient Independent Splitting strategy and the reported accuracy (, p < 0.0001). The strength of association was strong, as indicated by Cramér’s ; (IV) A statistically significant association between K-Fold Cross Validation strategy and the reported AUC (, p = 0.0458). The strength of association was weak, as indicated by Cramér’s ; (V) A statistically significant association between Train Test Splitting strategy and the reported accuracy (, p = 0.0004). The strength of association was weak, as indicated by Cramér’s ; and (VI) A statistically significant association between Train Test Splitting strategy and the reported AUC (, p = 0.0001). The strength of association was weak, as indicated by Cramér’s .

Table 1.

Statistical tests of association between validation strategies and performance metrics

Association Type	df	By Paper (n = 48)				By Study Observation (n = 145)
		Type of test	p-value	Cramer’s V	Conclusion	Type of test	p-value	Cramer’s V	Conclusion
LOPO V.S. Reported ACC	1	χ2 test	0.0006	0.4952	Sig. Assoc. (Mod.)	χ2 test	p < 0.001	0.5806	Sig. Assoc. (Strong)
LOPO V.S. Reported AUC	1	χ2 test	0.0798	0.2529	No Sig. Assoc.	χ2 test	0.0076	0.2218	Sig. Assoc. (Weak)
PIS V.S. Reported ACC	1	χ2 test	0.0005	0.4992	Sig. Assoc. (Mod.)	χ2 test	p < 0.001	0.5941	Sig. Assoc. (Strong)
PIS V.S. Reported AUC	1	Fisher’s Exact	0.1736	N/A	No Sig. Assoc.	χ2 test	0.1350	0.1241	No Sig. Assoc.
K-Fold CV V.S. Reported ACC	1	χ2 test	0.0152	0.3503	No Sig. Assoc.	χ2 test	0.2783	0.0900	No Sig. Assoc.
K-Fold CV V.S. Reported AUC	1	Fisher’s Exact	0.3037	N/A	No Sig. Assoc.	χ2 test	0.0458	0.1659	Sig. Assoc. (Weak)
TTS V.S. Reported ACC	1	χ2 test	0.1689	0.1986	No Sig. Assoc.	χ2 test	0.0004	0.2955	Sig. Assoc. (Weak)
TTS V.S. Reported AUC	1	Fisher’s Exact	0.1626	N/A	No Sig. Assoc.	χ2 test	0.0001	0.3209	Sig. Assoc. (Mod.)

Note: df = degree of freedom, Sig. Assoc. (Mod.). = Statistically Significant Association of Moderate Strength, Sig. Assoc. (Strong) = Statistically Significant Association of Strong Strength., Sig. Assoc. (Weak) = Statistically Significant Association of Weak Strength, No Sig. Assoc. = No Statistically Significant Association. LOPO=Leave One Patient Out, PIS = Patient Independent Splitting, TTS = Train Test Splitting

Data leakage and its influence on reported metrics

In the 48-paper study, a Fisher’s exact test revealed a significant association between the Data Leakage Risk and the reported accuracy (p = 0.0002) as well as Data Leakage Risk and the reported accuracy rank (p = 0.0001) (Table 2). In the study observation, a chi-square test of independence revealed a statistically significant association between the Data Leakage Risk and the reported accuracy (, p < 0.0001). The strength of association was strong, as indicated by Cramér’s . A Fisher’s exact test in the study observation also revealed a statistically significant association between the Data Leakage Risk and the reported other metrics (p = 0.0001) as well as Data Leakage Risk and the reported accuracy rank (p = 0.0002) (Table 2).

Table 2.

Statistical tests of association between data leakage risk and performance metrics

Association Type	df	By Paper (n = 48)				By Study Observation (n = 145)
		Type of test	p-value	Cramer’s V	Conclusion	Type of test	p-value	Cramer’s V	Conclusion
Data Leakage Risk V.S. Reported ACC	2	Fisher’s Exact	0.0002	N/A	Stat. Sig. Assoc.	χ2 test	p < 0.001	0.6114	Sig. Assoc. (Strong)
Data Leakage Risk V.S. Reported AUC	2	Fisher’s Exact	0.1522	N/A	No Sig. Assoc.	χ2 test	0.1822	0.1533	No Sig. Assoc.
Data Leakage Risk V.S. Reported Other	2	Fisher’s Exact	0.2823	N/A	No Sig. Assoc.	Fisher’s Exact	0.0001	N/A	Stat. Sig. Assoc.
Data Leakage Risk V.S. ACC Rank	10	Fisher’s Exact	0.0001	N/A	Stat. Sig. Assoc.	Fisher’s Exact	0.0002	N/A	Stat. Sig. Assoc.
Data Leakage Risk V.S. AUC Rank	10	Fisher’s Exact	0.5146	N/A	No Sig. Assoc.	Fisher’s Exact	0.5069	N/A	No Sig. Assoc.

Note: df= degree of freedom, Stat. Sig. Assoc. = Statistically Significant Association., Sig. Assoc. (Strong) = Statistically Significant Association of Strong Strength., No Sig. Assoc. = No Statistically Significant Association,

Openness of code and data: associations with metric reporting and data leakage

A Fisher’s exact test revealed a significant association between the Dataset Availability and the reported accuracy (p = 0.0063) using the 48-paper study (Table 3). A chi-square test of independence based on the study observation revealed a statistically significant association between the Code Availability and the reported accuracy (,p = 0.0177). The strength of association was weak, as indicated by Cramér’s . It also revealed a statistically significant association between the Code Availability and the Data Leakage Risk (, p = 0.0005). The strength of association was moderate, as indicated by Cramér’s . While a Fisher’s exact test showed a significant association between the Dataset Availability and the reported accuracy (p = 0.0164), the Dataset Availability and the reported AUC (p = 0.0003), and between the Dataset Availability and the Data Leakage Risk (p = 0.0025) (Table 3).

Table 3.

Statistical tests of association between openness and performance metrics, data leakage

		By Paper (n = 48)				By Study Observation (n = 145)
Association Type	df	Type of test	p-value	Cramer’s V	Conclusion	Type of test	p-value	Cramer’s V	Conclusion
Code Availability V.S. Reported ACC	1	Fisher’s Exact	0.3037	N/A	No Sig. Assoc.	χ2 test	0.0177	0.1969	Sig. Assoc. (Weak)
Code Availability V.S. Reported AUC	1	Fisher’s Exact	1.0000	N/A	No Sig. Assoc.	χ2 test	0.5181	0.0537	No Sig. Assoc.
Code Availability V.S. Data Leakage Risk	2	Fisher’s Exact	0.2874	N/A	No Sig. Assoc.	χ2 test	0.0005	0.3230	Sig. Assoc. (Mod.)
Dataset Availability V.S. Reported ACC	2	Fisher’s Exact	0.0063	N/A	Stat. Sig. Assoc.	Fisher’s Exact	0.0164	N/A	Stat. Sig. Assoc.
Dataset AvailabilityV V.S. Reported AUC	2	Fisher’s Exact	0.7665	N/A	No Sig. Assoc.	Fisher’s Exact	0.0003	N/A	Stat. Sig. Assoc.
Dataset Availability V.S. Data Leakage Risk	4	Fisher’s Exact	0.0968	N/A	No Sig. Assoc.	Fisher’s Exact	0.0025	N/A	Stat Sig. Assoc.

Note: df = degree of freedom, Stat. Sig. Assoc. =Statistically Significant Association., Sig. Assoc. (Mod.). = Statistically Significant Association of Moderate Strength, Sig. Assoc. (Strong)= Statistically Significant Association of Strong Strength., Sig. Assoc. (Weak) = Statistically Significant Association of Weak Strength, No Sig. Assoc. = No Statistically Significant Association

Structural analysis of experimental factors affecting model performance

Our structural equation model, assessed using Partial Least Squares (PLS-SEM), provided key insights into the predictive relationships among latent constructs: Validation Strategy, Open Data (Code), Data Leakage, and Reported Performance (Fig. 7). The construct Validation Strategy emerged as a strong and statistically significant predictor of Data Leakage (p < 0.001), explaining 59.8% of its variance (R² = 0.598). In contrast, the path from Open Data (Code) to Data Leakage was not statistically significant (, p = 0.290), indicating a limited contribution of Open Data (Code) to explaining Data Leakage once the Validation Strategy is considered. Validation Strategy also failed to significantly predict Open Data (Code) (β = 0.252, p = 0.295). The explained variance in Open Data (Code) was negligible (R² = 0.082), indicating that Open Data (Code) is weakly determined by Validation Strategy and may not be functioning as an effective mediating or dependent construct within the model. However, Data Leakage was found to significantly predict Reported Performance (, p = 0.005), with an R² of 0.202, reflecting a moderate explanatory effect.

Fig. 7.

PLS-SEM modelling and experimental results. p-values are reported on the structural paths. Paths with are considered statistically significant; paths with p > 0.05 are not statistically significant. Values shown inside the endogenous constructs represent the coefficient of determination (R²). Standardized path coefficients (β) are reported in the text

Overall, the model supports a robust Validation Strategy, Data Leakage, and Reported Performance pathway, highlighting Validation Strategy (, p < 0.001, R² = 0.598) as a primary antecedent and Data Leakage as a meaningful intermediate variable. The construct Open Data (Code) (, p = 0.290, R² = 0.082), by contrast, exhibits limited structural relevance under the current model specification and may require re-evaluation or refinement in future iterations. However, the path coefficient from Validation Strategy to Data Leakage (, p < 0.001, R² = 0.598) was unexpectedly large in magnitude. Despite low multicollinearity (all VIFs < 1.4), this may indicate a suppressor effect or model misspecification. Further exploration of potential mediation and the measurement quality of latent variables is recommended.

Discussions

Using 48 papers and 145 unique study observations, the study systematically examines the relationships among validation strategies (data partitioning), open code (data), data leakage, and reported model performance in EEG-based neonatal seizure detection research.

Despite 87.69% of the studies using public datasets to train neonatal seizure detection EEG models, only 22.9% reported their codes as publicly available. On the one hand, there is a standard use of public datasets in the field, and on the other hand, there are limited proprietary options. This discrepancy highlights a crucial gap or bottleneck between reproducibility and transparency, undermining efforts to validate the neonatal EEG models, reproduce experimental training, and ensure consistency across findings. Furthermore, this limits progress in benchmarking and fair model comparison. Additionally, there are only a few publicly accessible neonatal EEG datasets, with limited seizure episodes compared to the number of seizure occurrences—1–3 per 1000 live births in high-income countries and substantially higher in low- and middle-income countries  [43].

Out of 145 study observations, we found that 22.8% (33/145) used the LOSO and PIS validation strategies. These results highlight a preference for patient-independent splitting methods, particularly LOSO and PIS, while also showing that many studies either do not specify their validation approach or use methods outside of these common categories.

Despite increased awareness of evaluation transparency in neonatal EEG seizure detection models, our analysis revealed significant inconsistencies in how model performance metrics are reported. Only 47.6% (69/145) of the study observations presented the full set of metrics, while 20.7% (30/145) did not report AUC, a critical metric for imbalanced classification in neonatal seizure detection. Furthermore, only 11.0% (16/145) reported AUC, 1.4% (2/145) reported only ACC, and some metrics can be misleading when the class distribution is skewed (Fig. 5a). Overall, we find that current reporting practices lack consistency, which hampers reproducibility, overestimates model effectiveness, and limits the field’s ability to perform meaningful benchmarking and clinical translation.

While studies on low data leakage risks were common, we found that a notable proportion of studies (i.e., 16.7% of the papers or 31.7% of the study observations) showed a significant vulnerability to data leakage. The chi-square test revealed that a significant association was found between the Leave One Patient Out strategy and the reported accuracy, as well as the Patient Independent Splitting strategy and the reported accuracy. We also observed that Fisher’s exact test revealed a significant association between the Data Leakage Risk and the reported accuracy. Conversely, no significant dependencies between code availability and data leakage risks, suggesting independence between these factors in our review. These findings reinforce growing concerns in neonatal EEG seizure detection models regarding the replicability crisis and the need for validation protocols. Therefore, to develop a robust, generalizable, and clinically valuable neonatal seizure detection model, adopting a methodological approach is essential, including: (I) mandatory subject-wise data splitting; (II) early splitting; (III) appropriate cross-validation techniques; (IV) strategic use of data augmentation; and (V) addressing inter-subject variability through domain adaptation and transfer learning.

To evaluate the hypothesized relationships among validation (data partitioning) strategies, open data (code), data leakage, and reported performance (Fig. 1), we applied PLS-SEM. The model shows that the validation strategy significantly predicts Data Leakage, explaining 59.8 of its variances, making a strong relationship. Notably, the path from validation strategy to data leakage was both statistically significant and negative, suggesting that robust validation strategies are associated with lower data leakage risk. In contrast, open code (data) does not significantly affect data leakage, nor significantly predicted by the validation strategy, suggesting open code (data) is not playing a meaningful role in the model. Data leakage significantly affects reported performance, accounting for 20.2% variation, while open code (data) appears weak. The findings support our hypotheses and emphasize the key role of validation strategy, showing a direct effect of validation strategy on data leakage, with limited evidence for mediation or indirect effects through open code (data). Collinearity diagnostics indicated acceptable levels (VIFs < 3.3), confirming the stability of the model estimates. In all, the PLS-SEM finding shows that the model explains a large portion of data leakage, validation strategy was significantly associated with data leakage, whereas open code (data) was not, despite the model explaining a substantial portion of data leakage variance (R² = 0.598). Furthermore, the moderate R² for reported accuracy suggests that leakage has a real but not overwhelming effect on performance, still meaningful, especially in the neonatal EEG seizure detection model. The non-significant path from open code to data leakage indicates that transparency alone is an insufficient safeguard against methodological flaws. While valuable, code availability does not guarantee the correct implementation of validation protocols; improper data partitioning can persist and be propagated even within public repositories. These findings argue that open science initiatives must be augmented with stringent validation standards to ensure the reproducibility and reliability of reported outcomes.

Our findings underscore three critical needs in the application of the neonatal seizure detection model: (I) The need for an EEG seizure detection evaluation framework. Clinical researchers should interpret the predictive model with caution that the validation strategy has explicitly accounted for data leakage and temporal drift, since an improper validation strategy leads to “mirage” the performance of the model. For clinicians, reproducibility and transparent reporting are crucial to trust the adoption of the model in clinical settings and make the model interpretable; (II) The need for systemic incentives for transparency and reproducibility, including open code (data). Journals and conferences should require methodological checklists that include reproducibility checklists, model cards, and the release of open-source code and data . Funding agencies should prioritize grants that commit to open-source development and rigorous evaluation. Introducing benchmarking initiatives for EEG-based seizure detection model with standard datasets and transparent leaderboards is important; and (III) The need for future research directions that integrate tooling and education for widespread adoption of best practices, such as an automatic leakage detection tool and validation-aware model wrappers that check the leakage and enforce subject (or patient) independent splitting by default. Moreover, organizing workshops, tutorials, and modules is a core skill, particularly in the domain of EEG detection models.

While our study offers systematic insight into the methodological landscape of AI models in neonatal seizure detection, several limitations warrant consideration. First, our systematic review analysis relied on data extracted from published literature in the English language, which may affect reporting bias and incomplete methodological disclosures, including the accuracy of the validation strategy and data leakage risk latent constructs. Second, while the PLS-SEM model was suitable for exploratory structural assessment and testing hypothesized directional influences, the analysis may be sensitive to indicator selection, and potential confounding factors were not fully accounted for. Third, our scope is limited to neonatal EEG-based studies, which limits the generalizability potential to other biomedical domains. Fourth, to enhance statistical power, our unit of analysis was the individual experiment rather than the publication, which captures the methodological heterogeneity within papers but introduces potential non-independence, as experiments from the same study may be correlated. While we acknowledge this clustering effect as a limitation, future investigations could employ multilevel modeling to explicitly account for this hierarchical data structure. Finally, we acknowledge that AUC can be especially helpful in imbalanced datasets, such as seizure detection, and future research could consider treating AUC and Accuracy as separate outcomes to evaluate the robustness of our results.

Conclusions

The study systematically evaluates the relationships between validation strategies (data partitioning), open code (data), data leakage, and reported model performance in EEG-based neonatal seizure detection literature. More specifically, exploring the effect of data leakage on reported performance. Our analysis highlights that most studies utilize public datasets, the low rate of code sharing, and incomplete metric reporting raise concerns about reproducibility and transparency. The PLS-SEM results and chi-square test align well, supporting that rigorous data partitioning reduces leakage and that leakage adversely affects performance. The systematic assessment of the AI model in the neonatal seizure detection model validation strategy emerged as a strong and statistically significant predictor of data leakage and reported accuracy. The chi-square test showed a statistically significant association between the reported accuracy and the Leave One Patient Out strategy, as well as the Patient Independent Splitting strategy. Fisher’s exact test also revealed a significant association between the data leakage risk and the reported accuracy. The PLS-SEM model also explained a substantial portion of the variance in risk of data leakage (R² = 0.598) and modest variance in reported performance (R² = 0.202), underscoring the critical role of validation strategy in ensuring a credible and generalizable neonatal seizure detection model. Overall, the evidence leans toward the generalizability of the relationship between leakage and performance, but with some caution due to variability in accuracy outcomes across studies.

Our findings support and underscore the need for methodological standards to mitigate leakage and improve transparency in the neonatal EEG seizure detection model. To move beyond “mirage” by minimizing inflated reported performance and ensure generalizability in EEG seizure detection models, clinical research communities and conference organizers should emphasize transparency and methodological standards; journals require explicit leakage audits and a reproducibility checklist; and funding agencies should prioritize proposals that demonstrate strong methodological robustness.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Author contributions

GST conceived and designed the study, carried out the experiments, analyzed the data, and wrote the initial manuscript draft. HS contributed to data acquisition, analysis, interpretation, and reporting. TW provided overall supervision, intellectual guidance, and critical revisions and proofreading of the manuscript. All authors read and approved the final manuscript.

Funding

This publication has emanated from research supported under the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 101,034,252.

Data availability

No datasets were generated or analysed during the current study.

Materials and code availability

All materials, methods, and data supporting the findings of this study are available within the main manuscript and the supplementary appendix (Appendices A, B, C, and D). No unique materials were generated or used that require additional access.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent of publication

The authors have reviewed, approved, and agreed to publish this work.

Competing interests

The authors declare no competing interests.