Houston, We Have AI Problem! Quality Issues with Neuroimaging‐Based Artificial Intelligence in Parkinson's Disease: A Systematic Review

Abstract

In recent years, many neuroimaging studies have applied artificial intelligence (AI) to facilitate existing challenges in Parkinson's disease (PD) diagnosis, prognosis, and intervention. The aim of this systematic review was to provide an overview of neuroimaging‐based AI studies and to assess their methodological quality. A PubMed search yielded 810 studies, of which 244 that investigated the utility of neuroimaging‐based AI for PD diagnosis, prognosis, or intervention were included. We systematically categorized studies by outcomes and rated them with respect to five minimal quality criteria (MQC) pertaining to data splitting, data leakage, model complexity, performance reporting, and indication of biological plausibility. We found that the majority of studies aimed to distinguish PD patients from healthy controls (54%) or atypical parkinsonian syndromes (25%), whereas prognostic or interventional studies were sparse. Only 20% of evaluated studies passed all five MQC, with data leakage, non‐minimal model complexity, and reporting of biological plausibility as the primary factors for quality loss. Data leakage was associated with a significant inflation of accuracies. Very few studies employed external test sets (8%), where accuracy was significantly lower, and 19% of studies did not account for data imbalance. Adherence to MQC was low across all observed years and journal impact factors. This review outlines that AI has been applied to a wide variety of research questions pertaining to PD; however, the number of studies failing to pass the MQC is alarming. Therefore, we provide recommendations to enhance the interpretability, generalizability, and clinical utility of future AI applications using neuroimaging in PD. © 2024 The Author(s). Movement Disorders published by Wiley Periodicals LLC on behalf of International Parkinson and Movement Disorder Society.

Parkinson's disease (PD) is an age‐related neurodegenerative disease associated with debilitating motor deficits but also accompanied by nonmotor symptoms. ¹ Approximately 1% of the elderly population develops PD, hallmarked by a gradual loss of dopaminergic neurons in the substantia nigra. ² Diagnosis relies on clinically detectable motor symptoms, with the Movement Disorder Society‐Unified Parkinson's Disease Rating Scale (MDS‐UPDRS) serving as a clinical tool to assess motor (II–IV) and nonmotor (I) symptoms. ³ Despite the high prevalence of PD, efficient early or differential diagnoses, prognoses, and treatment strategies remain a major clinical challenge. The heterogeneous disease phenotype and overlapping symptoms with atypical parkinsonian syndromes (APS) contribute to high misdiagnosis rates ⁴ , ⁵ and hamper accurate prognoses. Moreover, by the time of a PD diagnosis, up to 70% of dopaminergic neurons are already degenerated, ² , ⁶ significantly reducing the potential effectiveness of intervention therapies. For this reason, various neuroimaging modalities were proposed and validated to enhance diagnostic accuracy. ⁷ , ⁸ , ⁹ Unfortunately, interpreting neuroimaging data, especially when considering additional risk factors for understanding individual PD trajectories, can be a highly complex task and may require advanced automated solutions.

“Artificial intelligence” (AI) is an umbrella term for powerful tools (eg, from machine or deep learning) capable of integrating and interpreting the information of diverse clinical datasets. AI can be defined as the ability of computers to learn and perform tasks without specific instructions based on data patterns. ¹⁰ In recent years, AI has garnered increasing attention in the field of neuroimaging in PD, particularly with increasing availability of data and advancing computational resources. With their strong potential for detecting complex associations within multidimensional (eg, imaging) and/or multimodal data (eg, incorporating imaging, genetic, and demographic data), AI can aid diagnosis, estimate disease course trajectories, or contribute to treatment planning. This can pave the way for more patient‐tailored (eg, precision medicine) rather than uniform approaches in PD. ¹¹

However, with the ever‐increasing number of studies using neuroimaging‐based AI techniques in PD, maintaining an overview of potentially helpful applications for daily practice is becoming more challenging. It has been repeatedly recommended that quality standards be ensured among medical AI studies ¹² , ¹³ , ¹⁴ to achieve generalizable results and interpretable decision‐making. Without proper regulation, overfitting can cause issues of generalizability. Overfitting occurs when AI algorithms capture noise that is specific to the training data, resulting in strong performance during training but poor performance at the validation stage and real‐world settings. ¹⁵ Another common problem is data leakage, which happens when information from data used for validation is introduced into the training process. ¹⁶ , ¹⁷ , ¹⁸ Additionally, overly complex approaches that are trained on a relatively large number of features are prone to overfitting as they are likely to fit noise in the training data rather than disease‐specific patterns. ¹⁹ , ²⁰ Incomplete reporting of model performance further prevents interpretability of a model's strengths and pitfalls. ²⁰ Together, these (mal)practices have likely impeded the implementation of AI into clinical routines. Although these issues are well known, the quality of neuroimaging‐based AI studies in PD has not yet been systematically analyzed.

Therefore, the aims of this systematic review were threefold: (1) to compile a comprehensive overview of AI applications based on neuroimaging in PD, (2) to evaluate the quality of existing AI research studies using minimal quality criteria (MQC), and (3) to suggest actionable recommendations to enhance the quality of medical research utilizing AI.

Methods

All decisions were performed by two independent raters; in cases of disagreement consensus was reached by discussion. This review was not registered.

PubMed Search

A PubMed search was conducted on January 1, 2024, with the search term shown in Figure 1 (full search term available in the Supplementary Materials, M1), yielding 810 articles. The search term was deliberately broad to capture an inclusive range of literature. We established the following exclusion criteria based on consultations with experts in the fields of PD/neurology (T.E.) and AI (E.D.): articles had to be original research articles (eg, no reviews or case reports), written in English, and not be retracted. Moreover, the study had to be done in humans, and PD had to be a stand‐alone diagnosis (ie, PD could not be grouped with other diseases as “parkinsonian syndromes”). Similarly, neuroimaging had to be the primary input modality for the AI model, which needed to adhere to our definition of AI and serve a clinically relevant research aim. Consequently, studies on AI‐based preprocessing or symptom classification and those lacking a defined outcome were excluded.

FIG. 1.

Visualization of PubMed search term and the respective exclusion criteria for current systematic review. Top: the search term incorporated three main domains with the respective possible expressions: (1) the clinical condition of interest (Parkinson's disease [PD]), (2) the techniques of interest (artificial intelligence [AI]), and (3) the neuroimaging modality (eg, MRI [magnetic resonance imaging]). Bottom: exclusion criteria and number of studies failing the criteria. [Color figure can be viewed at wileyonlinelibrary.com]

Categorization of Research

According to their aims, papers were categorized into diagnosis, prognosis, and intervention, with some papers fitting into multiple categories.

The first category was diagnosis, which was further subdivided into six subcategories:

1. Classification of PD versus healthy controls (HC, ie, a cohort without neurological impairment)

2. Classification of PD versus APS (where PD was a distinct group)

3. Identification of PD subtypes (eg, clinically established subtypes like tremor dominant or new biological PD subtypes; biological subtypes needed to be related to clinically relevant explanatory variables like genetic variants)

4. Disease staging and symptom severity estimation

5. Classification of cognitive impairment

6. Classification of freezing of gait (FOG)

The second category was prognosis. Four subcategories were defined:

1. Prediction of conversion to PD (ie, from prodromal to clinical PD)/monitoring of PD (ie, changes from clinical to biomarker‐confirmed diagnosis)

2. Prediction of disease and symptom severity

3. Prediction of symptom occurrence

4. Conversion to PD dementia

The third category was therapeutic interventions. Two subcategories were formed:

1. Drug therapy

2. Deep brain stimulation (DBS)

MQC

We defined MQC and applied them to all included papers. The criteria were based on commonly applied evaluation methods to assess medical AI approaches and cover validity of methods at the implementation level (MQC 1–3) and diligence of reporting at the interpretation level (MQC 4 and 5):

1. Train/test split: data were split into separate cohorts for training and evaluating the model. This criterion was fulfilled if (at least) a training and a test set existed, or if cross‐validation was applied. This criterion did not apply to unsupervised algorithms.

2. No data leakage: no information from the test set or target variable was available to train the algorithm or transform the training data (eg, this criterion was not passed when feature selection was based on the whole dataset rather than only on training data). ¹⁶ , ¹⁷ , ¹⁸

3. Minimal model complexity: the number of features had to be smaller than that of individuals, or samples included in the training set. ¹⁹ , ²⁰ This criterion did not apply to deep learning–based models.

4. Generalizability: test or average cross‐validation performance was reported to infer model performance on unseen data, ¹⁶ without selective reporting of favorable results. This criterion did not apply to unsupervised algorithms.

5. Indication of biological plausibility: outlining how strongly features contributed to the model's performance to ensure that these features can be interpreted in a biologically meaningful context. ²⁰

MQC could be rated as “passed” (✓), “failed” (✗), “not applicable” (NA), or “unclear” (?). Unclear indicated that the raters could not rate the MQC due to the paper lacking adequate information. MQC were evaluated only for algorithms specific to the research aim of the respective study. The full MQC details are presented in the Supplementary Materials (section M3).

Meta‐Data

Next to assessing the quality of included studies, we also recorded the aim (ie, outcome), modality, best‐performing model, most effective features, sample size, and overall performance of each study's best model. For comparability, we reported accuracy (the most common metric), where available, and otherwise area under the curve (AUC) for classification; and r, r², or the mean absolute error (MAE) for regression approaches. If cross‐validation and test results were available, according to best practices, we reported the test performance of the model that performed best during cross‐validation. Although reporting of performance on external test sets demonstrates generalizability, only a few studies did so. Therefore, we reported performance on test (ie, validation data derived from the same cohort as the training data) and external test set (ie, additional validation data derived from a different cohort than the training data) when available. For comparability, we specifically noted usage of Parkinson's Progression Markers Initiative (PPMI) data.

Statistical Analysis

Statistical analyses were conducted per study (ie, each unique digital object identifier, n = 244), per subcategory entry (ie, counting a study allocated to two subcategories as two unique subcategory entries, n = 284), or per study aim (ie, counting each of a study's aims within or across subcategories, n = 327). For an example, see Supplementary Materials, section M4. Within the “classification of PD vs. HC” subcategory, we analyzed whether model accuracy was different between model types or modalities using a t test weighted by sample size (at the level of study aims). Only studies reporting an accuracy were included in this analysis.

To assess the methodological quality of studies, we evaluated the percentage of studies meeting MQC at the subcategory level (the same study never differed in MQC across study aims within one category). To assess whether recency or journal quality influences MQC adherence, we assessed frequencies of MQC adherence by year and impact factors, respectively, across subcategory entries. Impact factors were retrieved via Clarivate's Journal Citation Reports (https://jcr.clarivate.com/jcr/) for the publication year and arranged into quantiles. Finally, we examined which MQC indicate overfitting by comparing the accuracies of study aims (noting that accuracies might vary across multiple aims within the same subcategory) that passed versus those that did not pass each implementation‐level MQC (MQC 1–3). This comparison was made using a 1‐tailed t test (test: “failed”/“unclear” > “yes”).

Bias Assessment

Our bias assessment strategy comprehensively addressed three critical dimensions: comparability of included studies, generalizability of results, and reproducibility of findings.

First, when retrieved studies were screened, we included only studies with a clearly defined AI method, with neuroimaging features, in a distinct cohort of clinically diagnosed PD patients, to address clinically relevant diagnostic, prognostic, or interventional aims. Whereas studies without neuroimaging features were excluded, those that complemented neuroimaging with other clinical features were not excluded. Second, we investigated the generalizability of study results by applying defined MQC, as described in the “PD‐ versus HC‐Specific Results” section, and by recording sample sizes. We expanded the investigation of generalizability beyond our five MQC by documenting metrics (balanced accuracy or sensitivity + specificity) or practices (eg, subsampling) appropriate for handling imbalanced data. ²¹ We assessed whether external test sets were used for validation and analyzed accuracy drops between test and external test sets using a paired t test. Finally, reproducibility with respect to study setup was scrutinized, including positron emission tomography (PET) tracer and scanner type, magnetic resonance imaging (MRI) scanner type, field strength, sequence, and electrode system and recorder type for electroencephalography (EEG) and magnetoencephalography (MEG). In the cases of functional imaging, information regarding activity (ie, resting or task) and eye status (ie, eyes open or closed) was required.

Results

Identification of Studies for Current Review

We included 244 of the initially retrieved 810 studies in the current systematic review (Fig. 1). Most commonly, we excluded articles that were not original (n = 108), did not focus on PD/include PD as a separate entity (n = 118), or did not use AI methodology (n = 147).

Description of the Categorization and Content of Studies

A total of 86% of studies reported at least one diagnostic aim, and 8% and 9% reported a prognostic and an interventional aim, respectively. In total, 284 papers were allocated to appropriate subcategories (several subcategories per study were possible). PD versus HC was the most prominent subcategory, with 54% (n = 131) of studies assigned to this aim, followed by PD versus APS (25% of studies, n = 60). Much fewer studies were categorized as diagnostic, prognostic, or interventional (<8% of studies per subcategory, n ≤ 18; Fig. 2). All studies in the individual subcategories, including outcomes, implementation details, and performance, along with the MQC ratings can be accessed in the Supplementary Materials, Tables R3–R14 (diagnosis: R3–R8, prognosis: R9–R12, intervention: R13–R14). The most frequently used model types were support vector machines (SVMs, 46% of studies, n = 131), convolutional neural networks (CNNs, 17%, n = 47), and ensemble learning (14%, n = 39). 88% (n = 250) of subcategory entries used unimodal, and 12% (n = 34) used multimodal neuroimaging data as input.

FIG. 2.

Quantity of studies categorized under diagnosis, prognosis, and intervention, including subcategories. Some studies had multiple aims and were thus assorted to several subcategories. APS, atypical parkinsonian syndromes; HC, healthy control; PD, parkinson's disease [Color figure can be viewed at wileyonlinelibrary.com]

Diagnosis

Studies categorized under “classification of PD versus HC” achieved accuracies of up to 100%, ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ which is further discussed in section "PD‐ versus HC‐Specific Results". The subcategory “classification of PD versus APS” included mainly binary approaches (67%), aiming to differentiate PD from specific or multiple combined other APS. In a comprehensive study, Huppertz et al demonstrated that PD can be distinguished not only from individual APS (multiple systems atrophy‐cerebellar type, multiple systems atrophy‐parkinsonian type, progressive supranuclear palsy‐richardson syndrome) dichotomously (balanced accuracy ≥85%) but also in a multiclass paradigm (all class accuracies ≥75%). ³⁰ Another study utilized an ensemble of probabilistic binary classifiers to build a multiclass paradigm, which successfully distinguished between PD and APS forms (accuracy = 88%–94%). ³¹ Studies labeled as “identification of PD subtypes” applied either supervised methods to classify known clinical phenotypes of PD (eg, postural/gait vs. tremor dominant) ³² or unsupervised methods to identify new subtypes associated with the clinical characteristics of PD. ³³ Moreover, multiple studies demonstrated the potential of using AI for “disease staging and symptom severity estimation,” with accuracies of up to 99% for classifying different stages (Hoehn & Yahr) ³⁴ and correlation coefficients of up to 0.75 between true and estimated current symptom severity. ³⁵ In the “classification of cognitive impairment” subcategory, studies showed that clinical stages of cognitive impairment can be classified with high accuracy (accuracy = 92%–100%), ³⁴ , ³⁶ , ³⁷ , ³⁸ , ³⁹ and severity of global cognitive impairment (Mini‐Mental State Examination) can be estimated (r = 0.55). ⁴⁰ Finally, studies demonstrated the potential of using AI on EEG data for real‐time “classification of freezing of gait” with accuracies of up to 86%. ⁴¹ , ⁴² In summary, neuroimaging‐based AI methods accurately captured different clinical aspects and may support differential diagnosis, disease subtyping, and staging in PD.

Prognosis

Two studies categorized under “conversion to PD/monitoring of PD” outlined that [123I]‐FP‐CIT SPECT scans may contain information to determine if subjects without evidence of dopaminergic deficit (SWEDD) will be recategorized to biomarker‐confirmed PD in the future. ⁴³ , ⁴⁴ Moreover, CNNs based on [18F]‐FDG‐PET may predict whether individuals with rapid eye movement behavioral disorder (RBD) will progress to PD with moderate accuracy (AUC = 72%). ⁴⁵ Similarly, multiple studies labeled with “prediction of disease and symptom severity” predicted either motor symptom progression after 4 years (UPDRS‐III decline, MAE ≤4.7) ⁴⁶ , ⁴⁷ or overall disease progression after 3 years (Hoehn & Yahr stages, accuracy = 84%). ⁴⁸ Additionally, five studies investigated “prediction of symptom occurrence.” Three studies predicted (accuracy = 73%–77%) the occurrence of FOG within 5 seconds based on EEG. ⁴⁹ , ⁵⁰ , ⁵¹ Prediction into the more distant future was achieved for the development of FOG ⁵² or RBD. ⁵³ Finally, the development of mild cognitive impairment (accuracy = 87%) ⁵⁴ or the conversion from mild cognitive impairment to dementia (accuracy = 74% ⁵⁵ /AUC = 88% ⁵⁶ ) was anticipated using imaging in the “prediction of cognitive decline” subcategory. To summarize, neuroimaging‐based AI anticipated short‐ and long‐term symptom and disease development, illustrating great potential for prognosis and intervention.

Intervention

Studies in the subcategory “drug therapy” used neuroimaging‐based AI models to identify characteristic changes in brain activity (EEG/MEG) in response to dopaminergic treatment, ²⁵ , ⁵⁷ , ⁵⁸ , ⁵⁹ , ⁶⁰ to classify or predict the occurrence of levodopa (l‐dopa)–induced dyskinesias, ⁶¹ , ⁶² , ⁶³ or to classify or predict motor improvement from l‐dopa intake. ³³ , ⁶⁴ , ⁶⁵ , ⁶⁶ , ⁶⁷ Similarly, UPDRS‐III improvement after “deep brain stimulation” was highly accurately classified ⁶⁸ , ⁶⁹ or even predicted from preoperative data ⁷⁰ in several studies. ⁷⁰ , ⁷¹ Notably, SVM based on preoperative T1‐weighted MRI and demographic data predicted motor, cognitive, and behavioral improvements after DBS. ⁷² Moreover, AI showed potential for the localization of stimulation sites from EEG, ⁷³ functional MRI (fMRI), ⁷⁴ or microelectrode recordings. ⁷⁵

PD‐ versus HC‐Specific Results

Because PD versus HC was the most studied aim, we considered all 113 studies reporting accuracy in this subcategory to further explore which models and modalities best captured PD‐specific brain changes. Frequently used model types were SVMs (41% of studies), CNN (24%), ensemble learning (14%), and logistic regression (7%), and their mean sample sizes were 155, 466, 213, and 204 individuals, respectively. Weighted by sample size, ensemble learning methods (μ_weighted = 94.1%, SD_weighted = 4.9%) outperformed other methods in the classification of PD, as assessed using weighted t tests with Bonferroni correction (Fig. 3A; Supplementary Materials Figure R1 and Table R1). The most frequently used modalities were T1‐weighted MRI (20% of studies), EEG (17%), [123I]‐FP‐CIT SPECT (13%), and resting‐state fMRI (10%), with average sample sizes of 354, 73, 662, and 102 individuals, respectively. Studies using [123I]‐FP‐CIT SPECT (μ_weighted = 97.4%, SD = 2.1%) were significantly more accurate compared to any other modality after Bonferroni correction (all P < 0.005; Fig. 3B; Supplementary Materials Figure R2 and Table S2).

FIG. 3.

Accuracy of PD (Parkinson's disease) versus HC (healthy control) classification studies by (A) model type and (B) modality. (A) Accuracy of studies, sorted by model type (color) and sample size (size). Model types are sorted in alphabetical order. Studies of the same model type were also summarized to a weighted mean and weighted standard deviation, with sample size serving as weights (indicated by vertical lines). *Ensemble models performed significantly better (surviving Bonferroni correction, P < 0.005) than the remaining model types. (B) Accuracy of studies, sorted by modality type (color) and sample size (size, same as in A). Modalities are sorted in alphabetical order. Studies of the same modality were also summarized to a weighted mean and weighted standard deviation, with sample size serving as weights. *Significant differences (surviving Bonferroni correction, 10 comparisons ‐ P < 0.005) were detected between accuracies obtained using [123I]‐FP‐CIT SPECT and any other modality using weighted t tests. Notes for (A) and (B): sample sizes ranged from 21 to 2077 and were log transformed for plotting. Legend shows true (nontransformed) sample sizes. Only studies that reported accuracy were considered. If ranges were reported by studies, this figure shows the mean accuracy of these ranges. Only four most frequent modalities/models are shown explicitly, whereas the remaining ones were grouped as “others.” Studies of all model and modality types (including those summarized as “other”) are shown in Supplementary Materials Figures 1 and 2. CNN, convolutional neural network; SVM, support vector machine.

Differences in accuracy between other modalities were not significant. However, most (15 of 17) [123I]‐FP‐CIT studies used PPMI data, where (ab)normality is among the inclusion/exclusion criteria for both groups, leading to a strong circularity. The remaining two studies used clinically diagnosed PD patients without biomarker confirmation and still yielded accuracies of 96% ⁷⁶ and 100%, ⁷⁷ respectively. Supplementary Materials section R1 exhibits the performance of all model or modality types, that is, including those summarized as “other.”

MQC Results

Only 20% of subcategory entries passed all MQC, with MQC 2 (no data leakage, 49% passed), MQC 5 (biological plausibility, 57%), and MQC 3 (minimal model complexity, 57%) passed least frequently (Fig. 4A). MQC 1 and 4 were passed by 95% and 92% of subcategory entries, respectively. Failure to pass implementation‐level MQC (MQC 1–3, 33% passed) was more frequently observed compared to failure to pass interpretation‐level MQC (MQC 4 and 5, 53% passed). Most subcategory entries passed between three and four MQC (Fig. 4B).

FIG. 4.

Minimal quality criteria (MQC) analysis. (A) Percentage of studies passing individual or all MQC. (B) Violin plot of the accumulated number of MQC passed by each individual study. (C) Number of total studies published (gray shaded area) and percentage of studies passing all MQC (red line) over the years. (D) Percentage of studies passing all MQC by impact factor. (E) Differences in accuracy between studies passing or failing individual or all implementation‐level MQC. [Color figure can be viewed at wileyonlinelibrary.com]

MQC 2 frequently failed due to test data influencing feature selection, for example, because feature selection was performed prior to splitting the data into training and test sets, prior to cross‐validation, or as recursive feature elimination in the absence of a test set. In studies where multiple samples were acquired per subject, such as EEG and fMRI, MQC 2 was also often failed when authors did not report whether data of the same subject was exclusively allocated to the training or test set. MQC 3 was failed in numerous cases due to incomplete descriptions of feature extraction/selection workflows. MQC 5 was failed in many cases where authors listed the selected features before model training but did not explain their importance to the final model performance (eg, through feature weights or permutation testing).

Investigating study quality over time, we observed that there was no increase in the percentage of studies passing all MQC per year, whereas there was a steep increase in the quantity of studies that were published (Fig. 4B). We also assessed whether MQC adherence covaried with the general level of quality control, as indicated by the impact factor of the journals. This analysis showed that, whereas there was a trend toward more studies passing MQC with higher‐impact factors, MQC adherence was generally low across bins (Fig. 4C). Finally, we investigated whether model accuracy was significantly different between study aims passing implementation‐level MQC. N = 225 study aims reported an accuracy of their results. We found that studies with data leakage yielded significantly higher accuracies compared to studies where no data leakage was identified (87.5 ± 10.9% vs. 84.6 ± 12.1%), indicating data leakage was a primary source of overfitting (t = −1.8, P = 0.035 [1 tailed], not significant after correction for multiple comparison [α = 0.017]). Other differences were not significant.

Bias Assessment beyond MQC

A total of 244 studies were deemed eligible for bias assessment. Importantly, we excluded 11 studies as they did not specify any research aim. Of the included studies, sample sizes ranged from 4 to 3188, with mean sample sizes for CNN, SVM, and ensemble learning approaches being 445, 128, and 246, respectively.

Generalizability was compromised due to few studies passing our MQC and because some classification aims (16%, n = 49) did not address data imbalance. Additionally, only 7% (n = 22) of aims validated their findings in external test sets. Unsurprisingly, mean performance in external test sets was significantly lower than that in test sets from the original dataset (mean difference = 5%, t = 3.1, P < 0.01).

Regarding reproducibility, several studies lacked essential details; 46% (n = 32) of PET/SPECT, 20% (n = 23) of structural MRI, and 6% (n = 2) of fMRI did not indicate the scanner type, of which 91%, 87%, and 100% used PPMI data, respectively. The tracer was not specified in 6% (n = 4 [75%, n = 3 PPMI]) of PET/SPECT studies. Field strength and sequences were not provided in 16% (n = 18) and 19% (n = 21) of structural, as well as 6% (n = 2) and 9% (n = 3) of fMRI studies. For fMRI studies, 9% (n = 3) lacked activity status, and 44% (n = 14) did not report eye status. Similarly, these details were missing in 5% (n = 2) and 21% (n = 8) of EEG and MEG studies, respectively. Additionally, 21% (n = 8) of EEG/MEG studies lacked information about the electrode system.

Discussion

Recent years have seen rapid development and increased accessibility of AI methods, leading to their widespread application in neuroimaging research. In this review, we systematically summarized and rated the quality of research papers using AI methods on neuroimaging data in PD, specifically focusing on diagnosis, prognosis, and therapeutic intervention outcomes. 54% of studies were concerned with the differentiation of PD and HC, whereas differential diagnosis and especially prognostic or intervention applications were rarely investigated. To the best of our knowledge, we are the first to assess the quality of such studies, revealing that the majority of studies—an alarming 80%—failed to pass all common and minimal quality standards of AI.

Methodology Misalignment and Clinical Impact of PD‐AI Research

With more than 50% of studies covering the “issue” of differentiating PD from HC, the potential of AI and its actual fields of application appears to be strongly misaligned. Multiple studies claimed that PD versus HC classification could support the early diagnosis of PD, that is, the early identification of PD‐related brain changes in asymptomatic individuals. However, PD patients included in such studies were often already in advanced stages and showed clinically overt symptoms. To evaluate the effectiveness of PD versus HC models in identifying asymptomatic individuals at risk for PD, clinical follow‐up for diagnosis verification is necessary. However, none of these studies included longitudinal assessments. The clinical utility of papers in this category is therefore severely limited. In our analyses, we showed that ensemble learning as a model and [123I]‐FP‐CIT SPECT as modality achieve almost perfect accuracy. Importantly, when stratification into PD or HC groups is already supported using [123I]‐FP‐CIT SPECT, as is the case in the PPMI study (https://www.ppmi-info.org/study-design/study-cohorts#overview/), their distinction based on the same modality is highly circular, and clinical relevance is not granted. Moreover, HCs in studies hardly represent a population likely to consult a neurologist for a potential diagnosis of PD. ⁷⁸ Among individuals presenting to movement disorders clinics, major challenges rely on correct diagnosis of a particular parkinsonian syndrome, that is, differential diagnosis, ⁴ , ⁵ personalized assessments/subtyping and prognostic assessments, ⁷⁹ , ⁸⁰ or the prediction of therapeutic efficiency. ⁸¹ Unfortunately, such clinically relevant and more challenging aims were rare in the current literature. The potential of AI for multimodal integration and pattern recognition could be more strongly exploited for these tasks. PD is known to affect various levels of brain organization ⁸² , ⁸³ ; thus, unimodal neuroimaging information, used by the majority of studies assessed here, may restrict information content on individual patients compared to multimodal assessment.

Current State of Quality in PD‐AI Research

Beyond a descriptive representation, we performed quality assessments of all included studies in two areas, that is, methodological soundness at the implementation level and diligence of reporting at the interpretation level. At the implementation level, we demonstrated that studies frequently committed data leakage or failed to provide sufficient evidence of avoidance thereof (MQC 2). Moreover, numerous studies applied overly complex models or lacked a clear description of the number of features and participants used in their workflows (MQC 3). Although advanced techniques such as partial least squares regression have been proposed to handle high dimensionality in small training sets more efficiently than conventional AI algorithms, this remains a topic of debate. ⁸⁴ Importantly, the application of more advanced methods may only partially resolve the issue. Therefore, minimal model complexity remains an important aspect when model validity is assessed. Studies passing implementation‐level MQC exhibited significantly lower accuracies than studies with (potential) data leakage. This finding underscores the assumption that data leakage induces overfitting; ie, it artificially inflates performance. ¹⁷ , ¹⁸ A major source of data leakage was the allocation of multiple samples per subject to both the training and test set. Indeed, a previous study showed that this practice can cause an overestimation of accuracy by about 50% in classifying PD versus HC. ¹⁸ Including overfitted AI models in clinical practice, thus, could decrease rather than increase diagnostic accuracy, as clinicians may not be aware of the erroneous behavior and adapt their diagnosis based on the model output ⁸⁵ —a process known as “automation bias.” Beyond within‐study data leakage, the frequent usage of big databases such as PPMI also increases the risk for “cross‐literature data leakage”; that is, that feature selection for studies using some database is done based on prior literature employing partly the same data. However, this latter issue pertains to research in general and may even be less common in AI studies, where feature selection tends to be data rather than literature driven.

At the interpretation level, studies often did not report feature importance, thereby preventing the assessment of biological plausibility. Especially in clinical settings, it is important to understand and interpret the decision‐making process of models to confidently integrate AI‐generated information. ⁸⁵ Moreover, feature importance is critical to form a conceptual bridge between AI and neuroscience, allowing neuroimaging‐based models to deepen mechanistic insights. ²⁰ , ⁸⁶

Recommendations for Future Research

Based on the issues described earlier, we could identify a tremendous gap between the clinical needs of PD management and the use of scientific resources in the realm of neuroimaging‐based AI research. This gap pertains to both study aims and quality standards. Therefore, we recommend fostering interdisciplinary collaboration between clinicians and engineers, wherein clinicians should be consulted to identify relevant research questions, and engineers should ensure correct and high‐quality solutions to these needs. Admittedly, a comprehensive assessment of quality standards in AI‐based studies requires considerable specific expertise even when provided with a catalog like the MQC we put forward here. It may still be surprising that even high‐impact journals did not predominantly publish high‐quality AI studies. Therefore, journal editorial boards should also promote awareness on the importance of AI quality standards during the review process. For example, reviewers could be prompted to indicate if they recommend an additional AI expert review.

We demonstrated that a fifth of studies did not account for data imbalance and only a minority of studies used external test sets. Whereas our MQC aimed to investigate the bare minimum of AI quality standards, AI applications aimed to translate into clinical tools are critical to account for both these issues to prove the generalizability of the results. One should account for data imbalance, for example, by subsampling, ie, reducing the number of individuals from one class to that of the smallest class, or by reporting adequate metrics (eg, balanced accuracy, F1 score, or area under the precision–recall curve). External test sets, that is, test sets acquired at a different site with potentially different scanning protocols or diagnostic criteria, are of central importance to demonstrate the generalizability. In cases where external test sets are unavailable, permutation testing provides an alternative to test whether a model is overfitted. ⁸⁷ , ⁸⁸ Some train–test division procedures are more prone to overestimations of accuracy than others. It was proposed to prefer nested cross‐validation or train–test split without cross‐validation, especially in small datasets (n < 1000). ¹⁶ Moreover, instead of a simple train–test split, multiple splits (training/validation/test) are preferable. ⁸⁹ Generalizability is also constrained by different preprocessing/feature extraction techniques, which are often extensive and not well described. End‐to‐end AI solutions, wherein raw neuroimages are fed to AI models, would circumvent this source of bias. Finally, we recommend that analysis pipelines should be graphically visualized in AI publications, starting with feature selection, including train–test splitting or cross‐validation, and ending with the final outcome. This practice has immensely facilitated understanding different pipelines and assessing MQC adherence in our investigations.

Limitations

Some limitations of the current systematic review should be acknowledged. Even though we used a broad search term, we might have overlooked some new, highly specific neuroimaging‐based AI applications in this rapidly evolving field. Recent developments enabled non‐imaging PD biomarkers, such as from bodily fluids. Although neuroimaging offers the advantage of spatial information of abnormality, it may be worthwhile for future studies to extend the input modality (imaging or fluid biomarkers) depending on availability in clinical practice and the specific research objective. Although we provide a catalog of quality indicators, we recognize that the evaluation of MQC requires expertise, which may not be ubiquitously available. Given the extensive evaluation procedure, there is also a time lag between the conducted analyses and publication. Nonetheless, we have shown that quality remains low even in the most recent literature. Finally, our criteria are a first step and cover basic methodological requirements for AI studies rather than the whole picture. To ensure high‐quality research in the dynamic field of neuroimaging‐based AI research, there may soon be a need to update and complement such criteria.

Conclusion

In conclusion, we showed that there is a wide array of possible applications of neuroimaging‐based AI for PD, yet current resources are often used on tasks with questionable clinical relevance. It thus seems critical to enhance communication between engineers and clinicians. Moreover, in this first‐of‐its‐kind meta‐analysis, we demonstrated that the quality of neuroimaging‐based AI studies in PD is alarmingly low, especially at the implementation level. We therefore recommend to strengthen quality control of AI studies before they are published, optimally both during study design and at the review stage. By fostering high‐quality, interdisciplinary collaborations in AI research, more reliable, interpretable, and clinically relevant neuroimaging tools can ultimately advance clinical management of PD.

Author Roles

1. Research project: A. Conception, B. Organization, C. Execution; 2. Statistical analysis: A. Design, B. Execution, C. Review and critique; 3. Manuscript preparation: A. Writing of the first draft, B. Review and critique

V.D. 1A, 1B, 1C, 2A, 2B, 2C, 3A, 3B

E.D. 1A, 1B, 1C, 2A, 2B, 2C, 3A, 3B

H.E. 1C

A.S. 1C, 2A, 3B

D.V. 1C, 2A, 3B

K.S. 1C, 2A, 3B

T.v.E. 1A, 1C, 2A, 2C, 3B

Full financial disclosures of all authors for the preceding 12 months

T.E. receives honoraria for consulting or advisory roles from Lundbeck Foundation, Lundbeck Pharma, Orion Pharma, and GT Gain Therapeutics SA. A.P.S. is supported by the Canadian Institutes of Health Research (CIHR) (PJT‐173540) and the Krembil‐Rossy Chair program. K.S. receives funding from the National Institutes of Health (R01NS088160, R01NS124228, R01DC011805, R01DC012545, R01DC019353, P50DC01990, and R01DE030464) and the Department of Defense, serves on the scientific advisory board of the Tourette Association of America and the Voice Foundation, and received consulting fees from Jazz Pharmaceuticals Inc. and AbbVie Inc.

Supporting information

Data S1. PRISMA 2020 Checklist.

Data S2. Supplementary Materials.

Data Availability Statement

The data that supports the findings of this study are available in the supplementary material of this article.