Comment on: “A comprehensive landscape of AI applications in broad-spectrum drug interaction prediction: a systematic review” (Marzouk et al., 2025)

Abstract

Marzouk et al. reviewed 147 studies on artificial intelligence (AI) applications for predicting drug–drug, drug–disease, and drug–nutrient interactions, providing a broad overview of current machine learning and deep-learning approaches. However, several methodological and conceptual limitations reduce the reproducibility and interpretability of the review. The search strategy appears largely restricted to PubMed with title- and abstract-level filtering, while manual record removal is reported without explicit criteria defining “irrelevant” studies, limiting transparency and reproducibility. Protocol registration, duplicate independent screening, standardized extraction procedures, and formal bias assessment using established frameworks such as ROBIS, PROBAST+AI, and TRIPOD+AI were not clearly reported. The review reports performance metrics such as area under the receiver operating characteristic curve (AUROC), but does not provide a structured framework for interpreting or comparing metrics across heterogeneous datasets, prediction tasks, and evaluation protocols. Because the interpretation of AUROC and precision–recall metrics depends on class prevalence, outcome definition, and the intended prediction task, future reviews should report complementary discrimination metrics, calibration, uncertainty estimates, and external validation rather than assuming that any single metric is universally preferable. Claims of superior model performance should be supported by confidence intervals and statistical comparisons appropriate to the evaluation design, such as paired DeLong testing when applicable. Claims of superior model performance should also be supported by appropriate statistical testing, including methods such as the nonparametric DeLong test. Several conceptual clarifications are also warranted. AI models may prioritize hypotheses but do not replace experimental or clinical validation under current regulatory standards. Furthermore, AUROC should not be conflated with pharmacokinetic area under the curve, and SciBERT should not be characterized as a three-dimensional molecular graph framework. Future reviews should adopt transparent multi-database searches, structured bias assessment, and reproducible reporting practices.

Introduction

Artificial intelligence for interaction prediction, spanning drug–drug, drug–disease, and drug–nutrient pairs, has expanded rapidly across machine learning, deep-learning, and graph-based methods. Marzouk et al. synthesize 147 studies and map the main datasets, offering a useful landscape of current approaches [1]. Marzouk et al. describe a systematic review workflow involving database searching, study screening, eligibility assessment, and narrative synthesis of AI-based interaction prediction studies. The review also references systematic review principles and summarizes datasets, model categories, and reported performance metrics across included studies. However, several key methodological elements, including protocol registration, duplicate independent screening and extraction, structured risk-of-bias assessment, and explicit reproducibility measures, were either insufficiently reported or not incorporated. Clarifying these distinctions is important for evaluating the methodological robustness and interpretability of the review. As this evidence base grows toward clinical use, the methodological scaffolding of reviews becomes as important as the models themselves. Under the Journal of Cheminformatics standards, research and software should be fully reproducible by third parties, with accessible data, code, and analysis artefacts. Future syntheses should therefore publish complete search strings, standardized extraction tables, screening logs, and analysis scripts to enable independent verification. Knowledge-graph methods, which integrate heterogeneous entities and relations across biomedical sources, should be explicitly distinguished from text-only language-model approaches to avoid modality conflation in performance claims.

Because core computer science venues are not comprehensively covered by PubMed, single-database searches risk selection bias. A planned multi-database strategy that explicitly includes Embase, Scopus, and the Cochrane Library for biomedical coverage, together with IEEE Xplore and the ACM Digital Library for computing venues, is recommended. This aligns with Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 (PRISMA 2020) and PRISMA-S guidance on transparent, reproducible search reporting [2, 3]. Registering the protocol on PROSPERO and depositing deduplication logic will further enhance auditability.

To support reliable decision-making in pharmacovigilance and translational informatics, future reviews should be aligned with established reporting and appraisal frameworks. Use PRISMA 2020 and PRISMA-S for search and reporting, Risk Of Bias In Systematic reviews (ROBIS) for review-level bias assessment, and TRIPOD+AI with Prediction model Risk Of Bias Assessment Tool, AI extension (PROBAST+AI) for the evaluation and reporting of prediction-model studies [4–7].

Search strategy and study selection

Comprehensiveness is the foundation of a reliable systematic review. Restricting searches to a single biomedical database overlooks innovation presented in computer science journals and conference proceedings. PubMed indexes biomedical and life sciences literature but does not comprehensively cover engineering and computer science venues [8]. Recent empirical evidence confirms that bibliographic database coverage remains incomplete and varies substantially across research fields. In a large-scale analysis of studies included in Cochrane reviews, supplementing PubMed with Embase increased study-level coverage by 5.5 percentage points, resulting in a combined coverage of 88.0%; however, the incremental contribution of Embase differed considerably across review groups and over time [9]. In a subsequent cross-sectional analysis of 4,709 references from 274 systematic reviews, MEDLINE indexed 86.6% and Embase indexed 88.2% of the included references, whereas the combined indexing rate was 90.2%; notably, 9.8% of the references were indexed in neither database [10]. These estimates are context-specific and should not be interpreted as universal retrieval rates. Coverage also differs according to publication type and disciplinary scope. A recent cross-database analysis of 9,575,603 publications from 2012 to 2022 identified substantial differences in document-type classification across PubMed, Scopus, Web of Science, OpenAlex, and Semantic Scholar, including differences in the identification of conference-proceedings records [11]. These findings support the use of complementary sources when a review spans biomedical and computational disciplines. Accordingly, reviews of AI-based drug-interaction prediction should combine appropriate biomedical and multidisciplinary databases with computing-focused sources, such as IEEE Xplore and the ACM Digital Library, while prespecifying and justifying the selected database combination in the review protocol [2, 3, 5, 9–11]. Consequently, future reviews should couple PubMed with Embase and Scopus and explicitly search IEEE Xplore and the ACM Digital Library to capture models developed by computer scientists. Registries like PROSPERO and adherence to PRISMA-S guidelines should be used to predefine and publish full Boolean search strategies and deduplication methods [4]. Citation chaining (screening references and citations of included studies) further enhances completeness.

The original review excluded studies that analyzed interactions within specific disease contexts, such as COVID-19 or oncology. Although intended to avoid narrow clinical signals, this policy sacrifices valuable domain-specific information. Retaining these studies and stratifying results by disease or drug class preserves clinical relevance without compromising breadth.

Screening and data extraction

Robust screening and data extraction processes minimize selection errors and enhance reproducibility. The review does not clearly report whether screening and extraction were performed independently by multiple reviewers, with no evidence of independent duplicate review or consensus discussions with the corresponding author, raising concerns about potential missed studies and subjective decision-making. Methodological research demonstrates that duplicate independent screening detects more eligible records and reduces the median proportion of missed studies by roughly 5% [10]. Independent extraction using piloted forms and reconciliation further ensures accuracy. Although dual processes require more resources, they are the standard for decision-critical syntheses. Future reviews should report inter-reviewer agreement (e.g., κ statistics) and deposit screening logs as supplementary materials. Inter-reviewer agreement may be quantified using Cohen’s κ statistic:

$$\kappa =\frac{Po-Pe}{1-Pe}$$

where P_o represents observed agreement and P_e represents expected agreement by chance. For example, if two reviewers agree on 85 of 100 screened studies (P_o = 0.85) and expected chance agreement is 0.5, the resulting κ value would be 0.70, indicating substantial agreement. Commonly used interpretation thresholds classify κ < 0.4 as poor agreement, 0.40–0.75 as moderate-to-substantial agreement, and > 0.75 as excellent agreement. Screening logs should additionally document numbers of retrieved records, duplicate removals, title/abstract exclusions, full-text exclusions, and reasons for exclusion to improve transparency and reproducibility.

Risk of bias and applicability assessment

Assessing bias and applicability is vital to contextualize predictive performance. The PROBAST comprises 20 signaling questions across four domains, participants, predictors, outcomes, and analysis and was designed to evaluate risk of bias in prediction-model studies. PROBAST+AI, an extension tailored for machine learning models, is forthcoming [7]. At the review level, ROBIS can identify biases related to search, selection and synthesis procedures. Neither PROBAST nor ROBIS was applied in the original review. Beyond methodological reporting limitations, the review primarily provides descriptive summaries of included studies without sufficient critical appraisal of their methodological robustness, validation quality, clinical interpretability, or comparability. Important differences in model architectures, training strategies, dataset composition, external validation procedures, and theoretical strengths and limitations are not systematically examined. As a result, the review offers limited guidance regarding which AI approaches may be more reliable, generalizable, or clinically applicable across different interaction-prediction settings. Structured appraisal frameworks such as PROBAST+AI and ROBIS are valuable not only for formal bias assessment but also for improving interpretability and methodological contextualization of evidence synthesis. Future syntheses should integrate these tools and map results to reporting standards such as TRIPOD+AI to ensure transparent reporting of model development and validation [6]. Stratifying synthesis by risk of bias (e.g., high vs. low) aids interpretation (Appendix A and B). ROBIS primarily evaluates bias at the systematic review level, including concerns related to eligibility criteria, study identification, selection procedures, data collection, and synthesis methods. In contrast, PROBAST+AI is intended for study-level appraisal of AI-based prediction models and addresses methodological concerns such as overfitting, inadequate validation, inappropriate predictor handling, outcome definition, and analytical bias. TRIPOD+AI serves a complementary role as a reporting framework designed to improve transparency and completeness of prediction-model reporting rather than directly assessing bias. Alternative frameworks may also be relevant in specific contexts, including AMSTAR-2 for general systematic review methodology appraisal and QUADAS-2 for diagnostic prediction settings.

Synthesis framework and performance metrics

The review reports AUROC as a commonly cited performance metric across included studies; however, it does not provide a structured framework for interpreting or comparing these metrics across heterogeneous datasets, prediction tasks, and evaluation settings. Although AUROC quantifies a model’s ability to discriminate between classes, it can be misleading when positive cases are rare. Precision–recall curves emphasize performance on the positive class and are often more informative than ROC curves when the positive class is rare and correctly represents the minority class, although their relative utility depends on the specific prediction task and should not be considered universally superior for all imbalanced datasets [11]. Thus, PR-AUC should accompany AUROC to reflect both discrimination and positive predictive value. Calibration, the agreement between predicted probabilities and observed outcomes is equally important. Models with excellent discrimination but poor calibration may mislead decision-makers [12]. Calibration can be assessed using calibration plots or metrics such as Brier scores; reviewers should report these measures and evaluate whether studies performed external validation on independent datasets, which is essential for transportability.

Claims that one model outperforms another require statistical justification. When two models are evaluated on the same independent test sample, their correlated ROC curves may be compared using an appropriate paired method, such as the nonparametric DeLong test, together with a confidence interval for the AUROC difference [13]. Importantly, a higher AUROC point estimate does not by itself establish superior discrimination. For example, Zhao et al. reported validation AUROCs of 0.876 for a support-vector-machine model and 0.863 for a random-forest model, but the difference was not statistically significant according to the DeLong test (P = 0.825) [14]. Similarly, Li et al. reported AUROCs of 0.90 for a multimodal AI model and 0.88 for a clinical-information model, yet the difference was not statistically significant (P = 0.28) [15]. These empirical examples illustrate that ranking models solely according to AUROC point estimates can overstate evidence of superiority. Model comparisons should therefore report the AUROC difference and its confidence interval, apply a statistical method appropriate to the evaluation design, and additionally consider calibration, clinical utility, and external validity.

Conceptual clarity and technical corrections

A few conceptual inaccuracies in the original review warrant correction. First, AI models complement rather than replace experimental or clinical validation. Predictive algorithms can prioritize hypotheses but cannot substitute in vitro, in vivo or clinical studies that confirm causality [14]. Regulatory guidelines from the International Council for Harmonization and the FDA mandate experimental evidence for drug–drug interactions. Second, the term “area under the curve” has multiple meanings. In machine learning, AUROC refers to the area under the ROC curve; in pharmacokinetics, AUC denotes the integral of plasma concentration over time. Using explicit qualifiers (AUROC vs. pharmacokinetic AUC) prevents confusion. Third, SciBERT, a BERT-based language model pretrained on scientific text, is designed for natural language processing tasks and not for encoding three-dimensional molecular graphs [15]. Distinguishing between text-based models and 3D graph neural networks (e.g., SchNet) clarifies data requirements and appropriate validation methods [16].

Practical recommendations for future systematic reviews

1. Protocol registration and multi-database searching: Register a protocol (e.g., with PROSPERO) and execute comprehensive searches across biomedical and computer science databases. Publish full search strings and deduplication methods to enable reproducibility [4].

2. Duplicate screening and extraction: Employ at least two independent reviewers for screening and data extraction. Report agreement statistics and provide screening logs to increase transparency [10].

3. Quality assessment: Apply PROBAST+AI to assess study-level bias and applicability. Use TRIPOD+AI to ensure complete reporting. Employ ROBIS to evaluate bias at the review level [6, 7].

4. Comprehensive performance reporting: Report AUROC alongside PR-AUC for imbalanced outcomes and include calibration metrics. Use appropriate statistical tests (e.g., DeLong) to compare models and emphasize external validation [11–13]

5. Retain domain-specific studies: Do not exclude studies focused on specific diseases or drug classes. Analyze them as strata to preserve clinically relevant signals.

6. Use precise terminology and separate modalities: Use standardized terminology and clearly distinguish different AI model classes, evaluation metrics, and validation approaches to improve methodological transparency and facilitate comparison across studies [14]; and clearly differentiate language models (e.g., SciBERT) from 3D molecular approaches [15, 16].

Conclusions

Marzouk et al.’s review offers a valuable map of AI models for broad spectrum interaction prediction, but methodological shortcomings limit its reproducibility and translation. By adopting rigorous reporting standards, expanding search strategies, conducting duplicate screening, evaluating risk of bias with PROBAST+AI and ROBIS, and reporting comprehensive performance metrics, future reviews can generate decision grade evidence to support pharmacovigilance and clinical decision making. Precise terminology and inclusion of domain specific studies will further bridge AI innovations with safe drug therapy. Implementing these recommendations will help researchers harness AI’s potential to identify harmful interactions while maintaining scientific integrity and patient safety.

Abbreviations

AUC

Area under the curve

AUROC

Area under the receiver operating characteristic curve

PK AUC

Pharmacokinetic area under the curve

DDI

Drug–drug interaction

DNI

Drug–nutrient interaction

DDSI

Drug–disease interaction

LLM

Large language model

PR-AUC

Area under the precision-recall curve

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

PROBAST

Prediction model Risk Of Bias Assessment Tool

ROBIS

Risk Of Bias In Systematic reviews

TRIPOD+AI

Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (AI extension)

Appendix A: ROBIS Phase‑2—concerns about the review process

ROBIS domain	Concern level	Rationale (evidence‑based)
1. Study eligibility criteria	High concern	Inclusion/exclusion choices restrict scope (e.g., excluding disease‑specific topics such as COVID‑19/cancer); no protocol registration; eligibility not demonstrably pre‑specified; date windows vary across interaction types
2. Identification and selection of studies	High concern	PubMed‑only; title/abstract‑limited search; manual removal of records without explicit criteria defining “irrelevant” studies; no duplicate independent screening reported; language limits not stated; no additional sources (Scopus/Web of Science/IEEE/ACM/preprints)
3. Data collection and study appraisal	High concern	Extraction without standardized forms; number of reviewers and calibration not described; no formal risk‑of‑bias/quality appraisal of included modelling studies
4. Synthesis and findings	High concern	Narrative synthesis; emphasis on ROC‑AUC; calibration, PR‑AUC, external validation and sensitivity analyses not systematically addressed

Overall ROBIS judgement: HIGH RISK OF BIAS.

Appendix B: mapping to standard bias domains

Type of bias	Assessment	Rationale
Selection bias	High risk	Single‑database, title/abstract‑limited search; topic exclusions likely remove relevant AI‑DDI work; language limits unclear
Performance bias (review conduct)	High risk	Manual record removal without explicit criteria; no standardized forms; no explicit duplicate independent screening/extraction
Detection bias (appraisal)	High risk	No formal study‑level risk‑of‑bias assessment; appraisal tool not reported
Attrition bias (missing data)	Unclear risk	Handling of incomplete reporting and missing metrics not described; author contact not reported
Reporting bias (protocol/selective reporting)	High risk	No a priori protocol registration; departures from plan cannot be verified

Author contributions

All authors contributed to conception, critique, drafting, and approval of the final manuscript.

Funding

None.

Data availability

No datasets were generated or analysed during the current study.

Declarations

Competing interests

The authors declare no competing interests.