When Clinical and Metabolomics Data Work Together: A Comparative Framework for Multimodal Disease Classification across Machine and Deep Learning

Abstract

Disease classification using clinical and metabolomics data increasingly relies on multimodal integration, yet the complementary and comparative contributions of these modalities remain poorly understood. Most existing frameworks prioritize predictive performance without systematically examining how data modalities and modeling paradigms influence classification outcomes. Consequently, the relative value of individual modalities versus their integration, particularly in terms of model behavior, robustness, and interpretability, remains poorly characterized across machine learning (ML) and deep learning (DL) approaches. Here, we present a modality-aware comparative framework that enables direct, side-by-side evaluation of clinical-only, metabolomics-only, and combined modeling strategies within a unified pipeline. Unlike existing tools designed primarily for multiomics integration, this framework explicitly assesses when and how each modality contributes value across diverse classification scenarios. It supports the efficient use of available data while enabling a systematic comparison of model performance, stability, and interpretability across ML and DL methods. Rather than introducing a new classifier model, this work delivers a unified benchmarking workflow for evidence-based decision-making of modeling strategies in small-sample clinical metabolomics settings. Applied to two glomerulonephritis cohorts representing clinically-driven and metabolomics-driven classification settings, the framework revealed that the dominant contributing modality shifted by scenario, reflecting differences in disease context. These scenarios reflect realistic situations where discriminative signals arise from clinical variables, metabolic alterations, or their combination, as well as more complex cases involving subclass discrimination with overlapping profiles. While several models achieved comparable predictive accuracy, they differed in feature ranking stability, sampling sensitivity, and tendency to overfitting. Overall, this framework facilitates transparent, evidence-based selection of modeling strategies and data modalities suited to data complexity, sample size, and research objectives. Source code is available at https://github.com/kwanjeeraw/MMFramework.

Introduction

While genomics provides the individual’s blueprint, metabolomics reflects the functional state of biological systems. By capturing the dynamic interplay between biochemical reactions and environmental exposures, metabolomics provides a comprehensive biochemical snapshot of an individual’s physiological status. In a clinical context, this capability offers opportunities for more personalized treatment strategies and deeper insights into disease pathogenesis. Clinical diagnosis generally relies on observable symptoms, radiographic images, and a limited set of laboratory tests and molecular assays. While these methods are effective for many standard conditions, they often struggle to account for the complicated biological diversity found in complex and heterogeneous diseases. This frequently leaves clinicians with an incomplete picture of a patient’s unique stage.

Advances in high-throughput technologies now enable the simultaneous acquisition of diverse clinical and omics data, providing unprecedented opportunities to investigate disease biology at multiple levels. Integrating these complementary layers can yield a more holistic representation of disease processes. Integration strategies can be categorized according to data types (single- or cross-omics integration), statistical methodologies (e.g., empirical correlation or network-based integration), knowledge- or pathway-based approaches, and the stage at which data are combined (sequential or simultaneous approaches). These strategies are implemented through diverse statistical and computational frameworks, including multiblock chemometric techniques, , multiomics clustering, correlation-based network approaches, kernel-based methods, and other machine learning (ML) and deep learning (DL) models.

Data modalities often differ markedly in scale, dimensionality, structure, and statistical properties, raising fundamental questions about whether they should be analyzed independently or jointly within integrative frameworks. The choice of integration strategy may therefore depend on multiple factors, including the characteristics of the data sets, study design, and research objectives (e.g., disease subtyping, biomarker discovery, or mechanistic inference). These challenges underscore the importance of carefully considering the nature of the data in hand, appropriate modeling approaches, and suitable data analysis strategies for both independent and integrative workflows.

Most existing methods primarily focus on developing or evaluating integration algorithms to improve predictive performance, identify complementary information across omics layers, or discover multiomics biomarkers. In contrast, our study addresses a different practical question that is often overlooked in multiomics studies. Specifically, we investigate not only whether data integration improves predictive performance, but also under which classification settings clinical-only, metabolomics-only, or integrated models provide the greatest value. We hypothesize that, when properly evaluated, clinical-only, metabolomics-only, and integrative models can each provide meaningful and complementary insights for knowledge discovery. Evaluating these modalities side-by-side allows us to determine whether diagnostic predictions are primarily driven by physiological status (clinical information), metabolic shift (metabolomics), or their combination. Identifying concordant and discordant signals across clinical and metabolomics data is critical for improving diagnostic accuracy and advancing our understanding of disease-associated metabolic alterations. The objective of the present study is not to introduce a new integration algorithm but to establish a well-characterized data analysis framework that enables evidence-based selection of modeling strategies based on empirical performance, thereby facilitating the efficient use of all available data modalities to solve a classification problem for clinical metabolomics settings.

As a showcase application, we applied this framework to analyze real-world clinical metabolomics cohorts. We systematically evaluated how metabolomics data complement clinical information, and vice versa, in the classification and discrimination of glomerulonephritis (GN) patients. GN was selected due to its heterogeneous, immune-mediated subtypes with overlapping clinical features, making it a suitable test case for multimodal data integration. Prior studies have also highlighted metabolites in the kynurenine pathway as potential biomarkers for lupus nephritis (LN), a GN subtype.

Materials and Methods

Data Sets

We used data from two GN cohorts: a primary GN cohort comprising patients with immunoglobulin A nephropathy (IgAN) and membranous nephropathy (MN), and a secondary GN cohort comprising patients with lupus nephritis (LN) and healthy subjects. The study protocol was approved by the Siriraj Institutional Review Board, Faculty of Medicine Siriraj Hospital, Mahidol University (COA. Si 550/2023). The primary GN data set comprised 70 patients with IgAN, and 46 patients with MN, all diagnosed by renal biopsy at Ramathibodi Hospital, Bangkok, Thailand. The LN data set was obtained from Anekthanakul et al., who quantified urinary tryptophan metabolites using liquid chromatography-tandem mass spectrometry (LC-MS/MS) in 84 LN patients and 71 control subjects. The processed metabolite concentration values were used directly from that study without reanalysis of the raw data. LN patients were further classified into Class III/IV (n = 37) and Class V (n = 47). Clinical variables available for the LN cohort include age, gender, eGFR, serum creatinine (Se_Cr), urine creatinine (U_Cr), urine protein (U_Protein), and UPCR, consistent with those collected for the primary GN cohort. A full summary of clinical and metabolomics variables for both cohorts is provided in Table S1.

Quantification of Tryptophan Metabolites in the Kynurenine Pathway

Urine samples from GN patients were subjected to targeted analysis of nine urinary tryptophan metabolites, including kynurenic acid (KynA), kynurenine (Kyn), picolinic acid (Pic), quinolinic acid (Qui), tryptophan (Tryp), xanthurenic acid (Xan), 3-hydroxykynurenine (OHKyn), 3-hydroxyanthranilic acid (OHAnt) and serotonin (Se). Note that while the majority of these metabolites are kynurenine pathway intermediates, Se was produced via the parallel serotonin branch of tryptophan metabolism and is included here as a biologically relevant tryptophan-derived metabolite. The metabolites were quantified by adopting a previously published method. Briefly, 50 μL of urine was mixed with 200 μL of methanol containing 100 ng of picolinic acid-d4 as an internal standard. Analyses were conducted using a Xevo TQ-S MS/MS with an electrospray ionization source (Waters, MA). A 5 μL aliquot was injected, and targeted metabolites were separated on an HSS T3 column (2.1 × 100 mm, 1.8 μm; Waters, Milford, MA, USA) at 30 °C with a constant flow rate of 0.3 mL·min^–1. The MS was operated in positive multiple reaction monitoring mode with optimized MS parameters as reported in the previous study. Quantification was performed using a matrix-matched calibration method. All calibration curves yielded R² greater than 0.99. The mean recovery (mean ± standard deviation) of the internal standard in the urine samples was 95 ± 2.95% (n = 3). Although based on a limited number of replicates, this result was consistent with the validated method from which the protocol was adopted.

Data Processing and Statistical Analysis

The concentration of each metabolite was converted to μM before data analysis. Metabolites with less than 30% missing values were imputed on a group-wise basis using the mean concentration of the corresponding metabolite. While mean imputation is a conservative approach that may slightly underestimate within-group variance, it was selected for its simplicity and consistency with the original LN data set processing pipeline.

Metabolite-to-Tryp ratios (Metabolite_Tryp) were then calculated by dividing the concentration of each metabolite by the concentration of Tryp, and these ratios were used as metabolic features in subsequent analyses. Differences between groups were assessed using the Mann–Whitney U test, while gender distributions were compared using the chi-squared test (Table S1). Both clinical and metabolite data were log2-transformed prior to differential statistical analysis. To examine overall variance, the data were autoscaled and subjected to unsupervised principal component analysis (PCA). Autoscaling involves mean-centering each feature and dividing it by its standard deviation. Further details of model-specific preprocessing are described in the Analysis Workflow section. All data processing and statistical analyses were performed using R version 4.3.3 and Metabox 2.0.

Classification Models

Both classical ML and DL approaches were applied and systematically compared for classification tasks. Classical ML methods, such as partial least-squares discriminant analysis (PLS-DA) and random forests (RF), offer advantages in interpretability and computational efficiency for data sets with limited sample sizes and lower-dimensional feature spaces. In contrast, DL architectures leverage multilayer neural networks to learn complex and nonlinear representations, at the cost of increased data and computational requirements. In this study, both ML and DL models were implemented using consistent training and evaluation strategies to compare classification performance and feature importance across models.

ML Models

Two widely used ML algorithms, PLS-DA and RF, were explored. PLS-DA is a supervised, linear multivariate method commonly used in metabolomics for classification and feature selection. The method reduces high-dimensional data into a lower-dimensional latent space while maximizing class separation. Feature importance was evaluated using Variable Importance in Projection score (VIP) derived from PLS-DA models. VIPs summarize the contribution of each feature to class separation. Features with VIP greater than 1 were considered important contributors to the model.

RF is an ensemble-based classification approach that constructs multiple decision trees and aggregates their predictions. This enables robust classification and effective modeling of nonlinear relationships while remaining relatively insensitive to outliers and noise. RF feature importance was assessed using the mean decrease in classification accuracy, which reflects the reduction in predictive performance when a feature is permuted. Feature importance scores were scaled to a maximum of 100, with higher values indicating greater importance.

DL Models

To evaluate DL-based classification, convolutional neural networks (CNNs) were implemented using both one-dimensional (1D) and two-dimensional (2D) input representations. The 1D CNN processed tabular features in their original vector form, whereas the same features were transformed into 2D heatmaps for analysis with a 2D CNN. This design enabled comparison of how different CNN architectures handle specific input structures in terms of classification performance and feature interpretability. It also allowed evaluation of whether spatial feature arrangements in a 2D representation could capture feature co-occurrence patterns not accessible to 1D architectures.

For 1D CNN, log2-transformed features were reshaped into vectors of size p × 1, where p denotes the number of input features. The network was implemented using TensorFlow/Keras and consisted of two convolutional layers with 32 and 64 filters (kernel size = 2, same padding), each followed by rectified linear unit (ReLU) activation and max pooling. Feature maps were aggregated using global average pooling and passed to a fully connected layer with 64 units and ReLU activation, followed by dropout (rate = 0.2) and a softmax output layer. The model was trained using the Adam optimizer with the default learning rate of 0.001 and categorical cross-entropy loss. Early stopping was not employed.

Feature importance was assessed using SHapley Additive exPlanations (SHAP) with a gradient-based explainer. , SHAP estimates Shapley values, which quantify the contribution of each feature to individual model predictions. The training data were used as the background reference distribution. Global feature importance scores were calculated by averaging absolute SHAP values across training samples, analogous to VIP or RF feature importance scores. Class-specific importance profiles were generated by averaging SHAP values within each class. An additional test-set SHAP analysis was performed to assess feature contributions during model generalization.

For 2D CNN, log2-transformed features were reshaped into square heatmaps of size S × S, where S was the smallest integer greater than or equal to√p. This ensured consistent spatial dimensions across samples while retaining all features. The heatmaps were used as single-channel image inputs. The 2D CNN was implemented using TensorFlow/Keras and comprised two convolutional layers (16 and 32 filters, 3 × 3 kernel, ReLU activation), each followed by 2 × 2 max pooling. Feature maps were summarized using global average pooling and passed to a dense layer with 64 units and ReLU activation, followed by a softmax output layer. Model training was performed using the Adam optimizer with the default learning rate of 0.001 and categorical cross-entropy loss. No dropout regularization or early stopping was applied.

Feature importance was evaluated using Gradient-weighted Class Activation Mapping (Grad-CAM), which identifies input regions most influential for class predictions. Grad-CAM maps were computed by backpropagating class-specific gradients to the final convolutional layer. Activation values were then mapped back to the original input features based on their spatial positions in the heatmaps to obtain feature-level importance scores. These scores were averaged across training samples to derive overall feature importance, while class-specific profiles were obtained by averaging scores within each class. Grad-CAM analysis was also performed on the test set to assess feature contributions during model generalization. This approach allowed evaluation of whether Grad-CAM can yield meaningful feature-level importance scores when applied to pseudoimage omics data.

Analysis Workflow

Two data sets from GN cohorts were used to demonstrate the proposed framework. Each data set contains both clinical and metabolomics data, allowing systematic evaluation of clinical-only, metabolomics-only, and integrative modeling strategies across different classification approaches. A schematic overview of the analysis workflow is shown in Figure S1.

Each data set underwent identical data partitioning procedures, with 70% of samples assigned to the training set and 30% to the test set. The partitioning was repeated ten times to assess average model performance and model stability across resampling iterations. Three data modalities were analyzed independently: clinical data, metabolomics data, and integrated data, in which clinical and metabolomics features were combined into a single feature matrix.

All modeling approaches, ML (PLS-DA and RF) and DL (1D CNN and 2D CNN), were applied to each input modality. For data processing, log2 transformation was applied for all other models. The sole exception was PLS-DA, for which autoscaling was applied during model training instead. Autoscaling was used for PLS-DA to ensure that low-abundance features contributed proportionally to model variance and class discrimination. PLS-DA and RF models were optimized using 3-fold cross-validation (CV) on the training set. Given the limited sample sizes in these cohorts (approximately 28–35 subjects per group in the training set), a 3-fold CV was selected to balance reliable performance estimation with the risk of overly small fold sizes; leave-one-out CV was considered but deemed computationally prohibitive for the RF hyperparameter grid search. The RF model was implemented using the R caret package. RF hyperparameters, including mtry, were automatically tuned, while the number of trees was fixed at ntree = 500. The selected mtry value for each model was summarized in Table S2. DL models were trained with a 30% internal validation split, 200 training epochs, and a batch size of 32 to balance the trade-off between computational runtime and accuracy gains. It is noted that the combined effect of the 70/30 train-test split and the 30% internal DL validation split results in DL models effectively training on approximately 49% of the total available data, which is acknowledged as a constraint of this study design under limited sample sizes.

Model performance was evaluated on the test set using multiple classification metrics, including accuracy, F1-score, area under the receiver operating characteristic curve (AUC), sensitivity, and specificity. Performance metrics were summarized and compared across data modalities, and across modeling approaches to assess their relative predictive performance. Comparative heatmaps were used to visualize model performance across all evaluated combinations. In addition, the main performance summaries, including mean values, 95% confidence intervals (CI), standard deviations (SD), balanced metrics from class frequencies, and paired Wilcoxon comparisons across resamples, are provided in the Supporting Information.

Feature importance analysis was conducted in parallel to identify key contributors to disease prediction. Global feature importance rankings were generated by averaging importance scores across training samples. Cross-modal rank plots were used to visualize and compare feature rankings across all evaluated data modalities. To identify features with consistent importance across data modalities, feature importance scores derived from models trained on clinical, metabolomics, and integrated data sets were combined across resampling folds. Within each fold, features were ranked by descending importance, and a rank product (RP) was computed as the geometric mean of ranks across folds to identify consistently important features. Lower RP values indicate more consistently and highly important features. Statistical significance was evaluated using a permutation test and p-values were calculated with the false discovery rate (FDR) correction. Features with FDR < 0.05 were considered statistically significant. These analyses enabled a systematic comparison of predictive performance and feature relevance across clinical, metabolomics, and integrated data modalities, as well as across ML and DL models.

Results and Discussion

Multimodal Integration Framework for Clinical and Metabolomics Data

We proposed a multimodal integration framework for joint analysis of clinical and metabolomics data. The framework (Figure ) consists of three main components: an input module, model development, and class prediction with feature evaluation. It incorporates both ML and DL approaches to capture linear and nonlinear relationships among features. PLS-DA and RF serve as robust benchmark ML methods commonly used in metabolomics studies, while CNN-based models are employed for more complex patterns via multilayer neural architectures. The framework provides two key functionalities. First, it systematically evaluates how metabolomics data complement clinical information, and vice versa, in disease classification. By comparing clinical-only, metabolomics-only, and integrative models side-by-side, the framework reveals whether predictions are primarily driven by physiological status, metabolic alterations, or their combined effects. Second, it enables a systematic assessment of different classification approaches in terms of both predictive performance and feature importance.

1.

Multimodal integration framework for clinical and metabolomics data. The framework consists of three modules: (Left) Input module, which accepts three types of data sets: clinical data (C), metabolomics data (M), and integrated clinical–metabolomics data (I). (Middle) Model development module, where both ML models, PLS-DA and RF, and DL models, 1D and 2D CNNs, are applied. (Right) Class prediction and feature evaluation module, where model outputs are compared using multiple performance metrics and feature importance analyses (e.g., SHAP and Grad-CAM) to assess predictive performance and identify key clinical variables and metabolites contributing to model predictions.

Multiple classification metrics are provided to evaluate model performance, and model interpretability is supported through feature importance score extraction. These importance measures allow the identification of key metabolites and clinical parameters underlying model predictions, thereby facilitating biologically interpretable insights. This unified design further allows the integration of results from multiple models, increasing confidence through complementary evidence derived from different model learning approaches.

The framework was implemented using R and Python. DL models were developed in Python, whereas the remaining components of the framework were implemented in R. The framework is computationally efficient and can be executed on a standard desktop computer. All analyses were tested on an Apple M1 Pro system with 16 GB of RAM and an 8-core CPU. The source code is publicly available at https://github.com/kwanjeeraw/MMFramework.

Cohort Characteristics

Two clinical metabolomics cohorts were used as showcase examples for the proposed framework. GN represents a heterogeneous group of immune-mediated kidney disorders that are classified based on histological findings obtained from a kidney biopsy. LN, a secondary GN subtype, is associated with altered systemic immunity and predominantly affects young to middle-aged women. In contrast, IgAN and MN are considered manifestations of autoimmunity involving only the kidney. IgAN, the most prevalent GN subtype worldwide, is typically characterized by a microscopic hematuria and mild to moderate proteinuria, whereas MN is typically associated with a nephrotic syndrome marked by pronounced proteinuria. These differences arise from immune complex deposition at distinct locations within the glomerulus. , These diseases were selected because they represent a heterogeneous spectrum of immune-mediated kidney disorders with distinct pathogenic mechanisms but partially overlapping clinical manifestations, providing a suitable testbed for evaluating multimodal data integration.

In this study, Case 1 demonstrates a clinically-driven classification scenario using the LN cohort, comparing control subjects with LN patients as well as between two LN subclasses (LN class III/IV and class V). In contrast, Case 2 represents a metabolomics-driven scenario using the LN and GN cohorts to distinguish disease subtypes. This includes comparisons between GN and LN patients, GN-IgAN and LN, GN-MN and LN, and GN-IgAN and GN-MN. An overview of the clinical and metabolomic profiles is shown in Figure , and a summary of statistical testing is provided in Table S1. In Case 1, PCA showed clear separation between control subjects and LN patients (Figure A). However, substantial overlap was observed between LN class III/IV and class V, indicating similar clinical–metabolic profiles between LN subclasses. Univariate analyses further showed significant differences between control subjects and LN patients across several clinical parameters, particularly age, proteinuria measures, and gender (Table S1). In contrast, comparisons between LN subclasses revealed only limited significant differences, suggesting largely comparable clinical–metabolic profiles between LN subclasses.

2.

PCA score plots of clinical and metabolomic profiles. (A) PCA for Case 1, showing the distribution of control subjects and LN patients, including LN class III/IV and class V. (B) PCA for Case 2, illustrating the separation among LN and GN subtypes, including IgAN and MN.

In Case 2, PCA revealed overlap among disease groups (Figure B), although LN patients clustered distinctly from GN subtypes. The variance explained by the first two PCs in this scenario (PC1 + PC2 = 37.72%) was lower than in Case 1 (PC1 + PC2 = 41.33%), suggesting more similar clinical–metabolic profiles among patient groups compared with the healthy-disease contrast. GN patients differed significantly from LN patients across multiple clinical and metabolic variables, with particularly strong statistical values observed for metabolite features. Direct comparison between GN subtypes (GN-IgAN and GN-MN, Table S1) yielded statistically significant, though comparatively higher p-values for metabolic and clinical features, indicating partially overlapping yet distinct disease-specific profiles.

Overall, these results demonstrate that control-LN classification is primarily driven by clinical parameters, whereas discrimination among disease subtypes is more strongly associated with metabolomics and combined clinical–metabolic differences. This finding supports the rationale for both modality-specific and integrative modeling strategies.

Differential Contributions of Clinical and Metabolomics Features to LN Classification (Case 1)

Model Evaluation

Comparative evaluation of clinical-only, metabolomics-only, and integrated data modalities across ML and DL models revealed clear performance differences between disease detection and disease subclassification tasks (Figure and Tables S3–S5). Discrimination between LN patients and controls was robustly captured by clinical variables, with both ML and DL approaches achieving consistently high performance across all metrics (AUC > 0.90; Figure A). These results indicated that clinical features alone were sufficient for discriminating LN from controls. While metabolomics-only models showed the poorest performance across modeling approaches. Integrating clinical and metabolomics data achieved performance comparable to clinical-only models, suggesting that dominant disease-related signals were already well represented in standard clinical variables. The strong discrimination between LN patients and controls observed using clinical variables alone likely reflects the pronounced physiological differences between diseased and nondiseased groups. When established clinical biomarkers already capture major disease manifestations, both ML and DL models can achieve high predictive performance with limited additional benefit from metabolomics features. Similar observations have been reported in cardiometabolic studies where metabolomic biomarkers provided only modest improvements when added to models based on traditional clinical risk factors. , Together, these findings suggest that metabolomics may offer greater value for disease stratification or mechanistic characterization rather than for binary disease–control classification when dominant clinical signals are already present.

3.

Comparative classification performance across data modalities and modeling approaches. Two prediction tasks were evaluated: (A) control vs LN and (B) LN class III/IV vs class V. Models were trained using clinical-only (Cli; blue), metabolomics-only (Met; green), and integrated (Int; orange) data modalities with both ML and DL models. Heatmaps summarize prediction performance regarding accuracy, sensitivity, specificity, F1-score, and AUC. Color intensity represents model performance, with purple indicating higher performance and yellow indicating lower performance. Statistical summaries are provided in Tables S3–S5.

In contrast, classification of LN class III/IV vs class V showed substantially lower performance across all modeling strategies (Figure B), with AUC values ranging from 0.23 to 0.68. Clinical variables alone contributed little to prediction, reflecting considerable overlap in clinical profiles between LN subclasses and consequently limited discriminative power. PLS-DA performed the poorest overall. However, its integrated data outperformed single-modality approaches, with the integrated model achieving an AUC of 0.53, compared with 0.23 for the clinical-only model and 0.42 for the metabolomics-only model. Similarly, across other classification models, integrating metabolomics and clinical data yielded slight performance improvements (Figure B). We hypothesized that when clinical or metabolite features are used alone, informative signals may be masked by noise from many irrelevant variables, thereby reducing model performance. Data integration may allow the models to down-weight noise and emphasize a small subset of discriminative features.

Overall, PLS-DA performed worst in both classification tasks, whereas RF and DL-based methods demonstrated comparable performance across feature sets. The consistently poor performance of PLS-DA across all integration strategies highlights its sensitivity to class overlap. As a linear method, PLS-DA assumes linear class separation and maximizes covariance, making it particularly vulnerable when between-class variance is small relative to within-class heterogeneity, characteristic of LN subclassification. In contrast, RF and DL-based models exhibited greater robustness, likely due to their ability to model nonlinear interactions, and to extract complex patterns distributed across variables. Nevertheless, even these more flexible models did not achieve high discriminative power for LN subclassification, underscoring the intrinsic biological overlap between LN subclasses.

Feature Evaluation

Feature importance scores were evaluated for each modeling approach using VIP for PLS-DA, RF importance for RF, SHAP for 1D CNN, and Grad-CAM for 2D CNN. These scores were used primarily to quantify feature attribution to model predictions, which may aid biological or mechanistic interpretation. In addition, they were used to assess the stability and consistency of feature rankings across repeated model runs, and different modeling strategies.

Differences between LN patients and controls were primarily reflected by clinical variables, e.g., age and proteinuria measures (except in the Grad-CAM-CNN), which were consistently ranked among the most important predictors across modeling strategies and integration schemes (Figure ). This trend was supported by RP-based feature ranking, where age, UPCR, U_Protein (except PLS-DA), gender (only PLS-DA), and Pic/Tryp ratio showed significant RP values, indicating stable prioritization across modalities (Figure S2). Although Grad-CAM-based interpretation produced rankings that differed from the other methods, clinical variables remained the dominant predictors (Figure C and Figure S2C).

4.

Cross-modal comparison of feature importance for control vs LN classification. Feature importance rankings from models trained using integrated, clinical-only, and metabolomics-only inputs are displayed side-by-side. Panels correspond to (A) PLS-DA, (B) RF, (C) Grad-CAM CNN, and (D) SHAP-CNN models. Bars indicate the average feature importance across resampling runs, with importance scores shown at the end of each bar. Importance metrics include VIP, RF importance scores, Grad-CAM scores, and SHAP values for PLS-DA, RF, Grad-CAM CNN, and SHAP-CNN, respectively. “Cli” and “Met” denote clinical and metabolite variables, respectively.

For LN subclassification, metabolite features (e.g., Pic/Tryp and OHAnt/Tryp ratios) were ranked concomitantly with clinical variables (Figure S3) and remained among the most stable features by RP analysis despite variation in their importance scores across classification models (Figure S4). In this task, serum creatinine (Se_Cr) (in ML models, Figure S4A–B) or U_Protein (in DL models, Figure S4C–D) emerged as the only significant clinical feature. This pattern highlighted the relevance of metabolomics signals for disease subclass discrimination.

When examining cross-modal rank consistency, ML models demonstrated high agreement in feature ordering across data modalities, whereas DL models, especially the Grad-CAM-CNN exhibited lower rank consistency. For example, eGFR ranked first in the integrative model but only fifth in the clinical-only model, while the OHKyn/Tryp ratio ranked third in the integrative model but sixth in the metabolomics-only model (Figure C). The inconsistency of Grad-CAM-derived feature importance likely reflects its gradient-based nature, stochastic training effects, limited sample size, and the lack of true spatial structure in tabular or pseudoimage omics data. An example of a feature heatmap and Grad-CAM importance map is provided in Figure S5. Additionally, Grad-CAM exhibited a gradual reduction in importance scores across ranked features (Figure C), in contrast to ML-based approaches, which more clearly separated highly influential features from those of minor importance (Figure A-B). In contrast, SHAP-CNN demonstrated greater stability and closer alignment with ML-based rankings (Figure D). SHAP-based methods leverage a game-theoretic formulation to estimate feature contributions. The resulting Shapley values satisfy local accuracy and stability, making them less sensitive to gradient noise.

Collectively, these findings suggested that although integrating metabolomics did not markedly improve predictive accuracy, consistent prioritization of specific metabolites among top-ranked features provides complementary biological insight beyond clinical variables and supports the value of data integration for feature discovery and mechanistic interpretation.

Additionally, we evaluated the effect of input feature order on the stability of feature importance rankings. Clinical variables were initially ordered by renal function markers, while metabolite features were arranged according to key components of the kynurenine pathway. This baseline ordering was followed by four rounds of random feature shuffling to assess ranking robustness. For both the control vs LN classification (Figure S6) and LN subclassification (Figure S7) tasks, PLS-DA showed the highest ranking stability, followed by RF and SHAP-CNN. SHAP-CNN maintained relatively stable rankings for major features but showed greater variability among lower-ranked features. In contrast, Grad-CAM-CNN exhibited highly inconsistent rankings across feature reorderings.

Despite these ranking differences, the predictive performance of all models remained largely unaffected by feature reordering. As shown in Figure S8, strong correlations in model performance metrics were observed across all shuffling rounds for Grad-CAM-CNN. These findings suggest that although DL models can achieve robust classification performance, their feature importance rankings are less stable and reproducible than those generated by standard ML approaches.

Comparative Performance of Clinical and Metabolomics Features in GN Classification (Case 2)

Model Evaluation

In contrast to Case 1, discrimination between GN and LN cohorts was primarily driven by metabolomics features, with most ML and DL models achieving high performance across all metrics (AUC > 0.90; Figure A–C and Tables S6–S9). PLS-DA showed slightly lower performance compared with other models. These results indicate that metabolomic profiles alone were largely sufficient for discriminating between GN and LN. Clinical-only models achieved modest performance (AUC ranging from 0.70 to 0.83), and integrating clinical with metabolomics data did not further improve performance beyond metabolomics-only models. This suggests that dominant discriminative signals were primarily encoded in metabolite concentrations.

5.

Comparative classification performance across data modalities and modeling approaches. Four prediction tasks were evaluated: (A) GN vs LN, (B) GN-IgAN vs LN, (C) GN-MN vs LN, and (D) GN-IgAN vs GN-MN. Description as in Figure . Statistical summaries are provided in Tables S6–S10.

Because the GN and LN cohorts were derived from independent studies, the observed metabolomics signals potentially reflect a combination of disease-specific biology and batch/cohort effects. We addressed this by using metabolite-to-tryptophan ratios as metabolic features instead of raw concentrations, providing an initial layer of normalization against cross-study variance. However, other batch effects, artifacts, or variations in study design may still persist, highlighting the need for future downstream validation and full cross-study normalization. Consequently, the metabolomics-driven separation observed in this analysis should be interpreted as a joint reflection of disease pathology and cohort-related effects and should therefore be interpreted cautiously.

In contrast, classification of GN subtypes (IgAN vs MN) showed lower overall performance across all modeling approaches (Figure D, Tables S6 and S10). In this setting, clinical variables emerged as the dominant contributors to predictive performance, with clinical-only models achieving AUC values of 0.81–0.87 despite substantial overlap between subtypes. Metabolomics features contributed little to subtype discrimination.

PLS-DA and RF models exhibited relatively high sensitivity (approximately 90%; Figure D), indicating a preference for identifying IgAN cases, potentially reflecting the class imbalance rather than disease-specific biology (IgAN n = 70 vs MN n = 46). Such an imbalance can bias models toward the majority class, thereby increasing sensitivity for that class. Meanwhile, weighted sensitivity, which accounted for class frequencies, was lower (approximately 85%). Nevertheless, PLS-DA and RF models consistently achieved higher weighted sensitivity than the DL models (Table S6). The DL models appeared less sensitive to this imbalance and frequently produced identical sensitivity and specificity values. This pattern reflects the use of a fixed decision threshold in the DL models, resulting in equal false-positive and false-negative rates.

When the number of discriminant features was limited, PLS-DA frequently failed to identify a stable first principal component (Figure D), highlighting the intrinsic difficulty of the classification problem. In this context, the reduced performance of PLS-DA may provide a more accurate reflection of data complexity.

Feature Evaluation

Consistent with Case 1, ML models demonstrated high cross-modal rank consistency across clinical-only, metabolomics-only, and integrated modalities (Figures S9A,B–S11A,B), unlike DL models, particularly the Grad-CAM-CNN, which showed lower consistency (Figures S9C–S11C). Among the DL interpretation approaches, SHAP-CNN again exhibited greater stability and closer agreement with ML-derived rankings than Grad-CAM (Figures S9D–S11D).

Metabolite variables consistently distinguished GN from LN cohorts. RP-based analysis identified Pic/Tryp, OHKyn/Tryp (except in PLS-DA), and Xan/Tryp (only in PLS-DA) as important features across modalities (Table S11), and these remained significant in GN subtype versus LN comparisons (Tables S12 and S13). In contrast, Grad-CAM-based interpretation diverged from other methods, most notably by identifying gender, U_Protein, age, and U_Cr as top features, and Kyn/Tryp as the topmost metabolite (Figure S9C, integrated model). However, only U_Protein, gender, and Xan/Tryp showed significant RP values (Table S11).

This discordance indicates low agreement in feature ranking across modalities. For instance, gender ranked among the lowest features in the clinical-only model but highest in the integrated model, while Xan/Tryp ranked highest in the metabolomics-only model but among the lowest in the integrated model (Figure S9C). Similar patterns were consistently observed for Grad-CAM models across GN-IgAN vs LN, GN-MN vs LN, and GN-IgAN vs GN-MN classifications (Figures S10–S12, Tables S11–S13).

For GN subtype classification, clinical variables dominated the feature importance ranking. Albumin was consistently identified by ML models (Figure S12A–B), whereas metabolites such as Se/Tryp or OHAnt/Tryp were more frequently highlighted by DL models (Figure S12C–D). ML and DL rankings showed limited concordance in this task (Table S14), reflecting substantial biological overlap between GN subtypes and the difficulty in identifying stable discriminative features in this classification setting.

Model Stability and Variability across Resampling

We investigated accuracy trends across repeated resampling, which highlighted distinct resampling sensitivity among ML and DL approaches. For well-separated tasks (control vs LN and GN vs LN; Figure A and Figure S13A), all models achieved consistently high and stable accuracy, reflecting strong biological separation between classes. RF models achieved near-perfect training accuracy and highly stable test performance. The ability to construct near-pure decision trees enables RF to closely fit the training data and identify dominant discriminative structure in the data. PLS-DA also achieved high accuracy but exhibited greater sensitivity to resampling, consistent with its linear, projection-based nature. Because latent components in PLS-DA are derived from covariance structures, even small perturbations in sample composition can alter projection directions, leading to increased variability despite strong overall separation. The DL models showed minimal variability in these tasks. Given the relatively limited sample sizes (Case 1: n = 155; Case 2: n = 200), DL models may be overparameterized for such well-separated tasks. The apparent stability was likely due to task saturation, where classification boundaries were easily identified rather than improved model generalization.

6.

Accuracy curves across resampling iterations for Case 1 (integrated modality). Accuracy patterns for (A) control vs LN and (B) LN class III/IV vs class V classification tasks across repeated resampling. Solid lines represent overall trends. Mean (±SD) values of training and test accuracy are provided.

Under subtype classification tasks (LN class III/IV vs class V and GN-IgAN vs GN-MN; Figure B and Figure S13B), different patterns emerged. RF remained robust to resampling but exhibited clear train–test performance gaps, suggesting some degree of overfitting. PLS-DA showed pronounced sensitivity to sampling variability, consistent with its dependence on covariance structure, which becomes less reliable when between-class variance is small relative to within-class heterogeneity. DL models, particularly Grad-CAM-based CNNs, also became more sensitive to resampling, showing fluctuating accuracy and more pronounced discrepancies between training and test performance. These patterns suggested that CNNs may capture sample-specific structures that did not generalize well to unseen data.

Overall, the reduction in stability and generalization observed in subtype classification tasks likely reflect the intrinsic difficulty of identifying robust, discriminative features when disease subclasses share substantial metabolic and clinical characteristics, rather than methodological failure. These findings further support the broader observation that strong biological separation enables stable model performance across algorithms, whereas tasks involving subtle disease heterogeneity expose differences in model robustness and susceptibility to overfitting.

Conclusions

This study demonstrates the value of the proposed framework for multimodal integration in clinical metabolomics research. The methodological contribution of our framework lies in three aspects. First, it enables a systematic side-by-side comparison of clinical-only, metabolomics-only, and integrated models within a unified workflow. Second, model evaluation extends beyond predictive accuracy to include feature importance, resampling stability, and overfitting behavior, providing a more comprehensive assessment of model reliability and interpretability. Third, the framework supports both independent analysis of individual data modalities and integrated analysis of all available data sets within a single workflow, facilitating practical comparison of alternative modeling strategies.

By enabling comprehensive evaluation of integration strategies, predictive performance, and interpretability, the framework facilitates the combined use of classical ML and DL models. This allows model choice to be adapted to data complexity, sample size, and research objectives while leveraging congruent evidence across approaches.

Across the evaluated classification tasks, RF and DL models achieved strong predictive performance overall, whereas PLS-DA generally exhibited lower performance than other models. Notably, DL models showed strong predictive performance despite low consistency in feature importance rankings. This likely reflected decision saturation under limited sample sizes, suggesting that these models may be overpowered for this scale of the data. In contrast, ML approaches exposed uncertainty more transparently. RF showed a tendency toward overfitting, while PLS-DA was sensitive to sampling variability and, occasionally in extremely overlapping cases, failed to identify discriminant features. This sensitivity may therefore provide a more natural reflection of intrinsic data difficulty.

Taken together, classification performance was primarily driven by the strength of the underlying biological signal. Differences between models were expressed mainly through their sensitivity to resampling variation and training-set overfitting, rather than through discrepancies in overall generalization performance, as most approaches achieved comparable results. These insights into model behaviors would not have been apparent without the proposed framework.

Scope and Limitations

The minimum sample size of approximately 40 subjects per group is adequate for exploratory clinical studies but remains small for DL applications. Hyperparameter optimization was not the primary focus, and model configurations were selected to be executable on standard computational hardware. Despite these constraints, the framework was deliberately applied under realistic small sample conditions and allowed us to characterize distinct DL model behaviors commonly encountered in clinical metabolomics research, where large cohorts are often unavailable.

CNN architectures were selected because they enable evaluation of both 1D tabular inputs and 2D image-like representations of the same clinical and metabolomics data within a unified DL framework. This allowed direct comparison of the influence of input representation while minimizing architectural differences between the DL models, thereby enabling evaluation of whether CNN-based models can learn complex patterns from different data representations. Nevertheless, the relatively small sample size may limit the full potential of DL models. Simpler neural architectures or modern tabular-learning methods may be more suitable for structured data sets of this scale and should be explored in future studies.

Furthermore, as the GN and LN cohorts in Case 2 were derived from independent studies with potentially different preanalytical procedures and patient recruitment criteria, the possibility that cohort-level differences contributed to the observed metabolomics-driven separation cannot be fully excluded. Although we used metabolite-to-tryptophan ratios to mitigate cross-study variance, batch effects and baseline differences may persist. The metabolomics-driven result from this scenario likely reflects a combination of disease biology and study-specific confounding factors. These results should therefore be interpreted with appropriate caution.

Although additional ML approaches, such as support vector machine (SVM), gradient boosting algorithms, or regularized logistic regression, could be applied, the objective of this study was to compare modeling paradigms rather than to exhaustively benchmark all available classifiers. PLS-DA and RF were selected because of their extensive use in metabolomics. PLS-DA represents a linear chemometric approach, with high interpretability, and the ability to identify discriminative features through VIP scores. It also represents the class of linear classification methods, such as regularized logistic regression and SVM. RF was selected as a representative ensemble-based method that captures nonlinear relationships while providing measures of variable importance, thereby representing the tree-based ensemble algorithms, such as XGBoost and LightGBM. The 1D CNN and 2D CNN models were included to evaluate whether DL architectures could further improve predictive performance by learning complex feature representations from clinical and metabolomics data. Likewise, while many explainable artificial intelligence (XAI) methods exist, Grad-CAM and SHAP were chosen to support the interpretation of DL models, providing complementary gradient-based and game-theoretic perspectives on feature importance. Together, these four models represent distinct data analysis paradigms, enabling a systematic comparison of conventional chemometric, ensemble ML, and DL approaches. The framework is designed to be modular, and additional classification models or XAI methods could be incorporated in future extensions to broaden its applicability across different research contexts.

The absence of an independent external validation cohort limits confidence in the generalizability of the findings. Therefore, to provide a robust internal assessment, we performed repeated train-test splits and summarized model performance using mean values, 95% CI, SD, and balanced performance metrics. In addition, we evaluated feature importance consistency, resampling stability, and overfitting behavior to assess the robustness of each modeling strategy across repeated data partitions. Although these analyses cannot replace independent external validation, they provide complementary evidence regarding the stability and reliability of the proposed framework within the available data set.

The framework was implemented using R and Python. DL models were developed in Python, whereas the remaining components of the framework were implemented in R. The source code and the example data are publicly available at https://github.com/kwanjeeraw/MMFramework.

The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acsmeasuresciau.6c00108.

• Overview of the analysis workflow; feature importance analysis including RP-based feature ranking and cross-modal comparison of feature importance for Case 1; feature heatmap and Grad-CAM importance map; the effect of input feature ordering on the stability of feature importance rankings; feature importance analysis including RP-based feature ranking and cross-modal comparison of feature importance for Case 2; model stability and variability across resampling; statistical summary of pairwise group comparisons; selected mtry values for each model across 10 resamples; and RP-based feature ranking for Case 2 (PDF)

• Tables S3–S10: Statistical summaries of classification performance metrics including mean values, 95% CI, SD, balanced metrics, and paired Wilcoxon comparisons (XLSX)

K.W. and S.K. conceived and designed the study. K.W. developed the software and performed the data analysis. K.D. and N.K. conducted the experiments. S.K. supervised the analyses and provided scientific input. P.S., T.T., C.K., Y.S., and S.K. provided the resources for the study. K.W. created the illustrations. K.W. and S.K. wrote the original draft. All authors reviewed the manuscript and approved the submitted version.

This project received funding from the National Science, Research, and Innovation Fund (NSRF) via the Program Management Unit for Human Resources & Institutional Development, Research, and Innovation, Grant no. B36G660007 to S.K. This research project is supported by Siriraj research development fund, Grant number (IO) R016737002, Faculty of Medicine Siriraj Hospital, Mahidol University.

The authors declare no competing financial interest.