AR-CDT NET: a deep deformable convolutional network for gut microbiome-based disease classification

Abstract

Advances in metagenomic sequencing have increasingly implicated gut microbiome dysbiosis in numerous complex diseases, yet its application for precise differential diagnosis remains a major challenge. Existing computational approaches often show limited predictive performance and insufficient robustness when applied to large-scale, imbalanced microbiome datasets, and they typically lack mechanisms to effectively capture microbial community-level or functional guild interactions. To address these limitations, we developed AR-CDT Net, a novel deep learning framework that integrates a Multi-Scale Deformable Convolution (MS-DConv) module with a Channel-wise Dynamic Tanh (CD-Tanh) activation function to achieve more accurate and robust classification of host disease states. Evaluated on a large-scale cohort comprising over 8000 samples spanning eight disease phenotypes, AR-CDT Net demonstrated highly competitive within-cohort performance, outperforming nine representative models across the majority of classification tasks. Importantly, in a stringent cross-dataset generalization test, the model was trained on the highly imbalanced primary multi-disease cohort and validated on relatively balanced independent external cohorts. It achieved a statistically significant AUC of 0.7921 on the highly heterogeneous external T2D cohort, confirming that AR-CDT captures transferable biological signals rather than dataset-specific artifacts. Furthermore, by combining dimensionality reduction with SHAP-based interpretation of our One-vs-Rest (OvR) classifiers, AR-CDT disentangles disease-specific pathogenic signatures from the shared dysbiotic background among clinically distinct yet microbially similar diseases.

Introduction

The human gut microbiome’s pivotal role in host metabolism and immunity means structural dysbiosis is linked to numerous major diseases [1]. Distinct microbial shifts characterize various conditions: multiple sclerosis (MS) involves alterations in taxa like Methanobrevibacter and Butyricimonas [2]; coronary artery disease (CAD) is marked by an enrichment of opportunistic pathogens and a depletion of beneficial butyrate-producers like Roseburia [3]; and fatty liver disease shows increased abundances of Proteobacteria, Veillonella, and Streptococcus [4].

Various computational methods exist to classify diseases from these microbial shifts. Early tools, such as MetAML [5], applied traditional machine learning algorithms like Random Forests and Support Vector Machines with hyperparameter optimization via grid search. More recent approaches leverage deep learning: GDmicro [6] utilizes Graph Convolutional Networks combining semi-supervised learning and domain adaptation, while PopPhy-CNN [7] integrates phylogenetic data into a novel CNN framework.

Despite their success, existing computational tools face two primary challenges. First, they inadequately model complex microbial ecologies. Many methods treat microbial features as independent variables (e.g., MLPs) or use fixed structural assumptions (e.g., phylogenetic tree-based CNNs), thus ignoring the synergistic and antagonistic relationships that form functional guilds [8, 9]. Such simplistic assumptions hinder their capacity to capture intricate biological patterns.

Second, models struggle with robustness and generalization when applied to large-scale, real-world cohorts. These datasets are often characterized by significant class imbalance, such as a large majority of healthy controls, and inter-study technical variations resulting from the aggregation of multi-center cohorts. Moreover, models that perform well on smaller, balanced testbeds often suffer significant performance degradation when applied to such large-scale, real-world data, particularly in identifying rare disease classes.

To address these challenges, we propose AR-CDT, a novel deep learning architecture. To tackle the first challenge, the Multi-Scale Deformable Convolution (MS-DConv) module employs parallel branches with adaptive receptive fields to flexibly capture diverse microbial community structures. For the second challenge, the Channel-wise Dynamic Tanh (CD-Tanh) activation function enhances feature discrimination and improves the model’s robustness by maintaining stable performance across heterogeneous and noisy data, while also increasing its resilience to severe class imbalance. This combined design strengthens the model’s ability to generalize to large-scale, real-world microbiome cohorts. We systematically demonstrate AR-CDT’s superior performance and leverage its explainability to identify disease-specific microbial signatures on large-scale, imbalanced data.

Methods

In this section, we first describe the representation and preprocessing of the microbial abundance data. Next, we outline the proposed AR-CDT architecture, a deep learning model designed for microbiome-based disease classification. We then present our biomarker-discovery-oriented One-vs-Rest (OvR) framework and the subsequent SHAP-based interpretability analysis. Following this, we summarize the dataset sources and composition used in this study. Finally, we detail our comprehensive, multi-level validation framework, which evaluates within-cohort stability, cross-dataset generalization, and modular contributions through ablation studies.

Input representation and preprocessing

Microbial abundance data are inherently sparse and compositional. Directly using raw abundance values in machine learning models can lead to imbalanced feature distributions, spurious correlations, and numerical instability. The input for these models is the microbial abundance matrix , where N is the number of samples and F is the number of taxa.

To mitigate these challenges, particularly those posed by compositionality, we apply a Centered Log-Ratio (CLR) transformation [10]. The CLR transformation for a feature within a sample’s abundance vector is defined as:

1

where denotes the geometric mean of the vector, and is a small pseudo-count (e.g., ) added to handle zero values. This transformation projects the compositional data from a simplex into Euclidean space, making it suitable for standard feature comparison and distance-based machine learning models.

AR-CDT Net

Our proposed AR-CDT Net is a novel deep learning model designed for microbiome-based disease classification. The model processes multiple receptive fields of different sizes in parallel and dynamically learns the spatial offsets of sampling points for each scale path, thereby flexibly capturing microbial community signals of varying sizes and shapes. Next, the extracted high-order features are fed into a classifier that integrates our Channel-wise Dynamic Tanh (CD-Tanh) activation function to achieve precise classification. This design enables the model to comprehensively capture complex intra-sample microbial interaction patterns. A schematic diagram of the AR-CDT Net is shown in Fig.  1.

Fig. 1.

The overall workflow of AR-CDT Net. The input consists of microbial relative abundance data. Parentheses indicate the tensor dimensions, while the small red, yellow, and blue squares denote the positions of different elements within the tensor. The process enclosed by the dashed line illustrates the workflow of the Channel-wise Dynamic Activation module. F is the number of original input features (i.e., microbial taxa, as defined in Section “Input representation and preprocessing”). S is the side-length of the 2D pseudo-image (e.g., S = 8), which is generated from the F-dimensional features as detailed in Section “Model evaluation and validation” C is the number of output channels from the MS-DConv module (e.g., C = 64). H is the dimension of the hidden layer within the final classifier (e.g., H = 32)

Multi-scale deformable convolution (MS-DConv)

A core challenge in analyzing microbiome data lies in effectively modeling the complex interactions among microbial species. These species do not act independently; rather, they influence the host phenotype by forming synergistic or antagonistic functional guilds. While traditional fully connected networks can implicitly learn feature interactions, they cannot explicitly model the structure of these functional guilds. Standard convolutional neural networks, on the other hand, are not directly applicable, as microbial features lack the inherent spatial topology of image data. To address this limitation, we propose a novel Multi-Scale Deformable Convolution (MS-DConv) module, inspired by LDConv [11]. Our module dynamically learns sampling point offsets from the input features, enabling it to capture microbial functional guilds of variable scales and shapes. Unlike standard convolutions, which are constrained to a rigid, grid-like receptive field, the deformable nature of our module allows its sampling points during training to learn to move freely and focus on the most informative microbial features, thereby overcoming the limitations imposed by their fixed, arbitrary position in the feature vector. This effectively creates a data-driven, adaptive receptive field that can conform to the irregular, non-local structure of a true functional guild.

The original LDConv module is limited as it operates on a single, fixed receptive field scale. In complex microbial ecosystems, however, interactions occur across a spectrum of scales: some are localized among a few species, while others manifest as broader, community-wide synergistic effects. To capture these multi-scale dynamics, our proposed MS-DConv processes features through parallel branches, each with a different receptive field size, and employs an attention mechanism to adaptively fuse their outputs. This design allows the model to simultaneously capture both fine-grained local interactions and macroscopic community effects, yielding a more holistic feature representation. For example, a 3 × 3 receptive field might capture a compact functional guild of a few core species, while a 7 × 7 receptive field could cover a more diffuse ecological network involving many species. By moving beyond the single-scale limitation of the original LDConv, our approach is better aligned with the multi-level, multi-scale nature of microbial ecology. A schematic of the MS-DConv architecture is provided in Fig. 2.

Fig. 2.

The MS-DConv module. The module consists of two paths. In Path A, a convolution first learns 2D offsets, which are used to create a deformed sampling grid for each scale. This allows the module to resample features from key non-local regions. These resampled features are then processed by different-sized convolutional kernels (e.g., 3 × 3, 5 × 5, 7 × 7), and the resulting multi-scale outputs are fused via an SE attention mechanism. In parallel, Path B learns a modulation gate (m) and a bias gate (b). The final output is a gated combination of the two paths. The colored squares in the figure (red, yellow, blue) represent the sampling positions of the deformable points on the 2D pseudo-image. The visualization of different colors, for example red versus blue, is intended to illustrate how the sampling points learned by different scale branches can move freely and cover different regions

The MS-DConv module is composed of two parallel paths, Path A and Path B, which are designed to first learn an adaptive receptive field and subsequently process features through it. In the core deformable sampling path (Path A), a small convolutional sub-network dynamically learns a 2D offset field from the 2D pseudo-image features passed from the preceding layer. This offset field defines a deformed sampling grid. This grid liberates the subsequent convolutional operation from the fixed, rigid neighborhood constraints of the 2D pseudo-image, enabling it to sample from feature locations that are biologically relevant yet potentially spatially distant. The features sampled via this deformed grid are then processed by parallel convolutional kernels of three different sizes to capture multi-scale patterns, and their respective outputs are finally fused using a Squeeze-and-Excite (SE) attention mechanism. Concurrently, the parallel gating path (Path B) functions as a gating mechanism, employing two independent convolutions to learn a modulation gate () and a bias gate (). The final output of the module is a gated combination of these two paths, formulated as:

2

which balances the learned deformable features through this gating process.

Channel-wise dynamic tanh (CD-tanh)

Following feature extraction by the MS-DConv module, an effective classifier is required to map the resulting high-level features to the final classifications. Traditional classifiers often pair a fully connected layer with a standard normalization and a fixed activation function, such as BatchNorm and GELU. We contend, however, that such a static activation strategy is suboptimal for the complex, high-order features produced by MS-DConv. Our work therefore builds upon the dynamic activation layer proposed in [12], which introduces a learnable global parameter , to control the steepness of the function, thereby adding a degree of adaptability.

Despite this innovation, the original design is limited by its use of a single, shared steepness parameter across all feature channels, which precludes channel-specific modulation. To overcome this, we propose the Channel-wise Dynamic Tanh (CD-Tanh). Given an input feature vector , the CD-Tanh module computes its output as:

3

where denotes the element-wise product, and are learnable, channel-wise parameters for steepness, scaling, and shifting, respectively.

Our core innovation is the extension of the original global scalar into a learnable vector that allows for channel-wise steepness adaptation. This design enables the network to perform implicit feature selection during training. For channels deemed important for the classification task, a large learned value pushes their outputs toward the saturation regions of the function, creating a strong, switch-like signal. Conversely, for less relevant channels, a smaller value suppresses their outputs toward zero, effectively gating them off. This transition from global to channel-wise dynamics enhances the classifier’s flexibility and capacity to model complex features, with negligible added computational overhead.

Biomarker discovery oriented one-vs-rest (OvR) framework

In complex microbiome datasets containing multiple disease states, traditional multi-class models struggle to specifically identify microbial markers that can distinguish a single disease from other diseases and healthy states. To solve this problem and to systematically mine for microbial features related to the specificity of each phenotype, this study designed and employed a biomarker discovery-oriented One-vs-Rest (OvR) computational strategy. By training a dedicated binary classifier for each class to distinguish it from all other classes in the dataset, we force the model to learn the most specific microbiome signals for that class, laying a solid foundation for the downstream discovery of specific microbial markers.

SHAP analysis based on OvR classifier

Shapley Additive Explanations (SHAP) [13], proposed by Lundberg and Lee in 2017, is a method for interpreting the outputs of a wide range of complex models. SHAP is a method based on cooperative game theory that attributes the model’s prediction output to each input feature. The calculated values represent the effect of a feature pushing the model output higher or lower. In this study, we performed SHAP analysis on the trained OvR classifiers, aiming to identify the key microbes that specifically distinguish each class. To evaluate the global importance of each microbe, we ranked them by calculating the mean absolute SHAP value across all explained samples and visualized the top 20 most important features and their SHAP value distributions for each OvR model. To validate these findings, we performed a corresponding One-vs-Rest differential abundance analysis using Linear Discriminant Analysis Effect Size (LEfSe).

Model evaluation and validation

We designed a comprehensive, multi-level validation framework to rigorously assess the performance, robustness, and generalization capability of the AR-CDT architecture. Our primary performance evaluation was conducted on the large-scale, multi-study cohort using a 10-fold stratified cross-validation protocol. This procedure ensured that the original class distribution, including the severe imbalance ratio, was precisely preserved within each training and testing fold.

We acknowledge that, given this cohort integrates samples from multiple independent studies, a Leave-Study-Out (LSO) validation would represent the most stringent approach for assessing cross-study generalization. However, the LSO strategy was statistically infeasible for most disease tasks due to severe structural confounding between study origin and disease label in the public datasets. In many cases, the integrated cohort included studies that contained only a single class, either all disease samples or all healthy controls, which made the validation Area Under the Curve (AUC) computation mathematically impossible in those folds. Moreover, several disease tasks were derived from only one complete study, which made any cross-study partitioning impossible. Therefore, LSO evaluation was only conducted for disease tasks where multiple studies contained both cases and controls, while 10-fold cross-validation was used to evaluate within-cohort stability and relative performance across all tasks.

For example, in the T2D task, 66 out of 75 studies consisted exclusively of either patient or control samples. Even among the few studies that contained both cases and controls, the class distribution remained heavily imbalanced. This imbalance reflects the inherent heterogeneity and sampling bias of public microbiome datasets. Consequently, the stratified 10-fold cross-validation was adopted to provide a statistically stable evaluation of within-cohort performance, ensuring that each fold preserved the global disease-to-control ratio while allowing fair comparison against baseline models under identical conditions.

To provide a definitive and rigorous assessment of cross-dataset generalization, we further conducted an external validation experiment. Specifically, the T2D-specific model was trained on the T2D samples within the primary cohort and then evaluated on the independent MetAML T2D dataset. A stringent species-level feature intersection was applied beforehand to ensure consistent input dimensions and to completely eliminate any possibility of information leakage across datasets. Other disease tasks were excluded from this analysis because their external datasets did not meet the minimum case–control balance required for meaningful evaluation.

Furthermore, to demonstrate the robustness of the proposed architecture across datasets with varying characteristics, such as smaller sample sizes and more balanced class distributions, we also evaluated its performance using 10-fold cross-validation within each of the three public MetAML benchmark datasets. Finally, to quantify the individual contributions of the proposed MS-DConv and CD-Tanh modules, we performed a series of systematic ablation experiments, including removing each module independently, replacing the deformable MS-DConv with standard fixed-shape convolutions, and simplifying the multi-scale configuration from the full [3, 5, 7] design to dual-scale and single-scale variants.

The AR-CDT architecture first projects input microbial features into a 64-dimensional latent space using a fully connected (Dense) layer. This latent representation is then reshaped into an 8 × 8 pseudo-image, which serves as the input for the MS-DConv module configured with 64 output channels and kernel sizes of [3, 5, 7]. Model optimization and early stopping were performed based on the AUC. Given the significant global class imbalance in our primary cohort, we employed a Focal Loss (with a of 2.0) as our primary balancing strategy for all training experiments. All detailed training configurations are provided in the Supplementary Materials.

Datasets

For our primary analysis, we constructed a large-scale, multi-disease gut metagenomic cohort to systematically evaluate the performance of our proposed model. This cohort, integrated from the curatedMetagenomicData package [14], comprises 8376 samples spanning eight clinical phenotypes: 5000 Healthy Controls (HC) and 3376 patients with seven major diseases, including: type 2 diabetes (T2D), inflammatory bowel disease (IBD), colorectal cancer (CRC), multiple sclerosis (MS), coronary artery disease (CAD), fatty liver, cirrhosis.

In addition to this primary cohort, we used three independent public datasets for architecture validation. These datasets, sourced from the MetAML package [5], include cirrhosis, T2D, and obesity. In contrast to our primary cohort, they are characterized by smaller sample sizes and relatively balanced case–control ratios. A comprehensive summary of all datasets used in this study is presented in Tables 1 and 2. A detailed study-level breakdown of this primary cohort, showing the number of cases and controls contributed by each individual study, is provided in Supplementary Table S2.

Table 1.

Summary of datasets for primary performance evaluation

Experiments	Dataset	Total samples
Primary performance evaluation	Health	5000
	T2D	894
	IBD	704
	CRC	682
	MS	365
	CAD	363
	Fatty Liver	157
	Cirrhosis	131

Table 2.

Summary of datasets for validating model generalizability

Dataset	Total samples	Case samples	Control samples
Cirrhosis	232	114	118
T2D	344	170	174
Obesity	253	164	89

Results

We systematically evaluated the performance and robustness of the AR-CDT model on large-scale, class-imbalanced microbiome data. We evaluated its binary classification performance on single disease vs HC separately by 10-fold cross validation. To verify the advancement of the model we proposed, we selected nine of the most representative tools in the current field, including MetAML_rf, MetAML_svm  [5], GDmicro [6], PopPhy-CNN [7], MicroHDF [15], CNN1D [16], MLPNN [17], DeepMicrobiome [18] and LASSO (Least Absolute Shrinkage and Selection Operator), and conducted performance benchmark tests with our designed AR-CDT under the same evaluation protocol. To ensure the fairness of the comparison, all models were run with their best publicly recommended parameter configurations. Next, we present the results of our hierarchical generalization analyses. We conducted a LSO validation on the primary multi-study cohort to assess cross-study generalization within the same dataset. This evaluation provides a realistic measure of robustness under study-level domain shifts and is feasible only for disease tasks that include multiple mixed-type studies (T2D, IBD, and CRC). We then report the results of our cross-dataset generalization test, in which models trained on the large primary cohort were evaluated on independent MetAML datasets. This analysis serves as the most stringent assessment of real-world generalization. Subsequently, we evaluate AR-CDT under the more challenging One-vs-Rest (OvR) classification framework, which serves as the foundation for our interpretability analyses. Using this OvR framework together with t-SNE and UMAP visualizations, we explore the microbial similarities and differences among disease phenotypes. For diseases showing overlapping microbial signatures, we further apply SHAP-based feature attribution to disentangle their shared dysbiotic backgrounds from disease-specific pathogenic patterns. Finally, we summarize the results of our systematic ablation experiments, which quantify the contributions of the MS-DConv and CD-Tanh modules and confirm the importance of the model’s multi-scale, deformable architecture design.

AR-CDT model shows excellent within-cohort classification performance

To evaluate its performance, we benchmarked AR-CDT on a series of large-scale, class-imbalanced binary classification tasks, each designed to distinguish a specific disease from HC. These datasets are particularly challenging because their scale and imbalance are known to induce overfitting or majority-class bias in conventional models. Following previous studies [5, 6, 19, 20], we employed 10-fold cross-validation and used the AUC as the primary evaluation metric.

As shown in Table 3 and Fig. 3, AR-CDT demonstrated highly competitive performance, outperforming the nine representative models across the majority of classification tasks. Across all disease tasks, AR-CDT achieved excellent and stable discriminative performance, with particularly strong results on the cirrhosis classification task (mean AUC = 0.988 ± 0.009). These findings indicate that AR-CDT effectively mitigates the challenges posed by large-scale, imbalanced microbiome data and validate the overall robustness of its architectural innovations (specifically, the MS-DConv and CD-Tanh components) as a reliable framework for complex disease classification. To comprehensively assess practical applicability, we further compared the computational efficiency of all benchmarked models. A detailed comparison of trainable parameters, training time, and inference speed is provided in Supplementary Tables S4–S10.

Table 3.

Mean AUC (10-fold cross-validation) of 10 methods on 7 disease cohorts. Values in parentheses indicate the standard deviation. Bold: Best performance in each cohort. Underline: Second-best performance in each cohort

Dataset	AR-CDT	PopPhy-CNN	GDmicro	MetAML_rf	MetAML_svm	MicroHDF	CNN1D	MLPNN	LASSO	DeepMicrobiome
T2D	0.8787 (0.0225)	0.8016 (0.0437)	0.8234 (0.0313)	0.7789 (0.1350)	0.7254 (0.0862)	0.8838 (0.0305)	0.7996 (0.0357)	0.8027 (0.0342)	0.5000 (0.0000)	0.7856 (0.0374)
IBD	0.8877 (0.0145)	0.8475 (0.0227)	0.8452 (0.0233)	0.8663 (0.0313)	0.8364 (0.0271)	0.8866 (0.0248)	0.8440 (0.0250)	0.8460 (0.0181)	0.8035 (0.0280)	0.8416 (0.0253)
CRC	0.9279 (0.0167)	0.8097 (0.0399)	0.8322 (0.0314)	0.9069 (0.0199)	0.8384 (0.0287)	0.9038 (0.0211)	0.7915 (0.0478)	0.8080 (0.0352)	0.6935 (0.1080)	0.8003 (0.0203)
MS	0.9551 (0.0129)	0.9071 (0.0247)	0.8734 (0.0205)	0.7336 (0.2338)	0.7300 (0.1896)	0.9593 (0.0134)	0.8762 (0.0357)	0.8893 (0.0195)	0.5421 (0.0455)	0.9114 (0.0207)
CAD	0.9568 (0.0090)	0.9071 (0.0224)	0.8394 (0.0308)	0.9065 (0.0682)	0.8015 (0.1396)	0.9552 (0.0097)	0.8875 (0.0243)	0.8984 (0.0229)	0.5000 (0.0000)	0.9082 (0.0218)
Fatty Liver	0.9379 (0.0330)	0.8558 (0.0528)	0.8985 (0.0459)	0.7435 (0.1951)	0.7214 (0.1555)	0.9109 (0.0347)	0.8319 (0.0533)	0.8331 (0.0655)	0.5905 (0.0299)	0.8412 (0.0615)
Cirrhosis	0.9875 (0.0094)	0.9434 (0.0311)	0.9245 (0.0395)	0.9853 (0.0149)	0.9241 (0.0238)	0.9788 (0.0176)	0.9507 (0.0163)	0.9263 (0.0284)	0.7959 (0.1962)	0.9430 (0.0304)

Fig. 3.

AUC performance across seven disease phenotypes. Subplots a–g show the AUC performance of AR-CDT and nine baseline models on T2D, IBD, CRC, MS, CAD, fatty liver, and cirrhosis, respectively

Cross-study generalization assessed by LSO validation

We performed a LSO validation within the primary cohort to directly assess the model’s ability to generalize across different studies. In this evaluation, each study was iteratively held out for testing while the remaining studies were used for training, thereby simulating a realistic cross-study prediction scenario. The LSO analysis was statistically feasible only for disease tasks that included multiple studies containing both cases and controls. This condition was met for three diseases: T2D, IBD, and CRC. The corresponding LSO performance results are presented in Table 4, whereas Supplementary Table S2 provides the detailed study-level composition of the primary multi-disease cohort.

Table 4.

LSO cross-validation results on the primary cohort. The table summarizes the number of usable studies and the corresponding Mean Validation AUC for the three disease tasks where LSO evaluation was statistically feasible, namely those with multiple studies containing both case and control samples

Disease	Usable studies	Mean validation AUC
T2D	9	0.7617 ± 0.1006
IBD	6	0.73 ± 0.1113
CRC	11	0.7211 ± 0.1204

Across these three tasks, AR-CDT maintained consistent discriminative performance, achieving mean validation AUCs of approximately 0.72–0.76. Although these values are lower than those obtained under standard 10-fold cross-validation, this reduction is fully expected given the much harsher conditions of the LSO setting. In this framework, each held-out study often exhibits uneven class ratios and limited sample sizes. For instance, the case-to-control ratios ranged from approximately 1.6:1 to 4.6:1 in T2D, 0.1:1 to 3.8:1 in IBD, and 0.4:1 to 2.0:1 in CRC. Such extreme and heterogeneous imbalance across studies makes certain LSO folds statistically underdetermined, particularly when one class is represented by only a few samples. Together with the complete removal of any shared batch or study-level information between training and testing, these factors naturally constrain the achievable AUC while providing a far stricter assessment of true generalization. Nevertheless, AR-CDT maintained stable and biologically meaningful discrimination across these heterogeneous studies, demonstrating its ability to capture transferable microbial patterns rather than relying on study-specific artifacts.

For the remaining disease tasks (MS, CAD, fatty liver, and cirrhosis), LSO evaluation was not feasible because each dataset originated from a single study, which precluded cross-study partitioning.

AR-CDT demonstrates out-of-distribution generalization

To rigorously evaluate the model’s ability to generalize beyond its training distribution, we conducted a cross-dataset validation focusing on the T2D task. In this experiment, AR-CDT was trained on the T2D samples within the original large-scale, class-imbalanced and study-confounded cohort, and then tested on the independent external T2D dataset after strict feature alignment. As shown in Fig. 4, the model achieved an AUC of 0.7921 on this external dataset. Although the value is moderate, this result remains noteworthy given the considerable difficulty of this setting, since the model was trained on a large, class-imbalanced, and study-mixed primary cohort, conditions that typically hinder stable generalization, yet it still achieved consistent predictive ability when applied to an entirely independent external dataset. This demonstrates that AR-CDT successfully transfers biologically meaningful features learned from heterogeneous training data to unseen cohorts.

Fig. 4.

Cross-dataset generalization performance of AR-CDT. Receiver operating characteristic (ROC) curves for models trained on the T2D samples within the primary cohort and tested on independent external T2D datasets

Together with the complementary LSO validation results on the primary cohort (AUC 0.72 to 0.76 across feasible disease tasks), this external evaluation provides converging evidence that AR-CDT captures robust, biologically transferable patterns rather than dataset-specific artifacts, supporting its real-world generalizability.

Architecture robustness on public benchmark datasets

Furthermore, to demonstrate that the architecture’s robustness also extends to datasets characterized by smaller sample sizes and more balanced class distributions, we also assessed its performance using a 10-fold cross-validation within each of the three independent public MetAML datasets. As detailed in Table  5 and Fig.  5, AR-CDT exhibited superior and robust classification performance across all three datasets, consistently outperforming the competing models on the AUC metric.

Table 5.

Mean AUC (10-fold cross-validation) of 10 methods on 3 disease cohorts. Values in parentheses indicate the standard deviation. Bold: Best performance in each cohort. Underline: Second-best performance in each cohort

Dataset	AR-CDT	PopPhy-CNN	GDmicro	MetAML_rf	MetAML_svm	MicroHDF	CNN1D	MLPNN	LASSO	DeepMicrobiome
Cirrhosis	0.9636 (0.0511)	0.8772 (0.0846)	0.9038 (0.0652)	0.9273 (0.0600)	0.8927 (0.0784)	0.9345 (0.0480)	0.8077 (0.0957)	0.7982 (0.1317)	0.8592 (0.0719)	0.8684 (0.0839)
T2D	0.7660 (0.0571)	0.7031 (0.0993)	0.7568 (0.0592)	0.6935 (0.0979)	0.6177 (0.1737)	0.7536 (0.0934)	0.6620 (0.0747)	0.6928 (0.0635)	0.7303 (0.0939)	0.7111 (0.0954)
Obesity	0.7111 (0.0954)	0.5811 (0.0969)	0.6882 (0.0678)	0.6204 (0.1062)	0.6225 (0.0942)	0.6637 (0.0936)	0.6045 (0.0460)	0.6176 (0.0340)	0.5338 (0.0694)	0.5667 (0.1095)

Fig. 5.

Validation of model performance on external datasets. Subplots a–c show the comparative AUC performance of AR-CDT and nine baseline models on three independent public cohorts: a Cirrhosis, b T2D, and c Obesity. Performance was evaluated using a 10-fold stratified cross-validation protocol, and shaded areas represent the mean ± standard deviation

This outcome confirms the strong robustness of our proposed AR-CDT architecture. It demonstrates that the model’s superiority is not limited to its proficiency with large-scale, imbalanced data, and its unique components, the MS-DConv module and the CD-Tanh activation function, are also highly effective at capturing key biological signals in traditional, smaller-scale, balanced datasets.

OvR framework reveals heterogeneity in inter-class distinguishability

To further probe our model’s performance in a multi-class environment and to systematically evaluate the uniqueness of each phenotype’s microbial signature, we employed a OvR evaluation framework. Within this framework, we used 10-fold cross-validation to train a dedicated binary classifier for each of the eight classes.

The results, presented in Table 6 and Fig. 6, reveal significant heterogeneity in the microbial distinguishability of different clinical phenotypes. Overall, all OvR classifiers demonstrated strong performance (mean AUC ), confirming AR-CDT’s ability to learn class-specific signals. Some phenotypes exhibited highly distinct signatures; for example, the cirrhosis classifier achieved a mean AUC of 0.9782 ± 0.0161 and a recall of 0.7330 ± 0.0853, indicating that this disease is easily separable from all other conditions. Similarly, the HC classifier showed robust performance. Its recall (0.9410 ± 0.0262) was among the highest of all classes, suggesting a stable and discernible microbial community structure in the healthy state.

Table 6.

Performance metrics (AUC, F1, Recall) of the eight One-vs-Rest (OvR) classifiers. Values in parentheses represent the margin of error

OvR	AUC	F1	Recall
Healthy vs. Rest	0.8405 (0.0113)	0.8047 (0.0095)	0.9410 (0.0262)
T2D vs. Rest	0.8360 (0.0312)	0.4204 (0.0330)	0.7561 (0.0825)
IBD vs. Rest	0.8971 (0.0109)	0.4960 (0.0351)	0.6291 (0.0775)
CRC vs. Rest	0.9279 (0.0151)	0.5642 (0.0380)	0.6289 (0.0699)
MS vs. Rest	0.9126 (0.0207)	0.3638 (0.0529)	0.6878 (0.1671)
CAD vs. Rest	0.9257 (0.0193)	0.4450 (0.0338)	0.6038 (0.0922)
Fatty Liver vs. Rest	0.9240 (0.0405)	0.4027 (0.0766)	0.4796 (0.1495)
Cirrhosis vs. Rest	0.9782 (0.0161)	0.7851 (0.0520)	0.7330 (0.0853)

Fig. 6.

Performance of the eight OvR classifiers. Each classifier is presented as a group on the x-axis. Within each group, three separate bars represent AUC, F1-score, and Recall, respectively. The precise value is annotated on top of each bar, and error bars are included to show the confidence interval of the data

In contrast, the OvR classifiers for MS, CAD, and fatty liver showed relatively low F1-scores and recall rates despite maintaining high AUC values. This is a classic indicator of inter-class confusion, suggesting potential similarities among their microbial signatures.

Diagnostic and SHAP analysis of the confusable disease group

To investigate the inter-class confusion among MS, CAD, and fatty liver, we visualized their feature space. We extracted the relevant samples and applied two non-linear dimensionality reduction algorithms, t-SNE and UMAP, to project their CLR-transformed microbiome features into a two-dimensional embedding. As shown in Fig. 7, both algorithms produced consistent visualizations. A corresponding t-SNE plot of the entire cohort is provided in Fig. 8. In these plots, the sample points for MS and CAD formed a single, dense cloud with highly overlapping distributions, indicating substantial similarity in their microbial community structures. In contrast, while spatially proximal, the fatty liver samples aggregated into a partially separated cluster. This visual evidence corroborates our earlier inference that the classification difficulty between MS and CAD stems from a shared dysbiotic background, while the signature of fatty liver, though containing unique elements, is not distinct enough to overcome the overall classification challenge, explaining its lower OvR performance.

Fig. 7.

Comparison of t-SNE and UMAP visualizations showing the distribution of different diseases. The distribution patterns of different disease samples in the high-dimensional data space are visually compared using two dimensionality reduction techniques, t-SNE and UMAP. Each point represents an individual sample, and different colors indicate its corresponding disease class

Fig. 8.

t-SNE visualization of the primary cohort (N = 8376) illustrating the distribution of the MS, CAD, and fatty liver phenotypes in the context of the entire cohort. Samples from these three diseases are highlighted in distinct colors against a background of all other phenotypes (shown in gray). This visualization confirms the overlap between MS and CAD and the partial separation of the fatty liver cluster, even in the context of the full dataset

To identify the key microbial contributors to the observed similarities and differences, as presented in Fig. 9, we performed a SHAP analysis on the OvR models for MS, CAD, and fatty liver. For both MS and CAD, the analysis revealed a shared dysbiotic signature: a marked depletion of key butyrate-producing taxa, although the specific microbes involved differed. This finding aligns with the recognized role of impaired intestinal barrier function in driving the chronic systemic inflammation common to both autoimmune and cardiovascular diseases [21, 22]. Despite this shared feature, the models revealed disease-specific patterns. For example, while the depletion of the protective genus Ruminococcus was a significant risk signal in both models, the MS model specifically highlighted the loss of Butyricimonas. The CAD model, in contrast, emphasized the protective effects of several Bacteroides species. Thus, while the key taxa responsible differ, the functional consequence is the same: an impaired capacity of the gut microbiome to convert dietary fiber into butyrate, a key anti-inflammatory molecule [23]. The importance of this shared defect cannot be overstated. Butyrate is not only the primary energy source for colonocytes but is also crucial for maintaining intestinal barrier integrity and reducing systemic inflammation [24]; moreover, as an HDAC inhibitor, it promotes the differentiation of immunosuppressive regulatory T cells (T-regs) [25]. Therefore, the depletion of these butyrate-producers represents not just the loss of a few species, but a systemic collapse of a critical anti-inflammatory and immunoregulatory function. We propose that this shared functional impairment places both MS and CAD patients in a similar pro-inflammatory state, which is the underlying reason for their confounding in the feature space.

Fig. 9.

SHAP summary plots for the OvR models of MS, CAD, and fatty liver. Features on the y-axis are ranked by their global importance (mean absolute SHAP value). For each feature, a single point represents a sample, with its color indicating the feature’s relative abundance (red: high; blue: low) and its horizontal position showing the impact on the model’s output (the SHAP value). A positive SHAP value contributes toward predicting the disease state, while a negative value contributes toward the ’Rest’ state

Despite the shared dysbiotic foundation, the model’s high AUC scores indicate it successfully learned to leverage unique microbial signatures for each class. A prime example is Lawsonibacter asaccharolyticus, which exhibited a context-dependent role: in the MS model, its increased abundance was a risk signal, possibly reflecting opportunistic proliferation in a neuroinflammatory environment [26], whereas in the CAD model, it was protective, perhaps due to its beneficial butyrate-producing capacity in the context of metabolic inflammation [27]. The models also identified disease-specific risk markers consistent with existing literature. For the CAD model, an increased abundance of Collinsella emerged as a strong, unique risk signal, aligning with its established role in pro-inflammatory and metabolic syndromes like atherosclerosis [28, 29]. Conversely, for the MS model, Phascolarctobacterium faecium was identified as a risk-driving microbe, consistent with reports of elevated Phascolarctobacterium abundance in other neurodegenerative diseases [30]. It is precisely that these unique microbial features provide the biological foundation for the model’s accurate discrimination between these otherwise confounded diseases.

While fatty liver shares a dysbiotic background with MS and CAD, our SHAP analysis revealed its distinct microbial signature. Paradoxically, an increased abundance of typically beneficial, SCFA-producing genera like Ruminococcus, Lachnospira, and Coprococcus was positively correlated with disease risk. This finding is consistent with observations in advanced fibrosis, which involves not a simple depletion of beneficial microbes but a complex functional remodeling of core communities under pathological stress [4]. We hypothesize that in the context of a diet driving NAFLD, this reflects an over-fermentation that produces an imbalanced SCFA profile, where acetate and propionate can serve as substrates for hepatic de novo lipogenesis (DNL) and thus exacerbate fat accumulation in the liver [31]. Furthermore, our model identified risk microbes indicative of Gut-Liver Axis dysfunction, such as the proliferation of specific Clostridium members (e.g., Clostridium sp. CAG:58). This aligns with reports of endogenous alcohol production by gut bacteria, which can directly damage the liver and drive NAFLD progression [32]. Thus, the microbial signature of fatty liver is distinguished by a dual pathological mechanism: an aberrant shift in SCFA metabolism that provides pro-lipogenic substrates, combined with the production of hepatotoxic endogenous alcohol. This transformation of the gut-liver relationship–from symbiotic support to direct antagonism–likely contributes significantly to the pathogenesis of fatty liver disease.

A comparison of the top 20 SHAP-identified features with our independent LEfSe validation revealed a complex relationship (see Supplementary Tables S4–S6). While the two methods showed strong consistency for many core biomarkers (e.g., Phascolarctobacterium faecium in MS), SHAP also identified important model-exclusive features, such as Ruminococcus bromii, that LEfSe did not detect. Most strikingly, the analysis uncovered critical contradictions. A prime example is Butyricimonas virosa, which LEfSe found to be statistically enriched in the MS cohort, yet our model unequivocally assigned a protective role. This discrepancy strongly suggests that while linear statistical methods capture direct correlations, advanced architectures like AR-CDT are necessary to uncover the more complex, interaction-based effects that define a microbe’s context-dependent functional role.

In summary, our SHAP analysis provides a mechanistic explanation at the microbial feature level for the macro-level phenomena observed in the model’s performance and visualizations. It reveals that the difficulty in distinguishing between MS and CAD is not a limitation of the model architecture, but rather a consequence of their shared dysbiotic background. Fatty liver, conversely, is characterized by a core pathogenic signal centered on Gut-Liver Axis dysfunction, which allows its samples to form a partially distinct cluster. However, its shared dysbiotic foundation with MS and CAD explains why this cluster remains spatially adjacent to the primary MS/CAD cloud in the feature embedding.

Ablation study

To systematically quantify the contribution of each core innovative component, we conducted a series of rigorous ablation studies on the large-scale main dataset. We progressively removed or simplified the model’s key modules and directly compared their performance against the final, complete model (Full Model). The evaluation for all models was performed on seven binary classification tasks using a uniform 10-fold cross-validation strategy. The results of these ablation studies are clearly presented in Table  7.

Table 7.

Ablation study results on the primary cohort. The table quantifies the contribution of each key component in AR-CDT. The Full Model represents the complete architecture; No DynamicTanh and No MS-DConv remove the adaptive activation and deformable convolution modules, respectively; 1 MS-DConv and 2 MS-DConv test single- and dual-scale variants. All results are obtained via 10-fold cross-validation on seven disease-vs-HC tasks. Metrics are reported as Mean ± SD, with the best performance highlighted in bold

Dataset	Metric	Full model	No DynamicTanh	No MS-DConv	1 MS-DConv	2 MS-DConv
T2D	AUC	0.8801 ± 0.0206	0.8754 ± 0.0245	0.8821 ± 0.0243	0.8758 ± 0.0205	0.8747 ± 0.0196
	F1	0.7232 ± 0.0312	0.6768 ± 0.0968	0.6873 ± 0.0541	0.7207 ± 0.0319	0.7221 ± 0.0281
	Recall	0.7846 ± 0.0285	0.7183 ± 0.0634	0.7840 ± 0.0270	0.7851 ± 0.0273	0.7854 ± 0.0263
IBD	AUC	0.8837 ± 0.0163	0.8860 ± 0.0191	0.8916 ± 0.0174	0.8847 ± 0.0179	0.8850 ± 0.0172
	F1	0.7068 ± 0.0101	0.6880 ± 0.0536	0.7338 ± 0.0419	0.7112 ± 0.0282	0.7129 ± 0.0281
	Recall	0.7854 ± 0.0332	0.7911 ± 0.0260	0.7793 ± 0.0357	0.7969 ± 0.0253	0.7906 ± 0.0307
CRC	AUC	0.9259 ± 0.0157	0.9199 ± 0.0147	0.9249 ± 0.0151	0.9265 ± 0.0145	0.9267 ± 0.0143
	F1	0.7621 ± 0.0268	0.7389 ± 0.0527	0.7665 ± 0.0361	0.7626 ± 0.0244	0.7609 ± 0.0224
	Recall	0.8443 ± 0.0162	0.8151 ± 0.0578	0.8276 ± 0.0249	0.8448 ± 0.0267	0.8413 ± 0.0293
MS	AUC	0.9573 ± 0.0161	0.9410 ± 0.0145	0.9440 ± 0.0129	0.9507 ± 0.0127	0.9506 ± 0.0126
	F1	0.7596 ± 0.0425	0.7102 ± 0.0872	0.6526 ± 0.1240	0.7537 ± 0.0400	0.7487 ± 0.0482
	Recall	0.8566 ± 0.0594	0.8321 ± 0.0287	0.8260 ± 0.0585	0.8465 ± 0.0451	0.8448 ± 0.0412
CAD	AUC	0.9583 ± 0.0084	0.9465 ± 0.0111	0.9421 ± 0.0091	0.9499 ± 0.0140	0.9504 ± 0.0140
	F1	0.7625 ± 0.0287	0.7320 ± 0.0813	0.7290 ± 0.0912	0.7551 ± 0.0360	0.7492 ± 0.0448
	Recall	0.8602 ± 0.0304	0.7826 ± 0.0599	0.8253 ± 0.0421	0.8457 ± 0.0459	0.8496 ± 0.0392
Fatty liver	AUC	0.9393 ± 0.0321	0.9223 ± 0.0413	0.9438 ± 0.0319	0.9383 ± 0.0305	0.9392 ± 0.0304
	F1	0.6691 ± 0.0432	0.5709 ± 0.2420	0.6664 ± 0.1541	0.6683 ± 0.0605	0.6615 ± 0.0604
	Recall	0.8555 ± 0.0489	0.7646 ± 0.1468	0.7994 ± 0.0740	0.8304 ± 0.0645	0.8318 ± 0.0626
Cirrhosis	AUC	0.9858 ± 0.0098	0.9740 ± 0.0260	0.9884 ± 0.0109	0.9850 ± 0.0158	0.9858 ± 0.0140
	F1	0.7711 ± 0.0586	0.5720 ± 0.2339	0.6122 ± 0.2634	0.7361 ± 0.1361	0.7122 ± 0.1715
	Recall	0.9282 ± 0.0471	0.8226 ± 0.1297	0.8386 ± 0.1431	0.9074 ± 0.0726	0.8992 ± 0.0875

Upon removing the Channel-wise Dynamic Tanh (CD-Tanh) activation function, the model’s overall performance dropped significantly, with AUC, F1-score, and recall being lower than the full model across all seven tasks. This proves its significant role in enhancing the model’s discriminative ability. It is particularly noteworthy that this module is crucial for model stability; in the fatty liver and cirrhosis datasets, its removal increased the standard deviation of the F1-score from 0.043 and 0.059 to 0.242 and 0.234, respectively, fully demonstrating that the module effectively enhances model stability and robustness.

In comparison, removing the MS-DConv module, the core of our spatial feature extraction, led to a consistent decrease in recall across all seven datasets and a lower F1-score in most tasks, although its impact on AUC fluctuated. This strongly proves that MS-DConv, with its multi-scale receptive fields and adaptive deformation capabilities, plays an indispensable role in accurately identifying and recalling positive-class samples. Furthermore, we validated our multi-scale hypothesis by simplifying the Full Model (3-scale) to a 2 MS-DConv (2-scale) and 1 MS-DConv (1-scale) variant. The results confirmed this hypothesis: in 4 of the 7 tasks (MS, CAD, fatty liver, and cirrhosis), the Full Model achieved superior F1 and Recall performance compared to the simplified 1-scale and 2-scale versions.

In summary, the two innovative modules have unique and complementary functions. CD-Tanh is key to improving the model’s overall performance and stability, while the superiority of the MS-DConv module relies on both its deformable nature and its multi-scale architecture. Together, they form the solid foundation for our model’s high performance.

Discussion

In this study, we introduced AR-CDT, a novel deep learning architecture designed for classifying large-scale, imbalanced microbiome data, where existing methods often fail to generalize. The core innovation of AR-CDT lies in its feature-learning framework, which represents microbial abundance profiles as pseudo-images to uncover latent spatial patterns. The architecture integrates a MS-DConv module to capture multi-level microbial community signals and a CD-Tanh activation function to adaptively model their high-order nonlinear relationships.

A key contribution of this work is the establishment of a rigorous, multi-level validation framework that systematically evaluates both within-cohort performance and out-of-distribution generalization. While AR-CDT achieved highly competitive results under 10-fold cross-validation, we acknowledge that this evaluation mainly reflects within-cohort consistency. To further assess robustness, we performed a LSO validation to evaluate cross-study transferability and an external validation on an independent T2D dataset to test real-world generalization. Across these more stringent evaluations, AR-CDT consistently maintained reasonable discriminative power, with AUC values ranging from approximately 0.72–0.76 for LSO and 0.79 for external validation. Although these scores are lower than those from standard cross-validation, this outcome is expected because the model was trained on a large, highly imbalanced, and heterogeneous multi-study cohort and was still able to generalize to a comparatively balanced external dataset. This result demonstrates that AR-CDT effectively captures biologically meaningful and transferable representations rather than dataset-specific artifacts.

Beyond robust classification, AR-CDT was further combined with an OvR–SHAP interpretability framework, enabling the systematic identification of disease-specific microbial signatures and providing mechanistic insights into the dysbiotic patterns underlying complex diseases.

Our work has several limitations that also highlight important directions for future research. First, the validation of the primary cohort was constrained by the structure of publicly available data, where severe structural confounding made the gold-standard LSO validation statistically infeasible, because many disease tasks not only came from a single study rather than multiple independent cohorts, but some also included only one type of sample within a study. Future research should focus on building and curating large, multi-study microbiome cohorts with balanced case–control representation within each study. Such datasets will enable a more rigorous, fully independent validation of AR-CDT across multiple cohorts, extending our current two-tier generalization framework toward true population-level generalizability.

Second, our finding that the microbial similarity between MS and CAD arises from a convergent functional deficit–namely, the depletion of butyrate-producing taxa–suggests that abundance-based features alone may be insufficient for precise discrimination. Future work could extend AR-CDT into a multi-modal framework by integrating complementary data modalities such as metabolomics or host transcriptomics, thereby enabling the identification of non-microbial markers capable of distinguishing such confounded diseases.

Third, the current implementation is limited to cross-sectional data. Since disease progression is inherently dynamic, future adaptations of AR-CDT should incorporate temporal modeling modules, such as Recurrent Neural Networks (RNNs) or Long Short-Term Memory (LSTM) networks, to capture longitudinal microbial evolution and support predictive diagnostics.

Fourth, extending AR-CDT toward a multi-task learning (MTL) paradigm represent promising directions for improving model efficiency, interpretability, and biological insight.

In summary, although the current validation is constrained by existing public data, AR-CDT establishes a principled foundation for robust, generalizable, and biologically interpretable microbiome-based disease modeling.

Author contributions

JL contributed primarily to code implementation and manuscript writing. ZS critically revised the manuscript. JT conceived and supervised the project and guided the writing of the manuscript. SC, HL, and YW provided guidance on experimental design and methodology.

Funding

This work was supported by the Zhejiang Province Traditional Chinese Medicine Science and Technology Project (No. GZY-ZJ-KJ-23008) and Zhejiang Province Traditional Chinese Medicine Key Laboratory Project (No.GZY-ZJ-SY-2303).

Data availability

The primary cohort data used in this study were sourced from the curatedMetagenomicData Bioconductor package, which is publicly available from the Bioconductor repository (https://bioconductor.org/packages/curatedMetagenomicData) and on GitHub (https://github.com/waldronlab/curatedMetagenomicData). The validation datasets are part of the MetAML package, which is publicly available from the Segata Lab website (http://segatalab.cibio.unitn.it/tools/metaml/) and its corresponding GitHub repository (https://github.com/segatalab/metaml). The datasets and the code used in this study can be found at https://github.com/TianJingkuiLab/AR-CDT

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

All authors have read the manuscript and consented for publication.

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12859-025-06357-0.