A systematic longitudinal study of microbiome: integrating temporal-spatial dimensions with causal and deep learning models

Abstract

Longitudinal microbiome data provide a unique opportunity to explore dynamic interactions between microbial communities and disease progression. However, these data are often characterized by missing values, sparse signals, and limited interpretability, which impede effective biomarker discovery and accurate disease modeling. Therefore, we propose SysLM, a comprehensive deep learning framework for systematic analysis of longitudinal microbiome data. It comprises two synergistic modules: SysLM-I and SysLM-C. SysLM-I focuses on the task of missing-value inference, combines metadata and three feature enhancement strategies, and comprehensively captures temporal causality and long-term dependence through Temporal Convolutional Network and Bi-directional Long Short-Term Memory modules. SysLM-C integrates deep learning with causal inference modeling to construct three causal spaces to accomplish the tasks of classification and screening of multiple types of biomarkers, including differential biomarkers of microbiomes, network biomarkers, core biomarkers, dynamic biomarkers, disease-specific biomarkers, and shared biomarkers. SysLM demonstrates superior performance in imputation, classification, and biomarker discovery across multiple datasets. Importantly, it uncovers novel microbial mechanisms underlying ulcerative colitis, highlighting its value for precision medicine. By integrating deep learning with causal modeling, SysLM offers a promising approach to advance microbiome-based disease research and facilitate the development of targeted therapeutic strategies.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12864-025-12282-6.

Introduction

Advances in high-throughput sequencing have revolutionized microbiome research, highlighting the critical role of microbial communities in human health and disease, including gastrointestinal disorders, metabolic syndromes, immune dysregulation, and cancers [1]. In contrast to cross-sectional data, longitudinal microbiome data, which samples the dynamics of individual microbiomes through multiple time points, has provided a unique perspective for understanding the temporal changes in microbial communities and their relationship with disease development [2].

However, longitudinal microbiome datasets often contain missing values caused by technical limitations or irregular sampling, which can introduce bias and compromise downstream analyses [3]. Traditional methods usually rely on missing mechanism assumptions (e.g., MCAR, MAR, or MNAR), which may not adequately consider the complexity and time-dependence of the data [4]. In recent years, deep learning methods have performed well in inferring missing values, especially the BRITS model, which infers missing values through a bidirectional recurrent neural network without assuming the data distribution [5], and the CATSI model introduces “context vectors” to learn the global temporal dynamics and further improves the accuracy [6]. These methods effectively improve the accuracy of missing data inference by automatically learning the temporal patterns of the data.

Beyond missing data inference, effective analysis of longitudinal microbiome data requires integrated approaches that combine metadata incorporation, feature enhancement, and robust biomarker discovery. Existing models tend to address these challenges in isolation: DeepMicroGen model uses generative adversarial networks to infer missing values and classification tasks for longitudinal microbiome data, but lacks in-depth exploration of biomarker screening and functional analysis [7]. Fung et al. focus on classifying tasks by sequentially populating zero-values and integrating self-knowledge distillation with neural networks. However, they do not also explore further analyses [8]. MDITRE is an interpretable model designed to discover associations between microbiome dynamics and host status through human-readable rules, but it does not support causal inference [9]. Similarly, while some models have attempted to consider the increased use of metadata for classification, the models have also been limited to downstream classification tasks and have not been able to exploit the potential of microbiome data fully. For example, Chen et al. uses the FE_GRU model to predict host status based on longitudinal microbiome data in combination with phylogenetic trees and metadata [10]. phyLoSTM model combines a convolutional neural network and a long and short-term memory network, which integrates feature extraction with the environmental factors of the host and improves the model’s ability to classify [11]. Some models primarily focus on the classification task. For instance, Metwally et al. combines Minimum Redundancy Maximum Relevance (mRMR) and LSTM to classify food allergies [12].

Moreover, the complexity of longitudinal microbiome data and substantial inter-individual variability present significant challenges to model interpretability [13], which is crucial for clinical applications. Many existing approaches lack sufficient transparency, which to some extent restricts the extraction of meaningful biological insights and reduce their translational potential. To address these challenges, we propose SysLM, a systematic and interpretable deep learning framework. The framework combines the strengths of deep learning and causal modeling to effectively handle time-dependence and individual differences in longitudinal microbiome data. SysLM integrates feature augmentation and metadata with Temporal Convolutional Network (TCN) and Bi-directional Long Short-Term Memory (BiLSTM) networks in its SysLM-I module to accurately infer missing values while capturing temporal causality and long-term dependencies [14, 15]. On this basis, the SysLM-C module employs causal inference to construct multiple causal spaces, enabling comprehensive biomarker identification and elucidation of key microbial pathways linked to ulcerative colitis (UC). Extensive validation on diverse datasets demonstrates SysLM’s improved predictive accuracy, interpretability, and biological relevance, offering a promising tool for microbiome-based precision medicine.

Materials and methods

Datasets

To comprehensively assess model performance, we have collected six publicly available real datasets or research projects, namely DIABIMMUNE, BONUS-CF, DiGiulio, PROTECT, iHMP-IBD, and iHMP-T2D. Among them, the DIABIMMUNE project aims to unravel the biological mechanisms underlying the increased prevalence of autoimmune diseases and allergies. The data are collected from a total of 222 infants in three countries (Finland, Estonia and Russia) [16]. Different allergy types are used as classification targets to support early diagnosis of related diseases. The BONUS-CF project, a prospective multicenter study of infants with cystic fibrosis (CF), aims at identifying microbiological correlates of poor growth observed in infants with cystic fibrosis, includes a total of 232 subjects with sequenced samples, and in this paper, the presence or absence of cystic fibrosis is used as a classification result [17, 18]. The DiGiulio data is a case-control study of 40 pregnant women designed to explore the relationship between microorganisms and preterm labor. In this paper, the presence or absence of preterm labor is used as a categorization goal to better understand the impact of microorganisms on pregnancy outcomes [19, 20]. The PROTECT dataset covers follow-up data from 428 patients with new-onset pediatric UC, and in this paper, patients with UC are classified into active and inactive groups based on the subjects’ PUCAI scores [21]. The iHMP-IBD project collects multinomial longitudinal data from approximately 90 patients with inflammatory bowel diseases (IBD) to analyze the temporal changes in the gut microbiota, and we use the type of IBD patients as a classification outcome [22, 23]. The iHMP-T2D project establishes a cohort of about 100 diabetes-risk patients to study the biological changes associated with T2D, and we use the IR and IS of T2D patients as a classification outcome, which is intended to assist in the development of personalized treatment plans [22]. A description of the data preprocessing process is provided in the Supplementary Section 1.1.

SysLM

Compared to microbiome data from a single time point, longitudinal microbiome data, in which samples are collected at multiple time points, can provide detailed information on the dynamics of microbial communities and reveal their relationship with host health. However, longitudinal studies often compromise the accuracy and consistency of data due to missing samples during follow-up. Therefore, we propose the SysLM model, which contains two sub-models, SysLM-I and SysLM-C. The structure of the SysLM model is schematically shown in Fig. 1A. The SysLM-I model is mainly used to infer longitudinal microbiome data with missing values to maximize the recovery of missing values and improve the predictive performance and generalization ability of the model’s downstream analysis (Methodological details are provided in the Supplementary Sect. 1.2). Inferring missing values in data helps reduce data bias and inconsistency, improve analysis accuracy, and avoid wasting potential information by deleting samples with missing values. In our experiments, we compare SysLM-I with multiple benchmark methods, using MAE, MSE, RMSE, and R² as evaluation metrics, to validate its inferential performance.

Fig. 1.

A The flowchart of SysLM. B The flowchart of SysLM-C

To enhance the performance of the SysLM-I model on different datasets and to better capture the complex structure and distributional features of the data, we introduce two diversity loss functions based on the mean square error (MSE) loss. Firstly, loss_a is used to measure the difference between the alpha-diversity of the generated values and the true values, which is calculated based on the Shannon index. Secondly, loss_β is used to measure the difference between the beta-diversity calculated for the generated and true values, and this difference is calculated using the Bray-Curtis distance. That is, the loss of the model consists of the following three main components, which can be expressed as:

1

where, loss_r measures the error between the model generated values and the true values, H (∙) is the computed Shannon index, D_BC (∙) is the calculated Bray-Curtis distance. To effectively optimize the model, the differences in diversities are measured using the Frobenius norm. By optimizing the diversity of the generated data, the model can generate microbial data with higher biological plausibility and ecological diversity. w_a, w_β are their weighting coefficients in the range [1e-1,1e-2, ⋯,1e-5].

Currently, deep learning-based longitudinal microbiome studies mainly focus on inferring missing values and classification tasks based on subject status, and some studies have not deeply explored how to fully utilize the temporal properties in longitudinal microbiome data to identify potential microbial biomarkers. To address this issue, we propose the SysLM-C model, which is based on inferred data to classify subjects and find microbial biomarkers associated with subject status (Methodological details are provided in the Supplementary Sect. 1.3). Its performance has been assessed by comparing it with published classification models and using metrics such as AUC. The model can identify microbial biomarkers with potential causal relationships and provide insights into microbe-host health relationships. The SysLM-C model has enhanced interpretability in biomarkers discovery and can reveal potential causal relationships between microbes and host status, providing theoretical support for subsequent mechanistic studies and the development of personalized therapeutic regimens.

As shown in Fig. 1B, the SysLM-C model consists of three causal spaces. The first causal space is mainly used for subject classification and to improve model interpretability, and contains three loss functions, i.e., classification loss (used to train the classifier and differentiate between Case and Ctrl labels), reconstruction loss (ensures that the data features are reconstructed from the causal relationships and maintains the structural information), and causal loss (captures the causal paths and causal effects between semantic vectors to help the model understand the causal inference). The second causal space is the static causal space, which is designed to capture global causal relationships between microbes and subject states, and contains causal loss (learning causal paths and causal effects between microbes and subject states) and reconstruction loss. The third causal space is referred to as the dynamic causal space, which is designed to capture trends in the temporal evolution of diseases or health changes. This space incorporates dynamic causal loss, which captures causal relationships in time-series data, alongside causal loss and reconstruction loss. This combination ensures that the model comprehensively understands the causal structure over time, thereby enhancing the relevance of the dynamic relationships extracted from the dynamic causal map [24]. In addition, it is also necessary to constrain the consistency of the causal relations learned in the static and dynamic causal spaces with the reconstructed state semantic vectors, which helps the model to better understand the causal inference in the data at both the static and dynamic levels, and thus improves the generalization ability and accuracy of the model. Thus, the total loss of the SysLM-C model can be represented as follows:

2

where w_rec, w_dag, w_dyn, and w_con are the weights of the reconstruction loss, causal loss, dynamic causal loss, and consistency loss, respectively, in the range [1e-1,1e-2, ⋯,1e-5]. The formulas for the different types of losses mentioned above can specifically be expressed as follows:

3

4

5

6

7

where α_t​ is a balancing factor used to adjust the weights between categories. p_t is the predicted probability of the correct category. γ is the moderating factor, which is used to control the loss contribution to easily categorized and difficult to categorize samples. According to Lin et al. [25], α_t​ and γ are set to 0.25 and 2, respectively. T is the number of time points. , , represent the causal matrices of the three spaces, respectively. denotes the Hadamard product. q¹, q², q³ denote the number of variables in the three causal spaces, respectively. 𝜌 and α are the two hyperparameters of the causal loss, which are set to 1 [26]. In calculating the causal loss, we consider that the dimension of the causal space of the characteristics is too large, which may lead to the difficulty of numerical computation problems. Therefore, based on Zheng et al. [27] we normalize the Hadamard product of the causal matrix, which does not change the constraint of directed acyclicity of the matrix. The error term() compensates for numerical errors from normalization and relaxes constraints, ensuring stability and convergence in DAG learning. Here, Tr denotes the trace of a matrix. , are the Frobenius norm and L2 norm, respectively. and denote the reconstruction of the state semantic vector Case in the 3rd causal space and the 2nd causal space, respectively. Similarly, and are the reconstructions of the state semantic vector Ctrl, respectively.

Results

Parameter sensitivity analysis and ablation study

In microbiome data, there are more zeros at finer taxonomic levels (e.g. species or genus), whereas data at the phylum level are more complete. Therefore, to ensure robustness and generalizability, we conduct a parameter sensitivity analysis using the phylum-level DIABIMMUNE dataset.

For SysLM-I, we evaluate the impact of loss weights w_a and w_β using 5-fold cross-validation. The optimal configuration is determined as w_a = w_β=1e-5, which is then applied uniformly across all datasets and taxonomic levels to ensure consistency and biological plausibility (see Fig. 2A).

Fig. 2.

The Results of the analysis of parameter sensitivity. In the SysLM-C model, for clear visualization, we take a logarithm to base 10 for the parameter combinations

For SysLM-C, we tune four loss weights (w_rec, w_dag, w_dyn, and w_con) based on class imbalance ratios, optimizing primarily for AUC (see Fig. 2B-Fig. 2E.). Supplementary Table S3 summarizes representative weight combinations for different positive-to-negative sample ratios. This analysis highlights the importance of appropriately balancing optimization objectives under different data distributions. By tailoring weight configurations to specific imbalance ratios, we improve model performance and stability in classification tasks.

Ablation experiments further confirm the necessity of key modules and loss components for both SysLM-I and SysLM-C, with detailed results presented in Supplementary Sect. 1.4.

Performance comparison of missing value imputation methods

To comprehensively evaluate the performance of the SysLM-I model in inferring missing values from longitudinal microbiome datasets, we conduct extensive comparative experiments using 5-fold cross-validation. In each fold, the observed values in the test set are masked and treated as missing, and models are trained on the remaining training data. The imputation performance is assessed by calculating MAE, MSE, RMSE, and R² at the masked positions, and final results are averaged.

We compare SysLM-I against several baseline methods: deep learning models BRITS [5], CATSI [6], DeepMicroGen [7], as well as simple statistical approaches like Mean and Median imputation. The results, shown in Supplementary Fig. S2, demonstrate that SysLM-I generally outperforms or performs comparably to other methods across most evaluation metrics and taxonomic levels. Specifically, SysLM-I achieve the best MAE in 88.24% of experiments and the best MSE, RMSE, and R² in 91.18% of cases. This highlights the strength of SysLM-I in handling complex missing data across diverse microbial datasets.

Further, to assess whether the inferred values maintain ecological validity, we compare the alpha and beta diversity between inferred and true values across all methods. Due to the relatively poor performance of statistical methods in imputation, this diversity comparison focuses only on deep learning-based methods. As shown in Supplementary Fig. S3, SysLM-I demonstrates the best reconstruction of microbial diversity patterns, particularly from the phylum to genus levels, where community profiles tend to be more consistent and biologically informative. Performance at the species level is relatively less consistent, likely due to the high sparsity and variability inherent to this taxonomic resolution. Among other methods, BRITS performs reasonably well, while DeepMicroGen performs worse, possibly due to its reliance on post-processing strategies.

These results collectively suggest that SysLM-I not only achieves accurate numerical imputation but also preserves the underlying ecological structure of microbial communities, thus offering both quantitative and biological advantages.

Performance evaluation and method comparison in classification tasks

To comprehensively evaluate the effects of different missing value imputation methods and classification strategies on downstream classification tasks, we systematically compare various classification models and loss functions using longitudinal microbiome data with key metrics such as AUC, AUPR, ACC, and F1-score.

First, we compare the classification performance of different imputation methods using three classifiers: phyLoSTM [11], CNN-LSTM [8], and DeepMicroGen [7]. Results show that imputations from the SysLM-I model achieve the best and most stable performance across all classification models (see Supplementary Fig. S4). Other imputation methods show more fluctuation, indicating that SysLM-I provides more reliable inferred values to support accurate and robust classification.

Second, considering the severe imbalance of positive and negative samples, we compare various classification loss functions, including cross-entropy (CE), weighted cross-entropy (WCE), DICE loss, and Focal Loss. Using the gold standard dataset and 5-fold cross-validation, Focal Loss outperforms others on AUC and AUPR metrics, though it performs slightly worse on ACC and F1-score in the egg allergy classification task (Supplementary Fig. S5). This suggests that Focal Loss emphasis on hard-to-classify samples contributes to more stable and robust key metric performance.

Finally, we compare the SysLM-C model with several baseline models— phyLoSTM, CNN-LSTM, and DeepMicroGen—focusing on AUC performance across different taxonomic levels. SysLM-C achieves multiple best optimizations across phylum, class, order, family, genus, and species levels, demonstrating the advantages of its temporal and spatial modeling capabilities (Supplementary Fig. S6). Additionally, SysLM-C also shows improvements in AUPR, ACC, and F1-score (see Supplementary Fig.S6), confirming its robustness and superiority across datasets.

In summary, the SysLM series models achieve efficient, stable, and accurate predictions in classification tasks on longitudinal microbiome data through high-quality imputation, appropriate loss design, and effective classification architecture.

Statistical test

To further assess the necessity of introducing metadata, we conduct a mantel test to analyze the correlation between metadata and specific microbial taxa at the phylum level across all datasets. The results are shown in Supplementary Fig.S7, indicate statistically significant correlations between metadata and OTUs across all datasets. This implies that there may be some correlation or influence between these factors (factors such as country, gender, and race) and OTUs. These results support the necessity of incorporating metadata when inferring missing values for microbiomes as well as for classification tasks.

Based on the gold standard data, we further assess the reliability of the SysLM method by statistical tests. Firstly, a Shapiro-Wilk test has been used to determine whether the data satisfy the normal distribution (see Supplementary Table S4-S5, p-value < 0.01). If the data satisfy the normal distribution assumption, a one-way ANOVA (p-value < 0.05) is then used to compare the performance of the different models on the evaluation metrics. Statistical tests indicate that the SysLM method is significantly different from the other methods in the inference task and that the results of all inference methods conform to a normal distribution (see Table 1).

Table 1.

The results of one-way ANOVA test (p-value < 0.05)

Methods	R 2	MAE	MSE	RMSE	AUC	AUPR	ACC	F1-score
BRITS	7.48E-04	5.54E-05	7.64E-04	1.16E-04	-	-	-	-
CATSI	2.68E-17	3.81E-17	6.02E-13	3.82E-14	-	-	-	-
DeepMicroGen (inference task)	8.44E-16	3.24E-16	4.15E-13	2.15E-14	-	-	-	-
Mean	1.70E-20	2.14E-17	2.79E-14	5.13E-15	-	-	-	-
Median	4.56E-20	1.81E-17	3.20E-14	5.27E-15	-	-	-	-
CNN-LSTM	-	-	-	-	2.35E-03	0.074	2.91E-03	0.048
DeepMicroGen (classification task)	-	-	-	-	2.16E-02	0.11	1.39E-01	0.28
phyLoSTM	-	-	-	-	2.26E-02	0.035	2.68E-02	0.21

For the classification task, considering that the four-label classification task of the gold-standard dataset can be viewed as a multi-label task, we combine classification results to simplify the analysis process. Since DeepMicroGen does not conform to a normal distribution on the ACC metrics, we assume that it does to ensure a fair comparison. The results show that SysLM-C is significantly different from other methods on AUC metrics, while only part of the results is significantly different on AUPR, ACC, and F1-score metrics (see Table 1). To further enhance the statistical robustness of our results, we additionally conduct Wilcoxon signed-rank tests, and the corresponding results are provided in Supplementary Table S6. Some of the statistical test results are consistent with those obtained from the one-way ANOVA.

Interpretability of the model

As shown in Fig. 1B, the learned causal DAG graph derived from the semantic vector causal space offers a potential approach for interpreting model predictions. Specifically, when we obtain the optimized causal matrix , we will gradually remove edges in according to the weight size of the edges until satisfies the property of directed acyclic graph. This process ensures that the cropped DAG has a reliable and significant structure (The result after cropping is shown in Fig. 3A - K). These causal graphs appear to be interpretable, and in several cases, some inferred relationships align with prior domain knowledge or reported associations.

Fig. 3.

The results of DAG visualization for different tasks that explain possible causal relationships in the task

Specifically, as can be seen in Fig. 3A, for the classification task of milk allergy in the DIABIMMUNE dataset, it can be found that, which suggests that the microbial community, by regulating the function of the immune system, may influence the occurrence of milk allergic reactions to a certain extent [28–30]. Whereas, country factors of the subjects, such as dietary habits, environmental pollution and sanitary conditions, may play a role in the development of milk allergy by altering the composition of the microbial community or by directly influencing the immune response [31]. Gender may also influence susceptibility to allergy and the strength to immune response through physiologic and immunologic differences. For the classification task of egg allergy when it can be found (see Fig. 3B), this suggests that gender may influence egg allergy through differences in the immune system, whereas national factors may alter the composition of the microbial community through diet and environment. In addition, the occurrence of egg allergy conditions may lead to changes in the structure of the microbial community in the subjects [28, 32, 33]. Based on the classification task of CF, indicates that microbial community composition may be associated with subject-specific biological characteristics (see Fig. 3E). Furthermore, the status of CF patients may be associated with the composition of their microbial communities, suggesting that certain specific microbial communities could either contribute to the maintenance of health or interfere with it [34].

For the other classification tasks, we can also observe similar causal relationships, i.e., the interaction between the microbial community and the metadata may jointly influence the subject’s state. In addition, we also notice that some of the causal relationships are consistent with the mantel test results, which provides additional support for the interpretability and reliability of the model. In conclusion, through the causal graph analysis of the SysLM-C model, we can reveal the potential complex interactions between microbial communities and subject metadata (e.g., state, gender, etc.). This provides an important perspective for understanding the multilevel causal mechanisms of disease occurrence and the role of microbial communities in disease.

Microbial biomarkers

Leveraging the static and dynamic causal space (,), our model identifies six types of microbial biomarkers that provide insights into disease mechanisms. Below, we define and analyze each type with representative findings.

Differential biomarkers

Differential biomarkers are defined as microbial taxa that differ significantly between case and control groups. In this work, they are identified based on their simultaneous connections to both Case and Ctrl nodes in the static DAG. Compared to traditional statistical methods (e.g., Mann-Whitney U test, false discovery rate (FDR) < 0.05) [35], our method identifies a larger number of differential microbe’s biomarkers with some overlap (Fig. 4A–C; Supplementary Fig. S8), suggesting improved sensitivity.

Fig. 4.

The visualization results of differential biomarkers. A Differential biomarkers of milk allergy screened based on the statistical test method. B Differential biomarkers of milk allergy screened based on our method. C Comparison of the number of differential biomarkers at all taxonomic levels in all datasets between the statistical test method and our method

To further validate the reliability of the screened differential biomarkers, we compare the screening results with those reported in the literature. For example, p__Firmicutes and c__Clostridia are investigated as probiotic candidates for milk allergy treatment [30]. Meanwhile, g__Bifidobacterium and g__Coprococcus have been reported to show reduced abundance in patients with milk allergy [36], and f__Lachnospiraceae and f__Leuconostocaceae have shown significant differences between egg allergy patients and healthy people [28]. It has also been found that c__Clostridia showed significant differences between food allergy patients and healthy populations, especially in peanut allergy [37]. In addition, p__Actinobacteria has also been shown to be used in early infancy to prevent food allergies [33]. o__Pseudomonadales is strongly associated with CF, especially Pseudomonas aeruginosa which is one of the main causative organisms of lung infections in patients with cystic fibrosis [38]. It has been found that the abundance of p__Firmicutes is higher and p__Proteobacteria is lower in preterm pregnant women compared to those who delivered at term [39]. The higher abundance of g__Lactobacillus in CD patients compared to UC patients and non-IBD patients suggests that g__Lactobacillus may be associated with the pathologic process of CD [40]. The abundance of g__Faecalibacterium and g__Roseburia has been significantly reduced in UC patients [41], c__Actinobacteria has been significantly increased in abundance [42]. It has been confirmed that g__Flavonifractor is highly correlated with IS in T2D patients, and g__Coprococcus and g__Dorea are closely associated with IR [43]. These findings demonstrate partial agreement between our screened biomarkers and previously reported microbial alterations, thereby providing preliminary support for the potential applicability of our method across multiple diseases. However, further biological and clinical validation is required.

Network biomarkers

Network biomarkers are defined as microbial communities exhibiting coherent associations related to the phenotype of interest. They are identified through community detection performed on the static DAG using the Louvain algorithm. Modules containing both Case and Ctrl nodes highlight structural network changes across groups (Supplementary Fig. S9).

It can be observed that the microbial network structure of the Ctrl group is relatively denser, showing stronger intra-community interactions. In some Case groups, such as the CF group, we observe a denser network structure, which may reflect stronger interactions or more complex network formations among microbial communities in specific disease states. These observations indicate potential variations in microbial networks between individuals with certain diseases and health. In addition, distinct network patterns are observed across different taxonomic levels, highlighting the importance of multi-level taxonomic analysis.

To assess the reliability of the screened biomarkers, validation is performed based on published literature. For example, for milk allergy, network analysis shows that the p__Firmicutes and Ctrl groups show positive regulatory causality, which is consistent with the findings of Bunyavanich et al. [30]. In addition, the network suggests that the abundance of p__Firmicutes could potentially be modulated through the regulation of other connected microbes, which offer possible avenues for influencing the immune response in milk allergy. In the network structure of CF patients, we observe that the Case group positively regulated p__Actinobacteria, suggesting that the abundance or activity of this group may be up-regulated, which is consistent with the findings of Burke et al. [44]. In the network of CD patients, p__Firmicutes is negatively regulated, suggesting that the activity may be suppressed in CD patients, which is consistent with the findings of Loh et al. [45, 46]. In contrast, p__Actinobacteria is positively regulated in UC patients [46], suggesting that it may play a role in the pathological mechanism of IBD by promoting inflammation and exacerbating the disease. Screening of these network biomarkers provides potential insights into changes in disease-associated microbial communities, which may have implications for significant clinical.

Core biomarkers

Core biomarkers are microbes that appear consistently across time-resolved DAGs and are connected to Case at each time point. These stable associations suggest long-term involvement in disease progression (Table 2; Supplementary Table S7). For example, p__Cyanobacteria have been shown to be strongly associated with CD and UC [47, 48]. In addition, reduction of p__Proteobacteria has been found to improve the therapeutic efficacy of CD in two prospective studies [49], and reduction of p__Proteobacteria can alleviate the symptoms of UC [50]. However, phylum-level biomarkers, especially those of low abundance or unusual taxa, should be interpreted with caution, as they may result from sequencing noise or annotation artifacts. And o__Clostridiales has also been confirmed by Schirmer et al. to be associated with the severity of UC [21]. These findings may enhance our understanding of disease progression and could inform future studies on microbial markers for disease monitoring.

Table 2.

Core biomarkers at the phylum level for different datasets or different categories

Dataset	Categories	Core marker
DIABIMMUNE	Total	p__Fusobacteria
BONUS-CF	CF	p__Proteobacteria
iHMP-IBD	CD	p__Cyanobacteria
		p__Gemmatimonadetes
		p__Proteobacteria
		p__Planctomycetes
	UC	p__Fibrobacteres
		p__Chlorobi
		p__Tectomicrobia
		p__Proteobacteria
	Non-IBD	p__Tectomicrobia
		p__Cyanobacteria
		p__Tenericutes
		p__Armatimonadetes
		p__Latescibacteria
		p__Firmicutes
		p__Actinobacteria
		p__Euryarchaeota
PROTECT	Active	p__TM7
		p__Cyanobacteria

No core biomarkers found at the phylum level for category labels not shown

Dynamic biomarkers

Dynamic biomarkers are defined based on time-varying DAGs. OTUs showing substantial variation in causal weights or exhibiting time-associated trends in their relationship with Case are considered dynamic biomarkers. The results are shown in Supplementary Fig.S10. We observe that microbial biomarkers exhibit different trends across time and taxonomic scales, suggesting their involvement may vary at different stages of disease development. For example, the significant enrichment of f__Prevotellaceae in IBD disease and the increasing trend of its causal effect on IBD patients may imply its progressively increasing role in the disease process of IBD [51, 52]. g__Lachnospira’s abundance has been confirmed to be significantly reduced in CD patients [53], and we observe a decreasing trend in its causal effect, which may be associated with its reduced presence in disease progression. The p__Proteobacteria shows an increasing trend of causal effect in UC patients, which is also consistent with the previous findings that the alleviation of UC symptoms may be achieved by decreasing p__Proteobacteria [50]. In a longitudinal study, Zhou et al. have found that the abundance of g__Butyricicoccus is reduced in IR patients compared to IS typed T2D patients [54]. Further, in our study, g__Butyricicoccus is found to exhibit an incremental trend of a causal effect on IR, which may imply that its role in the disease is exacerbated. To further elucidate the evolving roles of key microbial taxa in disease progression, we visualize their dynamic causal trajectories (Supplementary Fig. S10M), which highlight the importance of temporal modeling in uncovering potential microbial mechanisms. The causal weights reflect the estimated regulatory effect of a taxon on disease outcome, which may not always directly follow the relative abundance trends in the original or imputed data. Supplementary Fig. S10N provides the temporal abundance trends for comparison, illustrating cases where causal effect and abundance trajectories are consistent as well as cases where they differ.

Disease-specific and shared biomarkers

Disease-specific biomarkers refer to microbes identified in only one disease group, while shared biomarkers refer to microbes that are present in at least two disease groups. For example, the presence of 15 shared biomarkers in the IR and IS subtypes of T2D disease can be observed in Fig. 5., with 38 specific microbial biomarkers in the IR group and 18 specific markers in the IS group. This finding suggests that different subtypes of T2D may have unique microbial ecological profiles, reflecting potential differences in disease mechanisms between these subtypes. Specifically, IR and IS subtypes may be influenced by different microbial communities, which could be associated with disease processes such as insulin resistance and glucose metabolism. Moreover, no shared microbial biomarkers were found between the two T2D subtypes and other diseases, which may indicate the microbial specificity of T2D and suggest its distinctiveness from other diseases in terms of microbiome composition. This finding may contribute to a better understanding of the pathogenic mechanisms of T2D and provides potential biomarker candidates for personalized therapy and precision medicine.

Fig. 5.

The visualization results of disease-specific and shared biomarkers based on all taxonomic levels. Shared biomarkers refer to microbes that are present in at least two disease groups

We also have observed the presence of shared microbial biomarkers for CF and IBD diseases, which may reveal a common pathogenesis for both diseases [55]. In addition, we have identified shared biomarkers for food allergies, including milk, egg, and peanut allergies, as well as IBD. This discovery could uncover common mechanisms of these immune-mediated diseases at the microbial level. Understanding these connections may offer novel insights into immune response mechanisms and help develop potential cross-disease therapeutic strategies [56–58]. Through this visualization, we can observe the microbial biomarkers specific to each disease as well as those that co-occur in multiple diseases, thus providing potential support for understanding the pathological mechanisms of diseases and cross-disease commonalities.

By identifying diverse types of microbial biomarkers, our framework provides multi-perspective insights into the roles of microbes in disease onset and progression, offering potential targets for diagnosis, monitoring, and intervention. Due to the high sparsity and large proportion of unclassified taxa at the species level—which may compromise biological interpretability—our biomarker analyses primarily focus on higher taxonomic levels, including phylum, class, order, family, and genus.

Functional analysis

In our preliminary analysis, we have screened different types of biomarkers that provide valuable insights for a deeper understanding of disease mechanisms. However, identifying microbial community composition alone is insufficient to fully elucidate their roles in disease progression, particularly about their functional potential. Therefore, to further explore the functional information of microbial communities, we use the phylogenetic investigation of communities by reconstruction of unobserved states (PICRUST) tool for functional analysis based on the screened differential biomarkers [59]. PICRUST can infer functional characteristics of microbial communities from 16 S rRNA gene sequences. Considering that the PICRUST relies on the Greengenes database and requires the ID of this database, we perform a functional analysis based on the PROTECT dataset and the results are shown in Supplementary Fig.S11-Fig.S12.

Our results demonstrate that different functions exhibit significant abundance differences at different taxonomic levels (see Supplementary Fig.S11A). In the KEGG function prediction analysis, the function Cell Motility shows a notably higher abundance at the family level compared to other taxonomic levels. These findings underscore the necessity of fine-grained, multi-level taxonomic analysis to better understand the relationship between microbial classification and function.

In addition, we have analyzed between-group differences (UC active vs. inactive state) in the functions of the samples at the L2 and L3 levels of the KEGG prediction results (see Supplementary Fig.S11B- Fig.S11C) using STAMP software (default Welch’s t-test, p-value < 0.05) [60]and the OmicStudio cloud platform (default t-test, p-value < 0.05) [61], respectively. The results of both analyses are consistent, further validating the reliability of the results of screening differential pathways. The results obtained through the OmicStudio cloud platform are presented in this study.

The results reveal significant functional differences between the UC active and inactive groups across various taxonomic levels. Specifically, at the L2 level, a total of 22 functional modules are significantly different, of which 18 functional modules are higher in the UC active group than in the inactive group, and 14 functional modules show the opposite trend. At the L3 level, a total of 92 functional modules shows significant differences, of which 72 functional modules are higher in the UC active group than in the inactive group, and 32 functional modules are lower than in the inactive group. These results further support the potential relevance of the screened biomarkers. In addition, we have observed inconsistency in the performance of different functional modules at different taxonomic levels. For example, Metabolism in the L2 level is higher in the active state of UC than in the inactive state at the phylum level, whereas the opposite trend is observed at the family level. This phenomenon may highlight the complexity of microbial community structure at different levels, indicating heterogeneous differences in microbiome data at various taxonomic levels. Analyzing microbial communities at different levels may provide valuable insights into their dynamics across various spatial scales, reflecting their complexity and adaptability in different disease states, and potentially aiding in understanding the underlying biological mechanisms.

To further explore the relationship between microbiomes and functional modules, we have conducted a spearman correlation analysis based on differential biomarkers and differential functional pathways, and the results are shown in Supplementary Fig.S12. By combining the results of STAMP intergroup analysis and Spearman correlation analysis, we observe that there is a strong link between the abundance of p__Actinobacteria and p__Bacteroidetes and the Carbohydrate metabolism pathway, and it has been shown that the Carbohydrate metabolism pathway is crucial in the development of UC [62]. These findings indicate that p__Actinobacteria and p__Bacteroidetes may contribute to the pathogenesis of UC by modulating Carbohydrate metabolism.

In addition, we have found that the Amino acid metabolism pathway is closely related to c__Bacilli and c__Erysipelotrichi, and the lipid metabolism pathway is closely associated with f__Ruminococcaceae, f__Bifidobacteriaceae, f__Erysipelotrichaceae, and f__Lachnospiraceae. It has been noted that the Lipid metabolism and Amino acid metabolism pathways undergo specific changes in UC activation [63], which is consistent with the results in our study, and further supports the possibility that these pathways may be involved in the pathophysiological processes of UC.

Meanwhile, we also identified a close correlation between the Energy metabolism pathway and p__Actinobacteria, p__Firmicutes, p__Tenericutes, and c__Betaproteobacteria. It has been shown that the Energy metabolism pathway is down-regulated in UC [64], and this finding is consistent with our findings, suggesting that these microbiotas may influence the onset and progression of UC by regulating the energy metabolism pathway. Finally, we also observe that the Protein kinases pathway shows significant differences in UC and is strongly associated with g__Blautia. The existing studies are consistent with the results of our study, which also indicate a potential involvement of the Protein kinases pathway in the pathogenesis of UC [65].

By deeply analyzing the functional differences at different taxonomic levels, this study reveals the role of microbiomes at different taxonomic levels and their impact on functional expression. Overall, this study mainly reveals the complex relationship between microbiome and functional modules in UC, especially in Carbohydrate Metabolism, Amino acid metabolism, lipid metabolism, Energy metabolism and Protein kinases pathways are discovered. These results provide new insights into the pathogenesis of UC and may inform the development of future therapeutic strategies targeting these pathways.

Execution time

We also count the run times based on the DIABIMMUNE dataset at different taxonomic levels. As can be seen from the results (see Supplementary Table S8-S9), the Mean and Median methods have negligible time in inferring missing values. For the deep learning methods, all the models basically show a tendency of prolonged running time with the subdivision of the taxonomic level. Overall, SysLM-I achieves a more significant advantage. SysLM-C has shown some efficiency improvement in classification tasks, especially in comparison with DeepMicroGen. However, we are also continuously optimizing its performance with a view to matching other state-of-the-art models in terms of speed. Note that for the inference task, all models are implemented based on the 11th Gen Intel(R) Core (TM) i7-11800 H @ 2.30 GHz. For the classification task, all models are implemented based on a 14 vCPU Intel(R) Xeon(R) Gold 6348 CPU @ 2.60 GHz. Their GPU acceleration is NVIDIA GeForce RTX 3060 Laptop GPU and A800-80GB respectively.

Discussion

Understanding dynamic host–microbiome interactions requires longitudinal data, but pervasive missing values and fragmented analyses often prevent comprehensive biological insights, calling for an integrated and interpretable framework. To address this challenge, we propose SysLM, a systematic framework integrating two complementary models: SysLM-I for missing data imputation and SysLM-C for causal analysis and biomarker identification. SysLM-I leverages advanced deep learning techniques to capture temporal dependencies and causal relationships, effectively restoring missing values while preserving data integrity. SysLM-C constructs interpretable causal spaces to explore associations between the microbiome and host health, supporting the identification and interpretation of potential disease-associated microbial biomarkers.

Through multi-scale temporal and taxonomic analyses from the phylum to genus levels, we identify biomarkers across different classification levels, contributing to a better understanding of potential pathological mechanisms. Functional pathway analyses further support the potential biological relevance of these biomarkers by implicating several metabolic pathways associated with ulcerative colitis, including Carbohydrate metabolism, Amino acid metabolism, Lipid metabolism, Energy metabolism, and pathways involving Protein kinases. These findings suggest that microbial communities may play an important role in disease progression and could inform future research into potential therapeutic strategies.

Despite the promising results demonstrated by the SysLM framework, several limitations should be acknowledged. First, the current model does not explicitly account for complex or varying missing mechanisms, and parameters tuned at higher-level taxonomic resolutions may not be optimal for sparse and variable lower-level data. Second, while our framework identifies biomarkers from the phylum to genus levels, species-level data are sparse and highly variable, which substantially reduces model performance and limits reliable species-level biomarker discovery. Additionally, further experimental validation is necessary, and future work will focus on integrating multi-omics data, expanding SysLM’s applicability, and optimizing the model to handle diverse missing mechanisms and sparse taxonomic levels, thereby enhancing both accuracy and generalizability.

Authors’ contributions

LW, GQ, YS, YM, JG designed the study. LW carried out analyses and wrote the program. LW and JG wrote the paper. All authors read and approved the final manuscript.

Funding

The work is supported by the National Natural Science Foundations of China (Grant No. 12271216, 11831015, 92370131).

Data availability

The raw data used in this study are described in the Materials and Methods section under Datasets. Specifically, the DIABIMMUNE dataset is accessible via DIABIMMUNE Microbiome Project (https://diabimmune.broadinstitute.org/diabimmune), with DOI: 10.1016/j.cell.2016.04.007 and MicrobiomeDB (https://microbiomedb.org), with DOI: 10.1093/nar/gkx1027. The BONUS-CF dataset is available from MicrobiomeDB (https://microbiomedb.org), with DOI: 10.1038/s41591-019-0714-x and 10.1093/nar/gkx1027. The DiGiulio dataset is available with DOI: 10.1073/pnas.1502875112 and 10.1186/s12859-020-03803-z. The PROTECT dataset is available with DOI: 10.1016/j.chom.2018.09.009. The iHMP-IBD dataset is available via IBDMDB (http://ibdmdb.org), with DOI: 10.1016/j.chom.2014.08.014 and 10.1038/s41586-019-1237-9. The iHMP-T2D dataset is available with DOI: 10.1016/j.chom.2014.08.014. Both iHMP datasets are described in the HMPDACC multi-omic data resource article (10.1093/nar/gkaa996). To facilitate reproducibility, a representative processed dataset has been deposited in Figshare (10.6084/m9.figshare.30443732). The source code is available in a public GitHub repository: https://github.com/wang-124/SysLM-model.git.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.