Integrating tongue image and health examination records for MASLD risk prediction via multimodal deep learning

Abstract

Metabolic dysfunction-associated steatotic liver disease (MASLD) is a growing global health concern, but its early detection remains limited by the cost and accessibility of conventional diagnostic tools. This study aimed to develop an interpretable multimodal deep learning framework for MASLD classification by integrating tongue images with routine health examination data. A total of 711 matched tongue images and health records were collected from Shanghai Changhai Hospital between June and December 2024. Residual Network-50 was used to extract image features, while a graph attention network modeled structured clinical data. An attention-based fusion module integrated the 2 modalities for classification. DeepMASLD achieved an accuracy of 92.04% and an F1 score of 0.91, outperforming the corresponding unimodal models. Triglycerides, alanine aminotransferase, and total cholesterol were among the most influential structured features, while class activation maps highlighted the posterior and lateral tongue regions. This study presents an accessible, interpretable multimodal framework integrating tongue images with clinical indicators for MASLD screening, achieving 92.04% accuracy and an F1 score of 0.91. In summary, DeepMASLD may support scalable early detection, pending multicenter and histopathological validation.

1. Introduction

Metabolic dysfunction-associated steatotic liver disease (MASLD) is a chronic liver disease closely associated with metabolic dysfunction and has become one of the most common liver diseases worldwide.^[1,2] The progression of MASLD can result in severe clinical outcomes, including hepatocellular carcinoma, cirrhosis, and hepatic fibrosis.^[3,4] It is also closely linked with a range of metabolic comorbidities such as type 2 diabetes, hypertension, cardiovascular disease, chronic kidney disease, and systemic inflammation.^[5,6] Consequently, early detection and timely intervention are essential to mitigate disease progression and minimize the risk of complications.

Despite its significance, the clinical screening and identification of MASLD remain challenging. The disease is frequently asymptomatic in its early stages, and conventional biochemical tests have limited sensitivity for detecting hepatic steatosis or inflammation. This limitation may result in a substantial number of high-risk individuals going undiagnosed. The standard diagnostic approach typically integrates clinical history, laboratory parameters, and imaging studies. Although liver biopsy remains the gold standard for definitive diagnosis, providing precise histological insights into steatosis, inflammation, and fibrosis,^[7] its invasiveness, cost, and associated risks hinder its feasibility for large-scale population screening.

In response to these limitations, recent advancements in artificial intelligence have led to growing interest in utilizing noninvasive biomarkers and machine learning algorithms for MASLD risk prediction. Early studies in this area have yielded encouraging results.^[8,9] However, many existing models suffer from notable limitations. Some models are tailored to specific populations,^[10] whereas others rely heavily on high-dimensional omics data or specialized biomarkers.^[8,9] Many are also limited by small sample sizes or insufficient consideration of population heterogeneity.^[11,12] As a result, there remains an unmet need for a robust, scalable, and generalizable MASLD risk prediction tool suitable for use in large-scale health screening across diverse populations.

In recent years, the widespread application of artificial intelligence in medical image analysis has led to growing interest in the predictive value of noninvasive imaging modalities for disease detection.^[13,14] Among these, tongue imaging, a fundamental component of diagnostic practice in traditional Chinese medicine, has attracted attention because it can reflect the functional state of internal organs.^[15] Advances in computer vision and machine learning have enabled the systematic analysis of tongue images. Preliminary findings indicate correlations between visual features, such as color, coating thickness, and texture, and various clinical biomarkers.^[16,17] Specifically, features of the tongue coating have been found to correlate with liver function indicators, including alanine aminotransferase (ALT), aspartate aminotransferase, and gamma-glutamyl transferase, as well as metabolic markers such as body mass index, lipid profile, and uric acid levels.^[18]

Building on this foundation, we propose DeepMASLD, a deep learning-based multimodal framework for MASLD risk prediction. This model integrates noninvasive tongue image data with structured physical examination records. In DeepMASLD, a convolutional neural network (CNN) extracts high-level visual features from tongue images, while a graph neural network (GNN) models the structured clinical data. A feature fusion module then enables joint representation learning across these heterogeneous data modalities, thereby supporting accurate and individualized MASLD risk prediction.

This approach offers a novel and scalable solution for noninvasive MASLD screening, potentially enhancing early risk stratification and clinical decision-making. Moreover, it provides both a theoretical framework and empirical evidence to support the integration of tongue image analysis into modern biomedical research and clinical practice.

2. Materials and methods

2.1. Study participants

This study was designed as a prospective cohort investigation to develop and validate a prediction model for MASLD. The derivation cohort was recruited from individuals undergoing routine health examinations at the Health Checkup Center of Shanghai Changhai Hospital between June 1 and December 31,2024. The study protocol was approved by the Research Ethics Committee of Changhai Hospital (approval no. CHEC2025-129). The study was conducted in accordance with the Declaration of Helsinki, and written informed consent was obtained from all participants before enrollment. Throughout the study, participant privacy and data confidentiality were strictly protected, and no interventional procedures were performed.

Participants were considered eligible for inclusion if they were between 18 and 65 years of age and fulfilled the diagnostic criteria for MASLD. Specifically, eligible participants reported no alcohol consumption or only light-to-moderate alcohol intake, showed ultrasonographic features consistent with fatty liver, and had received no previous treatment for liver disease.

2.1.1. Exclusion criteria

Participants were excluded if they were pregnant or breastfeeding; had comorbid malignancies or autoimmune liver diseases; had a history of viral hepatitis, autoimmune hepatitis, primary biliary cholangitis, or other chronic liver diseases; or had incomplete or missing clinical data required for subsequent analysis.

2.1.2. Control group

The control group consisted of healthy individuals undergoing routine clinical assessment at the same center during the study period, for whom no evidence of acute or chronic liver disease was identified.

By applying clearly defined inclusion and exclusion criteria and enrolling participants from a real-world health examination setting, this study established a stable and clinically relevant dataset for model development. This design was intended to support the robustness of the prediction framework and enhance its potential applicability in routine clinical practice.

2.2. Data collection and processing

To support model development, comprehensive physical examination data were extracted from the hospital’s electronic health record system. For each participant, a total of 137 features were collected, covering a wide range of clinical domains. These variables included demographic characteristics, anthropometric measurements, blood pressure measurements, cardiovascular indicators, complete blood counts, hepatic and renal function indices, glycemic parameters, diabetes-related biomarkers, lipid profiles, thyroid function indices, tumor markers, immune and inflammatory indicators, urinalysis findings, and ophthalmological assessments. In addition to these conventional clinical variables, tongue diagnosis features and meridian indicators reflecting traditional Chinese medicine perspectives were also included.^[19]

Tongue images were acquired using an Intelligent Constitution Identifier (model DS01-A; Shanghai Daosheng Medical Technology Co., Ltd.). The tongue imaging unit incorporated a face frame, chin support, and saliva collection tray. It used a simulated daylight light-emitting diode area-light source with a color rendering index of at least 90, a color temperature range of 4000 to 7000 K, and a nominal illuminance of 3000 ± 10% lx. All data were collected using the same device in an integrated acquisition environment. The enclosed acquisition space and integrated positioning supports were used to reduce variations in ambient lighting, head position, camera-to-subject distance, and acquisition angle. However, minor differences in lighting and other acquisition conditions could not be completely eliminated across participants. During image acquisition, participants were seated in front of the device, with their foreheads and chins positioned against the integrated supports. They were instructed to maintain a neutral facial expression and fully protrude the tongue. After acquisition, 2 qualified physicians independently reviewed the tongue and facial images and the device-generated results to verify data quality.

All collected variables were organized into a structured dataset for subsequent analysis. Integrating routine health examination indicators with tongue-related features enabled the construction of a multimodal data resource, providing the basis for downstream model training and validation.

2.3. Definition of MASLD

MASLD was diagnosed and classified based on the controlled attenuation parameter (CAP), a quantitative marker of hepatic steatosis measured using transient elastography. All enrolled participants underwent CAP assessment via FibroScan (Echosens), conducted by trained technicians in accordance with standardized operating procedures. CAP measurements were considered valid only when the interquartile range was < 40 dB/m, ensuring measurement reliability. A CAP value of ≥ 238 dB/m was used to define the presence of MASLD, whereas values < 238 dB/m indicated normal hepatic fat content.

2.4. Model framework

We propose a multimodal deep learning framework that integrates structured clinical features and tongue image data to classify and stage MASLD, with the overall architecture visualized in Figure 1. The architecture consists of 3 main modules: a GNN for structured data modeling, a CNN for image feature extraction, and an attention-based feature fusion classifier.

Figure 1.

Provides an overview of the proposed multimodal fusion framework for MASLD classification. The model integrates clinical examination indicators and tongue images as 2 complementary data modalities. A patient similarity graph is constructed from the structured clinical features and processed by a GNN to generate structured embeddings. Simultaneously, ResNet-50 extracts image embeddings from tongue images. The 2 types of embeddings are concatenated and passed through a neural network classifier to predict the stage of MASLD (normal, mild, moderate, or severe). ALT = alanine aminotransferase, AST = aspartate aminotransferase, BMI = body mass index, CNN = convolutional neural network, GAT = graph attention network, GNN = graph neural network, MASLD = metabolic dysfunction-associated steatotic liver disease, ResNet = Residual Network.

2.5. Graph construction and GNN modeling

2.5.1. Graph construction

Given N individuals with F-dimensional clinical features, the structured clinical data were represented as a feature matrix X ∈ ℝ^{N × F}, where each row corresponds to 1 participant and each column corresponds to 1 clinical variable. To explicitly model inter-patient relationships that may not be fully captured by independent tabular analysis, we constructed a patient similarity graph G = (V, E), in which each node v_i ∈ V denotes an individual participant. Edges in E were established using the k-nearest neighbors method based on cosine similarity.^[20]

The graph construction step was introduced to encode local neighborhood structure in the clinical feature space. In this setting, patients with similar metabolic, biochemical, and anthropometric profiles are connected, allowing the model to leverage clinically meaningful relational information rather than treating each sample as an isolated observation. Cosine similarity was adopted because it measures similarity in feature direction rather than absolute magnitude, which is advantageous when clinical indicators are heterogeneous in scale and units. The edge weight between nodes i and j was defined as:

$$\begin{array}{c}{\displaystyle }{A}_{ij}=\{\begin{array}{cc}\mathrm{c}\mathrm{o}\mathrm{s}({\mathrm{x}}_i,{\mathrm{x}}_j),& \mathrm{i}\mathrm{f}{\mathrm{x}}_j\in \mathrm{T}\mathrm{o}\mathrm{p}-k\left({\mathrm{x}}_i\right)\\ 0,& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}\end{array}$$ (1)

This formulation ensures that each patient is connected only to the most similar neighbors, thereby preserving local structure while reducing the influence of weak or noisy associations. Such a sparse similarity graph is beneficial for downstream graph learning because it improves computational efficiency and promotes the aggregation of information from clinically relevant neighboring samples.

2.5.2. Graph attention network (GAT)

To learn patient representations from the constructed graph, we adopted the GAT.^[21] Compared with conventional graph convolution methods that assign uniform or fixed normalized weights to neighboring nodes, GAT introduces an adaptive attention mechanism that enables the model to learn the relative importance of different neighbors. This design is particularly suitable for clinical data, in which not all similar patients contribute equally to the representation of a target patient.

For a node $i$, the unnormalized attention score between node $i$ and its neighbor $j$ was computed as:

$$\begin{array}{c}{\displaystyle }{e}_{ij}=\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left({\mathrm{a}}^T\left[\mathrm{W}{\mathrm{h}}_i||\mathrm{W}{\mathrm{h}}_j\right]\right)\end{array}$$ (2)

where ${\mathrm{h}}_i$ and ${\mathrm{h}}_j$ denote the input features of nodes i and j, respectively, W is a learnable linear transformation, a is the learnable attention vector, and ‖ denotes concatenation.

The attention scores were then normalized across the neighborhood of node $i$ using the softmax function:

$$\begin{array}{c}{\displaystyle }{\alpha }_{ij}=\frac{\exp \left(e_{ij}\right)}{{\sum }_{k\in 𝒩\left(i\right)}\exp \left(e_{ik}\right)}\end{array}$$ (3)

where $𝒩\left(i\right)$ denotes the set of neighboring nodes of node $i$. The updated representation of node $i$ was then obtained as a weighted aggregation of the transformed neighboring features:

$$\begin{array}{c}{\displaystyle }{{\mathrm{h}}_i}^{\prime }=\sigma \left(\sum _{j\in 𝒩\left(i\right)}{\alpha }_{ij}\mathrm{W}{\mathrm{h}}_j\right)\end{array}$$ (4)

where σ is an activation function, implemented as exponential linear unit in this study.

To enhance representation capacity, 2 GAT layers were stacked. The first layer employed an 8-head attention mechanism to capture diverse relational patterns from different attention subspaces, whereas the second layer used a 1-head mechanism to generate the final compact embedding. Batch normalization^[22] and dropout^[23] were applied between layers to improve optimization stability and reduce overfitting.

2.5.3. GNN output and loss function

The GNN produced node embeddings Z_gnn ∈ ℝ^{N × d}, which served as learned representations of the structured clinical modality. These embeddings were used both for direct classification and for subsequent multimodal fusion with image-derived features.

For supervised training, the graph branch was optimized using the cross-entropy loss:

$$\begin{array}{c}{\displaystyle }{L}_{\mathrm{G}\mathrm{N}\mathrm{N}}=-\sum _{i\in Y}\underset{c=1}{\sum ^C}{y}_{ic}\mathrm{l}\mathrm{o}\mathrm{g}\left({\hat{y}}_{ic}\right)\end{array}$$ (5)

where ŷ_ic denotes the predicted probability that sample i belongs to class c, y_ic is the corresponding ground-truth label, Y denotes the set of labeled samples, and C is the number of classes.

2.6. CNN-based image feature extraction

The tongue image modality was modeled using a pretrained Residual Network (ResNet)-50 backbone.^[24] We discarded the original classification head and retained the convolutional feature extractor to obtain high-level visual representations from tongue images.

The resulting 2048-dimensional feature vector f_img was flattened and projected into a 1024-dimensional latent representation:

$$\begin{array}{c}{\displaystyle }{\mathrm{z}}_{\mathrm{c}\mathrm{n}\mathrm{n}}={\mathrm{W}}_{\mathrm{c}\mathrm{n}\mathrm{n}}{\mathrm{f}}_{\mathrm{i}\mathrm{m}\mathrm{g}}+{\mathrm{b}}_{\mathrm{c}\mathrm{n}\mathrm{n}}\end{array}$$ (6)

where W_cnn ∈ ℝ^{1024 × 2048} is a learnable weight matrix, and b_cnn is the corresponding bias term.

2.7. Multimodal fusion and classification

2.7.1. Feature projection and alignment

Because the graph and image branches produce embeddings from different data sources and with different statistical properties, we projected them into a shared latent space before fusion. Specifically, the GNN and CNN embeddings were independently transformed into a common 256-dimensional space using linear mapping followed by LayerNorm and Rectified Linear Unit (ReLU) activation:

$$\begin{array}{c}{\displaystyle }{\mathrm{z}}_{\mathrm{g}\mathrm{n}\mathrm{n}}^{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{j}}=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\left({\mathrm{W}}_g{\mathrm{z}}_{\mathrm{g}\mathrm{n}\mathrm{n}}\right)\right)\end{array}$$ (7)

$$\begin{array}{c}{\displaystyle }{\mathrm{z}}_{\mathrm{c}\mathrm{n}\mathrm{n}}^{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{j}}=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\left({\mathrm{W}}_c{\mathrm{z}}_{\mathrm{c}\mathrm{n}\mathrm{n}}\right)\right)\end{array}$$ (8)

2.7.2. Attention-based feature fusion

After projection, the 2 modalities were integrated using an attention-based fusion mechanism. The attention weights α = [α_gnn, α_cnn] were computed as:

$$\begin{array}{c}{\displaystyle }\alpha =\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\mathrm{W}}_2\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left({\mathrm{W}}_1\left[{\mathrm{z}}_{\mathrm{g}\mathrm{n}\mathrm{n}}^{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{j}}||{\mathrm{z}}_{\mathrm{c}\mathrm{n}\mathrm{n}}^{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{j}}\right]\right)\right)\end{array}$$ (9)

where [·‖·] denotes concatenation. The final fused representation was then defined as:

$$\begin{array}{c}{\displaystyle }{\mathrm{z}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}={\alpha }_{\mathrm{g}\mathrm{n}\mathrm{n}}·{\mathrm{z}}_{\mathrm{g}\mathrm{n}\mathrm{n}}^{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{j}}+{\alpha }_{\mathrm{c}\mathrm{n}\mathrm{n}}·{\mathrm{z}}_{\mathrm{c}\mathrm{n}\mathrm{n}}^{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{j}}\end{array}$$ (10)

2.7.3. Fusion classifier

The fused representation was passed through an multilayer perceptron classifier to predict the target class:

$$\begin{array}{c}{\displaystyle }\hat{y}=\mathrm{M}\mathrm{L}\mathrm{P}\left({\mathrm{z}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}\right)\end{array}$$ (11)

The multilayer perceptron consisted of 3 fully connected layers with ReLU activation and dropout between layers.^[25] The final prediction was supervised using categorical cross-entropy:

$$\begin{array}{c}{\displaystyle }{L}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}=-\sum _{i\in Y}\underset{c=1}{\sum ^C}{y}_{ic}\mathrm{l}\mathrm{o}\mathrm{g}\left({\hat{y}}_{ic}\right)\end{array}$$ (12)

where ŷ_ic is the predicted probability for class c for sample i, and y_ic is the corresponding ground-truth label.

2.8. Training protocol

The training process was divided into 2 stages to improve optimization stability and enable the structured clinical branch first to learn informative graph-based representations before multimodal integration.

In Stage 1, the GNN was trained using structured clinical data only. The model was optimized for 600 epochs with a learning rate of 0.003 using the Adam optimizer, and early stopping was applied based on validation accuracy.

In Stage 2, after image features were extracted and pretrained GNN embeddings were loaded, the fusion classifier was trained for 1000 epochs with a learning rate of 1 × 10⁻⁴.

All models were implemented in PyTorch and PyTorch Geometric^[26] and were trained on NVIDIA graphics processing units.

3. Results

3.1. Model performance and class-wise analysis

To provide context for the subsequent class-wise performance analysis, we first examined the distribution of MASLD severity categories and their corresponding CAP values. Figure 2A shows the distribution of MASLD severity categories in the study cohort. Among the 711 participants, 29.1% were categorized as normal, 12.9% as mild, 22.3% as moderate, and 35.7% as severe MASLD, indicating a substantial representation of varying disease stages and supporting model evaluation across clinically relevant classes. Figure 2B shows the distribution of CAP values across the 4 severity categories, annotated with category-wise classification accuracy. The median CAP values increased progressively from the normal to severe groups, in accordance with clinical staging thresholds. Accuracy was highest in the Severe and Normal groups, while misclassification occurred more frequently in the mild and moderate stages, likely due to their proximity in CAP ranges.

Figure 2.

Distribution of MASLD severity categories and CAP values. (A) Proportions of participants classified as normal, mild, moderate, and severe in the overall study cohort; (B) distribution of CAP values across the 4 severity categories in the test set, together with category-specific classification accuracy. CAP = controlled attenuation parameter, MASLD = metabolic dysfunction-associated steatotic liver disease.

After examining the class-wise distribution and CAP characteristics, we further evaluated the effectiveness of the proposed multimodal model by comparing it with several widely used machine learning algorithms, including support vector machine,^[27] random forest,^[28] gradient boosting machine (GBM),^[29] light GBM,^[30] and eXtreme Gradient Boosting.^[31] Figure 3 shows that DeepMASLD achieved the best overall performance, with an accuracy of 0.92 and an F1 score of 0.91. Among the conventional baselines, eXtreme Gradient Boosting ranked second, followed by light GBM, GBM, and random forest, whereas support vector machine obtained the lowest scores. These results demonstrate the effectiveness of multimodal fusion for MASLD risk stratification.

Figure 3.

Performance comparison across multiple machine learning models for MASLD risk classification. The DeepMASLD model outperformed traditional machine learning baselines in both classification accuracy and F1 score. GBM = gradient boosting machine, LGBM = light gradient boosting machine, MASLD = metabolic dysfunction-associated steatotic liver disease, SVM = support vector machine, XGBoost = extreme gradient boosting.

3.1.1. Test distribution

Figure 4 provides a visual assessment of the proposed multimodal model on the internal test set. The predicted class distribution closely matched the true test-set distribution, yielding an overall accuracy of 92.04%. The t-distributed stochastic neighbor embedding projections revealed clear but partially overlapping class clusters, while the error analysis indicated that misclassifications were concentrated near class boundaries. The CAP-based visualization further confirmed the progressive separation of MASLD severity categories.

Figure 4.

Visual evaluation of multimodal model performance on the MASLD test set. The predicted label distribution closely matched the true label distribution, while most classification errors occurred between adjacent disease stages. CAP = controlled attenuation parameter, MASLD = metabolic dysfunction-associated steatotic liver disease.

3.2. Ablation study

To evaluate the contribution of different data modalities, we conducted an ablation study by comparing 3 variants of the proposed DeepMASLD model: DeepMASLD (GNN), which utilized only structured clinical data through a GNN; DeepMASLD (ResNet-50), which used only tongue image data via a ResNet-50-based feature extractor; and the full model DeepMASLD, which integrated both modalities using an attention-based fusion mechanism.

Figure 5 quantifies the predictive performance of all proposed DeepMASLD variants. The full DeepMASLD model achieved the highest performance with an accuracy of 0.92 and an F1 score of 0.91, demonstrating the advantage of multimodal fusion. In comparison, DeepMASLD (GNN), using only clinical features, achieved an accuracy of 0.90 and an F1 score of 0.87, indicating strong predictive power of structured data alone. However, DeepMASLD (ResNet-50), which relied solely on tongue images, performed substantially worse, with an accuracy of 0.34 and an F1 score of 0.30, suggesting that tongue images alone were insufficient for accurate MASLD staging.

Figure 5.

Performance comparison of DeepMASLD with its unimodal variants. The multimodal model outperformed models based only on structured clinical data or tongue images. MASLD = metabolic dysfunction-associated steatotic liver disease.

3.3. Model interpretability

To improve the interpretability of DeepMASLD, we applied feature attribution to the structured clinical data and visual explanation techniques to the tongue images.

For structured clinical data, we utilized analysis of variance to assess the contribution of each feature to model predictions. Figure 6 displays the top 15 features ranked by analysis of variance F-value scores. Notably, triglycerides, ALT, and total cholesterol were identified as the most influential predictors, consistent with known clinical risk factors for MASLD. In addition, meridian indicators such as Kidney Meridian and Pericardium Meridian also demonstrated considerable predictive value.

Figure 6.

Top 15 most important structured features ranked by ANOVA F-value scores. TG, ALT, and TC were among the most influential variables for MASLD prediction. ALT = alanine aminotransferase, ANOVA = analysis of variance, BMI = body mass index, FPG = fasting plasma glucose, LDLC = low-density lipoprotein cholesterol, MASLD = metabolic dysfunction-associated steatotic liver disease, TC = total cholesterol, TG = triglycerides, UA = uric acid.

For tongue image analysis, we adopted class activation mapping techniques^[32] to visualize the regions of interest that contributed most to model classification. Figure 7 shows a progressive change in feature representation across different network depths. The early feature maps primarily preserved global tongue contours, color, and texture information, whereas the middle feature maps produced more localized responses in the posterior, lateral, and central tongue regions. In the deep layers, the activation patterns became increasingly sparse and discriminative, indicating that the model selectively retained class-relevant regional information while suppressing less informative background features. The variation in activation locations across samples also suggests that the model captured heterogeneous visual characteristics rather than relying on a single fixed region.

Figure 7.

Class activation mapping visualizations of tongue images at different feature extraction stages. The model consistently focused on posterior and lateral tongue regions in moderate and severe cases.

4. Discussion

This study developed DeepMASLD, a multimodal deep learning framework for the staging of MASLD by jointly integrating structured clinical variables and tongue images. Compared with the corresponding unimodal models, DeepMASLD achieved superior predictive performance, indicating that heterogeneous data sources provide complementary information for disease characterization. The structured clinical branch contributed substantially to the final prediction, with triglycerides, ALT, and total cholesterol consistently identified as influential features. These findings are biologically plausible because abnormalities in lipid metabolism and liver enzyme levels are closely related to the development and progression of MASLD. In addition, several meridian-related features derived from traditional Chinese medicine exhibited relatively high importance scores, suggesting that these variables may contain potentially useful phenotypic information when incorporated into a modern data-driven diagnostic framework. However, their clinical interpretation requires further validation in larger and more diverse populations. Although the tongue-image model alone showed relatively limited discriminative ability, incorporating image representations extracted by ResNet-50 into the attention-based multimodal fusion module improved the overall staging performance. This result suggests that tongue images may not provide sufficiently strong diagnostic information in isolation but can contribute complementary visual cues when combined with metabolic and biochemical indicators. The attention mechanism enabled the model to adaptively weight information from different modalities, thereby enhancing the representation of clinically relevant features while reducing the influence of less informative signals.

Feature-attribution analysis further confirmed that the model relied on key metabolic and hepatic indicators rather than spurious variables, which improved the interpretability and clinical plausibility of the predictions. Meanwhile, class activation mapping revealed that the image branch repeatedly focused on the posterior and lateral regions of the tongue. According to traditional Chinese medicine theory, these regions are commonly linked to internal organ-related functional patterns. Although such spatial attention does not establish a direct physiological relationship, the consistency of the highlighted regions suggests that the model may have captured reproducible visual characteristics associated with disease stage. Overall, these findings support the feasibility of combining conventional clinical data with tongue-image features for MASLD staging and provide a foundation for developing more interpretable and clinically applicable multimodal diagnostic systems.

From a clinical implementation perspective, DeepMASLD has the potential to function as a clinical decision-support tool within routine primary care, preventive health examinations, and chronic disease management programs. Structured clinical variables required by the model could be automatically retrieved from electronic health records, laboratory information systems, or health examination databases, thereby minimizing additional data-entry burden. In parallel, standardized tongue images could be collected using dedicated imaging devices under controlled lighting, distance, angle, and color-calibration conditions. Following further technical and clinical validation, portable or smartphone-based image acquisition systems may also provide a more accessible option for community-based screening and remote healthcare settings. Once the required multimodal data are obtained, the model could generate an initial estimate of MASLD stage and present the result together with risk probabilities or clinically interpretable feature contributions. Such output should not replace liver imaging, laboratory interpretation, or specialist judgment. Instead, it could support preliminary risk stratification, help prioritize patients for additional assessment, and identify individuals who may benefit from ultrasonography, elastography, advanced biochemical testing, or hepatology referral. The system may be particularly valuable in settings where specialist resources are limited because it could assist primary care clinicians in recognizing patients with a higher likelihood of clinically significant disease. Successful translation into routine practice would nevertheless require several implementation safeguards. First, image acquisition protocols must be standardized to reduce variability caused by illumination, camera type, patient positioning, tongue movement, and operator technique. Second, the model should be interoperable with hospital information systems and use standardized data formats to ensure reliable extraction and transmission of clinical variables. Third, appropriate privacy protection, access control, encryption, audit logging, and informed data-use procedures should be established because the system processes both personal health information and identifiable images. In addition, clinicians and technicians would require training in data collection, model interpretation, and management of uncertain or discordant outputs. Most importantly, DeepMASLD should undergo prospective, multicenter, and externally validated evaluation across diverse populations and healthcare environments before clinical deployment. Future studies should also assess calibration, robustness, workflow efficiency, cost-effectiveness, patient acceptability, and the extent to which model-assisted screening improves referral decisions and clinical outcomes.

This study has several limitations. First, the dataset was collected from a single center and without external cross-center validation. This design may introduce center-specific selection bias and may not fully capture variations in patient demographics, disease severity, imaging equipment, image acquisition procedures, and clinical practice across different institutions. Consequently, the observed performance may partly reflect characteristics specific to the local cohort and may not be directly generalizable to other populations or clinical settings. Multicenter datasets would provide greater demographic, geographic, and operational diversity, allowing a more rigorous assessment of model robustness and generalizability across institutions. Future studies should therefore conduct prospective external validation in independent hospitals from different geographic regions. Prior to external validation, the model architecture, preprocessing workflow, classification thresholds, and decision-making algorithms shall be prespecified and frozen throughout the validation phase. Model performance should then be evaluated separately at each center and in the pooled cohort, including discrimination, calibration, and subgroup analyses. If substantial performance differences are observed across centers, model recalibration or domain-adaptation strategies should be considered before broader clinical deployment. Second, MASLD staging was based on CAP rather than histopathological confirmation. Although CAP is a validated noninvasive measure of hepatic steatosis, it does not provide histological information on hepatic inflammation or fibrosis and may introduce misclassification, particularly at the cutoff boundaries of successive disease stages. Such misclassification may affect model training and performance evaluation. Future studies should validate the model in cohorts with histopathological confirmation where feasible. In addition, image quality varied due to inconsistent acquisition settings. Future work should also incorporate additional modalities and refine model interpretability.

5. Conclusions

This paper provides a promising and interpretable multimodal approach (DeepMASLD) for noninvasive MASLD staging by integrating complementary clinical and visual information. The DeepMASLD achieved an accuracy of 92.04% and an F1 score of 0.91 on the internal test set, demonstrating the value of multimodal integration. In summary, these findings support the feasibility of DeepMASLD for comprehensive MASLD assessment. However, the single-center design and CAP-based staging without histopathological confirmation may limit generalizability and introduce potential misclassification. Multicenter external validation and validation against histopathological reference standards, where feasible, are therefore required before broader clinical application.

Acknowledgments

The authors thank Shanghai Changhai Hospital for supporting this work. This work was supported by the Traditional Chinese Medicine standardization program (grant no. 2025BZ007). The authors declare no conflicts of interest.

Author contributions

Conceptualization: Xiaohua Hu, Jing Qiu, Wenyi Pei.

Data curation: Xiaohua Hu.

Funding acquisition: Jing Qiu, Wenyi Pei.

Investigation: Jing Qiu.

Methodology: Xiaohua Hu, Jia Shi, Zhehong Zhang.

Project administration: Jing Qiu.

Resources: Xiaohua Hu.

Software: Jia Shi, Zhehong Zhang.

Supervision: Jing Qiu, Wenyi Pei.

Validation: Zhehong Zhang, Wenyi Pei.

Writing – original draft: Xiaohua Hu.

Writing – review & editing: Jia Shi.