Revealing hidden material composition via physics-induced deep learning for precision clinical diagnosis

Summary

Material-composition information extracted from spectral computed tomography (CT) images facilitates clinical diagnosis by characterizing pathological tissues. However, the complexity of human anatomy and the diversity of tissue types make precise material quantification from single-energy CT (SECT) highly challenging. Here, we propose an X-ray imaging induced multi-material decomposition (MMDX) framework. By integrating the physical principles of photoelectric effect and Compton scattering into a deep-learning network architecture, MMDX achieves superior performance on both phantom and clinical patient datasets, with a 12.92-dB improvement in peak signal-to-noise ratio (PSNR) and a 3.94% increase in volume fraction accuracy. To evaluate the clinical utility of MMDX, we constructed a large-scale clinical CT dataset with 1,637,738 CT images from 7,629 patients. MMDX demonstrates superior performance across 20 downstream tasks, including three diagnosis tasks, two prognosis tasks, and 15 biomarker prediction tasks, showing promise as an accessible AI tool for precision clinical diagnosis.

Highlights

• •MMDX is a multi-material-decomposition framework for clinical diagnosis
• •Physics-induced AI extracts material composition from single-energy CT images
• •Material information improves performance across 20 downstream tasks
• •Efficient computation supports scalable clinical deployment

The bigger picture

Computed tomography (CT) is a widely used medical imaging modality that enables non-invasive visualization of internal anatomical structures. CT scans acquire X-ray projections from multiple angles and reconstruct them into high-resolution cross-sectional images. These images provide detailed information about bones, organs, and soft tissues, making CT an essential tool for disease diagnosis, treatment planning, and longitudinal patient monitoring in clinical practice. While conventional single-energy CT provides detailed anatomical structures, many important clinical decisions also rely on information regarding tissue composition. Material composition can reveal subtle biological and pathological characteristics, including tissue density and variations in tissue constituents such as fat and muscle. However, due to the complexity of human anatomy and the heterogeneity of tissue types, precise material quantification remains challenging. This study introduces a material decomposition method that incorporates the physics of X-ray imaging. The ability to extract tissue information from routine CT scans bridges the gap between anatomical imaging and precision medicine, thereby enhancing the clinical utility of standard CT examinations, particularly in resource-limited settings where advanced imaging technologies are not readily available. More broadly, our work highlights the potential of integrating domain-specific physical knowledge with artificial intelligence to facilitate the development of clinically deployable and accessible precision healthcare solutions.

Liu et al. present a physics-induced deep-learning approach that extracts tissue composition information from routine single-energy CT scans. By incorporating X-ray imaging principles into model design, this approach improves material characterization and enhances performance across diverse clinical tasks. The framework expands the diagnostic value of standard CT imaging and supports more accessible precision medicine.

Introduction

X-ray computed tomography (CT) allows for the precise revelation of the internal structure of materials non-destructively, leading to its widespread application in biomedical engineering fields.¹ CT scanners utilize X-rays to perform cross-sectional scanning of the human body, thereby providing detailed anatomical information. However, relying solely on the morphological features of organs and tissues is insufficient to uncover the underlying mechanisms of diseases. Growing evidence suggests that alterations in material composition are closely associated with the development of various diseases.^2,3 Therefore, extracting material-composition information from X-ray CT images represents a more effective approach for precision clinical diagnosis.

CT scanners generate images of the linear attenuation coefficients of materials, displayed as Hounsfield unit (HU) values. The linear attenuation coefficient of a material depends on the effective atomic number, mass density, and X-ray energy level, providing the physical basis for material decomposition.^4,5,6,7 At a given X-ray energy level, the combined influence of effective atomic number and mass density can result in similar attenuation coefficients for different materials, thereby making material discrimination challenging in CT images.⁸ Human organs and tissues are composed of various typical materials, including blood, fat, muscle, water, and air.^9,10,11 A single image pixel may contain one or more material types, and its HU value reflects a mixture of the linear attenuation coefficients. Consequently, accurately differentiating and quantifying individual materials is inherently a highly ill-posed problem, particularly in single-energy CT (SECT) images. Several studies have explored material decomposition from SECT based on X-ray path-length estimation,^12,13 in which projection data and estimated path lengths are mapped to the corresponding path lengths of two basis materials to enable decomposition. In recent years, artificial intelligence (AI) has made remarkable advances in disease diagnosis, treatment planning, and prognostic evaluation, offering perspectives for various medical applications.^14,15,16 In this context, deep-learning-based methods have been introduced for SECT material decomposition to facilitate feature extraction and noise suppression,^17,18,19 which have demonstrated promising performance. However, these approaches mainly focus on a limited number of decomposable materials (e.g., water and iodine), which may restrict their applicability in clinical scenarios. The introduction of material sparsity constraints²⁰ has further enabled multi-material decomposition and improved decomposition flexibility. Nevertheless, the incorporation of multiple regularization terms may increase computational complexity and lead to sensitivity to initialization, thereby presenting challenges for practical deployment.

Dual-energy CT (DECT)⁴ acquires CT data at two distinct X-ray energy levels, thereby providing two independent attenuation measurements. Compared with SECT, the additional spectral information makes DECT more suitable for material decomposition, leading to widespread clinical applications including virtual monoenergetic imaging,²¹ CT angiography,²² gout differentiation,²³ silicone-implant detection,²⁴ and urinary-stone detection.²⁵ However, due to equipment complexity, sophisticated operational requirements, and high costs, DECT scanners remain largely unavailable in low- and middle-income countries and regions. Most existing multi-material-decomposition methods for DECT images rely on mathematical optimization and regularization theories,^{26,27,28,29,30,31} resulting in computationally intensive and time-consuming procedures that limit their clinical practicality. Recently, deep-learning-based approaches have been proposed to address the ill-posed problem.^32,33,34 However, existing methods remain limited by a restricted range of decomposable materials, insufficient validation on real-world CT data, and the lack of large-scale clinical trials, thereby hindering their clinical deployment.

In recent years, emerging studies have enhanced the representation capacity of deep-learning models by integrating physical laws and domain-specific constraints into the training process, demonstrating broad applicability in fields such as fluid mechanics,³⁵ materials science,³⁶ and biomedicine.³⁷ For real-world CT data, the photoelectric effect and Compton scattering play crucial roles in the X-ray imaging process. By introducing the physical laws into neural networks, it is possible to effectively constrain the complex solution space of multi-material-decomposition tasks, improving both the accuracy and physical plausibility of the results.

In this work, we propose an X-ray imaging induced multi-material decomposition (MMDX) framework, which consists of the DECT generation network, MMDX network, and precision clinical diagnosis system. Experimental results obtained from both phantom and clinical datasets demonstrate the effectiveness of MMDX. In addition, a wide range of clinical applications were conducted using a large-scale clinical CT dataset comprising 1,637,738 CT images from 7,629 patients. Extensive experiments across 20 downstream tasks demonstrate that the precision clinical diagnosis system, which utilizes material-composition information, significantly outperforms conventional SECT images and offers enhanced interpretability for clinical decision-making. Notably, MMDX outperforms existing CT foundation models at a significantly lower computational cost and also achieves performance superior to that of radiologists. The high-precision performance, combined with low computational resource requirements, supports the deployment of MMDX across diverse clinical settings and provides a practical solution to bridge the healthcare gap in resource-limited areas.

Results

Overall framework

The MMDX framework is presented in Figure 1. The precision clinical diagnosis system utilized CT data from The Cancer Genome Atlas (TCGA), The National Cancer Institute’s Clinical Proteomic Tumor Analysis Consortium (CPTAC), and other data sources, totaling 14,864 3D CT scans and 1,637,738 CT images from 7,629 patients. The DECT generation network and MMDX network were trained and evaluated using both phantom and clinical patient datasets. The phantom dataset consists of 1,000 pairs of DECT images and corresponding material labels,³⁸ while the patient dataset contains 5,753 pairs of clinical DECT images.³⁹ A summary of the CT datasets and key characteristics used can be found in Table S1.

Figure 1.

Overview of the MMDX framework

The workflow of MMDX includes three steps. First, we construct a large-scale clinical CT dataset, and the DECT images are obtained by DECT generation network. Then, DECT images are utilized to train the MMD network to generate MDIs. At last, the precision clinical diagnosis system validates the effectiveness of MDIs on diagnosis, prognosis, and biomarker expression prediction tasks, and provides decision interpretation of MMDX. SECT, single-energy CT; DECT, dual-energy CT; MDI, material-decomposition image; μ, attenuation coefficient; l, propagation distance; α, proportionality coefficient.

In the workflow of MMDX, SECT images are first processed by a pretrained DECT generation network to obtain DECT images, which are subsequently used for the MMDX network to produce material decomposition images (MDIs). The MDIs containing material-composition information provide auxiliary support for various downstream tasks. The MMDX network, driven by the photoelectric effect and Compton scattering of the dual-energy X-ray imaging process, enables spontaneous learning of imaging parameters and enhances the feature representation capability. The training process of the network integrates pseudo-label supervision and material reconstruction constraints. Specifically, the pseudo-labels are obtained using the direct inversion method²⁶ and provide supervision in the early stage of training, enabling the network to approximate the true material composition. Furthermore, the decomposition results and the linear attenuation coefficients of different materials are used to reconstruct DECT images. The discrepancy between the reconstructed and original DECT images guides the fine-tuning process, thereby optimizing the approximate solution. We evaluate the performance of MMDX in 20 downstream tasks, including three diagnosis tasks, two prognosis tasks, and 15 biomarker prediction tasks. MMDX significantly improves the diagnostic performance of SECT and presents high explainability for clinical decision-making. More details about the network architecture and training process are provided in the methods section.

Precision material decomposition on the phantom dataset

The performance of MMDX was first evaluated on the phantom dataset, which consists of 1,000 pairs of DECT images with four types of materials: adipose, fibroglandular, calcification, and air. Meanwhile, we compared MMDX with four existing material-decomposition methods (direct inversion,²⁶ BM3D,^40,41 PWLS-TNV-L0,³⁰ and NoL-MBMI³⁴) as well as two deep-learning methods (EMCAD⁴² and Med-SA⁴³). All the deep-learning methods were trained in a supervised manner, with quantitative evaluation performed using peak signal-to-noise ratio (PSNR)⁴⁴ and structural similarity index (SSIM).⁴⁵

The phantom experiment results indicate that MMDX significantly outperformed the other methods in terms of PSNR and SSIM. With decomposition results demonstrating the closest visual similarity to ground-truth labels, MMDX exhibited superior performance in noise suppression and structural detail reconstruction (Figure 2A). Notably, while calcification appeared as isolated pixels in the images, many methods failed to decompose them properly or introduced artifacts in their results. This may lead to missed diagnoses of critical lesions in clinical practice. In contrast, MMDX achieved precise decomposition results, demonstrating exceptional capability in identifying structural details (Figure 2B). Meanwhile, although all the methods produced visually similar results for air, MMDX achieved remarkable quantitative metrics of 59.93 dB PSNR and 0.999 SSIM, almost identical to the ground truth. Overall, the proposed method achieved an average PSNR of 43.56 dB and an SSIM of 0.995 across all four materials, representing significant improvements of 12.92 dB in PSNR and 0.053 in SSIM compared to the state-of-the-art method (Figures 2C and 2D). These results demonstrate the outstanding ability of MMDX in precise material identification and decomposition, which is critical for clinical diagnosis and greatly enhances its clinical usability.

Figure 2.

Visual and quantitative comparisons on the phantom dataset

(A) Input DECT images, corresponding material-decomposition results of MMDX and six comparison methods, and ground-truth images. The decomposition results range between 0 and 1, and different basis materials are represented using different colors.

(B) Zoomed-in view of calcification images.

(C and D) Comparison of PSNR and SSIM. Higher values indicate better performance.

Precision material decomposition on the clinical patient dataset

To further validate the effectiveness of MMDX on real-world clinical CT data, we conducted experiments on the clinical patient dataset, which contains 5,753 pairs of real-world DECT images. We additionally included EINet⁴⁶ and AirNet⁴⁷ as comparison methods compared to the phantom experiments. For quantitative evaluation, we selected regions of interest (ROIs)³⁹ with known dominant components. The mean value and standard deviation (SD) were calculated to provide assessments of decomposition accuracy and noise levels, respectively. Volume fraction accuracy (VFA)³⁴ was further employed to evaluate the overall performance across all ROIs.

The patient experiment results demonstrate the outstanding performance of MMDX on real-world CT data, with all four MDIs exhibiting excellent visual quality (Figure 3A). Precision clinical diagnosis requires extremely high decomposition accuracy, and even minor errors may lead to clinical misdiagnosis. While most comparison methods suffered from noise-induced artifacts that significantly degraded image quality, MMDX effectively preserved detailed structures and edge information while achieving remarkable noise reduction. Although PWLS-TNV-L0 and NoL-MBMI showed partial noise suppression capability, it was difficult for them to quantify materials with similar attenuation coefficients in real-world CT images, such as muscle and fat, resulting in erroneous voids in the decomposition results. Overall, MMDX achieved the highest mean values and lowest SDs across all ROIs, with a VFA of 99.70%, representing an improvement of at least 3.94% over the comparison methods (Figures 3B–3D). To further validate the fidelity and physical plausibility of the decomposition results, we conducted DECT image-reconstruction experiments. Specifically, reconstructed DECT images were generated based on the output MDIs and the corresponding attenuation coefficients and then compared with the input DECT images. As shown in Figure S3 and Table S2, MMDX achieved the best reconstruction performance, with an average SSIM of 0.919 and a PSNR of 32.03 dB. Moreover, the DECT images reconstructed by MMDX exhibited high similarity to the ground-truth DECT images, further confirming the superior decomposition capability of the proposed method. These findings underscore the potential value of MMDX in clinical applications.

Figure 3.

Visual and quantitative comparisons on the patient dataset

(A) Input DECT images, corresponding material-decomposition results of MMDX, and six comparison methods. The decomposition results range between 0 and 1, and different basis materials are represented using different colors.

(B‒D) Comparison of mean (higher values indicate better performance), SD (lower values indicate better performance), and VFA (higher values indicate better performance).

Achieving significant improvement in disease diagnosis

We first evaluated the performance of MMDX in various disease diagnosis tasks, including epidemic pneumonia diagnosis, malignant-nodule screening, and lung cancer subtype classification. In clinical practice, the rapid identification of high-risk pneumonia patients is challenging due to the diversity of symptoms and disease progression. Timely and accurate diagnosis is crucial for implementing early intervention strategies to prevent complications such as respiratory failure. Similarly, as a common pulmonary condition, pulmonary nodules have a high probability of evolving into malignant tumors. Timely diagnosis of pulmonary nodules makes early detection of lung cancer possible. Furthermore, non-small cell lung cancer (NSCLC), which accounts for approximately 80% of all lung cancers,⁴⁸ encompasses various subtypes such as lung squamous cell carcinoma (LUSC) and lung adenocarcinoma (LUAD). Accurate discrimination between these subtypes is critical for improving treatment outcomes. We utilized the Clean-CC-CCII dataset⁴⁹ for epidemic pneumonia diagnosis, the LUNA22-ISMI dataset⁵⁰ for malignant-nodule screening, and the Lung-PET-CT-Dx and NSCLC-Radiomics datasets⁵¹ for lung cancer subtype classification. Four popular 3D classification networks (ResNet50, DenseNet121, ResNeXt50,⁵² and Swin Transformer⁵³) were used as diagnostic networks to verify the effectiveness of material-composition information.

As presented in Figure 4A, MDIs achieved significant improvements over SECT images across all disease diagnosis tasks. Specifically, for the task of epidemic pneumonia diagnosis, MDIs achieved a macro-averaged area under the receiver operating characteristic curve (AUROC) of 0.995 (95% confidence interval [CI]: 0.992–0.997) with ResNet50, surpassing SECT images of 0.973 (95% CI: 0.965–0.980). Similarly, MDIs with DenseNet121 exhibited an AUROC of 0.997 (95% CI: 0.995–0.999), outperforming SECT images of 0.961 (95% CI: 0.951–0.969). For ResNeXt50, MDIs showed marked improvements, attaining an AUROC of 0.993 (95% CI: 0.989–0.996) compared to SECT images of 0.981 (95% CI: 0.974–0.987). With Swin Transformer, MDIs achieved an AUROC of 0.989 (95% CI: 0.983–0.993), exceeding SECT images of 0.977 (95% CI: 0.970–0.984). For malignant-nodule screening, MDIs demonstrated significantly improved diagnostic performance across four networks, achieving superior AUROCs of 0.955 (95% CI: 0.933–0.973) versus 0.918 (95% CI: 0.888–0.946), 0.979 (95% CI: 0.968–0.990) versus 0.906 (95% CI: 0.872–0.935), 0.938 (95% CI: 0.912–0.960) versus 0.877 (95% CI: 0.838–0.916), and 0.927 (95% CI: 0.895–0.953) versus 0.899 (95% CI: 0.866–0.927), respectively. For the task of lung cancer subtype classification, which is relatively challenging, MDIs demonstrated superior performance by achieving AUROCs of 0.869 (95% CI: 0.794–0.936), 0.848 (95% CI: 0.765–0.917), 0.848 (95% CI: 0.763–0.918), and 0.905 (95% CI: 0.839–0.957), representing notable improvements of more than 0.088, 0.111, 0.045, and 0.056, respectively, compared to SECT images.

Figure 4.

Disease diagnosis and prognosis performance of MMDX

(A) Comparisons of the receiver operating characteristic (ROC) curves and AUROCs with 95% confidence intervals between SECT images and MDIs for three diagnosis tasks: epidemic pneumonia diagnosis, malignant-nodule screening, and lung cancer subtype classification. Four diagnostic networks (ResNet50, DenseNet121, ResNeXt50, and Swin Transformer) were used in each task. Error bars indicate the 95% CIs, and statistical comparisons were conducted using the DeLong test (∗p < 0.05, ∗∗p < 0.01, ∗∗∗p < 0.001).

(B) Performance comparisons of two prognosis tasks: LARC treatment response prediction and 5-year survival status prediction.

(C and D) Average performance of the SECT images and MDIs in the diagnosis tasks and the prognosis tasks.

For the diagnosis tasks under study, MMDX brought an average improvement of 4.8%, 6.8%, 9%, and 3.5% with respect to AUROC, accuracy, sensitivity, and specificity (Figure 4C). The results demonstrate that MMDX can significantly enhance the ability of classification networks across various diagnosis tasks. Moreover, MMDX exhibits potential for improving the performance of more deep-learning diagnostic tools, providing valuable insights for clinical treatment plans and offering a pathway for more accurate and accessible healthcare solutions.

Improving predictive performance for prognosis tasks

To further evaluate the clinical usability of MMDX for prognosis tasks, we conducted experiments on two independent datasets. The LARC dataset⁵⁴ was used to predict locally advanced rectal cancer (LARC)⁵⁵ treatment response, and the task of 5-year survival status prediction was conducted on the TCGA-kidney renal clear cell carcinoma (KIRC) dataset.⁵¹

Total mesorectal excision (TME) combined with preoperative neoadjuvant chemoradiotherapy (nCRT) is the current important treatment for LARC. Patients achieving pathological complete response (pCR) after preoperative nCRT can avoid surgery and associated complications. Thus, it is crucial to predict the response to standard treatment. As shown in Figure 4B, the AUROCs of the diagnostic networks with MDIs and SECT images were 0.837 (95% CI: 0.660–0.985) versus 0.719 (95% CI: 0.512–0.903), 0.830 (95% CI: 0.602–0.992) versus 0.745 (95% CI: 0.533–0.917), 0.882 (95% CI: 0.727–0.983) versus 0.758 (95% CI: 0.528–0.940), and 0.810 (95% CI: 0.591–0.981) versus 0.739 (95% CI: 0.525–0.935) (all p < 0.05). Furthermore, the 5-year survival rate refers to the proportion of cancer patients who survive for at least 5 years after being diagnosed with a specific type of cancer. In clinical practice, this metric is commonly used as a key indicator for evaluating treatment efficacy and prognosis. Compared to SECT images, MDIs achieved significantly higher AUROCs of 0.800 (95% CI: 0.620–0.964), 0.836 (95% CI: 0.655–0.979), 0.887 (95% CI: 0.743–0.989), and 0.851 (95% CI: 0.687–0.978), with improvements of 0.221, 0.169, 0.092, and 0.051, respectively.

MMDX demonstrated average improvements of 11.7%, 11.8%, 10.3%, and 15.9% in terms of AUROC, accuracy, sensitivity, and specificity, respectively (Figure 4D). The experimental results highlight the excellent clinical benefits of MMDX, demonstrating its great potential in helping patients with low rectal cancer preserve the anal sphincter and improve their overall quality of life. Furthermore, MMDX exhibits superior performance in predicting 5-year survival status, making it a valuable tool for clinical treatment evaluation and decision-making.

Enhancing the performance of biomarker expression prediction

Biomarkers play a crucial role in the diagnosis, prognosis, and treatment outcome prediction for various cancer types.⁵⁶ We further explored the generalization ability of MMDX by conducting experiments on biomarkers including microsatellite instability (MSI), programmed cell death ligand 1 (PD-L1), and tumor mutational burden (TMB), which have significantly advanced cancer therapy in the past decade.^57,58,59 The ability of MMDX to classify gene expression was evaluated on five cancers, and the datasets utilized in this task comprise bladder urothelial carcinoma (BLCA), KIRC, liver hepatocellular carcinoma (LIHC), LUAD, and ovarian serous cystadenocarcinoma (OV).

As shown in Figure 5A, MDIs outperformed SECT images across all cancer types, achieving average AUROC improvements of 0.07 for BLCA, 0.094 for KIRC, 0.073 for LIHC, 0.081 for LUAD, and 0.092 for OV. Specifically, for the prediction of MSI, MDIs achieved AUROCs of 0.741 (95% CI: 0.625–0.850), 0.901 (95% CI: 0.825–0.958), 0.800 (95% CI: 0.697–0.895), 0.753 (95% CI: 0.630–0.865), and 0.860 (95% CI: 0.753–0.952), significantly outperforming the results of SECT images, which achieved AUROCs of 0.675 (95% CI: 0.551–0.803), 0.834 (95% CI: 0.743–0.914), 0.719 (95% CI: 0.592–0.835), 0.670 (95% CI: 0.539–0.793), and 0.812 (95% CI: 0.670–0.924). For predicting PD-L1 expression, MDIs demonstrated significantly superior performance compared to SECT images. Especially for LUAD, MDIs achieved an accuracy of 0.825 (95% CI: 0.719–0.912) and an AUROC of 0.910 (95% CI: 0.826–0.970), outperforming SECT images with an accuracy of 0.772 and an AUROC of 0.835. Furthermore, MDIs demonstrated consistent superiority over SECT images in the task of predicting TMB expression, with AUROCs of 0.798 (95% CI: 0.668–0.913), 0.859 (95% CI: 0.776–0.927), 0.751 (95% CI: 0.617–0.858), 0.841 (95% CI: 0.729–0.930), and 0.866 (95% CI: 0.712–1.000) (Figures 5B‒5G). Based on these results, we observed that material-composition information significantly contributes to the prediction of biomarker expression while also demonstrating strong generalizability across multiple cancer types. The superior performance of MMDX holds broad potential for clinical application, thereby facilitating cancer treatment.

Figure 5.

Performance of biomarker expression prediction

(A) Average performance of five cancer types on three biomarkers.

(B‒F) Comparisons of the ROC curves between SECT images and MDIs for three biomarkers across five cancer types on the TCGA-BLCA, TCGA-KIRC, TCGA-LIHC, TCGA-LUAD, and TCGA-OV datasets.

(G) Comparisons of AUROCs with 95% confidence intervals. Error bars indicate the 95% CIs, and statistical comparisons were conducted using the DeLong test (∗p < 0.05, ∗∗p < 0.01, ∗∗∗p < 0.001).

Outperforming foundation models across clinical applications

CT-based foundation models represent a cutting-edge advancement in AI-driven healthcare, revolutionizing medical imaging. By leveraging deep-learning techniques, these models excel at analyzing complex patterns within CT scans, enabling more accurate diagnosis of various diseases and offering profound insights for clinical decision-making. However, the training process of foundation models necessitates extensive datasets and substantial computational resources, posing significant challenges for their deployment in resource-limited areas where cost-effective diagnostic tools are urgently needed. We further compared MMDX with two recent CT-based foundation models, CT-CLIP⁶⁰ and Merlin.⁶¹ The foundation models offer two application modes: linear-probing (training only a linear classifier atop the pretrained backbone) and fine-tuning (training the entire network). We conducted a detailed comparison with both modes to comprehensively evaluate the superior performance of MMDX. Quantitative results are provided in Figures 6A–6F. For linear-probing, MMDX demonstrated significant improvements across all tasks. The average accuracy and AUROC of MMDX were 0.836 and 0.886, while the results for CT-CLIP and Merlin were 0.664 and 0.729, and 0.710 and 0.756, respectively. Notably, in the task of predicting biomarker expression, the performance of foundation models was similar to random classification, indicating the limited generalization capabilities on specific rare tasks. As for fine-tuning, MMDX showed significant performance improvements in the prognosis and biomarker expression prediction tasks. The average prediction accuracy for the task of LARC treatment response, 5-year survival status, and biomarker expression prediction increased by at least 7.7%, 13.3%, and 5.1%, respectively. Moreover, comparable or improved diagnostic performance was observed in the diagnosis tasks. These results indicate that MMDX, through material-composition characterization, exhibits superior generalization across diverse CT-based tasks while achieving performance that surpasses foundation models. Notably, MMDX achieves superior performance at a significantly lower computational cost, utilizing only one-fifth the number of parameters of foundation models, demonstrating its potential for deployment in resource-limited areas. The related hyperparameter settings and parameter counts are listed in Table S3.

Figure 6.

Clinical usability of MMDX

(A‒F) ROC curves of epidemic pneumonia diagnosis, malignant-nodule screening, lung cancer subtype classification, LARC treatment response prediction, 5-year survival status prediction, and biomarker expression prediction (PD-L1 prediction for TCGA-KIRC dataset). The foundation models were trained using linear-probing or fine-tuning mode.

(G‒I) Comparisons between MMDX and radiologists with different levels of expertise on epidemic pneumonia diagnosis, malignant-nodule screening, and lung cancer subtype classification. For each task, the macro-average ROC curve of the diagnostic networks is provided.

(J) ROC curve of different input modalities and training strategies for NTM diagnosis.

(K) Radar charts arranged from left to right indicate the average AUROC, accuracy, sensitivity, and specificity of MMDX on the external set.

Outperforming radiologists with enhanced clinical usability

To evaluate the potential of MMDX in clinical applications, we compared its disease diagnosis performance with four radiologists, including two junior radiologists with less than 5 years of experience and two senior radiologists with over 10 years of experience. Considering that the prediction of PD-L1 expression, rectal cancer treatment response, and survival status based on SECT images is nearly infeasible in clinical practice, we selected the other three common clinical diagnosis tasks (diagnosis of pneumonia, pulmonary nodules, and lung cancer). All radiologists were required to independently complete the tasks based on SECT images. Quantitative results are provided in Figures 6G–6I. MMDX outperformed the average performance of radiologists by 14.0% in sensitivity and 7.2% in specificity for epidemic pneumonia diagnosis, by a significant margin of 11.7% in sensitivity and 8.1% in specificity for malignant-nodule screening, and by 9.3% in sensitivity and 6.0% in specificity for lung cancer subtype classification. Overall, MMDX achieved an average diagnostic accuracy improvement of 10.5% (ranging from 7.5% to 14.1%). Notably, disease diagnostic networks using SECT images demonstrated performance comparable to that of the radiologists, surpassing junior radiologists and remaining inferior to senior radiologists. These results highlight the superior clinical applicability of MMDX, with its performance exceeding that of senior radiologists, suggesting broad potential for clinical diagnostic applications.

Validating generalizability in non-tuberculous mycobacterial pulmonary disease diagnosis

Non-tuberculous mycobacteria (NTMs) comprise a diverse group of mycobacterial species that are widely present in environments such as water and soil. As a relatively rare disease, NTM pulmonary disease is difficult to diagnose due to non-specific clinical manifestations.^62,63,64 We further evaluated the diagnostic performance of MMDX on NTMs using the mycobacterial dataset.⁶⁴ Consistent with the epidemic pneumonia diagnosis task, adipose, air, muscle, and fibrous tissue were selected as the basis materials. To assess the adaptability of MMDX across diverse clinical scenarios, we conducted a comparative analysis of different input modalities and training strategies, including SECT images, zero-shot inference (directly applying the MMDX model from the pneumonia diagnosis task), fine-tuning, and retraining (training from scratch).

As shown in Figure 6J, all three MDI-based strategies achieved superior diagnostic performance compared with SECT images. In the zero-shot setting, MDIs achieved an accuracy of 0.758 (95% CI: 0.708–0.812) and an AUROC of 0.777 (95% CI: 0.711–0.841), outperforming SECT images with an accuracy of 0.723 (95% CI: 0.669–0.773) and an AUROC of 0.748 (95% CI: 0.675–0.809). The results demonstrate that MMDX can be effectively generalized to the less common NTM disease, highlighting the strong generalizability in pulmonary disease diagnosis. Due to differences in data distribution and modality, retraining achieved better performance than both zero-shot and fine-tuning strategies, with an accuracy of 0.788 (95% CI: 0.735–0.838) and an AUROC of 0.812 (95% CI: 0.749–0.871). Nevertheless, the improvements observed with fine-tuning and retraining indicate that MMDX can effectively adapt to task-specific data, demonstrating its strong potential across diverse clinical scenarios. More detailed quantitative results can be found in Table S4.

Validating diagnostic performance on eight unseen external datasets

We further implemented external validation on an independent external test set, which includes 1,393 patients from eight unseen datasets. Detailed information about the datasets is provided in the methods section. As shown in Figure 6K, MDIs still exhibited superior overall performance compared to SECT images. For all downstream tasks, MDIs brought an average improvement of 10.3%, 11.8%, 7.5%, and 17.3% with respect to AUROC, accuracy, sensitivity, and specificity. We observed that, when using SECT images as input, the diagnostic networks performed similarly to a random classifier in some tasks, especially for the prediction of biomarker expression and survival status. This decrease in performance could be attributed to several factors, such as differences between CT datasets and the inherent complexity of the tasks. However, networks with MDIs still demonstrated satisfactory performance by extracting material-composition information closely related to the disease from SECT images. These results indicate that MMDX exhibits both robustness and generalization capability. More detailed results can be found in Tables S5, S6, S7, S8, S9, and S10.

Model interpretation for clinical decision-making

We utilized gradient-weighted class activation mapping (Grad-CAM)⁶⁵ and t-distributed stochastic neighbor embedding (t-SNE)⁶⁶ methods to analyze how MMDX improves the performance of SECT data. The results demonstrate that MDIs generated by MMDX provide inherent interpretability in highlighting lesions and diagnostic network feature embeddings. Grad-CAM generates heatmaps that highlight regions significant to the model decision in the input images (Figures 7A‒7F). For the epidemic pneumonia diagnosis task, diagnostic networks using MDIs as input show greater focus on consolidations, ground-glass opacities, and other specific lesion areas, which are important for distinguishing between different types of pneumonia.^67,68 For the malignant-nodule screening task, the attention maps are primarily concentrated on the center and surrounding areas of the nodule, which are more critical for classifying benign and malignant nodules. Furthermore, MDIs demonstrate stronger attention to the lung tissue in the cancer subtype classification task, while SECT images suffer interference from surrounding air regions, resulting in more dispersed attention. In the tasks of predicting survival status and biomarker expression for kidney cancer, MDIs exhibit significantly higher attention to the kidney region compared to other areas. Enhanced attention to the regions of solid masses and necrosis leads to more reliable results of diagnostic networks. In addition, the results of MDIs for rectal cancer treatment response prediction focus more on the rectal regions, capturing both local structural features and global information and assisting clinical decision-making. These results indicate the ability of material-composition information to assist diagnostic networks in extracting specific features.

Figure 7.

Decision interpretation of MMDX

(A‒F) Attention maps of networks with different inputs for epidemic pneumonia diagnosis, malignant-nodule screening, lung cancer subtype classification, LARC treatment response prediction, 5-year survival status prediction, and biomarker expression prediction. The attention maps generated using the Grad-CAM method indicate the significance of each area in the images, where red color represents an area more significant to the deep-learning model decision.

(G) Feature spaces of networks with different inputs across the diagnosis, prognosis and biomarker expression prediction tasks. The left feature spaces represent networks with SECT images, and the right feature spaces represent networks with MDIs.

Additionally, t-SNE was used to embed the high-dimensional deep features of the diagnostic networks into a two-dimensional plane (Figure 7G), enabling more intuitive visualization of the feature distributions from SECT images and MDIs. The results indicate that networks with MDIs achieve better intra-class aggregation, where data points from the same class are more tightly grouped together, while also enhancing inter-class separation and producing more distinct feature representations across different classes. Furthermore, we evaluated clustering performance using separation, silhouette coefficient, and Calinski-Harabasz index (higher values indicate better performance), as well as compactness and Davies-Bouldin index (lower values indicate better performance). As provided in Table S11, MDIs achieve overall superior performance compared with SECT images across all downstream tasks, indicating that MDIs significantly improve feature extraction in clinical applications, thereby enhancing diagnostic reliability.

Discussion

Over the past decades, numerous studies have indicated that there is a close relationship between the material composition and various diseases, providing avenues for enhancing the effectiveness of SECT in clinical applications. However, multi-material differentiation and quantification is a severely ill-posed problem, particularly in SECT images. In this work, we propose the MMDX framework for precisely quantifying material fraction from clinical CT images and generating corresponding MDIs. Furthermore, we established a precision clinical diagnosis system using large-scale clinical CT data, which generates DECT images and specified MDIs and assists deep-learning models in performing multiple downstream tasks. Comprehensive experimental results indicate that MMDX significantly improves diagnostic performance while mitigating the reliance on expensive medical equipment, providing an economical and effective AI-assisted diagnostic tool for clinical applications.

Challenges such as the diversity and complexity of the material composition, the lack of ground-truth labels, and the limited availability of DECT data have restricted the development of material-decomposition research. Conventional analytical methods rely on mathematical optimization and multiple regularization constraints to address the ill-posed decomposition problem. Although these methods can achieve acceptable performance, they are often associated with high computational complexity, pronounced sensitivity to initialization, and limited ability to preserve fine structural details. Existing deep-learning methods have achieved improvements in structural preservation and noise suppression. However, these methods primarily depend on image feature extraction and fail to account for the critical role of X-ray imaging principles in material decomposition. Meanwhile, the restricted range of decomposable materials further limits their applicability across diverse clinical scenarios. The introduction of MMDX effectively addresses these challenges with a deep-learning network architecture induced by the photoelectric effect and Compton scattering. By spontaneously learning X-ray imaging parameters during the network training process, MMDX obtains enhanced representations of DECT. Additionally, the proposed class-aware attention decoder enables the network to focus more on the category information of the pre-selected basis materials, thereby better representing the intricate relationships among different materials. Experimental results validated the effectiveness of the proposed modules, and MMDX significantly outperformed previous methods. Quantitative experiments on the phantom dataset showed that MMDX improved PSNR and SSIM by at least 12.92 dB and 0.053, respectively. On the clinical patient dataset, MMDX achieved a VFA of 99.70%, surpassing previous methods by at least 3.94%. The material-decomposition results of MMDX exhibit high visual quality and show significant advantages in maintaining structural details and noise suppression.

We constructed a large-scale clinical CT dataset consisting of 1,637,738 CT images from 7,629 patients for downstream tasks. The performance of the precision clinical diagnosis system was evaluated using three different diagnosis tasks (epidemic pneumonia diagnosis, malignant-nodule screening, and lung cancer subtype classification), two prognosis tasks (LARC treatment response prediction and 5-year survival status prediction) and 15 biomarker prediction tasks. The MDIs generated by MMDX showed significant advantages over SECT, with the average AUROC of each task type increased by 4.8%, 11.7%, and 8.5%, respectively. In addition, the external validation demonstrated an average accuracy improvement of 11.8%. Notably, MMDX also demonstrated robust performance in small-sample scenarios, with consistent performance improvements observed in tasks with limited training data and external validation cohorts containing only a few cases. The generalizability of MMDX was further evaluated in the NTM diagnosis task, where the model demonstrated strong diagnostic performance, highlighting its applicability across diverse clinical scenarios. Unlike task-specific models that are tightly coupled to training datasets, MMDX incorporates the universal principles of X-ray imaging into the architecture, enabling consistent modeling across different CT imaging systems. Moreover, the architecture is decoupled from the selection of basis materials, allowing flexible adaptation to different clinical requirements without modifying the network structure, and facilitating the learning of a transferable representation space for material decomposition across various diseases and anatomical regions. The results of t-SNE also revealed that MDIs had better class separability in the feature space of deep-learning models compared to the original SECT images. By precisely quantifying the composition of different materials, MDIs can extract more disease-related information from the original CT images, capturing the intricate relationships between diseases and material-composition changes. Furthermore, the proposed framework can be integrated into existing clinical workflows without additional radiation exposure. After routine SECT acquisition, MMDX can automatically generate MDIs to assist CT interpretation. Given its efficient inference performance, MMDX has the potential to be deployed within hospital picture archiving and communication systems (PACS), which would be particularly beneficial for institutions where DECT scanners are not readily available. By combining the expertise of radiologists, MMDX can improve healthcare outcomes and serve as an effective diagnostic assistance tool for regions with limited medical resources.

Despite the extensive evaluations of MMDX across various clinical tasks, certain limitations still remain in this study. First, this work primarily focuses on materials with significantly different attenuation characteristics. However, human tissue types are highly diverse and complex, and precise quantification of similar materials may be critical for the diagnosis of specific disease subtypes. Future studies will explore more decomposable material types, particularly tissues with subtle attenuation differences (e.g., gray matter and white matter in the brain). Second, as a retrospective study, our research mainly focuses on thoracic and abdominal diseases. Although a large-scale CT dataset was used in the clinical application experiments, further prospective studies are still required to fully validate the clinical effectiveness of MMDX. In particular, additional investigations on cross-device and cross-center data are necessary to systematically assess the adaptability of the proposed framework under variations in scanner types, imaging protocols, and clinical environments. Moreover, although MMDX has demonstrated promising performance across multiple diseases and clinical tasks, variations in patient characteristics, such as disease stage, pathological subtype, and treatment status, may influence the performance of material decomposition and downstream diagnostic tasks. Future work will focus on analyses of specific clinical subgroups to further improve performance consistency across heterogeneous patient populations.

In conclusion, our study reveals the feasibility of precisely quantifying material composition from SECT images using deep learning. MMDX achieves superior performance across large-scale, multi-center downstream tasks, highlighting the clinical value of material-composition information beyond conventional anatomical imaging. The MDIs generated by MMDX provide distinct advantages over traditional SECT, significantly enhancing performance across a wide range of clinical applications and providing a practical AI-assisted tool to support clinical decision-making.

Methods

Datasets

This study was conducted on 20 CT datasets, including two DECT datasets and 18 SECT datasets, which were used for model development and downstream tasks, respectively. We utilized data from the TCGA project via the National Cancer Institute (NCI) Genomic Data Commons Portal,⁶⁹ CPTAC project via the Cancer Imaging Archive (TCIA),⁵¹ and other open data sources, which comprise a total of 1,637,738 CT images from 7,629 patients. The datasets of each task were divided into training set (70%), validation set (10%), and test set (20%). Specifically, the Clean-CC-CCII and MosMedData datasets were used for epidemic pneumonia diagnosis, which contain 2,725 and 1,110 patients, respectively. The mycobacterial dataset used for NTM diagnosis contains 1,301 patients. The LUNA22-ISMI and SPIE-AAPM Lung CT datasets used for malignant-nodule screening contain 1,010 and 70 patients, respectively. The classification of lung cancer subtypes was conducted on the Lung-PET-CT-Dx, NSCLC-Radiomics, TCGA-LUAD, and TCGA-LUSC datasets, which comprise 300, 203, 67, and 37 patients, respectively. The LARC internal and external datasets used for LARC treatment response prediction contain 135 and 15 patients, respectively. The datasets employed for the prediction of 5-year survival status were TCGA-KIRC, TCGA-KIRP, TCGA-KICH, and CPTAC-CCRCC, which consist of 237, 23, 12, and 59 patients, respectively. The datasets utilized in the biomarker prediction tasks comprise TCGA-BLCA (n = 107), TCGA-KIRC (n = 237), TCGA-LIHC (n = 75), TCGA-LUAD (n = 67), and TCGA-OV (n = 143).

For model development, the first DECT dataset is a breast phantom dataset containing 1,000 DECT image pairs,³⁸ and each image pair represents a distinct imaging instance with different anatomical and material distributions. Four types of materials (adipose, fibroglandular, calcification, and air) were provided, and the corresponding distribution maps were used as the ground-truth labels for material decomposition. During the imaging process, the X-ray source potential alternates between successive views. A total of 512 projection views are generated, with 256 views corresponding to each voltage. DECT images are then reconstructed by applying the filtered back-projection (FBP) algorithm to the negative log-transformed transmission data. We randomly divided the phantom dataset into training and test sets containing 800 and 200 image pairs, respectively. The second DECT dataset was collected at Nanjing General PLA Hospital (NGPH), China,³⁹ which was also used for the development of the DECT generation network. Iodine contrast-enhanced DECT image pairs were obtained using a SOMATOM Definition Flash DECT scanner at 100- and 140-kVp energy levels. The data received institutional review board approval, and informed consent was obtained from all participants prior to examinations. The dataset comprises 5,753 DECT image pairs, and the FBP algorithm was used to reconstruct all CT images. The training and test sets include 4,793 and 960 DECT image pairs, respectively. A summary of used CT datasets and key characteristics can be found in Table S1.

Data preprocessing

Different data-processing strategies were employed based on the specific datasets and tasks. For DECT datasets, all images were resized to 512 × 512 pixels before training. After generating MDIs, we used identical preprocessing methods to ensure a fair comparison of the diagnostic networks using SECT images and MDIs. In the epidemic-pneumonia diagnosis task, we applied an automated pulmonary parenchyma segmentation method on each slice. Both raw and segmented images were resized to 160 × 160 pixels, followed by center cropping to 128 × 128 pixels, and they were combined to produce multi-channel images. We randomly sampled 32 slices per scan, and the input size of diagnostic networks was 128 × 128 × 32. For datasets utilized in the malignant-nodule screening task, CT scans were first resampled based on the average diameter of the nodules, and a central region of 48 × 48 × 48 was extracted as input for the diagnostic networks. All the other datasets underwent identical preprocessing protocols. After excluding slices with reconstruction errors, images were resized to 128 × 128 × 48 for the downstream tasks. In addition, random rotation and cropping of the images were applied to enhance the generalizability of the precision clinical diagnosis system.

DECT generation network

The DECT generation network was based on VMamba.⁷⁰ Leveraging the long-range dependency modeling capabilities of VMamba, we constructed a U-Net-based network architecture to learn the intricate texture details and energy information differences in CT images. During the training process, the network takes SECT images as input, which are obtained by fusing half of the DECT images, and generates the corresponding low- and high-energy CT images. To better adapt VMamba for DECT generation, we designed the ConVSS block, which introduces an additional convolutional branch into the original visual state space (VSS) block. Specifically, given the input feature map x_l of a certain layer l, the ConVSS block is formulated as follows:

$$\begin{array}{c}\text{ConVSS}\left(x_l\right)=x_l+\text{Conv}\left(\text{VSS}\left(x_l\right)+\text{ReLU}\left(\text{Conv}\left(x_l\right)\right)\right)\end{array}\text{,}$$ (Equation 1)

where Conv(·), VSS(·), and ReLU(·) denote the 1 × 1 convolution, VSS block, and rectified linear unit, respectively. The ConVSS block combines the global modeling capability of VMamba with the local feature extraction strength of convolution operations, allowing the network to simultaneously capture global anatomical structures and subtle local texture details. In addition, we further improved the training strategy by designing a dual-constraint loss function that combines mean squared error (MSE) loss and fusion loss. Specifically, the MSE loss is used to directly supervise the generated low- and high-energy CT images against the corresponding ground-truth DECT images, ensuring accurate generation of energy-specific image details. More importantly, we introduce a fusion loss by averaging the generated DECT images to reconstruct the original SECT image and then calculate the MSE between the reconstructed and input SECT images. This reconstruction constraint ensures that the generated DECT images preserve the same anatomical structures as the original SECT input while maintaining consistency in attenuation representation. As shown in Figure S1 and Tables S12 and S13, the HU values of the generated DECT images are highly consistent with those of the ground truth, and the proposed DECT generation network achieves SSIM of 0.992 and 0.991 and PSNR of 48.831 dB and 48.928 dB for the two energy levels, respectively. In addition, the visual results further demonstrate that the generated DECT images exhibit high fidelity, with only minimal discrepancies compared to the ground truth. We further conducted material-decomposition experiments based on the generated and real DECT images. As shown in Table S14, the material-decomposition results obtained from the generated DECT images are highly consistent with those from the real DECT images across all material types, with negligible performance differences. These results indicate that the proposed method effectively preserves the underlying energy-dependent attenuation characteristics required for accurate material decomposition, thereby providing reliable support for downstream tasks.

Basis material selection and pseudo-label generation

The direct inversion method was used to generate pseudo-labels through the following operations. Considering a triplet from the pre-selected basis materials, the material-decomposition process of a single pixel can be formulated as follows:

$$\begin{array}{c}{\mu }_L\left(E_j\right)=\sum _i{\alpha }_i{\mu }_{L,i}\left(E_j\right),\left(i=1,2,3;j=L,H;0\le {\alpha }_i\le 1\right)\text{,}\end{array}$$ (Equation 2)

where μ_L(E_j) denotes the linear attenuation coefficient of the mixture at low and high energy levels, μ_L,i(E_j) denotes the linear attenuation coefficient of a specific basis material at energy levels E_L and E_H, and α_i is the corresponding volume fraction. After introducing the volume conservation assumption ($\sum _i{\alpha }_i=1$),³⁹ a set of α_i can be obtained for every triplet of the pre-selected basis materials based on Equation 2. The set of volume fractions with the sum of absolute values closest to 1 was chosen as the final solution. For the four selected basis materials, once the volume fractions of a given triplet were determined, the volume fraction of the remaining material outside the triplet was assigned a value of 0, thereby generating pseudo-labels for all four basis materials.

For the selection of basis materials, we followed the principles that the attenuation coefficients among the chosen materials should be as different as possible, and the disease-relevant materials are given priority. In the material-decomposition experiments, two DECT datasets were utilized. The breast phantom dataset consists of four types of materials (adipose, fibroglandular, calcification, and air) along with their corresponding ground-truth labels. Therefore, these four materials were directly chosen as basis materials. The clinical patient dataset comprises abdominal iodine contrast-enhanced DECT image pairs. Considering that the images in this dataset are predominantly composed of adipose, muscle, iodine, and air and do not include specific tissues such as fibrous tissue, we followed prior work^34,39 and selected these four materials as basis materials.

To obtain more stable decomposition results in clinical applications, we primarily selected materials that are most commonly observed in CT images and exhibit significant discrepancies in attenuation coefficients⁷¹ (air, −1,000 HU; bone, 1,000 HU; adipose, −190 to −30 HU; and muscle, 30 to 150 HU⁷²). These four basis materials were adopted in tasks including lung cancer subtype classification, LARC treatment response prediction, 5-year survival status prediction, and biomarker expression prediction. Additionally, the materials related to the specific disease were also considered. For example, fibrous tissue was selected as a basis material in the epidemic pneumonia diagnosis task, as lesion regions typically exhibit a higher volume fraction of fibrous tissue. To further investigate the impact of material selection on the performance of diagnostic networks, we conducted task-specific material-selection experiments for epidemic pneumonia diagnosis. As provided in Table S15, both removing fibrous tissue and replacing fibrous tissue with bone in the input MDIs lead to performance degradation, demonstrating the rationality and effectiveness of selecting disease-relevant basis materials. Furthermore, while the combination of SECT images and MDIs achieved performance improvements, MDIs alone consistently achieved the best results, indicating that material-decomposition images provide more discriminative and disease-relevant representations. The proposed MMDX framework enables flexible selection of basis material types and quantities, thereby achieving superior performance across extensive clinical applications.

MMDX network

Overall architecture

The proposed MMDX network is designed to resolve the nonlinear multi-solution material-decomposition problem by leveraging the physical process of X-ray imaging and deep learning. We adopt TransNeXt⁷³ as the backbone of the MMDX network, and both the encoder and decoder modules are hierarchical structures with four layers. TransNeXt is a visual backbone network that incorporates aggregated attention and convolutional gated linear unit, and it has demonstrated superior performance in image feature extraction. As shown in Figure 8, the dual-energy encoder takes DECT images as input, and the outputs are defined as dual-energy X-ray features. These features are high-dimensional latent representations that not only preserve anatomical structures but also encode energy-dependent attenuation information of tissues under different X-ray energy levels. After being processed by the feature aggregation module, these features are fed into the class-wise decoder via skip connection. There are two key components in this process: X-ray feature aggregation module (XFAM) and class-wise vision transformer (CWViT). The XFAM takes dual-energy X-ray features as inputs and aggregates them after learning the parameters of photoelectric effect and Compton scattering during X-ray imaging automatically, enabling the network to better capture the underlying attenuation information and improve the accuracy and interpretability of material-decomposition results. As the basis of the class-wise decoder, CWViT introduces class tokens in the ViT attention mechanism, perceiving the differences between material categories effectively and generating more precise decomposition results. In the following sections, we provide a detailed description of each component.

Figure 8.

Detailed structure of the MMDX network

(A) The MMDX network with dual-energy encoder and class-wise decoder.

(B) Class-wise vision transformer (CWViT) block.

(C) X-ray feature aggregation module (XFAM).

(D) X-ray imaging guidance block (XIGB).

(E) Multi-scale depth-wise convolution block (MDCB).

XFAM

The proposed XFAM is designed to model the physical attenuation characteristics of dual-energy X-ray imaging. During the feature extraction, XFAM aggregates features from one of the four layers of the dual-energy encoder by incorporating physics-induced attenuation modeling and multi-scale representation learning. As shown in Figure 8C, XFAM consists of two components: an X-ray imaging guidance block (XIGB) and a multi-scale depth-wise convolution block (MDCB). Considering a layer in the dual-energy encoder, the feature aggregation process is formulated as

$$\begin{array}{c}{F}_L^{\prime },F_H^{\prime }=\text{XIGB}\left(F_L,F_H\right)\text{,}\end{array}$$ (Equation 3)

$$\begin{array}{c}{F}_{\text{XFAM}}=\text{MDCB}\left(\text{Cat}\left(F_L+F_L^{\prime },F_H+F_H^{\prime }\right)\right)\text{,}\end{array}$$ (Equation 4)

where F_L and F_H are the output dual-energy X-ray features of the encoder, F_XFAM is the output of the XFAM, and Cat(·) indicates the concatenation operation.

According to the Lambert-Beer law, the attenuation of a single-energy X-ray as it passes through a homogeneous material is expressed by the following formula:

$$\begin{array}{c}F=F_0{e}^{-\mu l}\text{,}\end{array}$$ (Equation 5)

where μ denotes the attenuation coefficient, l denotes the propagation distance, and F₀ denotes the initial energy feature. In the diagnostic X-ray imaging energy range, the total attenuation coefficient μ is considered to be the combination of two primary mechanisms: photoelectric effect and Compton scattering. The photoelectric effect occurs when an incident X-ray photon is completely absorbed by an atom. This process is highly dependent on the atomic number of the material and the photon energy, making it particularly sensitive to variations in tissue composition. Compton scattering involves the inelastic collision between an X-ray photon and an electron, resulting in partial energy transfer. These two mechanisms together determine the energy-dependent attenuation characteristics of materials in CT imaging. Specifically, the attenuation of dual-energy X-rays is formulated as

$$\begin{array}{c}{F}_L=F_{OL}{e}^{-\left({\mu }_{LP}+{\mu }_{LC}\right)l}\text{,}\end{array}$$ (Equation 6)

$$\begin{array}{c}{F}_H=F_{OH}{e}^{-\left({\mu }_{HP}+{\mu }_{HC}\right)l}\text{,}\end{array}$$ (Equation 7)

where μ_ij(i = L,H;j = P,C) denote the attenuation contributions of photoelectric effect and Compton scattering at low and high energy levels. The relationship between F_L and F_H is further formulated as

$$\begin{array}{c}{F}_H=\alpha e^{\left({\mu }_{LP}+{\mu }_{LC}-{\mu }_{HP}-{\mu }_{HC}\right)l}{F}_L\text{,}\end{array}$$ (Equation 8)

$$\begin{array}{c}{F}_L=\frac{1}{\alpha }{e}^{\left({\mu }_{HP}+{\mu }_{HC}-{\mu }_{LP}-{\mu }_{LC}\right)l}{F}_H\text{,}\end{array}$$ (Equation 9)

where $\alpha =\frac{F_{OH}}{F_{OL}}$ denotes the proportionality coefficient. Considering the intricate relationship between the initial features at different energy levels and the pixel-wise heterogeneity of α, we designed a three-layer convolutional block, which is formulated as

$$\begin{array}{c}\alpha =\text{Sig}\left(\mathrm{LN}\left(\text{Conv}\left(\text{Cat}\left(F_L,F_H\right)\right)\right)\right)\text{,}\end{array}$$ (Equation 10)

where Sig(·), LN(·), and Conv(·) indicate the sigmoid function, layer normalization, and convolution, respectively. The above operations are repeated three times to model α at the pixel level. The remaining five parameters, μ_LP, μ_LC, μ_HP, μ_HC, and l, represent the attenuation coefficients under different energy levels and the propagation distance. Unlike fixed physical constants directly measured in CT imaging, these parameters follow the relationship between the dual-energy encoder outputs F_L and F_H, and are adaptively optimized during the network training process. This formulation enables data-driven parameter learning while being implicitly constrained by physical principles, thereby facilitating the modeling of energy-dependent attenuation characteristics. After obtaining all the X-ray imaging parameters, the outputs of the XIGB are calculated and defined as $F_L^{\prime }$ and $F_H^{\prime }$ from Equations 8 and 9. As energy features derived from the X-ray imaging process, $F_L^{\prime }$ and $F_H^{\prime }$ integrate pixel-wise attenuation parameters to approximate the underlying physical interactions between X-ray photons and biological tissues and provide enhanced energy information for the initial features via residual connections.

The compositional heterogeneity and the non-uniform attenuation characteristics of materials lead to spatial variations in X-ray energy features. To address this, we introduce MDCB to effectively aggregate the high- and low-energy features. Given the concatenated input feature $x=\text{Cat}\left(F_L+F_L^{\prime },F_H+F_H^{\prime }\right)$, the formulation of MDCB can be further elaborated as follows:

$$\begin{array}{c}\text{DC}\left(x\right)=\sum _{k\in K}\left(\text{ReLU}\left(\text{BN}\left(\text{DWConv}\left(x\right)\right)\right)\right)\text{,}\end{array}$$ (Equation 11)

$$\begin{array}{c}\text{MDCB}\left(x\right)=\text{Conv}\left(x\right)+\text{BN}\left(\text{Conv}\left(\text{CS}\left(\text{DC}\left(x\right)\right)\right)\right)\text{,}\end{array}$$ (Equation 12)

where DWConv(·), BN(·), ReLU(·), and CS(·) denote depth-wise convolution, batch normalization, rectified linear unit, and channel shuffle operation. The parameter k indicates the kernel size of depth-wise convolution, and we set K = [1, 3, 5]. The output of MDCB integrates multi-scale feature representations at different energy levels and is subsequently used as the input to the material-aware decoder.

CWViT

CWViT is designed to extract class-aware energy features. The attention mechanism of CWViT is adapted from aggregated attention,⁷³ which combines query-centered sliding-window attention and pooling attention. Considering a sliding window centered on the target pixel in the input feature map, pixels within the sliding window are defined as local pixels, and pixels obtained by pooling the input feature map are defined as global pixels. The similarity matrices S_l and S_g are calculated as follows:

$$\begin{array}{c}{S}_l=Q_{cws}{K}_l^T,S_g=Q_{cws}{K}_g^T\text{,}\end{array}$$ (Equation 13)

where Q_cws denotes the class-wise query of the target pixel in calculating similarity matrices, and $K_l^T$ and $K_g^T$ denote the transpose of local key and global key. Given a set of query embeddings (qe), the calculation of Q_cws is formulated as

$$\begin{array}{c}{Q}_{cws}=\text{Linear}\left(\text{Softmax}\left(\text{Cat}\left(Q+{qe}_1,Q+{qe}_2,\cdots ,{Q+qe}_n\right)\right)\right)\text{,}\end{array}$$ (Equation 14)

where Q is the original query of the target pixel and n is the number of selected basis materials. The introduction of Q_cws enables the network to focus more on the class information of basis materials during the decoding process. Then the weight matrices W_l and W_g are calculated as follows:

$$\begin{array}{c}{W}_l,W_g=\text{Split}\left(\text{Softmax}\left(\lambda \left(\text{Cat}\left(S_l,S_g\right)+B\right)\right)\right)\text{,}\end{array}$$ (Equation 15)

where λ and B denote the length-scaled cosine attention parameter and the position bias in aggregation attention. Finally, a set of learnable tokens (T) are also introduced in the calculation of class-wise attention (CWA), which is formulated as

$$\begin{array}{c}\text{CWA}=\left(W_l+Q_{cwa}\right)V_l+W_g{V}_g\text{,}\end{array}$$ (Equation 16)

where Q_cwa denotes the class-wise query in the calculation of final attention, and V_l and V_g denote the local value and the global value. Similarly, Q_cwa is calculated as

$$\begin{array}{c}{Q}_{cwa}=\text{Linear}\left(\text{Softmax}\left(\text{Cat}\left(Q{T}_1,Q{T}_2,\cdots ,Q{T}_n\right)\right)\right)\text{.}\end{array}$$ (Equation 17)

By leveraging Q_cwa, the network simultaneously enhances class discriminability of basis materials and learns more implicit relative positional information for the target query.

Loss functions

In the clinical patient experiment, we proposed a loss function (L_patient) incorporating pseudo-label supervision and material reconstruction constraints. Specifically, the L_patient is defined as follows:

$$\begin{array}{c}{L}_{reconstruction}=\frac{1}{N}\sum _i{\Vert \mu Y_i-X_i\Vert }^2\text{,}\end{array}$$ (Equation 18)

$$\begin{array}{c}{L}_{boundary}=\frac{1}{N}\sum _i\text{Softmax}\left(\left|\nabla \left(\mu Y_i\right)\right|\right)\left|\mu Y_i-X_i\right|\text{,}\end{array}$$ (Equation 19)

$$\begin{array}{c}{L}_{patient}=\gamma L_{pseudo}+\left(1-\gamma \right)L_{reconstruction}+\beta L_{boundary}\text{,}\end{array}$$ (Equation 20)

where N=H×W is the total number of pixels, $X\in {\mathbb{R}}^{2\times H\times W}$ denotes input DECT images, $Y\in {\mathbb{R}}^{n\times H\times W}$ denotes output MDIs and n is the number of selected basis materials, $\mu \in {\mathbb{R}}^{2\times n}$ denotes the attenuation coefficients of selected basis materials at high and low energy levels, and ∇(·) denotes the Laplace operator for calculating the second-order gradient. L_patient is composed of three components, where the pseudo-label loss L_pseudo represents the mean-squared error between outputs and pseudo-labels, the reconstruction loss L_{reconstruction} constrains the difference between reconstructed DECT images and input DECT images, and the boundary loss L_boundary maintains structural consistency between images by using weighted gradients. During the training process, the network was dominated by pseudo-label loss at first and transitioned to reconstruction loss gradually. The time-dependent attenuation function γ is defined as γ(t) = exp⁡(-5t/t_max), and β is the weight of boundary loss, which is set to 0.01.

In the phantom experiment, the distribution maps of materials were used as true labels. Therefore, the network was trained in a supervised manner. The loss function L_phantom is defined as

$$\begin{array}{c}{L}_{phantom}=L_{mse}+\beta L_{boundary}\text{,}\end{array}$$ (Equation 21)

where L_mse denotes the MSE between outputs and true labels.

Ablation studies

To assess the contribution of each component, we conducted ablation studies by sequentially removing XFAM, CWViT, and the boundary loss from the MMDX network and evaluating the resulting material-decomposition performance. Without individual modules, the network tended to ignore fine structural details, such as isolated material pixels and small calcification regions, and produced fragmented or blurred boundaries. In contrast, the complete MMDX model preserved these subtle structures more faithfully, confirming its superior capability in capturing microscale material characteristics. Quantitative results indicated that XFAM contributed most significantly to the performance, underscoring the importance of imaging physical principles for MMDX. Detailed results are provided in Figure S2 and Table S16.

The MDIs generated by MMDX have demonstrated superior performance across extensive downstream tasks. Nevertheless, in the material-decomposition experiments, several comparison methods also generated high-quality images, indicating their potential for clinical applications. To further evaluate the performance of different material-decomposition methods in downstream tasks, we conducted additional experiments on the epidemic pneumonia diagnosis task. Considering that the material-decomposition methods based on mathematical optimization, such as BM3D and PWLS-TNV-L0, typically require several minutes to process a pair of DECT images and are therefore difficult to apply in clinical practice, we compared MMDX with the deep-learning-based methods, including EINet, AirNet, and NoL-MBMI. As shown in Table S17, all the material-decomposition methods consistently outperformed SECT images, further underscoring the clinical value of material-composition information. The average AUROCs of the three comparison methods and MMDX were 0.982, 0.982, 0.991, and 0.994, respectively. These results indicate that MMDX not only achieves state-of-the-art performance in material decomposition but also delivers superior results in downstream tasks, demonstrating its potential as an AI tool for precision clinical diagnosis.

Implementation details

The models were developed and implemented using PyTorch⁷⁴ 2.6.0 on a platform with two NVIDIA RTX 3090 24GB GPUs. We constructed the DECT generation network based on VMamba⁷⁰ blocks, with both the encoder and decoder designed as four-stage hierarchical structures. The network consists of [2, 2, 9, 2] blocks across the four stages, with corresponding feature channel dimensions of [96, 192, 384, 768], while all other parameters remain at their default settings. The network was trained for 100 epochs with a batch size of four.

We adopted TransNeXt⁷³ as the backbone of the MMDX network. TransNeXt provides multiple backbone variants (micro, tiny, small, and base), all of which adopt a four-stage hierarchical architecture. Larger backbones contain more TransNeXt blocks per stage and higher-dimensional feature channels. To systematically determine the optimal parameters, we conducted ablation studies on both the backbone variants and the attention-window sizes. The results indicate that increasing the model capacity does not necessarily lead to performance improvement, while the attention window-size has a significant impact on decomposition accuracy. Considering both computational efficiency and model performance, we selected TransNeXt-Tiny with a window size of 3. Specifically, TransNeXt-Tiny consists of [2, 2, 15, 2] blocks across the four stages, with corresponding feature channel dimensions of [72, 144, 288, 576], while all other parameters remain at their default settings. Detailed results of the parameter ablation study are provided in Table S18. The direct inversion method was used to generate pseudo-labels for the clinical patient dataset. Based on the pseudo-labels of the selected basis materials, BM3D and PWLS-TNV-L0 further improve the results through mathematical optimization and denoising procedures to achieve material decomposition. The remaining deep-learning methods, as well as the proposed MMDX, were trained in a self-supervised manner. For the phantom dataset, all deep-learning methods were trained in a fully supervised manner using the ground-truth labels. The models were trained for 100 epochs with a batch size of two.

In the clinical application experiments, ResNet50, DenseNet121, ResNeXt50, and Swin Transformer were employed as diagnostic networks for the diagnosis and prognosis tasks, while DenseNet121 was used for biomarker expression prediction due to its stable performance. For each network, SECT images and MDIs were used as inputs, respectively, while other experimental settings, including dataset splitting, optimizer configuration, learning rate schedule, batch size, and number of training epochs, were kept identical. In addition, to ensure reproducibility, the hyperparameter settings of all diagnostic networks were consistent with those reported in their original publications. All the networks were trained for 200 epochs with a batch size of four. The early-stopping strategy was employed to mitigate overfitting, which terminated network training when the validation AUROC no longer increased for 20 epochs. We used the AdamW optimizer⁷⁵ with an initial learning rate of 1 × 10⁻³, and the strategy of cosine annealing was applied to schedule the learning rate with a minimum value of 1 × 10⁻⁵.

Inference efficiency

We evaluated the inference performance of MMDX across different hardware platforms to assess the deployment feasibility in clinical applications. As summarized in Table S19, the inference latency and frames per second (FPS) were measured at input resolutions of 256 × 256 and 512 × 512 pixels. On an NVIDIA GeForce RTX 4090 GPU, MMDX achieved inference speeds of 17.3 and 11.4 FPS at the two respective resolutions, demonstrating efficient real-time processing capability. In addition, MMDX achieved inference speeds of 12.5 and 8.7 FPS on an NVIDIA GeForce RTX 3090 GPU, and 4.9 and 1.1 FPS on a lower-power NVIDIA GeForce GTX 1650 Ti GPU, respectively. Furthermore, when deployed on a CPU-only platform, MMDX remained operational, achieving 0.4 FPS at a resolution of 512 × 512. Notably, compared with the several minutes typically required for CT acquisition, MMDX processes a CT volume within a substantially shorter time. Overall, the proposed framework demonstrates a favorable balance between diagnostic performance and inference efficiency, supporting its applicability in real-world clinical scenarios and resource-limited areas.

Statistical analysis

To assess the performance of MMDX, AUROC, sensitivity, specificity, and accuracy were utilized to provide comprehensive evaluations. All results were reported with 95% CIs computed by 1,000 bootstrap replicates. In addition, the DeLong test was employed to compare AUROCs between the diagnostic models with different inputs. The significance comparisons of sensitivity, specificity, and accuracy were conducted using permutation tests to calculate two-sided p values with 10,000 permutations. A threshold of p < 0.05 was considered statistically significant. All analyses were implemented with Python 3.10.

Resource availability

Lead contact

Requests for further information and resources should be directed to and will be fulfilled by the lead contact, Xiangzhi Bai (jackybxz@buaa.edu.cn).

Materials availability

This study did not generate new unique reagents.

Data and code availability

• •All original code has been deposited at https://github.com/Jack-Mars6/MMDX and is publicly available as of the date of publication.
• •The model weights and source code have been deposited at Zenodo (DOI: https://doi.org/10.5281/zenodo.21380025)⁷⁶ and are publicly available as of the date of publication.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under grant 62271016, and in part by the Beijing Natural Science Foundation under grant L242130.

Author contributions

Conceptualization, X.B., M.L., and J.L.; methodology, X.B., M.L., and J.L.; investigation, X.B., M.L., J.L., and Q.L.; writing – original draft, X.B. and M.L.; writing – review and editing, X.B., M.L., and Q.L.; funding acquisition, X.B.; resources, X.B.; supervision, X.B.

Declaration of interests

The authors declare no competing interests.

Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work, the authors used ChatGPT for language polishing. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

Published: July 28, 2026