Artificial intelligence-based multimodal model for the identification of ulcerative colitis with concomitant cytomegalovirus colitis

Abstract

Background:

Ulcerative colitis (UC), a chronic immune-mediated colon inflammation, impacts patients’ quality of life. Immunosuppressive-treated UC patients are prone to opportunistic infections like cytomegalovirus (CMV) infection, which exacerbates UC, causes steroid resistance, and elevates surgery and mortality risks. Identifying CMV colitis from UC exacerbation is difficult due to overlapping symptoms and low biopsy detection rates.

Objectives:

To develop an artificial intelligence (AI)-based multimodal model for early identification of UC with concomitant CMV colitis.

Design:

This was a retrospective diagnostic study.

Methods:

A total of 174 moderate to severe UC patients (87 with CMV colitis) from 2015 to 2023 in Peking Union Medical College Hospital were enrolled retrospectively. A total of 3345 colonoscopy images were collected. The dataset was split into training (70%) and testing (30%) sets. A multimodal dynamic affine transformation (DAFT) model integrating clinical biomarkers and endoscopic images was constructed, along with ResNet and SeNet models. Model performance was evaluated using accuracy, sensitivity, specificity, positive and negative predictive values from the confusion matrix.

Results:

UC patients with CMV colitis had distinct clinical characteristics. The multimodal DAFT model outperformed ResNet and SeNet in distinguishing UC with CMV colitis, with higher accuracy (0.91), sensitivity (0.87), and specificity (0.93).

Conclusion:

AI application offers a promising way to enhance early identification of UC with CMV colitis. The multimodal model combining clinical and endoscopic data can assist clinicians in accurate and timely diagnosis.

Plain language summary

Artificial intelligence based multimodal model for the identification of ulcerative colitis with concomitant cytomegalovirus colitis

Background: Ulcerative colitis (UC), a chronic immune-mediated colon inflammation, impacts patients’ quality of life. Immunosuppressive-treated UC patients are prone to opportunistic infections like cytomegalovirus (CMV) infection, which exacerbates UC, causes steroid resistance, and elevates surgery and mortality risks. Identifying CMV colitis from UC exacerbation is difficult due to overlapping symptoms and low biopsy detection rates.

Aim: To develop an AI-based multimodal model for early identification of UC with concomitant CMV colitis.

Methods: 174 moderate to severe UC patients (87 with CMV colitis) from 2015 - 2023 in Peking Union Medical College Hospital were enrolled retrospectively. 3345 colonoscopy images were collected. The dataset was split into training (70%) and testing (30%) sets. A multimodal DAFT model integrating clinical biomarkers and endoscopic images was constructed, along with Resnet and Senet models. Model performance was evaluated using accuracy, sensitivity, specificity, positive and negative predictive values from the confusion matrix.

Results: UC patients with CMV colitis had distinct clinical characteristics. The multimodal DAFT model outperformed Resnet and Senet in distinguishing UC with CMV colitis, with higher accuracy (0.91), sensitivity (0.87), and specificity (0.93).

Conclusion: AI application offers a promising way to enhance early identification of UC with CMV colitis. The multimodal model combining clinical and endoscopic data can assist clinicians in accurate and timely diagnosis.

Background

Ulcerative colitis (UC) is a chronic immune-mediated disease characterized by inflammation of the colon that can have a prolonged and unremitting course, severely impacting patients’ quality of life. Data from the National Urban Employee Basic Medical Insurance Program in China showed that the incidence of UC increased from 8.1/100,000 in 2013 to 16.6/100,000 in 2016. ¹ The rapid increase in the number of UC cases highlights the importance of prevention and active treatment. Patients with UC who receive treatment with steroids, immunosuppressants, and biologics have a 3–5 times higher risk of opportunistic infections. ² Of these infections, cytomegalovirus (CMV) infection in UC patients requires particular attention. Research has shown that CMV infection can induce disease activity in UC patients, cause steroid resistance, and increase surgery and mortality rates in patients with CMV colitis and UC. ³ Early recognition and treatment of CMV colitis are necessary to improve steroid resistance and reduce the risk of colectomy in UC patients.^4,5 However, acute UC flare and CMV colitis share very similar clinical presentations, including hematochezia, diarrhea, and fever, which increases the difficulty of identification. Currently, the diagnosis of CMV colitis relies on pathological examination, including searching for viral inclusions in colonic mucosal tissue using HE staining, immunohistochemical (IHC) staining, and CMV-DNA in situ hybridization. ⁶ HE staining has a high specificity, ranging from 92% to 100%, but its sensitivity is in a wide fluctuation from 10% to 87%, ⁷ and the diagnosis is observer-dependent. IHC staining increases the diagnostic sensitivity (78%–93%) and specificity (92%–100%) ⁸ but to some extent, it is time-consuming and also depends on the location and number of the biopsy. ⁹ Some characteristic endoscopic features like punched-out ulcers, irregular ulceration, and cobblestone-like appearance may imply CMV colitis. ¹⁰ A study investigating physicians’ opinions on the endoscopic features of CMV colitis with UC patients revealed that punched-out ulcerations were the most agreed-upon feature that indicated CMV colitis. ¹¹ Targeting these endoscopic features of CMV colitis when biopsy is benefit to increase the sensitivity of pathological examination. Identifying these features under endoscopy requires repeated training and enriched endoscopic experience, and inexperienced endoscopists may overlook these features. Therefore, a diagnostic tool is needed to provide clues for tissue biopsy that help better identify CMV colitis earlier.

In recent years, artificial intelligence (AI) technology has been widely utilized in the field of gastroenterology, greatly improving the clinical diagnosis efficiency of gastrointestinal diseases. In the field of endoscopy, AI has been used to identify early-stage tumors and depth of tumor invasion in the upper gastrointestinal tract, identify and classify colon polyps, and predict Helicobacter pylori infection. ¹² AI also has enormous potential in the field of inflammatory bowel disease (IBD). Cai et al. applied traditional machine learning algorithms, including random forests (RFs), support vector machines (SVMs), and eXtreme gradient boosting (XGBoost), to train to predict disease activity in IBD patients. ¹³ Compared to traditional machine learning, deep learning exhibits a powerful ability to learn the essential features of a dataset from a small sample set, with a higher accuracy for classification or prediction. Guimaraes et al. applied convolutional neural networks (CNNs), one of the representative algorithms of deep learning, based on endoscopic images and clinical data to differentiate inflammatory bowel disease, infectious colitis, and ischemic colitis. ¹⁴ However, there is little research on deep learning models for differentiation of UC and CMV colitis.

The purpose of this study is to establish a multimodal model trained with clinical biomarkers and endoscopic images, based on a deep learning algorithm for early recognition and diagnosis of CMV colitis.

Method

Dataset description

This study consecutively included 87 patients with moderate to severe active UC and 87 patients with moderate to severe UC combined with CMV colitis who underwent colonoscopy at Peking Union Medical College Hospital from July 2015 to June 2023. UC diagnosis was made based on the “Chinese clinical practice guideline on the management of UC (2023, Xi’an) ¹⁵ ” and “European Crohn’s and Colitis Organization (ECCO) Consensus on Diagnosis and Management of Ulcerative Colitis.” ¹⁶ Moderate to severe activity was defined as a modified Mayo score of ⩾6. ¹⁷ CMV colitis diagnosis was based on pathological examination: finding CMV inclusion bodies in colonic mucosal HE staining, positive CMV Immunohistochemical staining, or positive CMV-DNA in situ hybridization. A total of 87 patients with matched age and modified Mayo score who had moderate to severe UC without CMV colitis were also collected as a control group. Colonoscopy images and the following laboratory tests were collected for each patient: white blood cell count, neutrophil count, hemoglobin, eosinophil count, lymphocyte count, platelet count, erythrocyte sedimentation rate (ESR), and immunoglobulin A (IgA).

A total of 3345 colonoscopy images were included and classified into three categories by two experienced endoscopists: CMV colitis (CMV, n = 1115), UC without CMV colitis (UC, n = 1115), and normal (N, n = 1115). When developing and evaluating the model, all the images were randomly divided into two subsets: training set (70% of the sample size) and testing set (30% of the sample size). The definitions of the endoscopic classification are detailed in Figure 1. The flow chart of the study is shown in Figure 2.

Figure 1.

The endoscopic classification of each category.

UC + CMV, UC concomitant with CMV colitis.

CMV, cytomegalovirus; UC, ulcerative colitis.

Figure 2.

Study flow diagram.

Data extraction

Data extraction was performed by removing interfering information such as patient age, gender, and examination time from colonoscopy images. The specific steps involved findcontours function in OpenCV ¹⁸ to identify the contour of the colon, and obtain the coordinates of the rectangular box using the x, y coordinates of the point set. The region corresponding to the rectangular box with four vertices (x_min, y_min), (x_min, y_max), (x_max, y_min), and (x_max, y_max) was taken as the image used for diagnosis.

Model architecture

The model architecture consisted of a main network utilizing the structure of residual networks (ResNet): a 7 × 7 convolutional layer, batch normalization, ReLU activation function, and max pooling layer were used to extract shallow features of the image, followed by four layers to extract deep features. Each layer consisted of two Basic Blocks, of which the structure is shown on the right side of Figure 3. The feature maps from layer4 and layer3 were merged after being downsampled by residual structure, then global pooling, a fully connected layer, and a softmax function were applied to obtain the final classification results.

Figure 3.

Deep learning architecture.

The feature fusion module is shown in the purple area on the left side of Figure 2. First, AdaptiveAvgPool2d was used to produce the channel-level feature vector D(d1,d2. . .dc) of the feature map B generated by layer3, where c was the number of channels. Then, the feature vector A corresponding to the medical index was concatenated with D and fused by a fully connected layer to obtain feature vectors E(e1,e2. . .ec) and F(f1,f2. . .fc), where E was the bias vector and F was the channel weight vector. Finally, the feature map G = B × F + E was used as the final fused feature.

The dynamic affine transformation (DAFT) module (G = B × F + E) provides the foundation for model decision interpretability: clinical biomarkers generate scaling factors F and offsets E through fully connected layers, directly modulating the channel-wise expression of image feature maps B. This mechanism enables key clinical features, such as low eosinophil count, to quantitatively influence the model’s interpretation of imaging manifestations, allowing clinicians to trace how biomarkers assist in endoscopic diagnosis.

To validate the independent contribution of clinical biomarkers to classification, this study employed a progressive ablation strategy to train three models. ResNet served as the baseline, processing only endoscopic images without attention mechanisms or clinical data to evaluate the performance of pure imaging features. The Squeeze-and-Excitation Networks (SeNet) model embedded an Squeeze-and-Excitation (SE) channel attention module into the ResNet architecture to enhance perception of critical imaging regions while excluding clinical biomarkers to test the effect of attention-based feature optimization. Building upon SeNet, DAFTNet integrated eight clinical biomarkers—white blood cell, neutrophil, eosinophil, lymphocyte, and platelet counts; hemoglobin, ESR, and IgA—with attention-optimized imaging features through the DAFT module, aiming to quantify the incremental value of clinical biomarkers in multimodal diagnosis.

Model training

Training was performed using a GeForce 3060 graphics card with 12 GB of memory. The cross-entropy cost function was chosen as the loss function to supervise the optimization direction of the model. Adam optimizer was used with a fixed learning rate of 1e-4, and gradient clipping was applied to prevent shaking caused by excessive gradient during training. A total of 2340 samples were collected as training data and trained for 200 epochs with a batch size of 10. To avoid overfitting, random horizontal and vertical flips and rotations were applied to increase the diversity of the dataset during training, and the best-performing model parameters were selected as the final result after convergence of the network.

Outcome definition

In this work, we investigated three major outcomes, namely (1) distinguishing healthy samples from pathological samples (healthy), (2) distinguishing UC from healthy and CMV colitis (UC), and (3) distinguishing CMV colitis from UC and healthy samples (CMV).

Performance evaluation metrics

The metrics used for assessing the model performance in test stages are accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), which are computed from the values of the confusion matrix, true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). The reporting of this study conforms to the Standards for Reporting Diagnostic Accuracy (STARD) statement (Supplemental Material). ¹⁹

Results

Table 1 presents the clinical characteristics of UC patients and UC patients with concomitant CMV colitis. Compared with UC patients without CMV colitis, those with CMV colitis were characterized by elderly age, more severe anemia, lower eosinophil, and platelet count.

Table 1.

Clinical characteristics.

Characteristics	UC (n = 87)	UC + CMV (n = 87)	p Value
Age (years)	39.2 ± 14.0	48.6 ± 14.0	0.000
Gender (male/female, %)	49/38 (56%/44%)	56/31 (64%/36%)	0.278
White blood cell count (×109/L)	7.8 ± 2.8	7.9 ± 3.9	0.958
Neutrophil count (×109/L)	5.3 ± 2.4	5.4 ± 3.2	0.727
Hemoglobin (g/L)	109.3 ± 24.4	101.2 ± 21.5	0.023
Eosinophil count (×109/L)	0.2 ± 0.2	0.1 ± 0.2	0.001
Lymphocyte count (×109/L)	1.8 ± 0.7	1.8 ± 1.0	0.851
Platelet count (×109/L)	364.6 ± 127.5	317.9 ± 137.2	0.022
Erythrocyte sedimentation rate (mm/h)	35.45 ± 28.2	36.35 ± 26.7	0.834
Immunoglobulin A (g/L)	2.3 ± 0.9	2.8 ± 5.5	0.523

UC + CMV: UC with concomitant CMV colitis.

CMV, cytomegalovirus; UC, ulcerative colitis.

Table 2 reports the overall results in terms of accuracy, sensitivity, specificity, PPV, and NPV of the three models. Figures 4–6 show the performance of the model for all classification tasks. The SeNet model added an attention module SE Block on the basis of the classic ResNet model, and as Table 2 shows, the performance of the model for three tasks was generally improved. Compared to the ResNet and SeNet models, the DAFT Net ²⁰ led to superior classification performance.

Table 2.

Metrics values for the trained models.

Task	Data type	Model	Accuracy	Sensitivity	Specificity	PPV	NPV
healthy	Image-only	ResNet	0.837	0.803	0.854	0.733	0.897
		SeNet	0.851	0.818	0.867	0.755	0.905
	Image + table	DAFTNet	0.865	0.788	0.903	0.802	0.895
UC	Image-only	ResNet	0.741	0.588	0.818	0.618	0.799
		SeNet	0.761	0.57	0.857	0.666	0.799
	Image + table	DAFTNet	0.825	0.743	0.866	0.735	0.871
UC + CMV	Image-only	ResNet	0.833	0.725	0.887	0.762	0.866
		SeNet	0.845	0.797	0.869	0.752	0.895
	Image + table	DAFTNet	0.91	0.869	0.931	0.864	0.934

CMV, cytomegalovirus; NPV, negative predictive value; PPV, positive predictive value; UC, ulcerative colitis.

Figure 4.

Healthy pathological classification results: distinguish healthy samples from pathological samples.

Figure 5.

UC classification results: distinguish UC from healthy and CMV colitis.

CMV, cytomegalovirus; UC, ulcerative colitis.

Figure 6.

UC + CMV classification results: distinguish CMV colitis from healthy and UC.

CMV, cytomegalovirus; UC, ulcerative colitis.

Ablation experiments demonstrated that clinical biomarkers were core determinants of model performance improvement. In the task of distinguishing UC from CMV colitis, DAFTNet achieved a 6.5% increase in accuracy and 7.1% increase in specificity compared to SeNet. This improvement confirms that biomarkers such as low platelet count provide independent diagnostic signals unrevealed by endoscopic images alone, playing a critical role in reducing false-positive rates.

To further interpret how DAFTNet utilizes clinical biomarkers, we quantified the importance of each feature in the dynamic affine transformation. As shown in Figure 7, platelet count consistently demonstrated the highest importance across all diagnostic classes (Normal: 0.264, UC: 0.218, UC + CMV: 0.228), with statistically significant differences between Normal and inflammatory groups (p < 1e-10). Hemoglobin showed higher importance in Normal diagnosis (0.098) compared to UC (0.081) and UC + CMV (0.072) (p < 1e-8), suggesting its role in ruling out anemia-related mucosal changes. ESR importance was significantly elevated in UC + CMV (0.030) versus other classes (p < 0.01), reflecting systemic inflammation in CMV infection. The reduced importance of eosinophil count in UC + CMV (p < 0.05) aligns with the known phenomenon of eosinopenia during viral infections.

Figure 7.

Clinical feature importance by diagnosis class.

Specifically, we can see from Table 2 and Figure 6, the DAFTNet model exhibited better performance in distinguishing UC combined with CMV colitis from UC and healthy samples, with a high accuracy of 0.91. The overall sensitivity of the model was relatively low; however, the sensitivity to identify CMV colitis reaches as high as 0.869.

The ROC analysis showed that the area under the curve of the three classification tasks was 0.90, 0.86, and 0.95, respectively (Figure 8), which implied a considerably excellent performance of our model in terms of predicting UC and CMV colitis.

Figure 8.

Performance curves.

Receiving operating characteristics (ROC) for the testing set.

Discussion

In clinical practice, the occurrence of concomitant CMV colitis in patients with UC is relatively common, which has an impact on the treatment and prognosis of UC. However, due to the similar clinical manifestations between the two, it is often challenging for clinicians to differentiate CMV colitis from UC flares. In this study, we have successfully developed three deep learning-based diagnostic models (ResNet, SeNet, and DAFTNet) for the identification of UC and CMV colitis. Among them, the DAFTNet model demonstrates better performance in distinguishing UC combined with CMV colitis from UC and healthy samples, with an accuracy of up to 0.91, and the sensitivity for identifying CMV colitis reaches 0.869. In addition, through comparison, our multimodal model (DAFT) outperforms the image-based ResNet and SeNet unimodal models with attention modules in all tasks and metrics, indicating that the multimodal model that combines clinical biomarkers and endoscopic image information can more effectively assist clinicians in making accurate diagnoses.

The application of AI in various tasks is becoming increasingly widespread in IBD. ²¹ Kraszewski et al. established a machine learning prediction model based on blood, urine, and fecal markers with 97% and 91% mean average precision for CD and UC, respectively. ²² Gui et al. developed a CNN-based model to detect neutrophils in biopsy slide images of UC patients and classify images into either a histologic remission or non-remission. ²³ Chen et al. utilized multi-omics data to develop a diagnostic model for CMV reactivation in UC, identifying eight immune-related biomarkers via machine learning. ²⁴ In this study, we have successfully developed three innovative deep learning diagnostic models focusing on real-time endoscopic guidance to identify UC and CMV colitis.

In our deep learning model, the recognition of images is based on the endoscopic features of CMV colitis. Yang et al. discovered that characteristic endoscopic features such as punch-out ulcerations, irregular ulcerations, and a cobblestone-like appearance may help predict the presence of CMV colitis. ¹⁰ Furthermore, we found that UC patients with CMV infection had a tendency to exhibit elderly age, more severe anemia, eosinopenia, and thrombocytosis. Previous studies showed that decreased peripheral blood absolute lymphocyte count, eosinophil count, IgA level, and higher platelet to lymphocyte ratio may serve as a predictive marker for CMV infection,^25–27 which is consistent with our study. Therefore, clinical biomarkers may help predict CMV infection.

Our image-based ResNet and SeNet models performed similarly with previously established image-based CNNs in identifying UC ¹⁴ but they exhibited better performance in identifying CMV colitis. We acknowledge Kim et al.’s ²⁸ valuable contribution to CMV detection using deep learning, where their ensemble approach achieved a sensitivity of 0.91 and an accuracy of 0.83. However, it is important to note that their best-performing single model achieved an accuracy of 0.792, comparable to our unimodal imaging models (ResNet: 0.833, SeNet: 0.845). This comparison underscores that ensemble techniques, while effective for performance gains, were essential for Kim et al.’s approach to outperform standalone models. By contrast, our multimodal model, DAFTNet, achieves competitive accuracy (0.91), specificity (0.931), and balanced sensitivity (0.869) by integrating multimodal data: endoscopic features (3345 images from 174 patients) and clinical biomarkers. This innovation addresses critical limitations of Kim et al.’s study, which relied solely on endoscopic features with a small sample size (n = 34, 86 images). Our DAFT module dynamically fuses these complementary data sources, enabling the model to leverage clinical biomarkers to refine image-based predictions. The 6.5% accuracy improvement of DAFTNet over SeNet confirms that our multimodal integration strategy—rather than model ensembling—drives performance gains, providing a more scalable and interpretable solution for clinical translation. We acknowledge that low sensitivity and PPV for the UC and CMV colitis groups are the main limitations of these two models. The multimodal model (DAFT) addressed this deficiency. Multimodal involves the collaborative analysis of various types of data, such as images, text, etc., to help AI more accurately complete classification and recognition tasks. The DAFT model presents several innovative aspects. ²⁹ First, it enables a two-way exchange of information between image and tabular data by dynamically scaling and shifting the feature maps in a convolutional layer based on both the patient’s endoscopic images and clinical tabular information. This allows the CNN to leverage the complementary information from tabular data for improved prediction. Second, DAFT creates a bottleneck structure to compress the spatial dimensions of the image feature map through global average pooling, and then concatenates it with the tabular data. Through a series of fully connected layers and nonlinear transformations, it obtains the scaling and offset vectors for feature map transformation, which can dynamically adjust the feature maps according to the specific image and tabular information to better fuse the two types of data. In addition, DAFT is a versatile module that can be easily applied to any type of CNN for tabular information fusion, showing wide applicability in different application scenarios. Extensive experiments, including comparisons with multiple baseline methods, ablation studies, analysis of the contribution of tabular data features, and generalization and repeatability experiments on different datasets, demonstrate the effectiveness and superiority of DAFT in fusing image and tabular data, as well as its robustness to design choices. ²⁹ In this study, our multimodal model (DAFT) integrates information from two modalities: clinical biomarkers and endoscopic images. The clinical biomarkers include those mentioned above that help predict cytomegalovirus infection: white blood cell count, lymphocyte count, eosinophil count, platelet count, and IgA level. The two modalities reflect complementary aspects of UC and CMV colitis. Obviously, our results demonstrate that the multimodal model, taking into account both image data and clinical biomarker data, outperforms the unimodal, image-based ResNet and SeNet with attention module across all tasks and metrics, which is consistent with a prior work utilizing a similar multimodal model. ²⁰ Although a previous study has considered combining image and clinical parameters as a hybrid algorithm to improve the CNN’s poor sensitivity for IBD and low specificity for infectious colitis, ¹⁴ our multimodal model (DAFT) is superior in distinguishing CMV colitis and UC, with higher accuracy and ROC curve area.

The model’s primary goal is to guide endoscopists in identifying suspicious CMV colitis lesions during live examinations, ensuring no high-risk areas are missed. In this context, sensitivity—defined as the model’s ability to detect true-positive cases—directly addresses the clinical imperative to minimize underdiagnosis. A higher sensitivity reduces the likelihood of overlooking CMV-infected tissues, which could lead to delayed antiviral therapy. While specificity (0.931) remains important to limit unnecessary biopsies, the trade-off favors sensitivity in this pre-biopsy screening role, where the consequences of missed detection far outweigh the risks of additional sampling.

In the future, it is expected to develop a software system for identifying CMV colitis in UC patients based on existing models. This software can be used during colonoscopy to identify the manifestations of CMV colitis in real time, guide targeted biopsies, and improve the detection rate of CMV colitis. However, it is important to acknowledge the several limitations of the model.

Currently, despite the outstanding performance of the model, it cannot replace pathological examination as a gold standard in the diagnosis of CMV colitis, but it is an initial step in the process of development. The lack of external validation using independent datasets may affect the generalizability of our model. To establish more robust algorithms and validate the generalizability of our model, multicenter datasets with diverse endoscopic appearances and patient demographics are essential. In this study, we merely utilized images taken by endoscopists in real-life scenarios, without consistent information on localization. Our model is developed based on high-quality images, and its application under real-world suboptimal conditions, such as a blurry field of view due to poor intestinal preparation, may be affected. Furthermore, still images lack important information regarding the distribution of findings throughout the colon. Alternatively, implementation of video-based algorithms could be considered, but large datasets have yet to be established and linked to clinical data. It is important to bear in mind that in real-life scenarios, other infectious pathologies ³⁰ (such as Epstein-Barr virus-induced colitis) may have a similar clinical and endoscopic manifestation to the CMV colitis studied here, which makes the classification task even more complex.

Conclusion

The increasing application of AI provides a promising opportunity to improve the diagnostic capabilities and clinical management of UC and coexisting CMV colitis. By combining clinical data and endoscopic images, the proposed multimodal model has the potential to assist clinicians in accurate and timely recognition and improving the prognosis of UC patients with concomitant CMV colitis.