Development of a multimodal large language model for early warning and diagnosis of chronic ocular GVHD

Abstract

Chronic ocular graft-versus-host disease (coGVHD) after allogeneic hematopoietic stem cell transplantation (allo-HSCT) may lead to irreversible ocular surface damage and even vision loss. Current management of coGVHD faces challenges, with frequent missed or misdiagnosed cases. This study aimed to leverage a multimodal large language model (MLLM) to develop an early warning and diagnostic system for coGVHD. A total of 666 post-allo-HSCT patients (early warning model) and 805 post-allo-HSCT patients (1574 eyes, diagnostic model) were enrolled for construction, internal validation, and external validation of the corresponding models. We proposed the GVHD-MLLM, a multitask multimodal network that fused latent representations from four modal sequences to provide high-precision, real-time predictions for two tasks. The GVHD-MLLM achieved high performance in internal testing, with AUROCs of 93.44% (95% CI: 91.85–95.03%) for early warning, 98.98% (95% CI: 98.59–99.36%) for diagnosis, and 98.24% (95% CI: 98.05–98.43%) for disease severity grading. In external validation, the early warning AUROC was 83.45%, while diagnostic AUROCs across three external sites were all above 96.0%. The disease severity of patients seeking medical treatment after using the early warning model was significantly lower. Junior ophthalmologists also improved diagnostic accuracy using the model as an auxiliary tool. The GVHD-MLLM can process rich multi-modal information collected in clinical practice, and is expected to become an effective tool for managing coGVHD.

Introduction

Graft-versus-host disease (GVHD) is the most common and serious complication occurring after allogeneic hematopoietic stem cell transplantation (allo-HSCT)¹. It is a series of clinical symptoms caused by the overreaction of donor-derived lymphocytes to host antigens². Ocular graft-versus-host disease (oGVHD) often occurs within 6 months to 3 years after transplantation and can occur in 40–60% of allo-HSCT patients³. Patients with chronic ocular GVHD (coGVHD) suffer from lacrimal fibrosis, meibomian gland dysfunction, and conjunctival goblet cell loss, resulting in tear depletion and tear film instability, leading to severe ocular surface dryness and further corneal injury, including punctate keratopathy, filamentous keratitis, corneal ulcer and perforation, and corneal vascularization. Dry eyes, increased fibrous discharge, photophobia, and absence of tears are the most common symptoms^4–6. Chronic ocular GVHD features a complex pathogenesis and diverse clinical manifestations, making it not only difficult to diagnose but also highly challenging to treat. Notably, coGVHD differs significantly from simple dry eye disease (DED) in four key aspects: etiology, pathophysiological mechanisms, disease severity, and treatment regimens. Failure to achieve early identification and timely standardized treatment of coGVHD will lead to severe impairment of patients’ quality of life^7,8.

The onset of coGVHD is often insidious and is easily overlooked by patients in the early stage. In addition, most of the patients after allo-HSCT have poor systemic conditions, and when the eyes are uncomfortable, they are often accompanied by systemic GVHD affecting other organs, making it difficult to conduct timely and frequent ocular examination. Thirdly, many hematologists and ophthalmologists are unfamiliar with the disease, which makes it easy for doctors to misdiagnose coGVHD as another disease. These factors may lead to delayed diagnosis and treatment, which could potentially result in irreversible pathological changes, such as lacrimal gland hypofunction, meibomian gland dysfunction, conjunctival subepithelial fibrosis, and corneal opacity and scarring⁹. The average lag time of diagnosis from noting of ocular symptoms and confirmed diagnosis of coGVHD is 6.3 months. Moreover, patients with long-term delayed diagnosis tend to have more severe ocular symptoms and signs¹⁰. In clinical practice, we find that many patients who come to the hospital after allo-HSCT have severe coGVHD, and have already developed severe visual impairment, even blindness, seriously affecting patients’ quality of life⁸. Early disease intervention can effectively help these patients maintain tear function, thereby preventing deterioration due to coGVHD. Therefore, we urgently need a reliable method to help these patients after allo-HSCT carry out early warning and intervention steps anytime and anywhere, so as to detect the disease at an early stage and facilitate early intervention and treatment by physicians¹¹.

Besides, diagnosing coGVHD remains a complex clinical challenge that often requires the expertise of a highly specialized ophthalmologist, as well as the synthesis of a range of clinical information. Lack of clinicians specializing in oGVHD is a global barrier to management. The diagnostic criteria for coGVHD are mainly the consensus of the National Institutes of Health (NIH) in 2014¹ and the consensus of the International Chronic Ocular Graft-vs-Host-Disease Consensus Group (ICCGVHD) in 2013¹². The NIH consensus diagnostic criteria are easy to use and master, but they cannot distinguish GVHD-induced dry eye from ordinary dry eye, and the diagnostic false positive rate and false negative rate are both high². The ICCGVHD consensus diagnostic criteria are complex and are not widely used in the clinic but mainly used in clinical trials. Furthermore, routine conjunctival biopsy is not beneficial for the early detection of oGVHD, and routine lacrimal gland biopsy is not recommended². Due to the variety of diagnostic criteria and the lack of a diagnostic gold standard, the diagnosis of coGVHD still mainly relies on the clinical experience of doctors, which has a large subjectivity and variability. In addition, due to the atypical clinical features and the lack of understanding of the disease by primary ophthalmologists and hematologists, oGVHD is prone to missed diagnosis and misdiagnosis. Therefore, there is a great need for a useful tool that can assist ophthalmologists at all levels in the accurate diagnosis and severity grading of coGVHD, thereby enabling the effective management and treatment.

In recent years, artificial intelligence (AI) technology has developed rapidly, and its application in the medical field has become more and more extensive. The emergence of neural networks, Deep Learning (DL), and Visual Large Language Model (VLLM) has made major breakthroughs in image recognition, speech, and word processing, which brings opportunities for intelligent diagnosis and treatment of ocular diseases. Since the first article on artificial intelligence in the field of ophthalmology was published in 2016¹³, AI research in the field of ophthalmology has developed rapidly, involving many aspects such as screening assessment, assisted diagnosis, assisted treatment, monitoring, and follow-up^14–17. So far, artificial intelligence has been applied in diabetic retinopathy (DR)¹⁸, glaucoma¹⁹, keratoconus²⁰, infectious keratitis²¹ and many other ophthalmic diseases²², and the application range is constantly expanding. However, in the field of oGVHD, no specific research related to artificial intelligence has been reported yet. In addition, multimodal fusion has gradually become the development trend of medical research. Multimodal fusion models based on Transformer or CNN networks have developed rapidly^23–25. This technology has been clinically applied in the diagnosis, etiological identification, prognosis assessment, risk prediction, or management recommendations of a variety of diseases, including: pulmonary infections²⁶, dementia²⁷, papillary thyroid carcinoma²⁸, interstitial lung disease²⁹, ovarian cancer³⁰, prostate cancer³¹, diabetes³² and so on. However, these methods are mainly oriented towards single tasks, and their modality categories are relatively limited. The emergence of Multimodal Large Language Models (MLLMs) provides a new and effective method for multimodal data fusion^33–35. Different from traditional methods, it uses separate encoders and alignment technology to simultaneously process multimodal data such as text, images³⁶, speech³⁷, and tabular data³⁸, so as to align all data into a unified representation space (usually a text space) after tokenization. It reveals a new intelligent paradigm through the powerful generalization and cross-modal fusion capabilities of large language models. There has been an increasing amount of work applying MLLM technology to the medical field^39–41.

In this study, we proposed the GVHD-MLLM, a multitask multimodal network designed to fuse latent representations across four modal sequences. The model delivered high-precision, real-time prediction results for two core tasks, early warning and diagnosis, and showed potential as an effective tool for the management of coGVHD and the improvement of patient prognosis.

Results

Patients’ characteristics

A total of 506 patients were enrolled for the model construction and internal testing of early warning. The median age of patients was 17.5 years, with 298 males and 208 females. Of these, 354 patients (70.0%) were at high risk for coGVHD. Compared with low-risk patients, high-risk patients showed statistically significant differences in age (p = 0.008), donor gender (p = 0.020), history of cutaneous GVHD (p = 0.027), oral GVHD (p < 0.001), hepatic GVHD (p = 0.043), OSDI (p < 0.001), and tear presence when crying (p < 0.001) (see Supplementary Table 1 for details). Besides, we included 160 patients after allo-HSCT as the external validation dataset. The median age of patients in the external validation group was 17 years, comprising 83 males and 77 females. Of these patients, 100 (62.5%) were classified as high risk for coGVHD. Specific patient characteristics related to the early warning tasks were detailed in Table 1.

Table 1.

Patients’ characteristics of the early warning datasets

	Model construction and internal testing dataset (N = 506 patients)	External testing dataset (N = 160 patients)
Demographics
Age (years, Mean ± SD)*	17.5 ± 27	17 ± 27
Gender (Male, %)	298 (58.9)	83 (51.9)
Risk evaluation
High risk of coGVHD (%)	354 (70.0)	100 (62.5)
Low risk of coGVHD (%)	152 (30.0)	60 (37.5)
Medical history
Donor gender (%)
Male	299 (59.1)	106 (66.3)
Female	179 (35.4)	41 (25.6)
Not filled	28 (5.5)	13 (8.1)
HLA matching (%)
Haploidentical	344 (68.0)	106 (66.3)
Nonidentical	32 (6.3)	13 (8.1)
Identical	96 (19.0)	26 (16.3)
Not filled	34 (6.7)	15 (9.4)
GVHD of other organs (Yes (%)/No (%)/Not filled (%))
Cutaneous GVHD	371 (73.3)/103 (20.4)/32 (6.3)	107 (66.9)/40 (25.0)/13 (8.1)
Oral GVHD	273 (54.0)/194 (38.3)/39 (7.7)	70 (43.8)/75 (46.9)/15 (9.4)
Intestinal GVHD	98 (19.4)/368 (72.7)/40 (7.9)	30 (18.8)/116 (72.5)/14 (8.8)
Pulmonary GVHD	104 (20.6)/361 (71.3)/41 (8.1)	28 (17.5)/117 (73.1)/15 (9.4)
Hepatic GVHD	156 (30.8)/308 (60.9)/42 (8.3)	38 (23.8)/106 (66.3)/16 (10.0)
Primary hematological diseases (%)
Acute myeloid leukemia (AML)	207 (40.9)	76 (47.5)
Acute lymphocytic leukemia (ALL)	147 (29.1)	30 (18.8)
Hybrid acute leukemia (HAL)	4 (0.8)	3 (1.9)
Chronic myeloid leukemia (CML)	15 (3.0)	5 (3.1)
juvenile myelomonocytic leukemia (JMML)	7 (1.4)	2 (1.3)
lymphoma	14 (2.8)	2 (1.3)
Myelodysplastic syndrome (MDS)	30 (5.9)	12 (7.5)
Aplastic anemia (AA)	37 (7.3)	3 (1.9)
Others	4 (0.8)	18 (11.3)
N.A. (not available)	41 (8.1)	9 (5.6)
Postoperative time of allo-HSCT (days, Mean ± SD)*	489 ± 476	544 ± 614
Symptoms and lifestyle habits
OSDI*	15.9 ± 34.1	11.4 ± 33.5
Whether there are tears when crying? (%)
Yes	227 (44.9)	82 (51.2)
No	276 (54.5)	75 (46.9)
Not filled	3 (0.6)	3 (1.9)
Average time spent on electronics per day (%)
Less than 2 hours	130 (25.7)	40 (25.0)
2–4 hours	119 (23.5)	39 (24.4)
4–6 hours	125 (24.7)	23 (14.4)
6–8 hours	67 (13.2)	33 (20.6)
More than 8 hours	51 (10.1)	17 (10.6)
Not filled	14 (2.8)	8 (5.0)

^* Median ± Interquartile Range.

For training and internal validation of the diagnostic model, this study enrolled 676 patients (1346 eyes) after allo-HSCT from PUH3. Additionally, external validation was conducted at three institutions: the Fourth Hospital Affiliated to Soochow University (60 patients/120 eyes), the First Affiliated Hospital of China Medical University (14 patients/28 eyes), and the Airport Yard of Peking University Third Hospital (55 patients/80 eyes). The patients’ characteristics were shown in Table 2. In addition, this study compared the characteristics of patients with chronic ocular GVHD and non-coGVHD in the model construction and internal testing dataset. Compared with non-GVHD patients, coGVHD patients were significantly older (p = 0.004) and had a significantly higher proportion of oral GVHD history (p < 0.001). Additionally, coGVHD patients had significantly higher OSDI (p < 0.001) and CFS scores (p < 0.001), while their tear meniscus height (p < 0.001), BUT (p < 0.001), and Schirmer tear secretion scores (p < 0.001) were significantly lower than those of non-GVHD patients. Statistically significant differences were also observed between the two groups in terms of “whether there are tears when crying” (p < 0.001) and “average time spent on electronics per day” (p = 0.005) (see Supplementary Table 2 for details).

Table 2.

Patients’ characteristics of the diagnostic datasets

	Model construction and internal testing dataset (N = 676 patients)	External testing dataset
		Center 1 (N = 60 patients)	Center 2 (N = 14 patients)	Center 3 (N = 55 patients)
Demographics and medical history
Age (years, Mean ± SD) *	28.0 ± 27.0	36.5 ± 14.6	36.5 ± 16.0	15.0 ± 20.0
Gender (Male, %)	385 (57.0)	28 (46.7)	4 (28.6)	36 (65.5)
GVHD of other organs (Yes (%)/No (%)/Not filled (%))
Cutaneous GVHD	454 (67.2)/119 (17.6)/103 (15.2)	35 (58.3)/23 (38.3)/2 (3.3)	10 (71.4)/4 (28.6)/0 (0.0)	38 (69.1)/12 (21.8)/5 (9.1)
Oral GVHD	391 (57.8)/173 (25.6)/112 (16.6)	28 (46.7)/30 (50.0)/2 (3.3)	7 (50.0)/7 (50.0)/0 (0.0)	26 (47.3)/24 (43.6)/5 (9.1)
Intestinal GVHD	140 (20.7)/433 (64.1)/103 (15.2)	7 (11.7)/51 (85.0)/2 (3.3)	3 (21.4)/11 (78.6)/0 (0.0)	6 (10.9)/44 (80.0)/5 (9.1)
Pulmonary GVHD	114 (16.9)/454 (67.2)/108 (16.0)	8 (13.3)/50 (83.3)/2 (3.3)	2 (14.3)/12 (85.7)/0 (0.0)	14 (25.5)/35 (63.6)/6 (10.9)
Hepatic GVHD	188 (27.8)/382 (56.5)/106 (15.7)	20 (33.3)/38 (63.3)/2 (3.3)	6 (42.9)/8 (57.1)/0 (0.0)	21 (38.2)/29 (52.7)/5 (9.1)
Postoperative time of allo-HSCT (days, Mean ± SD) *	707 ± 731.75	/	/	/
Symptoms and lifestyle habits
OSDI*	24.6 ± 38.6	35.0 ± 45.4	27.3 ± 50.0	15.9 ± 27.3
Whether there are tears when crying? (%)
Yes	159 (23.5)	10 (16.7)	10 (71.4)	32 (58.2)
No	467 (69.1)	21 (35.0)	4 (28.6)	21 (38.2)
Not filled	50 (7.4)	29 (48.3)	0 (0.0)	2 (3.6)
Average time spent on electronics per day (%)
Less than 2 hours	112 (16.6)	1 (1.7)	2 (14.3)	8 (14.5)
2-4 hours	126 (18.6)	0 (0.0)	5 (35.7)	13 (23.6)
4-6 hours	119 (17.6)	0 (0.0)	3 (21.4)	9 (16.4)
6-8 hours	75 (11.1)	0 (0.0)	2 (14.3)	5 (9.1)
More than 8 hours	39 (5.8)	0 (0.0)	2 (14.3)	5 (9.1)
Not filled	205 (30.3)	59 (98.3)	0 (0.0)	15 (27.3)
Clinical examinations	Model construction and internal testing dataset (N = 1346 eyes)	External testing dataset
		Center 1 (N = 120 eyes)	Center 2 (N = 28 eyes)	Center 3 (N = 80 eyes)
Visual acuity (logMAR)*	0.22 ± 0.30	/	/	0.30 ± 0.43
Intraocular pressure (mmHg)*	15.0 ± 5.0	/	/	16.0 ± 5.8
CFS*	5.0 ± 7.0	9.0 ± 9.0	3.0 ± 5.0	2.0 ± 9.0
BUT, second(s)*	2.7 ± 2.0	3.0 ± 3.8	2.0 ± 1.8	3.0 ± 4.0
Schirmer tear secretion scores (mm)*	1.0 ± 3.0	/	1.0 ± 9.3	7.0 ± 12.5
Tear meniscus height (mm)*	0.10 ± 0.10	0.17 ± 0.14	0.20 ± 0.18	0.20 ± 0.19
Diagnosis (coGVHD, %)	1215 (90.3)	100 (83.3)	16 (57.1)	49 (61.3)

^* Median ± Interquartile Range. Center 1, the Fourth Hospital Affiliated to Soochow University; Center 2, the First Affiliated Hospital of China Medical University; Center 3, the Airport Yard of Peking University Third Hospital. CFS, Corneal fluorescein staining scores; BUT, Tear film break-up time.

GVHD-MLLM with multi-task learning shows superior and robust performance

In this study, multimodal fusion and multitask fusion were used successively to improve the model performance.

For the early warning task, the model with only tabular data achieved an AUROC of 86.14% (95% CI: 83.74–88.54%) while the model with only speech achieved an AUROC of 75.80% (95% CI: 72.53–79.06%) on the internal testing cohort. After the integration of tabular data and speech, the performance was improved, with the AUROC increasing to 88.46% (95% CI: 86.28–90.64%). This improvement demonstrated a statistically significant difference compared to the AUROC of the speech-only model (p < 0.001) (Fig. 1A and Table 3). By integrating structured tabular data with speech information, the model acquired richer feature representations, thereby enhancing its early-warning capability. Subsequently, we integrated the early warning and diagnostic tasks. Multitask learning uses the intrinsic correlation between multiple tasks to improve the system's performance. After multitask fusion, the sensitivity, specificity, accuracy, AUROC, F1 of the early warning model of coGVHD were 87.18% (95% CI: 85.15–89.22%), 87.33% (95% CI: 84.28–90.38%), 87.23% (95% CI: 86.13–88.33%), 93.44% (95% CI: 91.85–95.03%) and 90.54% (95% CI: 89.64–91.45%), which increased by 3.66%, 3.33%, 3.57%, 4.98% and 2.79% respectively compared with before multi-task fusion (Fig. 1B, Table 3). The AUROC of the model in the early-warning task after multitask learning was significantly higher than that before multitask learning (p < 0.001).

Fig. 1. Performance of the GVHD-MLLM for early warning, diagnosis and severity grading of coGVHD.

Comparison of the Receiver Operating Characteristic (ROC) curves for the multimodal model, model of only tabular data and model of only speech or images in the internal testing datasets for early warning (A) and diagnosis (D) of coGVHD. ROC curves of the early warning model after multi-task fusion in the internal testing datasets (B) and external testing datasets (C). ROC curves of the diagnostic model after multi-task fusion in the internal testing datasets (E) and external testing datasets (F). G–I ROC curves demonstrating the model’s ability to grade disease severity across mild, moderate, and severe levels.

Table 3.

Performance of the early warning and diagnostic model (95% CI)

			Sensitivity (%)	Specificity (%)	Accuracy (%)	AUROC (%)	F1 (%)
Early warning
Only tabular data			79.3 (76.50–82.09)	83.67 (81.19–86.14)	80.59 (78.85–82.33)	86.14 (83.74–88.54)	85.12 (83.55-86.69)
Only speech			72.68 (64.03–81.32)	68.33 (61.50–75.17)	71.39 (66.06–76.72)	75.80 (72.53–79.06)	77.48 (72.12-82.84)
Multimodal fusion			83.52 (80.98–86.06)	84.00 (81.28–86.72)	83.66 (82.07–85.25)	88.46 (86.28–90.64)	87.75 (86.40-89.09)
Multimodal and multitask fusion (GVHD-MLLM)	Internal validation		87.18 (85.15–89.22)	87.33 (84.28–90.38)	87.23 (86.13–88.33)	93.44 (91.85–95.03)	90.54 (89.64-91.45)
	External validation		79.9 (77.48–82.32)	79.17 (76.62–81.71)	79.62 (78.53–80.72)	83.45 (81.32–85.57)	83.02 (81.93-84.11)
Diagnosis
Only images			85.30 (72.67–97.93)	36.00 (11.40–60.60)	79.96 (71.31–88.61)	68.14 (64.93–71.36)	87.09 (80.28-93.89)
Only tabular data			91.34 (89.40–93.28)	91.33 (88.38–94.29)	91.34 (89.77–92.90)	96.13 (95.09–97.17)	94.93 (93.96-95.89)
Multimodal fusion			95.67 (94.21–97.13)	93.67 (91.19–96.14)	95.45 (94.28–96.63)	98.29 (97.83–98.74)	97.39 (96.69-98.09)
Multimodal and multitask fusion (GVHD-MLLM)	Internal validation		96.23 (95.27–97.20)	95.33 (93.59–97.08)	96.14 (95.33–96.95)	98.98 (98.59–99.36)	97.79 (97.32-98.27)
		Center 1	97.70 (97.19–98.21)	85.50 (73.50–97.50)	95.67 (93.90–97.44)	98.70 (98.15–99.25)	97.44 (96.44-98.44)
	External validation	Center 2	96.25 (92.98–99.52)	95.83 (90.25–100.00)	96.07 (93.22–98.92)	99.06 (98.11–100.00)	96.58 (94.15-99.01)
		Center 3	94.9 (93.67–96.13)	90.32 (88.44–92.21)	93.12 (92.72–93.53)	96.85 (95.79–97.92)	94.41 (94.07-94.76)

For the diagnostic task, the model with only 2 images of the ocular anterior segment achieved a sensitivity of 85.30% (95% CI: 72.67–97.93%) but an AUROC of 68.14% (95% CI: 64.93–71.36%). The model with only tabular data achieved an AUROC of 96.13% (95% CI: 95.09–97.17). After multimodal fusion, the AUROC of the model achieved 98.29% (95% CI: 97.83–98.74%) in the internal testing dataset, which was significantly different from both the image-only model and the tabular-only model (p < 0.001) (Fig. 1D, Table 3). The subsequent multitasking fusion further improved the performance of the diagnostic model (p = 0.025). The sensitivity, specificity, accuracy, AUROC, F1 of the diagnostic model of coGVHD were 96.23% (95% CI: 95.27–97.20%), 95.33% (95% CI: 93.59–97.08%), 96.14% (95% CI: 95.33–96.95%), 98.98% (95% CI: 98.59–99.36%) and 97.79% (95% CI: 97.32–98.27%) which increased by 0.56%, 1.66%, 0.69%, 0.69% and 0.40%, respectively compared with before multi-task fusion (Fig. 1E, Table 3).

Following the sequential integration of multi-modal and multi-task approaches, the early warning and diagnostic model demonstrated significantly enhanced performance. Ultimately, we adopted this optimized model as our final version and designated it as GVHD-MLLM.

External validation of GVHD-MLLM

In the external validation dataset, the GVHD-MLLM for early warning achieved an AUROC of 83.45% (95% CI: 81.32%-85.57%) (Fig. 1C, Table 3). The GVHD-MLLM for diagnosis achieved AUROCs of 98.70% (95% CI: 98.15%-99.25%), 99.06% (95% CI: 98.11–100.00%), and 96.85% (95% CI: 95.79–97.92%), respectively in three external validation ophthalmology centers (Fig. 1F, Table 3). The results demonstrated that the GVHD-MLLM exhibited stable performance and potential applicability in clinical settings.

Disease severity grading of GVHD-MLLM

This study further assessed the performance of the constructed model in grading the severity of coGVHD in patients. The results showed that with single image modality input, the AUROC of the model for severity grading was 66.04% (95% CI: 63.80–68.27%); with single tabular modality input, the AUROC of the model for grading was 95.18% (95% CI: 94.71–95.65%). After multimodal fusion, the grading performance of the model was significantly improved, with the AUROC reaching as high as 98.24% (95% CI: 98.05%-98.43%). In addition, the ROC curves indicated that the model exhibited good classification ability for mild, moderate, and severe coGVHD (Fig. 1G–I). Finally, we validated this diagnostic grading model on external datasets from three centers, and the AUROCs were 99.79% (95% CI: 99.69–99.89%), 99.65% (95% CI: 99.36–99.94%), and 92.31% (95% CI: 91.75–92.87%), respectively. These results confirmed that the model has strong robustness and generalizability in terms of severity grading (Supplementary Table 3). In a supplementary analysis, the reconstructed Lee ocular subscale score showed significant positive associations with both the study-adopted coGVHD severity grades and the GVHD-MLLM-predicted severity categories, suggesting that the proposed severity stratification was aligned with patient-reported ocular symptom burden (Supplementary Note 5).

Performance of GVHD-MLLM on incomplete modality

We further verified the performance of the GVHD-MLLM in the absence of partial modal data. As shown in Fig. 2 and Supplementary Table 4, the GVHD-MLLM consistently performed well even when one data modality, either speech or image data, was missing. These findings showed that the GVHD-MLLM could make reliable predictions even when data modalities are incomplete, indicating its applicability across a variety of clinical scenarios in which a complete dataset was unavailable. However, the model’s performance was compromised in the absence of tabular data, underscoring the criticality of such data for both early warning and diagnosis.

Fig. 2. Additional Assessment of the performance of GVHD-MLLM.

ROC curves of GVHD-MLLM for early warning (A) and diagnosis (B) in the absence of partial modal data. C–F Comparison of disease severity with and without the implementation of the early warning model. G Evaluation of the diagnostic performance of GVHD-MLLM compared to ophthalmologists. OSDI Ocular Surface Disease Index, CFS corneal fluorescein staining, BUT tear film break-up time.

Comparison of disease severity with and without the early warning model

To further investigate the benefits for patients following the implementation of the early warning model in clinical practice, we compared the disease severity of coGVHD patients in the external validation group (186 eyes) with that of coGVHD outpatients at PUH3 in 2020 (283 eyes). The results demonstrated that, compared to the observations in 2020, the CFS (p = 0.007) and OSDI scores (p < 0.0001) of patients in the external validation cohort were significantly lower, while the BUT (p < 0.0001) and tear meniscus height (p < 0.0001) were significantly higher (Fig. 2 and Supplementary Table 5). This study found that after the implementation of the early warning model, the severity of the patients’ ocular conditions was significantly lower than that before the implementation.

Evaluation of the diagnostic performance of GVHD-MLLM compared to ophthalmologists

To further assess the practical value of the diagnostic model, we conducted this comparative study. The F1 score achieved by the GVHD-MLLM model for diagnostic purposes was 97.30%. The average F1 score of junior ophthalmologists was 86.65%, with individual scores of 82.54%, 84.71%, 87.32%, 88.57% and 90.14%. The average F1 score of senior ophthalmologists was 94.54%, with individual scores of 94.59%, 94.59% and 94.44%. After referring to the AI model, the average F1 scores of junior ophthalmologists improved to 94.43% (89.55%, 96.10%, 96.00%, 94.59% and 95.89% respectively), which was significantly higher than that at the initial diagnosis (p = 0.016). The F1 scores of senior ophthalmologists remained at approximately 95.50%, showing little change (p = 0.200) (Fig. 2 and Supplementary Table 6). These results indicated that the performance of the diagnostic model of coGVHD is superior to that of junior ophthalmologists and comparable to that of senior ophthalmologists. In practical clinical settings, referring to the results provided by the diagnostic model may improve the diagnostic accuracy of junior ophthalmologists.

Analysis of contributing features in the GVHD-MLLM

We used rollout attention values to visualize the features contributing most to the GVHD-MLLM’s predictions for the early warning and diagnosis of coGVHD (Fig. 3). For the coGVHD warning system, we found that the most important feature was “Uncomfortable in places with low humidity?”, followed by “Poor vision?”, “Uncomfortable in windy conditions?” and “Do you have tears when you cry?”. For the diagnostic system for coGVHD, we found that the most important features were “OSDI”, “CFS” and “BUT”. This was consistent with our clinical experience. For the rollout attention values of all features, please refer to Supplementary Tables 7 and 8.

Fig. 3. The contribution of factors in the GVHD-MLLM.

A Rollout attention values of the top ten factors in the GVHD-MLLM for early warning. For the rollout attention values of all features, please refer to Supplementary Table 7. B Rollout attention values of the top ten factors in the GVHD-MLLM for diagnosis. For the rollout attention values of all features, please refer to Supplementary Table 8. C–F Visualization of the images. In the heat map, regions that exhibit warmer color tones signify a more substantial impact on the final decision. * The complete question is: “If you have tears when you cry, do the volume of tears change?” # The complete question is: “If you don’t have tears when you cry, do you have moist sensation in your eyes?”.

Grad Cam was used to analyze important visual areas in the model prediction⁴². The study revealed that the GVHD-MLLM not only focused on the cornea, a key area routinely examined by ophthalmologists, but also attended to the conjunctiva and eyelid margin, which may receive less attention during clinical examinations (Fig. 3).

Discussion

In this study, we developed a Multimodal Large Language Model, designated GVHD-MLLM, capable of simultaneously performing both early warning and diagnostic tasks for coGVHD (Fig. 4). The GVHD-MLLM for early warning of coGVHD integrated both speech and structured tabular data, aiming to enhance the early detection of coGVHD in patients following allo-HSCT. The GVHD-MLLM for diagnosis of coGVHD integrated structured tabular data alongside anterior segment images, which was designed to assist ophthalmologists in achieving more accurate diagnoses and practical disease severity grading of coGVHD. The GVHD-MLLM demonstrated satisfactory predictive and diagnostic performance and showed potential as a valuable tool for the management of coGVHD, with possible benefits for patient prognosis.

Fig. 4. Expected application scenarios of GVHD-MLLM in real life.

Patients who have undergone allo-HSCT can conveniently complete the early warning questionnaire and input their speech via their mobile phones at home. The system will promptly assess the risk of coGVHD and recommend that high-risk individuals seek ophthalmological evaluation. Upon visiting the ophthalmology clinic, patients undergo comprehensive ocular examinations and anterior segment photography. By inputting the tabular data and images into the system, it can give the diagnosis of coGVHD or non-coGVHD, thereby assisting in the accurate diagnosis of coGVHD and supporting telemedicine applications. Allo-HSCT allogeneic hematopoietic stem cell transplantation, coGVHD chronic ocular graft-versus-host disease.

In order to achieve accurate early warning and diagnosis of coGVHD, our study adopted a variety of methods to improve the performance of the GVHD-MLLM. Firstly, our study built upon a Multimodal Large Language Model (MLLM). Unlike traditional LLMs that process only text⁴³, MLLMs can integrate multiple data types, enabling comprehensive understanding and reasoning for complex clinical information^43,44. Secondly, our model integrated multi-modal information of demographics, medical history, symptoms, lifestyle habits, clinical examination, speech, and images to enhance the performance of early warning and diagnosis. The integration of different modalities provided an opportunity to improve the robustness and accuracy of models, bringing AI closer to clinical practice⁴⁵. In the early warning task of this study, compared with the single modality of speech, the multi-modal system of tabular data and speech successfully improved the AUROC by 12.66%. In the diagnostic task of this study, compared with the single modality of images, the multi-modal system of images and tabular data successfully improved the AUROC by 30.15%. Last but not least, in order to further improve the performance of the system, we integrated the tasks of early warning and diagnostics. Multi-task learning is a joint learning paradigm that leverages the intrinsic correlation between multiple related tasks to improve performance, enhance generalization, and reduce overall computing cost⁴⁶. Following the implementation of multi-task fusion, the AUROC of the GVHD-MLLM for early warning and diagnosis improved by 4.98% and 0.69%, respectively.

Integrating patients’ speech into the early warning model offered several advantages. Firstly, it has been clinically found that speech can contain more detailed symptoms that could not be captured in a structured questionnaire. For patients who cannot accurately describe their symptoms, such as the elderly and children, relatives can also provide a richer description of their disease symptoms from the perspective of observers. Secondly, in clinical observations, we noted that patients with coGVHD exhibited severe symptoms such as dry eyes, photophobia, and visual impairment. These symptoms significantly impaired their ability to read and effectively respond to text-based questionnaires. Using speech as an interactive way to gather symptom information can also avoid aggravating patients’ eye discomfort. This speech-based data collection approach may help address health disparities by providing a more accessible assessment method for patients who have difficulty completing text-based questionnaires, such as those with low literacy levels or visual impairment. This study verified the feasibility and effectiveness of the speech in collecting relevant information for the early warning model among the adolescent and young adult patient population after allo-HSCT. Overall, the integration of speech into the large language model in this study was a meaningful practice.

In clinical practice, accurately differentiating coGVHD from dry eye disease (DED) poses a significant challenge, especially for young or grassroots ophthalmologists. Misdiagnosis of coGVHD as DED can lead to delayed diagnosis and treatment for the patient. Conversely, misdiagnosing DED as coGVHD may result in prolonged use of glucocorticoids or immunosuppressants, thereby increasing the risk of elevated intraocular pressure and ocular infections. To enhance the diagnostic accuracy of coGVHD, we successfully developed an artificial intelligence-based diagnostic model for coGVHD. The GVHD-MLLM exhibited commendable diagnostic performance and achieved robust results in external validations conducted across various regions and hospitals. Furthermore, to align with clinical practice requirements, we conducted additional training on the model to enable its disease severity grading capability. Results demonstrated that the model could accurately classify mild, moderate, and severe GVHD, achieving robust and consistent performance on both internal and external datasets, supporting its potential for real-world clinical application and early identification of severe cases that may require prompt intervention.

To further validate the diagnostic performance of the GVHD-MLLM, we conducted a comparison of diagnostic accuracy between the model and ophthalmologists. The F1 score of this model (97.30%) was higher than that of junior ophthalmologists (86.66% in average) and was comparable to that of senior ophthalmologists (94.54% in average). In addition, when junior ophthalmologists reconsidered and revised their diagnoses with reference to the GVHD-MLLM’s outputs, their F1 scores improved significantly (p = 0.016). Importantly, the second-round diagnoses of the junior ophthalmologists were not in complete agreement with the model, suggesting that their final judgments were not simply a direct acceptance of the algorithm’s outputs. This improvement could be attributed to the interactive process: discrepancies between the model’s predictions and the ophthalmologists’ initial assessments prompted them to conduct deeper case analyses, re-examine critical diagnostic features, and carefully confirm the presence of coGVHD. These results highlighted the potential of the GVHD-MLLM to support human-AI collaborative decision-making, which may help junior ophthalmologists improve diagnostic accuracy, reduce the risk of misdiagnosis, and facilitate more informed treatment strategies.

Furthermore, we made efforts to enhance the interpretability of the GVHD-MLLM. Based on rollout attention values of the GVHD-MLLM, we found that OSDI, CFS and BUT were the most important indicators for diagnosis of coGVHD. This suggested that in addition to CFS, patient symptoms and tear film stability also played important roles in the diagnosis of coGVHD. In addition, the Schirmer tear secretion score showed a relatively low attention value in the diagnosis of coGVHD in this study. The NIH diagnostic criteria mainly rely on Schirmer tear secretion scores for diagnosis; however, the Schirmer’s test is cumbersome, time-consuming, and uncomfortable for patients. Besides, previous studies have shown that Schirmer tear secretion scores do not adequately reflect the severity of coGVHD and cannot be used as a monitoring indicator for long-term follow-up⁴⁷. This study further suggested that the Schirmer tear secretion score had limited value in the diagnosis of coGVHD. Therefore, Schirmer’s test may not be necessary for every patient in routine clinical practice, which could streamline the diagnostic process and improve patient experience. In addition, the study found that the presence or absence of tears during crying was important for identifying coGVHD. One of the main challenges in diagnosing coGVHD is the lack of a clear distinction between early-stage coGVHD and dry eye disease, which is clinically important because the progression of coGVHD is an irreversible process. Our findings suggested a potential new perspective for the early warning and diagnosis of coGVHD by placing greater emphasis on changes in tear volume, particularly whether tears are present when a patient cries. Especially for children who can’t describe symptoms and cooperate with clinical examination, observing the change in tear volume is an easy indicator to manipulate.

The GVHD-MLLM demonstrates significant application value across diverse medical scenarios. For instance, at home, patients after allo-HSCT can conveniently use their smartphones to conduct an initial risk assessment of coGVHD through the GVHD-MLLM system, enabling timely medical consultation. In hematology clinics, hematologists can use this early warning model to quickly screen for patients at high risk of coGVHD and promptly refer these patients to ophthalmology specialists, thus preventing delays in diagnosis and treatment. In ophthalmology specialist clinics with a large number of patients and high diagnostic/treatment pressure, ophthalmologists can use this model for auxiliary diagnosis to provide efficient and accurate diagnosis of coGVHD. In remote areas where specialized expertise is scarce, the system can deliver coGVHD diagnoses with an accuracy comparable to that of experienced ophthalmologists, thereby mitigating disparities in healthcare provision. The aforementioned application scenarios cover the entire process of patients’ care, from home-based monitoring to specialist diagnosis. They fully highlight the application value of this tool in the full-process management of coGVHD, and it is expected to become an important auxiliary tool for improving patient prognosis.

This research had several limitations. Firstly, the size of our dataset was relatively small compared to previous AI-related research using tens of thousands of datasets. However, because coGVHD has a relatively low incidence, datasets comprising several thousand cases are rare in this field. Secondly, we did not develop an early warning and diagnostic model that included acute oGVHD. This was mainly because patients with acute oGVHD often presented with severe systemic GVHD at disease onset and were therefore unable to attend the ophthalmology clinic in a timely manner. In addition, the duration of acute oGVHD was relatively short. As a result, this study could not include a sufficient number of patients with acute oGVHD for model training. This limitation was largely due to objective clinical constraints. In future work, we will strengthen screening efforts to identify and include more patients with acute oGVHD, thereby further improving the model. Future research still needs to conduct large-scale, multi-center prospective longitudinal studies that include repetitive validation across diverse cohorts (e.g., different demographic groups), a broader age range (e.g., elderly patients with an average age of 60), and broader ophthalmic conditions (including infectious eye diseases, dry eye disease, and so on) to further confirm the clinical utility of this model and lay a more robust foundation for its potential deployment in real-world clinical settings. Furthermore, we plan to collaborate with departments such as dermatology, gastroenterology, and stomatology to enhance the model’s ability to predict the risk of GVHD involving multiple organs after allo-HSCT. In the future, a comprehensive artificial intelligence platform for collaborative GVHD management could be developed based on this model, thereby facilitating integrated patient management and enhancing prognosis.

In summary, we successfully developed a multimodal large language model (GVHD-MLLM) for the early warning and diagnosis of coGVHD, which showed potential as a valuable tool for the clinical management of patients with coGVHD. The GVHD-MLLM represented a meaningful application of multimodal large language models in the integrated management of ocular surface diseases. We anticipate that more impactful and clinically relevant AI models will be developed in the future, benefiting both patients and healthcare providers.

Methods

Study design

This study was a multi-center, cross-sectional study, which was carried out according to the flow chart (Fig. 5).

Fig. 5. Study flowchart of cohort construction for the early warning and diagnostic tasks.

A Flowchart showing the selection and allocation of participants in the internal and external cohorts for the early warning task. B Flowchart showing the selection and allocation of participants in the internal and external cohorts for the diagnostic task.

To construct an early warning model, we recruited 552 patients after allo-HSCT who were admitted to the Eye Center of Peking University Third Hospital (PUH3) from June 7 to November 30, 2021. 46 patients were excluded due to poor speech quality, no definite diagnosis, and incomplete tabular data, and 506 patients were finally included in the study. Patients with chronic ocular GVHD (coGVHD) in either eye were defined as high risk of coGVHD. We randomly assigned 405 patients to the Training Dataset and 101 patients to the Testing Dataset in a ratio of 8:2 with 10 resampling events.

To evaluate the effectiveness of the early warning model in practical applications, we implemented the early warning questionnaire within a mobile application (Yiduo Health) to provide immediate risk assessment for coGVHD starting in August 2022. Patients who had undergone allo-HSCT were recruited to complete the early warning questionnaire online through three channels: (1) referrals from hematology centers in China; (2) social media groups for HSCT survivors; (3) in-clinic promotions at PUH3. Between August 2022 and August 2023, a total of 400 individuals completed the questionnaire. Of these, 245 patients subsequently visited the outpatient department at PUH3. We utilized data from 160 individuals with complete case records as an external validation cohort for the early warning model. The inclusion criteria were as follows: (1) patients who have received allogeneic hematopoietic stem cell transplantation for serious blood diseases; (2) patients who are willing to participate in this study after learning about it; (3) patients who visited the ophthalmic clinic within half a month after filling out the early warning questionnaire. The patients were excluded if they (1) did not attend the ophthalmic clinic within half a month after filling out the early warning questionnaire; (2) lacked diagnoses or had unclear diagnoses; (3) provided poor-quality speech; (4) provided severely incomplete tabular data.

To construct the diagnostic model, 820 patients (1640 eyes) after allo-HSCT were recruited from Peking University Third Hospital, China, from June 3, 2020, to September 27, 2023. 294 eyes were excluded based on the exclusion criteria. Finally, 676 patients (1346 eyes) were recruited, and we randomly assigned 1069 eyes to the Training Dataset and 277 eyes to the Internal Testing Dataset in a ratio of 8:2 with 10 resampling events. For the external validation cohort, patients were recruited from three hospitals of China, including 60 patients (120 eyes) from The Fourth Hospital Affiliated to Soochow University (Center 1), 14 patients (28 eyes) from the First Affiliated Hospital of China Medical University (Center 2), and 55 patients (80 eyes) from Peking University Third Hospital Airport Yard (Center 3). The inclusion criteria of the diagnostic model were as follows: (1) patients who have received allogeneic hematopoietic stem cell transplantation for serious blood diseases; (2) patients attending the ophthalmic clinic; (3) patients who are willing to participate in this study after learning about it. Exclusion criteria of the diagnostic model included: (1) patients with infectious keratoconjunctivitis, allergic conjunctivitis, trichiasis or other serious ocular surface diseases; (2) patients with a history of eye surgery unrelated to the current disease within 1 year; (4) poor image quality or incomplete images; (5) lack diagnoses or the diagnosis is unclear; (6) provided severely incomplete clinical data. Due to the small number of patients with acute oGVHD and the atypical symptoms, patients with acute oGVHD were not included in our diagnostic model.

The patients in the early warning cohort and the diagnosis cohort partially overlapped, but were not identical. Specifically, 100 patients were included in both cohorts, 406 patients were exclusive to the early warning cohort, and 576 patients were exclusive to the diagnostic cohort. This partial overlap results from the two models having both differences and partial alignment in three aspects: recruitment timelines, inclusion/exclusion criteria, and data collection requirements.

Chronic oGVHD was diagnosed based on the 2014 National Institutes of Health (NIH) Consensus Criteria¹, the 2013 International Chronic Ocular GVHD Group (ICCGVHD) Consensus¹² and ophthalmologists with over 20 years of rich clinical experience in ocular surface diseases from Peking University Third Hospital and other hospitals for external verification.

The severity grading framework used in this study was developed mainly with reference to the key assessment domains emphasized by the 2013 ICCGVHD Consensus¹² and the 2014 NIH Consensus¹, together with the structure of our dataset and ophthalmic clinical experience. The grading mainly refers to the Ocular Surface Disease Index (OSDI) score, Corneal Fluorescein Staining (CFS) score, and Schirmer tear secretion score. The specific grading rules are as follows: (1) Scoring Criteria for Individual Indicators: The three indicators (OSDI, CFS, and Schirmer tear secretion score) are scored separately according to Table 4. (2) Calculation Rules for Comprehensive Grading: First, calculate the average score of the three indicators. If the value of any indicator is missing, that indicator will be excluded from the average calculation. The average is obtained by summing the scores of the valid indicators and then dividing by the number of valid indicators. Determine the severity of coGVHD based on the average score: Average score ≤ 1: Mild coGVHD; 1 < Average score ≤ 2: Moderate coGVHD; Average score > 2: Severe coGVHD. (3) Special Judgment Rule: If the CFS is ≥ 9, the case will be directly classified as severe coGVHD, regardless of the scores of OSDI, Schirmer tear secretion score, or the above-mentioned comprehensive average score.

Table 4.

Scoring criteria for individual indicators

	Normal (0 points)	Mild abnormality (1 point)	Moderate abnormality (2 points)	Severe abnormality (3 points)
Ocular Surface Disease Index (OSDI)	<13	≥13 & <23	≥23 & <33	≥ 33
Corneal Fluorescein Staining (CFS)	0	≥1 & <3	≥3 & <5	≥5
Schirmer tear secretion score	>10	>5 & ≤10	>2 & ≤5	≤2

To further evaluate the relationship between the severity stratification used in this study and patient-reported ocular symptom burden, an additional analysis using the Lee Symptom Scale ocular subscale was performed⁴⁸. Detailed methods and results are provided in the Supplementary Materials (Supplementary Note 5).

Ethics

This study was approved by the Medical Science Research Ethics Committee of Peking University Third Hospital (Protocol number: M20241030). This study was carried out strictly in accordance with the ethical guidelines of the Declaration of Helsinki, and all participants in this study provided written consent.

Data collection

In the early warning data collection phase, for each patient after allo-HSCT who planned to come to the oGVHD specialist clinic, we asked him or her to complete the Patient Self-Checklist Questionnaire and the Speech for Early Warning. The Patient Self-Checklist Questionnaire encompasses detailed information regarding demographics, medical history, symptoms, and lifestyle habits. The specific questionnaires and speeches are shown in Supplementary Tables 9 and 10. We collected tabular data, speech data and corresponding medical records of patients and conducted a secondary review of the diagnostic results to ensure the validity of the diagnosis. The speeches and questionnaires for younger children were completed by their parents.

In the diagnostic data collection phase, patients were required to fill out the Patient Self-Checklist Questionnaire and undergo clinical examinations. The severity of patients’ ocular symptoms was evaluated using the Ocular Surface Disease Index (OSDI)⁴⁹. The following data were collected for objective ocular examination⁴: visual acuity (measured in logMAR, for lower vision, the following criteria were used: Finger Count is 2.0, Hand Move is 2.3, Light Perception is 2.6, and No Light Perception is 2.9)⁵⁰, intraocular pressure (measured with rebound tonometer), corneal fluorescein staining (CFS) score (cornea was divided into five quadrants; for each quadrant, 0 point: no corneal epithelium stained, 1 point: 1-30 stained points, 2 points: >30 stained points without fusion, 3 points: punctate keratopathy fusion and/or corneal filiform paraphyte and/or corneal ulcer; final score for each eye is the sum of each quadrant’s score, the maximum of total score is 15 points)⁵¹, fluorescein tear film break-up time (BUT), Schirmer’s test score without anesthesia and tear meniscus height⁵². The tabular data for the diagnostic task include the Patient Self-Checklist Questionnaire and clinical examination results, which are presented in Supplementary Table 11. The clinical features of the anterior segment were photographed by slit-lamp microscopy. The examiners took images of the unstained anterior segment (including the patient’s cornea, conjunctiva, and palpebral margin) using diffused light, and then took images of the anterior segment stained with sodium fluorescein using cobalt blue light. Images were performed using Haag-Streit BX900® slit-lamp microscopy in the Peking University Third Hospital and the Fourth Hospital Affiliated to Soochow University, MediWorks S390H® slit-lamp microscopy in the First Affiliated Hospital of China Medical University, and TOPCON DC-3® slit-lamp microscopy in the Peking University Third Hospital Airport Yard. Each case should follow a standard imaging protocol that includes ambient indoor light (90-180 lux), a maximum width diffuse beam (30 mm), and a maximum light intensity within the patient’s tolerance range. Images were taken at the 10× magnification setting. The clearest images of each patient’s eyes before and after staining were selected for inclusion in the study. Sample photos are shown in Supplementary Fig. 1.

Multimodal framework and model design

In this study, we collected a comprehensive set of multimodal data, with each modality exhibiting distinct characteristics and functions that complement one another to enable high-performance task execution. Leveraging recent advances in multimodal large language models (MLLMs), many of which offer strong modality fusion and understanding capabilities, we selected a state-of-the-art open-source pre-trained MLLM as a basis to build our model. We then fine-tuned this model on the collected dataset to align it with our study cohort’s characteristics.

The early warning and diagnostic tasks involved diverse and abundant multimodal information related to coGVHD patients. Given the strong correlation between these two tasks, we adopted a multi-task learning strategy, allowing knowledge learned from one task to benefit the other, thereby improving the generalization performance across tasks. Furthermore, since the data volume for each individual task was relatively limited, integrating data from multiple tasks via multi-task learning provided the model with greater training diversity and enhanced its learning capability. Consequently, we developed a unified model that performed both early warning and diagnosis for coGVHD within a single multimodal framework.

In the early warning task, both structured tabular data and unstructured speech data were collected, while in the diagnosis task, structured tabular data and image data were collected. Accordingly, our MLLM was designed to process four modalities: text (task-specific prompts), speech, image, and professional tabular data, including patient questionnaires (used in both tasks) and ophthalmologist checklists (specific to the diagnosis task) (Fig. 6A). To achieve this, we adopt the open-source Qwen-VL³⁶ as our base, inheriting its architecture and pre-trained parameters, and modified its structure to accommodate our specific multimodal requirements. The overall architecture of our model is illustrated in Fig. 6B. It was designed to handle all modalities involved in both tasks. For each task, we constructed corresponding training datasets, with each input consisting of its own clinical information and a task-specific text prompt. The model then generated the final discriminative prediction for each task. Details of the data formats are provided in the Supplementary Notes 2 and 3.

Fig. 6. Model architecture and modeling strategy.

A Multimodal data, including speech, images, tabular data (demographics, symptoms, medical history, lifestyle habits, and clinical examination), and ophthalmic knowledge, are integrated to construct the large language model referred to as GVHD-MLLM. B shows the model construction process. (b1) shows the modeling strategy and process. In this study, multimodal fusion and multitask fusion are used successively to improve the model performance. (b2) shows the GVHD-MLLM for the early warning and diagnosis of coGVHD. It is based on one pre-trained model and constructs a specific architecture to accommodate all data, including images, speech, tabular, and text. At the same time, multi-task learning makes use of all integrated data, which involves data in the early warning task and data in the diagnosis task. GVHD-MLLM contains four feature extraction modules, each is constructed with a specific method. (b3) The backbone of MLLM, the visual embedding module and language embedding module are directly inherited from the base model (Qwen-VL). (b4) The speech recognition model is conducted using an external speech recognition model, Qwen-audio, which converts spoken language into text for subsequent processing. (b5) (b6) The initial structure and parameters of the tabular embedding module are inherited from the language embedding module, but due to the professionalism of tabular data, the tabular embedding module needs to be fine-tuned ahead of time in order to make tabular data align with knowledge-enhanced text. This stage occurs before the full model fine-tuning, which allows the final model to directly understand the tabular data in tasks. C Model validation comprises two primary tasks: early warning and diagnosis of coGVHD. The accuracy of the early warning and diagnostic tasks was separately verified, and rollout attention values were used to analyze the critical factors. Furthermore, the clinical characteristics of patients were compared before and after the implementation of the early warning system. Additionally, the accuracy of the diagnostic system was compared to that of ophthalmologists.

Model architecture and multimodal fusion mechanism

The model construction is illustrated in Fig. 6B. Our MLLM comprised four feature extraction modules: visual embedding module, speech recognition module, tabular embedding module, and language embedding module, followed by a transformer backbone for feature fusion. First, we adopt the open-source Qwen-VL³⁶ as the base, which inherently supports image and text data processing; thus, its visual and language embedding modules were directly inherited. For speech processing, we incorporated an external speech recognition module-the Qwen-audio model³⁷-known for its strong Chinese speech recognition performance. This module remained fixed during training to preserve its recognition capabilities. And to handle the tabular data unique to our tasks, we introduced a dedicated tabular embedding module. Architecturally, it mirrored the language embedding module and inherited its initial parameters. We converted tabular inputs into JSON format, treating them as structured text. However, these tabular data included specialized ophthalmic terms (e.g., ‘OSDI’: 34.09, ‘CFS’: 1, ‘BUT’: 2) and numeric values that were difficult to interpret directly. To address this, we first organized relevant ophthalmic knowledge and transformed these cryptic tabular entries into comprehensible text descriptions (see Supplementary Note 2). We fine-tuned the tabular embedding module in advance to align tabular data representations with the knowledge-enhanced textual descriptions, while keeping all other model parameters frozen during this stage (see Supplementary Note 2). Only after this targeted alignment do we proceed to jointly train the entire model for early warning and diagnosis tasks. During subsequent joint training, no additional external knowledge input was required, as the tabular embedding module had already internalized essential ophthalmic domain knowledge. By injecting explicit domain knowledge into the tabular embeddings, our approach enabled the multimodal LLM to effectively integrate complex, specialized structured data. Our design allowed the model to seamlessly and efficiently utilize domain-specific information, thereby enhancing its generalization performance under data-scarce conditions-a common challenge in rare diseases such as coGVHD.

Upon completing the construction of each embedding module, we connected them into the backbone. Our model backbone, comprising a substantial number of parameters, was directly inherited from the base model Qwen-VL³⁶. This transformer-based architecture is capable of processing and integrating multimodal information within a unified textual representation space. As illustrated in Fig. 6B, outputs from each modality encoder-text, image, speech, and tabular-were first tokenized and subsequently concatenated at the token level to form a unified multimodal input sequence. This sequence was then fed into the backbone, which jointly processes all embedded features. The model’s joint reasoning capability is crucial. In our framework, the transformer backbone conducted attention-based cross-modal reasoning across all tokens, enabling effective integration of heterogeneous multimodal information. Specifically, ophthalmic tabular data, patient speeches, and image features were contextualized with task-specific prompts, thereby allowing the model to infer early warning signals, diagnostic outcomes, or severity grading for coGVHD. The final model, termed GVHD-MLLM, thus exhibited the capability for integrated multimodal reasoning and simultaneous early warning and diagnosis within a unified framework. Details of the implementation are given in Supplementary Note 4.

Human–AI comparative evaluation protocol for coGVHD diagnosis

We evaluated the collaborative potential between the AI model and ophthalmologists in diagnosing coGVHD. To ensure an unbiased comparison, we recruited eight ophthalmologists who had not previously participated in the study. Ophthalmologists were divided into two groups based on their clinical experience: the junior group (J1–J5), consisting of ophthalmologists with less than 10 years of clinical experience, and the senior group (S1–S3), consisting of corneal specialists with more than 10 years of clinical experience. All ophthalmologists in the senior group had certain diagnostic and therapeutic experience in the field of ocular GVHD.

The study was conducted in two sequential stages to assess human–AI collaboration. In the first stage, ophthalmologists independently diagnosed 49 eyes using only tabular clinical data (questionnaires and examination results) and anterior segment images, without access to the AI model’s predictions. One week later, the same ophthalmologists were provided with the AI model’s diagnostic outputs (coGVHD/non-coGVHD) alongside the same data and images. They were instructed to review their initial diagnoses, re-evaluate each case, and adjust their conclusions if appropriate. F1 scores were then calculated for each ophthalmologist both before and after AI-assisted review, enabling a quantitative analysis of the impact of AI support on clinical decision-making.

Explaining and visualizing techniques for GVHD-MLLM

Due to the complex attention mechanism in the transformer network (basic structure of backbone), attention value in the model is difficult to explain the final attention distribution of the input information. Quantifying Attention Flow in Transformers⁵³ shows that the attention rollout method provides a more reasonable indicator to show the attention distribution, and it is a better way to attribute and visualize the final model. Therefore, we used rollout attention value to visualize the features contributing most to the GVHD-MLLM.

The general image analysis method Grad-Cam was used to analyze important visual areas in the model prediction⁴². Specifically, we used the GVHD-MLLM prediction results (coGVHD/non-coGVHD) to compute the gradients flowing into the last layer of the image encoder. Based on these gradients, we obtained a rough localization map and used visualization methods to highlight the important areas.

Statistical analyses

SPSS for Windows version 24 (SPSS, Inc., Chicago, IL, USA) and Python 3.10 was used for statistical analysis. The χ² test was conducted for the comparison of qualitative variables. The Shapiro-Wilk test was used to determine whether the continuous variables were normally distributed. Nonnormally distributed variables were expressed as the median ± interquartile range, and categorical variables were expressed as frequencies and percentages. For nonnormally distributed variables, the Mann-Whitney U test was used to compare whether there were significant differences between the two groups. Early warning and diagnostic performance were assessed by calculating the sensitivity, specificity, accuracy, area under the receiver operating characteristic curve (AUROC), and F1 with 95% confidence intervals (CIs). The 95% CIs were calculated using the nonparametric bootstrap method with 10 resampling events, while keeping a constant ratio of positive and negative cases. The DeLong test was used to compare the significance of the difference in area under the two ROC curves (AUROC). The p-values were bilateral, and less than 0.05 was considered statistically significant.

Author contributions

Shuwan Liu contributed to project design, data collection, data organization, data analysis, manuscript writing, and revision. Lili Cao was responsible for project design, model construction, data analysis, manuscript writing, and revision. Haoran Wu was responsible for project design, model construction, and data analysis. Rongmei Peng contributed to project design, data collection, and manuscript revision. Bohao Hu was involved in project design and data collection. Xiaofeng Zhang, Xinyu Zhuang, Yuzhao Sun, Zhan Shen, Jiao Ma, Rong Wu, Yinghan Zhao, Yinan Liu, Yi Wang, and Liang Han were responsible for data collection. Bo Xu and Jing Hong contributed to project design and manuscript revision.

Data availability

The datasets generated and analyzed during the current study are not publicly available due to institutional patient privacy regulations, but are available from the corresponding author on reasonable request and pending appropriate ethical approval.

Code availability

The training code for the GVHD model has been released for non-commercial academic purposes and is accessible at: https://github.com/xiaoxiu121/coGVHD.git. Comprehensive documentation of all deep learning techniques and software dependencies utilized in this work is provided, with hyperparameter configurations detailed in open-source scripts.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41746-026-02916-8.