Skip to main content
NCBI home page
Diagnostics logoLink to Diagnostics
. 2026 Sep 24;16(19):3100. doi: 10.3390/diagnostics16193100

Diagnostic Concordance Across Text and Image Workflows in Nine Consumer Multimodal Artificial Intelligence Systems for Cutaneous Vascular Anomalies: A Cross-Sectional Audit

Guoyong Wang 1,†, Hui Bi 2,*, Ye Zhang 1,†, Weixin Wang 1, Yingjie Zhu 1, Chaonan Wang 1, Xiaonan Yang 1,*
Editor: Choon Chiat Oh
PMCID: PMC13664238  PMID: 42882255

Abstract

Background/Objectives: The reliability of widely accessible multimodal artificial intelligence systems for rare or complex cutaneous vascular anomalies is uncertain. We evaluated diagnostic concordance and expert-rated management outputs across clinical-summary text, clinical-summary text plus images, and image-only workflows. Methods: In this cross-sectional audit, a purposively assembled, nonconsecutive challenge set of 511 published cases was evaluated with nine consumer web systems in new-conversation first-response sessions (13,797 final score records). Outputs were classified as top-1 concordance with the source-reported diagnosis, differential-list-only inclusion, or omission. Two vascular-anomaly specialists independently rated expected management appropriateness and safety on 5-point scales. Paired comparisons used exact McNemar, Friedman, and paired Wilcoxon tests. Results: Image-only top-1 concordance ranged from 0.4% to 7.0%, with omission rates of 71.0% to 97.8%; corresponding clinical-summary-text ranges were 32.9% to 53.0% and 11.5% to 29.4%. Image-only performance was lower than both text-containing workflows for every system. Text-plus-image workflows were not consistently superior to text alone. Conclusions: These date-stamped workflow contrasts do not estimate real-world diagnostic accuracy or isolate the causal contribution of images. They support preservation of clinical context and specialist oversight.

Keywords: artificial intelligence, cutaneous vascular anomalies, diagnostic concordance, multimodal systems, dermatology, clinical decision support

1. Introduction

Infantile hemangiomas range from self-limited lesions to those requiring early risk-stratified treatment [1,2]. Cutaneous vascular anomalies comprise tumors and malformations with heterogeneous biology, appearance, natural history, and complications [3,4].

Diagnosis and management may require clinical examination, imaging, pathology, or molecular findings. Severe infantile hemangiomas and lymphatic and venous malformations follow distinct care pathways [5,6,7]. Sclerotherapy and emerging PI3K/AKT/mTOR-targeted therapy illustrate disease-specific treatment [8,9]. Rare tumors such as kaposiform hemangioendothelioma may cause life-threatening coagulopathy [10,11]. Sturge–Weber syndrome may couple cutaneous findings with neurologic disease [12]. Complex cases may benefit from multidisciplinary care and rare-disease networks [5,6,7,13].

Multimodal artificial intelligence models can integrate text and images and are being explored for medical applications [14,15]. Large language models have demonstrated broad clinical knowledge and diagnostic capabilities [16,17]. Generalist biomedical and pathology systems further show cross-modal integration [18,19]. However, benchmark performance does not establish reliability in a rare dermatologic domain. Dermatology-specific systems such as SkinGPT-4 and PanDerm demonstrate the potential value of domain-oriented training [20,21].

Consumer systems may also retrieve or recall information from public sources rather than rely only on the submitted input. Retrieval-augmented generation can ground responses in external knowledge, but retrieval noise, domain shift, incomplete provenance, and limited transparency can compromise healthcare reliability [22]. These concerns are especially relevant when evaluation cases originate from published reports that may have been present in training corpora or accessible to platform retrieval.

Systematic reviews show that artificial intelligence performance in dermatology varies by disease class, image type, comparator, and external-validation setting [23,24]. Foundation-model translation introduces additional technical and safety concerns [25]. CLEAR Derm emphasizes transparent reporting of image-based evaluations [26]. Population diversity can affect model performance [27], and patient-level clinical context should be considered in evaluation [28]. A recent JAAD International comparison of GPT-4 with vision and dermatologists supports direct assessment of consumer visual systems [29]. Systematic evidence in rare or complex cutaneous vascular anomalies remains limited. We therefore conducted a date-stamped cross-sectional audit of nine systems across clinical-summary-text, text-plus-image, and image-only workflows. Because information content and prompt instructions differed, comparisons were interpreted as workflow-level contrasts rather than causal estimates of image effects.

2. Materials and Methods

2.1. Study Design and Reporting

This cross-sectional, simulation-based audit used published case reports and consumer web interfaces. Reporting followed STARD-AI where applicable and CLEAR Derm for image-based evaluation [26,30]. Broader reporting considerations drew on TRIPOD + AI [31].

The unit of analysis was the case–system–workflow interaction. The final score-level dataset contained all 511 × 9 × 3 records, with no missing score outcomes or duplicate case–system–workflow records.

2.2. Case Identification and Source Material

PubMed was searched for English-language case reports concerning cutaneous vascular tumors, vascular malformations, and related syndromes. The search had no date restriction and was completed before model testing; the exact final search date was not retained. Reports were eligible when they described a diagnostically challenging or clinically meaningful presentation and contained sufficient history, examination, and ancillary-test information. Guoyong Wang screened the retrieved literature and selected the cases. Reports outside the target clinical scope, reports for which full text could not be obtained, and reports with critical information missing, animal cases, or duplicate content were excluded. The search identified 1223 records, and the final challenge set comprised 511 cases. The number of included source reports and reason-specific exclusion counts could not be verified from the available records. The assembled set also included vascular-appearing mimics and related nonvascular conditions, as described in the post hoc case-spectrum summary. The cases formed a purposive, nonconsecutive challenge set for stress testing. The final diagnosis reported in each source article served as the comparator and was not independently re-established. The study flow is shown in Figure 1.

Figure 1.

Figure 1

Open in a new tab

Search, case assembly, and evaluation workflow. Search records (n = 1223) and analyzed cases (n = 511) are distinct counting units. The cases formed a purposively assembled, nonconsecutive challenge set. H, H + I, and I denote clinical-summary text, clinical-summary text plus images, and images only, respectively.

Clinical-summary-text inputs were assembled from discrete source fields for chief complaint, present illness, past medical history, physical examination, and ancillary examinations. Bibliographic titles, authors, journal names, figure captions or legends, explicit diagnosis fields, dedicated pathology and molecular-result fields, treatment, treatment response, and follow-up were not intentionally supplied. However, no case-level de-cueing audit was retained; diagnostic clues could remain within narrative fields or inside source images or annotations. Images were obtained from figures in the same reports and included clinical photographs and radiologic images. For each case, the same local image set was used across systems whenever accepted. Original format and resolution were preserved when possible, and no systematic cropping, annotation, contrast adjustment, denoising, or lesion marking was performed. Platform-side image processing was not observable.

To describe case spectrum, source-report diagnostic labels were grouped post hoc using a pragmatic label-term rule into four broad categories. This descriptive grouping was not an independent clinical or ISSVA reclassification and was not used for subgroup-performance analysis.

2.3. Systems, Workflows, and Prompting

Nine consumer-facing systems were accessed through web interfaces during January and February 2025: ChatGPT 4o, Claude 3.5 Sonnet, Gemini 2.0, Copilot, Wenxiaoyan, Kimi, Doubao, Tongyi, and Baixiaoying. Research application programming interfaces were not used. The archived records do not consistently preserve whether each 2025 session used a free/basic or paid subscription tier, so account tier cannot be compared. Exact per-interaction timestamps and a valid chronological variable were not uniformly preserved. Consequently, progressive improvement or ‘autolearning’ was not assessed. Platform labels describe the tested interfaces; immutable backend versions, routing, system prompts, temperature, top-p, and seed were unavailable.

Each case was evaluated under three workflows: clinical-summary text (H, the original dataset code), clinical-summary text plus images (H + I), and images only (I). Prompts were fixed across systems within each workflow but were modality-tailored rather than identical across workflows: prompts containing images requested a lesion description before diagnosis, whereas the text-only prompt did not. The exact prompts are reproduced in the Supplementary Methods. Each interaction used a new conversation and one first response; no regeneration, follow-up questioning, or system-specific prompt optimization was used. New conversations and available privacy/training opt-out settings reduced visible within-thread carryover and were intended to prevent training reuse, but they do not exclude cross-chat memory, personalization, account or browser state, platform retrieval, or temporal backend drift.

2.4. Outcomes and Rating

Ye Zhang and Weixin Wang, both physicians with more than 10 years of clinical experience, categorized diagnostic outputs against the source-reported final diagnosis. Each output was assigned one mutually exclusive category: top-1 concordance (the source-reported diagnosis was ranked first), differential-list-only inclusion (it appeared only in the differential diagnosis list), or omission (it was not mentioned). The clinicians used clinical judgment to assess whether synonymous or alternative vascular-anomaly terminology referred to the source-reported diagnosis. Ambiguous rankings and compound diagnostic wording were assessed through the same clinical judgment process. When Ye Zhang and Weixin Wang were uncertain about a diagnostic category or disagreed on its assignment, Xiaonan Yang made the final decision. The final category codes are provided in Supplementary Data S1.

Two vascular-anomaly specialists, Ye Zhang and Weixin Wang, independently rated each management recommendation using prespecified 5-point scales. The original dataset called the first dimension ‘effectiveness’; because no patient outcomes were observed, it is reported here as expected management appropriateness. Higher scores indicated closer alignment with accepted practice and fewer safety concerns. The arithmetic mean of the two ratings was used for workflow comparisons. Masking to system and workflow labels was incompletely documented.

2.5. Statistical Analysis

Proportions were calculated for each diagnostic category by system and workflow. Within each system and diagnostic endpoint, two-sided exact McNemar tests compared H + I with H, I with H, and I with H + I. Paired odds ratios were calculated from discordant pairs; 0.5 was added to both discordant counts when either count was zero for odds-ratio and interval estimation. Confidence intervals are pointwise, unadjusted 95% Wald intervals on the log-odds-ratio scale. The exact McNemar p-values were calculated from the uncorrected discordant counts and Bonferroni-adjusted across the three contrasts within each system-endpoint family. Statistical significance was assessed using the adjusted p-values.

Management ratings were compared across workflows within each system using Friedman tests. When the omnibus result was significant, paired Wilcoxon signed-rank tests without continuity correction were Holm-adjusted within each combination of system and management dimension (three workflow contrasts per family). Kendall W, standardized effect size r (the absolute standardized Wilcoxon statistic divided by the square root of the number of nonzero paired differences), paired mean differences, and percentile bootstrap 95% confidence intervals from 2000 case-level resamples were calculated. Interrater reliability was summarized using Cronbach’s alpha and a two-way mixed-effects, consistency, single-rater intraclass correlation coefficient [ICC(3,1)]. Reliability was summarized overall, by workflow pooling the nine systems, and separately by system pooling the three workflows. No formal sample-size calculation was performed; all 511 included cases were evaluated. Original analyses were performed in R version 4.3.1, and score-derived numerical results were independently checked from the final score-level dataset during revision.

2.6. Ethics and Data Governance

The study used curated case summaries and images derived solely from publicly published reports and involved no patient recruitment, intervention, contact, or access to nonpublic medical records. Public-source images were uploaded as evaluation inputs under available local/privacy controls, with training reuse disabled where the platform offered that option. Commercial-platform retention, secondary processing, cross-session memory, and deletion were not independently verified. Institutional review board approval was not obtained. Source articles and images were treated as third-party materials and are not redistributed; no source patient image or model-response text is reproduced in the manuscript, Supplementary Materials, or public release. The public score dataset uses numeric case identifiers.

3. Results

3.1. Study Sample and Diagnostic Concordance

The 511 cases were tested across nine systems and three workflows, yielding 13,797 final case–system–workflow score records. The post hoc label-based spectrum comprised 241 (47.2%) vascular/endothelial tumors or reactive proliferations, 137 (26.8%) syndromic or complex vascular/overgrowth presentations, 56 (11.0%) vascular malformations or related vessel anomalies, and 77 (15.1%) vascular-appearing mimics or related nonvascular conditions. These post hoc categories summarize the diagnostic spectrum of the challenge set. Diagnostic results are summarized in Table 1. Under the image-only workflow, top-1 concordance ranged from 0.4% to 7.0%, differential-list-only inclusion from 1.8% to 21.9%, and omission of the source-reported diagnosis from 71.0% to 97.8%. Under clinical-summary text, the corresponding ranges were 32.9% to 53.0%, 32.7% to 47.2%, and 11.5% to 29.4%.

Table 1.

Diagnostic concordance across input workflows (N = 511 cases per system–workflow).

System Clinical-Summary Text (H) Clinical-Summary Text + Images (H + I) Images Only (I)
ChatGPT 4o 41.3/47.2/11.5 42.3/45.0/12.7 7.0/21.9/71.0
Claude 3.5 Sonnet 53.0/32.7/14.3 50.7/30.9/18.4 2.0/6.5/91.6
Gemini 2.0 35.8/42.3/21.9 17.6/16.8/65.6 1.4/6.1/92.6
Copilot 32.9/39.1/28.0 20.2/20.2/59.7 2.2/4.7/93.2
Wenxiaoyan 33.1/37.6/29.4 25.2/33.1/41.7 0.8/8.0/91.2
Kimi 32.9/42.1/25.0 17.2/24.7/58.1 0.4/1.8/97.8
Doubao 38.7/36.8/24.5 31.7/30.9/37.4 4.9/12.9/82.2
Tongyi 37.0/43.1/20.0 23.3/40.3/36.4 2.5/8.2/89.2
Baixiaoying 33.7/43.8/22.5 22.9/34.8/42.3 1.8/11.2/87.1
Open in a new tab

Note: Values are percentages in the order top-1 concordance/differential-list-only inclusion/omission of the source-reported diagnosis. H, H + I, and I are original dataset codes. Category percentages may sum to 99.9% or 100.1% because of rounding.

For every system, image-only input had lower top-1 concordance than clinical-summary text (paired odds ratio range, 0.0019–0.069; all adjusted p < 0.001) and higher omission (paired odds ratio range, 22.07–166.00; all adjusted p < 0.001). The same direction was observed for images only versus clinical-summary text plus images (top-1 odds ratio range, 0.0020–0.062; omission odds ratio range, 20.08–125.67; all adjusted p < 0.001).

Exploratory H + I versus H comparisons were heterogeneous because prompt instructions differed. No statistically significant top-1 difference was detected for ChatGPT 4o or Claude 3.5 Sonnet; top-1 concordance was lower under H + I for the other seven systems. Omission was higher under H + I for eight systems, with no statistically significant difference detected for ChatGPT 4o. Figure 2 displays the paired comparisons.

Figure 2.

Figure 2

Open in a new tab

Paired diagnostic-category comparisons from the final score dataset. Forest plots show paired odds ratios and pointwise, unadjusted 95% Wald confidence intervals for I versus H (left) and H + I versus H (right) across nine systems. For each category, an odds ratio below 1 indicates that the category was less frequent in the first-named workflow; an odds ratio above 1 indicates that it was more frequent. Differential-list-only inclusion is interpreted alongside top-1 concordance and omission. Statistical significance is based on exact McNemar tests with within-family Bonferroni adjustment. The vertical dashed line denotes an odds ratio of 1. H, clinical-summary text; H + I, clinical-summary text plus images; I, images only. (A) Top-1 concordance, I versus H; (B) top-1 concordance, H + I versus H; (C) differential-list-only inclusion, I versus H; (D) differential-list-only inclusion, H + I versus H; (E) omission, I versus H; and (F) omission, H + I versus H. Panels (A), (C), and (E) (blue) show I versus H for top-1 concordance, differential-list-only inclusion, and omission, respectively; panels (B), (D), and (F) (teal) show the corresponding H + I versus H comparisons. Squares denote odds ratios, and horizontal bars denote confidence intervals. Blue denotes I versus H, and teal denotes H + I versus H. Squares show paired odds ratios, and horizontal bars show pointwise, unadjusted 95% Wald confidence intervals.

3.2. Management Ratings and Interrater Reliability

Across all 13,797 case–system–workflow interactions, interrater reliability was high for expected management appropriateness (Cronbach’s alpha = 0.947; ICC(3,1) = 0.898) and safety (alpha = 0.915; ICC(3,1) = 0.843). Table 2 presents workflow-stratified estimates pooled across the nine systems (4599 interactions per workflow); ICC(3,1) ranged from 0.727 to 0.912 for appropriateness and from 0.746 to 0.848 for safety. In a separate system-stratified analysis pooled across the three workflows (1533 interactions per system), ICC(3,1) ranged from 0.768 to 0.935 for appropriateness and from 0.807 to 0.862 for safety.

Table 2.

Interrater reliability of expected management appropriateness and safety ratings.

Category N Appropriateness Alpha Appropriateness ICC(3,1) Safety Alpha Safety ICC(3,1)
Overall 13,797 0.947 0.898 0.915 0.843
Clinical-summary text (H) 4599 0.842 0.727 0.899 0.816
Clinical-summary text + images (H + I) 4599 0.952 0.909 0.918 0.848
Images only (I) 4599 0.954 0.912 0.855 0.746
Open in a new tab

Note: Alpha indicates Cronbach’s alpha. ICC(3,1) is a two-way mixed-effects, consistency, single-rater coefficient. N denotes the number of case–system–workflow interactions. Overall estimates pool all nine systems and three workflows (511 × 9 × 3 = 13,797 interactions); each workflow-specific estimate pools all nine systems (511 × 9 = 4599 interactions). Each interaction was rated independently by two specialists. The separate system-stratified analysis pools all three workflows within each system (511 × 3 = 1533 interactions).

Friedman tests identified workflow differences in appropriateness and safety for every system (all p < 0.001; Table 3). Clinical-summary text scores exceeded image-only scores for appropriateness by 0.924 to 2.169 points and for safety by 0.494 to 0.841 points (all Holm-adjusted p < 0.001). Differences between H and H + I varied by system. For ChatGPT 4o, mean appropriateness/safety ratings were 4.03/4.14 under H, 4.03/4.14 under H + I, and 3.11/3.30 under I; no statistically significant difference was detected between H and H + I.

Table 3.

Expected management appropriateness and safety ratings by system and input workflow.

System H: Appropriateness/Safety H + I: Appropriateness/Safety I: Appropriateness/Safety Appropriateness p; W Safety p; W
ChatGPT 4o 4.03 ± 0.76/4.14 ± 0.78 4.03 ± 0.77/4.14 ± 0.78 3.11 ± 0.70/3.30 ± 0.55 p < 0.001; 0.416 p < 0.001; 0.382
Claude 3.5 Sonnet 4.17 ± 0.83/4.28 ± 0.79 4.05 ± 0.90/4.21 ± 0.81 2.62 ± 0.65/3.53 ± 0.51 p < 0.001; 0.678 p < 0.001; 0.249
Gemini 2.0 3.85 ± 0.81/3.96 ± 0.81 2.37 ± 1.52/3.44 ± 0.74 2.02 ± 1.08/3.12 ± 0.35 p < 0.001; 0.499 p < 0.001; 0.364
Copilot 3.79 ± 0.82/3.90 ± 0.82 2.52 ± 1.50/3.51 ± 0.75 1.91 ± 1.04/3.21 ± 0.43 p < 0.001; 0.498 p < 0.001; 0.227
Wenxiaoyan 3.78 ± 0.81/3.89 ± 0.80 3.29 ± 1.17/3.68 ± 0.79 2.72 ± 0.74/3.21 ± 0.39 p < 0.001; 0.381 p < 0.001; 0.217
Kimi 3.81 ± 0.80/3.90 ± 0.80 2.57 ± 1.42/3.54 ± 0.73 1.64 ± 0.73/3.41 ± 0.49 p < 0.001; 0.628 p < 0.001; 0.109
Doubao 3.89 ± 0.83/4.01 ± 0.82 3.45 ± 1.23/3.80 ± 0.84 2.78 ± 0.88/3.26 ± 0.51 p < 0.001; 0.388 p < 0.001; 0.266
Tongyi 3.88 ± 0.82/3.98 ± 0.80 3.41 ± 1.06/3.70 ± 0.76 2.68 ± 0.85/3.20 ± 0.43 p < 0.001; 0.446 p < 0.001; 0.305
Baixiaoying 3.83 ± 0.80/3.92 ± 0.80 3.21 ± 1.21/3.62 ± 0.75 2.59 ± 0.84/3.26 ± 0.43 p < 0.001; 0.473 p < 0.001; 0.208
Open in a new tab

Note: Values are mean ± standard deviation of the arithmetic mean of two independent ratings. The original data field labeled ‘effectiveness’ is reported as expected management appropriateness. p and Kendall W are from within-system Friedman tests across H, H + I, and I.

4. Discussion

This audit found a large and consistent performance gap between image-only workflows and workflows containing published clinical-summary text. Across all nine systems, image-only first responses frequently omitted the diagnosis reported in the source case and received lower offline expert ratings for expected management appropriateness and safety. The direction and magnitude of this finding were more consistent than differences among individual platforms.

The text-containing workflows included condensed clinical history, examination, and ancillary-test information, while images varied in modality and quality. Image-containing prompts also requested a lesion description. Accordingly, H versus H + I estimates a workflow-level contrast; the design does not separate the contribution of images from information content and prompting. The evaluated H + I workflow showed no consistent incremental benefit over H.

The image-only finding has clinical relevance. Vascular tumors and malformations may share surface color or morphology while differing in flow, deep extent, and biological behavior [3,4]. Rare tumors and syndromic disease may carry systemic consequences not evident from one image [10,11,12]. A plausible response can therefore omit a consequential diagnosis. Disease-specific pathways require clinical history and imaging [5,6,7]. Specialist review is multidisciplinary [13]. Domain-specific models may encode visual features differently [20,21], but image-set variation, metadata, and population diversity limit extrapolation to consumer platforms [27,28]. A JAAD International comparison similarly supports task-specific benchmarking [29].

Because the cases originated from public reports, memorization and retrieval may contribute to diagnostic concordance. The consumer interfaces did not expose whether a response relied on internal parametric memory, web search, a retrieval layer, or solely on the submitted prompt. Retrieval-augmented generation can improve grounding when authoritative evidence is retrieved, yet healthcare reviews also identify retrieval noise, domain shift, incomplete provenance, and limited pipeline transparency as reliability risks [22]. Concordance may therefore reflect a combination of clinical reasoning, prior exposure, and retrieval. Future audits should use prospectively assembled or otherwise nonpublic cases, predeclare and control retrieval states, retain citations and retrieval logs where available, and evaluate retriever and generator errors separately.

Management results followed the same workflow pattern. Ratings reflected expert-perceived appropriateness and safety, not patient outcomes. Missing clinical details may obscure disease-specific pathways [5,6,7], and selection of sclerotherapy or targeted therapy depends on subtype and biology [8,9]. Multidisciplinary input may be required [13]. High interrater reliability indicates consistency between the two specialists’ management ratings.

Generalizability is limited by the purposive, nonconsecutive selection of published challenge cases and associated selection and spectrum bias; this design does not estimate real-world diagnostic accuracy. Reason-specific exclusion counts were unavailable. Source diagnoses were not independently re-established, and the post hoc spectrum grouping was label-based. Inputs differed in clinical information, prompts, image modality and quality, and potential diagnostic cues. The study did not include a clinician comparator or prospective patient outcomes.

Run order, timestamps, and masking were incompletely documented. Immutable backend versions, routing, sampling settings, and platform-side image processing were unavailable, and single-response evaluation did not assess repeatability. Complete response text and contemporaneous adjudication records are not supplied, so output-to-score assignments cannot be independently re-adjudicated. Platform retention, secondary processing, and deletion of uploaded public-source images could not be independently verified.

Future studies should combine prospectively assembled cases with information-matched prompts, version-controlled systems, controlled retrieval, randomized run order, repeated generations, clinician comparators, and prespecified governance. Preserving prompts, outputs, adjudication records, and image metadata would improve auditability. Reporting can draw on CLEAR Derm, STARD-AI, and relevant TRIPOD + AI principles [26,30,31].

5. Conclusions

In conclusion, image-only workflows in this date-stamped audit showed substantially lower concordance with source-reported diagnoses and lower expert-rated management quality than workflows containing information-rich clinical-summary text. The findings support preservation of clinical context and specialist oversight, while highlighting the need for prospectively assembled, version-controlled, information-matched, and fully auditable studies.

Acknowledgments

During manuscript preparation and revision, OpenAI ChatGPT/Codex assisted with language editing, text organization, reviewer-response drafting, reference-format checks, and the development and checking of code for statistical verification and figure preparation. The authors are responsible for verifying the assisted work and for the final content of the article.

Abbreviations

AI, artificial intelligence; CI, confidence interval; CLEAR Derm, Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology; H, clinical-summary text (original dataset code); H + I, clinical-summary text plus images (original dataset code); I, images only (original dataset code); ICC, intraclass correlation coefficient; OR, odds ratio; STARD-AI, Standards for Reporting Diagnostic Accuracy Studies Using Artificial Intelligence; TRIPOD + AI, Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis plus artificial intelligence.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/diagnostics16193100/s1. Supplementary Methods (eMethods 1–8), including Table S1, “Broad post hoc distribution of source-report diagnostic labels in the 511-case challenge set”; Supplementary Data S1 (final diagnostic and management scoring results).

diagnostics-16-03100-s001.zip (852.5KB, zip)

Author Contributions

Conceptualization, G.W. and X.Y.; methodology, G.W. and X.Y.; formal analysis, G.W.; investigation, G.W., Y.Z. (Ye Zhang), W.W., Y.Z. (Yingjie Zhu) and C.W.; data curation, G.W., Y.Z. (Ye Zhang), W.W., Y.Z. (Yingjie Zhu) and C.W.; validation, G.W., Y.Z. (Ye Zhang), W.W., X.Y. and H.B. (second-round statistical-result checks); visualization, G.W. and X.Y.; resources, X.Y.; funding acquisition, X.Y.; project administration, G.W. and X.Y.; supervision, X.Y.; writing—original draft preparation, G.W.; writing—review and editing, G.W., H.B., Y.Z. (Ye Zhang), W.W., Y.Z. (Yingjie Zhu), C.W. and X.Y., with H.B. leading the second-round academic revision and contributing to interpretation of study findings, methodological clarification, and revision of the Discussion. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

No institutional review board approval was obtained. This study used information and images from publicly published reports and involved no patient recruitment, intervention, contact, or access to nonpublic medical records. Source articles and images are not redistributed in this article or its Supplementary Materials.

Informed Consent Statement

Not applicable. This study did not enroll or interact with participants, and no identifiable patient information or source patient images are reproduced in this article or its Supplementary Materials.

Data Availability Statement

The final de-identified diagnostic and management scores, scoring rubrics, and statistical materials are openly available in the Ydata-Vasc511 version 1.1.0 release at https://github.com/wang177777/Ydata-Vasc511/releases/tag/v1.1.0 (accessed on 10 August 2026). The score-only materials support statistical recalculation but do not permit independent re-adjudication of the original output-to-score assignments. Source patient images, full-text articles, platform-history records, complete model-response text, and contemporaneous adjudication records are not included in the manuscript, revision materials, or public release.

Conflicts of Interest

The authors declare no conflicts of interest.

Funding Statement

This research was funded by the National Clinical Key Specialty Construction Project, grant number 23003; the Plastic Medicine Research Fund of the Chinese Academy of Medical Sciences, grant number 2024-ZX-1-01; the Special Research Fund for Plastic Surgery Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, grant numbers YSZ2024CG007 and 2025-12M-C&T-B-067; and the Beijing Natural Science Foundation, grant number L256048. Additional support for student research-assistance costs and the article processing charge is provided by the Institutional Research Fund of the Plastic Surgery Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College (grant number YSZ2024XZ001; H.B.) and the Fundamental Research Funds for the Central Universities (Faculty–Student Joint Research Project, grant number 3332026021; G.W. and C.W.). The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

References

  • 1.Rodríguez Bandera A.I., Sebaratnam D.F., Wargon O., Wong L.F. Infantile hemangioma. Part 1: Epidemiology, pathogenesis, clinical presentation and assessment. J. Am. Acad. Dermatol. 2021;85:1379–1392. doi: 10.1016/j.jaad.2021.08.019. [DOI] [PubMed] [Google Scholar]
  • 2.Sebaratnam D.F., Rodríguez Bandera A.L., Wong L.F., Wargon O. Infantile hemangioma. Part 2: Management. J. Am. Acad. Dermatol. 2021;85:1395–1404. doi: 10.1016/j.jaad.2021.08.020. [DOI] [PubMed] [Google Scholar]
  • 3.Goldenberg D.C., Vikkula M., Penington A., Blei F., Schultze-Kool L., Wassef M., Frieden I.J., ISSVA Vascular Anomalies Classification Group Updated classification of vascular anomalies: A living document from the International Society for the Study of Vascular Anomalies Classification Group. J. Vasc. Anom. 2025;6:e113. doi: 10.1097/JOVA.0000000000000113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Wang M.X., Kamel S., Elsayes K.M., Guillerman R.P., Habiba A., Heng L., Revzin M., Mellnick V., Iacobas I., Chau A. Vascular anomaly syndromes in the ISSVA classification system: Imaging findings and role of interventional radiology in management. Radiographics. 2022;42:1598–1620. doi: 10.1148/rg.210234. [DOI] [PubMed] [Google Scholar]
  • 5.Diociaiuti A., Baselga E., Boon L.M., Dompmartin A., Dvorakova V., El Hachem M., Gasparella P., Haxhija E., Ghaffarpour N., Kyrklund K., et al. The VASCERN-VASCA working group diagnostic and management pathways for severe and/or rare infantile hemangiomas. Eur. J. Med. Genet. 2022;65:104517. doi: 10.1016/j.ejmg.2022.104517. [DOI] [PubMed] [Google Scholar]
  • 6.Ghaffarpour N., Baselga E., Boon L.M., Diociaiuti A., Dompmartin A., Dvorakova V., El Hachem M., Gasparella P., Haxhija E., Kyrklund K., et al. The VASCERN-VASCA working group diagnostic and management pathways for lymphatic malformations. Eur. J. Med. Genet. 2022;65:104637. doi: 10.1016/j.ejmg.2022.104637. [DOI] [PubMed] [Google Scholar]
  • 7.Dompmartin A., Baselga E., Boon L.M., Diociaiuti A., Dvorakova V., El Hachem M., Gasparella P., Haxhija E., Ghaffarpour N., Kyrklund K., et al. The VASCERN-VASCA Working Group diagnostic and management pathways for venous malformations. J. Vasc. Anom. 2023;4:e064. doi: 10.1097/JOVA.0000000000000064. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Cao J., Liu J., Zhang X., Wang Z. A systematic review and network meta-analysis of the effectiveness of sclerotherapy for venous malformation. J. Vasc. Surg. Venous Lymphat. Disord. 2023;11:210–218.e3. doi: 10.1016/j.jvsv.2022.08.004. [DOI] [PubMed] [Google Scholar]
  • 9.Zheng Y.Y., Hua C., Lin X.X. PI3K/AKT/mTOR axis in vascular malformations: From molecular insights to targeted clinical trials. Orphanet J. Rare Dis. 2025;20:624. doi: 10.1186/s13023-025-04115-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Atherton K., Hinen H. Vascular anomalies: Other vascular tumors. Dermatol. Clin. 2022;40:401–423. doi: 10.1016/j.det.2022.06.011. [DOI] [PubMed] [Google Scholar]
  • 11.Zhao J., Yuan J.J., Dong C.X., Zhang X., Li C., Sun Q., Wu G. Kaposiform haemangioendothelioma: Ultrasonographic features and risk factors for the Kasabach-Merritt phenomenon. Ultraschall Med. 2025;46:278–284. doi: 10.1055/a-2421-6047. [DOI] [PubMed] [Google Scholar]
  • 12.Yeom S., Comi A.M. Updates on Sturge-Weber syndrome. Stroke. 2022;53:3769–3779. doi: 10.1161/STROKEAHA.122.038585. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Schmidt J.L., Mohan V.C., Appel R., Rosenberg T.L., Ching J.A. Multidisciplinary team care in plastic surgery: Vascular anomalies. Semin. Plast. Surg. 2025;39:82–88. doi: 10.1055/s-0045-1808272. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Moor M., Banerjee O., Abad Z.S.H., Krumholz H.M., Leskovec J., Topol E.J., Rajpurkar P. Foundation models for generalist medical artificial intelligence. Nature. 2023;616:259–265. doi: 10.1038/s41586-023-05881-4. [DOI] [PubMed] [Google Scholar]
  • 15.AlSaad R., Abd-alrazaq A., Boughorbel S., Ahmed A., Renault M.-A., Damseh R., Sheikh J. Multimodal large language models in health care: Applications, challenges, and future outlook. J. Med. Internet Res. 2024;26:e59505. doi: 10.2196/59505. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Singhal K., Azizi S., Tu T., Mahdavi S.S., Wei J., Chung H.W., Scales N., Tanwani A., Cole-Lewis H., Pfohl S., et al. Large language models encode clinical knowledge. Nature. 2023;620:172–180. doi: 10.1038/s41586-023-06291-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Kanjee Z., Crowe B., Rodman A. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. JAMA. 2023;330:78–80. doi: 10.1001/jama.2023.8288. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Tu T., Azizi S., Driess D., Schaekermann M., Amin M., Chang P.-C., Carroll A., Lau C., Tanno R., Ktena I., et al. Towards Generalist Biomedical AI. NEJM AI. 2024;1:AIoa2300138. doi: 10.1056/AIoa2300138. [DOI] [Google Scholar]
  • 19.Lu M.Y., Chen B., Williamson D.F.K., Chen R.J., Zhao M., Chow A.K., Ikemura K., Kim A., Pouli D., Patel A., et al. A multimodal generative AI copilot for human pathology. Nature. 2024;634:466–473. doi: 10.1038/s41586-024-07618-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Zhou J., He X., Sun L., Xu J., Chen X., Chu Y., Zhou L., Liao X., Zhang B., Afvari S., et al. Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4. Nat. Commun. 2024;15:5649. doi: 10.1038/s41467-024-50043-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Yan S., Yu Z., Primiero C., Vico-Alonso C., Wang Z., Yang L., Tschandl P., Hu M., Ju L., Tan G., et al. A multimodal vision foundation model for clinical dermatology. Nat. Med. 2025;31:2691–2702. doi: 10.1038/s41591-025-03747-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Neha F., Bhati D., Shukla D.K. Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review. AI. 2025;6:226. doi: 10.3390/ai6090226. [DOI] [Google Scholar]
  • 23.Choy S.P., Kim B.J., Paolino A., Tan W.R., Lim S.M.L., Seo J., Tan S.P., Francis L., Tsakok T., Simpson M., et al. Systematic review of deep learning image analyses for the diagnosis and monitoring of skin disease. npj Digit. Med. 2023;6:180. doi: 10.1038/s41746-023-00914-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Salinas M.P., Sepúlveda J., Hidalgo L., Peirano D., Morel M., Uribe P., Rotemberg V., Briones J., Mery D., Navarrete-Dechent C. A systematic review and meta-analysis of artificial intelligence versus clinicians for skin cancer diagnosis. npj Digit. Med. 2024;7:125. doi: 10.1038/s41746-024-01103-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Gui H., Omiye J.A., Chang C.T., Daneshjou R. The promises and perils of foundation models in dermatology. J. Investig. Dermatol. 2024;144:1440–1448. doi: 10.1016/j.jid.2023.12.019. [DOI] [PubMed] [Google Scholar]
  • 26.Daneshjou R., Barata C., Betz-Stablein B., Celebi M.E., Codella N., Combalia M., Guitera P., Gutman D., Halpern A., Helba B., et al. Checklist for evaluation of image-based artificial intelligence reports in dermatology: CLEAR Derm consensus guidelines from the International Skin Imaging Collaboration Artificial Intelligence Working Group. JAMA Dermatol. 2022;158:90–96. doi: 10.1001/jamadermatol.2021.4915. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Daneshjou R., Vodrahalli K., Novoa R.A., Jenkins M., Liang W., Rotemberg V., Ko J., Swetter S.M., Bailey E.E., Gevaert O., et al. Disparities in dermatology AI performance on a diverse, curated clinical image set. Sci. Adv. 2022;8:eabq6147. doi: 10.1126/sciadv.abq6147. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Rotemberg V., Kurtansky N., Betz-Stablein B., Caffery L., Chousakos E., Codella N., Combalia M., Dusza S., Guitera P., Gutman D., et al. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Sci. Data. 2021;8:34. doi: 10.1038/s41597-021-00815-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Block B.R., Powers C.M., Chang A., Campbell C., Piontkowski A.J., Orloff J., Levoska M.A., Cices A., Fenner J., Talia J., et al. Diagnostic performance of generative pretrained transformer-4 with vision technology versus board-certified dermatologists: A comparative analysis using dermoscopic and clinical images. JAAD Int. 2025;18:142–144. doi: 10.1016/j.jdin.2024.10.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Sounderajah V., Guni A., Liu X., Collins G.S., Karthikesalingam A., Markar S.R., Golub R.M., Denniston A.K., Shetty S., Moher D., et al. The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nat. Med. 2025;31:3283–3289. doi: 10.1038/s41591-025-03953-8. [DOI] [PubMed] [Google Scholar]
  • 31.Collins G.S., Moons K.G.M., Dhiman P., Riley R.D., Beam A.L., Van Calster B., Ghassemi M., Liu X., Reitsma J.B., van Smeden M., et al. TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. doi: 10.1136/bmj-2023-078378. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

diagnostics-16-03100-s001.zip (852.5KB, zip)

Data Availability Statement

The final de-identified diagnostic and management scores, scoring rubrics, and statistical materials are openly available in the Ydata-Vasc511 version 1.1.0 release at https://github.com/wang177777/Ydata-Vasc511/releases/tag/v1.1.0 (accessed on 10 August 2026). The score-only materials support statistical recalculation but do not permit independent re-adjudication of the original output-to-score assignments. Source patient images, full-text articles, platform-history records, complete model-response text, and contemporaneous adjudication records are not included in the manuscript, revision materials, or public release.

Articles from Diagnostics are provided here courtesy of Multidisciplinary Digital Publishing Institute (MDPI)

RESOURCES