Figure 2: Zero-shot findings classification.

(a) Depicts how zero-shot classification is performed where text embeddings from disease presence prompts and disease absence prompts are compare to the image embedding. (b) We compare the performance of OpenCLIP,⁵³ BioMedCLIP,⁵⁴ Merlin on an internal dataset, and Merlin without radiology report splitting. We further evaluate Merlin on an external clinical dataset and the VerSe⁵⁵ external fracture detection dataset. (c) Performance of BioMedCLIP, Merlin, and Merlin without report splitting on the internal dataset, as well as Merlin on the external dataset, across 30 findings assessed on abdominal CT scans. (d) Merlin zero-shot classification performance improves with increasing pretraining dataset size. (e) An ablation study across various aspects of Merlin’s pretraining strategy. “Rpt.” is shorthand for “report” and indicated training with radiology reports only. Staged (Stg.) refers to performing weakly supervised training with EHR in a first training stage and then training with radiology reports in a second stage. This is in contrast to multi-task learning (MTL) where EHR and radiology reports are used for training simultaneously.