Quilt-1M: One Million Image-Text Pairs for Histopathology

Abstract

Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology, has halted comparable progress. To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering 1,087 hours of valuable educational histopathology videos from expert clinicians. From YouTube, we curate Quilt: a large-scale vision-language dataset consisting of 768,826 image and text pairs. Quilt was automatically curated using a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around 200K samples. We combine Quilt with datasets from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: Quilt-1M, with 1M paired image-text samples, marking it as the largest vision-language histopathology dataset to date. We demonstrate the value of Quilt-1M by fine-tuning a pre-trained CLIP model. Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new histopathology images across 13 diverse patch-level datasets of 8 different sub-pathologies and cross-modal retrieval tasks².

1. Introduction

Whole-slide histopathology images are dense in information, and even individual image patches can hold unique, complex patterns critical for tissue characterization. Summarizing this information into a single label is an oversimplification that fails to capture the complexity of the field, which covers thousands of evolving disease sub-types [58]. This highlights the need for more expressive, dense, interconnected representations beyond the reach of a singular categorical label. As such, natural language descriptions can provide this comprehensive signal, linking diverse features of histopathology sub-patch structures [20, 24].

If there were a large-scale vision-language dataset for histopathology, researchers would be able to leverage the significant advancements in self-supervised vision and language pre-training to develop effective histopathology models [48]. Unfortunately, there is a significant scarcity of comprehensive datasets for histopathology. Notable open-source contributions have been made with datasets like ARCH [20] and OpenPath [24]. Yet, these sources are still somewhat limited due to their size, as the former has only ≈ 8K samples and the latter (the largest histopathology vision-language dataset to date) has about 200K samples. Although recent efforts (e.g. PMC-15M [70]) curated 15M image-text pairs across a variety of different biomedical domains from Pubmed [50], whether their samples are specific to histopathology remains ambiguous; worse, their dataset is not openly available.

To address the need for a large-scale vision-language dataset in histopathology, we introduce Quilt: containing 419, 780 images aligned with 768, 826 text pairs. We draw on the insight that publicly available educational YouTube histopathology content represents an untapped potential. We curate Quilt using 1, 087 hours of valuable educational histopathology videos from expert pathologists on YouTube. To extract aligned image and text pairs from the videos, we utilize a mixture of models: large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition. Quilt does not overlap with any current open-access histopathology data sources. This allows us to merge our dataset with other open-source datasets available. Therefore, to create an even larger and more diverse dataset, we combine Quilt with data from other sources, such as Twitter, research papers, and the Internet, resulting in Quilt-1M. The larger Quilt-1M contains one million image-text pairs, making it the largest public vision-language histopathology dataset to date.

Using Quilt and Quilt-1M, we finetune vision-language models using a contrastive objective between the two modalities. We extensively evaluate it on 13 external histopathology datasets taken across different sub-pathologies. We report zero-shot classification, linear probe, and image-to-text and text-to-image retrieval tasks. Against multiple recently proposed baselines (CLIP [48], PLIP [24], and BiomedCLIP [70]), models trained with Quilt-1M outperform all others. Our ablations identify the importance of Quilt.

Quilt offers three significant advantages: First, Quilt does not overlap with existing data sources; it ensures a unique contribution to the pool of available histopathology knowledge. Second, its rich textual descriptions extracted from experts narrating within educational videos provide more expressive, dense interconnected information. Last, the presence of multiple sentences per image fosters diverse perspectives and a comprehensive understanding of each histopathological image. We hope that both computer scientists and histopathologists will benefit from Quilt’s potential.

2. Related work

We built upon a growing literature applying self-supervised learning and other machine learning methods to medical image understanding.

Machine learning for histopathology.

Early machine learning work in computational pathology primarily relied on weakly-supervised learning, with each whole-slide image (WSI) receiving a single label. The limited nature (single label to many patches) has produced sub-optimal models [12, 26]. Lately, a self-supervised learning approach, which learns useful representations from unlabeled data, has shown some success [26, 13, 12]. Most of this work has been unimodal. They use image augmentations similar to those used for natural images [14], mostly differing by way of consciously injecting domain knowledge. For example, they leverage the compositional nature of H&E stain information of whole-slide images [26], or inject hierarchical morphological information at different magnifications [13], or combine with other modalities like genomic features [12] or with descriptive text [20]. When text data is used, the objectives similarly use augmentations seen in natural language [52]. By contrast, we explore self-supervised mechanisms that learn better histopathology information representations that go beyond a single label, aided by language descriptions.

Medical vision-language datasets.

Learning vision-language representations demands a large dataset of images aligned with descriptive text, a resource that is notably lacking in histopathology. The MIMIC-CXR-JPG v2.0.0 dataset [29], for example, consists of de-identified hospital-sourced chest radiographs and reports. For histopathology, The Cancer Genome Atlas³ provides de-identified PDF-reports for a limited number of WSIs. Despite this resource, the enormous size of this data (reaching up to 120, 000² pixels) makes processing challenging, limiting its use to a small number of focused studies [41]. A majority of medical vision-language datasets are concentrated in the radiology sub-domain, due to the relatively straightforward process of collecting validated multimodal data [29]. Many models are trained on a subset of PubMed [50] or comparable radiology datasets [71, 23, 18, 45]. PMC-15M [70], a recent subset of PubMed not specific to histopathology, was used to train multiple models. While the models themselves are public, PMC-15M is not, making it hard to determine what portion of it is histopathology-relevant.

Vision-language pairs on histopathology.

One of the first histopathology vision-language datasets, ARCH, contains only 7, 614 accessible image-text pairs [20, 22]. Later on, [24] released OpenPath, a dataset of 200K image-text pairs extracted from Twitter. This was the largest histopathology dataset until Quilt-1M.

Video data for self-supervision.

Numerous recent studies have started to tap into video data. For instance, millions of publicly accessible YouTube videos were used to train a vision-language model [68, 69]. Similarly, a causal video model was trained by using sequential gaming videos [6]. Localized narratives [61, 46] provide another example of dense, interconnected supervision for a single image. Despite the untapped potential of video content, video often yields noisier datasets compared to static sources. Recently, the enhanced capabilities of automatic speech recognition models streamlined the curation of large-scale cleaner datasets from videos [68, 6, 70]. Furthermore, the growing versatility of large language models has shown promise as data annotators, information extractors [34, 62, 15, 21], text correctors [66], and as tools for medical information extraction and reasoning [1, 59].

3. Curating Quilt: Overview

Creating a vision-language dataset from videos is a significant undertaking, as not all videos are suitable for our pipeline. Many either lack voiced audio, are not in English, fail to contain medically relevant content, or have insufficient medical relevance—for example, videos that present static images of histopathology content on a slide deck, or those that briefly cover histopathology images in pursuit of a different objective. Conventional automatic speech recognition (ASR) systems also struggle with the specialized requirements of histopathology transcription, necessitating a non-trivial solution. The de-noising of text and image modalities adds further complexity as the videos are typically conversational and, therefore, inherently noisy. Instructors pan and zoom at varying speeds, recording a mix of relevant and irrelevant histopathological visual content in their videos. As such, trivially extracting frames at static intervals fails to capture the data appropriately. To collect Quilt we trained models and handcrafted algorithms that leverage the nuances in the instructors’ textual and visual behavior, ensuring accurate collection and alignment of both modalities.

3.1. Quilt: Collecting medical image and text pairs from YouTube

Our proposed dataset curation pipeline involves (1) gathering channel and video data covering the histopathology domain, (2) filtering videos based on a certain “narrative style”, (3) extracting and denoising image and text modalities from videos using various models, tools, and algorithms, (4) postprocessing denoised text by LLMs to extract medical text and finally, (5) splitting and aligning all modalities for curating the final vision-language pre-training (VLP) data. See Figure 1 (and A.1 in the Appendix) for a detailed overview of the pipeline.

Figure 1: Overview of Quilt curation pipeline.

We identify relevant histopathology YouTube videos in Search. For Image extraction, we find and de-noise histopathology frames using trained models. In Text section, we rely on a conventional Automatic Speech Recognition (ASR) model and leverage Unified Medical Language System (UMLS) and large language models (LLMs) for post-processing and ASR error correction. Relevant sub-pathology, medical and region-of-interest (ROI) text are extracted using an LLM. Finally, domain-specific algorithms are used to Pair images and text, eliminating duplicates to yield Quilt, a richly annotated image-text dataset for histopathology.

Collecting representative channels and videos.

Our pipeline begins by searching for relevant channels and video ids on YouTube, focusing on the domain of histopathology. Using keywords spanning 18 sub-pathology fields (see section A.4 in the Appendix), we search among channels before searching for videos to expedite discovery, considering that video searches are time-consuming and the APIs pose limitations on numerous requests [68]. Channels with subscriber count ≥ 300K are excluded to avoid large general science channels, as educational histopathology channels often have fewer subscribers. We then download low-resolution versions of all identified videos.

Filtering for narrative-style medical videos.

For each video within each channel, we exclude videos that are shorter than 1 minute, non-voiced, or have non-English audio. For videos meeting these heuristics, two decisions are made:

1. Do they have the required medical content, i.e., histopathology image-text pairs?

2. If so, are they in narrative style – videos wherein the presenter(s) spend a significant time panning and zooming on the WSI, while providing vocal descriptions of image content?

For (A) we automatically identify the relevant videos by extracting keyframes from a video. These keyframes are automatically extracted using FFmpeg ⁴, marking the beginning or end of a scene (frames containing significant visual changes). The software requires a threshold that determines the minimum amount of visual change required to trigger a keyframe. Through experimentation, we set different thresholds for various video durations, with smaller thresholds for longer videos. Next, we train and use an ensemble of three histopathology image classifiers to identify videos with histopathology images (See section A.3 in the Appendix).

For (B), in which we identify narrative-style videos, we randomly select keyframes predicted to be histopathology. For each such selected frame, we extract the next three histopathology key-frames and compute the cosine similarity between the selected frame and each of the subsequent three frames. If all three have similarity scores ≥ a preset threshold of 0.9, we count it as a narrative streak. A video is identified as narrative style if at least 10% of the selected frames exhibit a narrative streak. Consequently, we download all narrative-style videos at high-resolution. Narrative-style videos typically cover WSIs at various magnifications, hence, we train a tissue-image-magnification classifier to predict the following three scales: {(1 – 10)x, (> 10 – 20)x, (> 20)x}. This provides relevant metadata for downstream objectives.

Text Extraction using ASR and text denoising.

The high costs associated with private medical ASR APIs ⁵ necessitated the use of a more conventional ASR model: Whisper [49]. As anticipated, this model often misinterprets medical terms, thus requiring the use of post-processing algorithms to minimize its error rates.

We propose a four-step text de-noising and quality control pipeline: i) We utilize the Rake keyword extraction algorithm to extract keywords or key-phrases up to four words and refine them by eliminating stopwords [51]. ii) We then cross-check each refined entry against UMLS [7] using the SciSpacy entity linking package [43]. If an entry is not found within UMLS, we check for misspelled words within the entry using a spell-checking algorithm⁶, instantiated with a specialized list of histopathology terms curated from various histopathology ontology labels and definitions. iii) With this probable list of misspelled keywords, we condition and prompt the LLM with examples to correct the misspelled entry within its context (sentence), and secondly, we task the LLM with identifying additional unconditioned errors/misspelled entries. For both, we leverage a set of manually curated examples to prompt the LLM in-context. For more examples and failure cases, see Table 11 and Figure 9 in the Appendix. iv) Finally, to de-noise the text, we resolve the output mapping of incorrect ↦ correct entries by verifying the corrected words against UMLS and our curated list of histopathology words/phrases. Entries that pass this double-validation process are used to replace the initial noisy transcription. Leveraging domain-specific databases to extract the text and filter out noise allows us to bypass the correction of repetition errors and filler words, such as ’ah’, ’uhm’, ’the’, etc. in tandem, using LLMs allows us to concentrate on correcting medically-relevant misspelled words, rather than correcting non-medically-relevant terms.

From the ASR-corrected text, we extract medical text which describes the image(s) as a whole. Also, when the speaker describes/gestures at visual regions-of-interest through statements like “look here …”, we extract the text entity being described as ROI text. To filter relevant medical text and ROI text from the ASR-corrected text, we utilize LLMs (see Figure 9 in Appendix), a decision rooted in a few compelling reasons: 1) Curating pre-training datasets at a scale that can tolerate higher levels of noise, LLMs are more cost-effective than expert-human (medical) labor. 2) The task does not require LLMs to generate new information but instead they discriminate useful versus irrelevant signals, serving to improve the signal-to-noise ratio of the data. To extract relevant text, we prompt LLMs to filter out all non-medically relevant text, providing context as necessary. See Figure 2 for some example image-text pairs. Lastly, we instruct the LLMs to refrain from introducing any new words beyond the corrected noisy text and set the model’s temperature to zero. Finally, we use LLMs to categorize our videos into one of the 18 identified sub-pathology classes. Similar to the previous tasks, this categorization is done by conditioning with a few examples and prompting the LLM to predict the top three possible classes given the text. More details, prompts, and additional examples are presented in Figure 12 within the Appendix.

Figure 2: Quilt examples.

Input is the corrected ASR caption for the representative image. Output are the medical and ROI extracted text(s) paired with the image (see Section 3.1). In histopathology, understanding tissue characteristics often involves views from varying magnification levels. Thus, in Quilt we estimate an image’s magnification (indicated by the relative size of the microscope icon).

Image frame extraction and denoising.

For each video, we employ a similar method to that described in Filtering for narrative-style medical videos subsection to extract histopathology key-frames; our method leverages these frames’ times $t$ as beacons to break the entire video into time-intervals called chunks from which to extract representative image(s). Next, we extract the median image (pixel-space) of stable (static) frames in each chunk if they exists, else we de-duplicate the histopathology keyframes (beacons of the chunk). In essence, we use the extracted histopathology scene frames as guides for data collection, exploiting the human tendency in educational videos to pause narration during explanation, and we extract the relevant frame(s).

Aligning both modalities.

For each narrative-style video, we perform the following steps to align image and text modalities: First, we compute histopathology time chunks denoted as $\left[\left(t_1,t_2\right),\left(t_3,t_4\right),\cdots \left(t_{n-1},t_n\right)\right]$ from keyframes after discriminating histopathology frames using the histopathology ensemble classifier – (scene_chunks). Each scene_chunk is padded with pad_time to its left and right; see Figure 8 and Table 9 in the Appendix for more details.

1. Text: we use the ASR output to extract the words spoken during each chunk in scene_chunks. Using the method described in Text Extraction using ASR and text denoising subsection, we extract the Medical and ROI caption for this chunk.

2. Image: we extract representative image(s) for every chunk/time-interval in scene_chunks as described in Filtering for narrative-style medical videos subsection above.

Finally, each chunk in scene_chunks is mapped to texts (both medical and ROI captions) and images. Next we map each medical image to one or more medical text. Using the time interval in which the image occurs, we extract its raw text from ASR and then correct and extract keywords using the Rake method, which we refer to as raw_keywords. We extract keywords from each medical text returned using the LLM, and we refer to these as keywords. Finally, if the raw_keywords occur before or slightly after a selected representative image, and overlap with the keywords in one of the Medical/ROI texts for that chunk, we map the image to the medical/ROI text. Example. keywords: psammoma bodies, match with raw_keyword: psammoma bodies within the ASR-corrected text ‘Meningiomas typically have a meningothelial pattern with lobular-like arrangements and psammoma bodies.’Refer to Figure 7 and Figure 15 in the Appendix for a detailed explanation of the method and examples of aligned image and text.

3.2. Quilt-1M: Combining Quilt with other histopathology data sources

To create Quilt-1M, we expanded Quilt by adding other disparate histopathology image-text open-access sources: LAION, Twitter, and PubMed.

PubMed Open Access Articles.

We searched the PubMed open-access from 2010–2022, extracting 62,458 histopathology image-text pairs, using our histopathology classifier and multi-plane figure cropping algorithm. The images are categorized into (1) images that are fully histopathology, (2) multi-plane images that contain histopathology sub-figures, and (3) histopathology sub-figures cropped from (1) and (2). See Figure 16, and Section A.2.1 in the Appendix.

Histopathology Image Retrieval from LAION.

The Large-scale Artificial Intelligence Open Network (LAION-5B) [54] curated over 5 billion pairs of images and text from across the Internet, including a substantial volume of histopathology-related data. We tapped into this resource by retrieving 23,240 image and text pairs. See Section A.2.2 in the Appendix.

Twitter Data from OpenPath.

We utilized a list of tweets curated by Huang et al. [24], which totaled up to 55,000 unique tweets and made up 133,526 unique image-text pairs. This exhibits a one-to-many relationship where many images were matched with multiple captions; this differentiated our work from the OpenPath approach. To maintain comparability, we followed their text pre-processing pipeline [24]. See Section A.2.3 in the Appendix.

3.3. Quality

To evaluate our pipeline’s performance, we assess several aspects. First, we calculate the precision of our LLM’s corrections by dividing the number of conditioned misspelled errors replaced (i.e., passed the UMLS check) by the total number of conditioned misspelled words found, yielding an average of 57.9%. We also determined the unconditioned precision of the LLM, similar to the previous step, and found it to be 13.8%. Therefore, we replace our detected incorrect words with the LLM’s correction 57.9% of the time, and 13.8% of the time we replace the LLM’s detected errors with its correction (see Table 11 in the Appendix). To estimate the ASR model’s transcription performance, we compute the total number of errors replaced (both conditioned and unconditioned) and divide it by the total number of words in each video, resulting in an average ASR error rate of 0.79%. To assess the LLM’s sub-pathology classification, we manually annotated top-k $(k=1,2,3)$ sub-pathology types for 100 random videos from our dataset. The LLM’s accuracy for top-3, top-2, and top-1 was 94.9%, 91.9%, and 86.8%, respectively. Also note that, by prompting the LLM to extract only medically relevant text, we further eliminate identifiable information, such as clinic addresses, from our dataset.

3.4. Final dataset statistics

We collected Quilt, from 4504 narrative videos spanning over 1087 hours with over 438K unique images with 768K associated text pairs. The mean length of the text captions is 22.76 words, and 8.68 words for ROI text, with an average of 1.74 medical sentences per image (max=5.33, min=1.0). Our dataset spans a total of 1.469M UMLS entities from those mentioned in the text (with 28.5K unique). The images span varying microscopic magnification scales (0–10x, 10–20x, 20–40x), obtaining (280K, 75K, 107K) images from each scale respectively. Figure 14 (a, c) in the Appendix plots our dataset’s diversity across multiple histopathology sub-domains. This plot shows that the captions cover histopathology-relevant medical subtypes: findings, concepts, organs, neoplastic processes, cells, diseases, and a mix of laboratory and diagnostic procedures. Overall, across all 127 UMLS semantic types, our entities cover 76.2% of medically-related semantic types (e.g., findings, disease, or syndrome) and 23.75% non-medical (e.g., geographic area, governmental or regulatory activity).

4. QuiltNet: Experiments training with Quilt-1M

We use the Contrastive Language-Image Pre-training (CLIP) objective [48] to pretrain QuiltNet using Quilt-1M. CLIP takes a batch of $N$ (image, text) pairs and optimizes a contrastive objective to create a joint embedding space. The optimization process involves concurrent training of both image and text encoders to increase the cosine similarity of embeddings from aligned pairs, while decreasing it for unaligned pairs. The objective is minimized via the InfoNCE loss, expressed as:

$$\text{𝓛}=-\frac{1}{2N}\left(\sum _{i=1}^N\text{log}\frac{e^{\text{cos}\left(I_i,{\mathit{T}}_i\right)}}{{\sum }_{j=1}^N{e}^{\text{cos}\left(I_i,{\mathit{T}}_j\right)}}+\sum _{i=1}^N\text{log}\frac{e^{\text{cos}\left(I_i,{\mathit{T}}_i\right)}}{{\sum }_{j=1}^N{e}^{\cos \left(I_j,T_i\right)}}\right)$$

where $I_i$ and $T_i$ are the embeddings for the aligned $i$-th image and text, respectively. For the image encoder, we use both ViT-B/32 and ViT-B/16 architectures [16]. For the text encoder, we use GPT-2 [47] with a context length of 77, and PubmedBert [70]. We train QuiltNet by finetuning a pre-trained CLIP model on Quilt-1M to enhance its performance in histopathology. Once finetuned, we conduct experiments on two types of downstream tasks: image classification (zero-shot and linear probing) and cross-modal retrieval (zero-shot). We also compare the performance of fine-tuning a pre-trained CLIP model versus training it from scratch.

Downstream histopathology datasets.

We evaluate the utility of QuiltNet on 12 downstream datasets: PatchCamelyon [60] contains histopathology scans of lymph node sections labeled for metastatic tissue presence as a binary label. NCT-CRC-HE-100K [32] consists of colorectal cancer images and is categorized into cancer and normal tissue. For SICAPv2 [56] the images are labeled as non-cancerous, Grade 3–5. Databiox [8] consists of invasive ductal carcinoma cases of Grades I-III. BACH [4] consists of breast tissues labeled as normal, benign, in-situ, and invasive carcinoma. Osteo [5] is a set of tissue patches representing the heterogeneity of osteosarcoma. RenalCell [10] contains tissue images of clear-cell renal cell carcinoma annotated into five tissue texture types. SkinCancer [35] consists of tissue patches from skin biopsies of 12 anatomical compartments and 4 neoplasms that make up the SkinTumor Subset. MHIST [63] contains tissue patches from Formalin-Fixed Paraffin-Embedded WSIs of colorectal polyps. LC25000 [9], which we divide into LC25000 (Lung) and LC25000 (Colon), contains tissue of lung and colon adenocarcinomas. For more details on the datasets refer to C.1 and Table 15 in the Appendix.

Results using zero-shot learning.

Given the vast diversity of cancer sub-types in histopathology, it is critical that a model maintains comprehensive understanding without requiring specific data for retraining. Thus, we evaluate our model’s zero-shot performance against three state-of-the-art models: CLIP, BiomedCLIP, and PLIP. Our model demonstrates superior performance, as illustrated in Figure 3, where it outperforms the other models in all but two datasets, in which BiomedCLIP performs marginally better. See Table 17 for UMap visualizations and Figure 17 for cross-modal attention visualization comparison in the Appendix. The prompts used for these evaluations are presented in Table 16 in the Appendix. To ensure a fair comparison with BiomedCLIP, which uses a ViT-B/16 and PMB/256 (pre-trained with [70]), we trained three different variants of our model. For detailed insights into the results, please refer to Table 14 in the Appendix.

Figure 3:

QuiltNet, outperforms out-of-domain CLIP baseline and state-of-the-art histopathology models across 12 zero-shot tasks, covering 8 different sub-pathologies (accuracy percentage provided).

Results using linear probing.

We assess the few-shot and full-shot performance of our model by conducting linear probing with 1%, 10%, and 100% of the training data, sampled with three different seeds; we report the average accuracy and their standard deviation in Table 1. We deploy our evaluation across four distinct datasets, specifically those with dedicated training and testing sets among our external datasets. Remarkably, our model, utilizing the ViT-B/32 architecture with GPT/77, outperforms its counterparts, BiomedCLIP, PLIP, and CLIP, in most datasets. On the NCT-CRC and SICAPv2 datasets, our model surpasses even the fully supervised performance using only 1% of the labels. Also, note that for some results 10% does better than 100%; this is because we are sampling from each class equally, and thus the 10% subset contains a more balanced training set than 100%, for datasets that are very imbalanced, resulting in sub-optimal performance at 100%.

Table 1: Linear probing.

Classification results, denoted as accuracy % (standard deviation). Camelyon denotes the PatchCamelyon dataset. Supervised results are from each dataset’s SOTA models.

Dataset	%shot	ViT-B/32			ViT-B/16
		CLIP	PLIP	QuiltNet	CLIP	QuiltNet	BiomedCLIP	QuiltNet
Supervised(%)		GPT/77	GPT/77	GPT/77	GPT/77	GPT/77	PMB/256	PMB/256
NCT-CRC [32]	1	91.0 (0.10)	93.75 (0.09)	94.64(0.22)	90.96 (0.10)	93.36 (0.23)	92.14(0.12)	93.55 (0.25)
	10	92.02(1.30)	93.83 (0.06)	95.30 (0.03)	92.58 (0.12)	93.85 (0.04)	92.90 (0.07)	93.72 (0.08)
(94.0)	100	91.83 (0.01)	94.16 (0.01)	95.22 (0.01)	92.26 (0.09)	93.76 (0.02)	92.97 (0.05)	93.60 (0.01)
Camelyon [60]	1	80.38 (0.16)	87.26 (0.23)	87.62 (0.35)	80.28 (0.20)	84.78 (0.14)	83.63 (0.44)	83.48 (0.18)
	10	82.67 (0.19)	87.48 (0.08)	87.55 (0.03)	82.20 (0.04)	86.77 (0.09)	84.18 (0.15)	84.42 (0.10)
(97.5)	100	82.80 (0.01)	87.34 (0.01)	87.48 (0.01)	82.55 (0.02)	86.81 (0.04)	84.23 (0.01)	84.44 (0.02)
SkinCancer [35]	1	84.27 (0.22)	91.07 (0.25)	90.93 (0.25)	85.62 (0.16)	88.29 (0.15)	87.53 (0.21)	88.06 (0.20)
	10	89.0 (0.02)	93.39 (0.05)	92.99 (0.02)	90.28 (0.01)	91.20 (0.0)	89.23 (0.03)	90.03 (0.02)
(93.3)	100	89.02 (0.02)	93.29 (0.01)	93.03 (0.02)	90.29 (0.03)	91.20 (0.0)	89.16 (0.02)	89.91 (0.01)
SICAPv2 [56]	1	52.45 (2.41)	65.76 (2.65)	69.92 (1.02)	56.01 (0.66)	66.86 (1.16)	69.43 (1.03)	68.49 (1.06)
	10	62.24 (0.65)	69.23 (0.43)	74.14 (0.38)	63.70 (0.69)	72.37 (0.65)	71.61 (0.31)	72.48 (0.42)
(67.0)	100	65.75 (0.16)	73.0 (0.14)	75.48 (0.12)	68.74 (0.10)	74.14(0.16)	74.57 (0.04)	74.60 (0.17)

Results using cross-modal retrieval.

In our study, we evaluate cross-modal retrieval efficacy by examining both zero-shot text-to-image and image-to-text retrieval capabilities. We accomplish this by identifying the nearest neighbors for each modality and then determining whether the corresponding pair is within the top $N$ nearest neighbors, where $N\in \{1,50,200\}$. Our experiments are conducted on two datasets: our holdout dataset from Quilt-1M and the ARCH dataset. Results are in Table 2.

Table 2:

Cross-modal retrieval results on the Quilt-1M holdout set and ARCH dataset. In each cell, the results are displayed in the format (%/%), with Quilt-1M holdout results on the left and ARCH results on the right. The best-performing results are highlighted in bold text.

model	config	Text-to-Image (%)			Image-to-Text (%)
		R@1	R@50	R@200	R@1	R@50	R@200
CLIP	ViT-B/32|GPT/77	0.49/0.07	4.73/2.42	10.15/7.21	0.39/0.05	3.99/2.52	8.80/7.22
PLIP	ViT-B/32|GPT/77	1.05/0.56	10.79/13.10	21.80/29.85	0.87/0.74	11.04/13.75	21.63/29.46
QuiltNet	ViT-B/32|GPT/77	1.17/1.41	16.31/19.87	31.99/39.13	1.24/1.35	14.89/19.20	28.97/38.57
CLIP	ViT-B/16|GPT/77	0.83/0.09	5.63/2.73	11.26/8.72	0.66/0.13	5.02/3.09	10.82/9.04
QuiltNet	ViT-B/16|GPT/77	2.42/1.29	22.38/20.30	41.05/40.89	2.00/1.01	21.66/16.18	39.29/34.15
BiomedCLIP	ViT-B/16(224)|PMB/256	4.34/8.89	14.99/53.24	25.62/71.43	3.88/9.97	13.93/52.13	23.53/68.47
QuiltNet	ViT-B/16(224)|PMB/256	6.20/8.77	30.28/55.14	50.60/77.64	6.27/9.85	31.06/53.06	50.86/73.43

5. Discussion

Limitations.

Despite the promising results, Quilt was curated using several handcrafted algorithms and LLMs. Such curation methods, while effective, introduce their own biases and errors. For instance, our histopathology classifier had occasional false positives (≈ 5%) confirmed by human evaluation. Occasionally, ASR can misinterpret a medical term and transcribe it as a different existing term, such as transcribing ’serous carcinoma’ as ’serious carcinoma’. Unfortunately, such errors are not rectifiable using our current pipeline (see Table 11 in the Appendix). While not directly a limitation of our dataset, training a CLIP model trained from scratch underperformed compared to fine-tuning a pre-trained CLIP (see Table 14 in the Appendix). This suggests that a million image-text pairs may still not be sufficient.

Data Collection and Societal Biases

Aligning in strategies with [68], we release Quilt derived from public videos, taking structured steps to limit privacy and consent harms (see A.5 in the Appendix). Complying with YouTube’s privacy policy, we only provide video IDs, allowing users to opt-out of our dataset. Researchers can employ our pipeline to create Quilt. Regarding societal biases, a significant portion of our narrators originate from western institutions, a situation that is further amplified by our focus on English-only videos. Consequently, QuiltNet may exhibit inherent biases, potentially performing better on data associated with these demographics, while possibly underperforming when applied to other cultural or linguistic groups.

Conclusion.

We introduced Quilt-1M, the largest open-sourced histopathology dataset to date. Empirical results validate that pre-training using Quilt is valuable, outperforming larger state-of-the-art models like BiomedCLIP across various sub-pathology types and tasks including zero-shot, few-shot, full-shot, and cross-modal retrieval. We established a new state-of-the-art in zero-shot, linear probing, and cross-modal retrieval tasks in the field of Histopathology.