Core-Periphery Multi-Modality Feature Alignment for Zero-Shot Medical Image Analysis

Abstract

Multi-Modality learning, exemplified by the language-image pair pre-trained CLIP model, has demonstrated remarkable performance in enhancing zero-shot capabilities and has gained significant attention recently. However, simply applying language-image pre-trained CLIP to medical image analysis encounters substantial domain shifts, resulting in severe performance degradation due to inherent disparities between natural (non-medical) and medical image characteristics. To address this challenge and uphold or even enhance CLIP’s zero-shot capability in medical image analysis, we develop a novel approach, Core-Periphery feature alignment for CLIP (CP-CLIP), to model medical images and corresponding clinical text jointly. To achieve this, we design an auxiliary neural network whose structure is organized by the core-periphery (CP) principle. This auxiliary CP network not only aligns medical image and text features into a unified latent space more efficiently but also ensures alignment driven by principles of brain network organization. In this way, our approach effectively mitigates and further enhances CLIP’s zero-shot performance in medical image analysis. More importantly, the proposed CP-CLIP exhibits excellent explanatory capability, enabling the automatic identification of critical disease-related regions in clinical analysis. Extensive experiments and evaluation across five public datasets covering different diseases underscore the superiority of our CP-CLIP in zero-shot medical image prediction and critical features detection, showing its promising utility in multimodal feature alignment in current medical applications.

I. Introduction

Multi-Modality learning has emerged as a promising approach to enhance the understanding and analysis of complex tasks by leveraging information from multiple sources. There has been a growing research interest focusing on integrating textual modalities into vision models [17] [22]. The synergy between image and text modalities offers mutual benefits, enhancing modeling and reasoning capabilities, and aligns closely with the multimodal perceptual environment of the human brain [21]. One notable advancement in this field is the pre-trained CLIP (Contrastive Language-Image Pre-training) model, which has demonstrated remarkable performance in various tasks by jointly learning from language and image data [13]. The CLIP model aligns image and text embeddings in the latent space through contrastive learning on a dataset comprising 400 million image-text pairs sourced from a diverse range of publicly accessible online platforms. The fusion of text and image modalities has significantly improved zero-shot capabilities, allowing the model to generalize to unseen tasks or domains without explicit training [14] [6] [23].

Nevertheless, despite its remarkable accuracy and feature extraction capability, CLIP’s zero-shot performance heavily relies on large-scale, high-quality image-text paired datasets [24]. Creating such datasets poses significant challenges, especially in specialized domains like healthcare and radiology, where data is not only scarce but often presents distinct patterns in both image and text components compared to natural images and text that CLIP is trained on [25]. That is, there exists a significant domain shift between natural (non-medical) and medical images [20]. Additionally, CLIP’s reliance solely on contrastive loss for extracting image and text features imposes limitations on its ability to align these features effectively [4] [19] [26]. Thus, there is an increasing need to enhance CLIP with additional mechanisms that can not only improve the multimodality feature alignment between image and text features but also leverage CLIP’s zero-shot capability on downstream tasks with limited datasets [27] [28] [29].

To address the zero-shot performance degradation of the CLIP model in the medical imaging domain, our strategy aims to improve the effectiveness when aligning multimodal features in latent space by developing a novel information exchange mechanism for neural networks. This mechanism is inspired by the Core-Periphery (CP) organization that universally exists in brain functional networks of humans and other mammals [1]. It has been widely confirmed that the CP organization can effectively promote the efficiency of information transmission and communication for biologically integrative processing [5] [53]. In general, CP organization is composed of two qualitatively distinct components: a dense “core” of nodes that are strongly interconnected with one another, allowing for integrative information processing to facilitate the rapid transmission of messages and a sparse “periphery” of nodes that sparsely connected to the cores [30]. In this work, we aim to incorporate the CP principle into model design to effectively guide neural networks in aligning the text and image features extracted from CLIP, consolidating them into a unified latent space. Through this way, our CP-CLIP can align the critical features from multimodal data in latent space more efficiently with limited paired image-text samples, thereby alleviating performance degradation of original CLIP and simultaneously facilitating the identification of potential disease-related regions. The main idea of CP-CLIP is shown in Fig. 1. In vanilla CLIP, the image and text are encoded into the image feature space and text feature space separately by different encoders. The feature alignment is achieved through a contrastive loss. Instead, our CP-CLIP further encodes the image embeddings and text embeddings into a unified core-periphery aligned feature space using the designed core-periphery principle guided neural network, and then implements contrastive learning on the core-periphery aligned image and text features.

Fig. 1:

The distinction between CLIP (top panel) and CP-CLIP (bottom panel) lies in their approach to feature alignment. CLIP utilizes a single contrastive learning loss to regulate feature alignment, while CP-CLIP first maps image and text embeddings to a unified core-periphery aligned feature space and then regulates feature alignment accordingly.

We have applied our proposed CP-CLIP to five public medical datasets under zero-shot scenarios, and the experimental results show that CP-CLIP improves the zero-shot performance of CLIP by 13.34% on ChestXray, 2.50% on SIIM-ACR, 6.04% on INBreast, 0.40% on CheXpert5×200, and 2.72% on TMED. Additionally, it effectively identifies critical disease-related regions for the diseases. The primary contributions of this work are as follows:

• We bridge deep neural networks and brain science by applying the brain-inspired Core-Periphery structure to neural network design, resulting in a core-periphery principle guided neural network.

• We integrate the core-periphery principle guided neural network into the CLIP model, introducing CP-CLIP for aligning medical image and text features, thereby establishing effective multi-modal feature alignment.

• We demonstrate the improved zero-shot performance of CP-CLIP by implementing extensive experiments on five publicly accessible medical datasets, achieving enhanced zero-shot accuracy and the ability to identify disease-related areas. The interpretability of the CLIP model is significantly enhanced, enabling precise identification of disease-relevant regions.

II. Related Work

A. CP Structure

The core-periphery paradigm delineates a topology within a graph theoretic framework where central (“core”) nodes exhibit a high degree of interconnectivity, whereas peripheral nodes maintain sparse connections, both among themselves and with the “core”. This structural archetype has been extensively recognized and employed across a multitude of disciplinary boundaries, such as sociological network analysis [31] [2], economic systems [32], and the biological sciences, where it serves to model the intricate lattice of protein interactions [33]. Within the realm of brain science, empirical evidence substantiates the existence of a core-periphery schema underpinning cerebral dynamics [1]. Functional neural networks are similarly corroborated to manifest this architecture [7]. A vanguard investigation has illuminated this core-periphery dichotomy in the human brain through an anatomical lens [34], positing that gyri and sulci – the salient morphological features of cortical folding, synergistically form a core-periphery network. Such an organization is posited to augment the efficacy of neural signal propagation.

The Core-Periphery (CP) principle proposes that in complex networks such as brain networks, nodes (or neurons, in the context of neural networks) are organized into a densely connected “core” and a sparsely connected “periphery”. In the context of common latent spaces, such as those used in multimodal models like CLIP, these dynamics can influence how different modalities (e.g., text and images) are integrated. Nodes in the core are highly interconnected and responsible for preserving the most critical information across different modalities. In a common latent space, core-core interactions could represent strong, consistent features that are shared across modalities, ensuring robust representation. Core-periphery interaction reflects the exchange of information between central and peripheral nodes. In latent spaces, core-periphery dynamics might help in mapping unique features of one modality onto the shared space, supporting cross-modal translation or transfer learning. Nodes in the periphery are less connected and often represent specialized or less critical information. In a common latent space, these interactions may correspond to modality-specific nuances or less prominent features that do not strongly influence the shared representation.

B. Contrastive Learning

Contrastive learning is a machine learning technique that trains models to distinguish between similar and dissimilar data samples [35]. It is particularly powerful in the field of unsupervised and self-supervised learning, where labeled data is scarce or expensive to obtain. The core idea behind contrastive learning is to learn representations by enforcing that similar or related samples stay close to each other in the representation space, while dissimilar or unrelated samples are pushed apart [36].

Contrastive learning hinges on the definition of positive and negative pairs. Positive pairs are those data samples that are considered similar, such as different augmentations of the same image or sentence [37] [4] [38] [39]. Negative pairs, on the other hand, are dissimilar data samples, such as augmentations from different images or sentences. The contrastive learning then uses a contrastive loss function, such as the triplet loss [40], to optimize the distances between positive and negative pairs in the representation space. Contrastive learning has gained significant attention in the field of computer vision and natural language processing due to its ability to leverage large amounts of unlabeled data effectively. In computer vision, it has been used to develop state-of-the-art models for image tasks without the need for extensively annotated datasets [41] [13].

C. Feature Alignment

Feature alignment refers to the technology in which extracted features from different datasets or different views of the same dataset are compatible with each other. In other words, encoded features can match with others in the feature space according to their similarities. This is crucial for domain shift scenarios, where zero-shot capability is desired on samples with new distributions that a pre-trained model never saw before. Feature alignment plays an important role in generalization in multi-modality large language models [13]. Since the same concepts can be represented by different modalities, by processing and aligning these features, these models are able to understand information from multiple modalities and execute downstream tasks in new modalities.

One common idea of feature alignment is to learn a shared feature space where the representations of data from different modalities or views are indistinguishable. Techniques such as Canonical Correlation Analysis (CCA) [42] and Adversarial Discriminative Domain Adaptation (ADDA) [43] have been proposed to achieve this. These methods often minimize some measures of distance between feature distributions, such as the Maximum Mean Discrepancy (MMD), or leverage adversarial training to make the domains indistinguishable to a domain classifier.

D. CLIP for Medical Imaging-Text

CLIP (Contrastive Language–Image Pre-training) [13] is a multimodal pre-training algorithm proposed by OpenAI, continuing the legacy of the GPT series in leveraging substantial scale to achieve remarkable performance. This model represents a multimodal paradigm, concurrently processing image and text modalities. The training objective is constructed through the computation of similarity between feature vectors from these two distinct modalities. In an effort to train this model, OpenAI has amassed a dataset comprising over 400 million image-text pairs. Demonstrating exceptional performance across a variety of multimodal tasks, such as image retrieval and image classification, CLIP has shown that even through unsupervised learning, it can achieve outstanding performance comparable to those of mainstream supervised algorithms in many areas. The conceptual simplicity of CLIP, coupled with its impressive performance, underscores the substantial potential for future development in multimodal models [44].

In the rapidly evolving domain of medical imaging and diagnostics, the application of CLIP models has demonstrated significant potential [19] [45]. These models are adept at interpreting and understanding medical images in conjunction with textual descriptions, thus enabling nuanced recognition patterns that closely mimic the diagnostic process of medical professionals. However, the efficacy and robustness of such models are intrinsically tied to the volume and quality of the training data. However, acquiring large-scale, annotated medical datasets poses a critical challenge for successfully training CLIP models in the medical domain. These datasets should not only be large but also carefully curated to accurately represent the diverse range of clinical cases. This ensures the effective correlation between medical imaging features and their corresponding textual descriptions.

III. Method

The overview of CP-CLIP framework is shown in Fig. 2. The CP-CLIP comprises four essential components: the pre-trained image and text encoders, the generation of core-periphery graphs (CP graph), the CP graph guided neural network, and the integration of the CP-guided neural network into CLIP. The details of each component will be elaborated in the following sections.

Fig. 2:

The CP-CLIP framework. Part (a): The core-periphery principle guided feature alignment. Latent features extracted from the image and text encoders are subsequently projected into a unified latent space via the core-periphery principle guided multilayer perceptron neural network. Part (b): The communication of information and the pattern of neuron connections within the core-periphery principle-guided neural network is determined by the generated core-periphery graphs.

A. Pre-trained Image and Text Encoders

To leverage the feature extraction ability of pre-trained models and reduce computational costs, we adopt the pre-trained ResNet50 as the default image encoder and the pre-trained BERT as the text encoder [50] [47]. In the training stage, each subject encompasses two modalities: a medical image and the corresponding clinical report. For each subject, we have an image-text pair $\left(I,T\right)$.

Image Encoder

We encode the image into embedding ${\mathbf{u}}_I\in {ℝ}^D$ via the image encoder $E_I$. A projection head then maps the raw embedding to $\mathbf{u}\in {ℝ}^P$.

$$\left\{\begin{array}{c}{\mathbf{u}}_I=E_I\left(I\right),\\ \mathbf{u}=f_I\left({\mathbf{u}}_I\right)\end{array}\right.$$ (1)

where $f_I$ is the projection head of the image encoder.

Text Encoder

Similarly, we encode the clinical report into embedding ${\mathbf{u}}_T\in {ℝ}^M$ via the text encoder $E_T$, then we project ${\mathbf{u}}_T$ to $\mathbf{t}\in {ℝ}^P$, which can be formulated as:

$$\left\{\begin{array}{c}{\mathbf{u}}_T=E_T\left(I\right),\\ \mathbf{t}=f_T\left({\mathbf{u}}_T\right)\end{array}\right.$$ (2)

where $f_T$ is the projection head of the text encoder. This results in an embedding dimension $P$ identical to that of the image encoder, making it well-suited for contrastive learning. In the inference stage, as clinical reports may not be available for zero-shot medical datasets, we use the prompt “An image of [mask].” for classification tasks, where “mask” represents possible disease labels.

B. Core-Periphery Graph Generation

The core-periphery neural network in CP-CLIP is controlled by the generated Core-Periphery graphs (CP graphs). We introduce the CP graph generation process, which generates a diverse range of CP graphs within the graph space defined by core ratios. The core ratio is defined as the proportion of core nodes to all nodes. It is worth mentioning that in a vanilla neural network, neurons are fully connected, meaning each neuron is connected to all other neurons. Therefore, the connections in a vanilla neural network can be represented by complete graphs, with a core ratio of 1.0. To generate graphs with a Core-Periphery (CP) property [31] [2], we define CP graphs as having nodes categorized into core and periphery nodes. Core nodes, acting as information integration hubs, are connected to all nodes, while periphery nodes are solely connected to core nodes. Denoting the total number of nodes as $N$ and the core ratio as $p$, $p\in \left(0,1\right]$, we calculate the number of core nodes as $n=N\times p$ and the number of periphery nodes as $m=N\times \left(1-p\right)$. Note that when the core ratio equals 1.0, the CP graphs degrade to the complete graphs, implying that the CP-guided multilayer perceptron network reverts to its vanilla form.

Based on the above analysis, the adjacency matrix $A_{N\times N}$ of the generated CP graphs can be expressed as:

$$A\left(i,j\right)\left\{\begin{array}{ll}1\hfill & \text{if}\exists \left(i,j\right)\in n\hfill \\ 0\hfill & \text{if}\forall \left(i,j\right)\in m\hfill \end{array}\right.$$ (3)

where 1 signifies the presence of an edge between nodes $i$ and $j$, and 0 indicates no edge between the nodes. By employing various core ratios, denoted by different combinations of $n$ and $m$, a wide range of candidate graphs can be generated within the graph space. Examples of CP graphs and complete graph are shown in Fig. 3. As shown in the figure, there are different connection patterns of the CP graphs under different core ratios.

Fig. 3:

Examples of Core-Periphery Graphs with different core ratios. The first row displays the graphs, while the second row shows their corresponding adjacency matrices. The core nodes are shown in red, and the periphery nodes are shown in blue. In the adjacency matrices, the white area denotes 0, indicating no connection, while the black area denotes 1, representing connections between nodes.

C. Core-Periphery Principle Guided Neural Network

In this work, we integrate the core-periphery principle in neural networks through the Core-Periphery Graph Guided Multilayer Perceptron (MLP) Neural Network. In CP graphs, core nodes maintain connections to all other nodes, while periphery nodes only connect to the core nodes. To integrate the CP principle into the organization of the multilayer perceptron neural network, we reschedule the neuron connections based on the generated CP graphs. Here, neurons are considered as nodes, and connections between neurons are regarded as edges. This approach allows us to represent neural networks as graphs and utilize the generated Core-Periphery (CP) graph to guide connections between the nodes. Following this representation paradigm, a complete graph can represent vanilla multilayer perceptron networks. Similarly, we incorporate the Core-Periphery principle into the multilayer perceptron architecture by substituting the complete graph with the generated CP graphs. The new connection rules can then be redefined: CP graph can be represented by $\mathcal{G}=\left(\mathcal{V},\mathcal{E}\right)$, with nodes set $\mathcal{V}=\left\{{\nu }_1,\dots ,{\nu }_n\right\}$, edges set $\mathcal{E}\subseteq \left\{\left({\nu }_i,{\nu }_j\right)|{\nu }_i,{\nu }_j\in \mathcal{V}\right\}$, and adjacency matrix $A$. The information exchange in the CP graph guided multilayer perceptron network for a specific node $i$ at $r$-th layer is defined as:

$$z_i^{\left(r+1\right)}={\sigma }^{\left(r\right)}\left(\left\{\sum w_{ij}^{\left(r\right)}{z}_j^{\left(r\right)},\forall j\in \mathcal{N}\left(i\right)\right\}\right)$$ (4)

where $\sigma \left(·\right)$ is the activation function, $z_i^{\left(r+1\right)}$ and $z^{\left(r\right)}$ are the features stored in the nodes, $w_{ij}^{\left(r\right)}$ is the weight of the edge connecting the node $i$ and $j$, $\mathcal{N}\left(i\right)$ are the neighborhood nodes of node $i$. We can rewrite the Eq. 4 in matrix form as:

$${\mathbf{Z}}^{\left(r+1\right)}={\sigma }^{\left(r\right)}\left\{\left(A\odot {\mathbf{W}}^{\left(r\right)}\right){\mathbf{Z}}^{\left(r\right)}\right\}$$ (5)

where $\mathbf{Z}$ is the feature matrix, and $\mathbf{W}$ is the weight matrix, and $\odot$ is the element-wise matrix multiplication.

Each node corresponds to one or multiple neurons. We propose the following neuron assignment pipeline to map the original neurons to the nodes [51]: for a CP graph with $N$ nodes, each node will be assigned either $⌊M/N⌋+1$ or $⌊M/N⌋$ neurons, where $M$ is the dimension at a specific layer. For example, if we utilize a CP graph with 5 nodes for a layer with 196 dimensions, the 5 nodes will have 40, 39, 39, 39, and 39 neurons, respectively. Conversely, if we employ a CP graph with $M$ nodes for $M$ dimensions, each node will correspond to one neuron.

D. Core-Periphery Feature Alignment for CLIP

We integrate the CP graph guided neural network into the CLIP model to enhance the fusion of modality information from both images and texts, mapping them into a unified latent space. This framework, visually represented in Fig. 2, is termed CP-CLIP. After extracting features from images and texts using CLIP, their embeddings are fed into the CP network with shared weights. This process encourages the embeddings of both modalities to converge into a unified latent space, thereby facilitating the alignment of features between them.

For a mini-batch of images and texts $\mathbf{I}$ and $\mathbf{T}$, the embeddings extracted from CLIP are represented as ${\mathbf{Z}}_{\mathbf{I}}$ and ${\mathbf{Z}}_{\mathbf{T}}$. We refer to the core-periphery graph guided network as $f_{cp}$. Then, the CP network maps the text and image embeddings to a unified space as follows:

$$\begin{gathered} \left\{{\mathbf{Z}}_{\mathbf{I}}^{\prime }=f_{cp}\left({\mathbf{Z}}_{\mathbf{I}}\right) \\ {\mathbf{Z}}_{\mathbf{T}}^{\prime }=f_{cp}\left({\mathbf{Z}}_{\mathbf{T}}\right)\right. \end{gathered}$$ (6)

The logits are obtained from the cosine similarity between the embeddings of images and texts, which have been aligned by the CP network. This can be formulated as follows:

$$\mathbf{s}={\mathbf{Z}}_{\mathbf{I}}^{\prime }\cdot {\left({\mathbf{Z}}_{\mathbf{T}}^{\prime }\right)}^T$$ (7)

where $T$ means transpose. For an image $i$, the scaled cosine similrity is obtained by normalizing across the logits, which represent the scores or probabilities associated with the image’s relevance to various text descriptions:

$$y_{ij}^{\mathbf{I}\to \mathbf{T}}=\frac{exp\left(s_{ij}/\tau \right)}{{\sum }_{j=1}^{N_{batch}}exp\left(s_{ij}/\tau \right)}$$ (8)

where $\tau$ is the learnable temperature, similar to CLIP [13], $j\in \left[1,N_{batch}\right]$ correspond to the batch of texts. Likewise, we can computer the $y_{ji}^{T\to I}$, and reach the loss function:

$$\mathcal{L}=-\frac{1}{2}\sum \log y_{ij}^{\mathbf{I}\to \mathbf{T}}-\frac{1}{2}\sum \log y_{ji}^{\mathbf{T}\to \mathbf{I}}$$ (9)

IV. Results

In this section, we conduct extensive experiments to evaluate the performance of the proposed CP-CLIP model on various medical image datasets under zero-shot scenarios. We begin by introducing the datasets utilized in this study and providing training details. Subsequently, we apply CP-CLIP to five medical datasets in a zero-shot manner for image classification tasks and to identify critical areas potentially related to diseases. Additionally, we conduct ablation studies to investigate the influence of different core ratios and different vision backbones.

A. Datasets and Training Details

• MIMIC-CXR: The MIMIC Chest X-ray (MIMIC-CXR) database [11] offers a comprehensive public repository of chest radiographs in DICOM format, accompanied by free-text radiology reports. This extensive dataset comprises 377,110 images across 227,835 radiographic studies, positioning it as a pivotal resource for text-image pair analysis within the medical field.

• ChestXray: The NIH ChestX-ray dataset [18], encompasses an extensive collection of 112,120 chest X-ray images derived from 30,805 distinct patients, each annotated with disease labels. Predominantly, this dataset focuses on two critical categories: pneumonia and normal states.

• SIIM-ACR: The SIIM-ACR dataset [48], is facilitated by the Society for Imaging Informatics in Medicine (SIIM) in collaboration with the American College of Radiology (ACR). The dataset is composed of 12,047 radiographic images, among which 2,669 are annotated to delineate the presence or absence of pneumothorax, categorically differentiated into two distinct classes: normal lung function and collapsed lung, the latter indicative of pneumothorax.

• INbreast: The INbreast database [12] is a mammographic database, with images acquired at a Breast Centre, located in Hospital de São João, Breast Centre, Porto, Portugal. INbreast has a total of 115 cases (410 images) of which 90 cases are from women with both breasts (4 images per case) and 25 cases are from mastectomy patients (2 images per case).

• CheXpert5×200: The CheXpert dataset [10] is a large public dataset for chest radiograph interpretation, designed for developing algorithms to automate the reading and interpretation of chest x-ray images. This dataset contains 224,316 chest radiographs of 65,240 patients with both frontal and lateral views available. Labels were automatically extracted from radiology reports using a NLP tool. In this paper, we select 5 classes (atelectasis, cardiomegaly, consolidation, edema, pleural effusion), each class with 200 chest radiographs to form CheXpert5×200.

• TMED: The TMED dataset [9], short for Tufts Medical Echocardiogram Dataset (TMED), consists of 599 studies from 577 unique patients, some of whom underwent multiple studies on distinct days. All patients have aortic stenosis (AS) diagnostic label (none, early AS, or significant AS; for further details, see our severity diagnosis label primer). Additionally, some images from each study have view label annotations. After data clearing, there are a total of 24964 images available for zero-shot classification.

We initialize CP-CLIP with the pre-trained CLIP weights and subsequently train both CP-CLIP and CLIP models on the MIMIC-CXR dataset [11], which comprises medical textimage pairs. The image encoder in CLIP and the CP network are trained during fine-tuning, while the text encoder in CLIP remains frozen. Subsequently, we evaluate their zero-shot performance on five additional medical image datasets. Note that there are no available text descriptions for the other five datasets. For classification tasks, we create text descriptions in the format “An image of [mask].” Unlike supervised learning settings on the five datasets, we utilize all images in both the training and testing splits as testing images in the zero-shot scenarios. This significantly increases the size of the testing dataset, effectively demonstrating the efficacy of our CP-CLIP. The default number of layers for the CP graph guided MLP is set to 1. Training involves 10 epochs with a batch size of 128 on Titan GPUs, utilizing the AdamW optimizer and cosine learning rate scheduler [49]. We searched across a wide range of core ratios from 0.1 to 1.0 with an interval of 0.1. Learning rates are 1e−8 for the CLIP model and 1e−3 for the CP network.

B. Classification Results

We evaluate the trained CP-CLIP model under a zero-shot setting to assess the model’s multimodal representation robustness and generalizability. Specifically, we select five unseen medical imaging datasets covering breast images, chest X-ray images, and ultrasound heart images, and compare them against two baselines: CLIP and MedCLIP. Note that MedCLIP was well trained on the MIMIC-CXR dataset [19]. The zero-shot classification results are presented in Table I. CP-CLIP, along with two other baselines, was trained on the MIMIC-CXR dataset. However, as depicted in Table I, compared to vanilla CLIP, CP-CLIP demonstrates superior classification accuracy across the datasets, showing a 13.34% improvement on ChestXray, a 2.50% improvement on SIIM-ACR, a 6.04% improvement on INbreast, a 0.40% improvement on CheXpert5×200, and a 2.72% improvement on TMED, highlighting the efficacy of the CP principle guided feature alignment. Compared to the competitive MedCLIP, which is optimized for medical data, our CP-CLIP also achieves higher classification accuracy on four datasets.

TABLE I:

Comparison of zero-shot classification performance among CLIP, MedCLIP, and CP-CLIP across five medical image datasets. Pretrained ResNet50 and BERT are used as image and text encoders, respectively, in both CP-CLIP and CLIP. Results reported for CP-CLIP represent the best performance achieved across various core ratios. Balanced accuracy is reported in percentage. The best results are in bold.

Model	ChestXray (2 classes)	SIIM-ACR (2 classes)	INbreast (3 classes)	CheXpert5×200 (5 classes)	TMED (3 classes)
CLIP	49.50	47.50	34.67	21.80	33.00
MedCLIP	68.31	50.00	33.56	12.90	33.33
CP-CLIP	62.84	50.00	40.71	22.20	35.72

C. Critical Area Identification

To investigate how the CP mechanism enhances feature alignment, we utilized Grad-CAM [15] visualization on images from various datasets to analyze how the model processes core information during inference. Here, we assume that the core information refers to the critical area closely related to the disease status. We presented the visualization results in Figures 4, 5, and 6. As shown in Fig. 4, the original CLIP struggles to identify critical areas for disease identification. For instance, in the case of malignancy from the INbreast dataset, the area identified by CLIP is far from the cancer area and unrelated to breast cancer identification. In contrast, the CP-CLIP model effectively identifies the cancer area in the image, demonstrating effective feature alignment between the “malignant” in the text and the cancer areas in the image. Another example is the benign case in INbreast, compared to CLIP, our CP-CLIP not only focuses on lesion regions but also considers minor areas for comprehensive reasoning. For chest X-ray images, such as ChestXray and SIIMACR, CLIP tends to focus on the spine, whereas our CP-CLIP model can adjust to focus on the lung areas. In the CheXpert5×200 dataset, depicted in Fig. 5, CLIP often identifies unrelated areas such as the spine and background. In contrast, our CP-CLIP effectively mitigates CLIP’s tendency for shortcuts and accurately identifies disease-related areas. In TMED, as depicted in Fig. 6, CP-CLIP effectively focuses on the aortic valves across three different clinical statuses, whereas CLIP fails to identify these critical areas.

Fig. 4:

Visualization comparison of CLIP and CP-CLIP on the ChestXray, SIIMACR, and INbreast datasets. The left column displays the original images, the middle column illustrates the critical areas identified by CLIP, and the right column showcases the critical areas identified by CP-CLIP. CP-CLIP highlights critical disease-related areas, while CLIP tends to identify shortcuts.

Fig. 5:

The visualization comparison between CLIP and CP-CLIP on the CheXpert5×200 dataset. The left column displays the original images, the middle column illustrates the critical areas identified by CLIP, and the right column showcases the critical areas identified by CP-CLIP.

Fig. 6:

The visualization comparison between CLIP and CP-CLIP on the TMED dataset. The left column displays the original images, the middle column illustrates the critical areas identified by CLIP, and the right column showcases the critical areas identified by CP-CLIP.

The identification of critical areas demonstrates the strong interpretability of the proposed CP-CLIP model. This interpretability helps explain why CP-CLIP achieves higher classification performance, as reported in Table I, by effectively identifying the crucial areas related to disease status. These visualization results further demonstrate the effectiveness of leveraging our core-periphery principle guided feature alignment in conjunction with contrastive loss, as opposed to using contrastive loss alone.

D. Ablation Study

To comprehensively investigate the influence of different core ratios of CP graphs on classification performance, as well as the classification performance with different vision backbones, we conducted ablation studies on CP-CLIP with various core ratios and vision backbones. The results are presented in Fig. 7. As shown in Fig. 7, the impact of the core ratio on classification performance can vary depending on the dataset and model architecture, within the same vision backbone, different core ratios lead to varying zero-shot classification accuracy. For instance, in the upper left part of the RN50 backbone, the best core ratio on the ChestXray dataset is 0.7, whereas, on the INbreast dataset, the best core ratio is 0.5. We can also observe that different vision backbones impact the model performance. For example, among the four backbones, RN50 exhibits the best performance on the INbreast dataset in terms of the highest zero-shot accuracy, while ViT-B/32 demonstrates the best performance on the SIMM-ARC dataset, and RN101 performs the best on the CheXpert5×200 dataset. Another interesting finding is that the feature alignment guided by a CP graph with sparsity usually outperforms feature alignment guided by a complete graph without sparsity. This is particularly evident as the best zero-shot accuracy tends to appear under conditions where the core ratio is smaller than 1.0. For example, the highest zero-shot accuracy with ViT-B/16 occurs at a core ratio of 0.7 rather than 1.0. These results demonstrate the superiority of our proposed CP-CLIP model by outperforming the baselines across a wide range of core ratios. Additional zero-shot performance results for the original CLIP model across these five public datasets are presented in Table II. In Table II, CLIP is not fine-tuned on the MIMIC-CXR dataset, and our CP-CLIP also demonstrates superior performance compared to CLIP. We can conclude that CP-CLIP exhibits superiority over CLIP, regardless of fine-tuning CLIP on medical image datasets. We also present the best zero-shot accuracy achieved with different vision backbones on the five datasets in Table III, for quantitative visualization.

Fig. 7:

The impact of varying core ratios on zero-shot classification accuracy across the five datasets under different vision backbones.

TABLE II:

Comparison of classification performance between CP-CLIP and original CLIP on medical image datasets using RN50 as the vision backbone. CP-CLIP results represent the best performance across various core ratios. Balanced accuracy is presented in percentage.

Model	ChestXray (2 classes)	SIIM-ACR (2 classes)	INbreast (3 classes)	CheXpert5×200 (5 classes)	TMED (3 classes)
CLIP	50.00	50.00	33.26	20.00	33.36
CP-CLIP	62.84	50.00	40.71	22.20	35.72

TABLE III:

Classification comparison among various vision backbones. CP-CLIP results are based on optimal core ratios. Balanced accuracy is presented as a percentage.

Vision Backone	ChestXray (2 classes)	SIIM-ACR (2 classes)	INbreast (3 classes)	CheXpert5×200 (5 classes)	TMED (3 classes)
RN50	62.84	50.00	40.71	22.20	35.72
RN101	50.00	50.00	36.82	23.10	36.67
ViT-B/16	50.00	58.77	37.85	19.80	34.96
ViT-B/32	67.07	56.23	36.61	21.40	37.38

Furthermore, to evaluate the impact of different numbers of CP layers and to compare CP with other sparsity methods, we conducted an ablation study and presented the results in Table IV and Table V, respectively. CP-CLIP demonstrated superior performance on the SIIM-ACR, INbreast, and TMED datasets when utilizing a single CP layer. In contrast, the model performed better on the ChestXray and CheXpert5×200 datasets with three CP layers. Additionally, CP sparsity outperformed the other two sparsity methods, namely random sparsity and the Watts-Strogatz method [52]. To demonstrate the effectiveness of implementing the contrastive loss on the core-periphery aligned space, we compared it against applying the contrastive loss on both the CP aligned and CLIP space. The results are shown in Table VI. For most of the datasets, CP-CLIP achieved better performance with features solely aligned in the CP space, indicating that features were well-aligned by the core-periphery network.

TABLE IV:

Classification comparison with different numbers of CP layers using the RN50 as the vision backbone. Balanced accuracy is presented as a percentage.

CP Layer	ChestXray (2 classes)	SIIM-ACR (2 classes)	INbreast (3 classes)	CheXpert5×200 (5 classes)	TMED (3 classes)
1 layer	62.84	50.00	40.71	22.20	35.72
2 layer	51.10	44.85	34.71	20.40	33.33
3 layer	73.71	48.29	33.54	23.30	33.33

TABLE V:

Classification comparison of Core-periphery with other sparse methods. Using RN50 as the vision backbone. Balanced accuracy is presented as a percentage.

Sparse method	ChestXray (2 classes)	SIIM-ACR (2 classes)	INbreast (3 classes)	CheXpert5×200 (5 classes)	TMED (3 classes)
Core-periphery	62.84	50.00	40.71	22.20	35.72
Random Sparse	60.45	44.59	33.63	18.40	33.50
Watts-Strogatz	61.10	50.00	33.33	20.00	33.33

TABLE VI:

Classification comparison between contrastive loss on CP space versus CP and CLIP space. Using RN50 as the vision backbone. Balanced accuracy is presented as a percentage.

Alignment	ChestXray (2 classes)	SIIM-ACR (2 classes)	INbreast (3 classes)	CheXpert5×200 (5 classes)	TMED (3 classes)
CP Space	62.84	50.00	40.71	22.20	35.72
CLIP Space	50.00	50.00	33.26	20.00	33.36
CP Space+CLIP space	65.81	46.44	40.70	19.90	33.20

V. Discussion and Limitation

Since our core-periphery principle-guided network is inspired by brain networks, where different networks responsible for various tasks (such as vision and movement) have different core ratios [34], we have adopted a similar approach by experimenting with different core ratios to find the optimal ratio for specific datasets or tasks.

In this paper, we assign artificial network neurons to core and periphery nodes using an existing neuron assignment strategy [51], which treats all neurons equally. This approach may not be optimal. Exploring improved neuron assignment strategies is a promising direction for future research. Future work could consider the importance of neurons, assigning more critical neurons to core nodes and less important ones to periphery nodes. This paper focuses on integrating the brain-inspired Core-periphery principle into CLIP rather than exploring different neuron assignment strategies.

VI. Conclusion

In this study, we introduce CP-CLIP, a novel framework that incorporates the core-periphery principle into the CLIP model to enhance multi-modality feature alignment in medical applications. CP-CLIP constructs an auxiliary core-periphery graph guided neural network specifically tailored for zero-shot medical image analysis. This auxiliary network enhances the fine-grained alignment between image and text embeddings, guiding the model’s attention to focus on crucial information. Experimental results across five diverse medical image datasets validate the effectiveness of CP-CLIP in medical image analysis.