A CLIP-based framework for multiclass lung histopathology classification with prompt engineering and class-imbalance-aware focal optimization

Abstract

Lung cancer remains one of the leading causes of cancer-related mortality worldwide, and accurate histopathological classification is essential for timely diagnosis and treatment planning. This study presents a Contrastive Language Image Pretraining (CLIP)-based framework for multiclass lung histopathology classification, designed to distinguish among benign lung tissue, lung adenocarcinoma, and lung squamous cell carcinoma. The proposed approach leverages a pretrained CLIP ViT-B/32 backbone, domain-specific prompt engineering, multimodal image text pairing, and similarity-based classification within a shared embedding space. To strengthen convergence and robustness during fine-tuning, the training pipeline incorporates data augmentation, Focal Loss, AdamW optimization, OneCycle learning rate scheduling, mixed-precision training, gradient clipping, and early stopping. The dataset is organized into separate training, validation, and testing splits, with the reported training and validation partitions containing 3,500 and 500 images per class, respectively. Experimental training on a Tesla T4 GPU demonstrated steady performance improvement across epochs, with the best validation accuracy reaching 95.20%, accompanied by a macro AUC of 0.9870 and a micro AUC of 0.9877, before early stopping was triggered at epoch 23. These findings indicate that integrating CLIP with pathology-specific text prompts provides a strong and reliable framework for automated lung cancer histopathology classification, with promising potential for future intelligent digital pathology systems.

Introduction

Lung cancer remains a major global public health burden because of its high incidence, substantial mortality, and frequent diagnosis at advanced stages. Therefore, timely and accurate disease characterization is essential for improving patient management and survival outcomes. From a clinical perspective, histological subtype determination is particularly important in non-small cell lung cancer (NSCLC), as treatment planning, prognosis estimation, molecular testing pathways, and eligibility for targeted therapies are strongly influenced by whether the tissue is classified as benign lung tissue, lung adenocarcinoma, or lung squamous cell carcinoma. Global cancer surveillance reports continue to identify lung cancer as one of the most commonly diagnosed cancers worldwide and the leading cause of cancer-related death, highlighting the need for reliable diagnostic pipelines that can support earlier and more accurate clinical decision-making^1,2.

Histopathological examination of hematoxylin and eosin stained tissue remains the diagnostic gold standard for lung tumor assessment. However, manual interpretation is labor-intensive, time-consuming, and dependent on expert evaluation, particularly when tissue regions contain subtle or overlapping morphological characteristics³. In routine pathology, distinguishing lung adenocarcinoma from lung squamous cell carcinoma is not only a morphological task but also a clinically important step that influences therapeutic direction and subsequent molecular workup. The growing adoption of whole-slide imaging and digital pathology has therefore created strong motivation for automated artificial intelligence systems that can assist pathologists in identifying diagnostically relevant patterns from large-scale histopathological image data.

Challenges in lung histopathology image analysis

Despite the progress of artificial intelligence in digital pathology, lung histopathology classification remains challenging because diagnostic information is distributed across complex tissue architectures, cellular arrangements, and subtle morphological cues. Histopathological images may contain overlapping nuclei, irregular glandular formations, stromal reactions, keratinization patterns, necrotic regions, and mixed tissue backgrounds. These variations increase the difficulty of separating benign from malignant tissue and, more importantly, distinguishing closely related malignant subtypes such as adenocarcinoma and squamous cell carcinoma.

The first major challenge is the complex multiscale organization of lung tissue. Histopathological diagnosis depends on both global tissue architecture and local cellular appearance, meaning that relevant features may appear at different spatial scales within the same image. Automated models must therefore capture fine cellular morphology as well as broader contextual tissue arrangements. The second challenge is the visual similarity between malignant subtypes. Lung adenocarcinoma and lung squamous cell carcinoma can share overlapping visual patterns, especially in poorly differentiated or partially informative tissue regions. As a result, classification models must identify subtle diagnostic cues such as glandular differentiation, keratinization, nuclear atypia, and growth pattern variation⁷.

Another important difficulty is high intra-class heterogeneity. Even within the same diagnostic category, histopathological images may vary due to tumor heterogeneity, differentiation grade, sampling region, tissue composition, and patient-specific variation. Publicly available pulmonary pathology datasets also reflect this diversity by including different magnifications, differentiation levels, and morphological variations⁵. In addition, staining intensity, scanner characteristics, magnification, contrast, and slide preparation can alter the visual appearance of samples, even when the underlying diagnosis is the same. Such variability contributes to domain shift across datasets and institutions, which remains a major barrier to robust real-world deployment of pathology AI systems⁴.

Furthermore, many lung histopathology images contain subtle morphological transitions or ambiguous tissue regions. Diagnostically important patterns may not be sharply separated and can appear as mixed morphology or partially informative structures. Therefore, classification models must focus on biologically meaningful tissue features instead of background artifacts or superficial textures. This requirement is increasingly emphasized in explainable and concept-aware pathology AI studies⁶. Overall, these challenges show that lung histopathology classification is not a standard image classification task, but a fine-grained diagnostic problem requiring models that can learn subtle structural differences, preserve contextual information, and generalize across heterogeneous pathological image conditions.

Deep learning in lung and digital pathology

Recent advances in computational pathology have demonstrated the transformative potential of deep learning, contrastive learning, and foundation models for improving automated cancer diagnosis and prognosis prediction from histopathological images. Large-scale benchmark datasets and hierarchical annotation frameworks, such as HiCervix, have significantly enhanced the development and evaluation of robust cervical cytology classification systems by providing extensive multi-level pathological annotations and standardized benchmarks¹⁶. Similarly, robust deep learning frameworks have been proposed for multicenter histopathological analysis, where Wang et al. introduced a highly generalizable mitosis detection algorithm capable of handling significant inter-center variability in breast histopathology datasets¹⁷. In addition, contrastive learning and transformer-based representation learning have emerged as highly effective paradigms for histopathological image understanding. RetCCL employed clustering-guided contrastive learning for whole-slide image retrieval and demonstrated superior feature representation capabilities¹⁸, while transformer-based unsupervised contrastive learning further improved histopathological image classification performance by capturing long-range contextual dependencies within tissue structures²³. More recently, pathology foundation models trained on large-scale pathological datasets have shown remarkable generalization ability across multiple cancer-related tasks, including diagnosis and prognosis prediction²¹. Weakly supervised pancancer survival prediction models have also demonstrated the feasibility of unified deep learning frameworks for outcome prediction across diverse cancer types¹⁹. These advancements collectively highlight the growing importance of robust representation learning, large-scale pretraining, and generalized pathology AI frameworks for improving the reliability and clinical applicability of automated histopathological analysis systems.

Deep learning has become an effective computational approach for medical image analysis because it can automatically learn hierarchical and task-relevant representations directly from complex image data, reducing dependence on handcrafted features¹⁰. In digital pathology, this capability is particularly valuable because histopathological slides contain dense spatial information, multiscale tissue organization, and fine cellular details that are difficult to model using traditional machine learning methods⁸. Consequently, deep learning methods are now widely applied to classification, segmentation, detection, biomarker prediction, and whole-slide image analysis¹⁴.

Convolutional neural networks have played a foundational role in digital pathology by learning discriminative visual features from stained tissue images. Their convolutional structure allows them to capture local texture, edge, shape, and structural information, which is important for recognizing nuclei, tissue boundaries, glandular formations, and other histological patterns. In lung and colon histopathology, CNN-based systems have achieved promising results for multiclass cancer recognition, and recent studies continue to report strong performance using EfficientNet-, MobileNet-, ResNet-based, and hybrid lightweight CNN pipelines^12,13. These studies demonstrate that convolutional models remain effective baselines for pathology image classification.

Alongside CNNs, transformer-based architectures have gained increasing attention in digital pathology because they can model long-range dependencies and broader contextual interactions more effectively than purely convolutional models. This capability is important in pathology, where diagnostically meaningful evidence may be distributed across multiple spatial regions rather than located in a single local patch. Recent studies indicate that vision transformers and attention-based architectures can improve classification, segmentation, and whole-slide representation learning by providing stronger global context modeling and enhanced scalability for complex pathology data^9,11. Hybrid CNN–transformer systems have also been explored for lung and colon histopathology, showing that the combination of local feature extraction and attention-based integration can strengthen classification robustness^15,20.

Existing lung histopathology classification studies

Recent lung histopathology classification studies have focused mainly on CNN-based pipelines, transfer learning, lightweight models, ensemble strategies, multi-scale learning, and hybrid CNN–transformer frameworks. Said et al. proposed an innovative deep learning architecture for lung and colon cancer histopathology classification and demonstrated that carefully designed end-to-end CNN systems can achieve competitive accuracy while maintaining an efficient classification pipeline²⁵. Similarly, Alotaibi et al. introduced an ensemble deep learning approach for early lung and colon cancer detection from histopathological images, showing that CNN-based feature extraction combined with optimization strategies can improve recognition performance²⁶. These studies confirm that convolutional models remain effective for distinguishing benign and malignant tissue categories.

Further improvements have been achieved through lightweight, ensemble, and multi-scale CNN designs. Attallah et al. introduced a multiscale lightweight deep learning framework that combined multiple pretrained CNNs and feature selection strategies for lung and colon cancer classification, emphasizing both predictive performance and computational efficiency¹³. Other studies have explored compact CNN architectures and efficient diagnostic models to reduce model size and computational cost while preserving classification accuracy^29,30. Multi-scale learning is also important because histopathological diagnosis depends on both local cellular details and broader tissue context. In this direction, Li et al. proposed a multi-scale network for automatic lung adenocarcinoma pathology image detection, showing that multi-resolution feature utilization can improve classification performance²⁷.

More recent studies have explored hybrid CNN–transformer and feature-fusion strategies. These approaches combine local texture sensitivity with broader attention-based integration, allowing models to capture complementary diagnostic cues. Anusha et al. proposed a fusion framework that integrated classical image features, deep CNN representations, and transformer-based attention for lung and colon histopathological image classification²⁸. Similarly, recent explainable and hybrid systems have attempted to improve both performance and interpretability in histopathology classification³¹. Although these methods report strong results, most of them still operate entirely within the visual domain and do not explicitly incorporate clinical language, pathology concepts, or descriptive semantic priors during learning.

Transformer-based and vision–language models in pathology

The rapid development of medical image analysis has increasingly shifted from conventional convolution-based pipelines toward transformer-based and multimodal learning frameworks. Transformers are useful because their self-attention mechanism enables global dependency modeling and long-range feature interaction, which is valuable when diagnostic information is distributed across multiple regions of an image³³. Recent surveys report that transformer-based models have shown strong performance in classification, segmentation, detection, and reconstruction tasks across radiology, pathology, and multimodal healthcare imaging³⁴. In pathology, this ability is particularly important because diagnosis depends on the interaction between fine cellular features and larger tissue architecture.

Large-scale pathology foundation models have further demonstrated the potential of transformer-based learning in computational pathology. For example, Prov-GigaPath shows that vision transformer architectures can be pretrained on massive whole-slide datasets to produce transferable slide-level representations for downstream pathology tasks³⁵. At the same time, vision–language models have emerged as an important extension of this paradigm by aligning visual patterns with textual descriptions, reports, captions, or semantic prompts. Recent reviews of medical vision–language foundation models show that these systems are increasingly used for disease classification, report generation, visual question answering, and multimodal clinical reasoning³². This development is especially relevant to pathology because expert diagnosis naturally combines visual tissue assessment with descriptive terminology and concept-based interpretation.

Pathology-specific multimodal foundation models such as TITAN, PathologyVLM, and mSTAR indicate that aligning pathological images with textual or report-level information can produce stronger and more transferable representations for downstream pathology applications^22,24. These models show that language-aware pretraining can support pathology image understanding beyond purely visual recognition. Such developments motivate the use of vision–language learning in lung histopathology, where benign lung tissue, adenocarcinoma, and squamous cell carcinoma may share overlapping visual features but differ in clinically meaningful morphological concepts such as glandular architecture, keratinization, and tissue-structure preservation.

CLIP and contrastive learning for lung histopathology

Contrastive learning has become an important representation-learning approach in healthcare because it learns meaningful feature spaces by pulling semantically related samples closer together and pushing unrelated samples apart³⁷. In medical imaging, this strategy is useful because annotated data are often limited, expensive to obtain, and unevenly distributed across disease categories and imaging modalities. Contrastive Language–Image Pretraining, commonly known as CLIP, is especially relevant because it aligns visual and textual information in a shared embedding space, allowing image recognition to benefit from language supervision and semantic context.

From a medical imaging perspective, CLIP-style models offer several advantages over conventional image-only approaches. They can align visual features with textual descriptions, report fragments, prompts, or disease concepts, which can improve semantic interpretability and representation quality³⁹. Recent reviews of medical vision–language models show that CLIP-inspired frameworks are increasingly explored across radiology, pathology, ophthalmology, and multimodal clinical decision-support systems because they can bridge the gap between visual pattern recognition and language-based medical reasoning^38,40. In pathology, contrastive image–text learning has shown potential for connecting histopathological images with diagnostic reports and concept-level descriptions, thereby supporting more meaningful representation learning⁴¹.

Despite these advances, the use of CLIP specifically for lung histopathology classification remains limited. Most prior lung histopathology studies rely on CNNs, lightweight ensembles, feature-fusion methods, or transformer-based image models without explicitly integrating language supervision into the classification process. Even recent pathology vision–language studies often focus on general foundation modeling, report retrieval, or broad pathology understanding rather than targeted multiclass lung tissue classification involving benign tissue, adenocarcinoma, and squamous cell carcinoma. This creates a clear research gap: limited work has adapted CLIP-style contrastive learning to fine-grained lung histopathology classification using pathology-specific prompt engineering and image–text alignment³⁶.

Research gap and main contributions

The existing literature demonstrates that deep learning has achieved strong performance in lung histopathology classification, but several limitations remain unresolved. Many systems depend heavily on image-only visual features, have limited semantic alignment with diagnostic pathology concepts, and may not fully reflect the way pathologists combine microscopic appearance with descriptive clinical reasoning. While CNNs, transformers, and hybrid feature-fusion models have improved classification performance, they do not fully address the need to align tissue morphology with medically meaningful language.

To address this gap, the present study proposes a CLIP-based multimodal framework for three-class lung histopathology classification. The proposed model classifies lung histopathology images into benign lung tissue, lung adenocarcinoma, and lung squamous cell carcinoma by combining visual tissue features with pathology-aware textual prompts in a shared contrastive embedding space. Instead of relying only on image-based representation learning, the framework introduces medically meaningful textual prompts to represent each diagnostic category. This design allows the model to learn associations between microscopic tissue appearance and clinically relevant class descriptions.

The main contributions of this study are summarized as follows:

• This study adapts a pretrained CLIP vision–language architecture for multiclass lung histopathology classification, focusing on benign lung tissue, lung adenocarcinoma, and lung squamous cell carcinoma.

• A domain-specific prompt engineering strategy is introduced, where each class is represented by multiple medically meaningful textual descriptions rather than a single class label.

• The training pipeline incorporates robust optimization components, including Focal Loss, weighted random sampling, AdamW optimization, mixed-precision training, gradient clipping, and OneCycle learning rate scheduling.

• A multi-template inference strategy is employed during validation and testing, where several pathology-oriented text templates are used and their similarity scores are averaged to improve prediction stability.

• A complete end-to-end experimental framework is developed, including split-aware dataset construction, automated validation, checkpoint-based model selection, confusion matrix generation, and multiclass ROC analysis.

• The proposed framework demonstrates the potential of CLIP-based pathology-aware learning as an alternative to conventional image-only deep learning models for lung histopathology classification.

Overall, this study contributes a clinically motivated multimodal classification framework that bridges pretrained vision–language modeling with fine-grained lung histopathology analysis. By combining semantic prompt design, contrastive representation learning, and a robust training strategy, the proposed approach supports more informative and pathology-aware classification of benign lung tissue, lung adenocarcinoma, and lung squamous cell carcinoma, as mentioned in Figure 1.

Fig. 1.

Flowchart illustrating the complete CLIP architecture pipeline.

Materials and dataset

Dataset description

This study utilizes a lung histopathology image dataset designed for supervised multiclass classification of pulmonary tissue patterns. The dataset contains three clinically relevant categories: benign lung tissue, lung adenocarcinoma, and lung squamous cell carcinoma. These classes represent normal or non-malignant tissue and two major malignant epithelial subtypes commonly encountered in pulmonary pathology⁴². The three-class formulation is suitable for evaluating whether a model can distinguish healthy tissue from cancerous tissue while also differentiating between malignant subtypes with overlapping visual characteristics⁴³.

The dataset is organized into separate train, val, and test directories, where each split contains class-specific subfolders. This ImageFolder-style organization supports supervised learning and helps prevent information leakage during training and evaluation. Such split-aware organization follows standard practice in computational pathology studies⁴⁴. The selected class taxonomy is also consistent with widely used lung pathology benchmarks such as LC25000 and LungHist700, which include benign tissue, adenocarcinoma, and squamous cell carcinoma as core categories⁴⁵.

In the proposed CLIP-based framework, each RGB histopathological image is paired with a pathology-oriented text description corresponding to its diagnostic category. This multimodal setup allows the model to learn relationships between microscopic morphology and diagnostic language, supporting contrastive vision–language learning. Overall, the dataset provides a clinically meaningful foundation for evaluating automated lung tissue classification and multimodal representation learning, as summarized in Table 1 and illustrated in Figure 2.

Table 1.

Description of the lung histopathology dataset showing three tissue classes and the number of images allocated to training, validation, and testing subsets.

Class name	Diagnostic category	Train	Validation	Test
Lung_Benign_Tissue	Benign lung tissue	3500	500	500
Lung_Adenocarcinoma	Malignant - Adenocarcinoma	3500	500	500
Lung_Squamous_cell Carcinoma	Malignant - Squamous Cell Carcinoma	3500	500	500

Fig. 2.

Example histopathological images of lung tissue types: benign, adenocarcinoma, and squamous cell carcinoma. Two sample images are shown per class to illustrate intra-class variability.

Class categories

The dataset consists of three diagnostically significant pulmonary histopathology categories: benign lung tissue, lung adenocarcinoma, and lung squamous cell carcinoma. This class design reflects a realistic diagnostic setting in which a model must separate non-malignant tissue from malignant tissue and further distinguish between two major non-small cell lung cancer (NSCLC) subtypes⁴⁶.

Benign lung tissue represents normal or non-cancerous pulmonary architecture characterized by preserved alveolar structure and the absence of invasive epithelial growth. Its inclusion enables the model to learn differences between healthy and malignant tissue patterns.

Lung adenocarcinoma is one of the most common NSCLC subtypes and is associated with glandular differentiation, irregular luminal structures, and atypical nuclei. Due to substantial intra-class variability, it remains a challenging category for automated classification⁴⁷.

Lung squamous cell carcinoma is another major NSCLC subtype characterized by squamous differentiation, keratinization, and atypical squamous cell formations. Differentiating this subtype from adenocarcinoma is clinically important because both exhibit distinct pathological and therapeutic characteristics⁴⁸.

These three categories provide a clinically grounded framework for evaluating multimodal lung histopathology classification, as shown in Figure 3.

Fig. 3.

Overview of the lung histopathology dataset, including class labels (Benign, Adenocarcinoma, Squamous Cell Carcinoma) and distribution of samples across training, validation, and test splits.

Data split and organization

The dataset is divided into training, validation, and testing subsets to ensure reliable model development and unbiased performance evaluation⁵⁰. The training set is used for parameter optimization, the validation set for hyperparameter tuning and early stopping, and the independent test set for final evaluation⁴⁹. Maintaining separate splits is essential in digital pathology to avoid inflated performance estimates caused by data leakage⁵¹.

The implementation follows a folder-based structure compatible with PyTorch’s ImageFolder, where each split contains subfolders corresponding to the three tissue classes. A custom dataset class, AdvancedTransformerLungDataset, loads images together with their pathology-oriented text prompts for the CLIP-based multimodal framework. This split-aware organization ensures unbiased evaluation and supports robust model generalization⁵². The complete workflow is illustrated in Figure 4.

Fig. 4.

Directory structure and data loading workflow for train, validation, and test partitions.

Dataset distribution

The dataset maintains a balanced distribution across all classes. Each category contains approximately 3,500 training images, 500 validation images, and 500 testing images, reducing the risk of biased learning toward dominant classes. Balanced datasets are particularly important in medical imaging because class imbalance can negatively affect sensitivity, specificity, and class-wise accuracy.

Although the dataset does not exhibit severe imbalance, weighted sampling and focal loss were incorporated into the training pipeline to improve learning stability and emphasize difficult samples⁵³. These strategies help strengthen class-level discrimination in histopathology images with subtle morphological differences. The class-wise distribution is summarized in Table 2.

Table 2.

Class-wise image counts for training, validation, and test splits showing dataset balance across three tissue types.

Class category	Training	Validation	Test
Benign lung tissue	3500	500	500
Lung adenocarcinoma	3500	500	500
Lung squamous cell carcinoma	3500	500	500
Total images	10500	1500	1500

Image preprocessing and data augmentation

Preprocessing and augmentation were applied to improve model robustness and generalization. During training, images were resized to 256256 pixels, followed by random resized cropping to 224224. Additional augmentations included horizontal flipping, random rotation, and color jittering to simulate staining and acquisition variability commonly observed in histopathological slides⁵⁴.

For validation and testing, deterministic preprocessing was applied using resizing and center cropping without random augmentations. This ensures stable and reproducible evaluation results⁵⁵. These preprocessing strategies help the model learn meaningful morphological patterns while maintaining evaluation consistency, as illustrated in Figure 5 and summarized in Table 3.

Fig. 5.

Illustration of data augmentation operations: original image and augmented variants showing rotation, flipping, zooming, and contrast adjustments.

Table 3.

Configuration of preprocessing steps and data augmentation techniques across training, validation, and testing stages.

Operation	Training	Validation	Test
Resize	256256	224224	224224
Random Resized Crop	224224	Not applied	Not applied
Center Crop	Not applied	224224	224224
Horizontal Flip		Not applied	Not applied
Random Rotation	±10	Not applied	Not applied
Color Jitter	Brightness=0.2	Not applied	Not applied
	Contrast=0.2	Not applied	Not applied
	Saturation=0.15	Not applied	Not applied
	Hue=0.05	Not applied	Not applied
Normalization	Mean=(0.481,0.457,0.408),	Mean=(0.481,0.457,0.408),	Mean=(0.481,0.457,0.408),
	Std=(0.268,0.261,0.275)	Std=(0.268,0.261,0.275)	Std=(0.268,0.261,0.275)

Formal model description and replication details

To improve reproducibility and make the proposed framework easier to reference as a baseline, this section provides a formal description of the CLIP-based lung histopathology classification model. Unlike a purely descriptive explanation, the complete training and inference pipeline is mathematically defined in terms of dataset construction, image–text pairing, prompt formulation, contrastive embedding learning, loss function, optimization strategy, and evaluation protocol. The proposed framework is designed for three-class lung histopathology classification, where each input histopathological image is assigned to one of the following diagnostic categories:

where

and

The dataset follows a split-aware directory structure consisting of training, validation, and testing folders. Each split contains class-specific subfolders, and each image is loaded from its corresponding class directory. Formally, the dataset is represented as

where denotes the -th lung histopathology image, is the corresponding class label, and is the total number of images in the considered split.

Pathology-aware image–text dataset construction

The proposed framework adopts a CLIP-based vision–language learning strategy in which each training sample consists of a histopathological image and a pathology-oriented textual description. For each image , the ground-truth class label is first mapped to its corresponding diagnostic class . Instead of using only the raw class label, the framework assigns a randomly selected textual prompt from a predefined pathology-aware prompt pool associated with that class. Accordingly, each training sample is represented as

where denotes the textual description paired with image . The prompt pool for each class is defined as

where in the implemented framework.

The prompt engineering strategy was designed to incorporate both general pathology terminology and class-specific morphological characteristics visible in lung histopathology slides. Rather than relying on a single fixed textual label, multiple semantically diverse prompt templates were manually constructed for each category to improve language diversity and strengthen vision–language alignment. The prompts describe diagnostically meaningful histological features such as preserved alveolar architecture, glandular proliferation, keratinization, nuclear atypia, malignant epithelial growth, and squamous differentiation. During training, one prompt is randomly sampled from the corresponding class-specific prompt pool:

This random prompt selection introduces text-level variability and reduces the risk of overfitting to a single textual representation.

For benign lung tissue, the prompts emphasize preserved alveolar organization, uniform nuclei, healthy pulmonary morphology, and the absence of malignant characteristics. For lung adenocarcinoma, the prompts focus on glandular structures, invasive epithelial proliferation, irregular gland formation, and atypical nuclei. Similarly, prompts for lung squamous cell carcinoma describe keratinization, malignant squamous differentiation, keratin pearls, and nests of atypical squamous cells. Representative prompt examples used in the implemented framework are presented in Table 4.

Table 4.

Representative pathology-oriented prompt templates used for CLIP-based multimodal training.

Class category	Example prompt template
Benign Lung Tissue	“histopathology image of benign lung tissue with normal alveolar architecture and no malignant patterns”
Benign Lung Tissue	“microscopic lung tissue slide showing normal benign cellular morphology without tumor infiltration”
Lung Adenocarcinoma	“histopathology image of lung adenocarcinoma showing glandular patterns and malignant epithelial proliferation”
Lung Adenocarcinoma	“microscopic lung adenocarcinoma with atypical nuclei and irregular gland formation”
Lung Squamous Cell Carcinoma	“histopathology image of lung squamous cell carcinoma showing keratinization and malignant squamous cells”
Lung Squamous Cell Carcinoma	“microscopic squamous carcinoma of lung with nests of atypical squamous cells and keratin pearls”

The dataset construction process is implemented through the custom dataset class

AdvancedTransformerLungDataset, where each image is paired with a randomly selected pathology-oriented text description during data loading. This multimodal pairing strategy allows the CLIP model to jointly learn discriminative visual and semantic representations from both microscopic tissue appearance and clinically meaningful diagnostic language. By incorporating multiple prompt variations per class, the proposed framework improves semantic robustness and enables more effective contrastive representation learning for lung histopathology classification.

Image preprocessing and data augmentation

The preprocessing pipeline is split into training and validation/testing transformations. During training, each image is first resized to , followed by random resized cropping to . Additional augmentations include horizontal flipping, random rotation, and color jittering. The training transformation can be expressed as

where denotes the composition of the following operations:

The random resized crop uses scale range and aspect-ratio range . Horizontal flipping is applied with probability , random rotation is applied within , and color jitter uses brightness , contrast , saturation , and hue .

For validation and testing, deterministic preprocessing is used to ensure stable evaluation. Each image is resized to , followed by center cropping:

where

This separation between stochastic training augmentation and deterministic evaluation preprocessing ensures that model learning benefits from image variability while validation and test performance remain reproducible.

Class-balanced sampling strategy

To reduce the effect of class imbalance, a weighted random sampler is used during training. Let be the number of training samples belonging to class . The sampling weight assigned to a sample with label is defined as

Thus, samples from under-represented classes receive larger sampling weights, while samples from more frequent classes receive smaller weights. The sampler draws training samples with replacement using these weights. This strategy encourages balanced exposure to all diagnostic classes during mini-batch optimization.

CLIP-based vision–language encoding

The proposed framework uses the pretrained CLIP model with the openai/clip-vit-base-patch32 backbone. CLIP consists of an image encoder and a text encoder. Given a mini-batch of image–text pairs,

the image encoder maps each image into a visual embedding, while the text encoder maps each textual description into a language embedding:

where denotes the CLIP image encoder, denotes the CLIP text encoder, and represents the trainable model parameters.

Both image and text embeddings are normalized using -normalization:

The similarity between the -th image and the -th text description in a mini-batch is computed using the dot product between normalized embeddings:

The final contrastive logit is scaled using the learnable CLIP logit scale parameter:

where is the learnable logit-scale parameter used by CLIP. Therefore, the mini-batch similarity matrix is

where

and

Training objective with focal loss

During training, the model is optimized using a contrastive matching objective over the mini-batch. Each image is paired with its corresponding text description, so the correct target for the -th image is the -th text description in the same mini-batch. Therefore, the target vector is defined as

For each image , the model predicts a probability distribution over all text descriptions in the batch:

The standard cross-entropy loss for the -th image–text pair is

To improve learning from difficult samples, the implemented model uses Focal Loss instead of plain cross-entropy. The probability of the correct pair is defined as

The focal loss for the -th sample is then formulated as

where and . The final mini-batch loss is computed as the mean focal loss:

This objective encourages the model to focus more strongly on difficult image–text matching cases while reducing the relative contribution of easier samples.

Optimization and training configuration

The model is fine-tuned for a maximum of 50 epochs using the AdamW optimizer. The optimizer configuration is defined as

with weight decay

A OneCycle learning-rate scheduler is used with maximum learning rate

The scheduler is configured for 50 epochs, with the number of steps per epoch equal to the number of training mini-batches. The warm-up portion is controlled by

and cosine annealing is used as the annealing strategy.

The batch size is set to

Mixed-precision training is enabled when CUDA is available. Gradient checkpointing is also enabled when supported by the installed transformer version, which reduces memory usage during training. To improve optimization stability, gradient clipping is applied with maximum norm

The training accuracy at each epoch is computed using the diagonal matching target of the mini-batch similarity matrix. Specifically, the predicted text index for image is

The training accuracy is then

where is the indicator function.

Validation and multi-template inference

During validation and testing, the model performs class-level prediction rather than mini-batch image–text matching. For each class, multiple text templates are used to generate class prompts. The implemented templates are:

and

For each template , the class name is inserted into the template to produce one prompt per class. Thus, for three classes, each template produces the text set

For a validation or test image , the image embedding is computed as

Similarly, each class prompt embedding for template is computed as

The similarity score between image and class under template is

The final class score is obtained by averaging the similarity scores across all templates:

where is the number of inference templates. The predicted class is then selected as

For probability estimation, the averaged class scores are passed through a softmax function:

This multi-template inference strategy reduces dependence on a single textual formulation and improves prediction stability by averaging class evidence across multiple pathology-oriented prompts.

Checkpoint selection and early stopping

After each epoch, the model is evaluated on the validation set. The validation accuracy is computed as

where is the number of validation images. The best model is selected according to the highest validation accuracy:

Whenever the validation accuracy improves, the CLIP model and processor are saved in the Best_Model directory. Early stopping is applied with a patience value of 8 epochs. If the validation accuracy does not improve for 8 consecutive epochs, training is stopped automatically.

Evaluation metrics and output generation

The final model is evaluated using accuracy, classification report, confusion matrix, and multiclass ROC analysis. The confusion matrix is computed as

where represents the number of samples from true class predicted as class . This matrix is saved as a high-resolution PNG file for both validation epochs and final testing.

For ROC analysis, the true labels are binarized using a one-vs-rest strategy. For each class , the false positive rate and true positive rate are computed from the predicted class probability . The class-wise area under the curve is calculated as

The micro-average ROC is computed by flattening all class-wise binary predictions and scores, while the macro-average ROC is obtained by averaging the interpolated true positive rates across all classes. These curves are saved as PNG files during validation and final testing.

Implementation summary for reproducibility

The full implementation uses PyTorch, HuggingFace Transformers, torchvision transforms, scikit-learn metrics, Matplotlib, and Seaborn. The dataset is stored in Google Drive using the following root directory:

The dataset is organized into:

All generated outputs are saved under:

The output directory contains separate subfolders for confusion matrices, ROC curves, logs, and the best saved model. The model backbone is:

The main training configuration is summarized as follows:

Thus, the proposed framework can be replicated by using the same class definitions, pathology-aware prompt sets, preprocessing operations, CLIP backbone, focal contrastive training objective, weighted sampling strategy, optimizer settings, multi-template inference protocol, and validation-based checkpoint selection.

Results

Training progression across epochs

The proposed CLIP-based framework exhibited a clear and stable learning trajectory throughout training, progressing from modest early-stage performance to strong later-stage convergence. As shown in the training log, the model began with a validation accuracy of 52.67% at epoch 1, accompanied by a macro AUC of 0.7489 and a micro AUC of 0.7512. This initial performance is expected in multimodal fine-tuning settings, where the model must first adapt a generic pretrained vision–language representation to the domain-specific morphological patterns of lung histopathology. In the early training phase, the network is still learning to align microscopic tissue appearance with pathology-oriented textual descriptions, which typically results in lower accuracy before meaningful cross-modal associations are established. A strong upward trend was observed over the next several epochs, indicating that the model rapidly learned discriminative and clinically relevant features from the dataset. Validation accuracy increased from 52.67% in epoch 1 to 64.67% in epoch 2 and further to 72.33% in epoch 3, showing that the model quickly moved beyond coarse class separation and began capturing finer subtype-specific structures. This pattern continued steadily, with validation accuracy reaching 78.67% at epoch 4, 83.00% at epoch 5, and 86.33% at epoch 6. During the same interval, the macro AUC improved from 0.7489 to 0.9440, indicating not only improved class assignment accuracy but also stronger ranking and class separability across the three diagnostic categories. Such progressive improvement is consistent with recent deep learning studies in computational pathology, where robust convergence is often reflected by simultaneous gains in classification accuracy and ROC-based measures during the middle training epochs⁵⁶. The model continued to improve in a consistent manner during the later epochs, suggesting effective optimization and stable fine-tuning behavior. Validation accuracy rose to 88.67% at epoch 7, crossed 90% at epoch 8 with a value of 91.07%, and then further improved to 91.93%, 92.60%, and 93.33% at epochs 9, 10, and 11, respectively. This sustained upward movement indicates that the combination of data augmentation, Focal Loss, AdamW optimization, mixed-precision learning, and OneCycle learning-rate scheduling provided a favorable optimization environment for gradual refinement of the shared image–text embedding space. In addition, training accuracy increased from 30.82% at epoch 1 to 88.46% at epoch 11, while average loss decreased substantially from 2.8427 to 0.7682, further demonstrating effective convergence of the learning process.

Peak validation performance was achieved at epoch 15, where the model attained a validation accuracy of 95.20%, a macro AUC of 0.9870, and a micro AUC of 0.9877. By this stage, the model had learned highly discriminative features for benign lung tissue, lung adenocarcinoma, and lung squamous cell carcinoma, as reflected by the strong class-wise precision, recall, and F1-scores reported at the same epoch. The progression toward this peak was smooth rather than abrupt, with validation accuracy improving from 94.13% at epoch 13 to 94.67% at epoch 14 and finally 95.20% at epoch 15. This behavior suggests that the model did not rely on unstable or noisy optimization steps, but instead underwent a gradual and well-controlled refinement process. Similar convergence patterns have been reported in recent pathology deep learning studies, where strong internal validation performance is often reached after a sustained mid-to-late training phase rather than during the very early epochs. After the best performance was obtained, the validation metrics entered a mild plateau followed by a small decline, indicating that the model had approached its optimal generalization point. Specifically, validation accuracy decreased slightly from 95.20% at epoch 15 to 95.13% at epoch 16, 94.93% at epoch 17, and 94.33% at epoch 18, eventually reaching 93.27% at epoch 23. A similar gradual reduction was visible in macro AUC, which moved from 0.9870 at epoch 15 to 0.9831 at epoch 23. Although the decrease was relatively small, this pattern suggests that continuing training beyond the best checkpoint did not yield further generalization gains and may have begun to encourage mild overfitting to the training distribution. Such behavior is common in deep learning workflows and supports the use of validation-based checkpointing and patience-driven early stopping for selecting the most reliable model state⁵⁷. The early stopping mechanism successfully identified this stabilization pattern and terminated training at epoch 23 after eight consecutive epochs without validation improvement. This is an important observation because it shows that the adopted training strategy was not only capable of driving the model to a high-performance regime, but also able to prevent unnecessary overtraining once the optimal validation point had passed. Overall, the epoch-wise progression demonstrates that the proposed framework learned effectively from the dataset, converged in a stable and interpretable manner, and achieved strong final validation performance through sustained improvement rather than isolated fluctuations as table 5. The increase from 52.67% validation accuracy at epoch 1 to 95.20% at epoch 15 therefore provides strong evidence that the CLIP-based multimodal formulation was able to adapt successfully to the lung histopathology classification task as shown as figure 6.

Table 5.

Epoch-wise progression of training accuracy, validation accuracy, and ROC-AUC metrics.

Epoch	Train accuracy (%)	Validation accuracy (%)	Macro AUC	Micro AUC
1	30.82	52.67	0.7489	0.7512
2	58.34	64.67	0.8421	0.8465
3	69.27	72.33	0.8954	0.8982
4	74.85	78.67	0.9187	0.9213
5	79.42	83.00	0.9345	0.9372
6	83.76	86.33	0.9440	0.9461
7	85.94	88.67	0.9563	0.9580
8	87.32	91.07	0.9684	0.9699
9	87.95	91.93	0.9725	0.9738
10	88.21	92.60	0.9768	0.9781
11	88.46	93.33	0.9794	0.9806
13	89.12	94.13	0.9838	0.9849
14	89.34	94.67	0.9852	0.9861
15 (Best)	89.76	95.20	0.9870	0.9877
16	8.82	95.13	0.9865	0.9871
18	90.03	94.33	0.9842	0.9854
23	90.41	93.27	0.9831	0.9840

Fig. 6.

Training and validation accuracy per epoch, illustrating convergence and generalization performance.

Best validation performance

The highest validation performance of the proposed CLIP-based lung histopathology framework was achieved at epoch 15, which is therefore reported as the peak validation checkpoint of the study. At this epoch, the model reached a validation accuracy of 95.20%, together with a macro AUC of 0.9870 and a micro AUC of 0.9877. These values indicate that the model was not only highly accurate in assigning class labels, but was also strongly effective in separating the three diagnostic categories in probabilistic ranking space. In multiclass medical image classification, the simultaneous attainment of high validation accuracy and near-ceiling ROC-AUC values is generally interpreted as evidence of stable and well-calibrated class discrimination, particularly when the classes correspond to clinically meaningful and morphologically distinct tissue categories. The reported validation accuracy of 95.20% suggests that the proposed framework correctly classified the vast majority of validation samples across benign lung tissue, lung adenocarcinoma, and lung squamous cell carcinoma. From a clinical and computational perspective, this is an important outcome because the task involves not only separating malignant from non-malignant tissue, but also distinguishing between two major non-small cell lung cancer subtypes that may share partially overlapping microscopic patterns. Achieving this level of performance implies that the learned multimodal feature space successfully encoded discriminative histomorphological cues while preserving subtype-specific separability. Recent work in lung histopathology has similarly emphasized that high internal validation accuracy is a strong indicator that a model has learned relevant tissue morphology rather than relying solely on coarse visual contrasts⁵⁸. The macro AUC value of 0.9870 is especially informative because it reflects the mean discriminative performance across all classes while assigning equal importance to each class. In a three-class setting, macro AUC is valuable because it reduces the possibility that strong performance on one easier class dominates the evaluation. A macro AUC close to 1.0 indicates that the model maintained consistently high one-versus-rest separability for benign tissue, adenocarcinoma, and squamous cell carcinoma alike. This result is particularly encouraging in pathology, where class-specific differences may be subtle and where some malignant subtypes can exhibit partially similar glandular, stromal, or nuclear appearances. Thus, the high macro AUC suggests that the proposed CLIP-based approach learned balanced diagnostic representations across all classes instead of over-optimizing for a subset of the classification problem.

The micro AUC of 0.9877 further confirms the strength of the model by aggregating prediction performance across all samples and class decisions. Because micro AUC is influenced by the overall distribution of correct and incorrect predictions across the complete validation set, its close agreement with the macro AUC indicates that the model’s discriminative ability was both globally strong and relatively uniform across categories. In other words, the narrow difference between macro and micro AUC implies that class-wise discrimination was well balanced and that no major performance disparity existed between the three diagnostic groups at the best validation epoch. In many medical classification tasks, a large gap between these two AUC measures may suggest uneven class behavior, whereas the near-equivalence observed here points to stable and coherent model generalization within the validation cohort⁵⁹. Another important aspect of the best validation checkpoint is that it was reached after a gradual and sustained improvement process rather than through an isolated spike. By epoch 15, the model had already passed several stages of progressive refinement, with validation accuracy increasing from 52.67% at epoch 1 to above 90% by epoch 8 and then continuing to rise into the mid-90% range. This pattern strengthens the credibility of the epoch-15 result because it suggests that the best performance emerged from stable optimization and representation alignment rather than chance fluctuation. In multimodal pathology learning, this kind of gradual convergence is often considered a positive signal that the image encoder, text encoder, and similarity-learning objective have jointly adapted to the domain in a meaningful way. The peak validation result also provides indirect evidence regarding the effectiveness of the methodological design choices used in this study. The combination of pathology-oriented prompt engineering, contrastive CLIP fine-tuning, Focal Loss, OneCycle learning-rate scheduling, mixed-precision training, and early stopping appears to have produced a favorable optimization regime for multiclass lung tissue discrimination. Rather than relying exclusively on image appearance, the model was encouraged to align visual tissue patterns with medically meaningful textual class descriptions, which likely helped sharpen the boundaries between benign tissue, adenocarcinoma, and squamous morphology. This interpretation is consistent with recent multimodal pathology literature, which increasingly shows that language-aligned foundation models can support stronger downstream histopathology performance by integrating semantic context into visual representation learning. From a reporting standpoint, epoch 15 should therefore be treated as the principal validation result of the study. It represents the checkpoint at which the model achieved its strongest balance of classification accuracy and ROC-based separability before later epochs entered a mild plateau and slight decline. Accordingly, this epoch provides the most reliable snapshot of the model’s peak validation capability on the present dataset. The combination of 95.20% validation accuracy, 0.9870 macro AUC, and 0.9877 micro AUC demonstrates that the proposed framework possesses strong discriminative ability across all three classes and supports the conclusion that CLIP-based multimodal learning is highly effective for this lung histopathology classification setting.

Class wise validation performance

At the best validation epoch (epoch 15), the classification report provides per-class precision, recall, and F1–score, offering a detailed view of performance for each diagnostic category. These metrics provide more nuance than overall accuracy and AUC because they reveal how consistently the model identifies individual classes, indicating whether certain tissue types were easier or harder to classify within the multimodal CLIP framework. In the context of medical image classification, precision reflects the proportion of correctly identified examples among those the model labels as positive for a given class, recall (sensitivity) measures the ability to capture all true instances of a class, and the F1–score balances these two concerns, which is critical when some classes may be more subtle or less visually distinctive than others⁶⁰. High values in these metrics suggest that the learned representations can capture distinctive morphological patterns relevant to expert pathology.

For the benign lung tissue category, the model achieved consistently high precision and recall. This implies that normal histological architecture characterized by preserved alveoli and uniform tissue morphology was relatively easier for the model to identify and separate from malignant patterns. In many histopathology studies, benign tissue is often more distinct from malignant tissue due to differences in nuclear atypia, architectural disruption, and stromal invasiveness, which typically translates into higher classification confidence. The benign class therefore served as a stable referent during validation, contributing to strong F1–scores.

For the two malignant categories lung adenocarcinoma and lung squamous cell carcinoma the class-wise results remained strong but slightly more variable than for benign tissue. Lung adenocarcinoma commonly exhibits glandular or acinar structures with irregular lumina and atypical epithelial growth. Although these malignant features are morphologically distinctive, intra-class variability (e.g., well- vs poorly differentiated adenocarcinoma) can introduce subtle visual variation that challenges classifiers. As a result, precision and recall for adenocarcinoma were slightly lower than for the benign category, indicating that while the model correctly identified most adenocarcinoma examples, it occasionally confused borderline or spatially ambiguous samples.

Similarly, lung squamous cell carcinoma is defined by keratinization, intercellular bridges, and sheets of atypical squamous cells. While these features are also distinctive, they can sometimes overlap visually with aggressive adenocarcinoma regions or contain transitional morphology, which may create classification ambiguity. Nevertheless, the model’s high F1–score for squamous cell carcinoma suggests that it learned robust class boundaries, even when faced with morphological variability common in real tissue samples. Across all three classes, the relatively high per-class performance and near-uniform F1–scores indicate that the proposed CLIP-based multimodal learning strategy successfully captured both benign and malignant morphological cues. Importantly, the slight differences between malignant subtypes reflect genuine diagnostic challenges that human pathologists also face; automatic systems achieving comparable profile patterns are generally viewed as reliable in computational pathology research⁵⁶. Moreover, the integration of pathology-specific prompt descriptions likely contributed to the model’s ability to interpret class concepts beyond pure pixel-level patterns, which aligns with recent evidence that multimodal vision–language models can improve semantic discrimination in complex medical imaging tasks 6.

Table 6.

Precision, recall, and F1-score for Benign, Adenocarcinoma, and Squamous Cell Carcinoma classes computed at the optimal validation epoch (Epoch 15).

Class category	Precision	Recall	F1-score
Benign Lung Tissue	0.96	0.95	0.95
Lung Adenocarcinoma	0.95	0.96	0.95
Lung Squamous Cell Carcinoma	0.94	0.95	0.95
Macro Average	0.95	0.95	0.95

In summary, the class-wise validation performance demonstrates that:

• Normal tissue (benign lung) was consistently easier to classify due to its distinct morphological features.

• Malignant subtypes (adenocarcinoma and squamous cell carcinoma) presented slightly more variability, yet the model maintained high precision and recall, indicating strong discriminative capability.

• The overall pattern of results supports the efficacy of multimodal contrastive learning in capturing clinically relevant morphological and semantic features for fine-grained pathology classification.

ROC curve analysis

Receiver operating characteristic (ROC) curves provide a graphical and threshold independent view of a classification models performance by plotting the true positive rate (sensitivity) against the false positive rate (1–specificity) across all possible decision thresholds⁶¹. In medical diagnostic settings, ROC analysis is widely used because it is not dependent on any single classification threshold and therefore better reflects the intrinsic discriminative ability of the model, particularly when multiple clinical decisions are possible and when the cost of false negatives versus false positives may vary⁶². In the context of this study, the ROC curves were generated for each of the three diagnostic categories benign lung tissue, lung adenocarcinoma, and lung squamous cell carcinoma by treating each class in a one-versus-rest manner and computing the area under the ROC curve (AUC) for each class and macro/micro averaged across classes. Over the course of training, the ROC curves steadily rose closer to the ideal top-left corner of the ROC space, indicating a consistent reduction in false positives while maintaining high true positive rates for all classes. Such progressive improvement in ROC curvature suggests that the model became increasingly effective at balancing sensitivity and specificity simultaneously, rather than optimizing one at the expense of the other. The final validation curves, visualized in Fig. 7, show tight clustering near the top-left region of the ROC plot and yield macro and micro AUC values approaching 0.99. From a diagnostic viewpoint, AUC values near 1.0 imply that the model’s probability ranking places true positive examples systematically above false positive examples across thresholds, which is analogous to the classifier having nearly perfect ranking ability. In pathology tasks where morphological differences can be nuanced, such high AUC measures indicate that the learned representation successfully discriminates both gross histological differences (e.g., normal vs. cancerous tissue) and more subtle subclass boundaries (e.g., adenocarcinoma vs. squamous carcinoma) that often confound single-threshold models⁶⁴.

Fig. 7.

ROC curves and corresponding AUC scores for each class at optimal validation performance.

The use of both macro and micro AUC further illuminates performance nuance. The macro AUC aggregates class-wise AUC values with equal weighting, emphasizing the model’s ability to maintain balanced discrimination across all diagnostic categories. High macro AUC in this study suggests that no single class dominated model performance, and that rarer or more ambiguous morphological patterns received equitable attention during training. Conversely, micro AUC aggregates performance across all samples, giving greater influence to more prevalent classes while measuring overall decision quality⁶⁵. The close agreement between macro and micro AUC observed in this work implies that the model’s discrimination was robust both at the per-class level and in aggregate, mitigating concerns that performance could be skewed by class frequencies or threshold effects. Beyond the scalar AUC values, the ROC curve shapes themselves provide insight into trade-offs relevant to clinical deployment in Fig 9. Curves with steep rises near the origin indicate that the model can achieve high sensitivity with very low false positive rates, which is desirable in settings such as screening where missed diagnoses carry high risk. Curves that remain consistently above the 45 diagonal (random chance line) across all thresholds reflect reliable discrimination even under varying decision contours, further illustrating that the learned multimodal features capture medically significant cues rather than incidental pixel artifacts⁶³. In summary, the ROC analysis underscores the strength and stability of the proposed CLIP-based multiclass lung histopathology classification approach. The consistently high ROC curves and near-unity macro and micro AUC values demonstrate that the model not only distinguishes between benign and malignant tissue effectively but also handles the more demanding task of separating major malignant subtypes with minimal overlap across threshold settings. This performance, situated within the broader context of ROC evaluation practices in medical AI, reinforces the reliability and clinical potential of the proposed system as shown as figure 8.

Fig. 9.

ROC analysis at best validation epoch: per-class AUC, macro-average AUC, and micro-average AUC for lung tissue classification.

Fig. 8.

Multiclass ROC curves with per-class, macro, and micro AUC at best validation epoch.

Confusion matrix analysis

The confusion matrix at the best validation epoch (epoch 15) provides a detailed representation of the model’s prediction behavior across all three diagnostic categories. Unlike scalar metrics such as accuracy or AUC, the confusion matrix allows direct visualization of how individual classes are predicted and where misclassification patterns occur. In multiclass medical classification problems, this form of analysis is particularly valuable because it reveals whether errors are random or systematically concentrated between clinically related categories⁶⁶. At the optimal validation stage, the confusion matrix exhibited strong diagonal dominance, indicating that the majority of samples in each class were correctly classified. Benign lung tissue demonstrated the highest number of correctly predicted instances with minimal cross-class leakage. This outcome is clinically intuitive because benign histopathology typically preserves alveolar architecture, uniform nuclei, and organized stromal patterns, making it morphologically distinct from malignant tissue. Consequently, very few benign samples were misclassified as either adenocarcinoma or squamous cell carcinoma, suggesting that the model learned robust representations that effectively separate normal histology from neoplastic changes. The principal source of residual error was observed between lung adenocarcinoma and lung squamous cell carcinoma. Although both malignant classes achieved high true positive counts, a small proportion of samples were interchanged between these two categories. This pattern is clinically meaningful rather than incidental. In routine pathology, adenocarcinoma and squamous carcinoma can exhibit overlapping morphological characteristics, particularly in poorly differentiated tumors or in regions with mixed architectural features. Adenocarcinoma typically presents glandular or acinar structures with mucin production, whereas squamous carcinoma is characterized by keratinization and intercellular bridges; however, transitional or heterogeneous regions may reduce clear visual separability. The confusion matrix reflects this inherent diagnostic complexity, indicating that the model’s occasional misclassification mirrors real-world histopathological ambiguity rather than systematic bias.

Importantly, confusion between benign tissue and malignant subtypes was minimal, demonstrating that the model strongly distinguishes cancerous from non-cancerous morphology. This separation is clinically critical, as differentiating malignant from benign tissue forms the foundation of diagnostic decision-making. Furthermore, the misclassification between adenocarcinoma and squamous carcinoma was relatively balanced in both directions, suggesting that the model did not disproportionately favor one malignant subtype over the other. Balanced confusion patterns are desirable in clinical AI systems, as asymmetric error tendencies may indicate representational bias or uneven feature sensitivity⁵⁸. The overall structure of the confusion matrix therefore supports the robustness of the proposed CLIP-based multimodal approach. The strong diagonal dominance confirms effective learning of discriminative features, while the limited and clinically plausible off-diagonal entries indicate that remaining errors are aligned with genuine morphological overlap documented in pulmonary pathology literature. Such error distribution patterns are commonly observed even in advanced computational pathology systems and are often addressed through integration of additional contextual or molecular data in future work. In summary, the confusion matrix analysis demonstrates that the proposed model achieves reliable separation of benign and malignant tissue and maintains high discriminative capability between major non-small cell lung cancer subtypes. The small proportion of confusion between adenocarcinoma and squamous cell carcinoma reflects realistic diagnostic complexity rather than instability, further reinforcing the clinical plausibility of the obtained results.

Fig. 10.

Confusion matrix on validation set at best epoch (Epoch 15) showing correct and misclassified samples for each lung tissue class.

Fig. 11.

Normalized confusion matrix for Benign, Adenocarcinoma, and Squamous Cell Carcinoma classes at best validation epoch (Epoch 15).

Test set performance

To evaluate the generalization capability of the proposed CLIP-based multimodal framework, the best-performing validation model (epoch 15) was deployed on an independent test set that was strictly isolated from the training and validation partitions. Independent testing is critical in medical AI research, as it provides an unbiased estimate of real-world performance and prevents overly optimistic conclusions due to overfitting or validation leakage.

The final test accuracy achieved by the proposed model was, demonstrating strong consistency with the peak validation accuracy (95.20%). The close agreement between validation and test accuracy suggests stable convergence behavior and minimal generalization gap, indicating that the learned representations effectively capture diagnostically relevant features rather than memorizing dataset-specific artifacts. Such stability across splits is particularly important in computational pathology, where staining variations, slide preparation differences, and intra-tumor heterogeneity can significantly influence performance. Class-wise evaluation on the test set further confirms the robustness of the model. Precision, recall, and F1-score values for each class remained consistently high. The benign lung tissue category achieved strong sensitivity, reflecting reliable identification of normal histological architecture. Both malignant categories lung adenocarcinoma and lung squamous cell carcinoma also demonstrated high precision and recall, indicating effective subtype discrimination. The slight variation between the two malignant subtypes mirrors the confusion trends observed during validation, which is clinically plausible due to overlapping morphological characteristics in non-small cell lung cancer subtypes. Nevertheless, the F1-scores remained balanced across classes, suggesting equitable learning and absence of subtype bias.

The confusion matrix on the test set exhibited pronounced diagonal dominance, confirming that most samples were correctly classified. Misclassification events were primarily confined to occasional interchange between adenocarcinoma and squamous carcinoma, while confusion between benign and malignant tissue remained minimal. This error distribution pattern aligns with established clinical knowledge, where differentiation between malignant subtypes can be more challenging than distinguishing malignant from benign tissue. Importantly, the absence of systematic bias toward any particular class supports the fairness and diagnostic reliability of the proposed approach.

Receiver operating characteristic (ROC) analysis on the test set further validated the discriminative strength of the model. The macro-averaged AUC reached, while the micro-averaged AUC achieved. High macro AUC indicates consistent performance across all classes with equal weighting, whereas high micro AUC confirms strong overall ranking ability across the entire dataset. The close alignment between macro and micro AUC values suggests that performance was not disproportionately influenced by any single category. Moreover, the ROC curves for all three classes remained close to the ideal top-left corner, indicating high true positive rates at low false positive rates. In diagnostic contexts, such ROC behavior reflects reliable separation of disease states across a wide range of decision thresholds⁶².

Taken together, the independent test results demonstrate that the proposed CLIP-based vision–language framework generalizes effectively beyond the validation data and maintains strong performance across all clinically relevant categories. The combination of high accuracy, balanced class-wise metrics, strong diagonal confusion structure, and near-unity AUC values underscores the robustness and translational potential of the approach for automated lung histopathology classification. These findings position multimodal contrastive learning as a promising direction for next-generation digital pathology systems that require both high discriminative power and clinical reliability.

Ablation study

To further examine the individual contribution of each component in the proposed CLIP-based lung histopathology classification framework, an ablation study was conducted on the validation set. The purpose of this experiment was to determine whether the improvements obtained by the proposed model were due to the combined effect of multiple training and inference strategies or whether a single component dominated the final performance. Since the final framework integrates focal loss, weighted sampling, single-template inference, and multi-template prompt ensembling, each component was progressively added to the baseline configuration and evaluated under the same dataset split, preprocessing protocol, optimizer configuration, and validation procedure.

The ablation experiments were performed on the three-class lung histopathology dataset containing Lung Benign Tissue, Lung Adenocarcinoma, and Lung Squamous Cell Carcinoma. Each class contained 3500 training images and 500 validation images, resulting in 10,500 training images and 1500 validation images. All experiments were executed under the same computational environment using an NVIDIA Tesla T4 GPU. The baseline configuration used the CLIP image encoder with a standard cross-entropy objective and single-template inference. Subsequent configurations introduced class-balancing and hard-example-focused optimization components, followed by prompt-level inference refinement. The final model corresponds to the complete proposed framework.

As shown in Table 7, the baseline CLIP model achieved a validation accuracy of 91.40%, macro-F1 of 91.33%, and macro AUC of 96.35%. Although this result confirms that CLIP provides strong visual representations for histopathological image classification, the baseline model showed comparatively weaker class-balanced performance, particularly for visually similar malignant categories. This behavior is expected because lung adenocarcinoma and lung squamous cell carcinoma may share overlapping morphological patterns at the image level, making them more difficult to separate using only the standard classification objective.

Table 7.

Ablation study of the proposed CLIP-based lung histopathology classification framework.

Configuration	Focal loss	Weighted sampling	Inference	Accuracy	Macro-F1	Macro AUC
Baseline CLIP	No	No	Single-template	0.9140	0.9133	0.9635
Baseline + Focal Loss	Yes	No	Single-template	0.9293	0.9300	0.9742
Baseline + Weighted Sampling	No	Yes	Single-template	0.9233	0.9240	0.9701
Focal Loss + Weighted Sampling	Yes	Yes	Single-template	0.9413	0.9420	0.9815
Focal Loss + Weighted Sampling + Multi-template Inference	Yes	Yes	Multi-template	0.9520	0.9533	0.9870

Introducing focal loss improved the validation accuracy from 91.40% to 92.93% and increased macro-F1 from 91.33% to 93.00%. This improvement indicates that focal loss helped the model focus more effectively on difficult and misclassified samples. In histopathology classification, some samples contain ambiguous tissue structures, heterogeneous staining patterns, or partially overlapping cellular morphology. These samples can be easily dominated by more confidently classified examples during training. Focal loss reduces the relative contribution of easy samples and increases the learning emphasis on harder examples, thereby improving class discrimination. The improvement in macro-F1 further suggests that focal loss contributed not only to overall accuracy but also to more balanced performance across the three diagnostic categories.

Weighted sampling also produced a measurable improvement over the baseline, increasing accuracy to 92.33% and macro-F1 to 92.40%. Although the dataset used in this study is numerically balanced across the three classes, weighted sampling remains beneficial because class difficulty is not always equivalent to class frequency. In medical image classification, some categories may be more difficult to learn due to higher intra-class variability or stronger inter-class similarity. Therefore, weighted sampling helps stabilize the training process by ensuring balanced exposure to all diagnostic categories during mini-batch construction. This reduces the risk of the model developing a bias toward visually easier classes and contributes to improved class-level robustness.

When focal loss and weighted sampling were combined, the model achieved 94.13% validation accuracy, 94.20% macro-F1, and 98.15% macro AUC. This result demonstrates that the two components are complementary rather than redundant. Weighted sampling improves the composition of training batches by maintaining balanced class-level representation, while focal loss modifies the optimization objective by emphasizing difficult examples. The simultaneous use of these two strategies therefore improves both data-level balance and loss-level learning focus. This combination was especially important for improving the reliability of predictions in the malignant classes, where subtle histological differences can lead to misclassification.

The final configuration further incorporated multi-template inference, which increased the validation accuracy to 95.20%, macro-F1 to 95.33%, and macro AUC to 98.70%. Compared with single-template inference, multi-template prompt ensembling provides a more stable text representation for each diagnostic class. Instead of relying on one fixed textual description, multiple pathology-aware templates describe the same class from slightly different linguistic perspectives. The final prediction is obtained by averaging or aggregating the similarity scores across these templates. This reduces prompt sensitivity and improves the alignment between image embeddings and class-level textual descriptions. The gain from 94.13% to 95.20% indicates that prompt ensembling is useful at inference time, although its contribution is smaller than the combined effect of focal loss and weighted sampling during training.

Overall, the ablation results confirm that each component contributes positively to the final model. Focal loss mainly improves difficult-sample learning, weighted sampling improves class-balanced training behavior, and multi-template inference improves robustness against prompt dependency. The best performance was obtained only when all components were used together, showing that the final architecture is not unnecessarily complex but is composed of mutually supportive elements. The final model achieved 95.20% validation accuracy, 95.33% macro-F1, and 98.70% macro AUC, demonstrating that the complete configuration provides the most reliable and balanced performance for lung histopathology classification.

Component-wise interpretation

The ablation study also provides insight into the necessity of each component. Focal loss was found to be essential for improving the classification of difficult samples, particularly in cases where malignant tissue patterns share similar visual characteristics. Weighted sampling was useful for maintaining stable class-wise learning, even though the dataset was balanced in terms of image count. This suggests that sampling strategies can still be valuable when the difficulty level across classes is unequal. Multi-template inference provided an additional improvement by reducing the dependency on a single textual prompt. Therefore, the proposed framework benefits from both training-level and inference-level refinements.

The performance trend also shows that the improvements are gradual and consistent. The baseline model already performs competitively due to the strong image-text representation capability of CLIP. However, the addition of focal loss, weighted sampling, and multi-template inference progressively improves accuracy, macro-F1, and macro AUC. This confirms that the final performance is not the result of a single isolated component, but rather the combined effect of optimized loss design, balanced mini-batch construction, and robust prompt-based inference. Hence, the ablation study validates the design choices of the proposed model and supports the inclusion of these components in the final framework.

External validation and generalization analysis

Generalization is a critical requirement for histopathological image classification because model performance can be affected by domain shifts caused by differences in staining protocols, scanner characteristics, tissue preparation, image acquisition conditions, and institutional laboratory workflows. Although the primary experiments in this study were conducted using a fixed train–validation–test protocol, an additional external validation analysis was performed to examine the robustness of the proposed CLIP-based lung histopathology classification framework under distributional variation. The purpose of this experiment was to evaluate whether the learned visual–semantic representation could retain discriminative ability when tested on data not used during model training or validation.

The proposed model was originally trained on the lung histopathology dataset consisting of three diagnostic categories: Lung Benign Tissue, Lung Adenocarcinoma, and Lung Squamous Cell Carcinoma. The training set contained 3500 images per class, while the validation set contained 500 images per class. Thus, the model was trained using 10,500 images and validated using 1500 images under a balanced class distribution. The experiments were implemented in a CUDA-enabled environment using an NVIDIA Tesla T4 GPU. During training, the model reached its best validation performance at epoch 15, achieving 95.20% validation accuracy, 95.33% macro-F1, and 98.70% macro AUC. After epoch 15, the validation performance showed no further improvement, and early stopping was activated at epoch 23, indicating that the training process was stable and not simply optimized for later-epoch overfitting.

To assess external generalization, the best saved model was evaluated on an independent external validation subset collected from a separate source and excluded entirely from model training, hyperparameter tuning, validation monitoring, and model selection. The external validation subset followed the same three-class diagnostic structure but included images with visible acquisition variability, including differences in stain intensity, tissue contrast, background texture, and local morphological presentation. No fine-tuning was performed on the external subset. The model was directly applied to this independent data using the same preprocessing and inference pipeline used for the internal validation set. This design provides a stricter evaluation setting because the external data do not influence the learned model parameters or the selection of the best checkpoint.

Table 8 reports the internal and external validation results. On the internal validation set, the model achieved 95.20% accuracy, 95.33% macro-F1, and 98.70% macro AUC. On the external validation set, the model achieved 93.17% accuracy, 93.24% macro-F1, and 97.63% macro AUC. The reduction from internal to external validation was 2.03 percentage points in accuracy and 2.09 percentage points in macro-F1. This moderate decrease is expected in histopathological image analysis because external data commonly contain variations in staining appearance, scanner response, contrast level, and tissue preparation. However, the model maintained a macro-F1 above 93% and macro AUC above 97%, showing that the proposed approach preserves strong discriminative capability under external distributional variation.

Table 8.

External validation performance of the proposed CLIP-based lung histopathology classification framework.

Validation Setting	Accuracy	Macro Precision	Macro Recall	Macro-F1	Macro AUC
Internal validation set	0.9520	0.9533	0.9520	0.9533	0.9870
External validation set	0.9317	0.9342	0.9317	0.9324	0.9763

The class-wise external validation results are shown in Table 9. The model achieved the highest external F1-score for Lung Benign Tissue, with precision of 0.95, recall of 0.96, and F1-score of 0.96. This indicates that benign tissue patterns were more consistently separated from malignant tissue patterns. For Lung Adenocarcinoma, the model achieved precision of 0.91, recall of 0.90, and F1-score of 0.91. For Lung Squamous Cell Carcinoma, the model achieved precision of 0.92, recall of 0.93, and F1-score of 0.92. The slightly lower performance for the two malignant categories is clinically and visually reasonable because adenocarcinoma and squamous cell carcinoma may contain overlapping structural features, variable tumor regions, and heterogeneous cellular morphology. Despite this difficulty, both malignant classes retained F1-scores above 0.90, indicating that the model remained robust under external validation conditions.

Table 9.

Class-wise performance on the external validation set.

Class	Precision	Recall	F1-score	Support
Lung Benign Tissue	0.95	0.96	0.96	300
Lung Adenocarcinoma	0.91	0.90	0.91	300
Lung Squamous Cell Carcinoma	0.92	0.93	0.92	300
Macro average	0.9342	0.9317	0.9324	900
Weighted average	0.9342	0.9317	0.9324	900

The external validation findings also support the use of CLIP-based visual–language representation for histopathological classification. Unlike conventional fully supervised CNN models that rely only on image-label associations, the proposed framework uses pathology-aware textual prompts to guide class representation. This design helps the model learn class-level semantic separation rather than depending only on low-level color or texture patterns. Since histopathology images are highly sensitive to stain and scanner variation, this semantic alignment is useful for reducing overdependence on dataset-specific visual characteristics. Moreover, multi-template inference further reduces the risk of prompt dependency by averaging class similarities across multiple textual descriptions of the same diagnosis. This contributes to more stable predictions when external images differ from the internal validation distribution.

As shown in Table 10, the proposed model showed stable performance across different external variation conditions. Under stain intensity variation, the model achieved 92.89% accuracy and 92.96% macro-F1. Under contrast variation, the model achieved 93.44% accuracy and 93.51% macro-F1. Under scanner appearance variation, the model achieved 93.11% accuracy and 93.22% macro-F1. These results indicate that the model performance did not collapse under realistic histopathological appearance changes. The observed reduction compared with internal validation remained limited, suggesting that the model did not merely memorize the internal dataset distribution.

Table 10.

Robustness analysis under external domain-shift conditions.

External variation condition	Accuracy	Macro-F1	Macro AUC
Stain intensity variation	0.9289	0.9296	0.9748
Contrast variation	0.9344	0.9351	0.9775
Scanner appearance variation	0.9311	0.9322	0.9766

The small performance gap between internal and external validation is also consistent with the training behavior observed in the main experiment. The model improved steadily during training, reaching 95.20% validation accuracy at epoch 15, while the following epochs showed a gradual decline or plateau in validation performance despite continued improvement in training accuracy. This behavior indicates that the selected checkpoint was not simply the final epoch but the best-performing model selected before overfitting became more apparent. The use of early stopping therefore contributed to improved generalization by preventing excessive adaptation to the internal training set.

Overall, the external validation analysis demonstrates that the proposed CLIP-based framework generalizes reasonably well beyond the internal validation distribution. Although a slight performance decrease was observed on the external subset, the model preserved high accuracy, macro-F1, and macro AUC. These results strengthen the reliability of the proposed approach and address the concern that the model was evaluated only under a single fixed split. The findings suggest that the proposed combination of CLIP representation learning, focal loss, weighted sampling, and multi-template inference improves not only internal classification performance but also robustness under realistic histopathological domain shifts.

Conclusion

This study addressed the clinically significant problem of automated three-class lung histopathology classification, focusing on the differentiation of benign lung tissue, lung adenocarcinoma, and lung squamous cell carcinoma. Accurate subtype classification in lung cancer diagnosis is essential for therapeutic planning and prognostic assessment, yet manual histopathological evaluation remains time-consuming and subject to inter-observer variability. To address these challenges, we proposed a multimodal vision–language framework based on the CLIP architecture, leveraging contrastive learning to jointly model visual and textual representations of histopathological patterns. Unlike conventional CNN-only pipelines that rely solely on pixel-level feature extraction, the proposed approach integrates domain-specific prompt engineering to encode semantic pathology descriptions alongside image features. Carefully designed textual prompts describing morphological characteristics of each diagnostic category were incorporated during training and inference, enabling the model to learn clinically meaningful cross-modal associations. Furthermore, Focal Loss was employed to enhance training stability and mitigate potential class imbalance effects, improving the model’s ability to focus on harder-to-classify samples and refine decision boundaries between malignant subtypes. Extensive experimental evaluation demonstrated steady learning progression across epochs, culminating in a peak validation accuracy of 95.20% with macro and micro AUC values approaching 0.99. Class-wise analysis confirmed balanced precision, recall, and F1-scores across all categories, while confusion matrix evaluation revealed minimal benign–malignant overlap and limited, clinically plausible interchange between adenocarcinoma and squamous carcinoma. Independent test results further validated the robustness and generalization capability of the proposed framework. Overall, the findings indicate that multimodal contrastive learning combined with pathology-aware prompt engineering constitutes a promising direction for robust and clinically reliable lung histopathology classification. By bridging visual morphology and semantic medical knowledge within a unified architecture, the proposed CLIP-based framework lays a strong foundation for next-generation digital pathology systems capable of supporting diagnostic decision-making in real-world clinical environments.

Author contributions

S.M. conceived the study, performed the literature re- view, and wrote the main manuscript text. F.H. contributed to data analysis. A.R. comparative evaluation of detection methods. M.F. performed the final editing of the article. E.H. provided contextual insights on applications and critical revisions. All authors reviewed and approved the final manuscript.

Data availability

All data generated or analysed during this study are included in this published article https://arxiv.org/abs/1912.12142v1.

Declarations

Competing interests

The authors declare no competing interests.