Weakly supervised pathological differentiation of primary central nervous system lymphoma and glioblastoma on multi-site whole slide images

Abstract.

Purpose

Differentiating primary central nervous system lymphoma (PCNSL) and glioblastoma (GBM) is crucial because their prognosis and treatment differ substantially. Manual examination of their histological characteristics is considered the golden standard in clinical diagnosis. However, this process is tedious and time-consuming and might lead to misdiagnosis caused by morphological similarity between their histology and tumor heterogeneity. Existing research focuses on radiological differentiation, which mostly uses multi-parametric magnetic resonance imaging. By contrast, we investigate the pathological differentiation between the two types of tumors using whole slide images (WSIs) of postoperative formalin-fixed paraffin-embedded samples.

Approach

To learn the specific and intrinsic histological feature representations from the WSI patches, a self-supervised feature extractor is trained. Then, the patch representations are fused by feeding into a weakly supervised multiple-instance learning model for the WSI classification. We validate our approach on 134 PCNSL and 526 GBM cases collected from three hospitals. We also investigate the effect of feature extraction on the final prediction by comparing the performance of applying the feature extractors trained on the PCNSL/GBM slides from specific institutions, multi-site PCNSL/GBM slides, and large-scale histopathological images.

Results

Different feature extractors perform comparably with the overall area under the receiver operating characteristic curve value exceeding 85% for each dataset and close to 95% for the combined multi-site dataset. Using the institution-specific feature extractors generally obtains the best overall prediction with both of the PCNSL and GBM classification accuracies reaching 80% for each dataset.

Conclusions

The excellent classification performance suggests that our approach can be used as an assistant tool to reduce the pathologists’ workload by providing an accurate and objective second diagnosis. Moreover, the discriminant regions indicated by the generated attention heatmap improve the model interpretability and provide additional diagnostic information.

1. Introduction

Primary central nervous system lymphoma (PCNSL) and glioblastoma (GBM) are two important but quite different malignant brain tumors, accounting for 1.9% and 14.6% of all brain neoplasms, respectively.¹ The identification of the two is crucial as their prognosis and treatment differ substantially. For patients with PCNSL, high-dose methotrexate-based chemotherapy is the standard treatment, whereas patients with GBM usually undergo surgical resection followed by chemotherapy and radiotherapy with temozolomide.^2,3 The 5-year survival rate for patients diagnosed with PCNSL is 35.3%, whereas the rate is only 6.8% for patients with GBM.¹ Histopathology is considered the gold standard and mandatory in clinical diagnosis of PCNSL and GBM.^4,5 Precise and timely differentiation of PCNSL and GBM in histopathology may avoid unnecessary craniotomy through pre-operative stereotactic biopsy, assist in making intra-operative surgical strategies, and guide the postoperative immunohistochemistry and molecular testing. Morphologically, the hematoxylin and eosin (H&E)-stained PCNSL tissue exhibits the angiocentric growth pattern with sheets of tumor cells clustering within and around the blood vessels.² The PCNSL cells have monotonous nuclei with prominent nucleoli and scant basophilic cytoplasm.^6,7 By contrast, the GBM tissue often shows a glial fibrillary background and is composed of pleomorphic tumor cells with predominant astrocytic differentiation.⁸ Microscopically, GBM appears extremely heterogeneous and shows characteristics of cellularity, nuclear atypia, cellular pleomorphism, mitotic activity, microvascular proliferation, and palisading necrosis.⁹ Figure 1 shows typical PCNSL and GBM at the magnification of $400\times$. Manual examination of the tissues obtained in the surgical resection is challenging and might cause misdiagnosis because of the similarities between the histological characteristics of the two types of tumors. First, both PCNSL and GBM are highly proliferating, rich in cells, and diffusely infiltrate the brain parenchyma.^2,10 Second, in H&E-stained histology images, the mottled pattern of dense hyperchromatic tumor nuclei alternating with lightly stained neurofibrous stroma is typical for PCNSL but this feature imitates the necrosis which is common in GBM (Fig. 2). Third, unlike the lightly stained appearance of most GBM owing to the glial fibrillary background, small cell GBM appears hyperchromatic, which is easily confused with PCNSL (Fig. 2). All these issues pose challenges to the pathologists. Particularly, most pathologists in China are overworked and responsible for the diagnosis covering various systems of the human body due to the pathologist shortage;¹¹ the inadequate subspecialty training further increases the misdiagnosis rate, especially for rare diseases such as PCNSL.

Fig. 1.

Typical PCNSL (left) and GBM (right), both at $400\times$.

Fig. 2.

Typical mottled pattern in PCNSL (a) and necrosis in GBM (b) at $40\times$. Typical PCNSL (c) and small cell GBM (d) at $160\times$.

Computer-aided differentiation of PCNSL and GBM has received great attention in the last decade. However, the research mainly focuses on radiological differentiation, which mostly uses multi-parametric magnetic resonance imaging.^12–17 In addition, very recent work discriminates PCNSL from non-PCNSL lesions, especially glioma, using frozen whole slide images (WSIs).¹⁸ By contrast, pathological differentiation of PCNSL and GBM by conducting computational analysis of histology imaging is of high significance in assisting clinical diagnosis but has not yet been fully investigated.

Deep learning has been widely applied in histology image classification. To tackle the gigapixel size of the WSIs, the background has to be excluded and the remaining tissue region has to be cropped into small patches.^18,19 Some researchers assume that all patches share the same label with the WSI from which they are extracted.^18,20–22 Therefore, a patch-level classifier can be trained by feeding the extracted patches into deep learning algorithms such as Inception V3,²⁰ ResNet18,²¹ and other convolutional neural networks (CNNs).^18,22 Finally, the patch-level predictions are aggregated to generate the slide-level classification.^18,20 By contrast, more research considers the WSI classification problem under the theory of multiple instance learning (MIL), which assumes that the WSI should be classified as positive if at least one patch is positive and the WSI belongs to the negative class only if all patches are negative.²³ Many MIL variants have been proposed to make it more flexible and adaptive to binary and multi-class classification tasks.^19,24–30 These approaches often consist of two stages: patch feature extraction using CNN to reduce the data dimensionality and patch feature aggregation by applying attention-based MIL^19,24–26 or transformer²⁷ approaches for WSI diagnosis. Besides, some approaches select top-ranking patches with more representative features manually or by employing some selection strategies before applying the MIL-based patch aggregation for the WSI classification.^{24,26,28–30} Recently, self-supervised learning strategies such as SimCLR³¹ and SimSiam³² have drawn attention for pretraining a feature extractor on unlabeled patches.^33–38 Compared with a deep learning model pretrained on ImageNet such as ResNet50^19,25,27 or a custom CNN pretrained on the patches with WSI labels,²⁶ self-supervised feature extractor is expected to extract domain-invariant and intrinsic patch features, which potentially mitigates the data imbalance problem and benefits the generalization of the WSI classification model.³⁹

In this work, we propose a weakly supervised deep learning approach to differentiate the PCNSL and GBM WSIs obtained from postoperative formalin-fixed paraffin-embedded (FFPE) samples. To extract inherent morphological features from the patches and handle the data imbalance problem, a self-supervised feature extractor is trained in the SimSiam framework. Then, the patch features are fed into an attention-based MIL algorithm for the WSI classification. We validate our approach on the WSI datasets collected from three hospitals. Excellent and interpretable PCNSL/GBM differentiation is achieved in both the multi-center study and the validation of the combined dataset. The high performance suggests that our approach can be used as an assistant tool to reduce the pathologists’ workload by providing accurate and objective second diagnosis, which helps guide the postoperative immunohistochemistry and molecular testing. In addition, it is straightforward to extend our approach to differentiate the PCNSL and GBM WSIs of the biopsy and frozen samples, which potentially benefits the treatment planning in the pre- and intra-operative scenarios.

2. Methods

2.1. Datasets

Three WSI datasets were collected, namely JiNan, QingDao, and SiChuan datasets, respectively, according to the location of the hospital each dataset was collected from. All slides were obtained from H&E-stained postoperative FFPE samples with tumors clinically proven by histopathological and immunohistochemical examinations. Table 1 displays the number of PCNSL/GBM cases and slides included in each dataset. The WSI resolutions of the JiNan, QingDao, and SiChuan datasets are about 0.27, 0.27, and 0.24 microns per pixel (mpp), respectively. The size of the WSIs reaches the gigapixel level.

Table 1.

PCNSL/GBM data distribution and splitting for CLAM training/testing on each and the combined datasets.

Dataset	PCNSL cases/slides	GBM cases/slides	Folds	PCNSL cases for training/testing	GBM cases for training/testing
JiNan	45/163	416/417	4	36/9	315/101
QingDao	59/59	50/50	2	35/24	30/20
SiChuan	30/30	60/60	2	20/10	35/25
Combined	134/252	526/527	5	110/24	424/102

2.2. Tissue Segmentation and Patching

The tissue segmentation and patching process is illustrated in Fig. 3. To exclude the background including the whitespace, dirt, and artifact from the WSI, a tissue segmentation process is applied. The WSI is firstly downsampled to 1/32 of its original size and converted from RGB to the hue-saturation-value (HSV) color space. The tissue contour is then generated by successively applying median blurring, saturation channel thresholding, morphological closing, and area thresholding.¹⁹ Because the WSIs are gigapixel and the cellular morphology is the key to differentiate the PCNSL and GBM, the segmented tissue region is cropped into $256\times 256$ non-overlapping patches at the highest magnification for further processing. In our study, the number of patches extracted from the tissue region of each slide at the magnification of about $40\times$ varies between 248 and 107,475.

Fig. 3.

Tissue segmentation and patching.

2.3. Self-Supervised Feature Learning

The unlabeled patches are used to train the feature extractor in a self-supervised learning scheme named SimSiam,³² which has fewer requirements on the training data and a simpler architecture but achieves competitive performance compared with other schemes such as SimCLR,³¹ BYOL,⁴⁰ and SwAV.⁴¹ As illustrated in Fig. 4, SimSiam works with Siamese networks. For each training patch, two augmented views are generated and passed through the shared Siamese encoder consisting of a backbone and a projector. The standard ResNet50,⁴² specifically the five convolutional stages and the max-pooling layer, is used as the backbone, and thus, its output dimension is 2048. The projector consists of three fully connected layers and each layer contains 2048 neurons. Then, a predictor is followed on one branch and a stop-gradient operation is applied on the other branch. The predictor has two fully connected layers with 512 and 2048 neurons, respectively. During training, the model is optimized by maximizing the similarity between the outputs of both branches. When the model completes training, the first four convolutional stages of the backbone plus the max-pooling layer, as shown in Fig. 4, are used as the feature extractor in the subsequent WSI classification. It encodes each patch into a 1024-dimensional feature.

Fig. 4.

Self-supervised feature learning using SimSiam.

To enrich the training dataset and capture intrinsic patch representation, we design a sophisticated data augmentation strategy. After extensively investigating the commonly used data augmentation techniques,^43–45 a series of options relevant to histology image processing are selected and categorized into three groups. The first group contains basic transforms such as rotation, flipping, and identity (which means no transform). The second group consists of identity and transforms in sharpness (e.g., adding Gaussian noise and Gaussian blurring), contrast, and brightness. The third group includes image color transforms in the original RGB, HSV, and hematoxylin-eosin-DAB color spaces. For an augmentation operation, three transforms randomly selected from each of the three groups are successively applied to generate the augmented image. Figure 5 illustrates the effect of our augmentation strategy.

Fig. 5.

An original patch and its augmented versions.

For SimSiam training, the number of epochs was empirically set to 100, during which time the training loss tends to be stable as we observed. The batch size was set to 128 considering the limited computation resources. The stochastic gradient descent optimizer was adopted to update the model parameters by minimizing the cosine similarity loss. The initial learning rate was 0.025 and gradually decreased with a certain schedule while training proceeded. The momentum was set to 0.9 and the weight decay was 0.0001.

2.4. WSI Classification

For a WSI consisting of $K$ patches extracted from the tissue region, the $k$’th patch is encoded into its instance-level representation ${\mathbf{z}}_k\in {\mathbb{R}}^{1024}$ by the previously trained feature extractor. Then, a set of feature representations $\{{\mathbf{z}}_1,\dots ,{\mathbf{z}}_K\}$ are passed into the clustering-constrained-attention multiple instance learning (CLAM)¹⁹ method to perform the WSI classification. CLAM is a data-efficient algorithm with a simple architecture for weakly supervised WSI classification. As illustrated in Fig. 6, it employs an attention network to rank the feature representations and assigns scores according to their respective importance to the final prediction. First, a fully connected layer ${\mathbf{W}}_1\in {\mathbb{R}}^{512\times 1024}$ compresses ${\mathbf{z}}_k$ into a 512-dimensional vector ${\mathbf{h}}_k={\mathbf{W}}_1{\mathbf{z}}_k$. Then, the feature embeddings $\{{\mathbf{h}}_1,\dots ,{\mathbf{h}}_K\}$ pass through two parallel fully connected layers ${\mathbf{V}}_a\in {\mathbb{R}}^{256\times 512}$ and ${\mathbf{U}}_a\in {\mathbb{R}}^{256\times 512}$, and the element-wise product is calculated between their outputs. After that, an additional fully connected layer ${\mathbf{W}}_a\in {\mathbb{R}}^{1\times 256}$ is followed to generate the attention score $a_k$ for the $k$’th patch, which can be expressed as

$$a_k=\frac{\exp \{{\mathbf{W}}_a(\tanh ({\mathbf{V}}_a{\mathbf{h}}_k)\odot \mathrm{sigm}({\mathbf{U}}_a{\mathbf{h}}_k))\}}{\sum _{j=1}^K\exp \{{\mathbf{W}}_a(\tanh ({\mathbf{V}}_a{\mathbf{h}}_j)\odot \mathrm{sigm}({\mathbf{U}}_a{\mathbf{h}}_j))\}},$$ (1)

where $\tanh (·)$ and $\mathrm{sigm}(·)$ are tanh and sigmoid activation functions, respectively. Finally, the slide-level representation ${\mathbf{h}}_{\mathrm{slide}}\in {\mathbb{R}}^{512}$ is calculated as

$${\mathbf{h}}_{\mathrm{slide}}=\sum _{k=1}^K{a}_k{\mathbf{h}}_k.$$ (2)

Fig. 6.

WSI classification using CLAM.

The slide-level PCNSL/GBM prediction ${\mathbf{s}}_{\mathrm{slide}}$ is given by

$${\mathbf{s}}_{\mathrm{slide}}={\mathbf{W}}_c{\mathbf{h}}_{\mathrm{slide}},$$ (3)

where ${\mathbf{W}}_c\in {\mathbb{R}}^{2\times 512}$ is a fully connected layer with two neurons.

During inference, the trained CLAM model makes the diagnosis and the attention scores can be used to generate a slide heatmap, where high scores indicate the more relative regions to reveal the essential morphology for the diagnosis.

Empirically, the CLAM model was trained for at least 200 epochs and up to a maximum of 400 epochs, depending on the performance improvement within 20 continuous epochs. The batch size was set to 8. The Adam optimizer was adopted to update the model parameters by minimizing the cross-entropy loss. The learning rate was 0.0001, and the weight decay was 0.00001.

2.5. Dataset Splitting in Model Training and Validation

At first, we trained a feature extractor on the JiNan dataset and applied it to the other two datasets for feature extraction. Then, considering the variation in the WSI scanning and staining across different institutions, the models (i.e., the feature extractor and the prediction model) trained on one dataset might not generalize well on the other datasets. Hence, we performed a multi-center study, in which case the institution-specific feature extractor was trained for each individual dataset. For the SimSiam model training, to mitigate the data imbalance problem between the PCNSL and GBM classes in each dataset, the same number (specifically, 10 in our experiment) of PCNSL and GBM WSIs were randomly selected to extract patches. Each selected slide was taken from a unique patient to increase the data diversity. To capture the inherent morphological characteristics that are essential for PCNSL/GBM differentiation, the patches were extracted at the highest magnification of about $40\times$. Finally, 498,480, 547,902, and 703,348 patches were extracted to train SimSiam models for the three datasets, respectively.

To fully validate the algorithms, we also combined the three datasets and trained a multi-institution feature extractor on the entire dataset. A similar training procedure was followed as training the institution-specific feature extractors, except that 15 PCNSL and 15 GBM slides (five PCNSL and five GBM slides from each dataset) were used for SimSiam model training.

After feature extraction, the cross-validation was applied to each and the combined dataset to achieve the PCNSL/GBM classification. To avoid bias in the CLAM model training/prediction, the cases used for SimSiam training were first excluded and the remaining data were split into the training and test sets. Then, the excluded slides were combined into the training set because their labels were not exposed in training the feature extractor. Because of the very limited number of slides included in our datasets, we did not create a validation split to tune the hyper-parameters. The $k$-fold cross-validation was applied to reduce the impact of the variability in training and testing data splits, allowing for a more accurate assessment of the model’s generalization capabilities and thus avoiding potential overfitting. The test data were kept unseen in each fold model training. For the JiNan, QingDao, and SiChuan datasets, four-, two-, and two-fold cross-validation were applied, respectively, considering the different number of cases included in each dataset. For the combined dataset, five-fold cross-validation was applied. The patient-wise splitting makes sure that all slides belonging to one patient are included in either the training or test set. Table 1 displays the approximate number of PCNSL/GBM cases for training and testing in each fold.

2.6. Computational Hardware and Software

We implemented all the algorithms in Python and took advantage of image processing and deep learning libraries such as OpenSlide, PIL, OpenCV, and PyTorch. All the data including the WSI raw files, the temporary and results files, which take up about 2.5 TB, were stored on the hard drives of our local workstations.

The WSI segmentation and patching were performed on Intel Xeon CPUs (central processing units) on our local workstation. For a typical slide of size $\mathrm{66,304}\times \mathrm{76,544}$ in our datasets, the tissue segmentation from the thumbnail takes only 0.14 s. 46,233 patches can be extracted from this slide, and the patching takes about 0.61 s if only the patch coordinates are saved. If the patch images are saved, the patching takes about 6 min. The SimSiam model training was accelerated by utilizing two NVIDIA Tesla V100 for PCIe GPUs (graphic processing units) with 32 GB memory on each card. Training a model takes about 7 to 9 days. Both patch feature extraction and CLAM training/prediction were accelerated by two NVIDIA TITAN V GPUs with 12 GB memory on each card. For a typical slide, it takes about 65 s for feature extraction. The five-fold cross-validation of the CLAM model on the combined dataset takes about 35 h. Once the training is completed, the slide label prediction can be made in an instant.

3. Results

We validated the PCNSL/GBM differentiation of our approach on each individual and the combined datasets. Based on the same dataset splitting in the cross-validation of CLAM, we compared the performance of the JiNan-specific, institution-specific, and multi-institution feature extractors described in Sec. 2.5. Besides, we also compared our feature extractors trained through SimSiam with the clustering-guided contrastive learning (CCL)-based feature extractor pretrained on large-scale unlabeled histopathological images.⁴⁶ The CCL-based feature extractor also used ResNet50 as the backbone model.

The classification results of the PCNSL/GBM WSIs were measured by the area under the receiver operating characteristic curve (AUC) value and the accuracy (AC). Table 2 lists the overall classification performance measured by the AUC and AC and the classification AC of the PCNSL and GBM on each and the combined datasets using different feature extractors. By comparing the performance of the JiNan-specific and institution-specific feature extractors, it is demonstrated that for most metrics, especially for the discrimination accuracy of the PCNSL, training more specific feature extractors for different datasets produces better classification. The results also show that the institution-specific, multi-institution, and CCL-based feature extractors perform comparably with respective strengths in terms of certain metrics. We can find that the CCL-based feature extractor performs best in differentiating the PCNSL but worst in GBM while applying the multi-institution feature extractor generates the best discrimination in GBM in most cases but worst in PCNSL. By contrast, using the institution-specific feature extractors generally obtains the best overall classification AUC and AC with both the PCNSL and GBM classification accuracies reaching 80% for all three datasets. Regardless of the feature extractor used, for each dataset, the overall classification AUC exceeds 85% and AC is close to or exceeds 80%. When evaluating the combined dataset, employing either the multi-institution or the CCL-based feature extractor produces the overall AUC and AC close to 95% and 90%, respectively. The best classification results are achieved for the JiNan dataset, whichever feature extractor is used, probably because it has much more slides compared with the other two datasets and is more homogeneous than the combined dataset. For the JiNan, SiChuan, and the combined datasets, the classification AC of the GBM is much higher than that of the PCNSL, most probably owing to the severe data imbalance between the two classes.

Table 2.

PCNSL/GBM classification results on each and the combined datasets using different feature extractors.

Feature extractor	Dataset	AUC	AC	PCNSL_AC	GBM_AC
Institution-specific	JiNan	0.976 ± 0.011	0.940 ± 0.011	0.840 ± 0.075	0.968 ± 0.008
	QingDao	0.854 ± 0.004	0.809 ± 0.009	0.817 ± 0.017	0.800 ± 0.000
	SiChuan	0.934 ± 0.026	0.871 ± 0.014	0.800 ± 0.100	0.900 ± 0.020
JiNan-specific	QingDao	0.838 ± 0.002	0.764 ± 0.014	0.735 ± 0.015	0.800 ± 0.050
	SiChuan	0.914 ± 0.042	0.871 ± 0.014	0.750 ± 0.150	0.920 ± 0.040
Multi-institution	JiNan	0.972 ± 0.007	0.923 ± 0.047	0.781 ± 0.127	0.976 ± 0.015
	QingDao	0.865 ± 0.025	0.787 ± 0.031	0.736 ± 0.056	0.850 ± 0.000
	SiChuan	0.868 ± 0.096	0.829 ± 0.029	0.800 ± 0.100	0.840 ± 0.000
	Combined	0.940 ± 0.046	0.897 ± 0.048	0.778 ± 0.087	0.951 ± 0.023
CCL-based	JiNan	0.966 ± 0.019	0.909 ± 0.038	0.852 ± 0.145	0.949 ± 0.047
	QingDao	0.891 ± 0.016	0.809 ± 0.009	0.838 ± 0.038	0.775 ± 0.025
	SiChuan	0.872 ± 0.028	0.829 ± 0.029	0.800 ± 0.100	0.840 ± 0.080
	Combined	0.954 ± 0.033	0.904 ± 0.027	0.808 ± 0.109	0.947 ± 0.030

Each value represents the average and standard deviation within folds.

The value shown in bold indicates the best performance obtained across various feature extractors for the corresponding dataset with respect to the corresponding metric.

Figures 7 and 8 illustrate the attention heatmaps generated for the PCNSL and GBM slides of three datasets and representative patches from the selected region with the highest and lowest attention scores. The three slide-level attention heatmaps were produced by applying the institution-specific, multi-institution, and CCL-based feature extractors, respectively, for training the CLAM models on the corresponding dataset. It is shown that for each slide, the three heatmaps demonstrate generally consistent highly and slightly attended areas, which indicates that the models using the three different feature extractors produced consistent results. For both PCNSL and GBM, the hyperchromatic areas which contain mainly dense tumor cells are the most highly attended, whereas the relatively hypochromatic areas that are composed of neurofibrous stroma, blood vessels, hemorrhage, artifacts, necrosis, etc., receive the lowest attention. This indicates that our models primarily relied on the characteristics of the tumor cells to differentiate the PCNSL and GBM. This is understandable because the models were trained on the $256\times 256$ patches at the highest magnification, and the features at the cell level were more likely to be captured than those at the tissue level. From the representative patches with high attention scores displayed in the figures, we can see that the PCNSL cells have monotonous nuclei and high nuclear-to-cytoplasmic ratios, whereas the GBM cells exhibit high degrees of cellular pleomorphism and nuclear atypia.

Fig. 7.

Attention heatmaps and representative patches generated for example PCNSL slides of three sites. For each site, the top row shows the WSI thumbnail with the segmentation contours and three slide attention heatmaps respectively generated by the models trained on the corresponding dataset by applying the institution specific (the 2nd column), the multi-institution (the 3rd column) and the CCL-based (the 4th column) feature extractors. The slide heatmaps were produced by computing the attention scores for PCNSL over patches tiled with a spatial overlap of 50%, where the high-ranking and low-ranking patches are indicated in red and blue colors, respectively. To see more details, a representative region labeled by the black box in both the slide thumbnail and the heatmap (the 3rd column) is enlarged (the 5th column). Its corresponding regional attention heatmap (the 6th column) was generated using a 95% overlap and overlaid onto the original image. The bottom row displays the representative patches from the selected region with the highest and lowest attention scores indicated by the red and blue borders, respectively.

Fig. 8.

Attention heatmaps and representative patches generated for example GBM slides of three sites. Please refer to the caption of Fig. 7 for interpretation.

4. Discussion

Owing to the differences in scanners and staining protocols used for collecting the multi-site histopathology images, the classification model trained on one dataset did not perform comparably when directly transferred to other datasets. Therefore, we trained the models separately for each site and the combined multi-site datasets. To investigate the effect of the feature extraction on the final prediction, we compared the results generated by applying the institution-specific, multi-institution, and CCL-based feature extractors. Quantitative and qualitative results have shown that the models performed comparably and achieved satisfactory and interpretable classification.

Although our models have achieved good overall performance in all experiments, the misclassification is still non-negligible which is caused by several reasons. First of all, the PCNSL/GBM differentiation task itself is very challenging as explained before. A non-specialist without adequate training can hardly make the differentiation simply by examining the WSIs. Even the pathologists need to verify their diagnosis by conducting immunohistochemical analysis. It is expected that for deep learning models, it is also difficult to learn the essential differentiation characteristics from a huge amount of image content, especially considering that only the WSI labels are given for supervising the model training. Second, our datasets are relatively small for training deep learning models and severe data imbalance exists in the datasets due to the difficulty in histology image collection and the low PCNSL incidence rate. Moreover, it has to be noted that because all slides used in our experiments are taken from the postoperative FFPE samples, only the PCNSL cases which are misdiagnosed in all previous examinations, such as radiological imaging, pre-operative biopsy, and intra-operative frozen section examination, could be collected for our study. Compared with other feature extractors pretrained on thousands of WSIs,⁴⁶ only a small number of slides were selected for the self-supervised learning of our institution-specific and multi-institution feature extractors, which might be insufficient to capture the histopathological diversity of PCNSL and GBM. Furthermore, the datasets for training the classification models were also relatively small and severely imbalanced. These issues seriously affect the classification performance. In addition, some slides contain only a small amount of tissue and the artifacts within the tissue are difficult to remove in the preprocessing step. These problems also have an effect on the differentiation.

In this work, the attention heatmaps generated by the classification model improve the model interpretability. In the future, for the PCNSL/GBM differentiation, more specific and sophisticated cellular features, tissue patterns, and slide appearances have to be annotated by the pathologists to enhance the model explainability.⁴⁷ In terms of the methodology, we would like to investigate the domain generalization and stain normalization algorithms, which potentially benefit the domain adaptation in the multi-site study. The histopathological images can also be fused with radiological data such as the multi-parametric magnetic resonance images and the medical reports if available for training a more robust model. Besides, hierarchical attention mechanisms can be employed to focus on not only the discriminant patches but also the areas within the patches, which further enhances the model’s interpretability.

In summary, we investigated the computer-aided pathological differentiation of PCNSL and GBM using WSIs taken from the postoperative FFPE samples. The results generated for each site and the combined multi-site datasets have demonstrated that our approach has achieved excellent performance on the PCNSL/GBM differentiation task. The generated attention heatmaps enhanced the model interpretability and could provide the pathologists with additional diagnostic information. Our study has established a good foundation for the pathological differentiation of the PCNSL and GBM. By applying some domain adaptation algorithms, the approach can potentially be transferred to differentiate the PCNSL/GBM slides of the stereotactic biopsy samples and the intra-operative frozen sections, which are more challenging due to the limited size of the samples and the poor quality in histomorphology but more meaningful with regard to avoiding unnecessary craniotomy and making surgical strategies.

Biographies

Liping Wang is a lecturer at the School of Information Science and Engineering, Shandong Normal University. Her research interest lies in artificial intelligence in medical image analysis.

Lin Chen graduated from the Southwest Medical University of China. He is a resident neurosurgeon working at Provincial Hospital Affiliated to Shandong First Medical University and now is a PhD student supervised by Prof. Yingchao Liu. His research interest is neuro-pathology and deep learning applications in clinical medicine.

Kaixi Wei was a postgraduate at the Department of Neurosurgery of the Affiliated Hospital of Southwest Medical University. His research interests involve intracranial tumor, neuro-functional disease, and cerebral hemorrhage. He is currently working at Hejiang County Traditional Chinese Medicine Hospital.

Huiyu Zhou heads the AI and Machine Learning Group and leads the Biomedical Image Processing Lab at the University of Leicester, United Kingdom. He is the PGR director and deputy director of the Research Center for Artificial Intelligence, Data Analytics, and Modeling. Prior to this appointment, he was employed as a reader at the University of Leicester and a lecturer at the School of Electronics, Electrical Engineering, and Computer Science, Queen’s University Belfast. He has published widely in the field.

Reyer Zwiggelaar’s research interest concentrates on the development of computer vision and machine learning techniques applied to medical image analysis. He has published over 350 publications in JCR journals and peer-reviewed conferences. According to Google Scholar, he has an h-index equal to 38. He has attracted more than £5,000,000 in research funding, mainly as the principal investigator, from RCUK/HEFCW. He is an associate editor of the IEEE Journal of Biomedical and Health Informatics and Pattern Recognition.

Weiwei Fu, MD, is an associate deputy chief physician in the Department of Pathology, Affiliated Hospital of Qingdao University. She has been engaged in pathological diagnosis for nearly 20 years. Her current research interests are mainly in areas of neuropathology.

Yingchao Liu is a chief neurosurgeon in the Provincial Hospital Affiliated to Shandong First Medical University in Jinan, China. Besides the daily neurosurgical operation, he has also actively engaged in the field of developing brain functional MRI and Gama-knife radiosurgery treatment since 2013 and has published more than 40 peer-reviewed research articles in journals such as the nature biomedical engineering, radiology, and neuro-oncology related to the glioma, brain metastasis, and functional MRI.

Disclosures

The authors declare that they have no conflict of interest.

Ethics Approval

This research study was conducted retrospectively from data obtained for clinical purposes. Ethical approval was waived by the Institutional Review Board (IRB) in view of the retrospective nature of the study.

Consent to Participate

This study was performed on histopathological slides, where the participants’ information had been anonymized and the paper does not include any images that may identify the participants. In this case, the consent is not required according to the guidelines of the journal.

Code and Data Availability

The data are not currently accessible to the public due to their use in an ongoing study. It is anticipated that the interested parties can obtain the dataset from the corresponding author upon reasonable request once the study is completed. The code can be shared upon request, please contact Liping Wang at wangliping19872011@gmail.com.