Contribution of whole slide imaging‐based deep learning in the assessment of intraoperative and postoperative sections in neuropathology

Abstract

The pathological diagnosis of intracranial germinoma (IG), oligodendroglioma, and low‐grade astrocytoma on intraoperative frozen section (IFS) and hematoxylin–eosin (HE)‐staining section directly determines patients' treatment options, but it is a difficult task for pathologists. We aimed to investigate whether whole‐slide imaging (WSI)‐based deep learning can contribute new precision to the diagnosis of IG, oligodendroglioma, and low‐grade astrocytoma. Two types of WSIs (500 IFSs and 832 HE‐staining sections) were collected from 379 patients at multiple medical centers. Patients at Center 1 were split into the training, testing, and internal validation sets (3:1:1), while the other centers were the external validation sets. First, we subdivided WSIs into small tiles and selected tissue tiles using a tissue tile selection model. Then a tile‐level classification model was established, and the majority voting method was used to determine the final diagnoses. Color jitter was applied to the tiles so that the deep learning (DL) models could adapt to the variations in the staining. Last, we investigated the effectiveness of model assistance. The internal validation accuracies of the IFS and HE models were 93.9% and 95.3%, respectively. The external validation accuracies of the IFS and HE models were 82.0% and 76.9%, respectively. Furthermore, the IFS and HE models can predict Ki‐67 positive cell areas with R ² of 0.81 and 0.86, respectively. With model assistance, the IFS and HE diagnosis accuracy of pathologists improved from 54.6%–69.7% and 53.5%–83.7% to 87.9%–93.9% and 86.0%–90.7%, respectively. Both the IFS model and the HE model can differentiate the three tumors, predict the expression of Ki‐67, and improve the diagnostic accuracy of pathologists. The use of our model can assist clinicians in providing patients with optimal and timely treatment options.

The whole slide imaging‐based deep learning can contribute new precision to the diagnosis of IG, oligodendroglioma, and lower‐grade astrocytoma on intraoperative frozen sections (IFS) and postoperative hematoxylin‐eosin (HE)‐staining sections. The proposed model can help pathologists immediately discriminate tumor types during operations, while our HE model can assist pathologists with early postoperative treatment regimens.

1. INTRODUCTION

There were over 308,102 new cases of central nervous system (CNS) tumors diagnosed and approximately 251,329 cancer‐related deaths in 2020 worldwide [1]. Oligodendroglioma and low‐grade astrocytoma are the most common pathological types of CNS tumors known as the low‐grade gliomas [2, 3]. CNS germ cell tumor is the third most common CNS tumor diagnosed primarily during childhood and adolescence [4] with intracranial germinoma (IG) as its most common histological subtype [3, 5, 6]. Although IG, oligodendroglioma, and low‐grade astrocytoma all belong to the CNS tumor category and have similar cellular morphology, their therapeutic schemes and prognoses vary greatly. IG is highly radiosensitive, and whole‐brain radiotherapy with a boost to the tumor gross targeted volume (GTV) is regarded as the standard treatment [7, 8, 9]. In contrast, the maximum scope of surgical resection is recommended for oligodendroglioma and astrocytoma, with postoperative radiotherapy or/and chemotherapy prescribed based on the type and grade of the tumor [10]. Patients with IG show good prognoses, with a 5‐year overall survival (OS) rate of more than 90% [11], while 5‐year OS rates of astrocytoma and oligodendroglioma range from 48% to 66% [12]. Misdiagnosis leads to unnecessary resection of IG that may break the balance between different neurological functions and the extent of craniocerebral resection, as well as insufficient removal of astrocytoma or oligodendroglioma that may increase the risk of recurrence [7, 13].

The accurate diagnosis of IG, oligodendroglioma, and astrocytoma is of great importance and has been mainly based on three types of examinations: intraoperative frozen section (IFS) examination, hematoxylin–eosin (HE) staining‐based tumor section examination, and molecular testing. Molecular testing identifies the immunohistochemical (IHC) expression of specific proteins and gene mutation status [3, 14] for diagnosis, such as the positive expression of CD117 and placental alkaline phosphatase (PLAP) representing IG [15, 16], the presence of both the isocitrate dehydrogenase (IDH) mutation and the 1p/19q codeletion indicating highly potential diagnosed oligodendroglioma [3], and the IDH1/2 gene and the expressions of glial fibrillary acidic protein (GFAP) [17] and Olig2 [18] being closely correlated with astrocytoma. A biopsy specimen with pathological molecular testing is considered the gold‐standard diagnosis method for IG, oligodendroglioma, and astrocytoma. However, molecular testing is complex and time‐consuming (taking more than 1 week), thus unavailable for diagnosis during operations or soon after biopsies. Clinically, pathological examinations based on IFS and HE‐staining sections are preferable diagnosis methods. IFS is available for diagnosis within a few minutes during operations, which directly influences neurosurgeons' decision‐making. HE‐staining sections, which show better diagnostic value than IFS with clearer cellular structures, can be prepared within approximately 24 h after the biopsy, thus determining postoperative treatment regimens. Therefore, the intraoperative and postoperative treatments of IG, oligodendroglioma, and astrocytoma mainly depend on the pathological diagnosis of the IFS and HE‐staining section. However, even experienced pathologists cannot accurately distinguish the morphological features of IG, oligodendroglioma, and low‐grade astrocytoma based on microscopic examinations of the IFS and HE‐staining sections due to their cellular similarity.

Emerging whole‐slide imaging (WSI) technology largely promotes the development of digital pathology, making the pathological diagnosis more accurate, speedy, flexible, and convenient [19]. Deep learning (DL) is a computational analysis technique that can identify the features of WSI for tumor diagnosis and prognosis prediction, such as classifying non‐small cell lung cancer subtypes [20], grading the Gleason score in prostate cancer [21], predicting the microsatellite and treatment outcome of colorectal cancer [22, 23], evaluating the survival rate of soft tissue sarcoma [24], and exploring the origin of unknown cancer [25]. In terms of brain tumors, Jin et al. built an automatic analysis platform for the diagnosis of five types of CNS diffuse gliomas, achieving patient level accuracy of 87.5% [26]. Zadeh et al. developed a DL model for the segmentation of WSIs and identified novel tumor cell‐perivascular niche interactions that are associated with poor survival in glioblastoma [27]. Barker et al. used representative tiles in WSIs to automatically classify glioblastoma multiforme and lower‐grade gliomas with an accuracy of 93.1% [28]. Other research also utilized DL to grade gliomas, classify glioma subtypes, and predict prognosis and IDH mutation status for patients with lower‐grade gliomas based on WSIs [29, 30, 31, 32]. These studies demonstrate that WSI‐based DL models are able to identify digital histopathological features that are difficult for pathologists to pinpoint.

In the present study, we propose an automatic DL‐based pipeline for the pathological diagnosis of IG, oligodendroglioma, and low‐grade astrocytoma based on IFS WSI and HE‐staining section WSI. We also investigated the correlation between the DL models' output probability of diagnosis and the IHC expression of proteins. Furthermore, the diagnostic ability of DL models was compared with pathologists', and the impact of artificial intelligence (AI) assistance on the pathologists' diagnoses was evaluated.

2. MATERIALS AND METHODS

2.1. Patient cohort

This study retrospectively collected patients with CNS tumors from Xiangya Hospital of Central South University (Center 1) between 2010 and 2019, the Second Xiangya Hospital of Central South University (Center 2) between 2007 and 2021, the Third Xiangya Hospital of Central South University (Center 3) between 2010 and 2022, and the public the cancer genome atlas‐lower grade glioma (TCGA‐LGG) cohort [33, 34]. The data were approved by the institutional review boards of the three centers with waived signed informed consent. Inclusions were as follows: (1) Patients were confirmed as having CNS IG, oligodendroglioma, or low‐grade astrocytoma by the histological and molecular testing according to the WHO 2016 classification of CNS tumors for molecular pathological diagnosis [14] (details are in Supplementary Text S1 and Table S1); (2) had available IFS or HE slides that could be scanned to produce WSIs. Exclusions were as follows: (1) Patients were diagnosed as having a mixture of more than one tumor type. (2) The images of IFSs and HE‐staining sections were faded, damaged, or presented unclear tissue structure. All eligible IG patients were included in our study, and oligodendroglioma and astrocytoma patients within the same age range as IG patients (ranged: 6–35 years) were enrolled in analyses in order to avoid possible age‐related discrepancies. The representative slides of IFSs and HE‐staining sections were multilayer scanned at a high‐resolution of 0.25 μm/pixel (40× magnification) on a Pannoramic MIDI slide scanner (3DHISTECH Ltd, Budapest, Hungary), which produced WSI data for our analyses (Figure 1.I).

FIGURE 1.

The workflow of the analyses performed in this study. The whole‐slide imaging (WSI) of the intraoperative frozen section (IFS) and hematoxylin–eosin (HE)‐staining section was collected (I) and subdivided into small tiles (II). Then a deep learning (DL)‐based tissue tile selection model was trained to select high‐quality tissue tiles (II), which were then used to build a DL‐based tile‐level classification model (III). The diagnostic probability map showing the classification results of all tiles for each WSI was generated, and the majority voting method was used to determine the final diagnosis (IV). Finally, the diagnostic probability maps were provided to pathologists to investigate the effectiveness of AI assistance (V).

In addition, we collected pathological tiles from the public data set NCT‐CRC‐HE‐100 K‐NONORM [35], which contains eight types of tissue tiles and background tiles at 0.5 μm/pixel, for training the subsequent DL‐based tissue tile selection model.

2.2. Image preprocessing

As shown in Figure 1.II, a three‐step image preprocessing method was used to convert the original WSIs into small tiles. First, the original WSIs were converted from red, green, and blue (RGB) to gray, and a threshold of 220 was used to segment tissue regions on the gray WSIs [20]. Secondl the tissue regions were subdivided into non‐overlapping 224 × 224 tiles at a resolution of 0.5 μm/pixel (20× magnification). The tiles with less than 50% tissue were removed from analysis [20]. After these two steps, approximately 10,000,000 tiles were generated at all centers, among which there still existed many background tiles, that is, tiles that did not show clear cellular structure, such as blotchy tiles, dark tiles, and fuzzy tiles (Figure S1). Lastly, we established a DL‐based tissue tile selection model to further select the qualified tissue tiles, that is, the tissue tiles with a clear cellular structure, and remove background tiles. The model was established using 27,144 tiles, including 11,578 tiles from Center 1 (10,350 qualified tissue tiles and 1228 background tiles) that were picked manually by pathologists and 15,566 public tiles (5000 qualified tissue tiles and 10,566 background tiles) [35]. All tiles were randomly allocated into the training set (19,931 tiles, 73%) and the validation set (7213 tiles, 27%) to train a pretrained ResNet152 network and evaluate the tissue tile selection performance. The trained model was used to determine which tiles of our dataset were qualified for subsequent model development.

2.3. Model development and validation

To develop DL‐based diagnosis models, all patients at Center 1 were allocated into the training, testing, and internal validation sets at a ratio of 3:1:1 (HE: n = 130:43:43; IFS: n = 98:33:33), while patients at the other centers and TCGA‐LGG were set as the external validation set. The training set was used to optimize the hyperparameters with data augmentation applied to the samples. The testing set without applying data augmentation was used to select the models showing the best performance. The internal and external validation sets were used to validate the models' diagnostic ability. The tumor type of one patient was propagated to all his/her tiles as the training label, and tiles in the training set were sent to a pretrained ResNet152 network to execute the tile‐level classification task (Figure 1.III). We trained two independent models: one for the IFS‐based diagnosis task (IFS model), another for the HE‐based diagnosis task (HE model). After model development, the majority voting method was used to transfer the tile‐level classification results into the patient‐level diagnosis results [36]. For each patient, the tumor type that was predicted by the majority of the tiles was the final diagnosis result. The performance of DL models was evaluated using the accuracy of each tumor type and the total accuracy.

To intuitively illustrate the tile‐level classification results and the patient‐level diagnosis results, we generated diagnostic probability maps by fusing the output probabilities of the DL models with the original WSIs (Figure 1.IV). The output probabilities of IG, oligodendroglioma, and astrocytoma of each tile were set as the values of the green (G), blue (B), red (R) channels, respectively. The more the color of one tile approaches a primary color, the higher the probability it is to be one of the pathological types according to this key: green = IG; blue = oligodendroglioma; red = astrocytoma. The WSI at 4 μm/pixel (2.5× magnification) was converted to grayscale images, and the RGB value of each tile was added to the corresponding region of the grayscale image to generate a diagnostic probability map, with the fusion weights of RGB value and grayscale value of 0.55 and 0.45, respectively.

2.4. AI‐assisted diagnosis

We compared the diagnostic ability of DL models with pathologists and evaluated whether our models can assist pathologists with diagnosis (Figure 1.V). The HE (43 patients) and IFS (33 patients) WSIs in the internal validation set were reviewed by three board‐certified pathologists with 6, 15, and 25 years of experience (junior, deputy senior, and senior) with and without AI assistance. All WSIs were anonymized to ensure that pathologists were blinded to patients' clinical information. The pathologists reviewed WSIs on CaseViewer Version 2.4.0 (3DHISTECH Ltd, Budapest, Hungary) at any magnification they preferred. Pathologists were instructed to choose one of the three tumors as the diagnostic result, in order to ensure that DL models and pathologists faced the same diagnostic challenges. With AI assistance, pathologists were additionally provided the diagnostic results of the DL models and the corresponding diagnostic probability maps showing the distributions of the output probabilities on the original WSIs with a manual to explain their meanings.

2.5. Prediction of the expression of Ki‐67 protein

The IHC expression of Ki‐67 protein is highly correlated with tumor cell proliferation and has been considered a potential prognostic factor for patients with CNS tumors [37, 38, 39]. We investigated whether the pathologists' diagnoses (without AI assistance) and the diagnostic probabilities of the DL models had an association with the expression of Ki‐67 protein. We employed a multivariate linear model, as shown in Equation 1, to predict the Ki‐67 positive cell areas of patients in the internal validation sets.

$$Y={\mathrm{\beta }}_0+{\mathrm{\beta }}_1{X}_1+{\mathrm{\beta }}_2{X}_2+{\mathrm{\beta }}_3{X}_3$$ (1)

The response variable (Y) is the patient‐level Ki‐67 positive cell area. The ground truth of the Ki‐67 positive cell area was counted by a pathologist with 30 years of experience. For DL models, the predictor variables (X ₁ , X ₂ , X ₃ ) are the patient‐level diagnostic probabilities for IG, oligodendroglioma, and astrocytoma, respectively, which are the average of the output probabilities of all the tiles with the correct diagnosis for each patient. The regression coefficients (${\mathrm{\beta }}_0$, ${\mathrm{\beta }}_1$, ${\mathrm{\beta }}_2$, ${\mathrm{\beta }}_3$) are fitted by the linear model. For pathologists, the predictor variable (X ₁ ) is the tumor type they classified for each patient; ${\mathrm{\beta }}_0$ and ${\mathrm{\beta }}_1$ are regression coefficients while ${\mathrm{\beta }}_2$ and ${\mathrm{\beta }}_3$ in Equation (1) are equal to zero. The ordinary least‐squares method was used to minimize the sum of the residuals of points from the fitted curve, with the coefficient of determination (R ²) as a metric to evaluate the goodness of fit.

The analyses were performed using the Python package “statsmodels”.

2.6. Model implementation

The DL models were trained using Pytorch 1.9.0 on an Intel(R) Xeon(R) Silver 4210 CPU @ 2.20 GHz CPU and two NVIDIA GeForce RTX 2080 Ti GPUs (CUDA version: 10.2).

To enrich the diversity of tiles and avoid overfitting, data augmentation was applied to all DL models, including (1) random rotation within a range of −15 degrees to 15 degrees; (2) random horizontal flip; and (3) color jitter: brightness factor, contrast factor, and saturation factor varied uniformly from 0.5 to 1.5. Color jitter can enlarge the variations of the tile color so that the models can adapt to the differences of pathological staining.

For the tissue tile selection model, the training hyperparameters were as follows: (1) batch size: 64; (2) learning rate: 1 × 10⁻³; (3) optimizer: stochastic gradient descent (SGD) with a momentum of 0.9; (4) epoch: 100. The loss function used was cross entropy. For the IFS and HE diagnosis models, the training hyperparameters were as follows: (1) batch size: 128; (2) learning rate: 1 × 10⁻⁴; (3) optimizer: Adaptive Moment Estimation (Adam) with betas of [0.9, 0.999] and the weight decay of 10⁻³; (4) epoch: 10. The loss function used was cross entropy.

In addition, we have investigated the influence of the selection of different parameters, such as pixel sizes, loss functions, color jitter, and color normalization, on the classification performance of our models. Details are in Supplementary Text S2 and Table S2–S4.

3. RESULTS

3.1. Patient cohort and selected tiles

A total of 216 patients with three tumor types were enrolled at Center 1 (571 HE WSIs of 216 patients and 322 IFS WSIs of 164 patients). A total of 41 patients with three tumor types were enrolled at Center 2 (146 HE WSIs of 41 patients). A total of 12 patients with IG were collected from Center 3 (20 IFS WSIs of 12 patients). A total of 110 patients with oligodendroglioma and astrocytoma were collected from TCGA, providing 115 HE WSIs of 89 patients and 158 IFS WSIs of 110 patients.

The external validation set for HE models was composed of 146 HE WSIs from Center 2 and 115 HE WSIs from TCGA, while that of the IFS model was composed of 20 IFS WSIs from Center 3 and 158 IFS WSIs from TCGA (Table 1). Table 1 shows the characteristics and molecular testing results of all enrolled data.

TABLE 1.

Characteristics of data in this study.

Feature	Category	IG	Oligodendroglioma	Astrocytoma	Total
Numbers (n)		Total (Center 1 + Center 2 + Center 3 + TCGA)
Patient number (n)	IFS	69 (57 + 0 + 12 + 0)	46 (23 + 0 + 0 + 23)	171 (84 + 0 + 0 + 87)	286 (164 + 0 + 12 + 110)
	HE	74 (64 + 10 + 0 + 0)	71 (35 + 17 + 0 + 19)	201 (117 + 14 + 0 + 70)	346 (216 + 41 + 0 + 89)
	Total	86 (64 + 10 + 12 + 0)	75 (35 + 17 + 0 + 23)	218 (117 + 14 + 0 + 87)	379 (216 + 41 + 12 + 110)
WSI number (n)	IFS	135 (115 + 0 + 20 + 0)	83 (47 + 0 + 0 + 36)	282 (160 + 0 + 0 + 122)	500 (322 + 0 + 20 + 158)
	HE	200 (168 + 32 + 0 + 0)	224 (128 + 67 + 0 + 29)	408 (275 + 47 + 0 + 86)	832 (571 + 146 + 0 + 115)
Clinical factors
Gender (n)	Male/Female	58/28	42/33	115/103	215/164
Age (years)	Median(range)	16 (6–35)	30 (8–35)	22 (6–35)	24 (6–35)
WHO Grade (n)	I	/	0	7	7
	II	/	44	115	159
	III	/	31	96	127
Molecular profiles
IDH and 1p/19q co‐deletion subtype (n)	IDH‐wild‐type	/	0	21	21
	IDH‐mut‐non‐codel	/	0	187	187
	IDH‐mut‐codel	/	75	0	75
	NOS	/	0	10	10
GFAP	+	1	49	129	179
	−	74	3	2	79
	unknown	11	23	87	121
OLIG2	+	11	40	38	89
	−	3	0	1	4
	unknown	72	35	179	286
ATRX	wild‐type	0	22	31	53
	mutant	0	1	55	56
	+	2	30	10	42
	−	0	3	5	8
	unknown	84	19	117	220
P53	+	2	21	37	60
	−	4	26	28	58
	unknown	80	28	153	261
OCT‐3/4 (OCT4)	+	50	0	0	50
	−	4	0	1	5
	unknown	32	75	217	324
CD117 (c‐kit/CK)	+	68	0	0	68
	−	5	6	0	11
	unknown	13	69	218	300
PLAP	+	53	0	1	54
	−	2	1	1	4
	unknown	31	74	216	321

Abbreviation: IDHmut‐codel, IDH mutant and 1p/19q codeletion; IDHmut‐non‐codel, IDH mutant and not 1p/19q codeletion; IDHwt, IDH wild type; IG, intracranial germinoma; NOS, not otherwise specified, genetic testing not done or inconclusive.

The validation accuracy of the tile selection model was 99.51%, and the number of tiles in the internal validation set selected by this model are illustrated in Table S1.

3.2. Model performance

The internal accuracies of the IFS model and HE model were 93.9% and 95.3%, respectively. The external validation accuracy of the IFS model and the HE model were 82.0% and 76.9%, respectively (Table 2). Figure 2 illustrates the confusion matrices showing the diagnostic performance of the IFS and HE models. The IFS models distinguished IG cases from glioma cases (oligodendroglioma and astrocytoma) with accuracies of 100.0% and 99.2% in the internal and external validation sets, respectively. The HE model correctly identified 100.0% and 95.8% of IG and astrocytoma cases in the internal validation set, respectively, but its performance decreased in the external validation set (accuracy: 80.0% and 78.6%). Both IFS and HE diagnosis accuracies for oligodendroglioma in the internal validation set (100.0% and 83.3%) largely decreased in the external validation set (56.5% and 72.2%).

TABLE 2.

The diagnosis accuracy of the IFS and HE models in the internal and external validation sets.

Tumor	WHO grade	IFS model		HE model
		Internal validation set (n = 33)	External validation set (n = 122)	Internal validation set (n = 43)	External validation set (n = 130)
IG	/	100.0%	91.7%	100.0%	80.0%
O	II	100.0%	68.8%	100.0%	72.0%
	III	100.0%	28.6%	66.7%	72.7%
	Total	100.0%	56.5%	83.3%	72.2%
A	I	/	/	/	100.0%
	II	81.8%	90.7%	100.0%	84.4%
	III	100.0%	84.1%	90.0%	70.3%
	Total	88.2%	87.4%	95.8%	78.6%
Total	/	93.9%	82.0%	95.3%	76.9%

Abbreviation: A, Astrocytoma; IG, intracranial germinoma; O, oligodendroglioma.

FIGURE 2.

Confusion matrices showing the diagnosis performance of the IFS model (I) and the HE model (II) in the internal and external validation sets. Each column of the confusion matrix represents the instances in an actual class and their percentages in all datasets, while each row represents the instances in a predicted class and their percentages. Green color (green area and green number) represents correct diagnoses, and red color (red area and red number) represents incorrect diagnoses. The numbers in the right column and the lower row are the accuracy and loss for each row and each column, respectively. The number in the lower right corner is the total accuracy and loss. A: Astrocytoma; IG: intracranial germinoma; O: oligodendroglioma.

3.3. AI‐assisted diagnosis

Table 3 shows the improvement of the IFS‐based and HE‐based diagnosis accuracy of three pathologists with and without AI assistance.

TABLE 3.

The diagnosis accuracy of pathologists without and with AI‐assistance.

Category	Groups	Junior (6 years)		Deputy senior (15 years)		Senior (25 years)
		Without	With	Without	With	Without	With
IFS‐based diagnosis	IG	9.1%	100.0%	54.5%	100.0%	90.9%	100.0%
	Oligodendroglioma	0.0%	20.0%	0.0%	60.0%	0.0%	60.0%
	Astrocytoma	100.0%	100.0%	100.0%	100.0%	76.5%	94.1%
	Balanced accuracy	36.4%	73.3%	51.5%	86.7%	55.8%	84.7%
	Total accuracy	54.6%	87.9%	69.7%	93.9%	69.7%	90.9%
HE‐based diagnosis	IG	7.7%	100.0%	100.0%	92.3%	100.0%	100.0%
	Oligodendroglioma	66.7%	100.0%	16.7%	66.7%	33.3%	83.3%
	Astrocytoma	75.0%	79.2%	91.7%	87.5%	87.5%	87.5%
	Balanced accuracy	49.8%	93.1%	69.5%	82.2%	73.6%	90.3%
	Total accuracy	53.5%	88.4%	83.7%	86.0%	83.7%	90.7%

Note: The percentages are the diagnosis accuracy of each tumor and the total internal validation set. The balanced accuracy is the average of the accuracies of intracranial germinoma (IG), oligodendroglioma, and astrocytoma.

For the IFS‐based diagnosis, our IFS‐based models significantly improved the total diagnostic performance of the pathologists from 54.6%–69.7% to 87.9%–93.9%. With AI assistance, all pathologists accurately diagnosed all IG cases with an accuracy of 100.0%, while the accuracy without AI assistance ranged from 9.1% to 90.9%. The pathologists were not able to diagnose any oligodendroglioma cases without AI assistance but could correctly diagnose 20.0%–60.0% oligodendroglioma cases with AI assistance.

For the HE‐based diagnosis, the HE‐based total diagnostic accuracies of the junior, deputy senior, and senior pathologists were 53.5%, 83.7%, and 83.7%, respectively, without AI assistance. These were improved to 88.4%, 86.0%, and 90.7%, respectively, with AI assistance. The junior pathologist's diagnostic capability for all three tumors showed significant improvements. The deputy senior and senior pathologists' diagnosis accuracy for oligodendroglioma improved from 16.7% to 66.7% and 33.3% to 83.3%, respectively. The senior pathologist achieved 100.0% accuracy for IG with and without AI assistance, but he expressed much more confidence in his diagnostic result with AI assistance.

Figure 3 shows the chord diagrams showing the corrective effect of AI assistance for the three pathologists. The gray band indicates the pathologists' true diagnoses both without and with AI‐assistance (e.g., 22 cases in Figure 3.III); the purple band indicates that the pathologists corrected their diagnoses from the false group to the true group with AI assistance (e.g., eight cases in Figure 3.III); the orange band indicates the pathologists' misdiagnosed both without and with AI assistance (e.g., two cases in Figure 3.III); the red band indicates that the pathologists were misled by AI‐assistance, changing the diagnoses from true to false (e.g., one case in Figure 3.III). The wider purple band indicates the better efficacy of AI assistance; the narrower red band indicates a lower likelihood to be misguided by DL models. Our models corrected most of the misdiagnosed cases for the three pathologists (57.1%–80.0%), with only a few cases' diagnoses being misled by AI assistance (2.8%–8.3%). All pathologists stated they are willing to use our models to assist with both IFS‐based and HE‐based diagnosis in the future.

FIGURE 3.

The chord diagrams show the corrective effect of AI assistance for pathologists. The three pathologists reviewed the IFS WSIs (I‐III) and HE WSIs (IV‐VI) with and without AI assistance (the left and right sides of the circle). The orange and blue rings represent the incorrectly and correctly diagnosed patients that were included in the false and true groups, respectively; the connecting bands between the left side and the right side represent patients moving from one group to the other group with AI assistance. The gray band indicates the pathologists' true diagnoses both without and with AI assistance; the purple band indicates that the pathologists corrected their diagnoses from the false group to the true group with AI assistance; the orange band indicates the pathologists' misdiagnosed both without and with AI assistance; the red band indicates that the pathologists were misled by AI‐assistance, changing the diagnoses from true to false.

3.4. Prediction of the expression of Ki‐67 protein

The output probabilities of the IFS and HE models were predictive of Ki‐67 positive cell areas with R ² of 0.81 and 0.86 (p < 0.01), respectively. Three pathologists only showed limited predictive power of Ki‐67 positive cell areas, with R ² of 0.12–0.49 and 0.09–0.68 for IFS‐ and HE‐based diagnosis, respectively. Figure 4 shows the prediction of Ki‐67 positive cell areas based on the IFS and HE‐staining sections.

FIGURE 4.

The scatter diagram showing the actual Ki‐67 positive cell areas and those predicted by deep learning (DL) models (a) and junior (b), deputy senior (c), and senior (d) pathologists based on IFSs (I) and HE‐staining sections (II).

4. DISCUSSION

The pathological diagnosis of IG, oligodendroglioma, and astrocytoma directly determines patients' treatment options, but even experienced pathologists cannot give accurate diagnoses based on microscopic examinations of IFS and HE‐staining section (the pathologists in our study had an accuracy of 54.6%–69.7% and 53.5%–83.7%) due to the similarity of their cellular morphology. This greatly limits the planning of the optimal treatment regimens in clinical practice. Therefore, this study proposed a DL‐based automatic pipeline for the pathological diagnosis of IG, oligodendroglioma, and astrocytoma based on the IFS and HE WSIs. Both the IFS model and the HE model can accurately differentiate the three types of intracranial tumors. In addition, our models can further predict the expression of the Ki‐67 protein. We demonstrated that our models not only could diagnose the three tumors but also could reveal insights into the tumor cell proliferation and prognosis. Furthermore, we investigated the effectiveness of AI assistance for pathologists. Results showed that our models can correct many misdiagnosed cases and improve the diagnostic accuracy of pathologists, especially for IFS‐based diagnosis, by providing detailed diagnosis results on diagnostic probability maps. The diagnostic probability map generated by our model can visually and intuitively provide the diagnostic result of each small region on WSI and effectively locate the regions with high confidence for assisting pathologists in their diagnosis. Therefore, the DL models proposed in our study have the potential to be a useful tool for the pathological diagnosis of IG, oligodendroglioma, and astrocytoma.

During operations, pathologists can only differentiate normal brain tissues from malignant areas on IFS and cannot immediately discriminate tumor types, which may lead to inappropriate treatment options. Our research revealed that pathologists' IFS‐based diagnosis accuracies of IG, oligodendroglioma, and astrocytoma was only 54.6%–69.7%. In terms of identifying IG cases, the accuracies of three pathologists were 9.1%, 54.6%, and 90.9%, the result of which may lead to the misdiagnosis and unnecessary resection of IG. Whole‐brain radiotherapy with a boost to the GTV is regarded as the standard treatment for IG [7, 8, 9]. The radical resection of IG cannot offer more benefit than radio‐chemotherapy [13], and even increases the risk of intracranial tumor dissemination [40, 41]. Therefore, once IG was histologically confirmed intraoperatively on IFS, it's not recommended to continue the resection [7, 13]. Our IFS model can discriminate between IG and the glioma (oligodendroglioma and astrocytoma) with internal and external validation accuracies of 100.0% and 99.2%, and has improved the accuracy of IG from 9.1%–90.9% without AI assistance to 100.0% with AI assistance for all pathologists. The two misdiagnosed cases based on IFS WSIs occurred in the classification of astrocytoma and oligodendroglioma, which may be caused by atypical astrocytes. Although the same maximum of resection is recommended for the two tumors in the current guideline, it is meaningful to further intraoperatively differentiate these two tumors because of their different invasive abilities to the surrounding normal areas. Therefore, our IFS model can effectively assist neurosurgeons in selecting appropriate treatment options during operation for all three tumors.

Early postoperative treatment regimens (chemotherapy and/or radiotherapy) are determined by the pathological diagnosis of the HE‐staining section, which can be prepared within approximately 24 h after the biopsy. However, we found that the accuracy of pathologists for the diagnosis of IG, oligodendroglioma, and astrocytoma based on HE WSIs ranged from 53.5% to 83.7%. Our HE model can automatically diagnose the three tumors with internal and external accuracies of 95.3% and 76.9%, respectively, and can assist pathologists with the diagnosis. The early and accurate pathological diagnosis allows clinicians to plan optimal postoperative treatments in time and thus may improve positive outcomes.

We noticed that a big difference existed between the internal and external performance of the model. This may be caused by the different ways of resection and staining at different centers. Although color jitter was applied so that our models could adapt to the variations in the staining, the big differences existing in different centers may still influence the performance of our model. Besides, as automatic analyses were performed in our study and the WSIs were not manually segmented by pathologists, many causes may affect the prediction accuracy, such as angiogenesis, atypical astrocytes, tissue necrosis and deformation, and interstitial tissue. Detailed analyses of some typical errors in the representative WSIs of Center 2 are provided in Figure S2. Future studies will reduce or eliminate these error‐producing factors and further strengthen the differentiation of the three tumors in the external validation set. The generalization of the HE model should also be improved in the future, and a more advanced DL framework may be able to improve the generalization of DL models.

The output probabilities of both the IFS and the HE models were predictive of Ki‐67 positive cell areas with R ² > 0.8, which outperformed pathologists. Furthermore, we found that the output probabilities of the HE model showed similar distribution with the expression of some proteins. Figure S3 shows the diagnostic probability map of the HE model for a representative HE WSI (astrocytoma) and its corresponding IHC WSIs of Ki‐67, P53, and GFAP proteins. The regions with a higher probability of being the true diagnoses (red) exhibit higher Ki‐67 and P53 expression and lower GFAP expression compared to other regions, which indicate higher cellular proliferating speed, a higher malignant degree, and a worse prognosis [37, 42, 43]. This demonstrates that our model may be able to identify the morphological features of the tumor cells that are correlated with the expression of proteins. If tumor cells can be categorized into different subgroups with different expression of specific proteins on HE WSIs, it may contribute to the study of pathology‐based intra‐tumor heterogeneity and the design of an individualized treatment regimen in the future.

This study has several limitations. First, our models showed limited power for distinguishing oligodendroglioma from astrocytoma (Figure 2). Developing a specified binary model for classifying the two tumors may improve the classification performance. Second, our model did not further discriminate the WHO grades of oligodendroglioma and astrocytoma that are correlated with clinical treatment and prognosis. In future studies, we should enroll more tissue samples and further discriminate the WHO grades of oligodendroglioma and astrocytoma. Finally, among many relative proteins and genes, only the expression of the Ki‐67 protein was predicted using the output probabilities of the DL models due to the limitations of retrospective analysis. To increase the accuracy of the model, more molecular characteristics should be involved in future analyses.

In summary, we proposed an DL‐based automatic pipeline for the pathological diagnosis of IG, oligodendroglioma, and astrocytoma using the IFS WSIs and HE WSIs. Results show that both the IFS and HE models performed well in differentiating the three tumors, and the output probabilities of the models were predictive of the expression of Ki‐67 protein. Besides, we evaluated the effectiveness of AI assistance on the pathologists and found that our models can improve both the IFS and HE‐based diagnostic performance of pathologists at various levels of experience. Our DL models have the potential to benefit the perioperative pathological diagnosis of IG, oligodendroglioma, and astrocytoma, by which clinicians could provide patients with more optimal and timely treatment options than they could without using computer assistance.

AUTHOR CONTRIBUTIONS

Di Jing and Xin Gao conceived of the presented idea and supervised the project. Di Jing, Lin Shen, Bowen Yuan, Liangfang Shen, and Zhenzhen Liu prepared the sample and collected the data. Junming Jian organized the data. Liting Shi designed the computational model and analyzed the data. Wei Xia, Jiayi Zhang, Rui Zhang, and Keqing Wu checked and validated the results. Ke‐Da Yang, Yifu Tian, and Jianghai Huang performed the pathological diagnosis with and without AI‐assistance. Liting Shi and Di Jing wrote the manuscript. All authors read and approved the final manuscript.

FUNDING INFORMATION

This study is supported by the National Nature Science Foundation of China (No. 81803582), Key Research and Development Program of Shandong Province (No. 2021SFGC0104), Key Research and Development Program of Jiangsu Province (No. BE2021663), Hunan Provincial Natural Science Foundation of China (No. 2020JJ4896), Jiangsu Province Engineering Research Center of Diagnosis and Treatment of Children's Malignant Tumor, and Suzhou Science and Technology Plan Project (No. SJC2021014).

CONFLICT OF INTEREST STATEMENT

The authors declare no potential conflicts of interest.

Supporting information

Data S1 Supporting Information

Figure S1. The representative background tiles, including blotchy tiles (i), dark tiles (ii), fuzzy tiles (iii), that need to be removed from analysis.

Figure S2. The typical correctly diagnosed regions (T) and the typical misdiagnosed regions (F) of the HE WSIs (A) and the diagnostic probability maps (B) of astrocytoma (1) and intracranial germinoma (2 and 3). The region (1‐F) is mainly composed of neovascularization (1‐C); the region (2‐T) is mainly composed of IG cells (2‐C); the region (2‐F) is mainly composed of fibrous tissues and cells of which are deformed (2‐D); the region (3‐F1) contains many interstitial tissues and only a small number of tumor cells (3‐C); most of the region (3‐F2) is the hemorrhagic area (3‐D).

Figure S3. The immunohistochemical (IHC) whole slide images (WSIs) and hematoxylin–eosin (HE) WSI‐based prediction of one representative patient diagnosed as astrocytoma.

Shi L, Shen L, Jian J, Xia W, Yang K‐D, Tian Y, et al. Contribution of whole slide imaging‐based deep learning in the assessment of intraoperative and postoperative sections in neuropathology. Brain Pathology. 2023;33(4):e13160. 10.1111/bpa.13160

DATA AVAILABILITY STATEMENT

Data are available upon reasonable request from the corresponding authors.