Grading medullary thyroid carcinoma on fine needle aspiration biopsy specimens: a comparative study between manual counting and a deep learning platform

Abstract

Introduction:

The International Medullary Thyroid Carcinoma Grading System (IMTCGS) stratifies medullary thyroid carcinoma (MTC) using Ki-67 proliferation index, mitotic index, and tumor necrosis. However, the ability to predict MTC grade using fine needle aspiration (FNA) specimens is unclear. Artificial intelligence (AI)-based quantification tools are increasingly being adopted. We therefore evaluated the concordance of MTC grading on cytology using manual counting and an AI-based digital quantification platform.

Materials and methods:

Eighteen MTC cases with paired FNA and surgical resections (2013–2022) were retrospectively identified. MTC grading was assessed on FNAs and resections according to IMTCGS criteria. Manual cytologic Ki-67 proliferation index (MCPI) and manual surgical proliferation index (MSPI) were compared. Digital cytologic Ki-67 proliferation index (DCPI) was generated using DeepLIIF. Concordance between Ki-67 proliferation indices and overall MTC grade was evaluated using Cohen’s kappa.

Results:

Substantial agreement was observed between MCPI and MSPI (94% concordance; kappa=0.77). Overall grade concordance between cytology and surgical specimens was 83% (kappa=0.49), with discrepancies largely due to necrosis identified only in resections. Agreement between MCPI and DCPI was fair (61%; kappa=0.22), with digital analysis frequently overestimating Ki-67 proliferation index. Concordance between DCPI and MSPI was 67% (kappa=0.33), and overall grade agreement between digital cytology assessment and surgery was slight (56%; kappa=0.11).

Conclusions:

Grading of MTC on cytology demonstrates substantial concordance for Ki-67 assessment but only modest agreement for overall grade, primarily because necrosis is often only focally present. The limited performance of the AI-based digital quantification platform emphasizes the need for specimen-specific development and validation prior to clinical implementation.

INTRODUCTION

Medullary thyroid carcinoma (MTC) is a malignant tumor that arises from C cells in the thyroid gland and makes up approximately 3–10% of all thyroid carcinomas. Although less common than follicular cell derived thyroid carcinomas, it has a comparatively worse prognosis and a higher mortality rate.¹ However, the clinical course of patients with MTC is variable. Some of the prognostic variables that have been identified include Ki-67 index, mitotic count, presence of coagulative necrosis, stage at initial diagnosis, age and RET mutational status. ^1–3 Mitosis and tumor necrosis are independent predictors of poor patient outcome. ^1–3 To help identify patients with potentially more aggressive disease, the two-tiered International MTC grading system (IMTCGS) was developed and validated in 2022 as a means of stratifying MTCs through pathologic assessment using Ki-67 proliferation index, mitotic index, and tumor necrosis. MTCs are classified as high grade if they have at least one of the following features: Ki-67 proliferation ≥ 5%, mitotic index ≥ 5 per 2 mm², or tumor necrosis.² High grade MTCs had reduced overall survival and disease-specific survival and a higher likelihood of locoregional recurrence and distant metastasis compared to low grade MTCs.² Fine needle aspiration (FNA) biopsy of thyroid is a reliable, minimally invasive technique of tissue sampling and more often the primary sampling modality for thyroid nodules.⁴ A few other studies previously evaluated the feasibility of grading of MTCs using cytology samples.^5–7 The aim of our study was to determine whether grading of MTCs based on the IMTCGS can be accurately performed on cytologic samples using conventional methods (i.e. a manual count) as well as DeepLIIF, an artificial intelligence (AI)-based deep learning algorithm for digital quantification, which has previously been shown to have high concordance with MTC grading on surgical samples.^8,9

METHODS

We retrospectively searched for FNAs with corresponding surgical resections of MTCs in the pathology database from October 2013 to June 2022 at Memorial Sloan Kettering Cancer Center. All anatomic sites were included since the aim of this study was meant to establish concordance between cytologic and surgical specimens and not to determine correlation with clinical outcomes. For surgical resections specimens, Ki-67 immunohistochemical stain (MIB-1 clone, 1:1000 dilution, Dako antibody) was performed and the Ki-67 proliferation index was determined using a manual count in hot spot regions, which is the current gold standard.² For the cytology specimens, tissue for cell blocks were collected in either Cytolyt solution or formalin and then processed using HistoGel.¹⁰ Cases were only included if cell blocks had sufficient cellularity for evaluation, which was defined as containing at least 100 tumor cells. For all cell blocks, a Ki-67 immunohistochemical stain using the MIB-1 clone (1:1000 dilution, Dako) was performed. For Cytolyt-fixed cell blocks, Ki-67 immunohistochemical stain using both MIB-1 and 30–9 clones (1:5 dilution, Ventana) was performed. Manual cytologic and surgical counts of Ki-67 proliferation indices was performed on microscope camera-captured images and digitally annotated. All immunohistochemistry was performed on the Leica Bond RX platform. For cytologic smears, the presence of mitotic figures and the highest total count was evaluated by reviewing all available smears. All smears were reviewed by two of the authors (DLN and ML or EE). For cell blocks and surgical resections, the mitotic index and Ki-67 proliferative index were evaluated using the same methods proposed by the IMTCGS.² Briefly, measurements were obtained in areas showing the highest proliferative activity (i.e. “hotspots”). The mitotic index was assessed per 2 mm². For cell block sections that did not contain 2 mm² of tumor cells, the entire slide was evaluated. For the Ki-67 proliferative index, up to 2,000 tumor cells were counted per sample. In cell block sections with fewer than 2,000 tumor cells, as many tumor cells as possible were included.

To digitally quantify the Ki-67 proliferation index, an AI-based quantification platform (DeepLIIF) was applied, using the version available in September 2023.^8,9,11 DeepLIIF is a free, cloud-native, online platform (https://deepliif.org/), MSKCC, New York, NY, USA) and has been updated since this study was conducted. Slides were scanned with a whole-slide scanner (Hamamatsu Nano Zoomer S360) using a 40x objective (0.23um/pixel) and the whole slide images were uploaded to DeepLIIF. Hot spots/regions of interest were manually annotated, and adjustments made on the segmentation threshold and tumor cell size gating independently by two pathologists (IA and DLN). In cases where there was a disagreement, a third pathologist (BEK) was used as a tie breaker. The manual cytologic Ki-67 proliferation index (MCPI) and the digital cytology Ki-67 proliferation index (DCPI) were assessed independently and were then categorized as “low grade”, or “high grade” based on IMTCGS. Similarly, the manual surgical specimen Ki-67 proliferation index (MSPI) was assessed independently by another pathologist (BX) on the surgical specimen. Statistical analyses were performed on SPSS software, version 29.0.1.1 (IBM, Armonk, NY, USA). Agreement between DCPI and MCPI, as well as between DCPI and MSPI, was established using Cohen’s Kappa correlation coefficient. Kappa values of <0.00, 0.00–0.20, 0.21–0.40, 0.41–0.60, 0.61–0.80 and 0.81–1.00 were interpreted as poor, slight, fair, moderate, substantial, and almost perfect, respectively.¹²

This research study was approved by the Institutional Review Board (IRB) of Memorial Sloan Kettering Cancer Center, New York (protocol 17–634).

RESULTS

The clinicopathologic data are summarized in Table. 1. A total of 18 cases were identified from the pathology database, 9 of which were thyroid FNAs. The rest were from lymph nodes, parotid gland, bone, and thyroid bed. The median patient age was 64 years ± 16 (range 29–83). Out of the 18 cases, 10 cases were fixed in Cytolyt solution, 7 cases were fixed in formalin, and 1 case had two separate cell blocks, one of which was fixed in Cytolyt and another in formalin; Ki-67 immunohistochemistry was performed on both blocks. Cell block tumor cellularity was more than 500 cells in 44% (8/18) cases.

Table 1.

Summary of characteristics for all cases

Characteristic	N = 18
Median age (years)	64 ± 16
Gender
Male	10 (56%)
Female	8 (44%)
Anatomic Site
Thyroid	9 (50%)
Thyroid Bed	2 (11%)
Bone	2 (11%)
Lymph node	4 (22%)
Parotid gland	1 (6%)
Necrosis on cytology specimen
Present	1 (6%)
Absent	17 (94%)
Necrosis on surgical specimen
Present	3 (17%)
Absent	15 (83%)
Any mitotic figures on cytology specimen
Present	4 (22%)
Absent	14 (78%)
Cytology specimen mitotic index
<5/2mm2	18 (100%)
≥5/2mm2	0 (0%)
Surgical specimen mitotic index
<5/2mm2	17 (94%)
≥5/2mm2	1 (6%)
Manual cytology Ki-67 proliferation index (MIB-1 clone)
<5%	16 (89%)
≥5%	2 (11%)
Manual cytology Ki-67 proliferation index (30–9 clone)
<5%	10 (55%)
≥5%	1 (6%)
Not performed	7 (39%)
Digital cytology Ki-67 proliferation index
<5%	9 (50%)
≥5%	9 (50%)
Surgical specimen Ki-67 proliferation index
<5%	15 (83%)
≥5%	3 (17%)
Manual cytology grade
Low grade	16 (89%)
High grade	2 (11%)
Digital cytology grade
Low grade	9 (50%)
High grade	9 (50%)
Manual surgical specimen grade
Low grade	13 (72%)
High grade	5 (28%)

Rare mitotic figures were identified in 22% (4/18) of cases, 3 of which were categorized as high grade on the surgical specimen. None of the cytology cases had ≥5 mitotic figures in the entire specimen. Although tumor necrosis was noted in 1/18 cytology cases (6%), it was absent in the corresponding surgical sample but was still categorized as high grade because of the elevated Ki-67 proliferation index. For the 9 thyroid cases, all were categorized as low grade on both cytology and surgical specimens; there was 100% concordance between the cytology and surgical specimens for mitotic index, manual Ki-67 proliferation index, and absence of necrosis.

The percentage agreement and kappa values for all cytologic-histologic pairwise comparisons are summarized in Table 2.

Table 2.

Summary of percentage agreement and kappa values for all cytologic-histologic pairwise comparisons

Observation	Cases	Percent Agreement	Kappa (95% CI)	p-value
MCPI (MIB1 clone) versus MCPI (30–9 clone)	11	100%	1 (1, 1)	-
MCPI versus MSPI	18	94%	0.77 (0.33, 1)	<0.001
MCPI versus DCPI	18	6 1%	0.22 (−0.06, 0.50)	0.134
DCPI versus MSPI	18	67%	0.33 (0.01–0.66)	0.058
MCG versus MSG	18	83%	0.49 (0.04, 0.94)	0.034
DCG versus MCG	18	61%	0.22 (−0.06, 0.50)	0.134
DCG versus MSG	18	56%	0.11 (−0.3, 0.52)	0.599

Abbreviations: MCPI, manual cytology Ki-67 proliferation index; MSPI, manual surgical Ki-67 proliferation index; DCPI, digital cytology Ki-67 proliferation index; MCG, manual cytology grade; MSG, manual surgical specimen grade; DCG, digital cytology grade; 95% CI, 95% confidence interval

Because not all cytology specimens were fixed in formalin, and Cytolyt fixation is known to inhibit the immunoreactivity of the Ki-67 MIB1 clone¹³, we evaluated whether or not there was a difference between the Ki-67 proliferation index using the MIB1 versus 30–9 clone in Cytolyt fixed samples. Between the 11 available pairs, there was perfect agreement (kappa=1.00, p<0.001). Because of the high level of concordance, the MCPI, MDPI and MSPI were evaluated only using the MIB-1 clone.

The overall concordance between MCPI versus MSPI was 94% (17/18) (kappa=0.77, 95% CI 0.34–1, p<0.001). One discordant case was categorized as having a MCPI of <5%, but a MSPI of ≥5%. The concordance between MCG and MSG was 83% (15/18) (kappa=0.49, 95% CI 0.04–0.94, p=0.034), which had 2 additional discordant cases in which two cases were categorized as low grade by cytology but high grade based on the surgical resection. Although the proliferation and mitotic indices did not reach the threshold of high grade on the surgical specimens, areas of necrosis were identified.

Using the version of DeepLIIF available in September 2023, the overall concordance between MCPI and DCPI was 61% (11/18) (kappa=0.22, 95% CI −0.06–0.50, p=0.134). The comparison between DCG and MCG had the same results as the comparison between MCPI and DCPI. In all 7 discrepant cases, MCPI was <5%, but DCPI was ≥5%. When comparing DCPI and MSPI, the concordance was 67% (12/18) (kappa=0.33, 95% CI 0.01–0.66, p=0.058). In all 6 discrepant cases MSPI was <5%, but DCPI was ≥5%. Finally, the concordance between DCG and MSG was 56% (10/18) (kappa=0.11, 95% CI −0.3–0.52, p=0.599). Of the discordant cases, 6 were misclassified as high grade and 2 were misclassified as low grade on DCG because of the presence of necrosis seen only on the surgical specimen. Figure 1 shows a representative case.

Figure 1.

Representative cell block with clusters of medullary thyroid carcinoma. A) Hematoxylin and Eosin stain, 40x; B) Calcitonin immunocytochemistry stain, 40x; C) Ki-67 immunocytochemistry stain, 40x; D) DeepLIIF applied to Ki-67 immunocytochemistry stain, red outlines highlights positive tumor cells, blue outlines highlights negative tumor nuclei.

DISCUSSION

In this study, we assessed the ability to accurately grade MTC using FNA samples based on the criteria proposed by the IMTCGS. The ability to accurately grade MTC based on cytology samples may be useful for pre-operative patient management or counseling. Moreover, AI-based deep learning algorithms, such as DeepLIIF, have been proposed as tools to help expedite and standardize tasks such as quantification of the Ki-67 proliferation index, and have been shown to be highly effective in surgical specimens^11,14,15. Therefore, we also used both conventional methods (i.e. manual counting in hot spots) and compared it to Ki-67 proliferation index quantification by DeepLIIF.

Overall, our study noted a substantial level of agreement between manual counts in cytology and surgical specimens for the Ki-67 proliferation index. Only one FNA sample underestimated the Ki-67 proliferation. However, the level of agreement between overall grade was only modest, because grading on cytology samples heavily relies on Ki-67 proliferation index. As a result, 2 additional FNA cases were misclassified as low grade because necrosis was only seen on the surgical specimen. Prior studies on MTC and differentiated high grade thyroid carcinoma FNAs also noted that coagulative tumor necrosis and mitotic figures are challenging to identify and are frequently not seen in the cytology sample, even when present in the corresponding surgical specimen.^5,16 Our findings are also similar to prior studies which found that while there is high agreement between the cytology and surgical specimen when an MTC is categorized as high grade on cytology, but that cytologic assessment may under call many tumors as low grade.⁵

Moreover, studies evaluating the grading of pancreatic neuroendocrine tumors using cytology samples have reported similar findings.^17–20 In some of these studies, discrepancies in grading were attributed to tumor heterogeneity and sampling error.^19,21 Therefore, in clinical practice, one should recognize that because cytology material is frequently limited, focal features such as necrosis can either be easily missed or under-sampled. As a result, a designation of low grade on cytology material should be interpreted with caution and that definitive tumor grading should deferred to the surgical resection specimen.

Although manual counting remains the gold standard for determining grade based on Ki-67 proliferation index, it can be time intensive and subject to interobserver variability.^21,22 Manual counting of camera captured or printed images is highly practical and increases reproducibility, but is labor intensive.^21,23 The use of AI for quantification of Ki-67 proliferation index has been found to improve accuracy and provide faster turn-around time.²⁴ We evaluated whether digital Ki-67 quantification might be able to bypass some of the limitations of manual counting on cytology samples.^11,24 The DeepLIIF platform offers several advantages including its availability as a free online platform (https://deepliif.org/) as well as adjustable sliding scales for threshold adjustment of size, segmentation and positivity, intensity assessment.⁸ However, our study found the level agreement between DCPI and MSPI was fair, while the level agreement between DCG and MSG was only slight. Most of the discordant cases were a result of overestimation of the Ki-67 proliferation index. This finding is particularly notable because a prior study found that there was near perfect agreement between the DeepLIIF platform and manual counting in surgical MTC specimens.¹¹ While an incredibly powerful tool, DeepLIIF was primarily developed using surgical specimens and not cell blocks, which have significant differences. Cells blocks tend to be less cellular and can have disaggregated tumor cells with interspersed inflammatory and red blood cells. In our testing, it appeared that the AI-algorithm tended to detect a small subset of red blood cells and scattered cellular debris as Ki-67 positive cells. Another challenge in evaluating the Ki-67 proliferation index in cytology samples is that it is difficult to distinguish between inflammatory cells and tumor cells on the immunostained sections. One group proposed using a dual immunostain with Ki-67 and synaptophysin, which appeared to increase their total tumor count.²⁵ Therefore, some of the cells that were annotated in DeepLIIF may have been inflammatory cells rather than tumor cells. Our study shows that the results from histology to cytology even for the same tumor type cannot be extrapolated and that AI-based tools require development and validation using all intended sample types prior to implementation. It should be noted that DeepLIIF is a rapidly evolving platform and that the concordance data reflect the version available at the time of the study.

There are several limitations in this study. First, the study only included 18 samples, 9 of which were from thyroid. Because MTC is a relatively uncommon tumor, additional studies involving multiple institutions will be needed to determine the true concordance between MTC grading using FNA samples and surgical specimens. Another limitation is that cell blocks did not consistently used the same fixative. Some cell blocks used Cytolyt solution while others used 10% neutral buffered formalin. A prior study showed that Cytolyt fixation can cause substantial inhibition of the MIB-1 clone’s immunoreactivity, while the 30–9 clone does not appear as susceptible to inhibition.¹³ However, our evaluation of both clones on Cytolyt-fixed cell blocks appeared to have equal performance based on manual review. In addition, sufficient cell block cellularity was set to at least 100 tumor cells, which is a relatively low threshold and may limit the reproducibility of the Ki-67 proliferation index assessment. However, it did not appear that DeepLIIF tended to struggle more with lower cellularity cell blocks. Instead, one of the main factors that negatively affected the performance of DeepLIIF was when tumor cells were discohesive or scattered, regardless of cellularity. Finally, we did not use the most recent version of DeepLIIF that was available at the time of submission. Major updates have been made based on the feedback and observations made to the developers during this project to improve cell segmentation, ability to gate cells by size, and debris exclusion.

CONCLUSIONS

Although grading of MTC on cytology samples shows only modest agreement, it is highly accurate when classified as high grade. Most discrepancies are because of underestimation of grading on cytology, particularly for cases where necrosis is focal. While AI-based platforms can be potent tools, validation on cytology samples is critical prior to clinical deployment.

Data availability statement:

The data that support the findings of this study are available from the corresponding author upon request.