The 1000 Mitoses Project: A Consensus-Based International Collaborative Study on Mitotic Figures Classification

Abstract

Introduction. The identification of mitotic figures is essential for the diagnosis, grading, and classification of various different tumors. Despite its importance, there is a paucity of literature reporting the consistency in interpreting mitotic figures among pathologists. This study leverages publicly accessible datasets and social media to recruit an international group of pathologists to score an image database of more than 1000 mitotic figures collectively. Materials and Methods. Pathologists were instructed to randomly select a digital slide from The Cancer Genome Atlas (TCGA) datasets and annotate 10-20 mitotic figures within a 2 mm² area. The first 1010 submitted mitotic figures were used to create an image dataset, with each figure transformed into an individual tile at 40x magnification. The dataset was redistributed to all pathologists to review and determine whether each tile constituted a mitotic figure. Results. Overall pathologists had a median agreement rate of 80.2% (range 42.0%-95.7%). Individual mitotic figure tiles had a median agreement rate of 87.1% and a fair inter-rater agreement across all tiles (kappa = 0.284). Mitotic figures in prometaphase had lower percentage agreement rates compared to other phases of mitosis. Conclusion. This dataset stands as the largest international consensus study for mitotic figures to date and can be utilized as a training set for future studies. The agreement range reflects a spectrum of criteria that pathologists use to decide what constitutes a mitotic figure, which may have potential implications in tumor diagnostics and clinical management.

Introduction

The identification and enumeration of mitotic figures are essential for diagnosing, classifying, and grading of various tumor types. Mitotic figures are routinely assessed in tumor types that include neuroendocrine tumors,^1,2 breast carcinomas, ³ soft tissue sarcoma, ⁴ gastrointestinal stromal tumors (GIST),^5,6 cutaneous melanoma,^7,8 and thyroid tumors, ⁹ among others. Visual quantification of mitotic figures on hematoxylin and eosin (H&E) stained histologic sections serves as a surrogate measure for the proliferative capacity of tumors and has been shown as an independent prognostic factor of outcome in many cancer types.^10–12 Mitotic counts, traditionally quantified per high-powered microscope field, have evolved to standardized area measurements in the most current classification systems in order to address the variability with different microscope lenses. ¹³ Therefore, mitotic counts are typically defined as the number of mitotic figures in a given unit area and are expressed in discrete ranges that are used to guide clinical decision-making and risk stratification of patients.

The identification and quantification of mitotic figures can be both subjective and time-consuming, with poor reproducibility.^2,14,15 Previous studies have shown moderate agreement rates for mitotic counts, even within a standard area.^16–18 While pathologists are not required to characterize mitotic figures into various phases on histologic tumor slides in routine practice, the morphologies of mitotic figures may vary widely, with distinct appearances at each phase. A deeper understanding of these variations could lead to more accurate mitotic counts. Donovan et al defined morphologies of various phases of prophase, metaphase, anaphase, and telophase that occur during mitosis. Additionally, the authors described atypical mitotic figures, and potential mitotic figure mimics. ¹⁹ Examples of mitotic figure mimics include apoptotic figures, hyperchromatic or deformed nuclei, karyorrhectic debris, inflammatory cells, and tissue artifacts.^19,20 The process of distinguishing mitotic figures and atypical mitotic figures from mitotic figure mimics is challenging and is often subject to inter-observer variability.^19,21 Inaccuracies in mitotic figure interpretation and quantification of mitotic counts may lead to misdiagnoses and ultimately result in inaccurate classifications of tumors in clinical practice. ¹⁴ Thus, to produce consistent and reproducible mitotic counts, it is critical for pathologists to distinguish mitotic figures and atypical mitotic figures from mitotic figure mimics, and to recognize common pitfalls encountered in making this distinction.

There is increasing interest in developing AI-assisted computational pathology tools that can automate subjective and time-consuming tasks in current pathology workflows. ²² One such potential tool would identify and count mitotic figures across a whole-slide image (WSI), resulting in a holistic measure of mitotic count, in contrast to the current method of generating a mitotic count from a limited set of areas selected by the pathologist. Given the prevalence of mitotic figure datasets from grand challenges, ²³ automated systems are well positioned to solve the challenges of counting mitotic figures. However, their performance has been shown to decline in varied clinical settings due to the challenges in differentiating between mitotic figures and their mimics, which often have differing sizes, shapes, and structural elements. ²⁴ One limitation in developing robust machine learning applications that can accurately enumerate mitotic figures across a WSI is the lack of comprehensive diverse properly labeled datasets scored by a diverse panel of expert pathologists. ²⁵

There exists a paucity of data documenting the consistency in interpreting a large dataset of mitotic figures among pathologists from diverse backgrounds and clinical practice settings. With the availability of publicly accessible datasets and the increasing popularity of professional social media use among pathologists, we identified a unique opportunity to foster research collaborations extending beyond geographical and institutional barriers in order to generate novel insights and robust datasets. These datasets could then support the training of robust artificial intelligence-based tools within pathology workflows promising to automate tedious and error-prone tasks that can ultimately impact patient care. In this study, we leveraged publicly accessible datasets and social media to recruit an international group of 85 pathologists to collectively generate and score an image database of 1010 mitotic figures.

Design

Study Cohort

The study was announced (July 13, 2022) on Twitter (currently known as X) to recruit practicing pathologists across the world. Participants were screened based on their online professional profile and confirmed by their institutional affiliation to ensure they were at the level of a staff pathologist. No other exclusion criteria were utilized and pathologists from around the world were included.

Collection of 1000 Mitotic Figures

Each pathologist who completed the survey was sent a comprehensive protocol detailing the instructions for submitting their mitotic figures. Using the online dataset from The Cancer Genome Atlas (TCGA), each participant randomly selected a diagnostic slide from the dataset of their choosing. The selection criteria was open to all available datasets on the TCGA and not limited to specific tumors where mitotic counts are routinely assessed in clinical practice. Using QuPath, an open-source bioimage analysis tool, the pathologist annotated a 2 mm² area of tumor with the highest proliferative capacity on the tumor. ²⁶ Within this defined area, the pathologist annotated 10-20 mitotic figures that would have been included in a mitotic count to be used for patient care in their clinical practice. The first 1000 submitted mitotic figures (with 10 extra tiles included for redundancy) were used to create an image dataset, with each figure transformed into an individual tile at 40x magnification.

Scoring of Mitotic Figures

The dataset was split into four equal parts based on the order in which the tiles were submitted and uploaded into four separate Google Forms. For each tile, participants were asked whether the tile constituted a mitotic figure (Yes/No). At the end of each of the four surveys, participants were asked to provide qualitative feedback on tiles with uncertainty. The dataset was split into parts to create a workflow that saved the rater's progress and did not need to be completed in a single setting.

Determination of Mitotic Phase and Mitotic Figure Mimics

Each submitted mitotic figure was classified as the most probable phase of mitosis by a pathologist (MJC). Identification of cells in prophase is challenging with light microscopes, given the subtle morphologic changes in this phase. Cells in mitosis that appear as dark chromosome aggregates are most likely in the final stages of prophase, commonly referred to as prometaphase. ¹⁹ The term prometaphase will be used going forward to denote mitotic figures in the later phase of prophase that can be reliably identified on histologic sections. During the review by MJC, a potential mitotic figure mimic was assigned descriptively to a subset of the tiles. Given the lack of a gold standard for what constitutes a mitotic figure from a mimic, it is not possible to definitively enumerate the exact number of tiles in these categories. This descriptive analysis provides insight into the distribution of potential mimics in each phase of the cell cycle.

Statistics

Descriptive statistics were calculated at both the pathologist and tile level. At the pathologist level, an agreement rate was calculated by the proportion of tiles classified as positive over the total number of tiles rated. At the tile level, an agreement rate was calculated by the proportion of pathologists that classified the tile as a mitotic figure over the total number of ratings. Fleiss' kappa (κ) was used as a statistical measure of inter-rater agreement, where kappa values can be interpreted as poor agreement (<0), slight agreement (0-0.2), fair agreement (0.2-0.4), moderate agreement (0.4-0.6), good agreement (0.6-0.8), and almost perfect agreement (>0.8). ²⁷ Further, we utilized a one-way ANOVA and a subsequent Tukey's post hoc test to evaluate pathologist agreement rates based on the phases of mitosis. Additional unsupervised hierarchical clustering analysis was performed for both mitotic figures and pathologists, leading to the generation of corresponding heatmap denograms.

Results

Study Description

A group of 55 international staff pathologists from around the globe contributed to the mitotic figures dataset consisting of 1010 tiles, and 85 international staff pathologists scored the dataset. The group had varying years of practice and were from 24 countries (Figure 1). The dataset spans 19 distinct disease sites from the TCGA open-source database, with the largest contributors being cervical squamous carcinoma (120 tiles), sarcoma (104 tiles), and colorectal adenocarcinoma (91 tiles), followed by lung squamous cell carcinoma and breast carcinoma (both 80 tiles each) (Figure 2). The study spanned a total of 57 days from conceptualization and mitotic figure collection to scoring of the dataset.

Figure 1.

Demographics of pathologists scoring the mitotic figures dataset. Years of practice among the staff pathologists (A) and location of practice categorized by continent (B).

Figure 2.

Distribution of tumor types used in the 1010 mitotic figures dataset. Abbreviations: GBM, glioblastoma multiforme; H&N SCC, head and neck squamous cell carcinoma; HCC, hepatocellular carcinoma; DLBC, diffuse large B-cell lymphoma; Cervical SCC, cervical squamous cell carcinoma.

Assessment of Interobserver Variability

At the pathologist level, there was a median agreement rate of 80.2% of the mitotic figure tiles (mean 78.5 ± SD 12.4), with an agreement range of 42.0% to 95.7% (Figure 3A). At the tile level, there was a median agreement rate of 87.0% (mean 78.5 ± SD 22.2) with an agreement range of 1.2% to 100%, reflecting a subset of tiles with mixed or low agreement. Of all tiles submitted, 41.6% had a high agreement score in the 90%-100% category (Figure 4B). The average kappa value of 0.284 indicates fair inter-rater agreement across all tiles.

Figure 3.

Agreement rates among pathologists across the overall dataset (A). Mitotic figure tiles categorized based on agreement rate from 85 pathologists (B).

Figure 4.

Agreement rates with mitotic figures stratified based on phase of mitosis (A). Representative images of different mitosis phases of high and low agreement (B, 40x magnification).

Agreement Rate and Potential Mimics Based on Phase of Mitosis

Mitotic figures in prometaphase showed lower agreement rates (median 81.2%) compared to mitotic figures in metaphase (median 89.4, p < 0.0001), and anaphase/telophase (median 85.9%, p < 0.0001), with no differences between other groups (Figure 4). In order to better understand the variability in mitotic figure enumeration, each mitotic figure was assessed for potential mimics. A potential mimic was assigned in 467 out of 1010 (46.2%) tiles within the dataset. Mitotic figures with low agreement rates were most likely to resemble mitotic figure mimic, with the most common mitotic figure mimic being karyorrhectic debris, immune cells, stromal cells, and two cells in close proximity. All phases of mitosis could mimic karyorrhectic debris, whereas lymphocytes were most likely to resemble mitosis in prometaphase, and stromal cells were likely to mimic mitosis in metaphase (Figure 5).

Figure 5.

Mitotic figures within the dataset characterized by potential mitotic figure mimics (A). Representative microscopy images of mitotic figure mimics observed in the dataset (B).

Cluster Analysis of Pathologist Agreement in Mitotic Figure Classification

To better understand the clustering of mitotic figures and pathologists, heatmap denograms following unsupervised hierarchical clustering were generated (Supplemental Figures 1 and 3). Representative mitotic figures from the clusters are included in Supplemental Figures 2 and 4. Additional analysis of the results from the hierarchical clustering showed no association between the clusters of pathologists and their subspecialty, region of practice, or years of practice that was previously collected in the study.

Discussion

This study leveraged a social media platform to recruit a large cohort of pathologists to generate and score the largest dataset of mitotic figures to date, 1010 proposed mitotic figures. This approach allows for rapidly generating robust datasets generated by an international cohort of pathologists. In this study, we were able to move from conceptualization through recruitment, onboarding, and complete primary data collection in 57 days. The use of public datasets minimized barriers for data sharing, such as data transfer agreements and multi-institutional ethics approval. To our knowledge, this is the largest international consensus study for mitotic figures to date, with the overall goal of examining the consistency in interpreting mitotic figures among pathologists.

We found a spectrum of opinions on what constitutes a mitotic figure, which is consistent with existing literature.^15,16,19 A key finding was the lower agreement for prometaphase mitotic figures compared to other phases of mitosis. As all of the mitotic figures within the dataset were entities pathologists would classify as mitotic figures in their clinical practice, it is unsurprising that a vast majority of the dataset is in the upper agreement range (>80%). A subset of mitotic figures in the intermediate agreement range highlights the subjectivity of the task of identifying mitotic figures (Supplemental Figure 1). Individual pathologists also have a spectrum of agreement rates, with clusters of pathologists having variable thresholds for what constitutes a mitotic figure (Supplemental Figure 3). Despite having 85 pathologists that served as raters, the study was insufficiently powered to determine if this varying threshold was influenced by subspecialty, region of practice, or years of practice.

Pathologists’ interpretation of mitotic figures is inherently variable and previous studies have documented intermediate agreement rates for mitotic counts (κ = 0.34-0.64).^17,28,29 Our study presented a lower kappa value (κ = 0.284), indicating fair agreement. One possible explanation for this lower agreement rate is the vast diversity of our sample of pathologists compared to previous studies. A second explanation, highlighted by participant feedback, is the unique challenge of interpreting mitotic figures on WSI due to limitations of scan quality and the inability to refocus the specimen—a functionality that is possible on analog microscopes. Studies have previously documented the decrease in accuracy in detecting mitoses using WSI rather than microscopy.^30,31 Distinguishing mitotic figures and atypical mitotic figures from mitotic figure mimics, which include apoptotic bodies, hyperchromatic nuclei, karyorrhectic debris, inflammatory cells, two cells in close proximity, and tissue artifacts,^2,19 is one of the most error-prone steps in determining mitotic counts.

Due to the pitfalls surrounding the identification and enumeration of mitotic figures, other means of obtaining proliferation markers have been variably utilized. Immunohistochemistry with phosphorylated histone H3 (pHH3), a marker highly specific to mitotic figures, can yield improved mitotic figure counting accuracy and efficiency. ³² However, implementation is not practical in routine practice as this added step lends itself to diagnostic pitfalls with immunohistochemistry, increases in operational costs, and further delays in slide preparation. Further, studies defining the use of mitotic figures in grading and diagnosing cancers may need to be repeated to account for changes in the threshold for classification. The Ki67 proliferation index is routinely used in tumors, such as non-pulmonary neuroendocrine tumors, and is a more general marker of proliferation in which expression is not limited to cells undergoing mitosis.^33,34 However, Ki67 indices are not predictive for all cancer types, and there are similar challenges with the lack of standardization and variation in quantification and staining methodologies.^34,35

With major advancements in computational tools, automated mitotic figure detection and enumeration have garnered significant research interest. However, proposed models for automated mitotic figure classification have room for improvement and tend to lack generalizability in different disease sites and clinical settings.^25,36–38 A majority of these models are trained by limited publicly available mitosis detection datasets from grand challenges such as MITOS, ³⁹ AMIDA, ⁴⁰ MITOS-ATYPIA, ⁴¹ TUPAC, ⁴² and MIDOG, ²³ that provides standardized WSIs for researchers to train and test automated mitoses detection models, with recent models adopting increasingly complex neural networks using a combination of these datasets. Models are only as good as the data they are trained on, and datasets are often limited in the breadth of disease types and in selection of mitotic figures by a small number of observers. Ideally, automated mitotic figure models should be developed with datasets that account for mitotic figures within different disease sites, include mitotic figures in different phases of cell division, account for variations in tissue processing that may lead to staining variability, and allow variations in slide scanning quality. Previous datasets present binary information: each cell is classified as either a mitotic figure or not. Our dataset is unique as we have 85 observers, allowing for an agreement or consensus rate for each cell. Instead of training models with a simple binary system, our dataset affords a way for weighted confidence to be integrated into the training set, with high agreement mitotic figures given proportionally more weight.

Our study has several limitations. We recognize the challenge with assigning binary values to mitotic figures where uncertainty is present. In order to limit the number of repetitive inputs from staff pathologists who volunteered their time to contribute to this study, we did not capture confidence levels from pathologists when scoring the mitotic figures. Moreover, as there is no gold standard for what definitively does and does not constitute a mitotic figure, we were unable to include true negatives in our dataset which would have increased the robustness of the collected data and reduced bias when scoring the dataset. Further, it is possible that providing a dataset of images that other colleagues have denoted as mitotic figures might influence scorers to agree when evaluating tiles containing mitotic figures with nonspecific features. During the collection of mitotic figures, pathologists were instructed to choose a histologic slide from a disease site of their choosing. The use of TCGA slides was both a limitation and advantage for this study. Slides were prepared and scanned at various institutions, resulting in heterogeneous H&E appearance and scanning preparation. ⁴³ This heterogeneity may have made the identification of mitotic figures more challenging for participants, but it also helped provide a more diverse data set that can be utilized in subsequent training systems.

The field of pathology is becoming increasingly digitized, allowing pathologists novel ways to rapidly share data and contribute to international research collaborations with the promise of furthering patient care. ⁴⁴ In this study, we leverage an extensive international professional network of pathologists on social media to develop and score the largest dataset of mitotic figures to date and to capture the subjectivity in interpreting mitotic figures. This project stands as an example of providing equitable and inclusive opportunities for research among a group of scholars connected through social media. We anticipate the increasing adoption of technology-driven approaches to generate robust datasets that can fuel artificial-intelligence-based applications in pathology and drastically reduce the number of subjective and time-consuming tasks for pathologists. Initiatives, such as these, can assist in implementing well-defined guidelines to standardize mitotic counts and assist pathologists in tumor assessment.