Inter‐ and intraobserver agreement in whole‐slide digital ThinPrep samples of low‐grade squamous lesions of the cervix uteri with known high‐risk HPV status: A multicentric international study

Abstract

Background

High‐risk human papilloma virus (HR HPV) testing and liquid‐based cytology are used for primary cervical screening. Digital cytology, based on whole‐slide scanned samples, is a promising technique for teaching and diagnostic purposes. The aim of our study was to evaluate the interobserver and intraobserver variation in low‐grade squamous lesions, HR HPV status bias, and the use of whole‐slide scanned digital cervical cytology slides.

Methods

Fifteen expert cytopathologists evaluated 71 digitalized ThinPrep slides (31 atypical squamous cells of undetermined significance [ASC‐US], 21 negative for intraepithelial lesion or malignancy, and 19 low‐grade squamous intraepithelial lesion cases). HR HPV data were accessible only in the second round.

Results

In interobserver analysis, Kendall’s coefficient of concordance was 0.52 in the first round and 0.58 in the second round. Fleiss’ kappa values were 0.29 in the first round and 0.31 in the second round. In the ASC‐US category, Fleiss kappa increased from 0.19 to 0.22 in the second round and the increase was even higher expressed by Kendall’s coefficient: from 0.42 to 0.52. In intraobserver analysis, personal scores were higher in the second round.

Conclusions

The interobserver and intraobserver variability in low‐grade squamous lesions was within fair agreement values in the present study, in line with previous works. The comparison of two rounds showed that expert cytopathologists are generally unbiased by the knowledge of HR HPV data, but that being informed of the HR HPV status leads to a better agreement. Stain quality and back discomfort were highlighted as factors affecting digital cytopathology use.

Short abstract

The interobserver and intraobserver variability in low‐grade squamous lesions was within fair agreement values and knowing high‐risk human papillomavirus status leads to better agreement. Stain quality and back discomfort were highlighted as factors affecting digital cytopathology use.

INTRODUCTION

Since George Papanicolaou established the role of the Papanicolaou (Pap) smear in cervical cancer detection and prevention, ¹ several milestones have happened in cervical cytology. In particular, the introduction of liquid‐based cytology (LBC) and more recently high‐risk human papilloma virus (HR HPV) molecular tests have had a decisive impact on cervical cancer screening. ² In Europe, many countries have implemented HR HPV primary screening programs either regionally or nationally. ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ In the era of HR HPV primary screening, the number of cervical cytology slides has dramatically decreased, but there is a relative increase in atypical cytological findings at least in the first screening round. ⁶ , ⁷

Low‐grade changes of the squamous epithelia include both atypical squamous cells of undetermined significance (ASC‐US) and low‐grade squamous intraepithelial lesion (LSIL). The ASC‐US category is defined by The Bethesda System for Reporting Cervical Cytology (TBS) ⁹ as changes suggestive but not sufficient for an LSIL interpretation. Despite defined criteria of ASC‐US and particularly LSIL, several studies have shown that the intra‐ and interobserver agreement of these categories is low. ¹⁰ , ¹¹ In fact, all aspects of cervical cancer diagnostic workup (i.e., cytology, histology, and colposcopy) show interpretive variability. ¹⁰ , ¹¹ , ¹² , ¹³

Because most screening detected lesions are low‐grade lesions, particularly after the first HR HPV screening round, and low‐grade lesions are among those with the poorest agreement, the issue is one of high importance for cytologists and pathologists involved in screening programs.

Digital cytology has been widely used in cytology external quality assurance programs ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ and cytology education and training, ¹⁹ , ²⁰ , ²¹ but less so in routine cytology practice. Despite revolutionary developments such as the creation of large digital data sets coupled with deep learning methods and artificial intelligence, the use of digital cytology has lagged behind its use in surgical pathology. There are several technical challenges related to cytology image acquisition such as focusing on three‐dimensional cell groups and navigating slides at high magnification when screening impairing cytology workflow, the large file size and scanning times for whole‐slide images in cytology. ²² , ²⁵ , ²³ , ²⁴ Last, but not least, education in the use of digital cytology and ergonomics issues ²⁶ should also be taken into the account.

The aims of the present study were (1) to evaluate the interobserver and intraobserver agreement in low‐grade squamous lesions and (2) to evaluate HR HPV status biases among expert cytopathologists with HR HPV screening and/or triaging experience, and (3) to assess the ease of use and compliance with the digital platform.

MATERIALS AND METHODS

Selection and characterization of cases

Three expert pathologists (B.C.P., M.N., G.N.) retrieved 71 cervical LBCs (ThinPrep, Hologic, Marlborough, Massachusetts, USA) slides from the archives of their laboratories. The original cytomorphological diagnosis according to TBS were ASC‐US in 31 cases, NILM in 21, and LSIL in 19. According to HR HPV status, there were 20 HR HPV–positive and 11 HR HPV–negative ASC‐US cases, 16 HR HPV–positive and 3 HR HPV–negative LSIL cases, and 11 HR HPV–positive and 10 HR HPV–negative NILM cases, respectively. All enrolled cases had known HR HPV status and either histology or cytology follow‐up or 3 years of clinical follow‐up. All LSIL cases had histology follow‐up. Follow‐up details are in Figure 1.

FIGURE 1.

Heat map summarizing all answers in both rounds according to diagnoses. ASC‐H indicates atypical squamous cells – cannot exclude high‐grade squamous intraepithelial lesion; ASC‐US, atypical squamous cells – undetermined significance; HR HPV, high‐risk human papilloma virus; HSIL, high‐grade squamous intraepithelial lesion; LSIL, low‐grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy.

Scanning and digital platform

The slides were anonymized, assigned a unique study number, scanned by a Hamamatsu scanner at 40× magnification without any depth of focus (Hamamatsu Photonics, Hertfordshire, UK), and posted on internet platform (Eurocytology, www.eurocytology.cloud) accessible by individual password.

All slides were annotated with regions of interest with a total of 576 annotations (mean, 6.12 [range, 2–16] annotations per case).

The evaluation form included patient age and brief clinical data. Dedicated medical grade computer screen was used only by two participants (13%). HR HPV status was available for the observers only in the second round. The participants coded the cases according to TBS categories for squamous epithelia (0 negative, 1 ASC‐US, 2 LSIL, 3 atypical squamous cells, cannot exclude HSIL (ASC‐H), 4 high‐grade squamous intraepithelial lesion [HSIL]).

Rounds and participants

The first round was opened in January 2021 and the second round in March 2021. Each round was open for 6 weeks. Participants were 15 expert cytopathologists (eight females, seven males) from 13 European countries. The cytopathology, digital pathology, HR HPV screening/triaging experience, and compliance data were collected with questionnaires.

Statistical methods

Categorical variables were expressed as frequencies (and percentages) and numerical variables as means (and SD). Categorical variables were expressed as frequencies (and percentages) and numerical variables as means (and SD). Interobserver agreement was assessed by Fleiss’ kappa and Kendall's coefficient of concordance between all raters in two rounds. Bootstrap resampling was used to compute the SDs of the indices. To compare the indices in the two rounds (i.e., intraobserver variation for individual raters), the methods by Feldt, Woodruff, and Salih ²⁷ were used to provide p values. The free software R (http://www.r‐project.org/) was used for statistical analyses. The level of significance was set at p < .05.

The strength of association is defined for Fleiss' kappa as follows: 1 for perfect agreement, 0.81–0.99 for almost perfect agreement, 0.61–0.80 for substantial agreement, 0.41–0.60 for moderate agreement, 0.21–0.40 for fair agreement, 0–0.20 slight agreement, and < 0 poor agreement. The strength of association is defined for Kendall's coefficient of concordance as follows: ±0.30 or above strong, ±0.20 to 0.29 moderate, ±0.10 to 0.19 weak, and less than ±0.10 very weak.

Results

Participants

Eleven participants (73.3%) were from an academic hospital, three (20.0%) from other hospitals, and one (6.7%) from a private laboratory. The cytopathology practice of the participants was 24.20 ± 7.95 years on average ranging from 8 to 38 years. LBC experience was on average 13.47 ± 8.28 years with the longest period being 30 years. Cervical cytopathology on average comprised 33% of participants´ workload (with up to 70%) with an annual average of 5967 ± 4870.84 reviewed slides (range, 300–15000 slides).

The average experience time with the use of HR HPV in a screening context was 14.93 ± 5.91 years, with a range of 5 to 30 years. In particular, HR HPV primary screening practical experience was on average 1.50 ± 1.54 years, with the longest experience being 5 years. HR HPV triage experience was on average 11.07 ± 6.83 years with the longest experience of 25 years.

The washout period between the two rounds was 5.1 weeks on average.

Interobserver agreement

Kendall’s coefficient of concordance was 0.52 ± 0.04 in the first round and 0.58 ± 0.04 in the second round. Fleiss´ kappa values were 0.29 ± 0.03 in the first round and 0.31 ± 0.03 in the second round (Table 1; Figures 2 and 3).

TABLE 1.

Fleiss’ kappas/Kendall’s coefficients in total and in all three TBS categories according to the first and second rounds

Diagnostic category	First round (unknown HR HPV status) Fleiss’ kappa	First round (unknown HR HPV status) Kendall’s coefficient	Second round (known HR HPV status) Fleiss’ kappa	Second round (known HR HPV status) Kendall’s coefficient
All	0.29 ± 0.03	0.52 ± 0.04	0.31 ± 0.03	0.58 ± 0.04
NILM	0.32 ± 0.05	0.62 ± 0.09	0.28 ± 0.07	0.46 ± 0.08*
ASC‐US	0.19 ± 0.04	0.42 ± 0.05	0.22 ± 0.04	0.52 ± 0.05
LSIL	0.28 ± 0.06	0.47 ± 0.07	0.16 ± 0.04	0.38 ± 0.07

Abbreviations: ASC‐US, atypical squamous cells – undetermined significance; HR HPV, high‐risk human papilloma virus; LSIL, low‐grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy; TBS, The Bethesda System for Reporting Cervical Cytology.

^* p = .02.

FIGURE 2.

Kendall’s coefficients in the first round without HR HPV status and in the second round with HR HPV status knowledge in all cases and according to TBS categories. ASC‐US indicates atypical squamous cells – undetermined significance; HR HPV, high‐risk human papilloma virus; LSIL, low‐grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy; TBS, The Bethesda System for Reporting Cervical Cytology.

FIGURE 3.

Fleiss´ kappa values in the first round without HR HPV status and in the second round with HR HPV status knowledge in all cases and according to TBS categories. ASC‐US indicates atypical squamous cells – undetermined significance; HR HPV, high‐risk human papilloma virus; LSIL, low‐grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy; TBS, The Bethesda System for Reporting Cervical Cytology.

Senior observers, characterized as those with a working experience of more than 20 years, rated with Kendall’s coefficient of concordance 0.54 in the first round and 0.53 in the second round. Fleiss´ kappa values for senior experts were 0.30 in the first round and 0.31 in the second round.

The agreement evaluated as Fleiss´ kappa values/Kendall’s coefficients for all three Bethesda categories separately (i.e., for NILM, ASC‐US and LSIL in the first and second rounds) is shown in Table 1. Importantly, the highest agreement was seen for the NILM category followed by low‐grade dysplasia. In numbers, Fleiss´ kappa values ranged from 0.28 ± 0.07 to 0.32 ± 0.05 and Kendall’s coefficient ranged from 0.46 ± 0.08 to 0.62 ± 0.09 in NILM category. Fleiss´ kappa ranged from 0.19 ± 0.04 to 0.22 ± 0.04 and Kendall’s coefficient ranged from 0.42 ± 0.05 to 0.52 ± 0.05 for the ASC‐US category. Finally, Fleiss´ kappa values ranged from 0.16 ± 0.04 to 0.28 ± 0.06 and Kendall’s coefficient ranged from 0.38 ± 0.07 to 0.47 ± 0.07 for the LSIL category. A heat map summarizing all the respondent’s answers in both rounds according to diagnoses is shown in Figure 1. There is less uniformity in ASC‐US cases with more diverse answers in comparison to NILM and LSIL cases. Single cases with lower agreement were nevertheless present in all categories: case 32 in the NILM category with predominantly LSIL answers, case 2 in the ASC‐US category with mainly NILM answers, and case 42 in the LSIL category with some HSIL answers. Few ASC‐US cases had also trended to be answered as ASC‐H category.

Observers 7, 10, and 14 tended to more overdiagnosing and observers 4 and 9 to underdiagnosing. Nine (60%) participants admit overdiagnosing in uncertain cases.

Intraobserver agreement: HR HPV bias

In the NILM category, the knowledge of HR HPV status statistically decreased the Kendall’s coefficients (0.62 ± 0.09 in the first round vs. 0.46 ± 0.08 in the second round, p = .02). HR HPV status also influenced categorization in mild lesions because the agreement in the ASC‐US category improved with the knowledge of the HR HPV status: Fleiss kappa increased from 0.19 ± 0.04 without the HR HPV status knowledge to 0.22 ± 0.04 with a known HR HPV result and the increase was even higher with Kendall’s coefficient: from 0.42 ± 0.05 to 0.52 ± 0.05. On the other hand, the LSIL category decreased in agreement with HR HPV status knowledge: Fleiss´ kappa decreased from 0.28 ± 0.06 without HR HPV information to 0.16 ± 0.04 with the known status and Kendall’s coefficient decreased from 0.47 ± 0.07 without knowledge of the HR HPV status to 0.38 ± 0.07 with it. Overall, personal scores were higher (in better agreement) in the second round, when the HR HPV status was known: the statistical significance was p = .004753 for Fleiss´ kappa and p = .0005778 for Kendall’s coefficient.

Digital experience and compliance with digital platform

The majority of participants already had some experience with digital pathology (n = 12, 80%), mainly from research (histopathology, 80%; cytopathology, 73%), and, to a lesser extent, from routine work (histopathology, 27%; cytopathology, 13%).

The digital cytology experience and compliance in the present study was evaluated on average (scale 1–5, with 1 being the best) and the results are as follows: scores 2.8 for time consuming in comparison with conventional microscopy, 3.0 for eye discomfort, and 3.6 for back discomfort. Concerning technical aspects (scale 1–5, 1 every case, 5 no case), problems with image sharpness/focusing were rated as 3.7, problems with stain quality as 4.4, and problems with scan quality as 3.9. Overall, 47% of participants would recommend digital cytology in their department based on the experience in the present study.

Discussion

Since George Papanicolaou’s era, cervical cancer screening has changed massively: conventional smears have been widely replaced by HR HPV testing and liquid‐based cytology, whereas digitalization is regarded as a very promising technique for the future. In the modern setting of an organized, HR HPV‐based cervical screening, most screening detected lesions are incidental low‐grade lesions, particularly after the first HR HPV‐screening round. ²⁸ Because low‐grade lesions are among those with the poorest agreement, this is a potential issue for cytologists and pathologists involved in screening programs. The knowledge of the HR HPV status has been reported as beneficial for the cytological interpretation by some authors, ²⁹ whereas others have reported a potential issue with that. ³⁰

In the present study, the interobserver and intraobserver variability in low‐grade squamous lesions was within fair agreement (Fleiss´ kappas 0.29–0.31 and Kendall’s coefficient of concordance 0.52–0.58). These findings are in agreement with previously published results. ³¹ Several studies with quality control and assurance scope showed ranges for kappa value of 0.54–0.69 ³² and even 0.66–0.77 with improvement to 0.86–0.93. ³³

Nevertheless, the ASC‐US category failed in the majority of studies obtaining a kappa value of 0.40 ³² or even 0.16. ³¹ When the ASC‐US category was pooled into ASC‐US favor reactive versus ASC‐US favor SIL, the value was 0.23 and 0.36, respectively, and despite subcategorization of the values, the agreement remained poor to fair. ³¹ Pioneering BIRST‐1 and BIRST‐2 studies showed timeline improvement even in borderline categories including the ASC‐US category with an increase from 39.9% to 61.7% agreement in the BIRST‐2 study; ³³ , ³⁴ however, direct comparison to our results is not possible as different values of concordance have been used. The use of various concordance values makes comparison of various studies challenging. In our series, Fleiss´ kappa values ranged from 0.19 to 0.22 and Kendall’s coefficient ranged from 0.42 to 0.52 in the ASC‐US category. In contrast, Stoler and Schiffman showed a kappa of 0.64 in the ASC‐US category, superior to 0.51 in the LSIL and 0.56 in the NILM categories. ¹⁰ In the same study, colposcopic biopsy and loop electrosurgical excision procedure (LEEP) revealed comparable kappa values to liquid‐based cytology. ¹⁰ Variability is a problem in all diagnostic aspects of cervical cancer, involving cytology, histology, and colposcopy. All three methods are subjected to interpretative variability. ¹⁰ , ¹¹ , ¹² , ¹³ Practically, the management of a particular TBS category may vary in different countries, so there may be a tendency to categorize according to local clinical management and guidelines.

Bigras et al. analyzed ASC‐US–related cytomorphological features in detail and nuclear features had significantly lower reproducibility (kappa 0.46) with the lowest agreement for chromatin texture in comparison to other cytological findings. ³⁶ Interestingly, fellows´ conditional kappa value were 0.70 for ASC‐US in comparison to 0.60 for cytopathologists in another study showing the role of active education. ³⁷ From that perspective, it will be valuable in the near feature to analyze our samples for particular features related to the highest and lowest agreement. Endocervical atypical lesions with less agreement are often impaired by degenerative changes and lack of nuclear enlargement. ³⁸

When the two rounds were compared, expert cytopathologists were not biased by HR HPV data generally. Nevertheless, personal scores showed better agreement in the second round with known HR HPV status with statistical significance (p = .004753 for Fleiss´ kappa and p = .0005778 for Kendall’s coefficient). HR HPV known status influenced assessment in the ASC‐US category in the present study: the agreement increased from 0.19 to 0.22 by Fleiss´ kappa and grew even higher by Kendall’s coefficient from 0.42 to 0.52.

Several studies have reported HR HPV–positive cases to be highly likely reported as abnormal, so there is a bias of HR HPV knowledge in the interpretation of Pap smears or liquid‐based preparations. ²⁹ , ³⁸ , ³⁹ , ⁴⁰ The phenomenon was noticed both in cytopathologist and cytotechnologist studies. ²⁹ , ³⁸ , ³⁹ , ⁴⁰ The Tahoe study showed a significant difference in the ASC‐US, LSIL, and HSIL categories, ⁴¹ whereas we identified a p value significance only in the NILM category and personal scores. Generally, if the observer knows the HR HPV status, he or she tends to interpret minor cellular changes as abnormal findings upgrading the TBS category. Nevertheless, also downgrading of HR HPV–negative ASC‐US has also been observed. ³⁹ The HR HPV/Pap triage algorithm differs from a Pap smear–alone algorithm, so the diagnostic accuracy needs to be reevaluated in the HR HPV/Pap smear/liquid‐based preparation triage and Pap smear/liquid‐based preparation alone. ³⁰ In the analysis of the College of American Pathologists PAP Education program LSIL was misclassified in 2.4% of cases and ThinPrep LSIL slides were more likely to be misclassified than SurePath LSIL slides. ⁴²

HR HPV primary screening programs have shown an increase in colposcopy referral rates ⁶ , ⁷ and the role of cytology triage is pivotal. ⁶ Importantly, prior knowledge of HR HPV status improved CIN2+ detection ⁴³ in agreement with higher detection rates in HR HPV primary screening programs. ⁶

Although cytology still has a main role in cervical screening, different challenges need to be faced by the cytological community in the future. In the spite of efforts to promote the training of cytotechnicians, ⁴⁴ experienced cytologists are getting older and it will be increasingly difficult to replace them, particularly considering the suboptimal attention devoted to cytopathology education in most academic settings. Thus, the development of new digital systems based on whole‐slide technology seems very promising.

Digital platforms have been gradually used in research, quality assurance programs ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ and routine diagnostics despite technical challenges in focusing on three‐dimensional cell groups, large file size, and increased acquisition times for whole‐slide images in cytology. ²² , ²⁵ , ²³ , ²⁴ In our study, technical problems with image sharpness/focusing and scan quality were less common than with stain quality, suggesting a bright future for digital cytopathology. Notably, dedicated medical‐grade computer screens were used only on a few occasions, and their use would probably increase both ease of use and diagnoses. High staining quality and uniformity in routine stains represents the basis of digitalization. Indeed, pathologists are good in getting used to new systems and tools, and developing confidence, and 47% of participants would recommend digital cytology in their department based on their experience with the present study. Further standardization of digital cytopathology will reduce the problems and pitfalls.

Skills with digital pathology have been gained mainly from histopathology and research in the present study accompanying survey. The time needed to examine virtual gynecological cytology slides was higher than conventional microscopy slides in previous study, ²¹ our participants also subjectively scaled time demands at 2.8 in the 1 to 5 range, with 1 being best in mild preference of conventional microscopy. Ergonomic issues such as eye and back discomfort were graded 3 and 3.6, respectively. Pathology routine reporting is repetitive and continuous, so the risk of musculoskeletal injuries namely back and carpal tunnel syndrome is significant. In addition to visual fatigue, work environment, noise, and temperature should also be considered. ²⁶

In fact, whole‐slide digital cytology will profit enormously from artificial intelligence–based diagnostic algorithms. In the past, several attempts were made with automated screening systems, ⁴⁴ , ⁴⁵ , ⁴⁶ which were based on glass slides. Applying artificial intelligence to virtual slides may open new perspectives, reducing the strain of the screening activity and helping in the interpretation of slides in the context of large population‐based screening programs. Artificial intelligence–driven digital pathology has been raising several ethical challenges, namely privacy of data and form of consent when sharing of data for research, commercial benefit of data and related public trust, and biases of data series. Involvement of nonmedical experts in diagnostic workup may also awaken ethical and responsibility issues. ⁴⁷ , ⁴⁸

In conclusion, the interobserver and intraobserver variability in low grade squamous lesions was within fair agreement values in the present study in agreement with previous studies. The comparison of two rounds showed that expert cytopathologists are not generally biased by HR HPV data, but knowledge of HR HPV status leads to a better agreement. Stain quality and back discomfort were the main complaints in using digital cytopathology in our compliance survey. Following a good compliance with digital cytopathology, 47% of participants would recommend digital cytopathology in their department based on their experience in the present study.

AUTHOR CONTRIBUTIONS

Ivana Kholová: Conceptualization, data curation, formal analysis, investigation, methodology, project administration, resources, visualization, supervision, validation, writing‐original draft, and writing‐editing. Giovanni Negri: Conceptualization, data curation, resources, validation, and writing‐editing. Maria Nasioutziki: Conceptualization, data curation, resources, validation, and writing‐editing. Laura Ventura: Formal analysis, methodology, validation, writing‐original draft, and writing‐editing. Arrigo Capitanio: Methodology, project administration, resources, and writing‐editing. Massimo Bongiovanni: Conceptualization, data curation, validation, and writing‐editing. Paul A. Cross: Conceptualization, data curation, writing‐original draft, and writing‐editing. Claire Bourgain: Data curation and writing‐editing. Henrik Edvardsson: Data curation and writing‐editing. Rosario Granados: Data curation and writing‐editing. Artur Lipiński: Data curation and writing‐editing. Ellen Christina Obermann: Data curation and writing‐editing. Maurizio Pinamonti: Data curation and writing‐editing. Henrieta Sidlova: Data curation and writing‐editing. Margareta Strojan Fležar: Data curation and writing‐editing. Folkert J. van Kemenade: Data curation and writing‐editing. Danijela Vrdoljak‐Mozetic: Data curation and writing‐editing. Ambrogio Fassina: Conceptualization, methodology, project administration, resources, writing‐original draft, and writing‐editing. Beatrix Cochand‐Priollet: Conceptualization, data curation, resources, validation, supervision, and writing‐editing.

FUNDING INFORMATION

The study was initiated, organized, financed (sample shipping), and endorsed by European Federation of Cytology Societies.

CONFLICTS OF INTEREST

The authors made no disclosures.