Interobserver Variation in Vulvar Squamous Cell Cancer Depth of Invasion Using Two Measurement Methods

Abstract

Objectives

Depth of invasion (DOI) in vulvar squamous cell carcinoma (vSCC) predicts risk of nodal metastasis, with measurement >1 mm dictating the need for lymph node diagnostic procedures. In 2021, the International Federation of Gynecology and Obstetrics (FIGO) changed its advice on how pathologists measure DOI. Some organizations revised guidelines to this “New” method; others continued to endorse the FIGO 2009 “Old” method. This study compares interobserver variation of vSCC DOI using Old and New FIGO measurement strategies.

Methods

A single representative image was chosen from 50 consecutive vSCC excisions with reported DOI of 0.1–3 mm. Ten pathologists provided 2 electronic measurements for each, using Old and New methods. Statistical evaluation included analyses of variance, Student t-test, and the kappa statistic.

Results

The Old method yielded a larger mean DOI than the New [1.3 vs 0.9 mm; p < .001]. The Old method had a lower proportion of measurement disagreements spanning 1 mm (53%, κ = 0.65% vs 68%, κ = 0.6). Agreement by all pathologists of DOI being either ≤1 mm or >1 mm occurred in 29/50 cases (58%) using the Old and 26 (52%) using the New method. When at least 2 pathologists measured DOI >1 mm, interobserver variation was lower using the Old method in 30 (83%) of 36 cases [mean difference = −0.1 mm, t(280) = −2.78, p = .008].

Conclusions

The FIGO 2021 DOI measurement method has higher interobserver variation than FIGO 2009, with this difference arising from tumors with DOI >1 mm. This finding, combined with inadequate international consensus and scant clinical outcome data, should trigger reconsideration of 2021 FIGO staging guidelines.

Vulvar malignancies account for 4% of gynecologic cancers and >90% are squamous cell carcinoma (SCC).^1,2 Vulvar SCC (vSCC) is surgically staged using the International Federation of Gynecology and Obstetrics (FIGO) TNM system. Stage 1 is confined to the vulva and divided into stage 1A, with tumor size ≤2 cm and depth of invasion (DOI) ≤1 mm, while stage 1B includes tumors >2 cm or DOI >1 mm. Management of stage 1A vSCC is local excision because the risk of lymph node metastasis with DOI ≤1 mm is <1%.^3,4 Stage 1B carries a nodal disease risk >5% and treatment involves excision with either sentinel lymph node biopsy (SLNB) or lymphadenectomy.^5–7 Reliable DOI assessment permits detection and treatment of nodal metastases in higher risk cases while sparing lower risk patients the morbidity associated with lymph node procedures.

In 2021, FIGO changed its advice on how pathologists should measure DOI.^1,8 The conventional or “Old” method described in 2009 measures from the epithelial-stromal junction of the most superficial dermal papilla adjacent to the tumor extending to the deepest point of invasion.⁸ The 2021 alternative or “New” method measures “from the basement membrane of the deepest, adjacent, dysplastic, tumor-free rete (or nearest dysplastic rete peg) to the deepest point of invasion.”¹ This revised approach postulates the tumor origin arises from the deepest adjacent dysplastic rete ridge, thus reducing DOI by eliminating the length of that rete ridge. The New method mirrors the DOI measurement approach in most cervix SCC.⁹ Current staging and treatment guidelines are based on measurement data obtained via the Old method.

The 2021 FIGO staging guideline criticized the Old method as practical but not scientific. Measuring from the top of a papilla includes a rete ridge that may not contain SCC, perhaps overestimating DOI.⁹ The most adjacent superficial papilla may be distant from the tumor or difficult to select.⁹ Previous studies document only moderate agreement in DOI measured with the Old method.^1,10,11 Two retrospective studies compared Old and New methods and did not identify untoward outcomes for those “downstaged” with a New method DOI result.^1,9,12

Regardless of measurement technique used, histopathologic assessment of vSCC is a multistep process. The pathologist first identifies cellular atypia consistent with neoplasia, then determines if invasion is present. Tangential sectioning, dense inflammation, architecturally complex precursors, and branched or anastomosing rete ridges introduce difficulty into this step. After diagnosing cancer, the pathologist selects the deepest extent of malignant cells, sometimes assessing multiple invasive fields in a single specimen. Finally, they choose the optimal adjacent epithelial location from which to measure DOI. Tumor characteristics complicating this step include invaginated, ulcerated, or exophytic tumors and difficulty in distinguishing between precursor lesions and cancer.¹³ The New method introduces an additional decision about tumor origin (Figure 1). Each step in vSCC assessment is a potential source of intra- and interobserver variation. Inconsistency between pathologists in DOI measurement clouds the data reported to national and international databases, hampering the ability of researchers to draw meaningful conclusions about the optimal DOI threshold for sentinel node biopsy. For affected patients, high interobserver variation means clinical care may be determined more by the pathologist reading the slides than by tumor characteristics, potentially resulting in under- or overtreatment.

FIGURE 1.

Schematic comparing the Old and New measurement methods when applied to smaller versus larger tumors. The deep reference point for both methods is the deepest point of invasion (green line). A, The Old technique (red arrow) measures from the most superficial adjacent dermal papilla, whereas the New method (purple arrow) measures from the presumed tumor origin at the base of the deepest dysplastic rete ridge. In small cancers, there is little opportunity for interobserver variation in assessment of tumor origin. B, When applying the New method to larger cancers, there are 2 main possibilities for selection of tumor origin—the bottom of the mass (light purple arrow) or the top of the mass (pink arrow). This may introduce greater interobserver variation when compared to the Old method (red) of selecting the most superficial adjacent papilla.

The aim of this study was to compare interobserver variation in pathologic assessment of vSCC DOI using the 2009 and 2021 FIGO measurement strategies.

METHODS

Consecutive excisions of vSCC with a reported DOI of 0.1–3 mm were identified in the pathology database of a large Australian gynecologic oncology center. The mechanism of original DOI measurement was at the reporting pathologist's discretion. Cases included human papillomavirus (HPV)-associated and HPV-independent cancers, assessed with universal p16 and p53 immunohistochemistry. These specimens included initial cancers, recurrences, and lesions concurrent with a larger tumor. Punch or forceps biopsy specimens were excluded. Fifty consecutive cases between 2018 and 2024 were identified. The slides were reviewed to confirm the diagnosis and ensure adequate quality. Faded or damaged slides were replaced with new slides cut from the original tissue block. One case was assessed as having no invasion and was maintained in study. One representative slide from each case was selected, deidentified, and electronically scanned into a confidential online e-slide manager (Aperio e-slide Manager, Leica BioSystems) that contains a measurement recording function. The goal was to provide a range of early-stage vSCC appearances to pathologists that would mirror their day-to-day practice. This study was assessed as exempt from review by the local Human Research Ethics Committee(AU202301–01).

Ten pathologists with expertise in vulvovaginal disease agreed to provide measurements for the 50 cases. Four participating pathologists worked in the United States, 3 in Australia, 2 in Italy, and 1 in New Zealand. Pathologists provided 2 measurements of DOI for each case, 1 using the Old method and 1 using the New method. If invasion was not identified, the pathologist recorded zero. There was no clinical data provided with the images, nor an explanation of how cases were selected. Pathologists were unable to access the original reported DOI nor measurements made by other participants.

The measurement data was grouped into 3 categories. “Agree ≤1 mm” refers to cases in which paired Old and New method measurements were both ≤1, or all pathologists measured DOI ≤1 mm. “Agree >1 mm” refers to cases in which paired Old and New method measurements were both >1, or all pathologists measured DOI >1 mm. “Disagree” refers to paired Old and New measurements that span 1 mm, or when 1 or more pathologists’ measurements spanned 1 mm (Figure 2). Disagreement in paired measurements could occur in 2 ways: 1) a smaller measurement with the Old method or 2) a smaller measurement with the New method. Statistical analysis included descriptive statistics, calculation of means with standard deviation (SD), mean differences, and effect sizes. Differences across measurement methods and practitioners were assessed with analysis of variance tests (ANOVA). Comparison of paired samples and frequencies occurred with the Student t-test. Interobserver variation was also evaluated with the kappa statistic (κ).

FIGURE 2.

Example of high interobserver variation with the New method (white line with red outline) but not the Old method (solid blue line). The depth of invasion (DOI) of a vulvar squamous cell carcinoma is measured by 2 different pathologists (A and B) using the 2 methods. The pathologists agree that DOI is >1 mm using the Old method. They produce measurements spanning 1 mm using the New method, with Pathologist A measuring 0.18 mm from a separated invasive nest to the base of the complex mass, whereas Pathologist B regards the entire mass as invasive yielding DOI of 1.64 mm.

RESULTS

The mean age of patients was 66 years. Specimens represented the initial vSCC diagnosis in 35 (70%); HPV-independent vSCC comprised 15 (30%) cases. The mean DOI in original pathology reports was 1.5 mm (range 0.1–3 mm).

The participating pathologists' DOI measurements had a mean of 1.1 mm (range 0–4.5). In 32 of 500 (6.4%) measurement sets, the pathologist recorded a nil DOI. These zero measurements were excluded from further analysis. A repeated measures ANOVA assessed if differences in measurement were due to individual pathologists, the method used, or an interaction between the two. This analysis demonstrated no significant difference due to differences between pathologists and no significant interaction between the pathologists and measurement method, finding instead the primary driver of differences in measurement depth was use of Old versus New method [f(1466) = 469.64, p < .001].

Using the Old method, 281 of 468 (60%) measurements were >1 mm and 187 (40%) were ≤1 mm. The New method produced 155 (33%) measurements >1 mm and 313 (67%) ≤1 mm. The paired measurements were categorized as Agree ≤1 mm in 187 (40%), Agree >1 mm in 155 (33%), and Disagree in 126 (27%). The New method produced a larger DOI than the Old method in 2 (1.6%) of 126 Disagree pairs. The remaining 124 paired measurements that spanned 1 mm were labeled “Disagree Old >1 mm.”

The Old method yielded a larger mean DOI than the New method [1.3 mm (SD = 0.82) vs 0.9 mm (SD = 0.76); mean difference 0.4 mm, effect size −0.5, p < .001] (Figure 3). This was consistent across categories of Agree ≤1 mm, Agree >1 mm, and Disagree Old >1 mm (Figure 4).

FIGURE 3.

A box plot comparison of DOI mean and standard deviation (SD) shows larger measurements with the Old method than the New method.

FIGURE 4.

A box plot comparison of DOI mean and SD shows the Old measurement is larger than the New regardless of the Agreement category of paired measurements. The largest difference between means occurs when the Old measurement is >1 mm and the New is ≤1 mm.

The Old measurement method had a lower proportion of disagreement (53%, κ = 0.65), than the New method (68%, κ = 0.6). Agreement by all pathologists of DOI being either ≤1 mm or >1 mm occurred in 29/50 cases (58%) using the Old method and 26 (52%) using the New method. Among the 21 cases with measurements spanning 1 mm using the Old method, the number of pathologists who disagreed with the majority ranged from 1 to 4. Of the 24 cases that spanned 1 mm using the New method, the number who disagreed ranged from 1 to 5 of the 10 participants (Figure 5).

FIGURE 5.

A bar graph demonstrates universal agreement (zero disagreement) across the 10 pathologists that DOI is either ≤1 mm or >1 mm in 29 cases using the Old method and 26 cases using the New method. Among cases with DOI measurements spanning 1 mm, between 1 and 4 pathologists disagreed using the Old method and between 1 and 5 pathologists disagreed using the New method.

Overall interobserver variation in mean DOI without categorization into either ≤1 mm or >1 mm found slightly greater variation in measurements using the New method compared to the Old [mean difference = −0.054 mm, effect size 0.24, t(467) = −2.66, p = .01]. Analysis with a stratified t-test demonstrated this higher variation arises from cases with larger DOI. This analysis classified cases into DOI ≤1 mm or >1 mm through identifying when 2 or more pathologists produced a measurement of either ≤1 mm (29 cases), or >1 mm (36 cases); some cases fell into both categories (Figure 6). When at least 2 pathologists measured DOI ≤1 mm, variation within the 2 methods was the same [mean difference = −0.002 mm, effect size 0.05, t(186) = −0.2, p = .84)]. When at least 2 pathologists measured DOI >1 mm, interobserver variation was lower using the Old method in 30 (83%) of 36 cases [mean difference = −0.1 mm, effect size 0.34, t(280) = −2.78, p = .008].

FIGURE 6.

A box plot comparison of DOI mean and SD, stratified by at least 2 measurements of ≤1 mm or >1 mm, shows similar interobserver variation when DOI is ≤1 mm but more variation with the New method for deeper tumors.

DISCUSSION

Consistent detection, staging, and management of vSCC requires reliable histopathologic diagnosis of cancer with meaningful and reproducible DOI measurements. This study of 50 consecutive and unselected early vSCC specimens demonstrates lower interobserver variation with the FIGO 2009 method compared to the FIGO 2021 method. The 2 measurement strategies performed similarly when DOI is ≤1 mm, but deeper tumors showed increased variation. This result signals that selection of the superficial reference point is less reproducible when defined as the origin of invasion rather than the adjacent superficial papilla.

Since 2021, the FIGO alternative method was incorporated into practice guidelines by the International Collaboration on Cancer Reporting and British Royal College of Pathology (RCPath).^14,15 Both documents acknowledge a lack of prospective clinical data supporting the change. The RCPath suggests additional levels and multidisciplinary meeting discussion in “extremely exceptional cases” of adverse prognostic factors in tumors that would be upstaged using the Old method. The College of American Pathologists reporting protocol provides for documentation of measurements using both methods, noting the American Joint Committee on Cancer “recommends using the conventional method of measurement as an ancillary data point.”¹⁶ The European Society of Gynecological Oncology concluded there is insufficient evidence to adopt FIGO 2021 as the standard approach.⁵ Their vulvar cancer guideline recommends measuring by both methods, using the FIGO 2009 result to determine need for lymph node diagnostic procedures, documenting the measurement method(s) in the pathology report, and multidisciplinary team review of difficult or discrepant cases. The Royal College of Pathologists of Australia Structured Reporting Protocol advises use of the FIGO 2009 method and recommends the New method only when origin of invasion is clearly identified.¹⁷ The lack of consensus and varied use of caveats across leading professional organizations reflects persistent concerns about the reproducibility and feasibility of the New measurement method in daily pathology practice.

The variation between pathologists in selecting a superficial reference point across diverse vSCC morphologies is well documented.^10,11,13 In this study, cases with measurements spanning 1 mm did not usually arise from a single outlier; instead, between 2 and 5 of the 10 participating pathologists disagreed about DOI being either >1 mm or ≤1 mm. This suggests a broad swathe of pathologists find it challenging to select the superficial reference point and this variation is not readily explained by limitations in knowledge or experience. A 2013 study of 11 pathologists using FIGO 2009 identified up to 7 potential locations per case for the most superficial papilla and κ = 0.62 for overall DOI.¹⁰ A 2019 study of 10 pathologists using Old and New methods to evaluate DOI in 50 cases documented κ = 0.57 for the Old method and identified as sources of discrepancy the measurement of curved or ulcerated tumors and determination of the point of invasion.¹¹ This study identified that the Old technique produced a rightward skew with a wide range of larger values, but still showed less variation in cases with DOI >1 mm compared to the New method. In a qualitative evaluation of Old and New methods applied to 26 non-ulcerated vSCC cases, 10 pathologists identified the Old method as easy in 86.5% of cases and difficult or impossible in 6.9%.¹³ In contrast, the New method was deemed easy in 53% and difficult or impossible in 21.5%. Pathologists preferred to use the Old method for multiple fields of invasion, invaginated tumors, and tumors indistinguishable from adjacent precursors. They preferred the New method in small tumors with limited invasive nests. Participating pathologists expressed consensus that evidence was insufficient to adopt FIGO 2021 as the standard measurement approach. They instead advocated for flexibility in measurement strategy, documentation of the selected method on pathology reports, and prospective research combining interobserver variation assessment with long-term clinical outcomes.¹³

Prospective clinicopathologic research on this topic has pragmatic, methodologic, and ethical challenges. The 2021 FIGO revision was based on two retrospective studies that included patients with vSCC stage 1B or greater who underwent excision and lymph node assessment and subsequently had DOI remeasured using the Old and New methods.^9,12 In the 2015 study of 148 vSCC cases, 91% had a smaller DOI with the New method. Only 14 (9.5%) women were “downstaged” to stage 1A, of whom 1 had isolated tumor cells in a lymph node.⁹ In the 2020 publication, 18 of 100 selected stage 1 vSCC cases were “downstaged” to 1A based on pathologic tumor size reassessment combined with the New method DOI.¹² None of these patients had nodal disease and DOI measurement strategy was not associated with subsequent local or nodal recurrence nor recurrence-free survival. In that cohort, mean DOI of “downstaged” cases was 2.3 mm with the Old and 0.54 mm with the New methods.¹² Together these 2 studies document clinical outcomes of 32 women with the pertinent clinicopathologic situation of Old and New DOI measurement methods that span 1 mm and all but one of them could have safely avoided a lymph node procedure if DOI was measured by the New method. However, when DOI measurement <1 mm results in nondetection of lymph node metastasis, the dire prognostic impacts make this an unacceptable trade-off for most women.¹⁸ A prospective study to evaluate DOI method variation and clinical performance would need to enroll patients with the specific scenario of DOI likely to span 1 mm depending on measurement method used and either perform universal SLNB or pursue lymph node procedures based on other factors while relying on long-term follow-up to evaluate outcomes. Registry data may permit comparison of oncologic outcomes for Old and New methods if both are reported but cannot address concerns about measurement feasibility or interobserver variation.

Participating pathologists in this study had a low rate of disagreement about the presence of invasion. In contrast, a 2013 study of 11 gynecologic pathologists' assessments of 45 vSCC cases with reported DOI ≤5 mm documented that invasion was assessed as absent in 113 (23%) of 495 observations and as indeterminate in 15 (3%).¹⁰ This higher rate of disagreement may relate to their inclusion of a larger proportion of cases with DOI <0.5 mm and “preferential selection of diagnostically challenging cases.”¹⁰ Elaboration of criteria for diagnosis of invasion would enhance vSCC reporting guidelines and may contribute to reduced variation at this step in the assessment process. Although it is unlikely that changes to pathology training and reporting guidelines will eliminate variation in diagnosis of invasion, additional guidance and illustrations about DOI measurement strategies in complex cases may support pathologists in their work.

The dermatopathology experience with extragenital SCC does not directly extrapolate to vSCC assessment. Compared to vSCC, extragenital SCC is less prone to metastasis for a given DOI, so accuracy is less critical. Diagnosis of invasion in extragenital SCC relies on separated nests, which seems to support the concept of measuring from the bottom of the nest to the base of the lesion.^19,20 However, this method does not account for a malignant cell clone that expands in all directions, producing a large pushing tumor that could be regarded as noninvasive. As this morphology is not uncommon, extragenital SCC is usually measured with the Breslow thickness from the top of the granular cell layer to the deepest invasive cell, as in melanoma. This obviates the need for selection of the most superficial papilla or identification of a presumed origin of invasion. Cervix SCC likewise does not extrapolate well to vSCC because tumors may arise from squamous cells lining glands that extend deep into stroma, distant from the epithelial surface. In this situation, the best reflection of biologic truth is a measurement from the perceived tumor origin extending to the deepest invasive nest.²¹ The lack of comparability between cervical, extragenital, and vulvar SCC speaks to a need to tailor DOI measurement mechanisms to site while minimizing the number and complexity of steps in the assessment process.

A strength of this study is the search strategy targeting vSCC cases likely to have DOI measurements spanning 1 mm depending on measurement method used. However, the search criteria of reported DOI 0.1–3 mm resulted in reduced representation of HPV-independent tumors as they are more likely to present at a later stage.²² The statistical evaluation of DOI interobserver variation is difficult to convey; a mean DOI difference of −0.054 may be interpreted as irrelevant, but a more pertinent assessment of the 2 measurement methods is the frequency of agreement between pathologists when DOI is close to 1 mm. Participation of pathologists across multiple countries and work settings ensures the approach to DOI measurement was not influenced by local practice patterns or mentorship structures. The methodology did not allow for qualitative assessment of pathologists' feelings about measurement methods, nor elaboration of their decision-making processes. The unselected nature of cases means they are unsuitable for analysis of clinical outcomes, and this study cannot comment on measurement method validity as it relates to prediction of nodal metastasis.

In conclusion, the FIGO 2009 vSCC DOI measurement method has lower interobserver variation than the FIGO 2021 method, supporting reconsideration of the 2021 FIGO staging guidelines.