Inter-observer variability in upfront dichotomous histopathological assessment of ductal carcinoma in situ of the breast: the DCISion study.

Abstract

Histopathological assessment of ductal carcinoma in situ, a non-obligate precursor of invasive breast cancer, is characterized by considerable inter-observer variability. Previously, post hoc dichotomization of multi-categorical variables was used to determine the ‘ideal’ cut-offs for dichotomous assessment. The present international multi-center study evaluated inter-observer variability among 39 pathologists who performed upfront dichotomous evaluation of 149 consecutive ductal carcinomas in situ.

All pathologists independently assessed nuclear atypia, necrosis, solid ductal carcinoma in situ architecture, calcifications, stromal architecture and lobular cancerization in one digital slide per lesion. Stromal inflammation was assessed semi-quantitatively. Tumor-infiltrating lymphocytes were quantified as percentages and dichotomously assessed with a cut-off at 50%. Krippendorff’s alpha (KA), Cohen’s kappa and intraclass correlation coefficient were calculated for the appropriate variables.

Lobular cancerization (KA = 0,396), nuclear atypia (KA = 0,422) and stromal architecture (KA = 0,450) showed the highest inter-observer variability. Stromal inflammation (KA = 0,564), dichotomously assessed tumor-infiltrating lymphocytes (KA = 0,520) and comedonecrosis (KA = 0,539) showed slightly lower inter-observer disagreement. Solid ductal carcinoma in situ architecture (KA = 0,602) and calcifications (KA = 0,676) presented with the lowest inter-observer variability. Semi-quantitative assessment of stromal inflammation resulted in a slightly higher inter-observer concordance than upfront dichotomous tumor-infiltrating lymphocytes assessment (KA = 0,564 versus KA = 0,520). High stromal inflammation corresponded best with dichotomously assessed tumor-infiltrating lymphocytes when the cut-off was set at 10% (kappa = 0,881). Nevertheless, a post hoc tumor-infiltrating lymphocytes cut-off set at 20% resulted in the highest inter-observer agreement (KA = 0,669).

Despite upfront dichotomous evaluation, the inter-observer variability remains considerable and is at most acceptable, although it varies among the different histopathological features. Future studies should investigate its impact on ductal carcinoma in situ prognostication. Forthcoming machine learning algorithms may be useful to tackle this substantial diagnostic challenge.

Ductal carcinoma in situ (DCIS), a non-obligate precursor of invasive breast cancer, is a health problem worldwide.¹ The introduction of breast screening programmes in Western countries resulted in a significant rise of its diagnosis with a current incidence of approximately 20% of all breast cancers.^2–4 Prior to those screening programmes, symptomatic ductal carcinoma in situ accounted for around 1% of all breast cancer diagnoses.³ Little is known about its natural history, and hence its appropriate management.⁵

Many screen detected ductal carcinoma in situ patients are likely over-treated, as a consequence of identifying low risk ductal carcinoma in situ cases that would not otherwise be diagnosed outside the screening setting. To address this issue, four randomized clinical trials (LORIS, LORD, COMET and LARRIKIN) are currently ongoing to investigate the potential non-inferiority of active surveillance for patients with a biopsy diagnosis of low risk ductal carcinoma in situ by comparing watchful waiting with the current surgical standard of care.⁶ The single-arm Japanese LORETTA trial will investigate the value of endocrine therapy without surgery for estrogen receptor-positive, HER2-negative low risk ductal carcinoma in situ.⁷ The definitions of ‘low risk’ ductal carcinoma in situ vary slightly among these trials, but all take into account the degree of nuclear atypia (i.e. ductal carcinoma in situ grade).^5–7 This inclusion criterion signifies a major challenge, since previous studies have shown that histopathological assessment of nuclear grade is characterized by substantial inter-observer variability, regardless of the grading system used.^8–12

This variability is not restricted to study settings, as a nationwide evaluation of ductal carcinoma in situ grading in The Netherlands revealed significant variation among different laboratories.¹³ Similar variations at the population level were reported for grading of invasive breast cancers, and the presence of borderline features for grade resulting in discordant grades was associated with decreased disease-free survival.^14–16 Although the inclusion criteria of the aforementioned non-inferiority trials encompass strict theoretical definitions of ‘low risk’ ductal carcinoma in situ,⁶ the actual implementation of these definitions will probably result in variable eligibility rates among different laboratories, should their inclusion criteria be generalized to routine practice. The LORIS trial aims to reduce this variation by performing central review of all cases diagnosed as ductal carcinoma in situ.¹⁷ While this approach applies uniform histological criteria for confirming/refusing patient eligibility, it cannot be generalized to routine practice, where biopsies are often evaluated by only one or two pathologists.

We have previously shown that pathologists are better at distinguishing grade 2 from grade 3 ductal carcinoma in situ, than distinguishing grade 1 from grade 2 ductal carcinoma in situ.¹⁸ Similar differences between different cut-offs were noted upon post hoc dichotomization of the multi-categorical assessment of comedonecrosis, stromal inflammation and stromal architecture.¹⁸ We hypothesized that ad hoc dichotomization of multi-categorical histopathological features according to the ‘ideal’ cut-off might result in acceptable degrees of inter-observer variability. The goal of the current study was therefore to perform upfront dichotomous histopathological assessment of ductal carcinoma in situ and to explore the degree of inter-observer variability. Additionally, we aimed to assess pathologists’ concordance in quantifying tumor-infiltrating lymphocytes as a percentage and determine the cut-off for dichotomization characterized by the highest inter-observer agreement.

MATERIALS AND METHODS

Patient samples

A consecutive series of ductal carcinoma in situ was selected, based on organ (breast) and lesion codes in the electronic histopathological reports (LIS DaVinci, MIPS, Ghent, Belgium). All patients underwent breast-conserving surgery or mastectomy for ductal carcinoma in situ between January 1, 2014 and August 31, 2018 at the Cliniques universitaires Saint-Luc (Brussels, Belgium). Needle biopsies and vacuum-assisted core biopsies were excluded. Ductal carcinoma in situ with associated micro-invasive foci (≤1 mm) or frankly invasive carcinoma (invasive component >1 mm) were excluded. Hematoxylin and eosin stained slides were retrieved from the archives of the Department of Pathology (Cliniques universitaires Saint-Luc) and reviewed by one pathologist (H.D.), who selected one representative slide for each lesion.

Slides containing the biopsy site were avoided, as biopsy reactions hamper the assessment of myxoid stromal periductal changes and stromal inflammation. Resection specimens with limited amounts of residual ductal carcinoma in situ (i.e. one duct with ductal carcinoma in situ) were excluded from this study. Selected slides were scanned by an automated slide scanner with Z-stack feature (NanoZoomer 2.0-RS, Hamamatsu Photonics K.K., Hamamatsu City, Japan). Digital images were available on a pass-word protected online platform (DIH, Leica Biosystems, Dublin, Ireland). This study was approved by the Ethics Committee of the Cliniques universitaires Saint-Luc (2018/21NOV/443).

Participants

All participating pathologists (pseudonyms P1-P39) had to meet the following criteria: 1) being a board-certified pathologist with a special interest in breast disease, or equivalent; 2) actively working as a pathologist, either in an academic or non-academic pathology laboratory, or both; and 3) assess at least fifty primary oncologic breast cancer resection specimens per year, according to the EUSOMA-criteria for dedicated breast pathologists.¹⁹

DCISion set-up

All participants were invited to complete the DCISion questionnaire (Supplementary Fig. 1), which was partially based on the study of van Dooijeweert et al.¹³ Eleven questions were used to assess the experience (number of years in practice), work environment (academic and/or non-academic laboratory), daily work method (conventional light microscopy and/or digital pathology), weekly amount of time dedicated to breast pathology, the system used for ductal carcinoma in situ grading, and the habit of routinely reporting specific histopathological features. No training set was used. Instead, all pathologists were provided with the study protocol, which contained written definitions for each histopathological feature (Supplementary Fig. 2), the DCISion poster with exemplary photographs based on the previous study (Supplementary Fig. 3), and relevant literature concerning the applied histopathological definitions.^{20, 21} All participants received a log-in and password which allowed access to the digital slides during four months. Scores were entered in an Excel template. Completion of the informed consent form was a prerequisite for subsequent data processing.

Definitions for dichotomous histopathological assessment

Eight histopathological features were assessed dichotomously, based on previously determined cut-offs, and were illustrated in the DCISion poster.¹⁸ Nuclear grade was assessed as non-high versus high grade, after adaptation of the American Society of Clinical Oncology/College of American Pathologists (ASCO/CAP) protocol for examination of ductal carcinoma in situ specimens.²² While the architectural types include solid, cribriform, papillary and micropapillary growth,²³ in the current study, ductal carcinoma in situ architecture was assessed as predominantly solid (≥50% solid growth) or predominantly non-solid (<50% solid growth), as previously described.^{18, 24} Both ductal carcinoma in situ architecture and nuclear grade were assessed regardless of the presence or absence of necrosis. Necrosis was classified into two categories as previously described: no or single cell necrosis versus any amount of comedonecrosis.¹⁸ Comedonecrosis was defined by areas of confluent dirty necrosis, i.e. confluent eosinophilic material, often containing ghost cells and karyorrhectic debris, and easily detected at low magnification. Intraductal calcifications within the ductal carcinoma in situ were scored as present or absent, regardless of the size and the number of ducts with calcifications.

The architecture of the periductal stroma was recorded as either sclerotic or myxoid. Sclerotic stroma resembles the regular fibrous mammary stroma and consists of regularly arranged collagen fibers. Myxoid stroma was defined as loosely arranged collagen fibers, often interspersed with an amorphous, slightly basophilic substance, as illustrated in the DCISion poster and in previous reports.^{18, 25, 26} Stromal architecture was divided into 2 categories (< 33% or ≥ 33% of ducts surrounded by myxoid stroma), and preferentially assessed at low magnification. If the ductal carcinoma in situ was located within adipose tissue without any surrounding fibrous tissue, the case was considered as predominantly sclerotic. Lobular cancerization was defined as the presence of ductal carcinoma in situ tumor cells within breast lobules, with preservation of the normal lobular architecture. Lobular cancerization was assessed as either absent or present in the digital slide, regardless of its extent.

The presence and extent of chronic inflammatory infiltrates in the periductal stroma (regardless of its architecture) was recorded in a semi-quantitative manner as previously described and was preferentially assessed at low magnification, distant from the biopsy site (if present).^{18, 24, 25, 27} Low stromal inflammation was defined as periductal stroma that is not infiltrated by lymphocytes, or that was infiltrated by few loosely arranged lymphocytes with apparent intervening stroma.²⁷ High stromal inflammation was defined as periductal stroma containing a chronic inflammatory infiltrate that consists of at least one lymphoid aggregate (i.e. any infiltrate that consists of lymphocytes abutting one-another without intervening collagenous stroma).²⁷ Lymphoid follicle formation could be present but was not a prerequisite. Assessment of the stromal architecture could be hampered by the density of the inflammatory infiltrate.

Assessment of tumor-infiltrating lymphocytes

Tumor-infiltrating lymphocytes were assessed according to the standardized method proposed by the International Immuno-oncology Biomarkers Working Group.^{20, 21, 28} The ‘supplementary Figure 1’ of Pruneri et al. was used as a visual aid during assessment.²⁹ Tumor-infiltrating lymphocytes percentages signified the percentage of lymphocytes related to the total periductal stromal surface area, which served as a denominator.^{20, 28} The percentage of tumor-infiltrating lymphocytes was assessed without considering the score for ‘stromal inflammation’. Participants were asked to provide an average percentage for tumor-infiltrating lymphocytes surrounding all ducts affected by ductal carcinoma in situ in that particular slide.²⁸ Hotspots were not taken into account. Participants were also asked to assess tumor-infiltrating lymphocytes dichotomously, using an upfront cut-off (<50% versus ≥50%), as previously described by Pruneri et al.²⁹

Statistical analysis

Statistical analyses were performed with IBM SPSS statistics 25.0 (IBM Chicago, IL, USA) as previously reported.¹⁸ Pie charts were constructed in Excel (Excel Windows 10, Microsoft Corporation, Redmond, WA, VS). The arithmetic mean for each histopathological characteristic was calculated per lesion, to evaluate the distribution of each characteristic within the ductal carcinoma in situ cohort. The mean corresponds to the most commonly addressed category for a specific characteristic per lesion. Percentages of absolute agreement were determined, signifying the number of lesions with 100% concordance. Krippendorff’s alpha (KA) reliability estimates were calculated using the ‘Kalpha’ macro provided by Hayes and Krippendorff (http://afhayes.com/spss-sas-and-mplus-macros-and-code.html). This macro was introduced in SPSS to compute Krippendorff’s alpha for all dichotomous data to investigate overall inter-observer variability per characteristic.^{30, 31} The number of bootstrap samples was set at 10.000.

The intraclass correlation coefficient was calculated for tumor-infiltrating lymphocytes assessed as a percentage and was interpreted according to Koo and Li.³² Intraclass correlation coefficient settings were: two-way mixed, single measures, absolute agreement. Cohen’s kappa (Ƙ) values were calculated for all dichotomous variables for each observer duo (i.e. 741 kappa values for each dichotomous histopathological characteristic). The kappa’s distribution was visualized by box-and-whisker plots. Interpretation of the kappa values was performed according to Landis and Koch.³³ Pearson’s and Spearman’s correlation tests were performed when appropriate, to investigate correlations between the degree of inter-observer variability and any possible confounder mentioned in the DCISion questionnaire. Multiple linear regression analysis was performed to correct for multiple potential confounders. All tests were two-sided. The statistical significance level was set at 0,05.

RESULTS

Characteristics of the DCISion participants

In total, 47 pathologists were invited to participate in this study. Thirty-nine pathologists (83%) from nine different countries representing three continents and thirty different laboratories responded. The DCISion questionnaire was completed by 38 pathologists (97%); one participant had skipped four questions regarding habits of reporting (Table 1). The participants had been practicing as certified pathologists for 13,7 years on average (range 1–30 years, excluding the years of training). Twenty-five pathologists (64%) work in an academic laboratory, twelve pathologists (31%) work in a non-academic laboratory and two pathologists (5%) work in both an academic and non-academic laboratory. The average weekly time dedicated to breast pathology amounts to less than one day for seven participants (18%), one to two days for ten participants (26%), and two to three days for four participants (10%). Eight participants (20%) spend between three and four days on breast pathology, and ten participants spend more than four days on breast pathology (26%). Three participants (8%) use both conventional and digital microscopy in daily practice; thirty-six participants (92%) only use conventional light microscopy on a daily basis.

Table 1.

Distribution of the answers of the 39 participating pathologists regarding potential confounders that might influence the degree of inter-observer variability.

Multiple choice question	n (%)
In what kind of laboratory do you work?
Academic	25 (64)
Non-academic	12 (31)
Both academic and non-academic	2(5)
What kind of microscope do you use for daily practice?
Conventional light microscope	36 (92)
Digital and conventional light microscopy	3 (8)
How much of your time is dedicated to breast pathology?
< 20% (< 1 day per week)	7 (18)
≥ 20 and < 40% (between 1 and 2 days per week)	10 (26)
≥ 40 and < 60% (between 2 and 3 days per week)	4 (10)
≥ 60 and < 80% (between 3 and 4 days per week)	8 (20)
≥ 80% (> 4 days)	10 (26)
Which grade do you mention in case of heterogeneity?
Highest grade	35 (90)
Predominant grade	2 (5)
Both highest and predominant grade	2 (5)
Which grading system do you use for ductal carcinoma in situ grading?
ASCO/CAP	13 (33)
Holland classification	0 (0)
Lagios classification	0 (0)
Pinder classification	2 (5)
Van Nuys classification	4 (10)
WHO classification	16 (41)
Other	0 (0)
Van Nuys + WHO classification	2 (5)
Van Nuys + ASCO/CAP classification	1 (3)
No answer	1 (3)
Do you routinely report tumor-infiltrating lymphocytes?
Always mentioned in report	2 (5)
Sometimes mentioned in report	18 (46)
Never mentioned in report	18 (46)
No answer	1 (3)
Do you routinely report lobular cancerization?
Always mentioned in report	10 (26)
Sometimes mentioned in report	19 (49)
Never mentioned in report	9 (23)
No answer	1 (3)
Do you routinely report calcifications?
Always mentioned in report	36 (92)
Sometimes mentioned in report	2 (6)
Never mentioned in report	0 (0)
No answer	1 (3)
Do you routinely report comedonecrosis?
Always mentioned in report	35 (90)
Sometimes mentioned in report	4 (10)
Never mentioned in report	0 (0)
Do you routinely report ductal carcinoma in situ architecture?
Always mentioned in report	29 (74)
Sometimes mentioned in report	8 (21)
Never mentioned in report	2 (5)

ASCO/CAP: American Society of Clinical Oncology/College of American Pathologists; WHO: world health organisation

Case selection

Since a previous report has shown that histopathological diagnosis based on a single slide is characterized by substantial inter-observer variability regarding the distinction between benign breast disease, ductal carcinoma in situ and (micro-)invasive breast cancer,³⁴ a consecutive series of 149 slides encoded as pure ductal carcinoma in situ was provided to all participants. Upon completion of the digital histopathological assessment, all the participants’ comments regarding the eligibility of the ductal carcinoma in situ lesions were gathered and reviewed together with the digital images and the original glass slides and available archived immunohistochemical stained slides (performed by HD, CG and MRVB). In total, twenty-one lesions (14%) were removed from the series because of: 1) insufficient amount of ductal carcinoma in situ (i.e. one single duct) in one case; 2) insufficient size (<2mm), favoring a diagnosis of atypical ductal hyperplasia, in six cases; 3) presence of micro-invasive carcinoma in six cases; 4) presence of a 2 mm focus of invasive lobular carcinoma in one case; 5) lesions falling short of being designated ductal carcinoma in situ, favoring a diagnosis of usual ductal hyperplasia, flat epithelial atypia or apocrine atypia in seven cases. Eventually, 128 unequivocal pure ductal carcinoma in situ (86%) remained for the final analyses. This method aimed to prevent the introduction of a selection bias, which was likely to occur upon removal of specific cases judged by a single pathologist.³⁴

Absolute agreement

The mean histopathological scores were calculated and designated Px (Figure 1A). Absolute agreement reflects the number of cases that were rated identically by all pathologists (100% agreement) or all but one pathologist (97% agreement; Table 2). Absolute agreement was lowest for nuclear atypia and lobular cancerization (6% and 7%, respectively). Dichotomously assessed tumor-infiltrating lymphocytes showed the highest absolute agreement (59%). Of note, the high concordance was due to the rarity of ductal carcinoma in situ cases with ≥50% tumor-infiltrating lymphocytes (i.e. 12% of the cohort on average), resulting in high agreement on the cases with <50% tumor-infiltrating lymphocytes and no agreement at all on the cases with ≥50% tumor-infiltrating lymphocytes.

Figure 1.

(A) Pie charts illustrating the histopathological assessment of the ‘average pathologist’ (Px) as a percentage, based on the arithmetic mean of all pathologists’ scores per dichotomously assessed histopathological feature. KA signifies the Krippendorff’s alpha statistic per feature. (B) Box-and-whisker plots illustrating the differences in the distribution in Cohen’s kappa values per pathologist duo per histopathological feature. Circles represent outliers. DCIS: ductal carcinoma in situ. TILs: tumor-infiltrating lymphocytes.

Table 2.

Absolute agreement among pathologists regarding the evaluation of eight different histopathological features in ductal carcinoma in situ of the breast.

		All 39 pathologist agreed			At least 38 of 39 pathologists agreed
Histopathological feature		Cases n (%)	Absolute agreement n (%)		Cases n (%)	Absolute agreement n (%)
Nuclear atypia	Non-high	5 (3,9)	8 (6)		10 (7,8)	20 (16)
	High	3 (2,3)			10 (7,8)
Ductal carcinoma in situ architecture	Non-solid	29 (22,7)	35 (27)		44 (34,4)	56 (44)
	Solid	6 (4,7)			12 (9,4)
Comedonecrosis	Absent	13 (10,2)	21 (16)		23 (18,0)	40 (31)
	Present	8 (6,3)			17 (13,3)
Calcification	Absent	7 (5,5)	35 (27)		14 (10,9)	53 (41)
	Present	28 (21,9)			39 (30,5)
Stromal architecture	Sclerotic	14 (10,9)	17 (13)		23 (18,0)	30 (23)
	Myxoid	3 (2,3)			7 (5,5)
Stromal inflammation	Low	28 (21,9)	31 (24)		37 (28,9)	44 (34)
	High	3 (2,3)			7 (5,5)
Tumor-infiltrating lymphocytes dichotomous	<50%	75 (58,6)	75 (59)		76 (59,4)	90 (70)
	≥50%	0 (0)			14 (10,9)
Lobular cancerization	Absent	6 (4,7)	9 (7)		16 (12,5)	21 (16)
	Present	3 (2,3)			5 (3,9)

Overall inter-observer variability

Krippendorff’s alpha was lowest for lobular cancerization (Krippendorff’s alpha = 0,396), and slightly higher for stromal architecture (Krippendorff’s alpha = 0,450) and nuclear atypia (Krippendorff’s alpha = 0,422). The Krippendorff’s alphas were fairly high for stromal inflammation (Krippendorff’s alpha = 0,564), dichotomously assessed tumor-infiltrating lymphocytes (Krippendorff’s alpha = 0,520), and comedonecrosis (Krippendorff’s alpha = 0,539). The highest Krippendorff’s alphas were observed for solid ductal carcinoma in situ architecture (Krippendorff’s alpha = 0,602) and presence of intraductal calcifications (Krippendorff’s alpha = 0,676).

Comparison of pathologists duos

As Krippendorff’s alpha does not permit detailed evaluation of inter-observer variability among different pathologists, Cohen’s kappa values were determined for each pathologist duo (Supplementary Tables 1–8). These kappa values confirmed that evaluation of calcifications and lobular cancerization was characterized by the highest and lowest concordance, respectively (Table 3). The narrowest interquartile range was observed for evaluation of calcifications, whereas stromal inflammation showed the highest dispersion of kappa values (Figure 1B).

Table 3.

Descriptive statistics for the Cohen’s kappa values among 39 pathologists per histopathological feature of ductal carcinoma in situ of the breast.

Histopathological feature	Mean	SD	Minimum	Maximum	IQR	P5	P10	P25	P50	P75	P90	P95
Nuclear grade	0,430	0,124	0,091	0,790	0,171	0,207	0,272	0,344	0,437	0,515	0,586	0,627
Ductal carcinoma in situ architecture	0,600	0,012	0,272	0,873	0,156	0,391	0,444	0,524	0,617	0,680	0,727	0,766
Comedonecrosis	0,544	0,120	0,078	0,836	0,145	0,310	0,387	0,481	0,562	0,626	0,677	0,703
Calcification	0,675	0,094	0,345	0,969	0,110	0,510	0,554	0,624	0,682	0,734	0,791	0,817
Stromal architecture	0,461	0,122	0,142	0,824	0,166	0,256	0,309	0,377	0,458	0,543	0,622	0,673
Stromal inflammation	0,567	0,168	0,085	0,903	0,240	0,249	0,322	0,458	0,600	0,698	0,766	0,795
Lobular cancerization	0,403	0,130	0,059	0,765	0,192	0,172	0,218	0,314	0,414	0,506	0,564	0,598
Combined risk score	0,473	0,144	−0,079	0,881	0,204	0,235	0,280	0,373	0,480	0,577	0,653	0,692
Tumor-infiltrating lymphocytes dichotomous (10% cut-off)	0,553	0,186	0,036	0,874	0,261	0,188	0,261	0,438	0,603	0,699	0,750	0,776
Tumor-infiltrating lymphocytes dichotomous (20% cut-off)	0,668	0,121	0,020	0,948	0,159	0,434	0,505	0,594	0,689	0,753	0,809	0,836
Tumor-infiltrating lymphocytes dichotomous (30% cut-off)	0,637	0,124	0,255	0,924	0,171	0,405	0,465	0,555	0,656	0,726	0,781	0,811
Tumor-infiltrating lymphocytes dichotomous (40% cut-off)	0,595	0,132	0,178	0,871	0,184	0,351	0,401	0,509	0,610	0,693	0,752	0,788
Tumor-infiltrating lymphocytes dichotomous (50% cut-off)	0,556	0,148	0,080	0,920	0,182	0,246	0,353	0,477	0,573	0,659	0,725	0,761

IQR: interquartile range; P: percentile; P50: fiftieth percentile = median; SD: standard deviation.

Comparison of different cut-offs for tumor-infiltrating lymphocytes

The intraclass correlation coefficient was calculated for tumor-infiltrating lymphocytes assessed as a percentage, showing good overall agreement with an average of 0,821 (range 0,566 – 0,933; Supplementary Table 10). Since intraclass correlation coefficient and Krippendorff’s alpha are different statistical measures, their values cannot be mutually compared. Therefore, all participants were asked to rate stromal tumor-infiltrating lymphocytes ad hoc as a dichotomous variable by using a cut-off at 50%.²⁹ The tumor-infiltrating lymphocytes percentages were also dichotomized post hoc according to different cut-offs with 10% increments (10%, 20%, 30% and 40%). The number of cases with average low and high tumor-infiltrating lymphocytes according to each cut-off was compared with the number of ductal carcinoma in situ presenting with low and high ‘stromal inflammation’ (Figure 2), resulting in the following kappa values: 0,881 (10% cut-off), 0,837 (20% cut-off), 0,637 (30% cut-off), 0,437 (40% cut-off) and 0,245 (50% cut-off).

Figure 2.

Bar charts (A-E) and ROC-curve (F) illustrating the interrelationship between semi-quantitatively assessed stromal inflammation versus dichotomized tumor-infiltrating lymphocytes (TILs) assessed as a percentage in ductal carcinoma in situ (DCIS). The agreement between both parameters depends upon the cut-off used: Cohen’s kappa values (Ƙ) are mentioned for a threshold of 10% (A), 20% (B), 30% (C), 40% (D) or 50% (E). A ROC-curve (F) illustrates this interrelationship for each pathologist separately.

Similarly to the other dichotomously assessed histopathological features, Krippendorff’s alphas were calculated for tumor-infiltrating lymphocytes according to the different cut-offs: the Krippendorff’s alpha amounted 0,512 for the 10% cut-off, 0,669 for the 20% cut-off, 0,641 for the 30% cut-off, 0,604 for the 40% cut-off and 0,520 for the 50% cut-off. The kappa values were calculated for each cut-off per pathologist duo, which confirmed that the highest level of concordance was observed for the 20% cut-off (Figure 3).

Figure 3.

Box-and- whisker plots illustrating the differences in the distribution in Cohen’s kappa values per pathologist duo per cut-off for dichotomized tumor-infiltrating lymphocytes (TILs). The 20% and 50% cut-off for tumor-infiltrating lymphocytes assessed as a percentage showed the lowest and highest inter-observer variability, respectively. Circles represent outliers; asterisks represent extremes.

Concordance for a combined risk score

Previously, a combined risk score for ductal carcinoma in situ, based on dichotomously assessed nuclear atypia, stromal architecture and stromal inflammation, was shown to be associated with recurrence risk after breast-conserving surgery. We calculated the combined risk score for each participant, based on the available individual histopathological characteristics: ductal carcinoma in situ with a combination of high nuclear grade, predominantly myxoid stromal architecture and high stromal inflammation were considered to have a high combined risk score. When any of these features was lacking in a ductal carcinoma in situ lesion, it was considered to have a low combined risk score. The overall agreement for this post hoc determined dichotomous variable was rather low (Krippendorff’s alpha = 0,482). The kappa values were calculated for each cut-off per pathologist duo (Supplementary Table 9). Descriptive values for the kappa value of the combined risk score assessment are shown in Table 3.

Confounders

The influence of potential confounders on the (dis)agreement among pathologists was investigated, including: experience, time dedicated to breast pathology, work environment, classification system used for ductal carcinoma in situ grading (for nuclear atypia only), and habit of reporting the characteristic of interest. No significant associations were observed (p>0,05), except for two. Inter-observer variability for lobular cancerization was significantly lower for pathologists claiming to always mention this feature, and highest for pathologists stating never to report this feature (p=0,014). This observation was independent of the laboratory environment, experience and time dedicated to breast pathology. The degree of concordance for stromal inflammation was significantly higher for pathologists with more time dedicated to breast pathology (p=0,031), independent of the laboratory environment and experience. This association was not observed for tumor-infiltrating lymphocytes assessment. The influence of a ‘center effect’ (i.e. pathologists working together in one center might show higher concordance) was not investigated as the number of colleagues was too low to allow sufficiently powered statistical analysis.

DISCUSSION

Grading systems for ductal carcinoma in situ classify these lesions in three categories, analogous to the Nottingham grading system for invasive breast cancer.^35–38 Previous studies showed that ductal carcinoma in situ grading is characterized by substantial inter-observer variability, regardless of the grading system used.^{8–12, 39} The current ad hoc dichotomization as non-high versus high nuclear grade resulted in moderate agreement with an average kappa of 0,430, which is lower than the kappa values previously reported for post hoc dichotomization of ductal carcinoma in situ grade (0,55 by Rakha et al. and 0,53 by Van Bockstal et al.).^{18, 40} Nuclear atypia represents a biological spectrum, ranging from monotonous, slightly atypical nuclei to extreme pleomorphism. The upper ends of this spectrum are easily assessable, but a large grey zone exists in between. Heterogeneous morphology throughout a single ductal carcinoma in situ lesion can further hamper the adequacy and reproducibility of morphological assessment. Molecular and genomic studies provided evidence for a two-tier (low grade versus high grade) pathway in breast cancer development, wherein morphologic grade 2 ductal carcinoma in situ either cluster together with morphologic grade 1 or grade 3.^41–43 Two-tier classification systems for morphological evaluation of dysplasia are currently common practice in other organ systems, such as gastrointestinal epithelial neoplasia, cervical and vulvar squamous intraepithelial lesions.^{44, 45} These classifications improved interrater concordance.^{44, 45} Logically, the discordance among observers is generally larger when the number of available categories increases. We therefore hypothesized that two-tier assessment might improve the current inter-observer variability in ductal carcinoma in situ grading.

We previously examined the inter-observer variability among 13 pathologists for multi-categorical histopathological features and reported the ‘ideal’ cut-offs for two-tier assessment after post hoc dichotomization.¹⁸ The prognostic value of this two-tier assessment was subsequently investigated in a cohort of 211 ductal carcinoma in situ patients, wherein high grade nuclear atypia, high stromal inflammation and myxoid stromal architecture were associated with increased overall recurrence risk after breast-conserving surgery.²⁷

The DCISion study applied these binary cut-offs upfront, which resulted in overall moderate agreement for all individual histopathological features, as well as for the post hoc combined risk score assessment (i.e. average kappa values between 0,41–0,60). Despite dichotomization, the inter-observer variability remains considerable for all features except intraductal calcifications. Of note, the current study setting lacked some essential features of the ‘real-life setting’ of histopathological diagnosis, e.g. immunohistochemistry, deeper levels and multiple tissue blocks. In addition, only three pathologists reported that they used digital pathology on a regular basis. These factors might have negatively influenced the degree of inter-observer variability. Nevertheless, the observed discordance might explain the variable prognostic power of different histopathological features in previous reports. For example, some large retrospective study cohorts and randomized trials could not confirm the association between high nuclear grade and increased recurrence risk.^46–49 We were unable to investigate the impact of inter-observer variability on the recurrence risk stratification in this cohort, as most patients were recently diagnosed and adequate follow-up data were not available. However, it is likely that inter-observer variability has an effect on the association with recurrence risk, i.e. some pathologists’ scores might be associated with recurrence after breast-conserving surgery, and others might not. We aim to investigate this in the future, when the median follow-up time of this cohort reaches at least 5 years. Such a large-scale study would enable the definition of an allowed margin of error that is associated with a limited degree of discordance among pathologists, without significantly affecting recurrence risk stratification.

Robust prognostic histopathological markers require reproducibility of their assessment. Although features assessed as a continuous measure (such as stromal tumor-infiltrating lymphocytes) show overall acceptable concordance rates, the clinical decision making is often dichotomous (e.g. to treat or not to treat) and therefore generally requires a particular cut-off. For example, positive or negative hormone receptor status and HER2 status in invasive breast cancer will determine eligibility for (neo)adjuvant hormonal treatment and HER2-targeted therapy. Ductal carcinoma in situ patients who undergo breast-conserving surgery are often treated with adjuvant radiotherapy, and in some countries also with selective estrogen receptor-modulators or aromatase inhibitors. The choice for adjuvant treatment is mainly led by the recurrence risk, as up to 30% of non-irradiated ductal carcinoma in situ patients will develop a loco-regional recurrence after .⁵⁰ To date, margin size and histopathological features remain the cornerstone for recurrence risk stratification, although molecular tests such as Oncotype DX® DCIS (Genomic Health, Redwood City, CA, USA) are emerging.⁵¹ It is therefore important that pathologists speak the same language, to ensure that patients are treated in a consistent manner throughout different hospitals.

The DCISion study illustrates that the current histopathological evaluation, though acceptable, should improve. It is challenging to identify the precise causes of the observed disagreement. As mentioned above, discordance might have partly been induced by the use of digital histological images, since the majority of the participants commonly uses glass slides in daily practice. In general, diagnostic concordance between glass slides and digital slides is reported as high, but these studies usually investigate intra-observer equivalency.^{52, 53} However, inter-observer variability using glass slides will probably result in inter-observer variability using digital slides. Thus, the use of digital slides cannot entirely explain the observed discordance in the DCISion study. Intra-tumor heterogeneity, differences in selected regions of interest and different interpretations of the provided definitions may also account for increased discordance rates. The current study was limited to the evaluation of morphological features in a single H&E slide per lesion. Multiple H&E slides, additional levels and/or immunohistochemical markers might increase the degree of concordance in ductal carcinoma in situ grading. Future studies should explore whether there is a role for hormone receptor status and HER2 status in ductal carcinoma in situ risk stratification, as well as for immunohistochemical characterization of the stromal immune response.

Reproducibility and robustness of assessment are of particular importance for inclusion in the ongoing randomized trials that investigate non-inferiority of active surveillance. In these trials, risk assessment and eligibility are based on histopathological morphological features, although the LARRIKIN, COMET and LORETTA trials also take into account hormone receptor and HER2 status.^{6, 7} It will take several years before these trials’ findings are translated to routine clinical practice, but it is likely that watchful waiting will become a valid ‘treatment’ option. Deep learning algorithms or so-called ‘artificial intelligence’ might offer a solution to cope with the inter-observer variability in histopathological assessment, but we should be careful not to introduce inter-observer variability into these deep learning networks. If all the available deep learning algorithms are initially taught by only one or two pathologists, we might end up with similar inter-observer variability in artificial intelligence. It would be of interest to use the results of a multi-headed panel evaluation to enable the creation of a robust deep learning algorithm. The current DCISion study cohort might therefore be of interest, as it has been assessed by 39 different observers, representing thirty different laboratories. We aim to train an algorithm based on the ductal carcinoma in situ cases with 100% absolute agreement for a particular feature, and to explore its prognostic power in an independent ductal carcinoma in situ patient cohort with available follow-up data.

One limitation of the current study is the lack of a gold standard, since nobody knows who is ‘right’ or ‘wrong’ regarding the ductal carcinoma in situ cases with total lack of absolute agreement for a specific characteristic, such as a ductal carcinoma in situ lesion that was regarded as non-high grade by nineteen participants and as high grade by twenty other participants (Supplementary Fig. 4). A deep learning algorithm trained by using cases with 100% absolute agreement for a particular characteristic, might act as an artificial reference. This study also highlights that poor concordance (i.e. low kappa values) between two observers does not give any information on who is ‘right’ and who is ‘wrong’. It only indicates that this particular feature has been perceived differently by two observers, and this poor concordance might be due to unclear definitions of the investigated characteristics. For instance, Harrison et al. recently showed that the threshold for comedonecrosis is highly variable, even among experienced breast pathologists.⁵⁴ A discussion and developing national / international guidance on how to define or re-define particular histopathological features, including their applied cut-offs, should be highly ranked when setting a research agenda for ductal carcinoma in situ.

This is of particular interest for stromal inflammation or tumor-infiltrating lymphocytes. The International Immuno-oncology Biomarkers Working Group proposed to quantify tumor-infiltrating lymphocytes in as much detail as possible, i.e. as a continuous variable by using percentages.²⁸ This method allows in-depth analysis of potentially clinically relevant cut-offs. In the DCISion study, we asked all participants to quantify tumor-infiltrating lymphocytes both dichotomously and as a percentage by using previously published photographs as a visual aid (such as supplementary Figure S1 of Pruneri et al.).^{20, 21, 29} By using 10% increments, we identified the 20% tumor-infiltrating lymphocytes cut-off as the cut-off associated with the highest inter-observer agreement. However, the 10% tumor-infiltrating lymphocytes cut-off corresponded best with semi-quantitative assessment of stromal inflammation, implying that most ductal carcinoma in situ with ≥10% stromal tumor-infiltrating lymphocytes present at least one dense aggregate of lymphocytes (designated ‘high stromal inflammation’). Presence or absence of dense lymphoid aggregates (with or without tertiary follicle formation) might serve as a visual aid for pathologists to classify a particular ductal carcinoma in situ lesion as having low or high stromal tumor-infiltrating lymphocytes. Since inter-observer agreement seems to be the highest for this 20% tumor-infiltrating lymphocytes cut-off, future work should also focus on investigating its prognostic value for ductal carcinoma in situ recurrence risk stratification after breast-conserving surgery. Overall agreement on stromal inflammation and stromal tumor-infiltrating lymphocytes was moderate in the DCISion study, but we may improve the current degree of concordance by providing clear definitions and adequate visual aids to enable training of pathologists. The website of the International Immuno-Oncology Biomarker Working Group (www.tilsinbreastcancer.org) provides such a useful tumor-infiltrating lymphocytes training tool for pathologists. The added value of immunohistochemical characterization of the immune infiltrate should also be explored, as well as the role of the location of stromal tumor-infiltrating lymphocytes. Toss et al. have recently demonstrated that dense touching tumor-infiltrating lymphocytes (i.e. tumor-infiltrating lymphocytes touching the basement membrane or away from it with maximum one lymphocyte of thickness) are associated with decreased recurrence-free survival.⁵⁵ This promising observation warrants validation in an independent patient cohort, as it indicates that spatial arrangement of tumor-infiltrating lymphocytes might be more important than the quantity of tumor-infiltrating lymphocytes. Moreover, the assessment of touching tumor-infiltrating lymphocytes seems reproducible, because it showed higher inter-observer concordance than assessment of tumor-infiltrating lymphocytes as a percentage.⁵⁵

CONCLUSIONS

Despite upfront dichotomous evaluation, the inter-observer variability for histopathological assessment of ductal carcinoma in situ remains considerable and is at most acceptable, although it varies between the evaluated features. This large-scale international multicenter study allowed us to compare two different methods to assess the inflammatory response in the periductal stroma. This comparison suggests that a semi-quantitative method (absence or presence of lymphoid aggregates) corresponds best with tumor-infiltrating lymphocytes assessed as a percentage when a cut-off at 10% is used for the latter. Nevertheless, a post hoc applied cut-off at 20% for stromal tumor-infiltrating lymphocytes results in the highest inter-observer concordance. Future research should validate the upfront use of this 20% cut-off, and investigate its relation with post-operative outcomes. Furthermore, the impact of the current degree of inter-observer variability on ductal carcinoma in situ prognostication should be explored. Forthcoming machine learning algorithms and additional immunohistochemistry might be useful to tackle these substantial diagnostic challenges.

ABBREVIATIONS

ASCO/CAP

American Society of Clinical Oncology/College of American Pathologists

COMET

comparison of operative to monitoring and endocrine therapy trial

DCIS

ductal carcinoma in situ

Fig

figure

LARRIKIN

Low And InteR-mediate RIsK ductal carcinoma IN situ study

LORD

low-risk ductal carcinoma in situ trial

LORIS

low-risk ductal carcinoma in situ trial