Multi-institutional Assessment of Pathologist scoring HER2 Immunohistochemistry

Abstract

The HercepTest was approved 20+ years ago as the companion diagnostic test for trastuzumab in HER2 amplified/overexpressing breast cancers. Subsequent HER2 immunohistochemistry (IHC) assays followed, including the now most common Ventana 4B5 assay. While this IHC assay has become the clinical standard, its reliability, reproducibility, and accuracy have largely been approved and accepted based on concordance between small numbers of pathologists without validation in a real-world setting. In this study, we evaluate the concordance and inter-rater reliability of scoring HER2 IHC in 170 breast cancer biopsies by 18 breast cancer-specialized pathologists from 15 institutions. We used the ONEST (Observers Needed to Evaluate Subjective Tests) method to determine the plateau of concordance and the minimum number of pathologists needed to estimate inter-rater agreement values for large numbers of raters, as seen in the real-world setting. We report substantial discordance within the intermediate categories (<1% agreement for 1+ and 3.6% agreement for 2+) in the four-category HER2 IHC scoring system. The discordance within the IHC 0 cases is also substantial with an overall percent agreement (OPA) of only 25% and poor inter-rater reliability metrics (0.49 Fleiss’ kappa, 0.55 intraclass correlation coefficient). This discordance can be partially reduced by using a three-category system (28.8% vs. 46.5% OPA for four and three-category scoring systems respectively). ONEST plots suggest that the OPA for the task of determining a HER2 IHC score 0 from not 0 plateaus statistically around 59.4% at 10 raters. Conversely, at the task of scoring HER2 IHC as 3+ or not 3+ pathologists’ concordance was much higher with an OPA that plateaus at 87.1% with 6 raters. This suggests that legacy HER2 IHC remains valuable for finding HER2 gene amplified patients, but unacceptably discordant in assigning HER2-low or negative status for emerging HER2-low therapies.

Introduction

Accurate quantification of human epidermal growth factor 2 (HER2) expression levels is critical in the management of breast cancer patients. HER2 expression in breast cancer spans a large dynamic range of 3 logs so immunohistochemistry (IHC) cannot adequately assess HER2 concentrations throughout this dynamic range^1–3. Since only patients with amplified, over expressed HER2 benefitted from the initial HER2 axis drugs (e.g. trastuzumab)^4–9, subsequent commercial assays were designed to detect high HER2 expression. The current most common companion diagnostic test is the Ventana 4B5 assay and its dynamic range is best in tumors that have over 100,000 molecules of HER2 protein per cell. Even so, the current American Society of Clinical Oncology/College of American Pathologists Clinical Practice (ASCO/CAP) guidelines require reflex gene amplification testing by fluorescence in situ hybridization (FISH) of all IHC 2+ cases so that HER2 amplified tumors are not missed^10,11. More recently the landscape has changed as there are now new anti-HER2 drugs, for example, trastuzumab deruxtecan (T-DXd), that are effective in this low HER2 expressing subgroup^12–17. These recent clinical trials for T-DXd have attempted to define this low HER2 expressing subgroup as 1+ or 2+ cases without gene amplification using the 2018/current ASCO/CAP guidelines for the legacy HER2 assays¹¹, which were originally designed for detecting amplified HER2 expression. This raises new questions about the conventional FDA approved HER2 assays and their performance for both the historical drugs for which the assay was approved and for the new drugs for which it may be used.

The FDA granted approval for the conventional HER2 IHC assays based on the ability to detect positive or negative cases compared to HER2 gene amplification or the agreement with the original Dako HercepTest (only using a 0/1+, 2+, and 3+ scoring system). Additionally, all of these historical HER2 assays were approved with a relatively low inter-rater agreement requirements as can be seen in the FDA’s published SSEDs^18–20. Specifically, these assays were only required to be evaluated by 2 to 3 pathologists for FDA approval. The decision of whether a case was a score of 0 vs. 1+ (or “low” expressing) was not required to be accurate, reproducible, or concordant based on the FDA summary of safety and effectiveness datasheets (SSEDs) for these assays from over 20 years ago. The distinction between 0 vs. 1+ cases was simply not a meaningful category for FDA approval of these HER2 IHC assays, whereas the ASCO/CAP guidelines for pathologists and assay package insert information have featured the 0 and 1+ categories since the original Dako HercepTest. The general inattention toward reproducible scoring of the 0/1+ categories and subsequent “lumping” of these cases into a negative class presumably did not have significant clinical ramifications as the assays were “fit-for-purpose” to detect amplified cases²¹. However, now it is clinically relevant to distinguish “true negative” from HER2-low cases for these emerging therapies (namely antibody-drug conjugates including T-DXd), and the question is whether the legacy HER2 assays should be used for this task.

Early studies on the performance of these HER2 IHC assays focused on 3+ and 2+ scores, as these cut-points indicated trastuzumab therapy or reflex FISH testing based on the evolving FDA and ASCO/CAP guidelines (note, FISH as evolved to also included chromogenic in situ hybridization (CISH) and thus in the remainder of this work, we simply use ISH). Many independent studies demonstrated that HER2 IHC is a reasonable first test for HER2 overexpression with high negative predictive value as well as an acceptable positive predictive value when paired with reflex ISH testing for IHC 2+ cases^22–26. However, inter-observer concordance for HER2 IHC scoring is mixed with some studies reporting satisfactory agreement for positive (2+/3+) and negative (0/1+) cases^27–30, whereas others demonstrated significant discordance particularly on 2+ and negative (0/1+) IHC scoring^23,31–34. Even though HER2 ISH is used as the gold-standard reference assay for gene amplification in these studies, testing HER2 ISH for all breast cancer patients did not replace HER2 IHC testing likely due to the complexity and cost³⁵. Past studies have reported performing HER2 ISH testing for all patients including IHC 0 & 1+. While HER2 IHC 0 & 1+ patients with ISH amplification can benefit from anti-HER2 therapies^36,37, the prevalence of HER2 gene amplification with 0/1+ HER2 IHC expression is low (1.5% to 5% of 0/1+ are ISH positive compared to 20% to 30% of 2+ cases)^22,35.

More recent studies report substantial inter-rater discordance and poor reproducibility amongst low vs.” true negative” (IHC 0) cases when determining HER2 IHC status in breast cancer^32,34,38,39. Lambein et al. reported disagreement rates as high as 85% for HER2 0 IHC scores using the Ventana 4B5 assay between their local laboratory and central assessment³⁸. In a retrospective study investigating agreement of HER2 IHC classification across 5 breast cancer-specialized pathologists, discordance was mostly driven by 0 vs. 1+ cases (43% of all discordant cases, 15% of total cases)³⁹. Results from studies evaluating the evolution of HER2-low expression status in primary to recurrent/advanced breast cancer^40,41 or other associations in HER2-low breast cancer could be confounded by the high inter-rater discordance and poor reproducibility in the low expression range for the historical HER2 IHC assays. Despite these concerning results, these studies were only able to make limited conclusions on the inter-rater reliability of scoring HER2 IHC due to their small number of pathologists or cases. Furthermore, conventional methods of inter-rater reliability for a small number of raters can poorly generalize to the broad population of raters.

In the real world, there are not just 3 pathologist raters, but thousands of pathologists scoring these assays. There is no established method for examination of concordance between large numbers of observers nor is there an established statistical method to determine how many observers are needed to represent real world pathologist performance. Recently we have described a method to examine this issue. The ONEST method (Observers Needed to Evaluate Subjective Tests)⁴², allows us to assess the likelihood of this assay showing concordance amongst many pathologist readers. Our goal here is to use this method to better understand the past and future value of the conventional legacy HER2 IHC assays designed for detecting high HER2 expression. The ONEST method is based on calculation of the overall percent agreement within many combinations of pathologists/raters in order to determine if there is a plateau in overall percent agreement. The presence of a plateau strongly suggests that the metric will be stable even if the number of raters/pathologists continues to increase.

Here, we examine the concordance or overall percent agreement at the various cut-points used in the HER2 IHC assay. In this multi-institutional study, we evaluate the inter-rater reliability of scoring HER2 IHC in 170 breast cancer cases by a group of 18 breast cancer-specialized pathologists. We quantify inter-rater reliability with common metrics as well as the recently developed ONEST method to better generalize the performance of the legacy HER2 assay to larger populations of pathologists in routine clinical practice settings.

Materials and methods

Patient biopsies and immunohistochemistry

We retrospectively collected 170 breast biopsies from the archives of the Department of Pathology at Yale School of Medicine. These were all from patients with breast cancer seen in 2018. This set was enriched for HER2 positive cases defined as those with 3+ score by immunohistochemistry (IHC), or 2+ by IHC and HER2 positive by fluorescent in situ hybridization (FISH), as defined by American Society of Clinical Oncology/College of American Pathologists (ASCO/CAP) clinical practice guidelines^10,11. The archival hematoxylin and eosin (H&E) slides and HER2 IHC slides were reviewed, and quality checked by a board-certified pathologist to confirm stain integrity and checked for strong membranous staining of positive controls, that the slide and coverslips were not broken, and that all slides had enough tissue to assess. The slides were scanned using Aperio ScanScope Console (v10.2.0.2352) using bright field Whole Slide Scanning at 20× magnification and sent to eighteen board-certified pathologists, most with over 5 years’ experience. Pathologists scored the cases as HER2 0, 1+, 2+, or 3+ according to the current ASCO/CAP criteria¹¹.

Statistical analysis and Observers Needed to Evaluate a Subjective Test (ONEST)

All statistical analyses were performed in R 4.1.0⁴³. The ONEST package⁴⁴ was used to model and visualize the change in overall percent agreement (OPA) as a function of the number of pathologists scoring the cases. This method is described in detail in Han et al⁴². Briefly, for each subset of cases, we randomly select combinations of pathologists and calculate the OPA for each group (from sizes 2 to 18 in this study of 18 pathologists). 100 curves (of 100 combinations for each group size) were generated for plotting, and 1000 curves were used to estimate the mean and 95% confidence interval of the ONEST plot. The resulting ONEST plots descend and can reach a non-zero plateau that can be validated by estimating the parameters of the statistical model described in (Han G. et al., 2021)⁴². The ONEST model can also be used to estimate the number of raters needed to reach the plateau by calculating when the OPA difference between successive groups becomes clinically insignificant (less than 0.5%). As shown previously^42,45, if a test is easy to interpret and has high concordance, then the plateau will occur at a large OPA value with a small number of raters. Conversely, when there is low concordance for a test, the plateau occurs at low OPA values or may drop to 0 with a large number of raters. The raters package⁴⁶ and irr package⁴⁷ were used to calculate Fleiss’ Kappa and the intraclass correlation coefficient (ICC). ICC was calculated using a two-way random-effects model with absolute agreement as the relationship type and single rater as the measurement unit. The ggplot2 package⁴⁸ was used for plotting and data visualizations in this study.

Results

To assess HER2 IHC inter-rater reliability in this study, whole tissue sections of 170 independent breast cancer biopsy cases were evaluated for HER2 IHC by 18 pathologists from 15 institutions. Figure 1 displays stacked bar plots of the HER2 IHC score given for each case (Fig. 1A) or by each of these pathologists (Fig. 1B). For the 170 cases, 121 cases had disagreement on the IHC score amongst the 18 pathologists. The overall percent agreement (OPA), which is the percent of cases where all pathologists/raters in the group agree (in this case all 18 pathologists), for the 170 HER2 IHC cases was 28.8%. Discordance was observed for each cut-point, with the 0 vs. 1+ cut-point displaying the largest number of discordant cases. Of the 170 cases, 92 were read as 0 by at least one pathologist. 23 of these 92 IHC 0 cases were concordant (which corresponds to an OPA of 25%)(Fig. 1A; Table 1). 44 cases were read as 3+ by at least one pathologist. Again, only 22 of the 44 IHC 3+ cases were concordant amongst all readers (Fig. 1A; Table 1). When evaluating how each pathologist scored the 170 cases, the largest discrepancy is in the percent of cases scored as HER2 negative (IHC 0) vs. low (1+ or 2+); some pathologists scored 40% of the cases as IHC 0 whereas others scored 20% of the cases as IHC 0 (Fig. 1B).

Figure 1: HER2 IHC scores for 170 cases read by 18 pathologists.

HER2 IHC score of the whole tissue sections. A) Each case on the x-axis is shown as the percent of observers that called the case HER2 0, 1+, 2+, or 3+. Concordant (100% agreement) and discordant cases are indicated as bars above the plot. B) Percent of cases with HER2 IHC score assigned by each of the 18 pathologists.

Table 1:

Summary of inter-rater reliability metrics for different HER2 IHC groups amongst 18 pathologists in 170 cases of breast cancer

HER2 IHC group	Overall Percent Agreement (95% CI)	Fleiss’ Kappa (95% CI)	ICC (95% CI)
4 category (0, 1+, 2+, 3+)	28.82 (22.01, 35.63)	0.65 (0.64, 0.66)	0.88 (0.85, 0.90)
3 category (0, Low*, 3+)	46.47 (38.97, 53.97)	0.74 (0.73, 0.75)	0.83 (0.79, 0.86)†
Only including cases with this score by at least one pathologist
0 only	25 (16.15, 33.85)	0.49 (0.47, 0.50)	0.55 (0.47, 0.64)
1+ only	0.98 (0, 2.89)	0.35 (0.34, 0.36)	0.52 (0.44, 0.60)
2+ only	3.57 (0, 7.54)	0.46 (0.45, 0.47)	0.67 (0.59, 0.74)
3+ only	50 (35.23, 64.77)	0.63 (0.61, 0.65)	0.69 (0.59, 0.78)
Low* only	27.2 (19.4, 35)	0.47 (0.46, 0.48)	0.56 (0.49, 0.63)†
0 vs. not 0	59.41 (52.03, 66.79)	0.69 (0.68, 0.70)	0.69 (0.64, 0.74)‡
Low* vs. not Low*	46.47 (38.97, 53.97)	0.69 (0.68, 0.70)	--
3+ vs. not 3+	87.06 (82.01, 92.1)	0.89 (0.88, 0.90)	0.89 (0.87, 0.91)‡
< 2+ vs. ≥ 2+	64.12 (56.91, 71.33)	0.77 (0.76, 0.78)	0.77 (0.73, 0.81)‡

^*The Low category is the result of combining the 1+ and 2+ categories.

^†To calculate ICC for the 3 category score and Low only cases, the IHC scores were converted to 0, 1, or 2 to represent HER2 negative, Low, and 3+ cases respectively.

^‡To calculate the ICC for these groupings, scores for cases were converted to ordinal scores of 0 or 1 based on increasing IHC score (e.g. IHC 0 or not 0 was converted to 0 and 1 respectively).

Next, we wanted to explore whether these metrics of OPA across the 18 pathologists in this study were generalizable to a larger population of pathologists performing HER2 IHC in breast cancer. To do this, we used the ONEST technique⁴² of plotting OPA within different combinations of groups of pathologists to determine if there is a point where OPA plateaus. The ONEST method was developed to not only determine the number of observers needed for evaluation of a subjective test, but also to predict how the test/biomarker would perform in the real world with thousands of pathologist readers. The presence of a plateau strongly suggests that the metric will be stable even if you continue to increase number of raters/pathologists (since there are thousands of pathologists in practice reading IHC, this method has the potential to predict how the biomarker will perform with thousands of raters). Additionally, the point where the metric plateaus indicates the number of pathologists that are required to provide realistic concordance estimates for when the assay is broadly used.

The ONEST plots in Figure 2 of OPA show a decrease in OPA as the number of raters in the group increases, with each plot reaching a plateau between 6 to 12 raters. When considering the concordance amongst pathologists using a four-category score (0, 1+, 2+, 3+) compared to a three-category score (0, Low*, 3+), a three-category score yielded a higher OPA (28.8% OPA for four-category compared to 46.5% OPA for three-category)(Fig. 2A; Table 1). This can also be seen in Figure 1A, as combining 1+ and 2+ categories removes 30 discordant cases. Similarly, the Fleiss’ kappa increases using a three-category score compared to a four-category score for reading HER2 IHC (Fleiss’ kappa of 0.65 compared to 0.74 for four- and three-category scores respectively)(Fig. S1; Table 1). This suggests that there is substantial discordance when assessing whether a case is 1+ or 2+, and that combining 1+ and 2+ cases into a HER2-low category can result in increased concordance.

Figure 2: ONEST plots of overall percent agreement for different HER2 IHC groupings.

ONEST plot of HER2 IHC overall percent agreement (OPA) in a four category score (0, 1+, 2+, 3+) (a). OPA ONEST plots for the subset of cases that were read as HER2 IHC 0 (b) or 1+ (c) by at least one of the 18 pathologist raters. OPA ONEST plots for the task of determining HER2 IHC score of 3+ vs. not 3+ (d). One hundred curves were randomly generated from all possible combinations of pathologists for each HER2 IHC grouping. 0.5%).

We wanted to determine the OPA using the ONEST method for cases that were scored as 0 and cases that were scored as 1+ since this is the cut-point for determining whether a case is HER2-low and hence a candidate for HER2-low therapies including T-DXd. Other studies have reported that the bulk of discordance in HER2 IHC is driven by cases that were discordant between 0 vs 1+³⁹. Similarly, concerning disagreement rates of HER2 0 scores between local and central assessment (85%) have been reported³⁸. Figure 2B and 2C show the OPA ONEST plots when cases were scored as 0 and 1+ respectively by at least one of the 18 pathologists. The OPA for the cases that were scored as 0 plateaus at 25%, indicating that the pathologists disagreed in 75% of the cases that were scored as 0 by at least one pathologist. The pathologists’ discordance of the 0 cases was mainly between scores of 0 vs. 1+ (785/1656 of total ratings within 0 cases, 69/92 of 0 cases read as 1+ by another pathologist) and to a much lesser extent between scores 0 vs. 2+ (85/1656 ratings, 29/92 cases)(Table S1; Table S2). The OPA for the cases that were scored as 1+ reaches less than 1%. This is due to there being only 1 case that all 18 pathologists agreed that was 1+ out of the 102 cases that were scored as 1+ by at least one pathologist (Fig. 1A). The 2+ cases also had a very low OPA that reached 3.6% (Fig. S2-C,H; Table 1). Upon combing the 1+ and 2+ categories into a HER2 low category, the OPA for these low cases increases and plateaus at 27.2% (Fig. S2-D,I; Table 1). Also surprising, only 50% of the HER2 IHC 3+ cases were agreed upon by all of the 18 pathologists (Fig. S2-E,J; Table 1).

Since the emergence of lower levels of HER2 as a target for therapy, determining when a case is 0 vs. not 0 is an important clinical decision threshold for prescription of HER2-low therapies. This new threshold is added on to the existing threshold where trastuzumab is prescribed in HER2 amplified cases, defined as 3+ or 2+ and ISH+. Thus, we next evaluated the pathologists’ ability to make clinically impactful/significant reads at both clinical thresholds. First, for the task of determining cases with a 3+ score vs. not 3+ and then for cases with a 0 score vs. not 0. To do this, we grouped the HER2 IHC scores as 3+ or not 3+ and 0 or not 0 for analysis. In the HER2 IHC ONEST plot of the scores grouped as 3+ or not 3+ (Fig. 2D), there was an OPA of 87.1% that plateaued around 6 raters. This OPA demonstrates that this group of pathologists has high agreement for the task of determining 3+ cases from not 3+ cases. Correspondingly, in the ONEST plot of the scores grouped as 0 or not 0 (Fig. S3; Table 1) pathologists in the study had an OPA of 59.4% that plateaued around 10 raters. This observation agrees with previous studies that suggest that pathologists cannot agree on cases with a HER2 0 score, as there is up to a 40.6% disagreement for what cases are IHC 0 or not 0.

Discussion

This multi-institutional study assessing the inter-rater reliability of HER2 IHC scoring demonstrates several findings and offers generalizable concordance estimates for scoring HER2 IHC. The first finding is that the intermediate categories (1+ and 2+) in the four-category HER2 IHC scoring system are a large source of discordance (<1% agreement for 1+ and 3.6% agreement for 2+), and this discordance can be partially reduced by using a three-category system (28.8% vs. 46.5% OPA for four and three-category scoring systems respectively). Intermediate categories being less reproducible than the extreme categories is a trend that has been found in several other studies for different multi-category assays^{42,45,49–51}. The low agreement of 2+ cases in this study is due to discordance in scores of 2+ vs. 3+ (22/84 of 2+ cases read as 3+ by another pathologist), 1+ vs. 2+ (61/84), as well as a non-negligible number of discordant cases for scores of 0 vs. 2+ (29/84)(Table S2). The disagreement of cases at the 0 or 1+ vs. 2+ boundaries is concerning as these cases would not receive reflexive FISH testing (under the current ASCO/CAP guidelines) to check for HER2 gene amplification status if they were marked as 0 or 1+. Although the prevalence of HER2 gene amplification in IHC 0/1+ cases is much lower than IHC 2+ cases, past studies have demonstrated that these patients can have pathologic complete response to HER2 amplified therapy regimens^36,37.

We also found that there is a low concordance amongst pathologists in this cohort when evaluating breast cancer cases with HER2 IHC score of 0 and at the task of determining a score of 0 or not 0. This is a critical cut-point for the new HER2 antibody-drug conjugates. Other studies have also reported discordance for scoring HER2, particularly around the 0 to 1+ or 2+ cut-points^34,38,39. Despite knowing that there was potentially a high level of discordance, these studies did not show generalizability of their results to a large population of pathologists. The ONEST plots in this study suggests that the OPA for the task of determining a HER2 IHC score 0 from not 0 plateaus statistically around 59.4%. The agreement for assigning a HER2 IHC score of 0 vs. not 0 is only slightly better than a coin flip amongst these 18 pathologists in this multi-institutional study. Conversely, at the task of scoring HER2 IHC as 3+ or not 3+ pathologists’ concordance was much higher with an OPA that plateaus at 87.1%. These results indicate that the legacy HER2 IHC assay is largely valuable as is to find HER2 gene amplified patients for conventional HER2 targeted therapies, but unacceptably discordant for assigning HER2-low status for emerging HER2-low therapies.

This study has a number of considerations and limitations. One potential limitation is that the 18 pathologists that scored the biopsy cohort were not told that the 0 vs. 1+ concordance level would be assessed. In retrospect, many said they would have examined the low expressing cases more closely. Also, the set of breast cancer biopsies was enriched in 2+ and 3+ cases but most of the pathologists did not know this prior to reading the slides. This could have contributed to pathologists assigning HER2 negative and 1+ scores more frequently as these cases are more common in clinical practice. However, these considerations can be argued as strengths for this study, as compelling pathologists to provide additional scrutiny of the 0/1+ cases or informing them about the cohort composition beforehand would not have provided an accurate reflection of how pathologists really score these cases. Another limitation of this study is that we do not have quantitative molecular measurements of HER2 in the examined core biopsies. However, the absence of a criterion standard is not unusual in pathologist concordance studies.

Although the legacy HER2 IHC assay combined with ISH is the companion diagnostic for amplified HER2 therapies including trastuzumab, the legacy HER2 IHC assay’s high discordance and poor inter-rater reliability amongst pathologists for scoring IHC 0, 1+, and 2+ cases demonstrated in this study suggest that this assay will be problematic for the emerging HER2-low treatments. Finally, in agreement with other studies performed with other methods, the high level of discordance for pathologists scoring HER2 IHC 0 vs. not 0 (40.6% disagreement reported by ONEST in this study), suggests that the legacy HER2 IHC assay will likely be inaccurate and arguably insufficient for clinical decision making when prescribing HER2-low specific treatments (e.g. trastuzumab-deruxtecan).

Data Availability Statement

The datasets used and/or analyzed during the current study are available within the supplemental materials. The scripts used in the analysis are available at https://github.com/crobbins327/HER2-IHC-ONEST-Multi-Institutional-IRR. Additional datasets can be made available from the corresponding author on request.