Computer-aided scoring of erb-b2 receptor tyrosine kinase 2 (HER2) gene amplification status in breast cancer

Abstract

Background

Identification of HER2 protein overexpression and/or amplification of the HER2 gene are required to qualify breast cancer patients for HER2 targeted therapies. In situ hybridization (ISH) assays that identify HER2 gene amplification function as a stand-alone test for determination of HER2 status and rely on the manual quantification of the number of HER2 genes and copies of chromosome 17 to determine HER2 amplification.

Methods

To assist pathologists, we have developed the uPath HER2 Dual ISH Image Analysis for Breast (uPath HER2 DISH IA) algorithm, as an adjunctive aid in the determination of HER2 gene status in breast cancer specimens. The objective of this study was to compare uPath HER2 DISH image analysis vs manual read scoring of VENTANA HER2 DISH-stained breast carcinoma specimens with ground truth (GT) gene status as the reference. Three reader pathologists reviewed 220, formalin-fixed, paraffin-embedded (FFPE) breast cancer cases by both manual and uPath HER2 DISH IA methods. Scoring results from manual read (MR) and computer-assisted scores (image analysis, IA) were compared against the GT gene status generated by consensus of a panel of pathologists. The differences in agreement rates of HER2 gene status between manual, computer-assisted, and GT gene status were determined.

Results

The positive percent agreement (PPA) and negative percent agreement (NPA) rates for image analysis (IA) vs GT were 97.2% (95% confidence interval [CI]: 95.0, 99.3) and 94.3% (95% CI: 90.8, 97.3) respectively. Comparison of agreement rates showed that the lower bounds of the 95% CIs for the difference of PPA and NPA for IA vs MR were –0.9% and –6.2%, respectively. Further, inter- and intra-reader agreement rates in the IA method were observed with point estimates of at least 96.7%.

Conclusions

Overall, our data show that the uPath HER2 DISH IA is non-inferior to manual scoring and supports its use as an aid for pathologists in routine diagnosis of breast cancer.

Highlights

• •Image analysis algorithm for HER2 amplification using Bright-field ISH in Breast.
• •Automated tumor cell selection and quantitation within pathologist defined ROI.
• •The image analysis algorithm is non-inferior to manual scoring.
• •Integrated solution to support pathologists in determining HER2 gene status.

Background

Breast cancer is the most common cancer among women and the leading cause of global cancer mortality with more than 680 000 deaths among women in 2020.¹ Human erb-b2 receptor tyrosine kinase 2 (ERBB2 or HER2), a member of the epidermal growth factor receptor family, is overexpressed or amplified in 15%–20% of breast cancers^2,3 and overexpression/amplification is associated with a higher rate of recurrence and death.^4,5 Routine clinical use of trastuzumab and other HER2 targeted therapies have dramatically changed outcomes for patients with HER2 overexpression, providing improved survival, and quality of health.^6,7 Determination of HER2 positivity, by either overexpression or gene amplification, is fundamental to the current treatment paradigm, ensuring that patients receiving therapy can derive the most benefit from HER2-targeted therapies. In current practice, HER2 status is determined through immunohistochemical and/or in situ hybridization (dual color fluorescence [FISH] or dual bright-field chromogenic in situ hybridization [DISH]) assays.⁸ For in situ hybridization, the necessity of manual quantification of HER2 gene amplification, while reproducible, remains a time-consuming and expensive process.⁹ Furthermore, with pathologists facing ever-expanding work responsibilities, combined with a shrinking number of new pathologists entering the field, there is a growing unmet need for tools to assist pathologists in managing their daily work.10, 11, 12 Over the past several years, the increasing availability of slide scanners that can digitize a glass slide and produce a whole slide image (WSI) at high-resolution, combined with advances in computers and data storage has brought the promise of digital pathology into reality. In addition to full digitization of the pathologist workflow, the ability to leverage WSI and computer image analysis algorithms to aid pathologists with quantitative and qualitative tasks are also now possible. Indeed, the need for manual quantification of HER2 gene amplification in both FISH and DISH assays represents an ideal use case for image analysis algorithms to function as an aid for pathologists. As an example, automated image analysis has been used for FISH analysis and was found to be concordant with manual analysis,13, 14, 15 despite increased variability at higher HER2 counts.¹⁶

Here we have developed the uPath HER2 Dual ISH image analysis algorithm for breast (uPath HER2 DISH IA) as a clinical adjunctive aid for pathologists to score breast cancers stained with the VENTANA HER2 Dual ISH DNA Probe Cocktail assay. In this study, we demonstrate that the uPath HER2 DISH IA algorithm displays high inter- and intra-reader agreement between pathologists, and is concordant to manual assessment of the VENTANA HER2 Dual ISH DNA Probe Cocktail assay. Collectively our data show that the uPath HER2 DISH IA algorithm is robust, reproducible and non-inferior to manual pathologist scoring.

Methods

Tissue specimens, tissue processing, and ISH staining

Invasive breast cancer specimens used for validation and development of the uPath HER2 DISH IA algorithm were sourced from commercial vendors as formalin-fixed paraffin-embedded (FFPE) tissues and absent of identifying data, except for the histological diagnosis. All samples were reviewed by a Roche pathologist to confirm histological diagnosis and ensure the presence of viable tumor before use. FFPE breast cancers were sectioned at 4 μM, mounted onto positively charged glass slides and stored at room temperature until stained. Tissue sections were de-paraffinized, exposed to an antigen retrieval reagent, and stained with VENTANA HER2 Dual ISH DNA Probe Cocktail and counterstained with hematoxylin and bluing reagent on the BenchMark ULTRA platform (Ventana Medical Systems, Inc.) according to the recommended manufacturer staining procedure. The 220 de-identified, FFPE breast carcinoma tissue specimens used for the studies described below were sourced for an unrelated institutional review board (IRB) approved study assessing reproducibility of the VENTANA HER2 Dual ISH DNA Probe Cocktail assay as part of the regulatory approval for the assay.

uPath HER2 dual ISH image analysis algorithm development

The VENTANA HER2 Dual ISH DNA Probe Cocktail assay is an FDA approved/CE marked1 diagnostic device used for determination of HER2 gene status in breast cancer. Regulatory approval of the VENTANA HER2 Dual ISH DNA Probe Cocktail assay is based upon manual counting HER2 copies via silver in situ hybridization (SISH) and chromosome 17 (Chr17) copies via chromogenic red in situ hybridization (Re ISH) on a single slide as described in the VENTANA HER2 Dual ISH DNA Probe Cocktail interpretation guide for breast tissue specimens.¹⁷ The uPath HER2 Dual ISH IA is intended to assist in interpretation of VENTANA HER2 Dual ISH staining by automating cell selection, counting of HER2 gene and Chromosome 17 copies, and quantification of the HER2/Chr17 ratio (Fig. 2). The uPath HER2 Dual ISH IA is a ranking algorithm that utilizes a feature set, e.g. cell shape (roundness metrics, cell segmentation), to identify tumor nuclei and HER2 gene and Chr17 copies. Although the algorithm itself is not a deep learning algorithm, the feature set used for the ranking and selection of tumor nuclei was derived using an advanced machine learning approach during development. The uPath HER2 Dual ISH IA algorithm is composed of several individual image processing steps that are tightly integrated within the user workflow in the uPath Enterprise software. They include: (1) unmixing color images into blue, red, and black channels, (2) detecting “dots” (e.g. assay signals of HER2 loci and Chr17 copies) in the red and black channels, (3) segmenting cells into scorable cells, and (4) scoring the segmentation based on quality and ranking (Fig. 3). Dots are detected based upon measurement of the signal amplification and the center of the signal center. Cell segmentation is defined as “scorable” if they include both black and red signals and if the segmentation quality is found to be high. Scorable cells are ranked by segmentation quality (i.e. confidence), the number of detected signals per cell, and the likelihood of it being a tumor cell (as defined by cell area). Cell segmentation is performed via an adaptive voting-based algorithm that generates a quality score for each segmented cell. The algorithm ranks the individual cells throughout the region of interest (ROI) using the criteria 1–4 as described above. Twenty cells with the highest ranked scores are then selected by the algorithm for HER2/Chr17 assessment. More in-depth details of the development of the uPath HER2 Dual ISH IA algorithm are proprietary and are not publicly available at this time.

Fig. 2.

Overview of the uPath HER2 dual ISH image analysis algorithm workflow.

(A) Manual scoring workflow of the VENTANA HER2 Dual ISH DNA Probe Cocktail Assay. (B–E) The uPath HER2 DISH IA algorithm utilizes a heat map (B) in the slide viewer to orient the pathologist and highlight potential areas of HER2:Chr17 amplification. (C) An overlay of segmented tumor cells is provided to the pathologist to allow review of tumor cells and for placement of the region of interest (ROI) for analysis. Placement of the ROI can be performed at a magnification of 20x. Once placed, the uPath HER DISH IA algorithm will select 20 cells within the ROI and calculate the raw HER2 and Chr17 counts and the HER2:Chr17 ratio. (D) After analysis, HER2 and Chr17 counts for each individual cell and the calculated HER2:Chr17 ratio are displayed. The pathologist is able to review the selected cells and either remove and add new cells or confirm the analysis. (E) The result is then displayed and additional diagnostic information (e.g. heterogeneity, clusters) can be included for the final report.

Fig. 3.

Algorithm processing feature set.

The algorithm’s feature set is composed of several individual criteria which include: unmixing color images into blue, red, and black channels, detecting “dots” (e.g. HER2 loci and Chr17 copies) in the red and black channels, segmenting cells into scorable cells, scoring the segmentation based on quality and ranking and selecting the top 20 cells.

^aA scorable cell is defined as a cell with both red and black dots with a high segmentation confidence.

^bThe top 20 cells are selected based on a high segmentation confidence, a high number of dots, and the likelihood of the cell being a tumor cell.

The early feasibility studies utilized pathologist review (n=3) to establish the accuracy and precision of the cell ranking by the uPath HER2 Dual ISH IA algorithm. The algorithm’s accuracy was assessed at the cell level using a data set created from pathologist (n=1) selection of 50 non-overlapping ROIs from 50 images (1 ROI per image). For each ROI, the algorithm selected 20 cells that were within the cell segmentation boundary and had both HER2 and Chr17 signals (Fig. 3). A total of 1000 cells, generated from the 50 ROIs, were independently evaluated by a panel of 3 pathologists to ensure accuracy of cell segmentation, cell selection, and dot count accuracy.

To verify the algorithm’s ability to segment cells, pathologists visually assessed cells (n=20) selected by the uPath HER2 Dual ISH IA algorithm to determine if segmentation of each cell was acceptable or unacceptable. Segmentation was considered acceptable if the algorithm correctly incorporated the countable dots of a single cell in the segment and non-acceptable if algorithm did not incorporate the countable dots of a cell, or incorporated extra dots from an adjacent cell and/or included miscellaneous signals. To verify the algorithm’s ability to select cells, pathologists reviewed the algorithm’s selection of the 20 selected cells (out of all cells in the ROI) as either acceptable or unacceptable. Acceptable cells were tumor cells with the highest number of distinct black and red signals representative of that tumor area in a field of view (FOV). Unacceptable cells were lymphocytes or stroma cells that did not have distinct black and red signals, or were not one of the cells in a FOV with the highest number of both black and red signals. A minimum of 2 pathologists had to agree for a cell to be considered acceptable. In order to evaluate the acceptability of the algorithm’s selection of HER2 and Chr17 signals, pathologists reviewed the HER2 and Chr17 results generated by the algorithm in selected cells (n=20). Pathologists (n=3) reviewed the dot detection results and made corrections by either adding missing or removing incorrectly detected signals, i.e. algorithm dots. The total of HER2 and Chr17 signals after correction were considered the pathologist manual count for HER2 and Chr17 signals, respectively, for that ROI. The median number (generated from 3 different reads by 3 different pathologists) was considered the consensus count for either HER2 or Chr17. The R-squared (R²) values (Coefficient of determination) between the algorithm count (before correction) and Consensus (defined as a panel of 3 pathologists who provided the consensus scores) pathologist counts (after correction) for HER2 and Chr17 dots were compared. R² values greater or equal to 0.9 were considered acceptable.

Pathologist training

A total of 9 pathologists, who were previously qualified to manually evaluate VENTANA HER2 Dual ISH-stained breast cancer specimens for HER2 gene status, participated in the study. Three groups, Screening, Consensus, and Reader pathologists performed specific study activities (Fig. 1) and underwent training specific for the study protocol prior to performing study activities. Reader (pathologists who participated in digital and manual scoring) Pathologists (n=3) received training regarding the use the uPath HER2 Dual ISH IA algorithm. Training for the uPath HER2 Dual ISH IA algorithm was conducted by a Roche Pathologist involved in the development of algorithm. The training took place in multiple parts, including a group session, self-study (6 cases with consensus scores), mini test (6 cases with consensus scores), and a final proficiency test (12 cases with consensus scores). Reader Pathologists were required to complete a final proficiency test and achieve 90% overall percent agreement to consensus to ensure that they were comfortable and proficient in using the HER2 Dual ISH IA algorithm on Roche uPath enterprise software.

Fig. 1.

Study workflow.

Flowchart presenting the workflow of the validation studies.

ISH staining and case selection for validation studies

A total of 220 de-identified, FFPE breast carcinoma tissue specimens (§1.2.1) stained with the VENTANA HER2 Dual ISH DNA Probe Cocktail on BenchMark ULTRA instruments at external laboratories under an unrelated study protocol were retrieved from that study’s archives and used for the validation studies. VENTANA HER2 Dual ISH- and H&E-stained slides for all 220 selected cases were reviewed by the Screening Pathologists to ensure staining and adequate tumor tissue (Fig. 1). Slides that met the criteria and had a valid VENTANA HER2 Dual ISH staining run according to the package insert were included in the study. To establish the GT of each case, a panel of Consensus Pathologists (n=3) evaluated the HER2/Chr17 ratio for the selected cases and scored cases as shown in Table 1. GT status was established by two Consensus Pathologists independently scoring each case by manually counting cells on glass slides using a microscope according to the VENTANA HER2 Dual ISH Interpretation Guide¹⁷ (P/N 1018859, Cat.No.760-6072). Due to the inability to ensure that pathologists count the same exact cells and regions, consensus for cases was determined by agreement on amplification status (defined as non-borderline amplified, borderline amplified, borderline non-amplified, and non-borderline non-amplified) instead of exact HER2/Chr17 ratios. If the amplification status assigned by the 2 Consensus Pathologists agreed with each other for a given case, then the agreed status was assigned as the GT status. In the event that the Consensus Pathologists disagreed on the HER2 gene status for a given case, the third Consensus Pathologist independently scored the case in the same manner as the first two Consensus Pathologists. The majority amplification status among the 3 Consensus Pathologists was the GT status. However, if all 3 Consensus Pathologists disagreed (e.g., amplified, non-amplified, and unevaluable), then for those cases the 3 Consensus Pathologists met together under a multi-headed scope and achieved a GT status by consensus. Selected glass slides were scanned on the VENTANA DP 200 slide scanners2 at 40x magnification. Resulting digital slide images were checked for quality before transfer to the centralized server for analysis using the Roche uPath enterprise software and the uPath HER2 Dual ISH IA algorithm. The number of cases distributed to each score bin (non-borderline amplified, borderline amplified, borderline non-amplified, and non-borderline non-amplified) were determined based upon the reported clinical prevalence.^8,18, 19, 20, 21, 22

Table 1.

Summary of case qualification and enrollment.

Case bins	Number of cases enrolled
Non-borderline amplified cases (Ratio>2.6)	99
Borderline amplified cases (Ratio ≥2.0 and ≤2.5)	11
Borderline non-amplified cases (Ratio ≥1.5 and ≤1.9)	11
Non-borderline non-amplified cases (Ratio<1.4)	99
Total, no.	220

Method comparison study

The primary endpoint of this method comparison study was to determine whether the uPath HER2 Dual ISH IA algorithm scoring of breast cancer slides was non-inferior to manual pathologist scores. To accomplish this, HER2 scoring results from three Reader Pathologists’ IA and Manual Read (MR) modalities were compared against Consensus Pathologists’ GT HER2 gene status for the 220 selected cases. All pathologists who participated in the study were blinded to all previous HER2 gene status information of the cases. MR were conducted by three Reader Pathologists who independently scored each of the 220 cases using the microscope at a magnification of 600x per the VENTANA HER2 Dual ISH DNA Probe Cocktail assay interpretation guide (Fig. 2A).¹⁷ After completing a minimum 14 day washout following MR evaluation, the same three Reader Pathologists performed IA using the uPath HER2 Dual ISH IA algorithm reads according to the following steps: (1) selection of region of interest (ROI) containing 20 tumor nuclei with the help of the Heat Map (Fig. 2B), (2) the HER2/Chr17 ratio was recorded from the HER2 Dual ISH algorithm, (3) if deemed necessary by the pathologist, individual cells were edited within the ROI and replaced. The number of cells added or deleted was recorded. For cases where the HER2 score for the case was within 1.8–2.2 inclusive, the Reader Pathologist selected a second, non-overlapping ROI and applied the HER2 Dual ISH algorithm to another 20 nuclei and recorded the score from the 40 selected nuclei, in accordance with the FDA approved, CE marked VENTANA HER2 Dual ISH DNA Probe Cocktail assay.²³ Override scores at both the cell and overall score level were also captured and recorded. HER2 gene status was defined as either amplified (HER2/Chr17 ratio ≥ 2.0) or non-amplified (HER2/Chr17 ratio < 2.0) in accordance with HER2 scoring described in the VENTANA HER2 Dual ISH assay interpretation guide. HER2 gene status was used as a reference to determine the difference in agreement rates between MR and IA modalities. The difference in positive, negative, and overall agreement rates of HER2 gene status between IA vs GT and MR vs GT were calculated.

Inter-reader agreement for image analysis study

Inter-reader agreement was performed in addition to the primary analysis. The most common HER2 gene status given by the 3 Reader Pathologists (e.g. if a case is determined to be positive in 2 observations and negative in 1 observation from 3 IA reads, the status mode for that case would be positive) was calculated as the modal HER2 gene status. If the modal status for a case could not be determined, the HER2 gene status determined during case qualification was used as the reference. Inter-reader agreement (OPA, PPA, NPA) rates for each reader was compared to the modal status.

Intra-reader reproducibility and scanner precision studies

In addition to our primary objective to assess the non-inferiority of HER2 Dual ISH IA algorithm to manual scoring, we also assessed intra-reader reproducibility and inter-scanner precision in parallel studies to evaluate the robustness of the HER2 Dual ISH IA algorithm. To assess intra-reader reproducibility, a subset of primary breast cancer cases (18 Amplified; 2 Borderline amplified; 2 Borderline-non-amplified; 18 Non-amplified) were randomly selected from the 220 case set. A single pathologist assessed the digital images for each case in 3 separate reading rounds using the HER2 Dual ISH IA. A minimum of a 15-day washout period was required between each round. Next, the pathologist assigned HER2 status for each case as either amplified (HER2/Chr17 ratio ≥ 2.0) or non-amplified (HER2/Chr17 ratio < 2.0) in accordance with the HER2 scoring described in the VENTANA HER2 Dual ISH assay interpretation guide. Scanner precision was determined using glass slides from the intra-reader assessment. Slides were scanned 3 times on 3 different VENTANA DP 200 slide scanners to produce digital images. Each resulting digital image was analyzed by the HER2 Dual ISH IA to assign a HER2 score (HER2/Chr17 ratio) to the case. Agreement rates between IA evaluations of multiple scans of the same HER2 DISH-stained primary breast specimens from different scanners were compared against the case modal status.

Statistical analysis

All analyses were performed using SAS (version 9.4 or older) software. Ventana biostatisticians produced all analyses and summaries using SAS version 9.4. By-observation data listings were prepared for all data collected on data forms. The planned analyses and study results at times used ‘positive’ and ‘negative’ to denote ‘amplified’ and ‘non-amplified’, respectively. The aggregated scores from all 3 Reader Pathologists were used to determine the difference of PPA and NPA point estimates for the overall difference of agreement in HER2 gene status. The 95% CIs for these analyses were calculated using the percentile bootstrap method with cases stratified by the 4 screening bins. In order to demonstrate non-inferiority of IA to MR, the lower bounds of the difference of PPA and NPA 95% CIs had to be greater than or equal to -10%. The difference of PPA, NPA, and OPA point estimates and the difference of OPA 95% CI were calculated for descriptive purposes only. In order to demonstrate adequate inter-reader agreement, the point estimates for PPA and NPA had to be both ≥85%. During data analysis, a biostatistician assigned each case a HER2 gene status (amplified or non-amplified) based on its assigned HER2 score. Scoring used the HER2 scoring categories of amplified (HER2/Chr17 ratio ≥ 2.0) and non-amplified (HER2/Chr17 ratio < 2.0). For analysis purposes, an amplified gene status greater than or equal to 2.0 was positive and a non-amplified gene status less than 2.0 was negative.

Randomization and bias minimization

All study cases included in the study were randomized before the pathologist’s assessment began. Sample identification on glass slides were obscured and replaced by study case IDs. Several measures were employed to mitigate the risk of bias, including selection bias and recall bias. These measures included random selection of cases from tissue cohort (as specified in the study randomization plans), randomization in pathologist’s reading order, a 15-day washout period for intra-reader reading sessions, a 14-day washout period between IA and MR methods, and blinding of Reader Pathologists to the previous HER2 IHC/ISH results.

Results

uPath HER2 dual ISH image analysis algorithm development

The uPath HER2 Dual ISH IA algorithm is a ranking algorithm that uses a feature set of image criteria developed through a learning approach to rank cells based on dot detection, cell segmentation, and tumor cell probability. A description of the algorithm calculation of HER2 amplification status is described in the Methods section. To evaluate the algorithm’s capability to segment cells, HER2 DISH stained images were analyzed by the algorithm and reviewed by pathologists (n=3) to determine the correct segmentation of single cells. On average, there were 17.86 cells having acceptable segmentation by 2 out of 3 pathologists, which translated to 89% consensus accuracy. Pathologists (n=3) also evaluated the algorithm's cell selection accuracy by reviewing the algorithm performance in selecting the 20 highest ranked tumor cells, defined as tumor cells with distinct black and red signals and containing the highest number of both black and red signals representative of that tumor area. On average, there were 16.29 cells called acceptable by 2 out of 3 pathologists, which translated to an 81% consensus accuracy. To assess the algorithm’s performance in detection of individual HER2 and Chr17 signals (or dots), we compared the Coefficient of determination or R-squared (R²) between the algorithm count (before correction) and Consensus Pathologist counts (after correction) for HER2 and Chr17 signals. On average, the R² for red signal detection was 0.926 and R² for black signal detection was 0.99, indicative of a high level of correlation. Based upon these criteria, the uPath HER2 Dual ISH IA algorithm met the pre-defined criteria for feasibility testing. The code for the algorithm was locked and deployed for the method comparison studies.

Comparison of image analysis to manual read method and ground truth

Concordance between the uPath HER2 Dual ISH IA algorithm and MR methods was evaluated by comparing each method’s HER2 amplification status to GT HER2 amplification status established by three Consensus Pathologists (Fig. 1). A total of 220, FFPE breast carcinoma cases stratified into 4 case bins based on HER2 amplification status were used for these studies (Table 1). The distribution of HER2 status reflects the prevalence of amplified, borderline, and non-amplified breast carcinoma specimens in routine clinical practice.^8,18, 19, 20, 21, 22 The agreement rates for each Reader Pathologist were calculated in the direction of IA vs GT minus MR vs GT (Supplementary Tables 1–3). The aggregated HER2 amplification status from all 3 Reader Pathologists were used to determine the difference between MR vs GT scoring and IA vs GT scoring. (Table 2) The differences between IA and MR (IA - MR) agreement rates were 0.3% (1/321) (2-sided 95% CI:-0.9, 1.8) for PPA, -3.3% (-11/331) (2-sided 95% CI: -6.2, -1.2) for NPA, and -1.5% (-10/652) (2-sided 95% CI: -3.1, -0.2) for OPA, meeting the pre-determined acceptance criteria for the study. Overall, in terms of GT, there were 2 false negatives, where the HER2 Dual ISH IA read was negative and the MR was positive, vs 14 false positives, where HER2 Dual ISH IA was positive and the MR was negative. The net difference of false positives vs false negatives, (14 - 2 = 12) represents 12/652 = 1.8% of all observations.

Table 2.

Difference in agreement rates between manual reads vs. ground truth, and image analysis vs. ground truth.

		MR status
GT status	IA status	Amplified	Non-amplified	Total
Amplified	Amplified	309	3	312
	Non-amplified	2	7	9
	Total	311	10	321
	MR PPA n/N (%) (95% CI)	311/321(96.9)	(94.2, 99.0)
	IA PPA n/N (%) (95% CI)	312/321(97.2)	(95.0, 99.3)
	Difference of PPA (IA - MR) n/N (%) (95% CI)	1/321 (0.3)	(-0.9, 1.8)
Non-amplified	Amplified	8	11	19
	Non-amplified	0	312	312
	Total	8	323	331
	MR NPA n/N (%) (95% CI)	323/331 (97.6)	(95.8, 99.4)
	IA NPA n/N (%) (95% CI)	312/331 (94.3)	(90.8, 97.3)
	Difference of NPA (IA - MR) n/N (%) (95% CI)	-11/331 (-3.3)	(-6.2, -1.2)
Overall	Amplified	317	14	331
	Non-amplified	2	319	321
	Total	319	333	652
	MR OPA n/N (%) (95% CI)	634/652 (97.2)	(95.7, 98.6)
	IA OPA n/N (%) (95% CI)	624/652 (95.7)	(93.8, 97.5)
	Difference of OPA (IA - MR) n/N (%) (95% CI)	-10/652 (-1.5)	(-3.1, -0.2)

GT ground truth, IA image analysis, MR manual read; PPA positive percent agreement, NPA negative percent agreement, OPA overall percent agreement, CI confidence interval.

Note: 2-sided 95% confidence intervals were calculated using the percentile bootstrap method with cases stratified by the 4 screening bins.

Note: This analysis includes all 3 study Reader Pathologists' results.

Inter-reader agreement for image analysis

Inter-reader agreement for the IA reading method was assessed by comparing each reader’s HER2 gene status result for a case to the respective modal HER2 gene status. The aggregated reads from the Reader Pathologists (n=3) are presented in Table 3 and showed a high level of agreement for the IA reading method for readers to the modal HER2 gene status. Similarly, the agreement for the MR HER2 gene status compared to the inter-reader case-level was also very high with a PPA rate of 98.2% (319/325) (95% CI: 97.0, 99.4) and a NPA rate of 98.8% (328/332) (95% CI: 97.6, 99.7) (Supplementary Table 4).

Table 3.

Agreement of IA HER2 gene status in comparison to the inter-reader case-level.

		Modal result			Agreement
Reader	Reader result	Amplified	Non-amplified	Total	Measurement	% (n/N)	95% CIa
Overall	Amplified	322	11	333	PPA	98.8 (322/326)	(97.6, 99.7)
	Non-amplified	4	318	322	NPA	96.7 (318/329)	(94.8, 98.4)
	Total	326	329	655	OPA	97.7 (640/655)	(96.6, 98.6)
Reader 1	Amplified	109	5	114	PPA	100.0 (109/109)	(96.6, 100.0)
	Non-amplified	0	105	105	NPA	95.5 (105/110)	(89.8, 98.0)
	Total	109	110	219	OPA	97.7 (214/219)	(94.8, 99.0)
Reader 2	Amplified	105	0	105	PPA	96.3 (105/109)	(90.9, 98.6)
	Non-amplified	4	109	113	NPA	100.0 (109/109)	(96.6, 100.0)
	Total	109	109	218	OPA	98.2 (214/218)	(95.4, 99.3)
Reader 3	Amplified	108	6	114	PPA	100.0 (108/108)	(96.6, 100.0)
	Non-amplified	0	104	104	NPA	94.5 (104/110)	(88.6, 97.5)
	Total	108	110	218	OPA	97.2 (212/218)	(94.1, 98.7)

IA image analysis, PPA positive percent agreement, NPA negative percent agreement, OPA overall percent agreement, CI confidence interval.

^aFor reader overall result, the confidence intervals were calculated using the percentile bootstrap method. For each reader result, the confidence intervals were calculated using the Wilson Score method. Note: Only observations with non-missing evaluation results were included in this analysis.

Direct comparison of image analysis and manual reads agreement rates

Agreement between IA and MR was also compared directly between Reader Pathologists and to the overall (aggregated) scores (Table 4). Consistent with our other comparisons, the IA reading method displayed strong agreement to manual scoring, both as an overall aggregate score (OPA 97.5%, [95% CI: 96.0, 98.8]) and between individual readers (Reader 1-OPA 99.1% [95% CI: 96.7, 99.7], Reader 2-OPA 98.1% [95% CI: 95.3, 99.3], and Reader 3-OPA 95.4%, [95% CI: 91.7, 97.5]).

Table 4.

Agreement between manual read and image analysis algorithm.

		Manual read result			Agreement
Reader	IA result	Positive	Negative	Total	Measurement	% (n/N)	95% CIa
Overall	Amplified	317	14	331	PPA	99.4 (317/319)	(98.4, 100.0)**
	Non-amplified	2	319	321	NPA	95.8 (319/333)	(92.8, 98.2)**
	Total	319	333	652	OPA	97.5 (636/652)	(96.0, 98.8)**
Reader 1	Amplified	112	2	114	PPA	100.0 (112/112)	(96.7, 100.0)*
	Non-amplified	0	105	105	NPA	98.1 (105/107)	(93.4, 99.5)*
	Total	112	107	219	OPA	99.1 (217/219)	(96.7, 99.7)*
Reader 2	Amplified	101	3	104	PPA	99.0 (101/102)	(94.7, 99.8)*
	Non-amplified	1	111	112	NPA	97.4 (111/114)	(92.5, 99.1)*
	Total	102	114	216	OPA	98.1 (212/216)	(95.3, 99.3)*
Reader 3	Amplified	104	9	113	PPA	99.0 (104/105)	(94.8, 99.8)*
	Non-amplified	1	103	104	NPA	92.0 (103/112)	(85.4, 95.7)*
	Total	105	112	217	OPA	95.4 (207/217)	(91.7, 97.5)*

Note: Only observations with non-missing evaluation results were included in this analysis.

IA image analysis, PPA positive percent agreement, NPA negative percent agreement, OPA overall percent agreement, CI confidence interval.

^aThe confidence intervals were calculated using the percentile bootstrap method (**) or the Wilson Score method (*).

Image analysis slide level overrides

As part of the IA workflow, Reader Pathologists are allowed to override the HER2/Chr17 ratio calculated by the uPath HER2 DISH IA and enter a manual HER2/Chr17 ratio in instances where they disagree with the algorithm generated HER2/Chr17 ratio (slide-level override) as a quality check on the performance of the algorithm. To understand the disagreement between the pathologists and the uPath HER2 DISH IA, the frequency of IA slide-level overrides was calculated for each Reader Pathologist as the percentage of total cases evaluated by that Reader Pathologist using IA for which a slide-level override occurred (Table 5). A slide-level override would have occurred if the Reader Pathologist decided to ignore the IA HER2/Chr17 ratio completely and manually provided a HER2/Chr17 ratio significant enough to change the final amplification status after the HER2/Chr17 ratio was overridden. There was a single instance where Reader Pathologist 1 chose to override the IA HER2/Chr17 ratio due to disagreement with the IA results. No disagreement or slide overrides occurred for Reader Pathologists 2 and 3. In addition to the slide-level override, the uPath HER2 DISH IA allows the pathologist to delete and replace individual cells (within the ROI) selected by the algorithm (cell selection override) before algorithm computation of the HER2/Chr17 ratio. For an IA cell selection override to occur, at least one of the cells (within a single ROI) selected by IA had to be deleted and a new cell added by the Reader Pathologist. Out of a total of 693 ROIs, 472 cells (68.1%) were accepted by the Reader Pathologists (n=3) with no cells removed or added. In cases where the Reader Pathologists did perform a cell selection override, a total of 185 ROIs (26.7%) had 5 cells or less removed and replaced. Reader Pathologist 3 had the most IA cell selection overrides followed by Reader Pathologists 2 and 1. The frequency of IA cell selection overrides were summarized for each reader per ROI and provided in Supplementary Table 5.

Table 5.

IA score slide level overrides.

Description	Reader 1	Reader 2	Reader 3	Overall
	n (%)	n (%)	n (%)	n (%)
Final clinical status obtaineda	219	218	218	655
Confirm	218 (99.5)	218 (100.0)	218 (100.0)	654 (99.8)
Overrideb	1 (0.5)	0 (0.0)	0 (0.0)	1 (0.2)
Final clinical status not obtained	1	2	2	5

IA image analysis.

^aThe final clinical score from image analysis was obtained with the reader confirmation or override.

^bA slide-level override occurred if a reader decided to ignore the IA score completely and manually provided a slide score.

Intra-reader agreement for IA and scanner precision studies

As a secondary assessment of the performance of the uPath HER2 DISHIA, individual pathologist reads from the 3 reading rounds were used to evaluate intra-reader agreement of the HER2 Dual ISH IA (Table 6) and showed an overall OPA of 100% (120/120), (95% CI: 96.9, 100.0). We also assessed the performance of the uPath HER2 DISH IA between 3 different scanners, which displayed considerable agreement with a PPA and NPA rate of 98.9% (95% CI: 97.8, 100.0) and 100% (95% CI: 97.8, 100.0) respectively.

Table 6.

Agreement of IA HER2 gene status in comparison to intra-reader case-level.

		Modal result			Agreement
Round	Reader Result	Amplified	Non-amplified	Total	Measurement	% (n/N)	95% CIa
Overall	Amplified	60	0	60	PPA	100.0 (60/60)	(94.0, 100.0)
	Non-amplified	0	60	60	NPA	100.0 (60/60)	(94.0, 100.0)
	Total	60	60	120	OPA	100.0 (120/120)	(96.9, 100.0)
Round 1	Amplified	20	0	20	PPA	100.0 (20/20)	(83.9, 100.0)
	Non-amplified	0	20	20	NPA	100.0 (20/20)	(83.9, 100.0)
	Total	20	20	40	OPA	100.0 (40/40)	(91.2, 100.0)
Round 2	Amplified	20	0	20	PPA	100.0 (20/20)	(83.9, 100.0)
	Non-amplified	0	20	20	NPA	100.0 (20/20)	(83.9, 100.0)
	Total	20	20	40	OPA	100.0 (40/40)	(91.2, 100.0)
Round 3	Amplified	20	0	20	PPA	100.0 (20/20)	(83.9, 100.0)
	Non-amplified	0	20	20	NPA	100.0 (20/20)	(83.9, 100.0)
	Total	20	20	40	OPA	100.0 (40/40)	(91.2, 100.0)

IA image analysis, PPA positive percent agreement, NPA negative percent agreement, OPA overall percent agreement, CI confidence interval.

^aAll the confidence intervals were calculated using the Wilson Score method.

Discussion

In the current study, we describe the development and validation of the uPath HER2 DISH IA as a tool for the computer-aided evaluation of the VENTANA HER2 Dual ISH DNA Probe Cocktail assay in determining HER2 amplification status in breast cancer. Using image analysis and a feature ranking technique, the uPath HER2 DISH IA automatically identifies and segments tumor and non-tumor cells, selects individual tumor cells for analysis, identifies HER2 gene and Chr17 copies (dots), quantifies HER2 and Chr17 copies and calculates the HER2/Chr17 ratio. Critically, the algorithm relies on pathologist selection of a region of interest within the invasive breast tumor to perform these functions. Thus, accurate performance of the HER2 Dual ISH IA is inherently dependent upon the expertise of the pathologist. Using multiple pathologists and a large clinically relevant case cohort, we found that the uPath HER2 DISH IA is non-inferior to manual scoring and displays high inter- and intra-reader reproducibility. Collectively, these data suggest that the uPath HER2 DISH IA can be used in a clinical setting to support pathologists in determining HER2 gene status.

The uPath HER2 DISH IA algorithm is intended to function as part of a complete solution for determining HER2 amplification status, working in concert with the VENTANA HER2 Dual ISH DNA Probe Cocktail assay and the BenchMark staining platform, eliminating the need for internal validation of the algorithm and staining procedure. As such, the design of the uPath HER2 DISH IA algorithm’s workflow is in accordance with the CE marked/FDA approved VENTANA HER2 Dual ISH DNA Probe Cocktail assay manual interpretation. Because this study was designed to demonstrate the non-inferiority of the uPath HER2 DISH IA algorithm to manual evaluation of the VENTANA HER2 Dual ISH DNA Probe Cocktail assay, we did not perform any studies that compare against other assays (e.g. FISH) currently being used for determining HER2 gene status. However, understanding the performance of the uPath HER2 DISH IA algorithm in relation to other approaches for determine HER2 gene status is an important aspect of future work.

Application of image analysis algorithms as tools to help pathologists in routine time-consuming tasks, such as screening for invasive cancer, grading of prostate cancer biopsies and ISH analysis for HER2 quantification is an area of active work. A recently published study of a laboratory developed (LD) IA algorithm for HER2 DISH quantification showed that a LD IA algorithm displayed a 0.98 correlation coefficient for HER2 quantification compared with pathologist MR using a cohort of 22 breast cancers.²⁴ In a study of IA for FISH, authors reported moderate agreement between automated (tile image capture) vs manual FISH as measured by Cohen’s Kappa (0.48 [95% CI: 0.23, 0.72]).¹⁶ For comparison, we observed a high overall percent agreement (OPA) rate (95.7%) of image analysis vs GT, and more importantly displayed a narrow 95% confidence interval (95% CI: 93.8, 97.5). The image analysis-manual read difference in OPA was –1.5% (95% CI: –3.1, –0.2) and the inter-reader agreement rates for the uPath HER2 Dual ISH IA had point estimates of at least 96.7%. Finally, we observed that the frequency of slide level overrides significant enough to change the HER2 amplification status scored by the IA algorithm was extremely low with only 1 slide override out of 655 cases scored by the 3 pathologists. Notably our results were achieved using a large cohort of 220 FFPE samples and 3 pathologists. Overall, these data, performed with a large cohort of samples, suggest that the uPath HER2 Dual ISH IA is robust and could be used within routine clinical practice.

Increased work efficiencies is one of the commonly used reasons proposed for implementation of IA algorithms into routine clinical practice. Although our study was not designed to investigate potential gains in work efficiency, there are reports indicating that IA may help provide efficiencies in assessment of HER2 gene status. Indeed, a recent study found that application of IA in real-world clinical practice reduced diagnostic time by 65%.²⁵ Likewise, Hossain and colleagues reported that the application of the LD IA for HER2 DISH also reduced diagnostic time, from 3 to 20 min per manual score compared with 1 min for the algorithm to quantify the HER2/Chr17 ratio after pathologist annotation.²⁴ Thus, it will be critical for future studies to address the value of the uPath HER2 DISH IA in increasing workflow efficiencies, as demonstrating reduction of diagnostic interpretation time has substantial value for both the patient and oncologist.

Future directions

Although early guidelines recommended evaluation of a minimum of 60 cells²⁶ to accurately ascertain HER2 gene amplification using ISH, more recent updates have revised that number to a minimum of 20 cells (2018 and 2013 guidelines).^8,27 Despite widespread adoption of these guidelines, there is evidence that inter-reader agreement improves when larger (60 cells) numbers of cells are evaluated and could help reduce errors in diagnostic decision-making for borderline HER2 cases.^28,29 However, the need to assess ever larger numbers of cells would require a correlative increase in work for pathologists, which, when balanced with an ever-increasing workload, may not be possible. Again, the application of an IA algorithm and automation of the many tasks required for HER2 ISH assessment would directly address these workload demands, while enabling robust and accurate assessment of larger cohorts of cells. Moreover, implementation of IA to evaluate large cohorts of tumor cells may have added benefits towards improving patient diagnosis. We hope that the availability of the uPath HER2 Dual ISH IA algorithm may enable future studies into this topic.

Although evaluation of HER2 protein expression and gene amplification are routinely used to identify patients for HER2 targeted therapy, efforts have been made to utilize data regarding HER2 amplification and expression to provide additional diagnostic insights. Data suggest that increased HER2 amplification may predict pathologic complete response in the neoadjuvant setting,30, 31, 32 as well as predict progression-free survival in metastatic patients.³³ Similarly, characterization of HER2 intratumoral heterogeneity may act as a predictor for therapeutic response.34, 35, 36 However, the reliance on manual interpretation for these studies has complicated their ability to be implemented clinically. The ability of image analysis algorithms to compute large amounts of data may provide an avenue for quantitative assessment of HER2 amplification and HER2 gene heterogeneity.37, 38, 39 The application of IA algorithms might also provide more consistent quantitative measures for determining pathologic complete response and/or intratumoral heterogeneity. As the uPath HER2 DISH IA algorithm is implemented into clinical practice, we hope that future studies will address the role of IA in the clinical workflow, as well as the ability of IA to provide deeper insights that could improve overall HER2 diagnosis.

Sources of support

All the authors are employees of Ventana Medical Systems.

Competing Interests Statement

All the authors are employees of Ventana Medical Systems. LI, JRM, TKN, CG, KZ, RV, SG and SM are shareholders in Roche Diagnostics International, Ltd.

Authors’ Contributions

CG, JRM, SG, JM, KZ, TKN, SM, RV and CC performed study concept and design, analysis and generation of data; LI and AY performed interpretation of data and prepared original manuscript, CC performed data analysis. VM, CC, and CG assisted in drafting of the manuscript. All authors have read and approved the manuscript and have contributed sufficiently to the project to be included as an author.

Declaration of interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

Supplementary material