Commercial Laboratory Testing of Excision Repair Cross-Complementation Group 1 Expression in Non-Small Cell Lung Cancer

The authors reviewed the excision repair cross-complementation group 1 (ERCC1) expression results from three different laboratories. None of the laboratories’ assays could predict resistance to platinum-based therapies in non-small cell lung cancer with a specificity of greater than 50% and only 4 of 18 tumors examined in round-robin analysis were fully concordant. The authors conclude that commercial laboratories should not offer ERCC1 testing until assays are better standardized and results are more thoroughly validated.

Abstract

Introduction.

Excision repair cross-complementation group 1 (ERCC1) expression by non-small cell lung cancer (NSCLC) has been reported to predict resistance to platinum-based therapies. On this basis, several commercial laboratories have offered ERCC1 testing to facilitate clinical decision making, but the reliability of such assays has recently been called into question.

Methods.

First, three large commercial laboratories were queried for their cumulative ERCC1 test results in NSCLC patients to compare their independent rates of ERCC1 expression. Second, identical tumor blocks from individual NSCLC patients underwent round-robin analysis to evaluate interlaboratory concordance for ERCC1 expression. Third, a retrospective review of medical records from NSCLC patients identified those who were both highly responsive and resistant to platinum-based chemotherapies. Tumor blocks from these patients were then used in a gold standard analysis to determine individual laboratory sensitivity and specificity for ERCC1 results.

Results.

Significant differences were observed in independent laboratory ERRC1 expression rates (Clarient 70% vs. Genzyme 60% vs. Third Laboratory 44%, p < .0001 for all two-way comparisons). Only 4 of 18 tumors examined in round-robin analysis were fully concordant (κ ≤ 0.222 for all two-way comparisons). In preselected platinum responsive and resistant specimens, none of these three commercially marketed laboratory assays achieved a specificity of greater than 50%.

Conclusion.

The results of commercial laboratory testing for ERCC1 are inconsistent and unreliable. Better validation and postmarketing surveillance should be mandated before tumor biomarker assays are allowed to enter the clinical arena.

Implications for Practice:

Prior reports have suggested that clinical benefit from platinum-based chemotherapy may be predicted by determining tumor expression levels of the excision repair cross-complementation group 1 (ERCC1) enzyme. On this basis, ERCC1 testing has been recognized in consensus guidelines and offered by commercial laboratories. In this study, we compared ERCC1 expression levels on identical tumor specimens, as determined by three different commercial laboratories. We also evaluated each laboratory’s ERCC1 assay for its ability to correctly predict platinum resistance or sensitivity in tumor specimens that were preselected on the basis of clinically observed platinum responsiveness. ERCC1 testing was found to be both highly discordant and uniformly unreliable at all three laboratories. We conclude that these ERCC1 assays should not be used in routine clinical practice or recommended in current practice guidelines. Our experience also demonstrates how postmarketing surveillance may help to ensure the technical reliability and clinical utility of predictive tumor biomarker assays.

Introduction

Platinum-based chemotherapies represent the first-line metastatic treatment of choice [1, 2] and the only effective adjuvant option for most patients with non-small cell lung cancer (NSCLC) [3–5]. Even so, these therapies have modest benefits that must be weighed against significant treatment-related side effects [6]. Predictive markers are much needed to allow selective treatment of the subpopulation of NSCLC patients who are most likely to benefit from platinum therapies while sparing the remainder from treatment-associated morbidity. Excision repair cross-complementation group 1 (ERCC1) is one of the critical proteins involved in the process whereby cells normally repair platinated DNA and circumvent treatment-induced cytotoxicity [7–11]. Consequently, the upregulation of ERCC1 by tumors represents a plausible mechanism for their resistance to platinum-based chemotherapies [11–13].

Numerous prior reports have described an association between the tumor expression of ERCC1 and lack of clinical benefits following platinum-based chemotherapy [14–20]. In the adjuvant setting, Olaussen et al. reported that early-stage NSCLC patients whose tumors were “positive” for high levels of ERCC1 protein expression derived no benefit from platinum-based therapies (hazard ratio for death, 1.14, p = .40), whereas ERCC1 “negative” patients realized important platinum attributable benefit (hazard ratio for death, 0.65, p = .002) [18]. In the metastatic setting, Cobo et al. prospectively randomized NSCLC patients to standard platinum-based chemotherapy (control group) versus ERCC1-directed therapy (experimental group). Patients in the experimental group, who received platinum therapy only if their tumors demonstrated low ERCC1 mRNA expression and nonplatinum chemotherapy otherwise, experienced superior response rates (51% vs. 39%, p = .02) [15].

Based on such reports suggesting clinical utility of ERCC1 testing as well as its recognition in current National Comprehensive Cancer Network guidelines [21], several commercial laboratories have offered ERCC1 testing to assist medical oncologists and their patients when deciding whether to administer platinum-based chemotherapy regimens. However, ERCC1 testing methodology and criteria for high and low results are nonstandardized and may vary considerably between laboratories. In addition, technical problems with ERCC1 assays have become apparent and prior results suggesting clinical utility have not been reproducible [22]. This study was performed to evaluate ERCC1 testing offered by three large commercial laboratories.

Methods

Participating Commercial Laboratories

Clarient, Inc. (Aliso, CA, http://www.clarientinc.com), Genzyme Corporation (Cambridge, MA, http://www.genzyme.com), and “Third Laboratory” (requesting anonymity after learning of study results) agreed to participate in the conduct of this study and confirmed the accuracy of this report. Each of these laboratories offered ERCC1 testing to facilitate NSCLC patient management, and all three assays had been used by physicians at Winthrop-University Hospital (Mineola, NY) prior to the initiation of this study. All three laboratories agreed to perform ERCC1 testing without charge when requested specifically for the purposes of this study. The design and conduct of this protocol were approved by the Winthrop-University Hospital Institutional Review Board as well as by each of the participating commercial laboratories.

Study Design

ERCC1 testing was performed on tumor specimens from three separate cohorts of NSCLC patients. First, we compared the prevalence of “high” versus “low” ERCC1 expression in NSCLC as reported by each independent laboratory. Second, we chose 18 patients treated at Winthrop-University Hospital whose tumors had already been tested for ERRC1 expression at Clarient and arranged for retesting of their same tumor specimens at Genzyme and Third Laboratory.

Finally, we retrospectively reviewed the charts of more than 300 NSCLC patients treated at Winthrop-University Hospital to identify 12 who were “platinum responders” and another 12 who were “platinum nonresponders” to platinum-based chemotherapy administered as first-line treatment for metastatic disease. Platinum responders were required to have achieved a partial response, whereas nonresponders were required to have had progressive disease while receiving systemic platinum-based chemotherapy using either World Health Organization (WHO) [23] or Response Evaluation Criteria in Solid Tumors (RECIST) [24] criteria. In addition, charts were reviewed to confirm the impression of clinical benefit in responders and worsening in nonresponders. Slides were cut from a single pretreatment tumor block for each of these 24 tumors, divided equally, and sent to each of the three commercial laboratories for their independent ERCC1 testing. Specimens were deidentified and coded so that no laboratory knew tumor response status or any other tumor- or patient-related characteristics before reporting its ERCC1 assay results.

ERCC1 Testing Methodologies

Each of the three laboratories performed ERCC1 testing according to the same protocol and standards as incorporated in its own commercially offered test. Clarient and Genzyme determined ERCC1 protein expression by immunohistochemical (IHC) analysis using the same antibody reagent, designated 8F1, and protocol as reported by Olaussen et al. [18, 25] The proportion (expressed as a percentage) and the intensity (0, 1+, 2+, or 3+) of IHC-positive staining tumor nuclei were determined in identical fashion by both Clarient and Genzyme. However, each laboratory had developed its own criteria for reporting a final test result as “positive” or “negative” for ERCC1 expression. For Genzyme, a positive result required 3+ staining in at least 10% of nuclei. A positive result for Clarient required 2+ or 3+ staining in at least 50% of examined tumor nuclei. Importantly, after recognizing a change in the performance of newer 8F1 antibody lots, Clarient chose to modify its commercial assay by retitering the 8F1 antibody from 1:200 to 1:8000. These modified assay conditions took effect commercially in November 2009 and were applied to only the third cohort of patients in this study.

Third Laboratory determined ERCC1 mRNA gene expression by quantitative real-time polymerase chain reaction (RT-PCR) assay according to a proprietary procedure [26a] as previously reported by Lord et al. [26b] ERCC1 RNA levels were reported as the ratio of ERCC1 gene transcripts to β-actin reference gene transcripts. Ratios above 1:7 were reported as “positive” for ERCC1 expression and lower ratios were reported as “negative.”

We do recognize that because of posttranscriptional and posttranslational regulatory mechanisms, the accurate measurement of ERCC1 mRNA and protein expression for ERCC1 need not be concordant. However, all of these ERCC1 assays ultimately report a clinical result intended to reflect platinum sensitivity or resistance. We chose to compare all three laboratory results on the basis of this clinical predictive outcome measure.

Statistical Analysis

For cohort 1, involving separate patient populations tested at each of the three participating laboratories, descriptive statistics were analyzed by proportion. A two-sample test of proportions was used to compare these independent ERCC1 expression rates. For cohort 2, Cochran’s Q test was used to determine concordance for ERCC1 results in identical patients tested at all three laboratories [27]. McNemer’s test of paired proportions was applied to determine discordance between laboratories. After applying Bonferroni correction for multiple comparisons, none of our results changed from being statistically significant to nonsignificant using p < .05 as a measure of statistical significance. κ statistics were calculated to evaluate interlaboratory concordance. For cohort 3, clinically determined chemotherapy response provided a “gold standard” against which each laboratory’s results were compared using McNemer’s test. Because we were rigorous in our definition of platinum-sensitive and -resistant tumor specimens, our cohort 3 sample size was limited and no a priori power analysis was performed. As such, this gold standard analysis should be considered exploratory. Sensitivity, specificity, positive predictive value, and negative predictive value were calculated using exact binomial proportions [28]. Calculations were performed using SAS 9.3 (SAS Institute, Cary, NC, http://www/sas/com) and Stata/SE 10.0 for Windows (StataCorp, College Station, TX, http://www.stata.com).

Results

Aggregate ERCC1 Expression at Three Different Laboratories

Considerable variation in the prevalence of ERCC1 expression was noted at Clarient (70%, n = 1,083), Genzyme (60%, n = 1,070), and Third Laboratory (44%, n = 2,320). These differences in aggregate ERCC1 expression rates, as shown in Table 1, were highly significant (p < .0001 for all two-way comparisons) and motivated us to compare ERCC1 expression results that these three laboratories found on identical tumor specimens.

Table 1.

Commercial laboratory testing of ERCC1 in non-small cell lung cancer

ERCC1 Expression in Identical Tumor Blocks at Three Different Laboratories

Tumors from 18 NSCLC patients were tested for ERCC1 expression at all three commercial laboratories. Results are delineated in Table 1 and interlaboratory concordance is depicted in Figure 1. As shown, all three laboratories reported the same “positive” or “negative” result in only 4 of these 18 specimens. Clarient and Genzyme, the two laboratories using identical IHC testing reagents and methodology, reported concordant results in 11 of the 18 tumors (61%) tested. All 7 of the discordant cases were reported as positive by Clarient and negative by Genzyme. The results reported by these two laboratories were shown to correlate poorly (κ: 0.222; 95% CI: 0.017–0.55) and the difference between ERCC1 expression reported by Clarient (89%) and Genzyme (50%) on identical tumor blocks was highly significant (p = .015). Discordance could not be attributed to the different intensity/percentage staining criteria used by these two laboratories in determining ERCC1 expression. This was determined by repeating comparisons after applying Clarient criteria to Genzyme specimens and alternatively after applying Genzyme criteria to Clarient specimens. Third Laboratory’s ERCC1 results were concordant with those of Genzyme in only 39% (κ: –0.222; 95% CI: −0.646–0.202) and with Clarient in only 44% (κ: 0.117; 95% CI: −0.053–0.289) of cases.

Figure 1.

Laboratory concordance for excision repair cross-complementation group 1 (ERCC1) expression results. Venn diagram with overlap indicating concordant ERCC1 results. As depicted, all three laboratories were concordant for “positive” or “negative” ERCC1 result in only 4 of 18 identical tumor blocks examined in round-robin analysis.

ERCC1 Expression in Selected Platinum Responders and Nonresponders Tested at Three Laboratories

Specimens from 2 of the 12 selected platinum nonresponders and 1 of the 12 responders were deemed inadequate for ERCC1 determination by all three laboratories. Patient and tumor characteristics for each of the remaining 11 responders and 10 nonresponders are shown in Table 2. Responders and nonresponders were similar in age, gender, and the carboplatin-doublet chemotherapy regimens they received. Histology was adenocarcinoma in all patients, except for 2 nonresponders with squamous cell carcinoma. Per selection criteria for this study, all responders had partial response defined by WHO (n = 2), RECIST (n = 2), or both (n = 7) criteria, whereas all nonresponders had progression of disease defined by both WHO and RECIST criteria, except for 1 patient who had just 15% increase in maximum tumor diameter (RECIST criteria), but 42% increase in product of bidimensional target lesion measurements (WHO criteria) [24, 29]. Not surprisingly, responders received more cycles of carboplatin than nonresponders (median, 5 vs. 2.5 cycles, p = .003) and lived longer (median 25 vs. 9 months, p = .03). Only 3 nonresponders survived longer than 9 months, and all achieved major objective treatment response to second-line nonplatinum therapies.

Table 2.

Characteristics of platinum responders and nonresponders

The ability of commercial laboratory ERCC1 assays to correctly identify our platinum-responsive and -nonresponsive patients is shown in Figure 2 and Table 3. Results were reported by Clarient for 21 tumors and by Genzyme for 20, whereas Third Laboratory was able to provide results for only 8, reporting “insufficient material” in the remaining 13 cases. This was somewhat unexpected because one of the perceived advantages of Third Laboratory’s RT-PCR assay was that it might provide an enhanced ability to derive results from smaller specimens not amenable to IHC testing.

Figure 2.

Accuracy of excision repair cross-complementation group 1 results in predefined platinum responders and nonresponders.

Table 3.

Sensitivity, specificity, and positive and negative predictive values for excision repair cross-complementation group 1 predicting platinum resistance at three commercial laboratories

Initially, we were also surprised to find that Clarient, the laboratory that had previously been observed to report the highest rate of ERCC1 expression (Table 1), only reported one high ERCC1 expression result in this entire cohort of patients. Only after we queried Clarient about these unexpected results did we learn that its commercial assay had been deliberately modified to decrease the prevalence of high ERCC1 results, with that change having been implemented and applied between the second and third cohorts of this study (as above). Consequently, Clarient’s assay reported low ERCC1 results that correctly predicted response in all 11 platinum responders, but also incorrectly predicted platinum response in 9 of 10 nonresponders. Genzyme reported low ERCC1 results in 8 of 11 responders, but also in 5 of 9 nonresponders. Third Laboratory was unable to perform its RT-PCR-based mRNA assay on the majority of specimens, reporting low ERCC1 results in 3 of 4 responders and 2 of 4 nonresponders. Disappointingly, none of the three commercial assays reported a significantly higher frequency of low ERCC1 results in responders as compared with nonresponders in this selected cohort. The sensitivity, specificity, positive predictive value, and negative predictive value for each laboratory are shown in Table 3.

Discussion

Having entered an era in which cancer treatments are increasingly predicated on the results of tumor biomarker assays, the reliability of such assays is of the utmost importance. Our data demonstrate that ERCC1 testing, as performed in each of these three laboratories, is highly unreliable. Significant interlaboratory discrepancies in ERCC1 expression rates were confirmed by retesting specimens from the same tumor blocks at all three laboratories. Given that the test is reported as a simple dichotomous result (positive or negative), the degree of discordance that we observed between laboratories was both disappointing and alarming. Our results demonstrated that in 78% of cases, an ERCC1 result reported by one of these three laboratories would have been different if the test had been performed at one of the other two laboratories. In other words, if ERCC1 testing were used to decide whether to administer or withhold platinum-based chemotherapy, then that decision would most often be determined by which laboratory the test was sent to rather than any true difference in the biology of the tumor being tested. In the case of Clarient, the treatment decision would have likely been different if the test had been sent before or after their ERCC1 assay had been modified.

None of the ERCC1 assays that we studied could reliably distinguish between patients carefully selected as platinum responders and nonresponders. Because all patients were treated with platinum-doublet therapies, it is conceivable that responders who were determined to have high ERCC1 expression may have been truly resistant to the platinum therapy, but sensitive to the other coadministered cytotoxic agent. Nonresponder patients were, however, selected on the basis of being highly resistant to all administered agents in the platinum regimen, making their classification as “platinum nonresponders” more straightforward. Nevertheless, even in these refractory patients, none of the three commercially available assays correctly predicted refractory disease (high ERCC1) in more than half of tested specimens. In other words, none of the assays that we evaluated could predict platinum resistance on the basis of ERCC1 expression with a specificity of greater than 50%.

Recently, Friboulet et al. reported that none of the 16 currently available 8F1 antibody lots could reliably identify the ERCC1 isoform responsible for nucleotide excision repair and platinum resistance [22]. ERCC1 expression measured by these newer lots of 8F1 did not provide predictive utility in a newly studied cohort of patients receiving adjuvant cisplatin-based chemotherapy or in a reassessment of the original International Adjuvant Lung Trial cohort in which 8F1 had been previously validated. Importantly, when ERCC1 expression was measured with currently available 8F1 antibody lots, results were discordant with those previously observed with the original, and no longer available, 8F1 antibody. These authors suggest that the inadequacy of ERCC1 assays may reflect a change in the performance of currently available 8F1 antibody, but problems with ERCC1 measurement by 8F1 have been apparent since at least 2007 [30–32].

Prior reports have raised several other potential explanations as to why ERCC1 testing might not provide a reliable means of selecting patients for platinum or nonplatinum therapies. First, the very notion that measuring the expression of a single protein could adequately assess the complex process of DNA repair has been questioned [7, 32]. Second, low expression of ERCC1 has been shown to be associated with other genomic alterations, including those that affect sensitivity to nonplatinum chemotherapies [33, 34]. Third, others have previously noted that thresholds for ERCC1 “expression” had been arbitrarily assigned and inadequately validated [35]. Fourth, several prior studies had been unable to confirm the predictive utility of ERCC1 expression in platinum-treated lung cancer patients [31, 36]. Fifth, the measurement of ERCC1 protein by IHC and mRNA by RT-PCR has shown inconsistent correlation [37]. Sixth, the 8F1 antibody, most frequently used to measure ERCC1 protein levels, is unable to differentiate between normal and ERCC1-deficient cell lines, possibly due to excessive background cytoplasmic staining by 8F1 [30, 38, 39].

Based on such considerations, two consensus panels have appropriately recommended against the use of ERCC1 testing in routine clinical practice [2, 40]. Even the authors of studies previously reporting possible clinical utility of ERCC1 testing have responded to stated concerns by declaring that the test is still “not applicable for standard use in the everyday clinic” [41]. Yet, current National Comprehensive Cancer Network guidelines state that “Multiple translational investigations have provided evidence for the predictive use of ERCC-1 levels to assess the efficacy of platinum-based chemotherapies in NSCLC” [21] and the test remains commercially available.

Conclusion

This report provides direct evidence that currently marketed ERCC1 assays are unreliable. Initially perceived discrepancies between individual laboratory rates of ERCC1 expression suggested that some of these results were suspect. Discordance in laboratory round-robin analysis demonstrated that most were inconsistent. Sensitivity and specificity analysis among defined platinum responders and nonresponders confirmed that all were inadequate. We conclude that commercial laboratories should not offer ERCC1 testing until assays are better standardized and results are more thoroughly validated.

Two important lessons may be gleaned from the oncology community’s recent experience with ERCC1 testing. First, the process by which tumor biomarkers are validated and allowed to enter routine clinical practice is currently inadequate. Guideline committees, thought leaders, and clinicians should insist on more robust and consistent data before adopting tools that may alter patient care. Second, even after such tumor biomarker assays are incorporated into clinical practice, postmarketing scrutiny remains essential. This report suggests that such scrutiny may be achieved by simply demanding consistency of same-specimen results as well as by more ambitious efforts to develop gold standard specimens to ensure reliability of laboratory results.

Author Contributions

Conception/design: Jeffrey G. Schneider, Nosha Farhadfar, Abirami Sivapiragasam, Matthew Geller, Elena Selbs

Provision of study material or patients: Jeffrey G. Schneider, Nosha Farhadfar, Abirami Sivapiragasam, Matthew Geller, Elena Selbs

Collection and/or assembly of data: Jeffrey G. Schneider, Nosha Farhadfar, Abirami Sivapiragasam, Matthew Geller

Data analysis and interpretation: Jeffrey G. Schneider, Nosha Farhadfar, Abirami Sivapiragasam, Shahidul Islam

Manuscript writing: Jeffrey G. Schneider, Nosha Farhadfar, Abirami Sivapiragasam, Shahidul Islam, Elena Selbs

Final approval of manuscript: Jeffrey G. Schneider, Nosha Farhadfar, Abirami Sivapiragasam, Matthew Geller, Shahidul Islam, Elena Selbs

Disclosures

The authors indicated no financial relationships.