Immune Signature to Predict Trastuzumab Benefit: Potential and Pitfalls

To the Editor:

The article by Perez et al¹ presented a potentially exciting 14-gene algorithm that predicts benefit from trastuzumab in the adjuvant treatment of the breast cancer cohort they examined.

On careful review of the article and evaluation of the findings and method used in an independent data set from National Surgical Adjuvant Breast and Bowel Project (NSABP) trial B-31, we find problems with the study interpretation that may weaken the overall conclusion.

The 14-gene predictive model was built using gene expression data from formalin-fixed, paraffin-embedded tissues (FFPETs) from greater than 1,200 patients enrolled in the North Central Cancer Treatment Group NCCTG-N9831 trial using the whole transcriptome DASL (cDNA-Mediated, Annealing, Extension, and Ligation) platform. However, the model was not adequately cross validated, because feature selection was not performed independently in each set of the cross validation. Although the authors include this information in the article, it is known that such practice often results in overfitting of the data, leading to false-positive conclusions purely by chance, as described by Simon et al.²

To evaluate the procedure used to build the14-gene model in an independent data set, we developed models in Affymetrix gene expression data from 731 cases from NSABP B-31 trial using the same criteria for gene selection, but instead of selecting by immune-related genes, we randomly selected 14 genes that met the other criteria described by Perez et al.¹ In 10,000 tests, without any optimization of m and q cutoffs, 92% of the models had an interaction probability of less than 0.01, and 62% of the models had an interaction probability of less than 0.001. The conclusion that immune-related genes are driving the observation may not be valid because this criterion can be eliminated without effect on model performance.

Of equal importance is the lack of analytic validation of the 14 genes selected for the model. Gene expression profiling using microarray platforms with RNA extracted from FFPET has poor analytic performance that often leads to false-positive findings. On the basis of our internal data, comparing whole transcriptome expression platforms, including Affymetrix, Agilent, and DASL, with nCounter data (a gold standard for gene expression), less than 20% of genes correlate with nCounter data better than 0.7. Therefore, although microarray can be used to discover candidate genes, it is absolutely critical to analytically validate these genes using a technical platform, such as quantitative reverse transcriptase polymerase chain reaction or nCounter assays, which perform well in FFPET, before making any conclusions.

To illustrate the problem of using the original DASL microarray platform for validation, we examined the overlap of significant predictive genes between N9831 and our B-31 Affymetrix data, which also used FFPET. In B-31, 1,791 genes were predictive of trastuzumab benefit at P = .05; among them, only 136 genes (< 10%) were also significant in N9831. Similarly, less than 10% of genes in N9831 that were predictive of trastuzumab benefit were also significant in B-31. χ² tests suggested that the overlap is likely because of chance (P > .1). Furthermore, half of the genes that are predictive in both data sets showed opposing direction for expression-treatment interaction between the two studies. This indicates the intersection is not enriched for true positives.

The lack of analytic validation of the 14 genes chosen for the model seriously compromised the ability to develop a meaningful signature. In fact, in the NSABP B-31 Affymetrix data set, the 14-gene model failed to predict trastuzumab benefit (interaction P = .47; Fig 1). This does not mean that the14-gene model is wrong, because we do not know the analytic performance of the 14 selected genes in the Affymetrix platform. What we want to point out is that the 14-gene model has to be analytically validated using a proper technical platform. The complexity and small dynamic range of microarray data combined with the degraded and chemically modified nature of FFPET-isolated RNA make signature validation with microarray platforms unlikely.

Fig 1.

(A) Disease-free survival of patients in the National Surgical Adjuvant Breast and Bowel Project (NSABP) B-31 trial in the 14-gene not-enriched group according to treatment. (B) Disease-free survival of patients in the NSABP B-31 trial in the 14-gene enriched group according to treatment. ACT, doxorubicin, cyclophosphamide, and paclitaxel; ACTH, ACT plus trastuzumab.

In NSABP B-31, we first performed microarray profiling on a discovery cohort, which was half of the B-31 cohort, selected candidate predictive genes, designed a custom nCounter assay panel including candidate genes, re-examined the discovery cohort and built the eight-gene predictive algorithm and its cut points, and then examined the validation set.³ Although this model was validated in the B-31 independent cohort with nCounter data, we would not be too surprised to see our eight-gene predictive model not validated in the Perez et al¹ DASL data set on the basis of the arguments made above.

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Disclosures provided by the authors are available with this article at www.jco.org.

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Immune Signature to Predict Trastuzumab Benefit: Potential and Pitfalls

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or jco.ascopubs.org/site/ifc.

Patrick Gavin

No relationship to disclose

Nan Song

No relationship to disclose

S. Rim Kim

No relationship to disclose

Katherine L. Pogue-Geile

No relationship to disclose

Soonmyung Paik

No relationship to disclose