Clinical Validation of KRAS, BRAF, and EGFR Mutation Detection Using Next-Generation Sequencing

Abstract

Objectives

To validate next-generation sequencing (NGS) technology for clinical diagnosis and to determine appropriate read depth.

Methods

We validated the KRAS, BRAF, and EGFR genes within the Ion AmpliSeq Cancer Hotspot Panel using the Ion Torrent Personal Genome Machine (Life Technologies, Carlsbad, CA).

Results

We developed a statistical model to determine the read depth needed for a given percent tumor cellularity and number of functional genomes. Bottlenecking can result from too few input genomes. By using 16 formalin-fixed, paraffin-embedded (FFPE) cancer-free specimens and 118 cancer specimens with known mutation status, we validated the six traditional analytic performance characteristics recommended by the Next-Generation Sequencing: Standardization of Clinical Testing Working Group. Baseline noise is consistent with spontaneous and FFPE-induced C:G→T:A deamination mutations.

Conclusions

Redundant bioinformatic pipelines are essential, since a single analysis pipeline gave false-negative and false-positive results. NGS is sufficiently robust for the clinical detection of gene mutations, with attention to potential artifacts.

Next-generation sequencing (NGS), first launched as a commercial platform by Jonathan Rothberg's team in 2005, is revolutionizing biomedical research and the clinical practice of medicine, much like the invention of polymerase chain reaction (PCR) did two decades earlier.^1-3 Since its inception, NGS has undergone major improvements in read length, accuracy, and depth of coverage, in addition to remarkable reductions in cost, vastly exceeding Moore's law for semiconductors⁴ (http://www.genome.gov/sequencingcosts/, last accessed May 1, 2013). There are many NGS instruments currently in use, but among the most popular are the Roche 454 Sequencer and GS Junior (high and low throughput, respectively) (Nutley, NJ), the Illumina HiSeq and MiSeq (San Diego, CA), and the Ion Torrent Proton and Personal Genome Machine (PGM) (Life Technologies, Carlsbad, CA). The work we describe herein was performed on the Ion Torrent PGM, which is based on detecting the hydrogen ion that is released along with the pyrophosphate when a DNA polymerase adds a dNTP to a nascent strand of elongating DNA.⁵

In the field of cancer, NGS has substantially affected cancer predisposition gene discovery,^6-8 allowing comprehensive discovery of “all” driver gene mutations that cause a particular malignancy.^9-13 For patients suspected of harboring a germline predisposition to a particular type of cancer, panels can be constructed that contain all genes known to predispose to that cancer.¹⁴ NGS will ultimately play a major role in personalized chemotherapy and targeted therapy of patients. These will include gene mutations that predict response, and while relatively few paired gene-chemosensitivity associations are currently known,^15-18 this will be especially important if the chemosensitivity extends to mutations in all gene/pathway family members.^19,20 NGS-based chemoprediction will also include genes whose mutations predict lack of response and allow us to avoid a particular drug in which patients will only be subjected to unnecessary side effects, if treated.^21,22

For patients with cancer, NGS may be the basis of minimal residual disease (MRD) testing that may allow the clinical oncologist to avoid treating patients who test negative because they have been cured by surgery. It may also allow us to treat patients with MRD using second-line therapy more quickly based on residual tumor molecules, rather than waiting for radiographic progression.²³ NGS may also be the tool of choice for the early detection of cancer, which may likely be accomplished using a combination approach measuring critical oncogene activating mutations,²⁴ tumor suppressor gene deletions that create novel DNA-DNA junctions,²⁵ and chromosome arm imbalances.²⁶ Critical to implementing early detection will be detection in peripheral blood, a feat recently accomplished in a handful of patients with late-stage cancer.²⁶

While NGS holds tremendous promise for clinical patient testing, it is a new tool, and history tells us that, as such, it should be implemented cautiously.²⁷ The combination of inherent biologic variability and new technology has yielded surprising results in the past— for example, nearly diagnosing contaminating eosin dye as IGH monoclonality,²⁸ interpreting a single-nucleotide polymorphism (SNP) with no functional consequence as a factor V Leiden mutation,²⁹ and missing gene mutations because of an SNP underlying a primer binding site, resulting in selective amplification of only the wild-type allele.^30,31

Ion Torrent and Illumina both recently released nearly identical panels containing around 50 cancer driver gene targets, designated AmpliSeq and TruSeq respectively. Rather than attempting to validate all 50 genes in the AmpliSeq panel simultaneously, we opted to validate single-nucleotide variations in three clinically actionable genes—KRAS, BRAF, and EGFR—for which we had many previously characterized positive and negative control samples. We demonstrate the validation process for the six analytic performance characteristics, including accuracy, precision, analytic sensitivity, analytic specificity, reportable ranges, and reference ranges as recommended by the Next-Generation Sequencing: Standardization of Clinical Testing Working Group.³² We demonstrate that surprisingly high depths of coverage are required depending on the number of input genomes and percentage of malignant cells within the tumor sample. We demonstrate how to avoid false positives and false negatives by employing redundant bioinformatic pipelines.

Materials and Methods

Materials

Under approval by the institutional review board, the validation specimens consisted of seven cell lines (SW-1573, HCT-116, RKO, NCI-1975, NCI-H1650, OVCAR-8, and MOLT-4); four peripheral blood specimens from normal donors; 16 formalin-fixed, paraffin embedded (FFPE) lymph node tissues from patients with no known underlying malignancy; and 118 FFPE specimens with neoplastic tissues (lung cancers, colon cancers, malignant melanomas, papillary thyroid cancers, and other malignancies). Most of the genes tested within these 118 FFPE samples were wild type. Identity of cell lines was confirmed by microsatellite analysis using the AmpFlSTR Profiler kit (Applied Biosystems, Foster City, CA).³³ The tumor tissues were enriched by manual dissection of the targeted areas identified by pathologists as described previously.³⁴ FFPE tissues from three to 10 unstained, 10-μm-thick sections were macrodissected using Pinpoint reagents according to the manufacturer's protocol (ZymoResearch, Orange, CA). DNA was purified using the QIAmp DNA kit (Qiagen, Valencia, CA). All except two specimens contained more than 10% tumor cells. DNA was isolated from the peripheral blood or cell lines using the QIAamp DNA Blood Mini Kit (Qiagen).³⁵ Concentration of DNA was determined by a Qubit 2.0 Fluorometer (Life Technologies).

Validation Specimen Characterization

Among the 118 cancer specimens, 110 have been tested for KRAS by pyrosequencing, 96 for BRAF by pyrosequencing, and 47 for EGFR by Sanger sequencing. Samples were previously tested for KRAS and BRAF by pyrosequencing and for EGFR by Sanger sequencing. They include 31 KRAS gene mutations in 30 of 110 specimens, including three G12A, 10 G12C, three G12D, two G12R, three G12S, six G12V, and four G13D. One sample contained a double mutation (G12C and G12D). BRAF gene mutations were present in 19 of 96 specimens examined, including 17 V600E, one V600K, and one V600R. Among the 47 specimens examined by Sanger sequencing for EGFR, 17 specimens were positive for EGFR gene point mutations, including two cases with G719A, one with A722V, one with G724S, nine with L858R, and four with double mutations (one with G719C and S768I, one with V834L and L858R, and two with T790M and L858R). The codon 834 within exon 21 was not covered by the NGS amplicons. This mutation, therefore, was disregarded when the Sanger results were compared for the analytic performance characteristics of the NGS assay. Pyrosequencing of the KRAS and BRAF genes confirmed a G12C homozygous/hemizygous KRAS mutation of the SW-1573 cell line, a G13D KRAS heterozygous mutation of the HCT-116 cell line, a heterozygous V600E BRAF mutation of the RKO cell line, and absence of KRAS and BRAF gene mutation of the other four control cell lines (NCI-1975, NCI-H1650, OVCAR-8, and MOLT-4). Sanger sequencing confirmed T790M and L858R mutations within the NCI-1975 cell line.

Pyrosequencing (KRAS and BRAF)

Pyrosequencing for detecting codon 12 and 13 mutations of the KRAS gene was performed as described previously.^34,36 Pyrosequencing for detecting codon 600 mutations of the BRAF gene was conducted using forward primer 5’-GAAGACCTCACAGTAAAAATAG-3’ and biotinylated reverse primer 5’-ATAGCCTCAATTCTTACCATCC-3’ for PCR and primer 5’-GACCTCACAGTAAAAATAGGTGA- TTTTG-3’ for sequencing. PCR conditions were 95°C for 15 minutes; 42 cycles of 95°C for 20 seconds, 53°C for 30 seconds, and 72°C for 20 seconds; and 72°C for 5 minutes.

The nucleotide dispensation order for pyrosequencing was 5′-GTAGCTAGCTATCAGCATCGACTCTCGATGAGTG-3′. The limit of detection by pyrosequencing was 5% mutant alleles. Complex pyrosequencing results were resolved with the aid of Pyromaker software (http://pyro-maker.pathology.jhmi.edu/; last accessed April 30, 2013).³⁶

Sanger Sequencing (EGFR)

EGFR mutations within exons 18 to 21 (codons 688-875) were examined by Sanger sequencing. Primer pairs were 5’-CTGAGGTGACCCTTGTCTCTG-3’ and 5’-CCAAA- CACTCAGTGAAAC-3’ for exon 18, 5’-TGCCAGTTAA- CGTCTTCCTT-3’ and 5’-CAGGGTCTAGAGCAGAGCAG-3’ for exon 19, 5’-CATTCATGCGTCTTCACCTG-3’ and 5’-TTATCTCCCCTCCCCGTATC-3’ for exon 20, and 5’-TGATCTGTCCCTCACAGCAG-3’ and 5’-GGCTGACCTAAAGCCACCTC-3’ for exon 21. All primers were located within the introns and tailed with M13 sequences. PCR conditions were 95°C for 5 minutes; 48 cycles of 95°C for 30 seconds, 55°C for 30 seconds, and 72°C for 30 seconds; and 72°C for 10 minutes. PCR products were purified using USB ExoSapit (GE Healthcare, Uppsala, Sweden) and cycle sequenced using the BigDye Terminator version 3.1 cycle sequencing kit (Applied Biosystems) according to the manufacturer's protocol and resolved on an ABI 3500xL sequencer (Applied Biosystems). Sequences were analyzed using Sequencher software (Gene Codes Corp, Ann Arbor, MI). In general, the limit of detection of clinical assays by Sanger sequencing is approximately 20% to 25% mutant alleles, although a lower percent mutant allele rarely may be detected depending on the context of the targeted sequences.

NGS Platform

The NGS was conducted by using the Ion AmpliSeq Cancer Hotspot Panel (v2) for targeted multigene amplification (207 amplicons covering approximately 2,800 Catalog of Somatic Mutations in Cancer [COSMIC, http://cancer.sanger.ac.uk/cancergenome/projects/cosmic/; last accessed May 1, 2013]) mutations from 50 oncogenes and tumor suppressor genes), Ion AmpliSeq Library Kit 2.0 for library preparation, Ion OneTouch 200 Template Kit v2 DL and Ion OneTouch ES Instrument for emulsion PCR and enrichment, Ion PGM 200 Sequencing Kit, Ion 318 Chips, and the PGM sequencing platform for massive parallel sequencing (Life Technologies), as recommended by the manufacturers’ protocols without modification. The DNA input for targeted multigene PCR was 10 to 30 ng. Eight specimens were barcoded using Ion Xpress Barcode Adapters (Life Technologies), pooled, and run on a single Ion 318 chip. We attempted a parallel validation using the TruSeq panel but were unable to complete it due to the requirement of 250 ng of input DNA, which we were unable to obtain for most solid tumors.

Analysis Pipeline and Annotation of Mutation

Sequencing data of the three targeted genes (KRAS, BRAF, and EGFR) were analyzed using Torrent Suite (version 3.2.0; Life Technologies). All other genes except KRAS, BRAF, and EGFR were masked for this analysis in Torrent Variant Caller using specific browser extensible data files and in Ion Reporter using filters. Mutations were identified and annotated through Torrent Variant Caller (version 3.2.45211) and Ion Reporter (version 1.2) Figure 1. Additional analysis using visual inspection of the binary sequence alignment/map file on the Broad Institute's (Boston, MA) Integrative Genomics Viewer (IGV) was implemented (early in the validation process) after we found that Torrent Variant Caller and Ion Reporter missed a case with the most common EGFR point mutation in a lung cancer (L858R). IGV was used to determine the coverage of each specific exon and to display the number of reads of the variants. The Torrent Variant Caller plugin called the nucleotide variants along with their Hg19 genome position. Ion Reporter identified nucleotide variants with integrated annotations and provided a COSMIC ID if available. Annotation was confirmed by direct inspection of the nucleotide sequences and amino acid sequences. The annotations from these three pathways were compared.

Figure 1.

Analysis pipeline. Bioinformatics for point mutation detection using three parallel analysis tools: Torrent Variant Caller, Ion Reporter (Life Technologies, Carlsbad, CA), and Integrative Genomics Viewer (IGV; Broad Institute, Boston, MA). BAM, binary sequence alignment/map; COSMIC, Catalog of Somatic Mutations in Cancer.

Modeling Power for NGS Experiments to Determine Appropriate Read Depth

To estimate power, we modeled the NGS experiment as a two-stage process with random variation introduced at each stage. In the first step, we use n_g to denote the number of effective genomes drawn from the FFPE sample, assuming that the mutant allele count at this point follows a binomial distribution that depends on the actual mutant allele frequency p. Note that if all cancer cells are heterozygous at the position being queried, then p will be half the cellularity of the specimen. In the second step, we use n_d to denote the sequencing read depth, which also follows a binomial distribution, this one conditional on the outcome of the first step. Combining the two and propagating the errors from the binomial distribution gives the final model, which is well approximated by a normal distribution with mean E[x] = p and a two-term variance $[\mathrm{var}\left(E\left[x\right]\right)=n_d^2(p(1-p)∕n_g+p(1-p)∕n_d)]$. For several combinations of the parameters, the model was compared with the much more computationally intensive Monte Carlo simulations based on permutations of the data and found to agree very closely (results not shown).

Results

Limit of Detection of the NGS Platform

To determine where to set the limit of detection, we first quantified the “noise” level in deidentified negative control samples. Among the 16 cancer-free lymph node FFPE samples, we graphed the mean plus 3 SD for all common KRAS (codons 12 and 13), BRAF (V600E), and EGFR (T790M and L858R) gene point mutations Figure 2A. The highest value was 1.2% for the T790M EGFR resistance mutation, and all others were below this level. We performed a similar analysis on peripheral blood samples representing high-quality DNA Figure 2C. The noise level from the FFPE samples was approximately twice that of the peripheral blood samples (mean, 0.072% vs 0.038%, respectively). Because the noisy bases were qualitatively the same between peripheral blood and the first FFPE library, we wanted to further test how consistent these effects would be. A second independent FFPE library preparation and sequencing of the same samples produced a qualitatively identical pattern Figure 2B and an overall mean of 0.086% and the highest value (mean plus 3 SD) of 1.3% at the same position, T790M (see Discussion). Serial dilution of the HCT-116, RKO, and NCI-1975 cell lines at levels of 20%, 10%, 5%, 2%, and 1% cell mixtures demonstrated that the NGS platform could consistently detect the G13D, V600E, T790M, and L858R mutations at a 1% level of mutant alleles (not shown). A limit of detection of 2% was chosen as a conservative initial value above the maximum measured baseline noise (Figure 2B; T790M, mean plus 3 SD of 1.3%).

Figure 2.

Background noise. For each of the common mutations within KRAS (codons 12 and 13), BRAF (V600E), and EGFR (T790M and L858R) genes among the 16 formalin-fixed, paraffin-embedded (FFPE) lymph node control first library (A), a second FFPE library (B), and peripheral blood (C) specimens, the mean plus 3 SD is plotted. Note that the patterns of quiet bases and noisy bases in A and B are qualitatively similar and that background signal in FFPE is substantially higher than in the peripheral blood. Down arrows designate consistently quiet mutations. Asterisks indicate particularly noisy mutations that have arisen due to a C:G→T:A mutation (resulting in either a C→T or a G→A on the sense strand).

Depth of Coverage Required to Confidently Detect Mutations in Malignant Cells Within a Tumor

Tumors contain a variety of nonmalignant cells, including stromal cells, lymphocytes, and blood vessel cells. Adequate sampling of tumor DNA requires that a minimum number of tumor genomes are pipetted into the assay and that they are studied at a sufficient depth of coverage. For example, if we imagine a tumor that is only 20% malignant cells and that the malignant cells contain a heterozygous mutation, then the mutant alleles are present at a 10% level. If we were to then sequence this tumor DNA to only a depth of coverage of 10 sequences, this would be clearly insufficient, since the mutant allele may or may not be represented among these 10 sequences. We modeled this problem using a binomial statistic and required a 99% probability of detecting the mutation, and we graphed the depth of coverage required as a function of the percent of malignant cells within the tumor and the number of tumor genomes tested Figure 3. A tumor with 10% malignant cells (5% mutant alleles) and in which 1,000 genomes are analyzed (6 ng functional DNA input) would require around 400 times depth of coverage Table 1. If 20% malignant cells were present, this would likely be detected at a depth of coverage of only around 100 times. To detect 10% cancer cells at a sensitivity of 99.9%, 10,000 genomes must be used, which must be sequenced to around 500 times depth of coverage (Figure 3B). Looking at the data in a different way, analyses of tumors containing only 10% cancer cells require that a minimum of 1,000 functional genomes be studied. The FFPE process, which damages DNA largely through cross-linking, can be thought of as a bottleneck that reduces the number of functional DNA molecules that are added into the reaction. The input of 10 to 30 ng DNA may not be all functional, and this may require laboratories to qualify their stated limit of detection.

Figure 3.

Required depth of coverage. Depth of coverage required as a function of the number of genomes analyzed and the percentage of cancer cells in the sample, at a diagnostic sensitivity of 99% (A) and 99.9% (B). The effect of formalin-fixed, paraffin-embedded fixation is modeled as a decrease in the number of functional genomes analyzed. Lines that terminate (eg, arrow for 30 genomes in A) do so because the required sensitivity cannot be achieved, as the percentage of cancer cells within the tumor decreases.

Table 1.

Depth of Sequencing Required to Achieve a Probability of Detecting 99% Mutation^a

Cellsb	30 GE	100 GE	300 GE	500 GE	1,000 GE	10,000 GE	∞
5	NA	NA	NA	NA	NA	NA	10,000
10	NA	NA	10,000	10,000	397	296	286
20	NA	300	100	93	88	84	81
30	NA	79	48	46	44	43	42
40	237	39	33	32	32	31	31
50	67	28	25	25	24	24	24
60	33	22	20	19	19	19	19
70	23	17	16	16	16	16	16
80	18	14	13	13	13	13	14
90	14	12	11	11	11	11	12
100	11	10	10	9	9	9	11

GE, genome equivalents; NA, not achieved.

^aBold numbers represent commonly used cut points (see text).

^bCalculations are based on a heterozygous mutation within the cancer cell component of the tumor.

Accuracy, Analytic Sensitivity, and Analytic Specificity

All of the 118 validation specimens were examined by NGS. By pyrosequencing and Sanger sequencing, these included 31 KRAS, 19 BRAF, and 21 EGFR gene mutations in 30, 19, and 17 samples, respectively, since some samples contained two mutations (see Materials and Methods). Point mutations of the KRAS, BRAF, and EGFR genes were detected in 33, 19, and 17 of the 118 cases by NGS, respectively. Two A146T mutations and one Q61L mutation of the KRAS gene and one R108K and one A289V of the EGFR gene were identified by NGS. These mutations were confirmed by either pyrosequencing or Sanger sequencing. Mutations at these positions, however, were not covered by our conventional assays, and therefore only tested in retrospect, and were accordingly not counted as positives while validating the analytic performance characteristics of the NGS assay. One specimen with an A289V mutation and another with a R108K mutation also contained a G724S mutation and a G719A mutation, respectively, which were identified by Sanger sequencing. Nucleotide variation detection and amino acid annotation by Torrent Variant Caller, Ion Reporter, and direct visualization under IGV were all consistent except a specimen with double mutations of T790M and L858R. In this specimen, L858R was clearly present in IGV at 18% but was not detected by the Torrent Variant Caller or Ion Reporter (data not shown), possibly related to the two mononucleotide repeats nearby. We also noted false positives in Variant Caller and Ion Reporter that in IGV appeared as short reads, where all the reads were in a single direction.

Nucleotide variations detected by the assays and amino acid changes annotated by the analysis pipelines were compared with those by conventional assays. The results were all consistent with both analytic sensitivity and specificity of 100%, when only the variations within the target regions covered by both assays were compared Table 2. Genotypes of SNP rs1050171 within exon 20 of the EGFR gene determined by Sanger sequencing and NGS were also consistent Table 3. Since both pyrosequencing and NGS are quantitative assays, accuracy was also examined by comparing the mutant allele percentage of KRAS Figure 4A and BRAF Figure 4B gene mutations detected by pyrosequencing and NGS. Concordance between the two assays was remarkably high (R² correlation coefficients >0.93 and >0.97, respectively).

Table 2.

Analytic Sensitivity and Specificity

	Conventional Assays, No.
Characteristic	Mut	wt
KRAS
NGS (mut)	30	0
NGS (wt)	0	80
BRAF
NGS (mut)	19	0
NGS (wt)	0	77
EGFR
NGS (mut)	17	0
NGS (wt)	0	30

mut, mutation; NGS, next-generation sequencing; wt, wild type.

Table 3.

Number of Genotypes of SNP rs1050171 Within Exon 20 of the EGFR Gene

	Sanger Sequencing
NGS	A/A	A/G	G/G
A/A	16	0	0
A/G	0	20	0
G/G	0	0	11

NGS, next-generation sequencing.

Figure 4.

Accuracy of next-generation sequencing (NGS). Correlation of percent mutant allele of the KRAS (A) and BRAF (B) genes determined by the NGS platform (Personal Genome Machine [PGM] %) compared with the “gold-standard” pyrosequencing (Pyro %). A, y = 0.9786x – 0.0309; R² = 0.93421. B, y = 0.9819x – 2.2742; R² = 0.97221.

Precision

The NGS assay was performed in duplicate using five wild-type specimens and 11 mutation-bearing specimens (seven KRAS, two BRAF, and two EGFR mutations). When all three analysis pipelines were applied, results were consistent, including the correlation of mutant allele percentage (R² > 0.98) Figure 5A. Precision of the pyrosequencing assay has been demonstrated during the validation of KRAS and BRAF mutation detection by pyrosequencing. The mutant allele percentage was highly correlated between duplicated runs (R² > 0.99; data not shown). The genotyping and allele frequencies of SNP rs1050171 within exon 20 of the EGFR gene were also consistent within the different runs (R² > 0.99) Figure 5B.

Figure 5.

Precision between next-generation sequencing runs. Assessed by comparing the percent mutant allele (A) and allele frequencies of single-nucleotide polymorphism rs1050171 (B) from replicate runs. A, y = 1.0778x – 1.363; R² = 0.98916. B, y = 0.9946x – 0.603; R² = 0.99424.

Uniformity of Coverage

The average coverage and uniformity of coverage of the whole panel (207 amplicons from 50 genes) for each FFPE lymph node control ranged from 225 to 2,683 (mean ± SD, 1,768 ± 862) absolute read number and from 81.8% to 98.1% (94.1% ± 6.3%), respectively. The mean number of reads for each amplicon of the KRAS, BRAF, and EGFR genes was all more than 1,000, with sufficient uniformity in coverage across the amplicons (data not shown).

Reportable Range and Reference Range

The reportable range of bases sequenced was determined by direct visual inspection of amplicon reads in IGV Table 4. The demonstration of accuracy, precision, analytic sensitivity, and analytic specificity from results of validation specimens containing a variety of common mutations, along with the adequate average coverage and relative uniformity of coverage of the entire panel and each amplicon within the KRAS, BRAF, and EGFR genes, supports this reportable range. Within the reportable range, rs1050171 was the only SNP with a minor allele frequency of more than 1% in the general population according to the National Center for Biotechnology Information database.

Table 4.

Reportable Ranges

Gene	Oncogene/Tumor Suppressor	Chromosome	Exon	Starta	Stopa	Reportable Codon Range
KRAS	Oncogene	12	2	25398237	25398305	5-38
KRAS			3	25380258	25380338	39-70
KRAS			4	25378521	25378590	114-150
BRAF	Oncogene	7	11	140481395	140481504	439-474
BRAF			15	140453120	140453223	581-612
EGFR	Oncogene	7	3	55211024	55211132	93-124
EGFR			7	55221796	55221866	279-297
EGFR			15	55232962	55233053	575-602
EGFR			18	55241644	55241717	693-727
EGFR			19	55242416	55242540	729-761
EGFR			20	55248999	55249072	762-800
EGFR			20	55249109	55249187	802-823
EGFR			21	55259432	55259539	853-875

^aStart and stop of the nucleotide positions.

Discussion

We conclude that the Ion Torrent AmpliSeq panel on the PGM can be used in a Clinical Laboratory Improvement Amendments (CLIA) environment. Interestingly, cross-validation of documented mutation-positive and mutation-negative samples showed perfect concordance, since potential false positives and false negatives were eliminated through redundant analysis. Accuracy of the percent mutant allele was outstanding (R² > 0.90) compared with pyrosequencing. The limit of detection, based on “noise” levels in negative control FFPE samples, is 1.3% or less, which is superior to that of pyrosequencing. To confidently (>99%) detect a mutation present in a tumor with 10% malignant cells, at least 1,000 functional genomes must be analyzed to a depth of coverage of at least 400 times. These data clearly have implications for the recently released lung cancer testing guidelines, which recommend that laboratories employ tests “that are able to detect mutations in specimens with as little as 10% cancer cells.”³⁷ Independent validation of NGS for clinical cancer testing by other groups has been reported for the AmpliSeq panel on the Ion Torrent PGM^38-40 and other systems.^40-42 We consider multiple independent validations to be helpful for new and disruptive technologies, as discussed earlier, and some new findings are presented herein.

As with any technology, there are concerns, limitations, and unanswered questions. Because the system throughput vastly exceeds the required depth of coverage, samples are barcoded and intentionally mixed, a process that is normally avoided in clinical laboratory testing (except in the area of transfusion-transmitted disease testing). The major technical limitation of the system is initialization failures, estimated to occur approximately around 10% of the time. With regard to bioinformatics, we currently use three independent methods for analysis because false negatives and false positives occur if only one type of analysis is performed. Coverage is sufficiently uniform between amplicons to get to the required read depth but probably insufficient to use it for copy number change detection. When we challenged the system with samples containing insertions (FLT3 internal tandem duplications) or deletions (EGFR), they were not detected consistently, so we are currently detecting them using PCR and sizing by capillary electrophoresis. In this regard, improved chemistry, detection, and analysis should allow the bioinformatic mapping tools to become more data centric rather than target centric (eg, currently mapped to the human genome, Hg19).⁴³ Since a minimum number of input molecules are critical to prevent allele dropout, it may be important to quantify DNA functionally (eg, by ABI Quantifiler [Applied Biosystems]) for NGS applications to prevent a “bottleneck” phenomenon, in which the number of functional input molecules is reduced from the number determined fluorometrically. Finally, like pyrosequencing, an intrinsic limitation with ion-based sequencing is with mononucleotide repeats, and so this system would not be the tool of choice to diagnose frame shift mutations in the TGFBR2 gene or microsatellite instability in the BAT26 mononucleotide repeat microsatellite locus.

With regard to baseline noise of the system, it appears that this is primarily due to cytosine deamination. In Figure 2A, we denote these mutations with asterisks that arise from a C→T on the sense strand (T790M) or appear as G→A mutations on the sense strand (the four KRAS mutations). Since the G→A mutations on the sense strand could have arisen from a C→T mutation on the antisense strand, these mutations should be collectively designated C:G→T:A, and the five mutations are significantly increased (P = .0026, Wilcoxon rank sum test) from the other mutations in Figure 2A. This is also the case for an independent library preparation of the same 16 FFPE cases (Figure 2B; P = .00067, Wilcoxon rank sum test). These mutations are higher in the FFPE samples than in peripheral blood (Figures 2A and 2B vs Figure 2C; P = .063, Wilcoxon rank sum test). Williams and colleagues⁴⁴ made similar observations about formalin-induced C:G→T:A transitions, and others have suggested pretreatment of FFPE samples with uracil-N-glycosylase to eliminate this artifact by inactivating the uracil-intermediate containing DNA strands.⁴⁵ However, these base positions also appear to be about two times higher in peripheral blood samples (Figure 2C; P = .0013, Wilcoxon rank sum test), consistent with the observations of Hadd and colleagues.³⁸ These mutations must be either biologic (intrinsic to the sample) or an artifact of the molecular biology. Because the “mutation frequency” is remarkably higher than what we believe baseline mutation frequencies to be,⁴⁶ we hypothesize that these are most likely due to some intrinsic element in the system such as polymerase-induced errors, lack of DNA repair, or possibly spontaneous deamination from the heat associated with thermocycling.⁴⁷

Currently, the clinical utility of the NGS platform has mainly focused on multigene panels targeting mutations that predict responsiveness of therapies,^38,40,41,48 including the KRAS/BRAF mutation in colorectal cancer, EGFR mutation in lung cancer, and BRAF mutation in melanoma and other genes for a variety of ongoing clinical trials of small molecular inhibitors and monoclonal antibodies.^18,49,50 The NGS platform can be applied to the clinical diagnostic assays at different levels of complexity, including tumor-specific gene panels, small cancer panels, whole-exome sequencing, and possibly whole-genome and/or whole-transcriptome sequencing. We consider the AmpliSeq panel at the arrowhead along this spectrum. As the use of NGS expands, we consider it unlikely that laboratories will be able to maintain and quality control multiple tumor-specific panels. In the current study, we applied the traditional validation processes as well as the NGS-specific analytic parameters to validate KRAS, BRAF, and EFGR point mutations for standard clinical care of patients with lung cancer and/or colorectal cancer and demonstrated that the NGS platform is sufficiently robust for detection of these gene mutations. Validation of the other 47 genes is ongoing.

NGS may be the tool of choice for the early detection of cancer due to its ability to perform massively parallel DNA sequencing. Proof of principle for early detection comes from molecular clocking experiments that demonstrate that during the life span of a given cancer, more than half of that time is confined to the organ,^51,52 as well as the well-established concept that patients with cancer do well when organ-confined cancers are surgically resected at early stages. Methods exist to circumvent the high mutation rates inherent to NGS,^53,54 and early experiments using NGS show promise to detect driver mutations, novel DNA-DNA junctions, and chromosome arm copy number changes.^24,26 Further clinical studies are needed to demonstrate the clinical utility of the NGS platform for the early detection of cancer.

The CLIA requirement for validation of a clinical diagnostic assay includes confirmation of the analytic performance characteristics and the clinical performance characteristics.⁵⁵ Integration of an NGS platform into a clinical assay, however, is complex and requires validation processes specific to NGS technology in addition to conventional validation approaches. A Next-Generation Sequencing: Standardization of Clinical Testing Working Group was convened by the Centers for Disease Control and Prevention to establish quality management systems for the clinical application of NGS technology.³² Although the working group focuses mainly on medical genetics, in which NGS validation is more advanced,^14,56-59 the recommendations can also be applied to molecular microbiology and oncology. The working group defined the six key analytic performance characteristics (accuracy, precision, analytic sensitivity, analytic specificity, reportable range, and reference range) and also divided the validation process into three interconnected components: platform validation, test validation, and informatics validation. Briefly, platform validation is the process to establish that the NGS system can correctly identify each type of variant that the assay is designed to test, test validation confirms the ability of the NGS platform to identify disease-associated sequence variants within specific regions of the genome under investigation, and informatics or analysis pipeline validation documents the capacity of the software settings to generate accurate sequencing data and to detect variants within the targeted genomic regions.

In summary, we demonstrated validation of the KRAS, BRAF, and EGFR gene point mutation detection by using the Ion AmpliSeq Cancer Hotspot Panel on the PGM sequencing platform and combining traditional processes for conventional molecular assays with the recommendations from the Next-Generation Sequencing: Standardization of Clinical Testing Working Group specific for NGS. We conclude that the platform and kit are sufficiently robust for clinical testing, provided that one appreciates the limitations and caveats of interpretation with the system.