Precision Medicine Requires Precision Laboratories

Molecular pathologists have been champions in advancing Next Generation Sequencing (NGS) assays for use in clinical applications such as diagnosis and targeting therapy. The report by Lih et al in this issue of The Journal of Molecular Diagnostics presents the latest chapter in this story [1]. Unlike the earlier low and medium throughput technologies, the massively parallel sequencing by NGS offers the opportunity for simultaneous targeting of different genetic alterations that include single nucleotide variants (SNVs), insertions/deletions (indels), copy number variations (CNVs), and gene fusions. However, validating NGS assays remains challenging when it involves processing tissues from a variety of cancer types, different sequencing chemistries and instruments and complex methods for data analysis applicable to different types of genetic variants in multiple barcoded samples [2]. Guidelines and recommendations for NGS assay validation have been developed by a variety of organizations including the College of American Pathologists [3], the American College of Medical Genetics and Genomics [4] and the Centers for Disease Control and Prevention [5]. These general guidelines are useful to laboratories as they seek to provide the quality metrics required to validate NGS assays, but details of the study design to assure that the NGS assays are fit-for-purpose are still works in progress.

Molecularly Targeted Cancer Therapeutics

Precision medicine is the use of individualized genetic information to select drugs that are most effective and least toxic. The ability of precision medicine to improve cancer therapy requires the availability of drugs that are active against known genetic changes. Over the years, clinical trials have been painstakingly designed to prove that drugs are therapeutically effective for tumors with specific mutations. Regulatory approval of these drugs are often linked to analytically validated assays for the targeted gene or genes. To date most of these companion diagnostic assays have focused on relatively few analyte targets. However, the total number of mutations that have evidence for activity as a driver of tumor progression and that can be specifically targeted by a specific drug has grown to several thousand, shifting the precision medicine paradigm from 1-gene-1 drug to a multigene-many drugs model [6]. Multiplex testing strategies are needed to be developed to increase the number of tests that can be performed on limited biopsy samples. Furthermore, these methods need to be streamlined to provide and validated results in clinically relevant turn-around times.

Precision Medicine Enables “Genotype to Phenotype” Oncology Trials

The National Cancer Institute (NCI) refers to trials that match molecular features of a tumor to a candidate drug as “genotype to phenotype”, i.e. a particular genetic profile predicts response to a drug. These trials all require accurate and rapid detection of specific driver mutations so that the appropriate therapy can be selected and administered in a timely fashion. All require representative tissue biopsies, use complex and sophisticated NGS assays, and pipeline bioinformatics to generate reports as well as precision laboratories that have validated the assays.

In these clinical trials the NGS assays are used to assign treatment and therefore are considered integral assays and require some level of review and regulatory approval. The specifics of the study design to validate these assays and the findings about the parameters evaluated are novel and merit publication in the scientific literature to provide guidance to others. The approach to analytic validation of the targeted NGS assay used in the NCI-MPACT trial was recently reported in this journal [7]. It is clear that findings from validation of the NCI-MPACT assay were helpful to Lih et al in their approach to the NCI-Molecular Analysis for Therapy Choice (MATCH) Trial which presented an even broader challenge for the laboratories [1].

Unique Challenges of the NCI-MATCH Trial

The NCI-MATCH trial is the largest trial to date to attempt to match tumors to therapeutic agents based solely on the identification of presumed driver mutations amenable to specific drugs, the actionable mutations of interest (aMOIs) [8]. This phase 2 study shares features with other trials that NCI has designated as “genotype to phenotype”, such as ALCHEMIST (testing inoperable non-squamous, non-small cell lung cancers), Lung-MAP (testing advanced squamous cell lung cancer), and NCI-MPACT (testing solid tumors). Whereas one central laboratory could be used in the other trials, NCI-MATCH required four clinical molecular diagnostics laboratories (Frederick National Laboratory for Cancer Research, Massachusetts General Hospital, the University of Texas MD Anderson Cancer Center, and the Yale School of Medicine) accredited through the Clinical Laboratory Improvement Amendment. This is because it is being conducted at many clinical sites across the United States that participate in NCI’s Clinical Trials Network (as many as 2,400 different clinics). The broad geographic distribution of sites contributing tissue for analysis limited the feasibility of having testing done at only one central laboratory.

The NCI-MATCH trial targets the largest number of aMOIs to date. The therapy match is based on the identification of 4066 MOIs, a subset of which are aMOIs targeted by 24 different drugs. The full range of genetic alterations are included in these aMOIs - SNVs, indels, CNVs, and gene fusions. To capture the full range of these changes, a targeted DNA and cDNA based NGS panel is required, thus requiring both DNA and RNA extracts. While the sheer number of aMOIs and drugs suggests that precision medicine is poised to have a large impact on most cancer patients, the reality is that most aMOIs are rare, and the study’s coordinators estimate that nearly two-thirds of tumors screened will not have an aMOI identified and will not be able to be matched to a therapy. For the laboratory this presents challenges of validating rare aMOIs as well as increasing the capacity for testing the large number of tumors required to identify those with a matched drug.

Finally, the NCI-MATCH trial targets the largest variety of tumors to date. The trial is open to adults with solid tumors, lymphomas and myelomas that no longer respond to standard therapy. Drugs are matched independent of the origin and histology of the tumor, thus allowing the promise of targeted therapy to be extended to rarer tumor types and tumors that have a high degree of genetic variability between patients. This means that assay validation needs to account for the wide variety of tissue types to be extracted.

Overview of NGS Assay Validation

It is encouraging that NGS-based assays for clinical cancer genomic profiling have increasingly been the focus of validation studies [7; 9–11]. Although it is clear that assay validation is essential for clinical implementation, it could be questioned what each contributes to the literature. To place the study of Lih et al [1] in the appropriate context, some of the more recent validation studies with closest similarity to that of Lih et al will be reviewed.

A validation study reported by Frampton et al. relied on reference samples of pooled cell lines to determine the accuracy of mutant allele frequency, indel length and gene amplification in 287 cancer related genes using a capture-based NGS assay. They reported 95–99% sensitivity ( >99% for SNVs, 98% for indels and 95% for CNAs) and high specificity (positive predictive value >99%) [9]. Test accuracy on clinical material was evaluated using 249 formalin-fixed paraffin-embedded (FFPE) cancer samples that had base substitutions, indels or CNAs identified by alternative clinical diagnostic technologies. Agreement was good (96% concordant for base substitutions and indels and 95–100% concordant for CNAs) as was reproducibility. Interestingly they noted that 25% of concordant samples had an MAF ≤ 10%, emphasizing the need for high sensitivity. Applying this assay to 2,221 solid tumors of diverse origin submitted for clinical evaluation, 95% were successfully tested and 76% of those tumors had a genetic alteration that was associated with a targeted treatment option (either clinically available or currently in clinical trial).

In a 2013 Journal of Molecular Diagnostics publication, Singh et al. 2013 reported clinical validation of an Ion Torrent Ampliseq cancer panel interrogating 740 mutational hotspots in 46 cancer related genes using the Ion Torrent Personal Genome Machine (PGM)[10]. They applied the assay to FFPE from 70 solid tumor samples with known mutations identified with orthogonal methods. The NGS assay identified all but one of 66 expected nucleotide variants, but was not successful at detection of indels. Based on serial dilution of cell line DNA with known allele frequency the assay could reliably detect variants at 10% frequency with high intra- and inter-run reproducibility.

There have been two reports of NGS validation for use in other precision oncology clinical trials. The Memorial Sloan Kettering-Integrated Mutation Profiling of Actionable Cancer Targets (MSK-IMPACT) assay uses a custom hybridization capture panel for the targeted sequencing of all exons and selected introns of 341 genes using the Illumina sequencing platform [11]. Validation was based on analysis of DNA from 284 FFPE tumor samples with known mutations and serial dilutions to establish threshold for detection. The NCI-MPACT trial uses an NGS assay to evaluate 380 aMOIs in 20 genes in three targeted pathways. These genes were added to the commercially available 46-gene Ion AmpliSeq Cancer Hotspot Panel version I, resulting in final primer design encompassing 62 unique genes and 383 amplicons [7]. The validation included 191 DNA samples including 120 cell lines, 4 spike-in of plasmid controls including MOI, 17 xenographs and 50 clinical specimens. While the study will use FFPE, 118 of the validation samples were fresh frozen. Analytic validation included assessment of sensitivity, specificity, accuracy and reproducibility on targets sample subsets. Ten proficiency samples were prepared from clinical samples previously analyzed by a conventional mutation-detection assay in another CLIA laboratory. To establish the NCI-MPACT assay as “fit-for-purpose”, 10 DNA samples were prepared according to the NCI-MPACT protocol and assayed four times with the NCI-MPACT assay (two independent operators each performing assay twice). The identified aMOIs were verified by Sanger sequencing. Both MSK-IMPACT and NCI-MPACT validation studies reported high sensitivity and reproducibility for detection of different variants.

Unique Features of NCI-MATCH Validation Study

The validation plan for the NCI-MATCH assay reported by Lih et al [1] included the need to assure inter-laboratory comparability between the four laboratories. Several factors probably contributed to the success of these networked laboratories. First was the use of a commercial assay that was optimized for the MATCH trial. The details of the design and the initial validation of what is now known as the Oncomine™ Cancer Panel (OCP, Thermo Fisher Scientific) NGS assay has been described [12]. Briefly, there are several key features of this assay. The assay was designed to accommodate limitations imposed by FFPE tissues with limited quantity and quality of DNA and RNA. Sequencing results from the Ion Torrent Personal Genome Machine™ (PGM, Thermo Fisher Scientific) are linked to an informatics pipeline using the Torrent Suite (4.4.2) and Ion Reporter (4.4.2) to identify the relevant mutations. The initial validation study used 300 FFPE tumor samples and established accuracy >95% for selected mutations and gene fusions[12].

The four Clinical Laboratory Improvement Amendment laboratories each conducted preliminary testing with the OCP assay system and then held face-to-face meetings to develop locked SOPs using the validated PGM instruments for all steps of the OCP using to assure uniformity. This included preanalytic considerations of tumor quality and processing, as well as extraction, library preparation, sequencing and data analysis. Samples included in the validation were selected to represent 256 MOIs targeted by the OCP and previously verified by analytically validated assays. A total of 186 clinical samples and 12 cell lines, all FFPE, were assayed in a total of 455 sequencing runs. In addition to selecting samples based on MOIs, consideration was given to the range of tumor types and tissue likely to be submitted in the NCI-MATCH trial. The clinical samples included tumors originating from different tissues, including pancreas, melanoma, skin and bone that may present particular challenges for sequencing; 18 different tissues sources for DNA and 5 for RNA sequencing. Acceptable assay performance characteristics were pre-defined prior to beginning of validation which included (1) an acceptable assay sensitivity pre-specified ≥95% for SNVs, and ≥90% each for other variant types within each clinical laboratory and all four laboratories combined, (2) acceptable assay specificity pre-specified as ≥99.9% for SNVs, ≥99.0% for indels and large indels, ≥97.0 for CNVs, and ≥99.0% for gene fusions within each laboratory, and (3) acceptable intra-and inter-operator overall concordance pre-defined ≥99% for each laboratory as well as for combined laboratory results.

The validation study indicated the OCP assay as performed by the four collaborating laboratories in NCI-MATCH trial met the expected performance characteristics for its intended use in the precision medicine trial. The study used the MATCHBox data management system to upload, analyze and report results, verifying the fully functional analysis pipeline. Results indicated high reliability (96.98% sensitivity for 265 known mutations, and 99.99% specificity), and high reproducibility (99.99% mean inter-operator pairwise concordance across four laboratories). The study further demonstrated that the assay can be used with multiple tumor tissue types with remarkable level of lower level of detection (LOD) for each variant types (2.8% for SNVs, 10.5% for indels, 6.8% for large indels, and four copies for CNVs).

Fit-for-purpose assessment of the full assay was based on prospective testing of twenty two tumors system, from biopsy collection through reporting. These tumors included 16 different tumor types. Evaluation criteria required that the NGS data passed quality control metrics, results were successfully uploaded into MATCHBox, and variant calls and treatment assignment were correctly reported. In a total of 32 runs (some samples were analyzed by multiple laboratories), 29 runs (90.6%) passed quality control metrics. In the 6 samples assayed by multiple laboratories, 11 MOIs (8 SNVs, 1 indel, and 2 CNVs) were detected at similar allele frequency or copy number in all laboratories (100% concordance).

Conclusion

The validation study reported by Lih et al [1] is another step moving the field closer to the time when precision medicine will generate the expected benefits in improved clinical outcomes. The NCI-MATCH is the most ambitious “genotype to phenotype” study to date, requiring laboratories to achieve the precision required to have a fully transferrable NGS assay and pipeline applicable to a broad range of tumors and tissues. As remarkable as this achievement is, it is clear that the field is still young. Even with the large number of MOIs evaluated and aMOIs matched to drugs, the majority of patients screened will be ineligible for the trial. Achieving the full clinical impact of genetically matched therapies has complicated uncertainties in interpreting which genetic changes are driver mutations and by development of drug resistance. A major limitation of single agent-targeted treatment is the inevitable development of resistance, which can be assessed on NGS platforms at the time of initial presentation and on subsequent analyses but is not addressed in current NGS trials or this commentary. While the success of the NCI-MATCH trial cannot be assured, linking precision laboratories to precision medicine trials assures that data used for drug assignment will be reliable. Furthermore, the use of a commercial platform and integrated analysis and reporting pipeline will greatly facilitate broader translation of any successes. However translation to clinical use will require each laboratory to validate the assay in their hands. Published validation studies, such as the one by Lih et al, will serve to point out critical components of that process [1].