Assuring the quality of next-generation sequencing in clinical laboratory practice

To the Editor:

We direct your readers’ attention to the principles and guidelines (Supplementary Guidelines) developed by the Next-generation Sequencing: Standardization of Clinical Testing (Nex-StoCT) workgroup. These guidelines represent initial steps to ensure that results from tests based on next-generation sequencing (NGS) are reliable and useful for clinical decision making. The US Centers for Disease Control and Prevention (CDC) convened this national workgroup, which collaborated to define platform-independent approaches for establishing technical process elements of a quality management system (QMS) to assure the analytical validity and compliance of NGS tests with existing regulatory and professional quality standards. The workgroup identified and addressed gaps in quality practices that could compromise the quality of both clinical laboratory services and translational efforts needed to advance the implementation and utility of NGS in clinical settings.

The workgroup was composed of experts with knowledge of and experience with NGS and included clinical laboratory directors, clinicians, platform and software developers and informaticians, as well as individuals actively engaged in NGS guideline development from accreditation bodies and professional organizations. Representatives from US government agencies also participated.

These guidelines address four topics that are components of quality management in a clinical environment: (i) test validation, (ii) quality control (QC) procedures to assure and maintain accurate test results, (iii) the independent assessment of test performance through proficiency testing (PT) or alternative approaches and (iv) reference materials (RMs). Discussions were limited to the analytic and informatics processes required for accurate variant calling. The workgroup did not address how variants are prioritized, interpreted or reported.

The workgroup recommendations are summarized in Table 1. Although the workgroup focused on detection of DNA sequence variations associated with heritable human disorders, many of the principles and recommendations described are also relevant to the application of NGS to other areas of laboratory medicine, including the diagnosis, prognosis and treatment of cancer and infectious-disease testing.

Table 1.

Selected workgroup recommendations for establishing NGS test systems for clinical use

Requirements for test establishment	Objective	NGS-specific recommendationsa
Validation	Document reliability of the platform, test, and informatics pipeline before testing of patient specimens	Platform validation: establish that the system provides reliable sequence analysis across the genomic regions targeted by the test. Test validation: establish that the system correctly identifies disease-associated (and other) variants in targeted regions of the genome (Supplementary Guidelines, section 4). Informatics pipeline validation: establish that the algorithm(s) reliably analyze platform data to produce an accurate sequence. Establish and validate alternate methods (for example, Sanger sequencing) to derive high-quality sequence data for problematic genomic regions.
Quality control	Document reliability of the sequence analysis during patient testing	Utilize a combination of QC materials, both intrinsic and/or spiked in, that mimic genomic complexity and the types of mutations the test is designed to detect. During patient testing, quality metrics (for example, quality scores, depth and uniformity of coverage, mapping quality, GC bias and transition/transversion ratio) should be assessed and compared to those established during validation. Clinically actionable findings should be confirmed by independent analysis using an alternate method.
Proficiency Testing	The independent assessment of test performance	PT challenges should target the analysis of both disease-associated and naturally occurring sequence variations across the genomic regions targeted by the test to measure the reliability of sequence analysis. Electronic sequence files may permit a comparison of alignment and variant calling methods across laboratories but will require additional consideration of platform differences. PT programs should consider the different genomic regions targeted by each recipient laboratory’s assays to properly compare interlaboratory performance.
Reference Materials	The use of materials for quality management of the analytical phase of testing	RMs with both naturally occurring and disease-associated sequence variations are needed for test validation, QC procedures and the independent assessment of test performance. Synthetic DNA and electronic reference data files may serve as RMs for rare or challenging sequence variations. Efforts should be undertaken to establish a suitable NGS RM and the sequence of the RM should be refined as the technology changes. Such a RM should be annotated to indicate regions of high and low sequence reliability.

^aSee Supplementary Guidelines for complete recommendations. RM, reference material.

Validation is the process of establishing analytical performance specifications for a clinical test system developed in house to confirm that the system is suitable for its intended use¹. During the validation process, the laboratory must demonstrate that the assay functions as expected and provides reliable results. The workgroup considered the requirements of the Clinical Laboratory Improvement Amendments (CLIA) and provided recommendations for validation of clinical NGS tests. The validation process can be divided into three stages: platform, test-specific and informatics pipeline validation (Supplementary Fig. 1). Platform validation provides evidence that the assay can deliver reliable sequence data within the regions of the genome targeted for analysis. Test-specific validation demonstrates that the assay can detect clinically important sequence variations for the intended application. Validation of the informatics pipeline establishes the software settings necessary to ensure that the test can reliably provide accurate sequence data and detect variations. Although each stage of validation is considered separately, they are interdependent. Validation requires application-specific considerations for whether the test targets a gene panel, the exome or the whole genome, as well as the types of sequence variations that are detected.

In the United States, diagnostic tests that are provided to clinical laboratories are regulated by the US Food and Drug Administration (FDA). To date, no NGS technologies have been approved or cleared by the FDA. These tests are developed in house as laboratory-developed tests and are regulated under CLIA². The CLIA regulations define evaluation of analytical reliability and limitations and require laboratories to establish specifications for performance characteristics for each test system developed in house. The characteristics evaluated to establish the analytical validity of test results include accuracy, precision, analytical sensitivity, analytical specificity, reportable range, reference range or reference intervals, and other relevant performance metrics. Laboratories in the United States and other countries may also comply with the QMS standards described in ISO 15189 (ref. 3). The performance characteristics defined in CLIA² and professional guidance documents⁴ (ACMG standards, guidelines, and policies available at; CAP NGS checklist available to subscribers at http://www.cap.org/) do not readily translate to NGS testing practices owing to the complexity of the technology and the informatics analyses required for large-scale genome analyses. Therefore, the workgroup adapted the definitions of these performance characteristics to better fit the use of NGS in the clinical laboratory (Table 2). A comparison between the CLIA definitions and those developed by the workgroup is presented in Supplementary Table 1, and the unique metrics for NGS that laboratories should establish and monitor to assure high-quality analytical results are presented in Supplementary Table 2. For example, the depth of coverage, or the number of independent reads assessed when making a base call, is a crucial metric for establishing the accuracy, analytical sensitivity and analytical specificity of an NGS test. Owing to the high costs and extensive data analyses required for these tests, it is challenging to establish the precision (e.g., repeatability of testing replicates) of an NGS assay by determining concordance of sequencing results among a large number of samples. The workgroup proposed an alternative approach in which additional metrics such as the depth and uniformity of sequencing coverage would be incorporated to limit the number of samples required to establish precision. The workgroup redefined ‘reportable range’ and ‘reference range’ in terms of the qualitative nature of these DNA test results. This guideline classifies reportable range as the region of the genome from which sequence of an acceptable quality can be derived by the laboratory test, and reference range as the spectrum of nonpathogenic base changes observed in a population. Sequence variations outside this spectrum could be disease associated, but further investigation may be necessary to confirm disease association. The determination of reference range is problematic because the spectrum of sequence variations that can be defined as ‘normal’ or deleterious will vary across individuals and populations.

Table 2.

Workgroup definitions of CLIA performance characteristics for NGS

Performance characteristic	Workgroup established definition for NGS applicationsa
Accuracy	The degree of agreement between the nucleic acid sequences derived from the assay and a reference sequence.
Precision	The degree to which repeated sequence analyses give the same resultrepeatability (within-run precision) and reproducibility (between-run precision).
Analytical sensitivity	The likelihood that the assay will detect the targeted sequence variations, if present.
Analytical specificity	The probability that the assay will not detect a sequence variation when none are present (the false positive rate is a useful measurement for sequencing assays).
Reportable range	The region of the genome in which sequence of an acceptable quality can be derived by the laboratory test.
Reference range	Reportable sequence variations the assay can detect that are expected to occur in an unaffected population.

^aThese definitions may be applied to both NGS and Sanger sequencing. A more detailed comparison between the CLIA definitions and those developed by the workgroup is presented in Supplementary Table 1.

QC procedures monitor whether the components of an assay—including the reagents, specimen processing, instrumentation and data processing (the informatics pipeline)—are functioning properly and delivering accurate results during testing of patient samples. The workgroup considered NGS-specific QC metrics that are useful for monitoring the performance of the assay, including: depth of coverage and uniformity of coverage, quality scores for base calling and alignment, allelic read percentage, strand bias, GC bias and decline in signal intensity (Supplementary Table 2).

Proficiency testing and external quality assessment programs provide a formal mechanism for comparing inter-laboratory test performance and can help to identify analytical and interpretive errors and problems with QC, instrument calibration and assay design. Laboratories are encouraged to subscribe when such programs are available. PT programs typically provide several ‘blinded’ samples (PT challenge) to participating laboratories for analysis. The participants subsequently report their results, an interpretation and a brief description of their assay methods to the PT program. Data are compared among participating laboratories to assess inter-laboratory test performance, and a cumulative summary is shared with participants. At present, no formal PT programs exist for NGS, and development of a program faces four fundamental obstacles: (i) the absence of a defined scope for the challenge (which region(s) of the genome will be targeted); (ii) the absence of well-characterized PT materials suitable for a range of applications (for example, analysis of gene panels, the exome and the whole genome); (iii) the absence of standard metrics for use as comparators among participating laboratories and (iv) cost and time commitments for participants and the PT provider. To address these issues, the workgroup developed principles and guidelines for combining a formal PT challenge (when available) with an alternative assessment process, such as inter-laboratory exchange of previously characterized samples (this may include genomic DNA (gDNA) and/or electronic data files). This combined approach could reduce expenses and provide a flexible means to assess inter-laboratory test performance.

Reference materials are used during test validation, QC and PT to establish and monitor the quality of clinical laboratory tests^1,4. RMs are homogeneous and stable, and they have the particular property being measured, such as the presence of disease-associated sequence variations. Many different types of samples can be used as RMs for NGS, including characterized DNA derived from human cell lines or patient specimens, synthetic DNA or electronic data. The advantages and disadvantages of each of these RMs are described in Supplementary Table 3. Characterization of RMs for NGS is complicated by the size of the genomic region(s) targeted for analysis. Laboratories now use gDNA obtained from a variety of sources, including human cell lines characterized in the 1000 Genomes Project⁵, to develop and validate NGS assays. However, the large spectrum of disease-associated sequence variations cannot be represented in one or any practical number of gDNA RMs. The workgroup recommended that one or more RMs that contain a combination of disease-associated and naturally occurring sequence variations should be evaluated for use in assay validation, QC and PT. The workgroup suggested that electronic reference data files containing real or simulated sequence data can be used for establishing and monitoring the performance characteristics of the NGS informatics pipeline. This approach is useful because combinations of DNA sequence variations can be engineered into these files to assess the capability of the informatics pipeline to make accurate variant calls. The CDC’s Genetic Testing Reference Materials Coordination Program (http://wwwn.cdc.gov/dls/genetics/RMMaterials), the US National Institutes of Health and the US National Institute for Standards and Technology are developing and characterizing RMs for NGS to meet the many needs of clinical laboratories.

Sequencing technologies are evolving rapidly, and although the technical details may change, quality management requirements for test system validation, QC and PT will remain. The workgroup identified areas in which additional data collection and analysis are needed to assure the quality of clinical NGS (Supplementary Table 4). To our knowledge, the principles and guidance developed by the workgroup are the first efforts to establish consensus among a diverse group of stakeholders. These recommendations will further inform the deliberations of regulatory agencies, professional societies and accrediting organizations that are considering the application of QMS standards and guidelines to NGS for clinical applications. These guidelines will be updated and expanded as NGS technologies evolve. Information about the continued activities of the working group is available at http://www.cdc.gov/osels/lspppo/Genetic_Testing_Quality_Practices/Nex-StoCT.html. We encourage collaborations and ongoing discussions among laboratories, clinicians, manufacturers, service providers, software developers, professional organizations and government agencies to ensure the quality of clinical NGS tests.