Development and Characterization of Reference Materials for Genetic Testing: Focus on Public Partnerships

Abstract

Characterized reference materials (RMs) are needed for clinical laboratory test development and validation, quality control procedures, and proficiency testing to assure their quality. In this article, we review the development and characterization of RMs for clinical molecular genetic tests. We describe various types of RMs and how to access and utilize them, especially focusing on the Genetic Testing Reference Materials Coordination Program (Get-RM) and the Genome in a Bottle (GIAB) Consortium. This review also reinforces the need for collaborative efforts in the clinical genetic testing community to develop additional RMs.

INTRODUCTION

In clinical laboratories, molecular genetic tests are increasingly being used for diagnosis and characterization of tumors and heritable disorders. Molecular genetic tests can also aid in prognostic assessment, to guide the selection of therapeutic strategies, and to monitor response to treatment. As genomic sequencing is applied more widely, questions have been raised about how to standardize assays across laboratories, assess analytical accuracy, and implement quality control (QC) procedures comparable to other laboratory tests [1,2]. Because the human genome contains approximately 3 billion bases that can be screened for variants, assuring the quality of human DNA analysis presents great challenges [1,2,3,4].

Reference materials (RMs) and QC materials are essential for many aspects of genetic testing. These materials, which are tested alongside patient samples, act as controls and permit the detection of errors due to test system failure or operator error. In addition, RMs are needed for test development and validation, lot-to-lot comparisons, and for proficiency testing/external quality assessment programs [5,6,7].

Ideally, a RM should resemble a patient sample, and contain the variant alleles or mutation types that the assay is designed to detect. Clinical genetic testing laboratories typically use genomic DNA from residual patient samples or from characterized cell lines that contain variants relevant to the disorder being assessed by the assay [6,7]. Other types of RMs are also used, such as plasmids or synthetic DNA.

Until recently, efforts have largely centered on the development of RMs for specific genetic tests involving only a single gene. In May 2015, the National Institute of Standards and Technology (NIST) released the first "human genome" DNA RM, NIST RM 8398, that laboratories can use to evaluate the accuracy of next generation sequencing (NGS) assays. Laboratories can use RM 8398 to develop and optimize new NGS assays and assess the performance of their NGS library preparation methods, sequencing technology, and bioinformatics processes [8].

There are a number of groups working to produce RMs for DNA-based analysis. In this review, we will address the recent advances in the development and characterization of various types of RMs with an emphasis on their application toward human DNA sequencing and molecular oncology.

TYPES OF REFERENCE MATERIALS

1. Variety of reference material characterization

The term RM refers to a wide variety of material types. These materials can be certified or non-certified, qualitative or quantitative, and can be characterized to different degrees. The International Organization for Standardization (ISO) defines a RM as "a material that is homogeneous and stable with respect to one or more specified properties, which has been established to fit for its intended use in a measurement process" (ISO Guide 30:2015) [9] (Table 1).

Table 1. Definitions according to draft International Organization for Standardization (ISO) guide 30:2015.

Reference materials (RM)	Material, sufficiently homogeneous and stable with respect to one or more specified properties, which has been established to be fit for its intended use in a measurement process
Certified reference materials (CRM)	RM characterized by a metrologically valid procedure for one or more specified properties, accompanied by a certificate that states the value of the specified property, its associated uncertainty, and a statement of metrological traceability
Characterization of RM	Determination of the property values or attributes of RM, as part of the production process
Value assignment	Process by which RM property values or attributes obtained by characterization are combined and expressed in accompanying RM documentation
Property value of RM	Value corresponding to a quantity representing a physical, chemical or biological property of [certified] RM
Nominal property value of RM	Value corresponding to a qualitative attribute representing a physical, chemical or biological property of [certified] RM

Certified RMs (CRMs) are characterized by at least one metrologically valid procedure for at least one specified property with a stated uncertainty. They come with a certificate providing the value of the specified property, its associated uncertainty, and a statement of metrological traceability. CRMs are characterized for composition and are traceable to the International System of Units (SI) without reference to a calibrator, and are produced by a variety of accredited organizations. The NIST releases CRMs under the trademark "standard RMs (SRMs)" [10] and must meet additional NIST-specific criteria. European RMs (ERMs) are CRMs produced by the European RM consortium [11]. Other RMs, including the whole genome NIST RM 8398, are characterized for homogeneity and stability, but have no certified values. The World Health Organization (WHO) disseminates a variety of RMs that are widely used to harmonize clinical assay testing [12]. WHO standards are typically prepared in a natural matrix, such as plasma or serum, facilitating compatibility between assays. The values assigned to WHO International Standards are determined by the consensus of a number of laboratories that test the samples, rather than a reference method [7]. Other commonly used RMs include characterized genomic DNA from cell lines or residual patient DNA. These RMs, while often less formally characterized and distributed than other RM types, are useful for assuring the quality of laboratory tests when more characterized materials are not available.

2. Higher order reference materials

A number of "higher order" RMs are available for molecular genetic testing [13,14]. These include CRMs, SRMs, and WHO standards for Factor V Leiden; Fragile X; BCR-ABL1; prothrombin; Huntington's disease; mitochondrial DNA; the human genome; blood group genotyping; Hepatitis A, B, and C; cytomegalovirus (CMV); HIV; human papilloma virus (HPV); and a number of bacteria and yeast. In May 2015, NIST released RM 8398, Human DNA for Whole-Genome Variant Assessment [8]. This RM has been characterized for homogeneity and stability, and has reference values for single nucleotide polymorphisms (SNP), small insertions, and deletions (indels), and homozygous reference genotypes for approximately 77% of the human genome [15]. Variants are reported with respect to the GRCh37 Human Reference Assembly. However, CRMs, SRMs, and WHO standards are relatively expensive and are only available in small quantities. They are intended as primary standards that can be used to produce secondary standards traceable to the RM, which are then used as daily controls.

3. Other types of reference materials

Although "higher order" RMs are available for a small number of tests and often in small amounts, many other RMs are available in larger quantities and are appropriate for use on a daily basis as QC materials and for test development and validation [6,7]. These materials include commercially available controls, synthetic controls, and characterized genomic DNA and cell lines. Some of these materials are well-characterized, produced in accordance with good manufacturing practices and/or ISO guidelines, and/or approved or certified by national standards organizations (eg. Conformité Européenne [CE]-marked, Food and Drug Administration [FDA]-cleared). Other less-characterized materials are also sold as reference standards. Some materials, such as cell lines and genomic DNA, may be minimally characterized or uncharacterized and require further validation before use in clinical laboratories.

GeT-RM PROGRAM

1. The GeT-RM program addresses gaps

Clinical laboratories use RMs for a variety of purposes, including test development, validation, QC, and proficiency testing. Use of these materials is essential for assuring the quality of clinical tests, and their use for assay validation and quality assurance is required by regulatory and accreditation bodies and is generally recommended by professional guidance [6,16,17,18,19,20,21]. Although clinical genetic tests are available for over 3,000 conditions [22], RMs are not available for the vast majority of these assays. In the absence of publicly available RMs, clinical laboratories often use DNA derived from leftover patient specimens as controls. These RM specimens are not easily available, nor are they renewable, and they must be validated by multiple methods prior to use.

The US Centers for Disease Control and Prevention (CDC), in partnership with the genetic testing community, established the Genetic Testing Reference Materials Coordination Program (Get-RM) [23] in 2004 to address the urgent need for characterized genomic DNA RMs. The GeT-RM program is coordinated by the CDC, but all the decisions about RM priorities, specimen collection, material development, and characterization occur through voluntary collaborations with laboratories in the clinical genetic testing community. Cell lines with confirmed genotypes are the preferred type of control for DNA-based genetic testing as they closely resemble an actual patient specimen. Thus, the GeT-RM has focused on characterizing genomic DNA from publicly available cell lines.

2. How GeT-RM works

GeT-RM RM characterization projects typically begin with identifying the need for a RM to carry out tests for a single gene disorder (eg. fragile X) or a gene panel (eg. Ashkenazi Jewish [AJ]) (Fig. 1). Decisions about variants to be included among the characterized genomic DNA materials are made on the basis of consultations with clinical laboratories, commercial test developers, researchers, and cell repositories. Panels of genomic DNA samples are assembled to include many or all common variants or variant types known to cause the disorder(s), important combinations of variants, heterozygotes, homozygotes, compound heterozygotes, samples from males and females (X-linked disorders), non-pathogenic polymorphisms that affect assay performance, and variants at or near clinical decision points. Samples are also selected on the basis of the consideration of the assay technologies used [23].

Fig. 1. Steps in the GeT-RM Process. Information about reference material (RM) needs is obtained from the genetic testing community. Once needs have been identified, appropriate cell lines from the Coriell Cell Repositories are identified or created through collaboration with patient groups and academic laboratories. Coriell prepares and ships cell line-derived DNA samples to clinical genetic testing laboratories and test manufacturers who have volunteered to characterize the blinded samples. Different laboratories are selected to maximize the variety of testing methods used. Laboratories test the DNA samples and report their results back to the CDC for analysis and arbitration of discrepant results. A consensus genotype is assigned to each sample and the results are published, presented at meetings, and provided on the GeT-RM and Coriell websites.

Once a list of desired samples is compiled, cell lines containing these variants are selected from the Coriell Cell Repositories [24]. If the repository does not have cell lines with the desired variants, appropriate cell lines can be found or created. Cell lines positive for the requested variants can be transferred from research laboratories to the Coriell repository. In addition, the GeT-RM, in partnership with the Coriell Repositories, has created over 100 cell lines from blood samples collected from consenting patients who were contacted through patient advocacy organizations or clinical research projects. These newly created cell lines provided variants that were not already available from the Coriell Repository.

Once the selected cell lines have been identified or created, genomic DNA samples are sent to three or more volunteer laboratories for characterization. The laboratories are selected on the basis of the assay technology used to detect the variants and their willingness to test the samples. A variety of assay technologies and methods are used to characterize the samples to assure that the expected variant can be detected, and that the samples are commutable for use with current clinical assays. Results are sent to the GeT-RM, and a consensus genotype is assigned to each sample. The GeT-RM program publishes characterization studies [25,26,27,28,29,30,31,32,33,34] and the data are also available on the GeT-RM [23] and Coriell [24] websites.

3. Summary of GeT-RM projects for rare heritable disorders

Since 2004, the GeT-RM program has characterized more than 4,600 loci in over 400 cell line-based genomic DNA RMs for a number of genetic disorders including: fragile X [25], disorders on the AJ Panel [29] (Bloom syndrome, Canavan disease, Fanconi anemia, familial dysautonomia, Gaucher disease, mucolipidosis IV, Niemann Pick disease and Tay Sachs disease), cystic fibrosis [26], Huntington disease [27], methylenetetrahydrofolate reductase (MTHFR)-related homocysteinemia, α1-antitrypsin deficiency, multiple endocrine neoplasia, BRCA1- and 2-related cancers [28], Duchenne muscular dystrophy [30], myotonic dystrophy [31], Rett syndrome [32], and two large-scale DNA studies from 244 cell lines for a number of polymorphisms in 28 pharmacogenetic loci [33,34]. Each of these genomic DNA materials was tested in 3-10 clinical genetic laboratories by using a variety of genetic assays, including DNA sequence analysis. All cell lines and genomic DNA samples characterized by GeT-RM are publicly available from the Coriell Cell Repositories [24]. Information about genomic DNA samples characterized by the GeT-RM, as well as other information about other available RMs, is available from the GeT-RM website [23].

REFERENCE MATERIALS FOR NGS

1. Genome in a Bottle (GIAB) Consortium

NIST hosted the first meetings of the GIAB Consortium in 2012 to help develop RMs, reference data, and reference methods for assessing NGS performance. The GIAB RM Selection and Design Working Group recommended human cell lines for NIST to develop into RMs. The consortium selected NA12878 from the Coriell Cell Repositories as a pilot sample. This genome was widely used in public studies and was extensively characterized by the 1000 Genomes Project, sequencing technology developers, and other laboratories. The publicly available data for this genome was used to develop methods for integrating multiple datasets to establish high-confidence SNP, indels, and homozygous reference genotype calls [15]. In addition, NIST and GIAB have developed high-confidence calls for large deletions and insertions. The Illumina Platinum Genomes along with Real Time Genomics have developed high-confidence phased calls for this genome by using the extensive (17-member) pedigree to which this sample belongs and phasing variants through hereditary tracks.

NIST released the pilot RM 8398 in May 2015 [8]. Then, 8,300 vials of this RM were available for purchase, each containing approximately 10 µg of DNA extracted from a large homogeneous culture produced by the Coriell Repositories. Homogeneity of the DNA aliquots was verified by sequencing six randomly selected vials of the RM to approximately 300X. No differences in SNP, indels, or copy number variants were detected among DNA samples sequenced. Stability was assessed by measuring DNA fragment size distributions with pulsed field gel electrophoresis under a variety of conditions, including storage at elevated temperatures, as a proxy for accelerating aging.

In addition to the pilot RM 8398, GIAB has selected an AJ trio (i.e., samples from a child and both parents) and an Asian trio from the Personal Genome Project for characterization [35]. These genomes are also available from the Coriell Repositories as Epstein-Barr virus (EBV)-immortalized B lymphoblastoid cell lines and extracted DNA. Four of these genomes will be available as components of NIST RMs, with a planned release in mid-2016. The genomic DNA from the son of the AJ trio will be distributed as RM 8391, the DNA from the AJ trio (parents and son) will be distributed as RM 8392, and the DNA corresponding to the son of the Asian trio will be distributed as RM 8393 (note that the genomes of the Asian trio parents are only available from Coriell). Similar to the pilot genome, these NIST RMs consist of DNA extracted from a large batch of cells. These genomes were consented under the compressive Personal Genome Project [35] consent, which permits broad application of the genomes as scientific resources. For example, commercial redistribution of these samples is permitted, and two companies have already produced products that contain materials derived from biological samples of the NIST AJ son from whom RM DNA is prepared.

GIAB has established a collection of publically available data for these genomes from a comprehensive suite of technologies and methods. GIAB has formed an Analysis Group to coordinate analyses and integration of these diverse data types. The Analysis Group has the goal of forming high-confidence phased variant calls of all types and sizes for as much of the genome as possible.

These whole genome RMs are characterized for more than 2 billion bp. Therefore, instead of testing a laboratory's accuracy for a single pathogenic variant, these RMs can be used to establish test accuracy for different types of variants in different genome contexts. Standardizing definitions and computation of genome-scale performance metrics is complex, so GIAB and the Global Alliance for Genomics and Health formed a Benchmarking Team [36]. This team is developing standardized definitions for performance metrics along with open-source tools to calculate these metrics, even in difficult cases where different tools produce different representations of complex variants (i.e., from nearby SNPs and/or indels). In addition, the team is developing methods to stratify performance by variant type, genome context, functional context, and data characteristics. For example, they have compiled standardized bed files with different types of repetitive regions, GC content, etc. for stratification by genomic context. These files can facilitate the understanding of genome regions with relatively high or low accuracy, as well as regions in which the RMs have few or no examples of mutations. The performance metrics definitions, variant comparison tools, and stratification methods developed by the Benchmarking Team will help laboratories characterize the performance of variant calls in an accurate and standardized way.

GIAB is considering developing other types of RMs, including synthetic spike-in DNA that could be added into any sample to test the mutation detection accuracy. These spike-ins could be particularly useful for detecting somatic mutations at different allele frequencies in tumor samples. For germline sequencing, these spike-ins might include mutations that are difficult to detect (e.g., truncating indels and complex variants) in exons of genes that are typically included in targeted gene panels. Characterizing an assay's capability of identifying difficult mutations in disease-related genes is important because many disease-related mutations are difficult to detect, and few or no examples of these mutations exist in disease-related genes in the "normal" genomes that GIAB is developing as RMs.

2. GeT-RM browser

NGS assays are becoming common in clinical laboratories. Use of this technology allows the rapid, economical analysis of multi-gene panels, exomes, or genomes for disease-causing variants. Before the introduction of NGS technology, clinical laboratories analyzed single genes or small gene panels using Sanger sequencing. Validating the Sanger assays that covered small, well-studied genomic regions could be achieved by careful characterization of a relatively small number of residual patient samples, or characterized genomic DNA derived from cell lines containing variants in the gene(s) of interest. Through analysis of these samples, laboratories could determine their capability to detect the expected variants, measure the number of false positive or negative results, and study the repeatability and reproducibility of the assay, as required by regulatory requirements and recommended by professional guidelines [16,17,18,19,20,21,37,38]. It is much more difficult to perform analytical validation of large NGS gene panels and exome or genome tests than it is to validate simpler tests. It is not possible to obtain or analyze genomic DNA samples that contain pathogenic variants in every gene of the human genome. It is also extremely difficult to obtain samples without variants that are sufficiently well-characterized for regions included in assays to allow for the calculation of false positive and negative rates. The CDC Next-generation Sequencing: Standardization of Clinical Testing [Nex-StoCT] workgroup [4], the American College of Medical Genetics [39], and the College of American Pathologists (CAP) [18] recommend that laboratories use well-characterized genomic DNA RMs with both naturally occurring and disease-associated variants. The genomic DNA RMs created by NIST helps to fill this need.

Laboratories that use NIST RMs need tools to assess the capability of their methodology to detect variants and accurately sequence non-variant genomic regions. The GeT-RM program has worked with the National Center for Biotechnology Information (NCBI), in collaboration with members of the genetic testing community, academic and clinical genetic testing laboratories, in vitro diagnostics (IVD) manufacturers, professional organizations, other government agencies, and the Coriell Cell Repositories to create an interactive web portal called the GeT-RM Browser [40]. This tool allows for comparison of the NA12878 sequence generated by laboratories either from the NIST reference material, or from genomic DNA purchased from Coriell, to the high confidence variant calls that are associated with the NIST RM 8398 [15]. The GeT-RM Browser also contains a number of high quality, publicly available datasets, including data from the Illumina Platinum Genome [41], 36 datasets from NA12878, and 21 datasets for NA19240 generated by eight clinical laboratories and four research laboratories as part of a GeT-RM project. The browser allows easy access to and visualization of public variant call sets and the alignments used to generate them. The GeT-RM Browser also provides a number of other data tracks that show features such as segmental duplications, reference SNPs, and genomic regions that are difficult to sequence, that laboratories can also use to assess the quality of their data. Data from the browser can be downloaded and compared to user data, or the user can upload NA12878 or NA19240 sequence data and use the browser to visually compare their calls to those in the available data sets. New datasets for NA12878, as well as datasets and high-confidence calls from the candidate NIST RMs from the Personal Genome Project, will be added to the browser as they become available.

REFERENCE MATERIALS FOR MOLECULAR ONCOLOGY

The rapidly changing field of oncology is driven by many new therapeutics coming to the market that target genetic mutations. Selecting the most appropriate patient therapy requires robust and well-controlled NGS assays. Thoroughly characterized RMs are needed to assure quality, and to compare and standardize the clinical NGS assays used to select treatments. Oncology-related RMs should include the common variants used to initiate clinical patient management (i.e. treatment selection), as well as the types of mutations encountered when sequencing cancer tissue, including single nucleotide variants (SNVs) or multinucleotide polymorphisms (MNPs), and large and small indels. Many newly developed clinical NGS assays are able to detect gene amplifications and deletions (copy number variations [CNV]), thus, RMs containing these variant types are also needed.

There are several approaches that are currently being used to develop RMs for molecular oncology. One company has developed an engineered cell line containing a variety of oncogenic variants. These engineered cell lines consist of a specific number of mutations, including translocations, at defined variant allele frequencies (VAFs). Cell line RMs can be treated and processed in a similar manner to clinical samples, thereby better serving as controls for all stages of the assay. In addition, several groups have developed synthetic DNA RMs, including plasmids and synthetic oligos that are spiked into a well-characterized human genomic DNA sample such as NIST RM 8398.

Synthetic plasmids have been used successfully as control materials at the National Cancer Institute (NCI)'s molecular characterization laboratory (Fredrik National Laboratory for Cancer Research [FNLCR]). In yet unpublished data, a set of plasmids using the same plasmid backbone was generated, and can be easily manufactured with clonal purity [42]. Each plasmid consists of single known variant which is described in the Catalogue of Somatic Mutations in Cancer (COSMIC) database, located approximately in the middle of a 1 kb DNA fragment [43]. A mixture of these synthetic plasmids was added to genomic DNA from cell line NA12878 at quantities representing different VAFs. NA12878 was chosen because this cell line is well-characterized and is the basis for the NIST RM 8398 [8,15]. The pelleted cells containing plasmid DNA were treated with neutral buffered formalin (NBF), and processed as formalin-fixed paraffin-embedded (FFPE) samples to resemble FFPE clinical samples. The performance of NA12878 mixed with synthetic plasmids was compared to the results from cell line genomes harboring the same mutations, similarly prepared. Virtually no difference in mutation detection was seen between DNA derived from FFPE-treated cell lines containing variants and mutant synthetic plasmid mixed with normal FFPE DNA. Experiments also demonstrated that RMs derived from the synthetic plasmids can be successfully analyzed by using different sequencing chemistries, platforms, and bioinformatics pipelines [42]. Plasmid-based controls have been successfully used as positive controls in a clinically validated NCI-MPACT assay [44].

Another type of synthetic RM utilizes a single mix of multiple synthetic oligos. One company produces an oligo mixtures representing 550 commonly occurring cancer mutations diluted into genomic DNA derived from GM24385, one of the genomes being characterized by GIAB, at VAFs. These synthetic RMs also perform well when analyzed using different NGS platforms.

SUMMARY AND FUTURE PERSPECTIVE FOR REFERENCE MATERIALS

NGS assays are becoming increasingly common because the commercial availability of affordable, pre-configured gene panels and analytic pipelines has facilitated the introduction of this technology into clinical laboratories. In addition, clinical laboratories continue to increase the number of genes included in multigene panels and are beginning to offer assays that assess both the exome and the genome. These developments have accentuated the need for a wide variety of RMs and RM types.

RMs are critical for the successful development, validation, comparison, and standardization of clinical NGS assays and bioinformatics pipelines. RMs are also needed as positive controls to monitor assay performance. Laboratories need a sufficient number of samples containing a mixture of known variants, common pathogenic variants, and variant types (eg. SNVs, indels) in genomic regions covered by the test to determine the analytical sensitivity. RMs with difficult-to-sequence variants, such as mutations in pseudogenes, or large deletions or duplications that may interfere with assay performance, are especially important. Laboratories have also expressed a desire for synthetic RMs that can be used to simultaneously detect multiple variants in a cost-effective manner. Electronic RMs may also be used to assess sequencing tests [4]. Although there are a number of RM development efforts currently underway by government and commercial entities, many more RMs are urgently needed to facilitate the implementation of these assays and to assure their quality.