A Comprehensive Workflow for Read Depth-Based Identification of Copy-Number Variation from Whole-Genome Sequence Data

Abstract

A remaining hurdle to whole-genome sequencing (WGS) becoming a first-tier genetic test has been accurate detection of copy-number variations (CNVs). Here, we used several datasets to empirically develop a detailed workflow for identifying germline CNVs >1 kb from short-read WGS data using read depth-based algorithms. Our workflow is comprehensive in that it addresses all stages of the CNV-detection process, including DNA library preparation, sequencing, quality control, reference mapping, and computational CNV identification. We used our workflow to detect rare, genic CNVs in individuals with autism spectrum disorder (ASD), and 120/120 such CNVs tested using orthogonal methods were successfully confirmed. We also identified 71 putative genic de novo CNVs in this cohort, which had a confirmation rate of 70%; the remainder were incorrectly identified as de novo due to false positives in the proband (7%) or parental false negatives (23%). In individuals with an ASD diagnosis in which both microarray and WGS experiments were performed, our workflow detected all clinically relevant CNVs identified by microarrays, as well as additional potentially pathogenic CNVs < 20 kb. Thus, CNVs of clinical relevance can be discovered from WGS with a detection rate exceeding microarrays, positioning WGS as a single assay for genetic variation detection.

Introduction

Genetic variations are generally grouped by size and class, and all forms have been associated with disease. Smaller “sequence-level” variations include single-nucleotide variations (SNVs) and insertions/deletions < 50 bp (indels).¹ Larger “structural variations” (SVs) are comprised of copy-number variations (CNVs—unbalanced changes from 50 bp to entire chromosomes), balanced translocations, inversions, and combinations thereof. Individual genome sequences have ∼3.2 million SNVs, ∼760,000 indels, tens of thousands of SVs, and very rarely larger balanced structural changes when compared with the human reference genome.^{2, 3, 4, 5, 6, 7}

In medical genetics, detection of large genomic changes like trisomy 21 has been accomplished by karyotyping for 50 years.^{8, 9} Subsequently, chromosomal microarrays (CMAs) became the first-tier clinical test for detecting disease-causing CNVs,¹⁰ followed by exome sequencing for finding SNVs and indels affecting the <2% of the genome that encodes genes.¹¹ Whole-genome sequencing (WGS) promises to eventually supplant combination genetic testing using karyotyping, CMAs, and/or exome sequencing, since, in principle, it covers all types of genetic variations. Further, it provides higher diagnostic yields than targeted panels,¹² can reduce health-care costs by providing earlier diagnoses and reducing the need for other tests,¹³ and may have benefits for the primary care of healthy individuals.¹⁴

For extracting SNVs and indels from short-read WGS data (the current industry standard¹⁵), there is already a well-accepted “best practices” workflow, which involves mapping reads using Burrows-Wheeler Aligner (BWA)¹⁶ and detecting variations using the Genome Analysis Toolkit (GATK).^{17, 18, 19} In contrast, there are >50 heterogeneous algorithms for detecting SVs from sequence data (Figure S1).^{20, 21} The large number of methods, combined with the distribution of their citations (the most-cited algorithm has <12% of the total citations; Figure S2), show that there is currently no widely accepted method for identifying SVs from WGS data. The absence of best practices for identifying SVs extends to other steps in the workflow, including DNA library preparation, sequencing, reference mapping, and quality control.

Here, we describe the development and evaluation of a detailed workflow for the read depth-based identification of germline CNVs from short-read WGS data. We first focus on CNVs because they comprise the majority of SVs, which has led to CMAs being used worldwide in diagnostic settings.^{10, 22, 23, 24} We restrict our evaluation to algorithms that are primarily based on read depth because (1) they can be analyzed cohesively (i.e., we expect them to exhibit similar issues, such as with amplification bias in sequence data from PCR-based DNA library preparations); (2) they are conceptually the most direct analog for CMAs; (3) there is a wealth of CMA data for comparison and benchmarking; and (4) they are the method of choice for detecting larger CNVs, which are more likely to be clinically relevant. (Strategies for detecting smaller CNVs and other types of SVs based on paired-end mapping or split reads will be the focus of a separate study.) Although our workflow does not detect all possible CNVs (we restrict the analysis to CNVs > 1 kb, as these are most amenable to detection by read depth), it comprehensively addresses all stages of the process of detecting CNVs from short-read WGS data based on read depth. Specifically, in addition to evaluating the accuracy of read depth-based CNV detection algorithms, we investigate the effects of DNA library preparation (including insert size and PCR-based versus PCR-free protocols), sequencing depth, algorithm parameters, read-mapping software, and choice of reference genome. We also explore filtering strategies, quality-control metrics, and deletion breakpoint accuracy. Using data from a large cohort of individuals with autism spectrum disorder (ASD) and their family members, we demonstrate that our workflow has a low false discovery rate (FDR) for clinically relevant CNVs.

Material and Methods

Study Subjects

HuRef DNA was the same used for the original sequencing and assembly of the HuRef genome.³ DNA from individual NA12878²⁵ was purchased from Coriell. In addition, 43 DNA samples were from participants in the Personal Genome Project Canada (PGPC), 3,369 from participants in the Autism Speaks MSSNG autism WGS project (1,846 individuals with ASD and 1,523 parents of those individuals),²⁶ 111 from pediatric development research (unpublished data), and 8 from children referred for pediatric clinical genetics testing.²³ All DNA was blood derived. All samples analyzed in this study were collected under approved protocols through The Hospital for Sick Children and its Research Ethics Board.

DNA Library Preparation and Sequencing

DNA was quantified using the Qubit dsDNA HS Assay and sample purity checked by OD₂₆₀/OD₂₈₀ ratio using a Nanodrop. For PCR-based libraries, 100 ng of DNA were used for the Illumina TruSeq Nano DNA library preparation following the manufacturer’s protocol. Briefly, DNA was fragmented to an average size of 350 bp on a Covaris S2 or LE220 instrument, end-repaired, and A-tailed. Indexed TruSeq Illumina adapters were added by ligation, followed by six or eight PCR cycles. For PCR-free libraries, 100, 500, or 1,000 ng of DNA were used for library preparation using either the TruSeq Nano DNA library preparation protocol (omitting the PCR step) or the Lucigen NxSeq AmpFREE Low DNA library preparation protocol, following the manufacturers’ instructions. All libraries were assessed using a Bioanalyzer High Sensitivity DNA Chip and quantified by qPCR using the KAPA Library Quantification Illumina/ABI Prism protocol (Kapa Biosystems). Validated libraries were pooled in equimolar quantities and sequenced on a HiSeq X following Illumina’s recommended protocol. DNA library preparation and sequencing were performed either by The Centre for Applied Genomics (TCAG) or Macrogen.

HuRef CNV Benchmark

Previously, our laboratory compiled an extensive dataset of variations in the HuRef genome^{4, 27, 28} using a number of technologies and strategies, both CMA based (Affymetrix Genome-Wide Human single nucleotide polymorphism [SNP] 6.0; Agilent 24M comparative genomic hybridization [CGH];²⁹ Nimblegen 42M CGH;³⁰ and Illumina BeadChip 1M) and sequencing based (comparison of de novo-assembled sequences; mate pair- and read depth-based analysis of data from Complete Genomics; and mate pair- and split read-based analysis of Sanger reads). While standard CGH microarrays typically detect CNVs ≥20 kb, the NimbleGen 42M and Agilent 24M have more probes and thus can identify CNVs as small as 500 bp.^{29, 30} Although sequencing based, the Complete Genomics variations can be considered orthogonal to those from Illumina WGS data, with different library preparation, sequencing chemistry, read size, alignment method, and variation-detection methods. For this study, we are interested only in CNVs, so other variations (e.g., insertions) were removed from the benchmark. Only CNVs ≥1 kb were retained, because three of the six CNV-detection algorithms tested did not report smaller CNVs (see below) and because smaller CNVs are more readily detected by strategies not involving read depth. Overlapping CNVs from the same technology were manually merged. CNVs based on mate-pair analysis of Sanger data were removed, as the breakpoints and CNV sizes could not be precisely determined. If the coordinates for a given benchmark CNV were relative to a reference assembly other than the one to which reads were mapped (GRCh37/h19 or GRCh38/hg38), then they were converted using the University of California Santa Cruz (UCSC) Batch Coordinate Conversion (LiftOver) tool.³¹ Because CGH microarrays rely on a reference sample, they are susceptible to falsely identifying duplications for regions deleted in the reference sample and vice versa.^{24, 32} Thus, we removed any benchmark CNV (CGH microarray or otherwise) that had ≥50% reciprocal overlap with an opposite-direction CNV of ≥30% frequency in either the Database of Genomic Variants Gold Standard³³ or in MSSNG parents (detected by both the CNVnator and Estimation by Read Depth with Single-nucleotide variants [ERDS] methods).²⁶ Files S1 and S2 contain the full HuRef benchmarks for GRCh37/hg19 and GRCh38/hg38, respectively.

NA12878 CNV Benchmark

A previously published benchmark of NA12878 variations³⁴ was downloaded (Web Resources) and filtered to retain only deletions ≥1 kb (File S3).

AK1 Reads and CNV Benchmark

HiSeq X reads from the DNA of individual AK1 were downloaded in FASTQ format from the NCBI Sequence Read Archive.³⁵ To make the average depth consistent with our HuRef WGS data, as well as with the depth at which genomes are typically sequenced, we used just one of the two available read sets (accession number SRA: SRR3602759). A benchmark of variations in the AK1 genome, derived by comparing the AK1 assembly with the human reference assembly, was obtained from the supplementary materials of Seo et al.³⁵ and filtered to retain only deletions ≥1 kb (File S4).

Preprocessing of Sequence Data

For all CNV-detection algorithms except Canvas, base-calling was performed using bcl2fastq2 v2.17.1.14 and reads were mapped to the reference genome using BWA-MEM v.0.7.12 (for GRCh37/hg19) or v.0.7.15 (for GRCh38/hg38).¹⁶ BWA-MEM v.0.7.15 was used for GRCh38/hg38 because it fixed an error related to ALT-aware mapping that was present in v.0.7.12; the output from the two versions is identical in the absence of ALT contigs (GitHub BWA webpage, Web Resources). Read quality was checked using FastQC prior to mapping. GRCh37/hg19 was used as the reference genome except where specified otherwise. Duplicate reads were marked using Picard v.1.133. Base quality score recalibration, indel realignment (no longer necessary for GATK v.3.6 and later; see Web Resources), and SNV and indel detection were performed with GATK v.3.4-46 using best practices recommendations.^{17, 18, 19} SNV and indel detection were performed because ERDS requires these variations in variant call format (VCF) as part of its input. We tested two mapping pipelines designed to handle contigs representing alternate haplotypes: one described by BWA author Heng Li and the other by the developers of GATK at the Broad Institute (Web Resources). For Canvas, HiSeq Analysis Software v.2.5.55.1311 (Illumina) was used for base calling and Isaac version iSAAC-SAAC00776.15.01.27 for alignment. For all alignments, binary alignment/map (BAM) file manipulation, including read subsampling, was performed using SAMtools v.0.1.19-44428cd.³⁶ Read-depth statistics were calculated using GATK’s DepthOfCoverage function or the “depth” function in SAMtools. Commands for performing these tasks, as well as scripts for performing other analyses described in this paper, are available on GitHub (Web Resources).

Definition of Repetitive and Low-Complexity Regions of the Human Genome

Both array- and sequencing-based methods of CNV detection can be confounded by repetitive and low-complexity regions (RLCRs).²⁴ To define a comprehensive set of RLCRs, we combined four datasets: (1) the set of assembly gaps defined by UCSC, which includes centromeres, telomeres, constitutive heterochromatin domains, gaps between or within clones and contigs, and the repeat-dominated short arms of chromosomes 13, 14, 15, 21, and 22; (2) the UCSC list of segmental duplications; (3) the pseudoautosomal regions of the sex chromosomes; and (4) repeat regions as defined by RepeatMasker version open-3-2-7 (see Web Resources) using the –s (sensitive) option with release 20090120 of library RepeatMaskerLib.embl. The number of regions present in each data source, as well as the total number of base pairs, is given in Table S1. Links to download the full RLCR definition, as well as a version that omits the RepeatMasker definitions, are provided in Web Resources.

Filtering out CNVs with Substantial Repeat Content

For generating filtered sets of CNVs, a given CNV was removed if ≥70% of the CNV overlapped with RLCRs. This overlap could be comprised of just a single continuous region or multiple regions. To illustrate, if a CNV spans coordinates 100,000–105,000 of chromosome 1 and two RLCRs span coordinates 100,000–102,000 and 103,000–105,000 of chromosome 1, then 80% of the CNV overlaps with RLCRs.

CNV-Detection Algorithms

The CNV-detection algorithms tested were Canvas v.1.3.5,³⁷ Copy Number estimation by a Mixture Of PoissonS (cn.MOPS) v.1.16.2,³⁸ CNVnator v.0.3.2,³⁹ ERDS v.1.1,²⁰ Genome STRucture in Populations (Genome STRiP) v.2.00.1696,⁴⁰ and Read Depth eXplorer (RDXplorer) v.2.0 release 3.⁴¹ Seven other read depth-based algorithms (Figure S2) were considered but not tested because they were specific to certain experimental designs (e.g., matched tumor-control samples) or had no implementation available. All algorithms were initially evaluated using default parameters. Genome STRiP contains two separate, complementary pipelines, but we evaluated only the read depth-based “CNV Discovery Pipeline” using parameters recommended for genomes sequenced at 30–40× average depth (Web Resources). Both cn.MOPS and Genome STRiP require multiple genome sequences as input. For cn.MOPS, the authors recommend ≥6 genomes,³⁸ while the authors of Genome STRiP recommend ≥20 (Web Resources); thus, we conservatively used 30 genome sequences from MSSNG participants (in addition to the genome sequence of interest) as input to both algorithms. CNVs from Genome STRiP that overlapped one another were merged to form a single CNV. Supplementary alignments were removed from BAM files prior to running ERDS.²⁰ The raw output files from all six CNV-detection algorithms for the HuRef, NA12878, and AK1 genomes are included in File S5.

Concordance of CNV-Detection Algorithms

To determine sets of equivalent CNVs, CNVs were iteratively added to sets as follows. For each possible pair of CNVs (where each CNV in a pair was identified by a different algorithm or HuRef benchmark method), we determined whether they satisfied the 50% reciprocal overlap criterion. If so, and if neither CNV was yet in a set, then a new set was created including both CNVs. If only one of the CNVs was already in a set, then the other CNV was added to it. If both CNVs were already in different sets (a rare “conflict”), then they were left as such. When determining how many unique benchmark CNVs fell into each size bin, the size of a particular unique benchmark CNV was calculated as the median of the sizes of all the CNVs in the corresponding set. The overall similarity of two sets of CNVs A and B, where A and B were detected by two different methods, was measured using the Jaccard index $J=\left(A\cap B/A\cup B\right)$. As not all algorithms reported absolute copy number, it was not taken into account when comparing CNVs.

Accuracy of CNV-Detection Algorithms

The accuracies of the CNV-detection algorithms were evaluated according to their sensitivity and FDR when compared with a benchmark of known CNVs (HuRef, NA12878, or AK1 benchmarks). A CNV detected by an algorithm and a benchmark CNV were deemed to be the same if the first CNV overlapped with at least 50% of the second CNV and vice versa (50% reciprocal overlap).⁴² Sensitivity was defined as TP / (TP + FN), where TP (“true positives”) is the number of benchmark CNVs identified correctly and FN (“false negatives”) is the number of benchmark CNVs not detected by the algorithm. FDR was defined as FP / (FP + TP), where FP (“false positives”) is the number of identified CNVs not found in the benchmark. For the HuRef benchmark, which contains CNVs identified by multiple technologies and strategies, sensitivity was determined for CNVs having 50% reciprocal overlap with a benchmark CNV from at least one technology (denoted “1+”) or at least two technologies (“2+”).

Deletion Breakpoint Verification Assays

PCR primers were designed to span the putative breakpoint and the deletion-specific PCR product was sequenced by Sanger sequencing.

Read-Depth Uniformity

Read-depth uniformity was measured by the inter-quartile range (IQR). Let R_a denote the read depth for which a% of the bases in the reference genome have smaller depths. Then IQR = $R_{75}-R_{25}$. The GitHub repository for this paper (Web Resources) provides scripts for calculating IQR from BAM files.

Visualizations

Set intersections were visualized using UpSetR v.1.3.3.⁴³ Circle diagrams were created using circos v.0.69-3,⁴⁴ and other plots were generated using ggplot2 v.2.2.1.⁴⁵

Detection of Rare, Genic Variants in the Autism Speaks MSSNG WGS Dataset

Two algorithms, ERDS and CNVnator, were used for detecting CNVs in the MSSNG WGS dataset. Both programs were run using default parameters. For CNVnator, CNVs with q0 > 50 were removed except for homozygous deletions or hemizygous X-linked deletions in males (normalized read depth < 0.03). Adjacent CNVs of the same type were merged when separated by a gap in the genome assembly or separated by a small region (sum of the lengths of individual CNVs > 70% of the merged CNV length). The ERDS and CNVnator CNVs were merged independently and the merged ERDS CNVs were annotated with CNVnator CNVs having 50% reciprocal overlap.

BAM Confirmation

To manually assess the accuracy of predicted CNVs, Integrative Genomics Viewer⁴⁶ was used to visually compare the read depth of the CNV with that of the surrounding regions, with specific attention to deviations in read depth corresponding to the predicted change in copy number (e.g., a 50% reduction for a heterozygous deletion or a 50% increase for a heterozygous duplication). Predicted CNVs with clear start and end breakpoints and that were supported by split-read and/or read-pair information were deemed more likely to be correct. A more detailed description of how we used BAM confirmation to evaluate putative CNVs is given in the Supplemental Tutorial.

Results

An overview of this study is given in Figure 1.

Figure 1.

Overview of the Three Stages of This Study

In stage 1 (“algorithm selection”), three WGS datasets and corresponding CNV benchmarks (HuRef,^{3, 4, 28} NA12878,³⁴ and AK1³⁵) were used to assess the accuracy of six read depth-based CNV-detection algorithms—Canvas, cn.MOPS, CNVnator, ERDS, Genome STRiP, and RDXplorer. In stage 2 (“workflow development”), other factors influencing CNV detection were evaluated in the context of the most accurate algorithms identified in stage 1. Based on results from the first two stages, we propose a comprehensive workflow for detecting CNVs from short-read WGS data. In stage 3 (“workflow evaluation”), we show that our workflow can accurately identify clinically relevant CNVs. Green parallelograms represent data, and gray rectangles represent actions. The blue shape represents the CNV detection workflow developed from the results of the first two stages.

Stage 1: Algorithm Selection

Concordance of CNV-Detection Algorithms

We determined the concordance in the CNVs detected by the six CNV-detection algorithms when applied to WGS data from the HuRef-Free-500 DNA library. File S6 contains information on this library and the other libraries used in this study, including library preparation methods and common sequencing metrics. The algorithms exhibited only moderate agreement in terms of number (Table 1), size distribution (Figure S3), and overlap (Figures 2, S4, and S5) of detected CNVs. Concordance was higher for deletions than for duplications (Figure S5). The low concordance was unsurprising given that different algorithms for SNV and indel detection also exhibit modest agreement,⁴⁷ and similarly with different CMA platforms and algorithms.⁴²

Table 1.

Sensitivities and False Discovery Rates (FDRs) of the Six CNV-Detection Algorithms when the CNVs Identified in the HuRef Genome Were Compared with Those in the HuRef CNV Benchmark

	Unfiltered				Filtered
	n	Sensitivity		FDR	n	Sensitivity		FDR
Algorithm		1+	2+			1+	2+
Deletions
Canvas	493	0.23	0.44	0.56	253	0.35	0.64	0.53
cn.MOPS	386	0.16	0.28	0.62	93	0.19	0.38	0.33
CNVnator	2,093	0.35	0.57	0.84	356	0.44	0.76	0.59
ERDS	680	0.39	0.69	0.46	251	0.51	0.89	0.32
Genome STRiP	582	0.25	0.42	0.60	241	0.37	0.68	0.49
RDXplorer	12,685	0.55	0.76	0.96	3,232	0.66	0.94	0.93
Duplications
Canvas	225	0.06	0.62	0.89	56	0.11	0.67	0.77
cn.MOPS	421	0.05	0.38	0.95	61	0.08	0.33	0.84
CNVnator	337	0.06	0.33	0.93	27	0.07	0.33	0.70
ERDS	303	0.07	0.50	0.91	48	0.09	0.53	0.77
Genome STRiP	499	0.03	0.21	0.98	110	0.03	0.20	0.96
RDXplorer	7,500	0.10	0.46	1.00	3,052	0.15	0.47	0.99

For each algorithm, we determined the total number of CNVs detected (n), the number identified by at least one method in the HuRef benchmark (1+), and the number identified by at least two benchmark methods (2+). The algorithms’ accuracies varied substantially, both in terms of sensitivity (the proportion of benchmark CNVs identified by an algorithm) and FDR (the proportion of CNVs identified by an algorithm that were not in the benchmark). “Unfiltered” refers to all CNVs ≥1 kb, while “filtered” refers to CNVs that had <70% overlap with repetitive and low-complexity regions of the genome. The apparent superiority of Canvas for duplications is somewhat misleading, as it was the most accurate for only some sizes of CNVs (shown subsequently).

Figure 2.

Overlap in the CNVs Detected by the Six Algorithms

The bottom-left bar chart shows the number of CNVs identified by each algorithm. The remainder shows the number of CNVs detected by various intersections of the algorithms; for instance, the far-left bar for deletions represents the number of CNVs detected by RDXplorer only, while the far-right bar represents deletions detected by Canvas, cn.MOPS, CNVnator, and RDXplorer but not ERDS or Genome STRiP. Due to the log scale, zero-height bars represent a count of 1.

Accuracies of CNV-Detection Algorithms

The CNVs detected by each algorithm in the sequencing data from the HuRef-Free-500 library were compared with those in the HuRef CNV benchmark (Tables S2 and S3) using both unfiltered CNVs and filtered CNVs (those with <70% overlap with RLCRs). For unfiltered deletions, the most accurate algorithm was ERDS, which had high sensitivity and a low FDR (Table 1; Figure S6). For unfiltered duplications, ERDS and Canvas had the highest sensitivities (although much lower than the highest sensitivities for deletions), and FDRs were uniformly high. Accuracies for filtered CNVs were consistently better than for unfiltered CNVs, and the algorithm rankings remained the same. Thus, subsequent analyses use only filtered CNVs unless specified otherwise.

Detection was more accurate for larger CNVs (Table S4). For deletions, ERDS was the most accurate in all size bins. The most accurate algorithm for duplications varied: Canvas for 1–5 kb, cn.MOPS for 5–10 kb, and ERDS for 10–100 kb.

Combining Algorithms

We tested a scoring procedure in which a particular CNV was considered to be detected only if it was identified by at least m algorithms. Overall, this strategy was no better than the most accurate individual algorithm alone (Tables 1 and S5). Considering deletions, the most accurate pair of algorithms was CNVnator and ERDS, which had a lower FDR than ERDS alone, but lower sensitivity (Table 1; Figure S7). For duplications, several pairs of algorithms had lower FDRs than any individual one, but again at the expense of sensitivity.

Additional Genomes and Benchmarks

To ensure the HuRef results were representative of genomes in general, we used the above strategies to analyze deletions detected from Illumina HiSeq X reads from the NA12878 and AK1 genomes (Tables S6 and S7). The results were highly concordant with those from the HuRef genome and benchmark (Tables S8–S11).

Reproducibility of CNV Detection

To evaluate reproducibility, we analyzed monozygotic twins and their parents. Two metrics were used: percentage of CNVs detected in both twins (“twin concordance”) and percentage consistent with Mendelian inheritance (“parent concordance”). Most algorithms exhibited high concordance (≥80%) for both filtered and unfiltered deletions ≥5 kb (Tables 2 and S12)—better than the reproducibility of CMAs.⁴² Lower concordances were observed for deletions <5 kb and duplications.

Table 2.

Reproducibility of the Filtered CNVs Identified in the Genomes of a Mother, Father, and Their Monozygotic Twins

	[1 kb,5 kb)		[5 kb,10 kb)		[10 kb,100 kb)		[100 kb,1,000 kb)		[1,000 kb,…)
Algorithm	n	%	n	%	n	%	n	%	n	%
Deletions: Twin Concordance
Canvas	26	42.3	38	55.3	32	68.8	4	25.0	18	55.6
cn.MOPS	54	61.1	34	91.2	19	73.7	0	N/A	0	N/A
CNVnator	264	39.0	49	83.7	31	80.6	2	100.0	0	N/A
ERDS	303	55.1	39	84.6	24	87.5	2	100.0	0	N/A
Genome STRiP	484	35.5	26	88.5	20	70.0	1	100.0	0	N/A
RDXplorer	33,849	17.8	104	47.1	30	93.3	1	0.0	0	N/A
Deletions: Parent Concordance
Canvas	26	53.8	38	63.2	32	87.5	4	25.0	18	61.1
cn.MOPS	54	68.5	34	85.3	19	89.5	0	N/A	0	N/A
CNVnator	264	54.9	49	91.8	31	93.5	2	100.0	0	N/A
ERDS	303	63.0	39	92.3	24	100.0	2	100.0	0	N/A
Genome STRiP	484	42.1	26	84.6	20	75.0	1	100.0	0	N/A
RDXplorer	33,849	38.8	104	61.5	30	93.3	1	100.0	0	N/A
Duplications: Twin Concordance
Canvas	22	22.7	12	50.0	13	84.6	0	N/A	0	N/A
cn.MOPS	29	62.1	13	69.2	10	60.0	0	N/A	0	N/A
CNVnator	0	N/A	10	60.0	14	92.9	3	100.0	0	N/A
ERDS	19	63.2	13	61.5	11	81.8	4	100.0	0	N/A
Genome STRiP	79	55.7	29	65.5	21	76.2	0	N/A	0	N/A
RDXplorer	19,753	1.2	21	57.1	31	80.6	1	0.0	0	N/A
Duplications: Parent Concordance
Canvas	22	36.4	12	58.3	13	76.9	0	N/A	0	N/A
cn.MOPS	29	51.7	13	53.8	10	90.0	0	N/A	0	N/A
CNVnator	0	N/A	10	70.0	14	85.7	3	100.0	0	N/A
ERDS	19	73.7	13	76.9	11	81.8	4	100.0	0	N/A
Genome STRiP	79	68.4	29	75.9	21	71.4	0	N/A	0	N/A
RDXplorer	19,753	3.6	21	76.2	31	80.6	1	100.0	0	N/A

n refers to the number of CNVs in a given size bin that were detected in at least one of the twins. “%” refers to the percentage of those CNVs that were also detected in the other twin (“twin concordance”) or that were also detected in at least one of the parents (“parent concordance”).

CNVnator and ERDS Excel

Overall, ERDS was the most accurate algorithm for deletions (Table 1). Although the ranking of the remaining algorithms for deletions was less clear, CNVnator exhibited several advantages over its closest competitors (Canvas and Genome STRiP): it had particularly strong sensitivity for deletions ≥5 kb, making it complementary to ERDS; its reproducibility was generally better than Canvas and Genome STRiP; and it is comparatively easy to use and has modest compute time and storage requirements.

Given the lesser accuracy for duplications among all algorithms, and that sample sizes were smaller, there was insufficient evidence to justify using additional algorithms beyond CNVnator and ERDS. Thus, in stage 2 of this study, CNVnator and ERDS are used as the basis for examining other steps in the CNV-detection workflow.

Stage 2: Workflow Development

CNVnator and ERDS Parameters

We investigated two parameters potentially influencing the sensitivity/FDR tradeoff of CNVnator and ERDS: the window size and fraction of multi-mapping reads (q0). For filtered CNVs, there was limited or no benefit to modifying default parameters, but for unfiltered CNVs, using a q0 cutoff of ∼0.5 for CNVnator gave an excellent sensitivity-FDR tradeoff versus no cutoff (Figures S8–S10).

Read Depth

The optimal sensitivity-FDR tradeoff was achieved at average depths of 25–40× for both CNVnator (Figure S11) and ERDS (Figure S12); however, only CNVs 1–5 kb were particularly sensitive to depth.

Sequence Read Mapping

To determine how CNVnator and ERDS are affected by different read-mapping software, we tested them using alignments produced by Isaac⁴⁸ and NovoAlign (Web Resources) but found they were most accurate using BWA¹⁶ (Figure S13).

Library Preparation Methods

We generated four HuRef DNA libraries—two using PCR-free protocols and two using PCR-based protocols. All were similar in terms of typical quality metrics (File S6); however, the WGS data from the PCR-based libraries exhibited lower read-depth uniformity (RDU), which appeared to cause the detection of false CNVs (Figure S14). Regions of high or low depth were consistent in the two PCR-based libraries, suggesting a role for systematic PCR biases. Read depth in the PCR-free libraries was largely unrelated to guanine/cytosine (GC) content, while they were positively correlated (except at high GC content) in the PCR-based libraries (Figure S15). CNVnator had a substantially higher FDR for deletions in the PCR-based libraries (Figure S16).

Quality-Control Metrics

To investigate quality-control metrics and to further compare PCR-based with PCR-free libraries, we used DNA from 43 Personal Genome Project Canada (PGPC) participants. For 22 participants, a single library was sequenced (11 PCR-based and 11 PCR-free); for the remaining 21, both a PCR-based and a PCR-free library were sequenced. For each WGS dataset, we determined its RDU as measured by IQR. All WGS datasets with high IQR (≥15) were generated from PCR-based libraries and had an excess of CNVs detected by CNVnator (and often ERDS) (Figure S17). Despite the high false positive rate for CNVs detected by ERDS and CNVnator alone in WGS datasets with high IQR, these datasets had very high false negative rates when considering CNVnator and ERDS in combination (i.e., there were very few CNVs detected by both algorithms; Figure S17), further highlighting the poor quality of the sequencing data and the resulting CNVs. Overall, both IQR and CNV count are useful quality metrics—the accuracy of read depth-based algorithms on WGS datasets with high IQR is problematic, even in the absence of aberrant CNV counts; further, if excess CNVs are detected by ERDS or CNVnator (or very few CNVs are detected by both), then it is reasonable to assume that the data are of poor quality regardless of IQR. Based on Figure S17, samples with more than ∼400–500 filtered CNVs detected by either ERDS or CNVnator, or fewer than 100 CNVs detected by both, could reasonably be categorized as outliers. In the future, it would be desirable to develop a statistically rigorous method of detecting outliers in CNV count data, especially for clinical purposes. However, such a method would undoubtedly be complex—due not only to the inherent difficulty of outlier detection, but also due to ethnicity considerations^{49, 50} and because counts from ERDS and CNVnator appear to follow different statistical distributions (ERDS appears to be Gaussian, whereas CNVnator appears to be Poisson). In the absence of such a method, and in the presence of larger datasets, we suggest performing the analysis depicted in Figure S17 to intuitively identify reasonable cutoffs.

Effect of Alternate Haplotypes in the Reference Genome

Unlike GCRh37/hg19 (used throughout this study), GCRh38/hg38 contains “ALT contigs” representing alternate haplotypes, which could interfere with read depth-based CNV detection by artificially sequestering reads from the primary assembly. This is best handled using ALT-aware workflows (Figures S18–S20), with differences between builds mainly consisting of more false duplications detected when using GCRh38/hg38.

Other Technical Issues

In running ERDS on various WGS datasets, we observed that the number of deletions (but not duplications) detected depended on the median insert size of the DNA library (Figure S21) and that this problem could be best corrected via a simple modification of ERDS (Figure S22). In addition, ERDS consistently detected more deletions in WGS data from v.2.0 of the HiSeq X flow cell than v.2.5 (Figure S21), which were mostly false (shown subsequently).

Deletion Breakpoint Accuracy

We selected 18 deletions between 3 and 468 kb identified by both CNVnator and ERDS and verified their breakpoints by Sanger sequencing. ERDS’ breakpoints were generally close to the experimentally determined ones, while those determined by CNVnator were substantially less accurate (Table S13).

Best Practices Workflow

A “best practices” workflow based on the above observations is given in Figure 3.

Figure 3.

Recommended Workflow for Use of Read Depth-Based Algorithms for Detecting Germline CNVs from Short-Read WGS Data

The green and blue shapes represent the beginning and end of the workflow, respectively. Red rectangles represent quality-control steps, and other actions are colored in gray. Yellow diamonds represent decision points. For maximum stringency, the action “Remove CNVs with ≥70% overlap with RLCRs” may be performed using the full RLCR definition, including RepeatMasker (as in the algorithm selection and workflow development sections). For increased sensitivity, such as when examining rare, genic CNVs, it may be performed using the RLCR definition that omits RepeatMasker, as was done in the workflow evaluation section.

Stage 3: Workflow Evaluation

Identification of Clinically Relevant CNVs

To test our workflow (Figure 3) for CNV detection in medical genetics (where CMAs are the standard), we identified rare (<1% frequency in unrelated, unaffected individuals²⁶ and absent from the Database of Genomic Variants³³), genic CNVs detected by both CNVnator and ERDS in HiSeq X WGS datasets from 1,846 individuals with ASD.²⁶ Preliminary analysis suggested that for rare, genic CNVs, filtering based on RepeatMasker reduced sensitivity slightly (by approximately 5%) and did not reduce FDR, so we filtered out CNVs only if they had ≥70% overlap with the non-RepeatMasker portion of our RLCR definition. The resulting CNVs were grouped into three categories: (1) ≥3 Mb, (2) representing a known genomic disorder, or (3) overlapping putative ASD-risk genes (File S7). We experimentally tested all CNVs in the first two categories (n = 77) and a subset of those in the third (n = 43/313 total) using either CMAs or PCR-based methods, and all were successfully confirmed, giving an FDR of 0% (Table 3; File S8). The high confirmation rate for duplications differs from the high FDR observed in our HuRef analysis (Table 1), indicating that the detection of rare, genic duplications may be more reliable than duplications in general. In addition, all clinically relevant CNVs identified by CMA (≥20 kb) were detected via WGS.

Table 3.

Confirmation of Potentially Clinically Relevant CNVs Detected by Both CNVnator and ERDS in the Genomes of Participants in the Autism Speaks MSSNG WGS Project

	≥3 Mb	ASD-Risk Genes	Genomic Disorders
# deletions identified	10	143	17
# deletions successfully confirmed/total tested	10/10	27/27	17/17
# duplications identified	16	170	34
# duplications successfully confirmed/total tested	16/16	16/16	34/34

CNVs that were both ≥3 Mb and represented a genomic disorder were assigned to the latter category. Most CNVs were tested as part of another study,²⁶ and the remainder were tested specifically for this study.

To evaluate our workflow’s ability to accurately detect clinically relevant CNVs in the sub-CMA size range, we used the same cohort and selected rare CNVs <20 kb detected by both CNVnator and ERDS that overlapped disease-associated genes from the Clinical Genomics Database (CGD), resulting in 41 deletions and 30 duplications. Although these CNVs are not expected to contribute to ASD risk, we deemed them representative of pathogenic CNVs. All CNVs appeared correct based on “BAM confirmation” (manual read pileup inspection) (File S8), and 6/6 deletions and 2/2 duplications selected for PCR confirmation were deemed correct.

As another test, we used the HiSeq X to sequence the DNA of eight children referred for pediatric clinical genetics testing, from which nine clinically relevant CNVs between 300 kb and 90 Mb had previously been detected using clinical CMAs and Complete Genomics WGS (Table 2 of Stavropoulos et al.²³). All nine CNVs were detected by our workflow, with breakpoints close to those of the other technologies (Table S14).

ERDS Plus CNVnator versus ERDS Alone

Our evaluations thus far did not clearly establish whether accuracy improved when using CNVnator in addition to ERDS (Table 1; Figure S7). We therefore evaluated the accuracy of rare, genic ERDS-only CNVs (i.e., <50% reciprocal overlap with a CNVnator CNV). ERDS-only deletions exhibited several indications of systematic biases (detailed in Figure S23), and only 8/40 ERDS-only deletions overlapping CGD genes selected for verification appeared to be real by BAM confirmation (File S8). ERDS-only duplications had more favorable confirmation rates (35/40 by BAM confirmation). Of the 80 ERDS-only CNVs described above, 14 were tested using PCR, and the PCR result matched the BAM result in all but one case (File S8). Finally, ERDS-only CNVs were more sensitive to technical confounders, as shown in the workflow development section. For this reason, we recommend using CNVnator and ERDS in combination. However, to improve sensitivity, ERDS-only duplications can be used as a “discovery set,” for which the correctness of any duplications of potential clinical or experimental importance should be verified either experimentally or using BAM confirmation.

De Novo CNVs

We identified genic CNVs that were detected by both CNVnator and ERDS in a given ASD-affected proband but neither algorithm in both parents. This was done for the 959 ASD-affected case subjects in the MSSNG dataset for which the proband and both parental samples were sequenced on HiSeq X (total corresponding parental samples = 1,523). Of 71 such CNVs, 50 were confirmed as de novo after BAM confirmation and experimental validation. Of the 21 that were not de novo, five were due to false positives in the proband and 16 were parental false negatives (File S8). Our detection rate for genic de novo CNVs (approximately one per 19 children with ASD) was similar to that of previous studies from our group and others, several of which used much more complex and labor-intensive sets of algorithms and methods than suggested here.^{51, 52, 53, 54} For falsely detected de novo CNVs that were due to parental false negatives, the raw read depth of the CNV region in one parent was almost always clearly different from the surrounding region (Table S15), suggesting an easily automatable method for filtering putative de novo CNVs. Scripts for comparing the read depth of a CNV with the surrounding region are provided on the GitHub page for this paper (Web Resources).

Small CNVs

Earlier, we showed that FDRs are generally higher for CNVs <5 kb (Tables S4, S10, andS11).^{30, 55} To determine whether this pattern applies to rare, genic CNVs, we specifically examined the CNVs analyzed above that were <5 kb. Three such deletions overlapping ASD genes and 18 overlapping CGD genes were tested, and all were successfully confirmed (File S8). Further, 11 of the putative de novo deletions were <5 kb, and all but two were successfully confirmed as CNVs in the proband (although five of these CNVs were missed in a parent, suggesting that—consistent with Table S4—sensitivity may be lower for small CNVs). There were no duplications <5 kb in these categories (both ERDS and CNVnator were required to have detected these CNVs, and CNVnator rarely detects duplications <5 kb); however, eight of the rare, genic, ERDS-only duplications were <5 kb, and all but one were successfully confirmed. The observed FDR for rare, genic CNVs <5 kb was thus quite low; nonetheless, given the higher FDRs observed for general CNVs in this size range, we recommend that small CNVs that appear to be clinically relevant be subjected to additional validation measures (BAM confirmation or laboratory testing).

Discussion

We have developed a robust workflow for applying read depth-based computational algorithms to short-read WGS data in order to identify all CNVs, and more, detected by CMAs. Coupling this workflow with the established best-practices pipeline for SNV and indel detection^{16, 17, 18, 19} positions WGS as a single experiment capable of replacing combined testing using CMAs and exome sequencing, and for the most part karyotyping. However, while our workflow is comprehensive in terms of the steps required to detect CNVs >1 kb using read depth, it does not yet address the detection of several other classes of variation. For instance, most CNVs are <1 kb in size,⁶ and these are more readily detected from short-read data using algorithms involving paired-end mapping and split reads. These strategies are also more suitable for detecting other types of structural variations, such as translocations and inversions. Additionally, our workflow undoubtedly misses some CNVs >1 kb, as evidenced by our own comparisons to CNV benchmarks (Table 1) and because long-read sequencing data detects some such CNVs not discovered by short-read data (though these are mostly <5 kb⁶). Finally, our workflow is suitable only for detecting deviations from diploid copy number (or for deviations from one copy on sex chromosomes in males). For the precise detection and characterization of multiallelic CNVs, more specialized methods are required, such as that developed by Handsaker et al.⁵⁶ Overall, further work is necessary to establish workflows that cover the full repertoire of human genetic variation.

In our recommended workflow (Figure 3), we have suggested redoing library preparation and sequencing when quality-control metrics such as IQR and CNV count are poor. However, we recognize that this may be impractical (e.g., due to cost or lack of DNA); thus, an important area of future work is developing methods for making the best use of suboptimal sequencing data.⁵⁷

The HuRef CNV benchmark that formed the core of our analysis has several advantages: the technologies used to generate it detect a wide spectrum of CNV sizes,⁴ it allowed us to identify high-confidence CNVs (those detected by multiple technologies), and it was not used in the development of the six algorithms evaluated, ensuring an unbiased comparison. The NA12878 and AK1 deletion benchmarks were useful for ensuring that our findings generalized to other genomes. However, a difficulty in this study was the lesser quality and quantity of duplication benchmark data.^{42, 58} The marked difference in accuracy between deletions and duplications suggests that the HuRef benchmark duplications are less complete and have a higher false-positive rate than the deletions (Table 1). We are unaware of any other complete duplication benchmark; however, we partially addressed this deficiency by showing that many clinically relevant duplications were detected with a low FDR in WGS data from ASD-affected individuals (Table 3). In the future, our ability to perform even more thorough testing of variation-detection methods will rely on the creation of additional comprehensive, multi-technology, well-validated benchmarks of all types of variations (especially duplications).^{34, 59}

This study also sheds light on limitations inherent in current methods for the read depth-based detection of CNVs from WGS data. Although our workflow has a low FDR, particularly for rare, coding CNVs (Table 3), there is room for improvement with respect to sensitivity (Tables 1, S4, and S15). Although we did not examine the detection of germline mosaic CNVs, imperfect sensitivity for heterozygous CNVs suggests that sensitivity for mosaic CNVs would be low. The window-based approaches used by most CNV-detection algorithms are not well suited to aneuploidy detection, as read depth comparisons are made only within chromosomes. More advanced algorithms that are less dependent on parameters, windows, and thresholds may be able to address the above issues, as well as others observed in this study (e.g., repeat content, insert-size dependency, and PCR-amplification noise).

In addition to improved algorithms, CNV and SV detection from WGS data will be aided by new sequencing and library preparation technologies that provide long reads or other mechanisms of obtaining long-range genetic information.¹⁵ Already, such technologies have been used to create high-quality genome assemblies and to catalog SVs that would be difficult to resolve using short reads.^{60, 61} However, so far these technologies are costlier, provide lower coverage, and have higher error rates than the industry-standard short-read technology, limiting broad applicability. Thus, our workflow promises to have impact in high-throughput clinical genomic^{12, 23, 26, 51, 62, 63} and population-based^{7, 64} studies for the foreseeable future.

Published: January 4, 2018

Web Resources

• BWA, https://github.com/lh3/bwa/releases

• BWA/GATK ALT-aware, https://github.com/lh3/bwa/blob/master/README-alt.md

• Clinical Genomic Database, http://research.nhgri.nih.gov/CGD/

• Database of Genomic Variants (DGV), http://dgv.tcag.ca/dgv/app/home

• FastQC, http://www.bioinformatics.babraham.ac.uk/projects/fastqc

• GATK, https://software.broadinstitute.org/gatk/documentation/article.php?id=8017

• GATK Blog, https://software.broadinstitute.org/gatk/blog?id=7712

• Genome STRiP, http://software.broadinstitute.org/software/genomestrip

• Genome STRiP CNV Discovery Pipeline, http://software.broadinstitute.org/software/genomestrip/org_broadinstitute_sv_qscript_CNVDiscoveryPipeline.html

• Human Genome Reference Consortium, https://www.ncbi.nlm.nih.gov/grc/human

• MSSNG, https://www.mss.ng

• NA12878 variation benchmark, ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/technical/svclassify_Manuscript/Supplementary_Information/Personalis_1000_Genomes_deduplicated_deletions.bed

• Novoalign, http://www.novocraft.com/products/novoalign

• PGP Canada, https://personalgenomes.ca

• RepeatMasker, http://www.repeatmasker.org

• Repeat Masker repeat definition, http://hgdownload.soe.ucsc.edu/goldenPath/hg19/bigZips/chromOut.tar.gz

• Repetitive and Low Complexity Region (RLCR) definition (complete), http://tcag.ca/documents/projects/RLCRs.zip

• Repetitive and Low Complexity Region (RLCR) definition (no RepeatMasker), http://tcag.ca/documents/projects/RLCRs_no_Repeat_Masker.zip

• Sequence Read Archive (SRA), http://www.ncbi.nlm.nih.gov/sra

• TCAG-WGS-CNV workflow, https://github.com/bjtrost/TCAG-WGS-CNV-workflow

• UCSC assembly gaps, http://hgdownload.soe.ucsc.edu/goldenPath/hg19/bigZips/chromAgp.tar.gz

• UCSC segmental duplications, http://hgdownload.soe.ucsc.edu/goldenPath/hg19/database/genomicSuperDups.txt.gz