Balancing efficient analysis and storage of quantitative genomics data with the D4 format and d4tools

Abstract

Modern DNA sequencing is used as a readout for diverse assays, with the count of aligned sequences (read depth) representing the quantitative signal for each underlying cellular phenomena. Existing data formats for quantitative genomics assays are, however, limited in either the analysis speeds they enable, the disk space they require or both. We have developed the dense depth data dump (D4) format and tool suite, with the goal of balancing improved analysis speeds with file size. The D4 format is adaptive in that it profiles a random sample of aligned sequence depth from the input sequence file to determine an optimal encoding that enables fast data access. We demonstrate that the D4 format offers substantial speed improvements over existing formats for random access, aggregation and summarization, while also achieving better or comparable file sizes. This performance enables scalable downstream analyses that would be otherwise difficult.

Given the use of DNA sequencing to study a wide range cellular phenomena, aligned DNA or complementary DNA sequence depth has become an essential quantitative signal used to study genetic variation¹, differential gene expression^2,3 and chromatin binding⁴, and for critical evaluations of data quality control⁵. For example, the count of cDNA reads aligned to a given gene is a proxy for the overall expression of that gene in the population of assayed cells. Similar quantitative measures derived from the count or depth of sequence alignments are used for scores of assays used in laboratories throughout the world.

Despite the ubiquity of quantitative genomics datasets, the two most widely used data formats for storing quantitative datasets, bigWig⁶ and bedGraph⁷, have limitations that inhibit analysis speed, require excessive storage or both. Each of these formats are designed to represent the depth of aligned DNA or cDNA reads at each position in the relevant genome. While the binary bigWig format provides a space-efficient solution for storing quantitative genomic data as a file, it requires considerable memory to construct the R-tree index it uses to enable queries of the underlying data. Furthermore, its complex file structure has limited the development of more accessible programming interfaces or improvements to the format. Despite its storage efficiency, analyses of quantitative signals from many genomic regions are rather slow with the bigWig format. At the other end of the spectrum is the widely used, bedGraph format which, owing to the fact that it is a simple, four-column text file, is easy to understand, yet both slow to parse and consumes substantial disk space. The hierarchical data format, version 5 (HDF5)⁸ format is a widely used strategy for representing large-scale quantitative datasets, but it is a general-purpose file format that doesn’t provide functionality that caters to the specific analytical demands of genomics research.

To provide improved analysis speed over existing file formats, we have developed the dense depth data dump (D4) format and software suite. The D4 format is motivated by the observation that depth values often have low variability, which enables a specialized strategy for encoding quantitative genomics data. We demonstrate that the D4 format’s encoding enables very fast analysis of the underlying data, while also providing file sizes that are better than or comparable to compressed bedGraph, bigWig⁶ and HDF5⁸ formats. The D4 format and associated tools support fast random access, aggregation, summarization and extensibility to future applications. These capabilities facilitate a larger scale of genomic analyses that would be otherwise slower.

Results

The D4 format.

Devising a disk- and computation-efficient file format to store quantitative data that can take on a very large range of finite values is a difficult task. Thankfully, the sequencing depths observed in modern genomics assays often have little variance, thus yielding a limited range of discrete values. For example, consider a human whole-genome sequence (WGS) assay yielding the typical target of 30-fold average read depth (that is, on average, each genomic base pair is sampled with 30 independent DNA fragments). In a typical WGS dataset, more than 99% of the observed depths fall between 0 and 63 (Fig. 1a). Therefore, it is possible to encode >99% of the data using only 6 bits per base, since 2⁶ equals 64. Similarly, more than 50% of genomic positions have a sequence depth of 0 in typical RNA-sequencing (RNA-seq) experiments, as a small portion of the coding genome is assayed. On the other hand, RNA-seq yields a much broader range of non-zero depths than WGS, reflecting the highly variable degree of isoform expression from gene to gene. Nonetheless, the range of observed values is both finite and redundant (Fig. 1b).

Fig. 1 |. Depth distribution for WGS and RNA-seq datasets.

a,b, The global depth histogram (grey) and cumulative percentage of bases (red) for whole-genome (a) and RNA-seq (b) data. For typical WGS datasets, more than 99% of observed depth values are <63, indicating that the majority of data can be encoded in 6 bits. The variance in observed depths is much greater for RNA-seq data and other quantitative assays; however, the range of values is finite and amenable to encoding. The WGS dataset is sample HG002 from the Genome in a Bottle Project, and the RNA-seq dataset is sample ENCFF976QSN from the ENCODE project¹⁶.

The D4 format utilizes an encoding scheme that takes advantage of the finite range of depth values observed in most genomics assays to enable storage efficiency (Fig. 2). D4 uses a dense array as a primary table where each position in a given chromosome uses k bits (Fig. 2a, where k = 6). The index i into this dense array provides a lookup for the chromosomal position i. That lookup is then used as a key into a k-bits code table mapping 2^k k-bit codes to the 2^k distinct values (Fig. 2b). We note that the code table could be used to encode other values besides sequencing depths (for example, labels, P values and so on), but the number of distinct codes in the code table is governed by the choice of k. Any value less than 2^k − 1 in the code table encodes the actual depth for position i. However, if the encoded value is exactly 2^k − 1, then either the value is encoded by the code 2^k − 1, or its precise value is stored in the sparse secondary table (Fig. 2c). Therefore, when a code 2^k – 1 is seen, the sparse secondary table must be queried. If this query of the secondary table does not return an entry for that position, one can conclude that the actual value is the value encoded by the code 2^k − 1. Otherwise the actual value is returned from the secondary query. In effect, if a value cannot be encoded to a k-bit code by the code table, the code 2^k − 1 is emitted to the primary table as a placeholder, and the actual value is stored in the sparse secondary table instead.

Fig. 2 |. The D4 format encoding strategy.

a, The D4 dense primary table. The D4 format uses a dense array as a primary table that contains one entry per base in each chromosome. Each array entry consumes k bits, and the values stored in each entry range from 0 to 2^k−1. In this hypothetical example, k is 6. b, The k-bits code table. After a D4 file is created and when one wants to learn the depth of coverage at a particular genome position, one looks at the code stored at the position in the primary table. If the value of the code is less than 2^k−1, then one can look up the actual value (for example, sequencing depth) in the k-bits code table. For example, the code for position 1,000,000 is ‘011011’, which is less than 2⁶−1 (63). Therefore, the code table is used to look up the encoding for ‘011011’, which, in this case, is 27. c, The sparse secondary table. There is more work to be done in cases where the code stored in the primary table is exactly equal to 2^k−1. This scenario indicates that either the value for that position is encoded in the last possible entry of the code table, or that the value exceeds the range of distinct values that can be encoded by the choice of k. To distinguish the two scenarios, one must first lookup the genome coordinate in the sparse secondary table. If, as in the case of coordinate 1,000,010, an entry does not exist in the secondary table (denoted by the red ‘stop sign’), then we can infer that the actual value can be determined by looking up the code in the code table. If, however, an entry does exist, as in the cases of coordinate 1,000,010 (depth 64, which exceeds the range of 0 to 2^k−1) and 1,000,020 (depth 100), then the value for that coordinate is stored directly in the secondary table.

For a typical 30-fold coverage WGS dataset, 99% of the observed depths are encoded in the primary table, allowing for rapid queries of observed depths at most genomic positions. Queries of the secondary table occur less than 1% of the time and therefore the sparse secondary table often contains less than 1% of the data. This efficiency improves query speeds and saves disk space as the primary table uses k bits per base, and the secondary table usually contains a small percentage of the data but uses 80 bits to store the combination of the chromosomal range and the observed depth at that range of positions. In this way, we estimate that chromosome 1 of the human genome, which contains 249 million bases, will consume 212 megabytes for a D4 encoding of WGS data, as (249,000,000 × 6 bits + (0.01 × 249,000,000 × 80 bits)) < 181 MiB. In contrast, if we were to store each depth as an unsigned 32 bit integer, 996 megabytes would be required for the same data. Therefore, we can achieve a 1:5.2 lossless encoding ratio with the D4 encoding strategy with k equal to 6. Usually for WGS datasets, the secondary table encodes only a small portion of the values, but for RNA-seq datasets the optimal k is 0, which means we have a zero-sized primary table and all the data are completely stored in the secondary table (discussed further below). To further reduce file size, D4 allows an optional compression for the secondary table. When this is enabled, all the secondary table records are compressed in blocks, which usually reduces the secondary table size by more than 50% for RNA-seq datasets.

The d4tools software.

The d4tools software suite provides a collection of tools for creating D4 files from BAM⁹, CRAM¹⁰ and bigWig input files, as well as tools for statistical summaries of DNA depth profiles in arbitrary genomic regions, and for creating genome-wide summaries of quantitative signals. Example commands for common operations to create, analyze and visualize D4 files are provided in Supplementary Table 1.

Adaptive encoding with the d4tools software.

The toy example in Fig. 2 assumes k is known ahead of time. However, for each dataset, there is an unknown, optimal k that maximizes the efficiency of the read depth encoding. This optimal k depends on the range and variance of depths observed in the dataset. The d4tools software we have developed to create D4 files samples the depth distribution to quickly determine the optimal k for a given dataset. The essential trade-off is that a smaller k necessarily means that each entry in the primary array will use fewer bits (Fig. 3). However, with a smaller k, a greater percentage of the observed depths will fall outside of the range supported by the code table given the choice of k (for example, depths 0 to 63 are supported by k = 6). Likewise, a larger k means that each base in the primary array will consume more bits, but that fewer values are needed in the sparse secondary table, which consumes 80 bits per entry. As mentioned, in the case of most 30-fold WGS datasets the optimal choice of k is 6 (Fig. 3a). However, the optimal choice of k for most RNA-seq datasets is 0. This is because the range of depths observed in RNA-seq data is wide enough that it’s not possible to choose a narrow range (small k) that encompasses the majority of the observed depths (Fig. 3b). When k becomes too large, each entry in the chromosome-length primary table will consume enough bits to offset any gain from the larger range.

Fig. 3 |. Optimizing the choice of k given the trade-off between the size of the primary and secondary tables.

a,b, For example WGS (a) and RNA-seq (b) datasets, the trade-off between the size of the primary and secondary tables k is varied. The optimal choice of k, which minimizes the total size of the resulting D4 file, is the point at which the black line has its lowest value. Each entry in the primary table will consume more bits as k increases, but the secondary table size will decrease. D4 finds the optimal k by randomly sampling the depth distribution from the input file. note that for RnA-seq (b), it is often optimal to have k of zero, indicating that only the sparse secondary table is used.

Enabling parallelism, modularity and extensibility in the D4 format.

The implementation of the D4 format uses a general data container that allows multiple, variable-length data streams to be appended to the container file in parallel (Methods). In particular, when a D4 file is being created, genome coordinates are split into small chunks. Each thread is responsible for several chunks of data, encoding and writing the data to the growing D4 file in parallel. The container file data structure is based on an unrolled list¹¹ with adaptive chunk sizes. When a D4 file is created, it uses the unroll list to allow data chunks to be added to the growing D4 file without blocking the other threads. When reading a D4 file, the unrolled list can be mapped to the main memory using the ‘mmap’ system call and manipulated with the central processing unit (CPU) single instruction, multiple data (SIMD) instruction set. D4 files are able to be extended without breaking any existing file format by adding more streams to the container file, for example, a precomputed depth statistic.

Computing depth of coverage.

To fully take advantage of the encoding efficiency of the D4 format, we developed an algorithm to efficiently profile the depth of coverage in input BAM files. We developed this algorithm because existing methods such as SAMtools⁹ ‘pileup’ consume memory in proportion to the maximum depth in a region. This approach is also slow in genomic regions exhibiting very high depth. The memory use for mosdepth¹², our previously published method for reporting per-base sequencing depth, is governed by the length of the longest chromosome, but, as implemented, it is not parallelized. In contrast, D4 introduces an algorithm that limits the memory dependency on depth and also facilitates parallelization of the coverage calculation. This algorithm uses a binary heap that fills with incoming alignments as it reports depth. The average time complexity of this algorithm is linear with respect to the number of alignments (see Methods for algorithm details and lower-bound analysis). Because there is little memory allocation, this algorithm can be parallelized by performing concurrent region queries in different threads. The algorithm uses the start and end of alignments such that it does not count, for example, soft-clipped regions, but it will also not drop the depth for internal CIGAR (Compact Idiosyncratic Gapped Alignment Report) events such as deletions. As a result, the d4tools ‘create’ command is able to encode new D4 files from DNA or RNA sequence alignments in BAM or CRAM files in less time than creating depth profiles with bigWig or other formats, especially when leveraging multiple threads.

Single-sample evaluation.

To compare D4 to existing solutions, we calculated aligned sequence depth profiles for the WGS and RNA-seq samples in Fig. 1 using the D4, BGZF¹³-compressed BedGraph, bigWig and HDF5 formats. We focus on the comparison with these formats because they are either designed specifically for quantitative genomics data or are used for this purpose in the research community. We did not compare the performance of D4 to specialized compression approaches as they are focused on reducing file size, rather than striking a balance between analysis speed and file size. We note however, that we did explore a comparison to the small-Wig¹⁴ format, but were unable to conduct such an analysis owing to software failures arising from a lack of support for datasets including more than 32 chromosomes or contigs.

In addition to comparing the file sizes for each approach, we evaluated the time required to create a depth profile from each aligned BAM file into the relevant file format, the time required to summarize the results in a full sequential scan of the entire file, and finally, the average time used to query the depth for a set of random genomic intervals (Fig. 4). For both WGS and RNA-seq datasets, D4 yielded a 10-fold faster file creation time, and, with the exception of the highly compressed HDF5 format, yielded the smallest file size. The high compression rate of the HDF5 format comes at the cost of much less efficient analysis times. Sequential access to the depth at every genomic position was 3.6 times faster than bigWig, 3.8 times faster than HDF5 with a single thread, and 31.5 and 32.4 times faster with 64 threads. Again using 1 and 64 threads, accessing 10,000 random genomic intervals is between 21.3 and 72.8 times faster than bigWig, between 130 and 446 times faster than block-GZIPed bedGraph, and between 18 and 64 times faster than HDF5.

Fig. 4 |. Performance of D4 compared with other formats.

a-d, The file size (a), total wall time for creation (b), sequential access (c) and random access (d) are reported for the WGS and RNA-seq datasets presented in Fig. 1. The ‘sequential access’ experiment iterates over each depth observed in the output file. Ten thousand random intervals of length 10,000 were used to assess the ‘random access’ time. Panels b-d depict eight experiments each, whereas a depicts the single file size of the file used for the experiments conducted in b-d. The point labels for b-d reflect the mean value of the eight experiments. All D4 files shown use a compressed secondary table. Comp. BedG, compressed BedGraph; Thrd, threads; NA, not applicable.

D4 also supports optional secondary table compression. For WGS datasets, the secondary compression usually doesn’t result in a noticeable performance and size difference, but for RNA-seq datasets, enabling the secondary table compression reduces the file size at the cost of modestly slower computation (Supplementary Fig. 1). Comparing a D4 file with the deflated and inflated secondary table, the D4 file is 54% smaller while the sequential access performance of a deflated D4 file is ~300% times slower with single thread and 44% slower with 64 threads. Similarly, the random access performance is slower by 17 times and 2 times. In practice, D4 files with deflated secondary tables can be used as a compact archive format for cold data and those with inflated secondary tables can be used as hot data format that allows high-performance data analysis.

Evaluation on large cohort.

To illustrate the types of analysis facilitated by the speed and efficiency offered in D4 format and tools, we performed an evaluation of depth on both the 2,504 samples from 1000 Genomes high-coverage whole-genome samples, and on 426 RNA-seq BAM files from the ENCODE project. We restricted our comparison to the bigWig format given its wide use in the genomics community, and the fact that its combined file size and performance are closest to that of D4. For WGS datasets, D4 files are consistently less than half the size of bigWig files and D4 file size is largely consistent across a wide range of input CRAM file sizes (Fig. 5a). As the mean depth of WGS datasets from the 1000 Genomes project increases, we observe a transition from 5 to 6 in the optimal choice of k, since the variance increases with the mean (Fig. 5a, inset). This results in a small increase in D4 file size, yet we observed a large jump in bigWig file size for WGS datasets with a mean depth near 60 or higher. Furthermore, we note that the male 1000 Genomes datasets had slightly larger D4 files than females owing to the fact that one X chromosome yields more positions with aligned sequencing depths that lie outside of the range encoded by k. Finally, while D4 files are much smaller than bigWig for WGS datasets, D4 and bigWig file sizes are much more similar for RNA-seq datasets (Fig. 5b).

Fig. 5 |. Comparison of file sizes and analysis times across multiple WGS and RNA-seq datasets.

a, Mean depth of the original alignment file (x axis) compared with the size of the resulting D4 and bigWig files (y axis). The inset depicts how the optimal choice of k changes as mean depth increases. b, Size distribution of the D4 and bigWig files converted from RNA-seq BAM files. c, Per-sample distribution of wall time required to identify regions of each genome greater than or equal to 1,000 base pairs where each position has a depth of coverage greater than 120. Of the D4 samples, 98.8% completed in under 2 s of wall time each. d, Per-sample distribution of wall time required to identify regions of each genome greater than or equal to 1,000 base pairs where each position has a depth of coverage greater than 10,000.

As an example of the type of practical analysis improved by the use of D4, we identify large regions in the WGS and RNA-seq datasets where samples had high coverage depths, as these often reflect atypically high-depth regions that result in incorrect variant calls¹⁵, and genes with high expression levels, respectively (Fig. 5c,d). We report the time to create each D4 file and the time to calculate the high-coverage regions. On average, D4 required only 3% of the time required by bigWig to conduct these analyses for the WGS datasets, and 18% of the time required by bigWig for the RNA-seq datasets. We emphasize that the D4 format is designed to enable parallel analyses and find that processing speed continues to increase even when 40 or more CPU cores are used (Supplementary Fig. 2).

Discussion

Looking forward, we emphasize that the flexible architecture of the D4 format will allow it to adapt to other applications such as representing signals from single-cell RNA-sequencing datasets where multiple layers of information (for example, cell ID, barcode and so on) are required. Furthermore, the D4 format’s analytical efficiency can also facilitate rapid data visualization for genome browsers, as well as rapid statistical analyses and dataset similarity metrics that leverage additional indices or precomputed metrics stored in the D4 format. For example, we have already incorporated D4 output into our depth-profiling tool, mosdepth¹², which minimizes run time and makes mosdepth’s output more amenable to custom downstream analyses.

Currently, the d4tools software provides functionality allowing researchers to store integer depths of DNA sequencing coverage profiles from a single dataset BAM, CRAM and bigWig formats. Future development will seek to provide support for sequence depth profiles from multiple samples in a single D4 file, as well as support for both discrete and continuous measurements.

Nonetheless, given its current speed, flexibility and the availability of associated Rust and Python programming interfaces, we anticipate that the D4 format will enable large-scale genomic analyses that remain challenging or intractable with existing formats.

Methods

D4tools implementation.

The d4tools software suite is written in the Rust programming language to facilitate safe concurrency and performance. Our implementation takes advantage of Rust’s trait system, which allows zero-cost abstraction and the borrow checker to ensure correctness of the memory management. More importantly, Rust provides a strong multithreading safety guarantee that allows us to implement D4 in a highly parallelized fashion. We expose an application programming interface for both the C and Python programming languages so that researchers can easily utilize the library.

Depth-profiling algorithm.

current = next_read(); 
read_heap = new_heap(); 
for pos in 0..chrom_size {
if current!= EMPTY && pos > current.start_pos { 
read_heap.insert(current.end_pos); 
current = next_read();
}
while read_heap[0] < pos { 
read_heap.pop();
}
report(read_heap.size);
}

For each iteration, the binary heap operation takes $\mathcal{O}(\log D)$ running time where D is the current read depth, thus, we can prove that:

$$T=\mathcal{O}\left(N\log D_{\max }\right)$$

where T is the running time, N is the genome size and D_max is the maximum depth. Now we prove a tighter bound for it.

For the theoretical lower bound, we should at least enumerate all the reads in the alignment file since we need to both scan the alignment file and report the per-base values even in the worst case. Therefore:

$$T\ge \mathcal{O}(S+N)$$

where S is the total number of reads in the alignment file. The accurate running time of our algorithm is:

$$T=\mathcal{O}(N+S)+k\sum _{\text{pos=0}}^S\log d_{\text{pos}}$$

where d_pos is the depth at position pos. Applying the inequality of arithmetic and geometric means, we can show that:

$$\begin{gathered} T=\mathcal{O}(N+S)+kS\log {\left(\prod _{\text{pos}=0}^S{d}_{\text{pos}}\right)}^{\frac{1}{s}} \\ \le \mathcal{O}(N+S)+kS\overline{d_{\text{pos}}} \\ =\mathcal{O}(N+S)+\mathcal{O}(N)=\mathcal{O}(N+S) \end{gathered}$$

which means that our depth-profiling algorithm is no worse than the theoretical upper bound, so that:

$$\mathcal{O}(S+N)\le T\le \mathcal{O}(S+N)$$

Therefore, we prove that:

$$T=\mathrm{\Theta }(S+N)$$

It’s possible to (1) simplify the alignment in the read_heap to a single end position and (2) collapse all the alignment ends at the same location. By doing so, the upper bound of memory efficiency is:

$$M=\mathcal{O}\left(\max \left(L,D_{\max }\right)\right)$$

where L is the maximum of the read length.

Thus, this algorithm is optimal in terms of time complexity while space complexity is better than the counter array approach. In turn, this algorithm allows the BAM file to be processed with multiple processors at the same time with limited memory usage.

Data availability

The 1000 Genomes data were downloaded from http://ftp.1000genomes.ebi.ac.uk/vol1/ftp/data_collections/1000G_2504_high_coverage/working/20190425_NYGC_GATK/. Bulk RNA-seq samples were downloaded from the ENCODE project website (www.encode.org) and a list of URLs to download all samples used is available at the D4utils GitHub repository (https://github.com/38/d4-format). The WGS dataset is sample HG002 from the Genome in a Bottle Project, and the RNA-seq dataset is sample ENCFF976QSN from the ENCODE project¹⁶. Source data are provided with this paper.

Code availability

The code underlying the software described and the data analyzed in this study are available at: D4utils, https://github.com/38/d4-format; D4 Rust API, https://docs.rs/d4; D4 Python API, https://github.com/38/pyd4. Code has also been deposited on Zenodo¹⁷ at https://zenodo.org/record/4684595#.YJQ272ZKidZ.