zDUR: reference-free FASTQ compressor with high compression ratio and speed

Abstract

Background

High-throughput sequencing technologies generate massive amounts of FASTQ data comprising nucleotide sequences, quality scores, and read identifiers, necessitating efficient compression to alleviate storage and transmission burdens. Compared to general-purpose compressors, specialized FASTQ compressors achieve higher compression performance by exploiting the inherent redundancy in FASTQ files. However, existing FASTQ-specialized compressors often suffer from limited data applicability and tend to over-optimize either compression ratio or compression speed at the expense of the other.

Results

We present zDUR, a reference-free FASTQ compressor designed for efficient and scalable handling of next-generation sequencing data across diverse platforms and sequencing data types. Benchmarking against six reference-free compressors on 15 representative datasets spanning four sequencing data types demonstrates that zDUR achieves a favorable overall balance between compression ratio and speed, with broad applicability across data types. In particular, on single-cell RNA-seq and spatial transcriptomics datasets, zDUR achieves over a tenfold increase in runtime performance while maintaining higher compression ratios than SPRING, one of the state-of-the-art reference-free FASTQ compressors.

Conclusions

zDUR offers a scalable and efficient solution for reference-free FASTQ compression, balancing performance, speed, and usability across diverse datasets.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12859-025-06364-1.

Background

High-Throughput Sequencing technologies generate vast amounts of data [1] and one of the most widely used formats for such data is FASTQ. FASTQ is a plain text format storing sequencing reads, each consisting of three major fields: a header with metadata, a nucleotide sequence, and a quality string encoding per-base error probability. As plain text files that often reach several hundred gigabytes in size, FASTQ files are space-inefficient, and their compression significantly reduces storage and transmission costs [2].

FASTQ compression tools can be categorized into lossless and lossy approaches. Lossless compressors preserve the exact original content of the FASTQ file and ensure full reproducibility in downstream analyses. In contrast, lossy approaches modify or smooth selected fields, most commonly quality scores, to improve compression ratios. Tools such as QualComp [3] and Quartz [4] show that lossy compression can reduce storage requirements by exploiting correlations of quality scores within or across reads. Lossy compression may also influence downstream results and often requires additional computation. Therefore, we focus on lossless FASTQ compression in this study.

The most widely used tool for lossless FASTQ compression is gzip, a general-purpose compressor commonly preinstalled on UNIX systems. In comparison, specialized FASTQ compressors that exploit field specific patterns, such as redundancy from overlapping sequences or autocorrelation in quality strings, can reduce file size by over fivefold compared to gzip [5].

Specialized FASTQ compressors can be divided into reference-based and reference-free approaches. Reference-based approaches typically perform approximate alignment of each read to a reference genome and encode it as a genomic coordinate with associated alignment differences. Representative reference-based tools include Genozip [6], GTZ [7], and AMGC [8]. Among reference-free approaches, some perform entropy encoding with statistical modeling, such as fqzcomp4 [9] and DSRC2 [10]. Other tools further exploit sequence redundancy by grouping similar reads through reordering or approximate assembly steps: Leon [11], FaStore [12], SPRING [5], and PgRC [13, 14]. A third kind is exemplified by repaq [15], which performs simple transforms for the major fields before LZMA compression.

Reference-based FASTQ compressors typically achieve higher compression ratios and faster compression speeds when applied to FASTQ files derived from genomic or transcriptomic sequencing of common species, such as Homo sapiens, Mus musculus, and Danio rerio. However, when a reliable reference genome is not available, such as in previously unsequenced species or metagenomic datasets, the compression ratio drops substantially. In contrast, reference-free compression tools do not rely on external reference genomes and instead exploit intrinsic redundancy among reads, which allows them to maintain stable compression ratios and speeds across a wide range of datasets, including those lacking well-characterized references.

Despite the availability of many reference-free FASTQ compressors, they are often limited by restrictive data requirements, such as fixed read lengths or strict FASTQ header formats. In addition, many tools suffer from poor robustness and limited scalability to large datasets. Some tools also prioritize either compression ratio or speed, which may come at the expense of the other. Furthermore, no existing tool has been specifically optimized for single-cell RNA sequencing (scRNA-seq) or sequencing-based spatial transcriptomics (sST), two rapidly growing sources of sequencing data in recent years. The FASTQ file structures of scRNA-seq and sST datasets differ substantially from those of conventional paired-end sequencing datasets, with Read 1 containing only short barcode sequences (e.g., cell or spatial IDs less than 50 nucleotides) and Read 2 storing the transcriptome sequences. Applying the same compression strategy to both barcode and transcriptome sequences may reduce compression performance, as the two exhibit different sequence characteristics.

In this work, we present zDUR, a reference-free FASTQ compressor designed for efficient handling of next-generation sequencing data from diverse platforms and sample types. zDUR achieves a favorable balance between compression ratio, (de)compression speed and ease of use, and demonstrates efficient scalability across a wide range of file sizes.

Implementation

Overview of zDUR compression modes

zDUR supports three compression modes for FASTQ files: similarity-tree mode (default), barcode mode (for scRNA-seq and spatial transcriptomics), and fast mode (optimized for speed at the cost of compression ratio). These modes differ in input requirements and sequence compression strategies. Similarity-tree mode accepts both single-end (SE) and paired-end (PE) inputs, applying the similarity-tree algorithm [12] to all sequence reads. Barcode mode supports only paired input, compressing Read 2 with the similarity-tree algorithm and Read 1 with range coding. Fast mode accepts SE and PE inputs and applies range coding to all sequences. Quality scores and headers are processed identically in all three modes.

Field-wise compression strategies

Compression of sequence strings

zDUR implements the similarity-tree algorithm originally introduced by Roguski et al. [12], which exploits local redundancy between sequencing reads. Each read is assigned to a bin defined by its lexicographically smallest canonical minimizer (the smaller of a k-mer and its reverse complement), which serves as a read signature for binning. Within each bin, reads are sorted so that potentially overlapping reads appear close in order. Sorting is performed by first comparing the position of the bin-defining canonical minimizer, followed by lexicographical comparison of the sequence suffix starting at this minimizer. If the suffixes are identical, the prefixes to the left of the minimizer are compared. When sequences cannot be distinguished by suffix and prefix comparison, ordering is finalized using selected header fields.

After sorting, parent selection and differential encoding proceed as follows. For each read r, a parent p is selected from a sliding window of the previous W reads (with W set to 256 by default). Only reads sharing at least one minimizer with r are considered as valid candidates for computing the edge cost. The edge cost between r and p is computed from minimizer-aligned offsets and mismatches in the implied overlapping region based on following equation:

A read is assigned to p if its edge cost falls below a dynamic threshold equal to half of its length (max_cost = |r|/2). Reads satisfying this criterion are differentially encoded relative to their parent using four components: (i) distance in sorted order, (ii) minimizer-position offsets, (iii) mismatches, and (iv) the non-overlapping tail. Reads exceeding the threshold become roots and are compressed independently. These parent–child assignments collectively form a forest of similarity trees. For reads that form part of a similarity tree, the non-overlapping tail is compressed using range coding, and other components are compressed with libbsc. Reads exceeding the threshold are compressed independently using range coding. zDUR omits the redistribution and contig-assembly stages described in Roguski et al., as these provide only limited compression benefit at substantially higher computational cost.

Compression of quality strings

Quality strings are compressed with range coding which uses contextual frequencies. zDUR selects one of the two context models: a sequence-based model that uses the corresponding base as a context, and a second-order model from fqzcomp4, which exploits the relationship between adjacent symbols and the trend within the quality string of a read [9].

To determine the optimal model, zDUR samples a subset of quality strings and evaluates the compression ratio of both approaches, applying the better-performing model to the full dataset.

Compression of header strings

zDUR performs format-aware preprocessing of read headers across datasets from different sequencing platforms such as Illumina, MGI, and public archives such as NCBI’s SRA database. Headers are split into structured fields (e.g., read ID, lane, position), grouped by type across reads, and compressed using libbsc. For unrecognized formats, zDUR applies a general-purpose parser that handles arbitrary header structures.

Barcode-aware compression for scRNA-seq and sST data

zDUR applies a specialized strategy for technical sequences such as cell barcodes and unique molecular identifiers (UMIs) in the Read 1 files of single-cell RNA-seq and spatial transcriptomics data. Similarity-based compression is disabled for these sequences due to their short length and high entropy, which yield minimal compression gains and increase computational cost. Only biological sequences in Read 2 files are used for clustering and similarity-tree construction, while technical sequences are compressed using range coding.

Optimization of computational and I/O bottlenecks

Two major bottlenecks of the similarity tree algorithm were improved in zDUR: CPU-intensive similarity-tree construction and I/O overhead during clustering. The edge cost computation was vectorized using AVX2 instructions to accelerate tree construction. To reduce temporary disk usage, zDUR bit-packs sequence data; for large datasets, where I/O becomes a limiting factor, zDUR also applies real-time compression (LZ4 or Zstandard) to headers and quality scores. Lastly, clustered reads are retrieved via memory-mapped files to further improve access speed.

Three compression modes of zDUR

zDUR provides three compression modes. c-simtree applies the full similarity-tree workflow to all reads, enabling maximal exploitation of sequence redundancy. c-stereoseq applies the similarity-tree workflow only to Read 2 while compressing Read 1 with entropy coding, a configuration suitable for datasets containing short technical barcodes such as scRNA-seq and sST. c-fast disables similarity detection and performs entropy-encoded compression only, providing substantially faster runtimes at the cost of reduced sequence-based redundancy utilization.

Results

We benchmarked zDUR against six publicly available reference-free FASTQ compression tools, including SPRING, the top-performing tool to date. Additionally, we included pigz – a parallel version of gzip, the most widely used tool for FASTQ compression. For a comprehensive evaluation, we selected datasets representing various sequencing data types (Fig. 1 and Supplementary Data Table S1). In contrast to previous studies, which primarily focused on whole-genome sequencing (WGS) and metagenomic data, we specifically included scRNA-seq and sST datasets which feature asymmetric paired-end structures. For these datasets, zDUR’s barcode mode was applied, using different compression strategies for Read 1 and Read 2. The dataset sizes used in benchmarking ranged from 17 GB to 207 GB.

Fig. 1.

Performance comparison of zDUR and other reference-free FASTQ compression tools. (A) Compression ratio versus compression speed (MB/s) for seven reference-free FASTQ compression tools on four representative datasets: scRNA-seq, sST, metagenome, and WGS. Each triangle represents the average performance of a tool on one dataset. Each dataset comprises 3–6 paired FASTQ files (Supplementary data). Breaks in the y-axes are shown in all subpanels to accommodate the substantially higher compression speed of DSRC compared to the other tools. Compression ratio was calculated as the uncompressed file size divided by the compressed file size. (B) Average decompression speed (MB/s) of each tool across the same datasets.

As shown in Fig.1 A, zDUR achieved higher compression ratios than SPRING on scRNA-seq and sST datasets, while delivering over tenfold faster compression speeds. On metagenomic and WGS datasets, zDUR retained the advantage in compression speed compared to SPRING but achieved notably lower compression ratios. Nevertheless, zDUR achieved higher compression ratios than the other five tools on scRNA-seq, metagenome, and WGS datasets, and was only 1.5% lower than fqzcomp5 on the sST data. In terms of decompression speed (Fig. 1B), zDUR ranked second on scRNA-seq, sST, and WGS datasets, following DSRC, which consistently achieved the highest speed but only about half the compression ratio of zDUR (see Supplementary Data Table S2, S3, and S4 for detailed results). For memory usage during compression and decompression, similarity-based compressors including zDUR and SPRING exhibit higher peak memory consumption on the largest datasets, whereas lightweight match-based tools such as pigz and DSRC show the lowest memory footprint (Supplementary Data Table S5 and S6).

To quantify the trade-offs among the three compression modes, we performed a small ablation study on representative scRNA-seq and metagenome datasets. Across the datasets, c-simtree achieved higher compression ratios than c-fast, while c-fast showed substantially shorter runtimes. In scRNA-seq datasets, c-stereoseq produced higher compression ratios and slightly shorter runtimes compared with c-simtree (Supplementary Data Table S7). The differences among the three modes are more clearly reflected in the compression ratio of the sequence stream. For tested datasets, the sequence compression ratio of c-simtree was approximately 2–3× higher than that of c-fast, and on the tested scRNA-seq dataset, c-stereoseq further improved the sequence compression ratio by about 18% compared with c-simtree (Supplementary Data Table S8). During benchmarking, we encountered compatibility issues with certain tools. For instance, Leon successfully compressed only 6 out of 15 datasets, while FaStore was able to compress only a single dataset (Supplementary Data Table S7) and was therefore excluded from the main figure.

Discussion

Efficient compression and storage of sequencing data have remained a key challenge over the past two decades, especially as the rapid growth of multi-omics platforms continues to accelerate data generation. Although many compressors have been proposed, most suffer from limited applicability and suboptimal user experience. As a result, the majority of researchers still rely on gzip for compressing FASTQ files, despite its low efficiency and the resulting waste of storage resources.

To address these limitations, we developed zDUR, a reference-free FASTQ compressor designed to achieve favorable compression ratios across diverse data types within relatively short processing times. We evaluated zDUR on representative datasets from four sequencing data types: scRNA-seq, sST, metagenome and WGS. zDUR demonstrated robust and consistent performance across all datasets. In particular, on the scRNA-seq and sST datasets, zDUR achieved higher compression ratios than SPRING, one of the state-of-the-art reference-free FASTQ compressors, while delivering over a tenfold increase in compression speed. Furthermore, zDUR ranked second in decompression speed among all tested tools, outperforming SPRING and pigz, and only behind DSRC, which offers faster decompression at the cost of significantly lower compression ratios.

One potential limitation of zDUR is that it reorders reads during compression, which prevents straightforward file integrity validation using standard methods such as MD5 checksums. This issue may be particularly relevant in regulatory settings, such as clinical data management, where strict file-level reproducibility is required. Although zDUR is supplied with a sorting-based integrity verification script, its relatively high computational overhead makes it impractical to apply this step to every dataset in routine workflows. Nevertheless, prior studies have shown that such reordering does not affect downstream analyses [12], and more recent work has demonstrated that controlled reordering can even improve computational efficiency in tasks such as read alignment [16]. Another limitation of zDUR is its current lack of support for long-read sequencing data generated by third-generation sequencing platforms. Efficient compression of reads exceeding 1 kilobase (kb) in length requires different strategies, as redundancy patterns between long and short sequences differ substantially. Extending zDUR to handle long-read data will likely require algorithmic adjustments to accommodate these distinct sequence characteristics. A further limitation is that zDUR requires more memory for larger datasets. In most practical settings, however, such datasets are processed in computation environments with moderate to high RAM availability. Nonetheless, further optimization to reduce memory usage will be needed in future versions of zDUR.

Conclusions

zDUR provides a practical and versatile solution for reference-free compression for FASTQ data, offering high compression ratios and fast processing across diverse sequencing data types. Compared with existing reference-free compressors, zDUR achieves a favorable balance between compression efficiency, (de)compression speed, usability, and scales effectively across datasets of varying sizes.

Availability and requirements

Project name: zDUR

Project home page: https://github.com/MGI-EU/zDUR

Operating system(s): Linux and macOS

Programming language: C++

Other requirements: zlib and a standard system with C++17-compatible libc/gcc.

License: Free for academic research use under a license agreement (key request required). Commercial use requires a separate license.

Any restrictions to use by non-academics: Commercial license needed

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

A.E. contributed to methodology (lead), investigation, software development (lead), data curation, and writing of the original draft. R.D. contributed to conceptualization, investigation, project administration, and writing (review and editing). Q.F. contributed to software development, data curation, resources, validation, and visualization. I.K. contributed to software development and methodology. E.F. contributed to software development, investigation, and methodology. E.M. contributed to software development, investigation, and methodology. M.N. contributed to funding acquisition and supervision. Y.H. contributed to funding acquisition and supervision. Y.Z. contributed to conceptualization, supervision, project administration, and writing (review and editing). All authors reviewed and approved the manuscript.

Funding

The study did not receive external funding and was supported by internal resources of the authors’ affiliated organization.

Data availability

All data generated or analyzed during this study are included in supplementary information file.The zDUR executables and usage documentation are available at https://github.com/MGI-EU/zDUR.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The software developed in this work has been registered for software copyright protection and is also covered by a pending patent application.