Skip to main content
NCBI home page
Bioinformatics logoLink to Bioinformatics
. 2024 Jul 30;40(8):btae477. doi: 10.1093/bioinformatics/btae477

AssemblyQC: a Nextflow pipeline for reproducible reporting of assembly quality

Usman Rashid 1, Chen Wu 2, Jason Shiller 3, Ken Smith 4, Ross Crowhurst 5, Marcus Davy 6, Ting-Hsuan Chen 7, Ignacio Carvajal 8, Sarah Bailey 9, Susan Thomson 10, Cecilia H Deng 11,✉
Editor: Pier Luigi Martelli
PMCID: PMC11333564  PMID: 39078114

Abstract

Summary

Genome assembly projects have grown exponentially due to breakthroughs in sequencing technologies and assembly algorithms. Evaluating the quality of genome assemblies is critical to ensure the reliability of downstream analysis and interpretation. To fulfil this task, we have developed the AssemblyQC pipeline that performs file-format validation, contaminant checking, contiguity measurement, gene- and repeat-space completeness quantification, telomere inspection, taxonomic assignment, synteny alignment, scaffold examination through Hi-C contact-map visualization, and assessments of completeness, consensus quality and phasing through k-mer analysis. It produces a comprehensive HTML report with method descriptions, tables, and visualizations.

Availability and implementation

The pipeline uses Nextflow for workflow orchestration and adheres to the best-practice established by the nf-core community. This pipeline offers a reproducible, scalable, and portable method to assess the quality of genome assemblies—the code is available online at GitHub: https://github.com/Plant-Food-Research-Open/assemblyqc.

1 Introduction

During the last 20 years, reference genome assemblies have been generated for over 700 plant species (Sun et al. 2022). As of May 2024, the Assembly Database hosted by the National Center for Biotechnology Information (NCBI) returned >2.3 million search results (NCBI 2024). This exponential growth in genome assemblies has been realized by the continuous and substantial decrease in the cost of whole-genome sequencing (NHGRI 2023), coupled with advancements of sequencing technologies and assembly algorithms (Agarwal et al. 2020, Dida and Yi 2021). This trend is expected to persist as the pursuit for high quality genomes remains a major goal (Rhie et al. 2020b). Moreover, the reduced costs allow an increasingly wider community of research labs to routinely assemble new genomes. Aligned with this, fast and standardized assembly quality assessment has become critical and indispensable.

There are three major aspects of assembly quality: contiguity, completeness and correctness (Wang and Wang 2023). The Earth Biogenome Project tracks an updated list of quality metrics and their minimum values which should be met by genome assemblies (EBP 2024). A plethora of bioinformatics tools are available, each focusing on one specific aspect. To comprehensively assess an assembly, the researcher usually needs to run multiple tools to cover various aspects. This is challenging not only because many tools require manual installation and configuration of correct dependencies, but also because it can be tedious and time-consuming to run different tools separately. Reproducibility of the quality assessment results is an even bigger challenge as the problem is compounded by varied runtime requirements across platforms and frequent version changes. For example, BUSCO (Simão et al. 2015, Seppey et al. 2019), which has been widely adopted to estimate the gene-space completeness of an assembly had 11 updates within a year (January 2022 to June 2023).

To facilitate a streamlined application of the quality assessment tools in a reproducible manner, GenomeQC was recently released (Manchanda et al. 2020) with publicly available source code and a free R/Shiny webapp. The Vertebrate Genomes Project (VGP) assembly pipeline provides wrappers to execute different tools, including running individual quality evaluation tools (Rhie et al. 2020b). Incorporating additional tools for a thorough quality assessment as compared to existing pipelines, we have developed AssemblyQC, which adopts the highly portable Nextflow workflow management system in combination with the community-curated nf-core framework (Di Tommaso et al. 2017, Ewels et al. 2020, Langer et al. 2024). AssemblyQC is a unified, fully automated, reproducible tool that can be executed on local machines, high-performance computers, or on the cloud to evaluate the quality of genome and transcriptome assemblies. Quality metrics chosen for report are based on community standards. The pipeline is implemented using nf-core modules, which are reviewed and regularly updated by a large open-source community (Langer et al. 2024).

2 Materials and methods

2.1 Design

The pipeline is designed to evaluate multiple genome or transcriptome assemblies in parallel. The pipeline is divided into four major sections as shown in the flowchart in Fig. 1. Within each section, data are processed in parallel where possible. In Section 1, the pipeline checks the input FASTA and GFF3 annotation files using py_fasta_validator, SekQit rmdup, and GenomeTools gt gff3validator (Gremme et al. 2013, Shen et al. 2016, Edwards 2019), to ensure integrity of the input files, detect duplicate sequences and prevent failure of subsequent tools. In Section 2, the pipeline uses NCBI’s Foreign Contamination Screen (FCS) and its database to detect contaminants such as adapters and sequences from foreign organisms (Astashyn et al. 2023). Assemblies that successfully pass these checks move on to additional quality checks in Section 3. The pipeline can be configured to skip quality checks for assemblies which contain contaminants.

Figure 1.

Figure 1.

Open in a new tab

Pipeline flowchart.

Section 3 executes the remaining quality assessment in parallel for each assembly. For scaffold and contig-level contiguity statistics associated with FASTA sequences, assemblathon2-analysis is used (UCDAVIS-Bioinformatics 2012). To break the assembly into contigs, the pipeline uses 100 ‘N’ bases as the default unknown gap size. This parameter can be changed and is quoted in the AssemblyQC final report atop the contig-level statistics. Statistics related to annotations in GFF3 files are computed with GenomeTools gt stat tool (Gremme et al. 2013). The Benchmarking Universal Single-Copy Orthologs (BUSCO) tool is used for estimating the gene-space completeness of each assembly (Seppey et al. 2019). The pipeline can be configured to evaluate each assembly against one or more BUSCO lineages in parallel. The quality of the repeat-space is evaluated with the Long Terminal Repeat (LTR) Assembly Index (LAI) (Ou et al. 2018). The LTR_retriever workflow consisting of LTR_FINDER and LTRharvest for de novo detection of LTRs (Ellinghaus et al. 2008, Ou and Jiang 2018, 2019). The LAI statistic is independent of the BUSCO statistics and can help improve overall assembly contiguity by isolating shortcomings in the repeat-space.

The pipeline can also assess chromosome completeness and correctness through telomere, Hi-C contact-frequency, and synteny visualizations. The Telomere Identification toolKit (tidk) is employed to estimate the presence of telomeres with a specified telomeric motif (Brown et al. 2023). To supplement assessment with this user-specified repeat motif, the toolkit also explores the most likely data-driven repeat sequence for each assembly from the assembly sequences. The results for TIDK are sorted in ascending order by sequence length using SeqKit (Shen et al. 2016). Presence of the telomeric repeats throughout a sequence often indicates assembly errors, and in many cases, these can be corrected by rearranging fragments in the sequence.

Where Hi-C data are provided, a contact-frequency map is generated by the pipeline and added to the report for visualization. FASTQC and FASTP are used to trim and quality check the Hi-C reads (Andrews 2010, Chen et al. 2018). The reads are then mapped to the assembly sequences using Burrow-Wheeler Aligner (BWA) (Heng Li 2013). The mapping quality is checked with hic_qc.py (Sullivan 2022), and finally converted into hic format through a workflow consisting of samblaster, samtools, and run-assembly-visualizer.sh (Faust and Hall 2014, Dudchenko et al. 2017, Danecek et al. 2021). The interactive visualization of the Hi-C map is added to the report using the Juicebox.js JavaScript library (Robinson et al. 2018). Atypical Hi-C contact-frequency patterns such as gaps or nondiagonal contact concentrations can indicate gaps in the assembly or misassignment of contigs to scaffolds.

Two sub-workflows, the pair-wise mode, and the chromosome-level comparison, are implemented in the pipeline to generate synteny plots between input assemblies. The pair-wise workflow creates synteny plots between each combination of input assemblies. Firstly, cross-mapping is performed with MUMmer4 (Marçais et al. 2018). The pipeline can be configured to include or exclude many-to-many mappings detected by MUMmer4. The mappings are then plotted with Circos (Krzywinski et al. 2009). A dot-plot can also be created which is helpful in identifying inversions (Cabanettes and Klopp 2018). This result makes it easy to visualize gaps and large structural variations (SVs) between assemblies and helps to consistently assign chromosomal numbers to those assemblies. Chromosome-level comparison sub-workflow maps corresponding chromosomes in all the input assemblies with Minimap2 (Li 2018), and generates synteny plots with plotsr (Li 2018, Goel and Schneeberger 2022), allowing chromosome-by-chromosome visualization of the key synteny blocks and large SVs from all input. This sub-workflow is helpful for creating a single summary synteny visualization.

Kraken2 is used to assign taxonomic labels to each assembly sequence (Wood et al. 2019) and an interactive report is produced with Krona (Ondov et al. 2011). This can be useful in detecting cross-domain contamination in the assemblies (Cornet and Baurain 2022).

Merqury is also included in the pipeline to facilitate k-mer analysis. The pipeline supports both mixed-haplotype and diploid assembly assessment with and without the availability of parental reads. In addition to k-mer completeness and consensus quality, Merqury is very useful in evaluating the extent of haplotype phasing (Rhie et al. 2020a).

In Section 4 of the pipeline, outputs from various assessment tools are gathered, parsed, and converted into a HyperText Markup Language (HTML) report using Jinja templating language in Python. Other than file format validation, the remaining assessment tools are optional and can be bypassed via settings in the configuration file.

2.2 Implementation details

The implementation of the pipeline follows the Nextflow good-practice developed by the nf-core community (Ewels et al. 2020, Langer et al. 2024). It is built with the nf-core pipeline template, and whenever feasible, utilizing the nf-core modules and sub-workflows. The nf-test is used for unit testing, with each module executing a single script or tool. The modules do not rely on installation of dependencies in the system environment; instead, they are implemented using version-locked Bioconda Docker/Singularity/Apptainer containers (Merkel 2014, da Veiga Leprevost et al. 2017, Kurtzer et al. 2017, Grüning et al. 2018) from public repositories including https://quay.io and https://hub.docker.com. Pipeline code, documentation and example reports are available on GitHub: https://plant-food-research-open.github.io/assemblyqc.

3 Conclusion

We have created a highly portable and reproducible pipeline for comprehensive assessment of genome and transcriptome assemblies. The pipeline evaluates contiguity, completeness, correctness, and contamination of assemblies in multiple ways with various well adopted bioinformatics tools. The quality assessment results are presented in a shareable HTML report. Furthermore, the actionable insights from the report such as locations of contaminants, locations and abundance of telomeric motifs inside chromosome sequences, presence of gaps or off diagonal contacts in the Hi-C map, and structural variations borne out by the synteny plots, can be used to guide iterative improvement of the assembly.

Acknowledgements

The plant-food-research-open/assemblyqc uses code and infrastructure developed and maintained by the nf-core community. We are thankful to Mahesh Binzer-Panchal from National Bioinformatics Infrastructure Sweden, Uppsala University, Sweden for his help in porting pipeline modules to the nf-core template. We acknowledge the Ministry of Business, Innovation and Employment (MBIE) for providing the Strategic Science Investment Funding (SSIF).

Contributor Information

Usman Rashid, Molecular & Digital Breeding, The New Zealand Institute for Plant and Food Research Limited, 1025 Auckland, New Zealand.

Chen Wu, Molecular & Digital Breeding, The New Zealand Institute for Plant and Food Research Limited, 1025 Auckland, New Zealand.

Jason Shiller, Molecular & Digital Breeding, The New Zealand Institute for Plant and Food Research Limited, 3182 Te Puke, New Zealand.

Ken Smith, Molecular & Digital Breeding, The New Zealand Institute for Plant and Food Research Limited, 1025 Auckland, New Zealand.

Ross Crowhurst, Molecular & Digital Breeding, The New Zealand Institute for Plant and Food Research Limited, 1025 Auckland, New Zealand.

Marcus Davy, Molecular & Digital Breeding, The New Zealand Institute for Plant and Food Research Limited, 3182 Te Puke, New Zealand.

Ting-Hsuan Chen, Molecular & Digital Breeding, The New Zealand Institute for Plant and Food Research Limited, 7608 Lincoln, New Zealand.

Ignacio Carvajal, Molecular & Digital Breeding, The New Zealand Institute for Plant and Food Research Limited, 1025 Auckland, New Zealand.

Sarah Bailey, Molecular & Digital Breeding, The New Zealand Institute for Plant and Food Research Limited, 1025 Auckland, New Zealand.

Susan Thomson, Molecular & Digital Breeding, The New Zealand Institute for Plant and Food Research Limited, 7608 Lincoln, New Zealand.

Cecilia H Deng, Molecular & Digital Breeding, The New Zealand Institute for Plant and Food Research Limited, 1025 Auckland, New Zealand.

Conflict of interest

None declared.

Funding

This work was supported by the Traits for Life Indoors (TFLI) programme in the Horticulture Production Goes Urban research direction supported by Strategic Science Investment Funding (SSIF) received by The New Zealand Institute for Plant and Food Research Limited (PFR).

Data availability

Pipeline usage documentation, parameter descriptions, and example outputs are available on GitHub: https://github.com/Plant-Food-Research-Open/assemblyqc/tree/main/docs. A preview report is also hosted online: https://plant-food-research-open.github.io/assemblyqc.

References

  1. Agarwal T, Suravajhala R, Bhushan M. et al. Recent Advances in Gene and Genome Assembly: Challenges and Implications. Advances in Synthetic Biology2020:199–220.
  2. Andrews S. FastQC: A Quality Control Tool for High Throughput Sequence Data. Cambridge, United Kingdom: Babraham Bioinformatics, Babraham Institute. 2010. [Google Scholar]
  3. Astashyn A, Tvedte ES, Sweeney D. et al. Rapid and sensitive detection of genome contamination at scale with FCS-GX. Genome Biology 2024;25(1):60. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Brown M, González De la Rosa PM, Mark B.. A Telomere Identification Toolkit. Zenodo (2023). 10.5281/zenodo.10091385. Code repository: https://github.com/tolkit/telomeric-identifier. [DOI]
  5. Cabanettes F, Klopp C.. D-GENIES: dot plot large genomes in an interactive, efficient and simple way. PeerJ 2018;6:e4958. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Chen S, Zhou Y, Chen Y. et al. fastp: an ultra-fast all-in-one FASTQ preprocessor. Bioinformatics 2018;34:i884–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Cornet L, Baurain D.. Contamination detection in genomic data: more is not enough. Genome Biol 2022;23:60. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. da Veiga Leprevost F, Grüning BA, Alves Aflitos S. et al. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics 2017;33:2580–2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Danecek P, Bonfield JK, Liddle J. et al. Twelve years of SAMtools and BCFtools. Gigascience 2021;10:giab008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Di Tommaso P, Chatzou M, Floden EW. et al. Nextflow enables reproducible computational workflows. Nat Biotechnol 2017;35:316–9. [DOI] [PubMed] [Google Scholar]
  11. Dida F, Yi G.. Empirical evaluation of methods for de novo genome assembly. PeerJ Comput Sci 2021;7:e636. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Dudchenko O, Batra SS, Omer AD. et al. De novo assembly of the Aedes aegypti genome using Hi-C yields chromosome-length scaffolds. Science 2017;356:92–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. EBP. Report on Assembly Standards Version 5.0. 2024. https://www.earthbiogenome.org/report-on-assembly-standards (4 June 2024, date last accessed).
  14. Edwards R. linsalrob/fasta_validator: a fast and efficient fasta validator written in pure C. 10.5281/zenodo.25320442019. [DOI]
  15. Ellinghaus D, Kurtz S, Willhoeft U.. LTRharvest, an efficient and flexible software for de novo detection of LTR retrotransposons. BMC Bioinformatics 2008;9:18. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Ewels PA, Peltzer A, Fillinger S. et al. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol 2020;38:276–8. [DOI] [PubMed] [Google Scholar]
  17. Faust GG, Hall IM.. SAMBLASTER: fast duplicate marking and structural variant read extraction. Bioinformatics 2014;30:2503–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Goel M, Schneeberger K.. plotsr: visualizing structural similarities and rearrangements between multiple genomes. Bioinformatics 2022;38:2922–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Gremme G, Steinbiss S, Kurtz S.. GenomeTools: a comprehensive software library for efficient processing of structured genome annotations. IEEE/ACM Trans Comput Biol Bioinform 2013;10:645–56. [DOI] [PubMed] [Google Scholar]
  20. Grüning B, Dale R, Sjödin A. et al. ; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods, 2018;15:475–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Krzywinski M, Schein J, Birol I. et al. Circos: an information aesthetic for comparative genomics. Genome Res 2009;19:1639–45. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kurtzer GM, Sochat V, Bauer MW.. Singularity: scientific containers for mobility of compute. PLoS One 2017;12:e0177459. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Langer BE et al. Empowering bioinformatics communities with Nextflow and nf-core. bioRxiv, 10.1101/2024.05.25.595877, 2024, preprint: not peer reviewed. [DOI]
  24. Li H. Minimap2: pairwise alignment for nucleotide sequences. Bioinformatics 2018;34:3094–100. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Li H. Aligning sequence reads, clone sequences and assembly contigs with BWA-MEM. arXiv, arXiv:1303.3997, 2013, preprint: not peer reviewed.
  26. Manchanda N, Portwood JL, Woodhouse MR. et al. GenomeQC: a quality assessment tool for genome assemblies and gene structure annotations. BMC Genomics 2020;21:193. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Marçais G, Delcher AL, Phillippy AM. et al. MUMmer4: a fast and versatile genome alignment system. PLoS Comput Biol 2018;14:e1005944. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Merkel D. Docker: lightweight linux containers for consistent development and deployment. Linux J 2014;239:2. [Google Scholar]
  29. NCBI. NCBI Assembly Database. https://www.ncbi.nlm.nih.gov/assembly/?term=all%5Bsb%5D (24 May 2024, date last accessed).
  30. NHGRI. DNA Sequencing Costs: Data. https://www.genome.gov/about-genomics/fact-sheets/DNA-Sequencing-Costs-Data (28 June 2023, date last accessed).
  31. Ondov BD, Bergman NH, Phillippy AM.. Interactive metagenomic visualization in a web browser. BMC Bioinformatics 2011;12:385. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Ou S, Jiang N.. LTR_retriever: a highly accurate and sensitive program for identification of long terminal repeat retrotransposons. Plant Physiol 2018;176:1410–22. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Ou S, Jiang N.. LTR_FINDER_parallel: parallelization of LTR_FINDER enabling rapid identification of long terminal repeat retrotransposons. Mob DNA 2019;10:48. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Ou S, Chen J, Jiang N.. Assessing genome assembly quality using the LTR assembly index (LAI). Nucleic Acids Res 2018;46:e126. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Rhie A, Walenz BP, Koren S. et al. Merqury: reference-free quality, completeness, and phasing assessment for genome assemblies. Genome Biol 2020a;21:245. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Rhie A, McCarthy SA, Fedrigo O. et al. Towards complete and error-free genome assemblies of all vertebrate species. Nature 2021;592(7856):737–46. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Robinson JT, Turner D, Durand NC. et al. Juicebox.js provides a cloud-based visualization system for Hi-C data. Cell Syst 2018;6:256–8 e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Seppey M, Manni M, Zdobnov EM.. BUSCO: assessing genome assembly and annotation completeness. Methods Mol Biol 2019;1962:227–45. [DOI] [PubMed] [Google Scholar]
  39. Shen W, Le S, Li Y. et al. SeqKit: a cross-platform and ultrafast toolkit for FASTA/Q file manipulation. PLoS One 2016;11:e0163962. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Simão FA, Waterhouse RM, Ioannidis P. et al. BUSCO: assessing genome assembly and annotation completeness with single-copy orthologs. Bioinformatics 2015;31:3210–2. [DOI] [PubMed] [Google Scholar]
  41. Sullivan S. hic_qc, 6881c33 edn. 2022. Code repository: https://github.com/phasegenomics/hic_qc.
  42. Sun Y, Shang L, Zhu Q-H. et al. Twenty years of plant genome sequencing: achievements and challenges. Trends Plant Sci 2022;27:391–401. [DOI] [PubMed] [Google Scholar]
  43. UCDAVIS-Bioinformatics. assemblathon2-analysis. https://github.com/ucdavis-bioinformatics/assemblathon2-analysis (28 June 2023, date last accessed).
  44. Wang P, Wang F.. A proposed metric set for evaluation of genome assembly quality. Trends Genet 2023;39:175–86. [DOI] [PubMed] [Google Scholar]
  45. Wood DE, Lu J, Langmead B.. Improved metagenomic analysis with Kraken 2. Genome Biol 2019;20:257. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Pipeline usage documentation, parameter descriptions, and example outputs are available on GitHub: https://github.com/Plant-Food-Research-Open/assemblyqc/tree/main/docs. A preview report is also hosted online: https://plant-food-research-open.github.io/assemblyqc.

Articles from Bioinformatics are provided here courtesy of Oxford University Press

RESOURCES