Quantitative evaluation of microbiome sequencing resolution under varying experimental conditions using defined mock communities

Abstract

Objective evaluation of sequencing resolution is crucial for comparing technologies and ensuring reproducibility in microbiome analysis. Specifically, a systematic approach is necessary to quantitatively assess the effect of various platforms and experimental conditions on species-level resolution. Therefore, this study quantitatively evaluated multiple strategies, including 16S V3–V4 (16P), full-length 16S rRNA gene (16F), and whole metagenome shotgun sequencing (WMS), using a commercial DNA-based mock community (MC) and a domestically developed whole-cell MC (Korea MC [KMC]). The WMS strategy included 12 combinations of input DNA concentrations and sequencing output levels. A total of 64 WMS libraries were constructed for KMC samples, and 112 sequencing datasets were analysed. Taxonomic resolution was assessed using an adjusted F1-score integrating detection sensitivity and abundance-level reproducibility. Qualitatively examining the detected species against the expected species across platforms, WMS showed a true positive abundance ratio of over 90%, 16F was observed to have an average of 60%, and 16P was observed to have an average of less than 10%. The combination of 10 ng input and 10 gigabases output consistently yielded the highest species-level resolution. However, reduced performance was observed in some MCs under 1 ng or 100 ng DNA input conditions. Detection sensitivity varied by taxon and condition. Specifically, Streptococcus pneumoniae and Cryptococcus neoformans were detected only under high-input or -output conditions, whereas Escherichia coli exhibited optimal accuracy at intermediate inputs. Acinetobacter species demonstrated reduced resolution as input DNA increased. KMC samples showed species- and format-specific variability in DNA extraction efficiency. This study presents a quantitative evaluation of species-level resolution across sequencing conditions using defined mock communities. The results highlight how sequencing configuration and taxon-specific characteristics can influence detection performance and provide insights for interpreting microbiome sequencing results under different experimental conditions.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-53382-x.

Introduction

As microbiome analysis technologies evolve, methodological discussions have increasingly focused on enabling high-resolution profiling of microbial communities and ensuring the reliability of analytical outcomes^1–3. Microbial community profiling is a foundational aspect of microbiome research and is typically conducted using either 16S ribosomal RNA gene sequencing (16S rRNA sequencing) or whole metagenome shotgun sequencing (WMS)^4,5. Amplicon sequencing is a cost-effective and relatively uncomplicated experimental procedure, but is typically limited to taxonomic resolution at the genus level^6,7. WMS enables accurate taxonomic and functional profiling at the species or even strain level⁸. High-resolution taxonomic profiling is crucial in contexts such as disease diagnosis, personalized medicine, and environmental monitoring, where distinguishing between closely related species or strains can impact. however, it involves higher costs and more complex analytical workflows than amplicon sequencing⁹. Moreover, WMS-based analyses are highly sensitive to technical variability introduced during sample preparation and sequencing^10,11. Even among studies analysing similar sample types, considerable differences in microbial composition estimates have been observed, stemming from heterogeneous protocols and inconsistent analytical pipelines^12–14. The variability poses a notable challenge to inter-study comparability and reproducibility, highlighting the critical need for technical standardisation and systematic validation throughout the microbiome analysis workflow^15,16.

Species-level resolution in microbiome analysis is influenced by both the sequencing platform and experimental variables such as input DNA concentration and sequencing output yield^17–19. For example, sequencing output yield, which is often limited by the high cost of sequencing, can reduce read depth, thereby lowering the sensitivity for detecting low-abundance taxa and ultimately constraining species-level resolution^18,20. Although these variables are known to affect microbial profiling, few studies have systematically evaluated their effects at the species level^21,22. Even when such assessments exist, they are often limited to dichotomous metrics such as recall or precision, which do not properly reflect changes in accuracy or resolution at the level of abundance^23,24. To rigorously examine these effects, sequencing performance must be tested against reference materials with known composition and abundance^25,26.

Mock communities (MCs), defined as artificial mixtures with known microbial compositions, serve as essential reference controls for evaluating methodological accuracy, reproducibility, and potential bias in microbial profiling workflows^13,27,28. Widely used commercial MC standards, such as the HMP-ATCC and ZymoBIOMICS, are designed based on balanced microbial compositions and serve as benchmarks for evaluating analytical performance^29,30. However, the HMP-ATCC mock community consists of a fixed set of strains derived from Western populations, which may limit its relevance for studies targeting different cohorts or microbiome contexts^31–33. As an alternative, researcher-constructed MCs allow customised designs, such as incorporating region-specific strains or adjusting abundance ratios, which may better suit specific research needs^28,34,35. When prepared in whole-cell format, these MCs also enable assessment of the full experimental workflow, including DNA extraction, providing a more comprehensive evaluation than DNA-based MCs^13,36.

Therefore, this study aimed to quantitatively evaluate the effect of various experimental conditions on species-level resolution and abundance accuracy using two types of MC formats (whole-cell and DNA-based), different sequencing platforms, input DNA concentrations, and sequencing output yields. This study sought to contribute to the establishment of reproducible and reliable microbial profiling guidelines.

Methods

Experimental framework and sequencing conditions

This study was designed to evaluate sequencing performance using MCs with different formats and microbial compositions. Two distinct MC formats were used: a commercially available DNA-based MC and a domestically developed whole-cell MC, referred to as the Korea MC (KMC). Each MC format included four body site-specific mock communities representing the gut, oral, skin, and vaginal environments (Fig. 1).

Fig. 1.

Overview of study design. This figure illustrates the experimental workflow designed to evaluate species-level resolution using different types of mock communities and combinations of sequencing strategies and conditions. (A) Composition of mock communities (MCs). Commercial DNA-based MCs and domestically developed whole-cell MCs (Korean MCs [KMCs]) were designed to represent four human body sites: gut, oral, skin and vaginal (KMCG, KMCO, KMCS, and KMCV, respectively). (B) Sequencing strategies applied included short-read amplicon (16S V3–V4; 16P) sequencing, long-read full-length 16S rRNA amplicon (16F) sequencing and short-read whole metagenome shotgun (WMS) sequencing. (C) Datasets consisted of three predefined input DNA concentrations (1, 10, and 100 ng) and four sequencing output yields (1, 5, 10, and 20 gigabases [Gb]), leading to 12 combinations. (D) A total of 112 datasets were generated: 16P (n = 8), 16F (n = 8) and WMS (n = 96) covering all mock types and conditions.

Sequencing was performed using three platforms: short-read amplicon sequencing targeting the 16S rRNA V3–V4 region (16P), long-read full-length 16S rRNA gene sequencing (16F), and short-read WMS sequencing (Fig. 1). For WMS, 12 experimental conditions were constructed by combining 3 predefined input DNA concentrations with 4 levels of sequencing output yield (Fig. 1). Each MC underwent sequencing under a single 16P condition, a single 16F condition, and 12 WMS conditions, resulting in a total of 14 datasets per mock community. Applied across 4 body sites and 2 MC formats, a total of 112 sequencing datasets were generated and analyzed (Fig. 1).

Overview of the MC samples

The DNA-based MCs were labeled MCG, MCO, MCS, and MCV, corresponding to the gut, oral, skin, and vaginal sites, respectively. They were derived from commercially available ATCC Bacterial Mix standards (MSA-2006™, MSA-2007™, MSA-2008™, and MSA-2009™, respectively). These four types of microbial whole cell mock communities were developed by lyophilizing standardized bacterial suspension as a part of the research and development program for clinical metagenomics funded by Korea National Institute of Health. This format facilitates room-temperature storage and enables the standardized assessment of DNA extraction efficiency across sample types. The species included in each MC are listed in Tables 1 and 2 and served as the ground-truth reference set for taxonomic accuracy assessment.

Table 1.

List of species for each KMC.

KMCG	(%)	KMCO	(%)	KMCS	(%)	KMCV	(%)
Klebsiella pneumoniae	20.6	Staphylococcus aureus	24.1	Staphylococcus capitis	37.0	Staphylococcus aureus	35.7
Vibrio parahaemolyticus	18.4	Acinetobacter baumannii	18.8	Staphylococcus epidermidis	16.1	Haemophilus influenzae	25.0
Salmonella enterica	15.1	Streptococcus pyogenes	18.4	Staphylococcus aureus	12.2	Streptococcus agalactiae	11.2
Enterococcus faecium	10.6	Neisseria sicca	16.5	Klebsiella pneumoniae	9.7	Klebsiella pneumoniae	6.3
Lacticaseibacillus rhamnosus	10.1	Klebsiella pneumoniae	9.6	Corynebacterium striatum	8.7	Listeria monocytogenes	5.8
Bifidobacterium longum	5.9	Haemophilus influenzae	4.0	Acinetobacter bereziniae	7.1	Streptococcus pneumoniae	5.8
Clostridium perfringens	4.5	Acinetobacter bereziniae	4.0	Enterococcus faecium	5.6	Lacticaseibacillus rhamnosus	4.0
Escherichia coli	3.5	Acinetobacter ursingii	2.6	Acinetobacter ursingii	3.1	Escherichia coli	3.7
Yersinia enterocolitica	3.5	Escherichia coli	1.2	Escherichia coli	0.8	Gardnerella vaginalis	1.7
Bacteroides ovatus	2.9	Streptococcus pneumoniae	0.0			Lacticaseibacillus paracasei	0.7
Clostridioides difficile	2.5					Cryptococcus neoformans	0.1
Lacticaseibacillus paracasei	2.1

Table 2.

List of species for each MC.

MCG	(%)	MCO	(%)	MCS	(%)	MCV	(%)
Bacteroides fragilis	8.3	Schaalia odontolytica	16.7	Acinetobacter johnsonii	16.7	Gardnerella vaginalis	16.7
Bacteroides vulgatus	8.3	Prevotella melaninogenica	16.7	Corynebacterium striatum	16.7	Ligilactobacillus gasseri	16.7
Bifidobacterium adolescentis	8.3	Streptococcus mitis	16.7	Micrococcus luteus	16.7	Mycoplasma hominis	16.7
Clostridioides difficile	8.3	Veillonella parvula	16.7	Cutibacterium acnes	16.7	Prevotella bivia	16.7
Enterococcus faecalis	8.3	Haemophilus parainfluenzae	16.7	Staphylococcus epidermidis	16.7	Streptococcus agalactiae	16.7
Lactiplantibacillus plantarum	8.3	Fusobacterium nucleatum	16.7	Streptococcus mitis	16.7	Ligilactobacillus jensenii	16.7
Enterobacter cloacae	8.3
Escherichia coli	8.3
Helicobacter pylori	8.3
Yersinia enterocolitica	8.3
Salmonella enterica	8.3
Fusobacterium nucleatum	8.3

DNA extraction and quality control

Genomic DNA from the KMC was extracted using the DNeasy PowerSoil Kit (QIAGEN, Hilden, Germany) according to the manufacturer’s instructions. DNA integrity was evaluated via 1% agarose gel electrophoresis. The concentration and purity of the extracted microbial DNA were assessed using both a Nanodrop 2000 UV–Vis spectrophotometer (Thermo Fisher Scientific, Waltham, MA, USA) and a Qubit 3.0 fluorometer (Thermo Fisher Scientific). Only samples that passed quality control thresholds were used for downstream library preparation.

Library preparation

16S rRNA V3–V4 region amplicon sequencing

Amplicon sequencing targeting the V3–V4 region of the 16S rRNA gene (16P) was performed using the primer pair 341F (5′-CCTACGGGNGGCWGCAG-3′) and 805R (5′-GACTACHVGGGTATCTAATCC-3′). PCR reactions were prepared according to the manufacturer’s instructions using a Herculase II polymerase-based master mix (Agilent Technologies, Santa Clara, CA, USA) and V3–V4-specific primers. PCR amplification was performed on an ABI GeneAmp® 9700 thermocycler (Applied Biosystems, Foster City, CA, USA) using the following cycling conditions: an initial denaturation at 95 °C for 3 min; 25 cycles of denaturation at 95 °C for 30 s, annealing at 55 °C for 30 s, and extension at 72 °C for 30 s; followed by a final extension at 72 °C for 5 min and a hold at 4 °C. A separate index PCR was performed with 10 additional cycles under the same conditions.

After amplification, the PCR products were purified using AMPure XP Beads (Beckman Coulter, Brea, CA, USA). Equimolar pooling was performed based on fluorescence-based quantification. Indexing was achieved using the Nextera XT Index Primer Set (Illumina Inc., San Diego, CA, USA) in a separate PCR reaction using the same enzyme and buffer system. The purified and pooled libraries were sequenced as 2 × 300 bp paired-end reads on the Illumina MiSeq platform (Illumina Inc.) per the manufacturer’s instructions.

16S rRNA full-length amplicon sequencing

Amplicon sequencing targeting the full-length 16S rRNA gene (16F) was performed using the universal primer pair 27F (5′-AGRGTTYGATYMTGGCTCAG-3′) and 1492R (5′-RGYTACCTTGTTACGACTT-3′). Sample-specific barcode sequences provided by the manufacturer were added to both primers to allow multiplexed sequencing. PCR reactions were prepared according to the manufacturer’s instructions using the KAPA HiFi HotStart ReadyMix (Roche, Basel, Switzerland). PCR amplification was performed using a Veriti™ Thermal Cycler (Applied Biosystems) with the following cycling conditions: an initial denaturation at 95 °C for 3 min; 27 cycles of denaturation at 95 °C for 30 s, annealing at 57 °C for 30 s, and extension at 72 °C for 60 s; followed by a final extension at 72 °C for 5 min and a hold at 4 °C. Amplicons were evaluated for quality using a TapeStation™ system (Agilent Technologies), and equimolar pooling of all samples was performed before library construction.

Libraries were constructed using the SMRTbell Express Template Prep Kit 2.0 (Pacific Biosciences, Menlo Park, CA, USA) according to the manufacturer’s instructions for full-length 16S library preparation. Library size and concentration were assessed using the Qubit Fluorometer with the Qubit™ 1X dsDNA HS Assay Kit (Invitrogen, Carlsbad, CA, USA) and the TapeStation system. Primer annealing and polymerase binding were performed using the Sequel II Binding Kit 2.1 and DNA Internal Control Complex 1.0 (Pacific Biosciences). Sequencing was conducted on a Sequel II system (Pacific Biosciences) using the Sequel II Sequencing Kit 2.0.

WMS sequencing

Shotgun metagenomic libraries were constructed using the TruSeq DNA Nano Library Prep Kit (Illumina Inc.) according to the manufacturer’s instructions. Input DNA (100 ng) was enzymatically fragmented to an average insert size of 200–400 bp using the Covaris LE220 system (Covaris, Woburn, MA, USA). Fragmented DNA was end-repaired, A-tailed, and ligated with Illumina adapter indices following the standard workflow.

After ligation, libraries were amplified using the Enhanced PCR Mix provided in the kit with eight PCR cycles under the following conditions: 95 °C for 3 min; eight cycles of 98 °C for 20 s, 60 °C for 15 s, and 72 °C for 30 s; followed by 72 °C for 5 min and a final hold at 4 °C.

PCR products were purified using AMPure XP Beads (Beckman Coulter) and eluted in 30 µL of resuspension buffer. Library size distribution was evaluated using the TapeStation™ system (Agilent Technologies), and library concentrations were determined using qPCR. The pooled libraries were sequenced on an Illumina NovaSeq 6000 platform (Illumina Inc.) to generate 2 × 150 bp paired-end reads.

To evaluate the effect of input DNA quantity and sequencing depth, three predefined input DNA concentrations and four target sequencing output levels were used, resulting in 12 shotgun library conditions. Six libraries were physically prepared and sequenced, whereas the remaining six datasets were generated via in silico subsampling of sequencing reads. Eight libraries were generated for each MC: one from 16P sequencing, one from 16F sequencing, and six from WMS sequencing. The six WMS libraries were designed using three predefined combinations of input DNA concentrations and two sequencing output yield levels. Thus, eight libraries were constructed per MC, resulting in a total of 64 libraries across all conditions.

Data processing and taxonomic profiling

Taxonomic assignment methods were selected according to the characteristics of each sequencing strategy and commonly used analytical workflows. Because each sequencing approach generates sequence data with different read lengths, target regions, and information content, taxonomic classification methods were applied using approaches appropriate for each data type. Accordingly, comparisons across sequencing strategies in this study should be interpreted as comparisons of end-to-end analytical workflows tailored to each sequencing type, rather than as a strict head-to-head evaluation under an identical taxonomic classifier.

16 s rRNA v3–v4 region amplicon sequencing

Raw reads were quality-checked using FastQC (Babraham Institute, Cambridge, UK), and primer sequences were trimmed with Cutadapt (v4.0, Germany). Denoising, dereplication, and chimaera removal were performed using the DADA2 plugin in QIIME2 (v2023.2, USA). Reads with an average quality score < Q20 and length < 250 bp were excluded before amplicon sequence variant (ASV) inference. ASVs were generated, and taxonomic classification was conducted using a naïve Bayes classifier implemented in QIIME2 and trained on the SILVA 138.1 database trimmed to the V3–V4 region. This classifier was selected because it is one of the most widely used and validated approaches for short Illumina 16S reads and ensures compatibility with ASV‑based workflows. Because species-level discrimination in the V3–V4 dataset is constrained by the sequence variability retained within the amplified region, species-level assignments were considered within the known resolution limits of short-read amplicon profiling.

16S rRNA full-length amplicon sequencing

Full-length 16S rRNA reads generated using the Oxford Nanopore Technologies platform were adapter-trimmed using Cutadapt (v4.0, Dortmund, Germany) and quality-filtered to retain reads with a minimum length of 1,200 bp and an average quality score of ≥ Q10. Kraken2 was used for taxonomic classification of full‑length 16S rRNA reads because its k‑mer‑based algorithm is optimized for long reads and provides reliable species‑level accuracy for full‑length sequences. A custom reference database was constructed from the full‑length 16S rRNA gene sequences of the SILVA 138.1 reference set to ensure consistency with the other amplicon analyses. Sequencing read-count summaries for the 16P and 16F datasets are provided in Supplementary Table 1.

WMS sequencing

Sequencing data underwent a systematic bioinformatics pipeline for quality control, trimming, host genome removal, and microbiome profiling. Initially, raw sequencing reads were subjected to quality control using FastQC (version 0.11.9) to evaluate sequence quality and adapter contamination³⁷. Adapter trimming and quality filtering were conducted with Trim Galore (version 0.6.7) Reads shorter than 50 bp after trimming or those with average quality scores below 20 were excluded from further analyses³⁸. Subsequently, trimmed reads were aligned against the human reference genome hg38 using BWA-MEM (version 0.7.17) to remove host-derived sequences. Reads not mapped to the human genome were extracted, converted to BAM format, sorted by name, and then converted into FASTQ format using SAM tools (version 1.11) for downstream metagenomic analysis^39,40. MetaPhlAn was selected for WMS data due to its marker gene-based framework optimized for shotgun metagenomic profiling, enabling specific and conservative species-level assignment in complex genomic mixtures. Four output yield conditions (1, 5, 10, and 20 Gb) were analysed to evaluate the effect of sequencing depth. Libraries corresponding to 1 Gb and 20 Gb were physically constructed and sequenced, whereas 5 Gb and 10 Gb datasets were computationally generated by read-based subsampling of the 20 Gb FASTQ files using BBTools (reformat.sh). Subsampling was performed without replacement while preserving paired-end read structure to achieve the target output yields (Additional File 1: Table S2).

Statistical analyses

Quantitative accuracy assessment

An adjusted F1-score was used to evaluate both the quantitative accuracy of relative abundance estimates and their agreement with the expected composition of each MC. For each taxon, the observed relative abundance (Y) was compared to the predefined expected value (M). A tolerance threshold (T), defined as 10% of M, was applied to identify the range within which deviations were not penalised^2,5,13,41,42. Deviations exceeding this threshold were penalised using a power function of the form (excess), where the excess was defined as the difference between the deviation and the threshold. The exponent p = 0.5 was selected to implement a sublinear power function, which increases at a slower rate than a linear function with respect to the excess deviation, thereby limiting the rate of penalty escalation for moderate errors. True positives (TP), false positives (FP), and false negatives (FN) were computed based on the following definitions:

Equation 1:

True Positive (TP)

False Positive (FP)

False Negative (FN)

Adjusted F1-Score

Adjusted F1-score incorporating tolerance-based penalisation of abundance deviation

The final score was computed as the harmonic mean of precision and recall, where precision was defined as TP divided by TP plus FP, and recall as TP divided by TP plus FN.

Data visualization

All visualisations were conducted using R version 4.3.1. Bar graphs were generated using the ggplot2 package to compare relative abundances across sequencing methods and conditions. Heatmaps were constructed using ComplexHeatmap, gplots, and pheatmap to visualise taxonomic distributions and sample clustering. The ComplexHeatmap package was primarily used to incorporate multiple metadata layers and customise layout configurations. Spider graphs were produced using the fmsb package to present multivariate comparisons across sequencing strategies.

Results

Qualitative species-level detection based on presence or absence across sequencing platforms

To compare the species-level qualitative detection capability across sequencing platforms, we compared the expected taxa present in each MC with the taxa detected in the analysis results (Tables 1 and 2, Additional File 1: Figure S1). The presence/absence data was used to construct a confusion matrix for each sample, enabling a visual comparison of detection performance (Fig. 2). These comparisons were defined at the species level; therefore, taxa not recovered as species-level matches in Fig. 2 may still have shown concordant genus-level assignments, as summarized in Supplementary Table 3.

Fig. 2.

Matching heatmaps of qualitative taxon detection across sequencing platforms based on KMC. Presence or absence-based heatmaps visualise the agreement between expected and observed taxa across three sequencing platforms for each MC. Panels represent (A) KMCG, (B) KMCO, (C) KMCS, and (D) KMCV, corresponding to gut, oral, skin, and vaginal MCs, respectively. Each panel includes results from 16P, 16F, and WMS sequencing. These heatmaps represent the qualitative matching between the expected community composition and the observed taxa. Green indicates taxa that were expected and correctly observed (true positive), whereas red indicates expected taxa that were not observed (false negative).

In KMCG, none of the 12 expected taxa were detected in 16P. Eleven were matched in 16F and 12 were matched in WMS (Fig. 2). In KMCO, 1 out of 10 species were detected using 16P, 8 species using 16F, and all 10 species using WMS (Fig. 2). In KMCS, 3 out of 9 species was detected using 16P, 8 species using 16F, and all 9 species using WMS (Fig. 2). In KMCV, 1 out of 11 species were detected using 16P, 9 species using 16F, and all 11 species using WMS (Fig. 2).

In MCG, 3 out of 12 species were detected using 16P, 11 species using 16F, and all 12 species using WMS (Additional File 1: Figure S2). In MCO, 2 out of 6 species were detected using 16P, 6 species using 16F, and all 6 species using WMS. In MCS, 1 out of 6 species was detected using 16P, 6 species using 16F, and all 6 species using WMS. In MCV, 3 out of 6 species were detected using 16P, 4 species using 16F, and all 6 species using. Overall, KMC’s WMS showed 100% concordance, 16F showed 86%, and 16P showed 12%. In MC, WMS showed 100% concordance, 16F showed 90%, and 16P showed 31%.

Quantitative species-level detection based on abundance across sequencing platforms

We evaluated the reproducibility of abundance across sequencing platforms by MC type. The total correct abundance was calculated by adding the relative abundance of items corresponding to the predefined constituent species among the species observed in each KMC. Since there was no difference between the expected and detected species, WMS was 100% for all, and 16F was about 80%, showing that 16F had almost no correct species (Fig. 3). When observing the number of species, WMS detected no species other than the expected species, so the number was the same as the expected species. 16F was observed more than the expected species, and 16P was observed almost none. The true positive abundance ratio for the detected species was also higher for the expected species than for the detected species in the order of WMS, 16F, and 16P. Finally, the false negative abundance ratio was higher in the order of 16P, 16F, and WMS, contrary to the true positive abundance. The results in MC were also observed in the same manner (Additional File 1: Figure S3). In particular, WMS showed 100% matching of detected species to expected species in qualitative observation, but KMC showed an average of 86% and KM showed an average of 93%.

Fig. 3.

Quantitative evaluation of taxon detection performance across sequencing platforms using KMCs Barplots show quantitative metrics used to assess detection performance across different sequencing strategies, based on the known MC composition. (A) Total relative abundance assigned to expected taxa in the MC. (B) Number of species observed per platform. The dashed line indicates the total number of species present in the MC. (C) Total relative abundance of true positive taxa, defined as taxa that were included in the MC and correctly observed. (D) Total relative abundance of false negative taxa, defined as expected taxa that were not observed.

Impact of WMS sequencing conditions on quantitative performance

To identify the variation in quantitative detection values of WMS, we compared species-level abundance reproducibility under various WMS sequencing conditions, focusing on the variation in input DNA concentration and sequencing output yield of WMS (Fig. 4).

Fig. 4.

Comparison bar graph of WMS sequencing conditions using KMCs Species-level abundance reconstruction was compared across 12 WMS conditions defined by combinations of input DNA concentration (1, 10, and 100 ng) and output yield (1, 5, 10, and 20 Gb). (A–D) Comparison of quantitative evaluations for all MCs according to input DNA volume. (E–H) Comparison of quantitative evaluations for all MCs according to output yield.

We created 12 different WMS conditions by setting the input DNA concentration to 1, 10, and 100 ng and the sequencing output to 1, 5, and 20 Gb. In the case of KMCS, more detected species were observed than expected species when DNA input was 1 ng, so the total correct abundance ratio and true positive abundance ratio were also observed to be the lowest (Fig. 4). In addition, an upward trend was observed for 10 ng, including KMCO with the highest true positive abundance ratio. This showed that the input DNA volume affected the results. On the other hand, when comparing the sequencing output, all four KMCs showed consistent total correct positives and species numbers, with no significant difference between the outputs. The number of true positives and false negatives also remained constant regardless of the output, confirming that the sequencing depth had a limited effect on the reproducibility under the tested conditions. In the case of DNA-based MC, it was observed to be relatively stable without any effect according to the WMS sequencing condition compared to cell-based KMC (Additional file 1: Figure S4).

Species-level abundance reproducibility across WMS sequencing conditions

The resolution was evaluated at the species level for each KMC by comparing the reproducibility of quantitative detection using the adjusted F1-score (Figs. 5 and 6). This metric was used to assess condition-specific variability in the recovery of individual reference species across WMS sequencing parameters. Overall, WMS sequencing achieved high reproducibility for most species, although notable variations in resolution were observed across sequencing conditions. Species-specific sensitivity differences were evident, indicating that optimal detection may vary by taxon and experimental setup (Additional File 1: Figures S5 and S6).

Fig. 5.

Spider chart of species-level resolution by input DNA concentration across four KMCs in WMS Spider charts are presented to identify taxa that exhibit output-dependent sensitivity across WMS sequencing conditions in KMCs. Radial axes represent adjusted F1-scores ranging from 0 to 100, with values closer to the outer boundary indicating higher quantitative agreement with expected abundance. Each colored polygon corresponds to a distinct sequencing output yield (1, 5, 10, and 20 Gb), enabling direct comparison of condition-specific resolution patterns within each KMC type.

Fig. 6.

Spider chart of species-level resolution by sequencing output yield across four KMCs in WMS Spider charts are presented to identify taxa that exhibit input-dependent sensitivity across WMS sequencing conditions in whole-cell KMCs. Radial axes represent adjusted F1-scores ranging from 0 to 100, with values closer to the outer boundary indicating higher quantitative agreement with expected abundance. Each colored polygon corresponds to a distinct input DNA concentration (1 ng, 10 ng, and 100 ng), enabling direct comparison of condition-specific resolution patterns within each KMC type.

In KMCG, most species showed high reproducibility, but Lacticaseibacillus paracasei consistently showed low scores with an average score of 77 points (Fig. 5). Clostridium perfringens showed no deviation according to the degree of output yield with an average score of 96 points in the conditions of 1 ng and 10 ng DNA input, but it was lowered to 86 points in all output conditions when the input was 100 ng. When observed by output, differences in input were observed in all output conditions, so C. perfringens seemed to be affected by the amount of DNA input. In KMCO, Streptococcus pyogenes, Escherichia coli, Klebsiella pneumoniae, and Streptococcus pneumoniae showed low reproducibility, and in particular, S. pneumoniae was close to 0. S. pyogenes, E. coli, and K. pneumoniae showed improved reproducibility only in the 10 ng condition. In particular, E. coli showed high resolution only when the input was 10 ng, regardless of the output conditions. S. pneumoniae showed that the resolution improved as the input level increased and the output level increased, and the highest resolution was shown in the combination of input 10 ng and output 20 Gb. KMCS displayed the lowest overall reproducibility among the four KMCs, with no consistent improvement by output level. At a DNA input of 1 ng, Enterococcus faecium, Corynebacterium striatum, Staphylococcus epidermidis, and S. aureus exhibited poor reproducibility, which improved at input levels ≥ 10 ng. In contrast, Acinetobacter bereziniae and A. ursingii demonstrated high reproducibility at the 1 ng DNA input condition; however, their reproducibility declined as input DNA concentration increased. In KMCV, most species exhibited high reproducibility, except for Cryptococcus neoformans. The reproducibility of C. neoformans was shown to be affected by both input DNA level and output level, and improved as input DNA level and output level increased. In particular, the combination of 100 ng of input DNA and 20 Gb of output showed the most improved resolution.

When analyzing DNA-based MC, relatively little variability was observed compared to cell-based KMC. Acinetobacter johnsonii showed high resolution at an input level of 10 ng (Additional File 1: Figures S5 and S6).

Comparative resolution evaluation across KMC types

The final species-level resolution scores for each WMS sequencing condition are summarised in Fig. 7. In KMCG, all condition combinations yielded consistently high F1-scores, with values exceeding 95, regardless of the input and output settings (Fig. 7).

Fig. 7.

Heatmaps summarising species-level resolution scores across WMS conditions using KMCs Heatmaps present the final resolution scores calculated for each WMS condition, based on MCs and stratified according to MC type. Panels show results for (A) KMCG, (B) KMCO, (C) KMCS, and (D) KMCV. Each cell represents the resolution score derived from a specific combination of input DNA concentration and sequencing output yield.

In KMCO, 10 ng input DNA generally resulted in scores > 90 across output levels (Fig. 7). In KMCO, high-resolution scores were mainly related to output yields ≥ 10 Gb, regardless of input DNA concentration. In KMCS, conditions with ≥ 10 ng of DNA input showed high resolution scores overall at all output levels, while conditions with ≤ 10 ng showed relatively low scores (Fig. 7). In KMCV, high resolution was observed when the input level was ≥ 10 ng, and high resolution was observed when the output level was ≥ 10 Gb. Overall, the combination of 10 ng of input DNA and 10 Gb or more of output DNA consistently showed the highest resolution scores in most KMCs. MCG showed a resolution of 100 points in all conditions (Additional File 1: Figure S7). Next, MCV showed 99 score, and MCO and MCS also showed high scores of 97 scores on average in all conditions.

Discussion

Overall, WMS consistently yielded the highest reproducibility across all mock communities, while the amplicon-based strategies showed lower performance. Among the tested WMS conditions, the combination of 10 ng input DNA and ≥ 10 Gb sequencing output produced the most robust and consistent results. However, certain mock communities, such as KMCS, exhibited persistently low detection accuracy for specific taxa or under specific conditions. These findings demonstrate that both platform type and sequencing configuration critically affect species-level sensitivity, warranting further evaluation of the underlying biases using more refined metrics.

In this study, species-level resolution refers to the combined capacity to correctly identify and quantitatively reconstruct expected taxa, emphasizing compositional fidelity rather than merely detecting the presence of low-abundance species. Accurate quantitative representation of community structure is essential for meaningful biological interpretation. Assessing taxonomic resolution based solely on species presence or richness is insufficient to capture quantitative detection accuracy. Although traditional metrics like precision or recall partially reflect presence or recovery, they fail to account for deviations in abundance, such as species-specific over- or under-detection. To address this limitation, we employed an adjusted F1-score that incorporates quantitative imbalances among true positives, false positives, and false negatives. As formally defined in Eq. (1), the adjusted F1-score was calculated by defining true positives (TP) as the minimum of expected (M) and observed (Y) abundance, while false positives (FP) and false negatives (FN) were penalised only when the deviation exceeded the tolerance threshold (T). Deviations beyond T were weighted using a sublinear power function (exponent p = 0.5), allowing moderate abundance discrepancies to contribute proportionally without over-penalisation. The final adjusted F1-score was computed as the harmonic mean of precision and recall derived from these adjusted TP, FP, and FN values. The tolerance threshold (T) was set to 10% for the expected value (M) in the adjusted F1 score. There is still no consensus on the acceptable level of deviation in microbial community studies^1,2. The tolerance threshold (T) for strict quantitative comparison in the diagnostic field is set to < 5%⁴¹. In technical reproducibility tests for library prep, sequencing repeats, etc., it is set to ± 10–20%^1,42. In studies related to taxonomic resolution, it was set to around 10% and compared and analyzed⁴³. Considering these references and the aim of evaluating species-level abundance fidelity rather than strict diagnostic quantification, a 10% tolerance was selected as a balanced threshold that allows biologically meaningful variability while still penalising substantial deviations. This threshold reflects a compromise between overly strict diagnostic criteria (< 5%) and broader technical reproducibility margins (± 10–20%). Adjusted F1-scores were interpreted as continuous performance measures. Because the adjusted F1-score is inherently continuous, we avoided imposing arbitrary categorical thresholds and instead focused on comparative performance across sequencing conditions. In this framework, values approaching 1.0 indicate minimal deviation from expected abundance profiles, whereas progressively lower scores reflect increasing quantitative discrepancy. This metric assigns weighted penalties based on the numeric difference between observed and expected abundances, enabling an integrated assessment of both detection sensitivity and abundance fidelity. For example, KMCS displayed full species recovery in a qualitative presence/absence comparison (Fig. 2); however, under the 1 ng input condition, false negative abundance was notably high (Fig. 4). This approach enables quantitative comparisons of subtle differences in reproducibility, species-specific biases, and condition-driven sensitivity shifts that are not captured by qualitative analysis. It is particularly useful for evaluating the influence of library preparation and sequencing conditions on the effective resolution for each taxon.

The systematic comparison of input and output combinations revealed that the 10 ng DNA input with the 10 Gb output yield produced the most consistent and accurate results. Both lower (1 ng) and higher (100 ng) inputs were associated with reduced resolution. Interestingly, this pattern was not uniform across all taxa. In most cases, input amounts of 1 ng of DNA can lead to PCR bias and reduced library complexity due to limited starting material. However, Acinetobacter species in KMCS exhibited relatively higher resolution at the 1 ng input condition compared with higher input levels. This may be related to genome structural features and amplification dynamics. Acinetobacter genomes are moderately sized with relatively stable GC content, which may allow more uniform amplification under reduced template competition. At lower input levels, decreased template complexity may reduce inter-species competition during amplification, thereby improving relative abundance reconstruction for specific taxa. This observation suggests that amplification dynamics and genome-specific properties may interact with template concentration in a taxon-dependent manner, indicating that optimal input levels may vary according to species-specific characteristics rather than following a simple linear relationship with DNA quantity. In contrast, excessive input (100 ng) could result in nonspecific amplification or enzyme saturation, thereby decreasing quantification accuracy. These findings underscore that increased DNA input does not necessarily yield better resolution, highlighting the importance of optimizing concentration based on the analytical objective.

A similar plateau effect was observed for sequencing output, with little improvement noted beyond 10 Gb. Excessive sequencing depth beyond this threshold can lead to redundancy, decreased efficiency, and no further gain in taxonomic resolution. In some cases, resolution scores at a sequencing depth of 20 Gb were even lower than those observed at 10 Gb. Moreover, some species were detected only under specific input–output combinations, indicating that sequencing conditions directly influence species-specific detection sensitivity. Notably, DNA-based mock communities often exhibited higher resolution consistency compared to whole-cell mocks. This may be attributed to the absence of DNA extraction variability, as DNA mocks provide purified, high-quality input DNA, ensuring uniform library preparation efficiency^13,26. In contrast, whole-cell mocks are more susceptible to taxon-specific extraction efficiency and matrix effects, leading to greater variability in quantitative detection.

Resolution patterns also varied across MCs and among taxa under the same sequencing conditions. Differences in DNA extraction efficiency may partially explain the variability observed among taxa in the whole-cell mock communities. Gram-positive bacteria possess thick peptidoglycan cell walls that can make cell lysis more challenging during extraction, potentially reducing DNA recovery. In addition, certain organisms such as Cryptococcus neoformans have prominent polysaccharide capsules that may further hinder efficient DNA extraction. These biological characteristics may contribute to the reduced detection performance observed for some taxa in the cell-based mock communities. For example, S. pneumoniae in KMCO achieved detectable resolution only under the 100 ng-20 Gb combination, whereas the signal was nearly absent under all other conditions. In contrast, E. coli in the same community exhibited optimal performance at the 10 ng DNA input, with sharp declines at other concentrations. Similarly, C. neoformans in KMCV demonstrated stable resolution only under long-read platforms and high-output levels. This suggests that the fungal cell structure and large genome size of C. neoformans may affect DNA extraction and library preparation efficiency. In KMCS, A. bereziniae and A. ursingii exhibited the highest resolution at the 1 ng DNA input, with declining accuracy at increasing concentrations. In contrast, the resolution of other taxa in KMCS improved with increasing input. Collectively, our results emphasize the difficulty of applying a single resolution criterion across all taxa. Depending on the analysis goal or target species, sequencing and library conditions should be flexibly optimized. Overall, accurate performance evaluation requires comprehensive consideration of mock matrix composition, genome properties, extraction efficiency, and interactions with sequencing platforms. Importantly, the lower species-level performance observed for the 16S V3–V4 datasets relative to full-length 16S should not be interpreted as a pure platform effect alone, because the compared approaches differ in both target-region length and taxonomic assignment workflow.

C. perfringens of KMCG showed a decrease in resolution at 100 ng input (Fig. 5). Since C. perfringens has a genome structure with a high proportion of accessory genes, excessively high DNA amount can lead to PCR enzyme saturation or overamplification of specific sequences^44,45. This can result in strain-specific genes or low-frequency genes being relatively underamplified and not detected, which can result in a decrease in species-level abundance reproducibility (F1-score). E. coli of KMCO showed high resolution only at 10 ng input regardless of output conditions, and in particular, S. pneumoniae showed a reproducibility close to 0 in most conditions except this. This is because the starting amount of DNA has a significant impact on the uniformity of PCR amplification, and coverage bias occurs when it is too low or too high, so that PCR efficiency and library complexity are balanced at this concentration and various fragments are amplified without bias^44,46. A. bereziniae and A. ursingii of KMCS were more sensitive to this bias, suggesting that nonspecific amplification may have excessively generated short fragments when high-concentration DNA input was used^47,48. In contrast, C. neoformans of KMCV showed low reproducibility overall, which supports previous studies that it has a thick cell wall and a capsule composed of polysaccharides, making DNA extraction difficult regardless of the DNA extraction kit, which may lead to a decrease in sequencing resolution⁴⁹. In the results of this study, the resolution of C. neoformans increased as the input and output levels increased, indicating that high-concentration DNA input and high sequencing output are necessary to improve the resolution of C. neoformans⁴⁸. Finally, L. paracasei of KMCG consistently showed low resolution scores. The GC content of the 16S rRNA gene of L. paracasei is approximately 46.2–46.6%, which may affect the binding efficiency of primers during PCR amplification. When the GC content is high, the stability of double-stranded DNA increases, making denaturation difficult, which may reduce the efficiency of PCR amplification⁵⁰. It also suggests that the quantification of L. paracasei may be difficult due to the presence of other microorganisms in various sample matrices and chemical substances in the sample⁵¹. Collectively, these findings underscore the inherent difficulty of applying a uniform resolution criterion across diverse microbial taxa. Optimal sequencing and library preparation parameters appear to be species-dependent and must be tailored to the specific analytical goals. Consequently, rigorous performance evaluation requires the integrative consideration of multiple experimental and biological variables, including mock matrix composition, genome complexity, DNA extraction efficiency, and platform-specific biases.

Previous studies using MCs have primarily focused on qualitative assessments of sequencing platforms or analytical tools, often evaluating taxonomic resolution based on the presence or absence of a species or detection capability^43,52. However, few studies have quantitatively evaluated accuracy at the species level or systematically measured detection deviations^53,54. Instead, most investigations have relied on simple detection rates or descriptive differences in relative abundance, without incorporating metrics that capture abundance-level biases^55,56. Although prior research has reported platform-specific biases in short-read (16P; 16S V3–V4) and long-read (16F; full-length 16S) amplicon sequencing, the combined influence of sequencing input/output conditions and library construction parameters has been largely overlooked^28,57,58. To address this gap, we incorporated both amplicon- and WMS-based approaches and systematically evaluated 64 distinct input/output combinations across 112 datasets. This allowed for a comprehensive analysis of taxonomic resolution beyond platform-level comparison, enabling quantification of the influence of various experimental parameters on performance at the species level. A distinguishing feature of our study is the use of both commercially available DNA-based MCs (ATCC) and domestically developed whole-cell bead-based MCs (KMCs). The KMCs were formulated to mimic the matrix-bound state of microbes in clinical samples, allowing for assessment of DNA extraction efficiency and variation. These whole-cell mocks also offer practical advantages, such as room-temperature stability and long-term preservation⁵⁹. In addition, the microbial composition was designed to reflect Korean-specific body site microbiota, increasing the biological relevance for population-specific applications^60,61. This dual mock system enhances the interpretability of resolution outcomes, distinguishing our study from prior work. Notably, by integrating a multi-layered experimental design, our study provides objective evidence for selecting optimal sequencing configurations and contributes practical guidance for reproducible and accurate microbial analysis under variable conditions.

This study has several limitations. Although defined MCs provide a controlled framework for evaluating taxonomic resolution under controlled conditions, they are inherently less complex than real clinical or environmental samples and lack host-derived components or interspecies microbial interactions. In addition, because each sequencing condition was represented by a single dataset rather than technical replicates, statistical estimates of variance or confidence intervals were not assessed. In addition, taxonomic classification tools differ in their underlying algorithms and can influence the outcome of 16S analyses. The naive Bayes classifier used for V3–V4 amplicons and Kraken2 applied to full‑length 16S reads represent widely adopted pipelines for short‑read and long‑read sequencing data, respectively. These methodological differences reflect the need to match classifier design to read length and sequence characteristics. Although this choice ensures optimal accuracy for each dataset, it may also contribute to minor differences in species‑level resolution across sequencing strategies. Recognizing these analytical distinctions is important when interpreting comparative results. The choice of taxonomic profiling tools can also influence species-level resolution in metagenomic analyses. Kraken2 employs a k-mer–based classification strategy that assigns reads by exact matches to reference sequences, which can enable sensitive detection but may increase susceptibility to spurious assignments when reference databases contain closely related taxa. In contrast, MetaPhlAn relies on clade-specific marker genes to estimate taxonomic composition, which can improve specificity but may miss taxa lacking well-characterized marker sequences. These methodological differences highlight that tool selection can influence both detection sensitivity and taxonomic resolution. Therefore, species-level results from the V3–V4 datasets should be interpreted as practical outcomes of the selected short-amplicon workflow within the resolution limits of the target region, rather than as definitive evidence of species-level recoverability for every mock community member. However, negative controls consisting of distilled water were included in triplicate within each sequencing batch to monitor potential contamination. Consequently, resolution performance under mock conditions may differ from that observed in actual biological samples. Importantly, the non-linear relationship observed between template concentration, sequencing depth, and species-level resolution reflects fundamental amplification dynamics in mixed-template systems. Because similar template competition and amplification effects operate in complex microbial communities, the qualitative patterns identified here are likely to extend beyond defined mock communities, even if absolute performance values differ. This study did not include independent biological or technical replicates for each sequencing condition, and therefore the findings reflect condition-specific performance rather than variability across repeated experiments. Moreover, in some species, we noted discrepancies between input concentration and detection, likely owing to matrix effects and sensitivity in DNA extraction. Nevertheless, our results suggest that experimental limitations such as low input DNA or limited sequencing depth do not necessarily prevent meaningful analysis. Rather than discarding such samples, researchers may consider retaining them by aligning analytical strategies with specific biological goals and taxon characteristics, thereby enabling more informed and context-sensitive decision-making. In addition, the mock communities used in this study consist of a limited number of species compared with real-world microbiomes, which often contain hundreds to thousands of taxa. While defined mock communities provide controlled reference systems for evaluating methodological performance, their simplified composition does not fully represent the complexity of natural microbial ecosystems. Therefore, caution is required when extrapolating these results to highly diverse microbiome samples.

For example, in several MCs, reproducible performance was maintained at 10 Gb sequencing depth without requiring higher input, highlighting the feasibility of cost-effective analysis. Moreover, the use of whole-cell MCs provided valuable insights into the influence of DNA extraction and matrix structure, serving as a benchmark for optimising real-world workflows. The evaluation metrics and analytical framework proposed in this study can be applied to clinical or environmental samples to validate reproducibility under more complex conditions in future studies. Moreover, further investigations could integrate genome assembly accuracy, metagenome-assembled genome recovery, and functional gene resolution to advance efforts in developing practical and consistently high under tested conditions guidelines for sequencing strategy selection across diverse microbiome research objectives. Species-level resolution in the present study reflects the combined effects of multiple technical steps across the workflow, including sample preparation, DNA recovery, and taxonomic assignment, in addition to the sequencing configuration itself.

While the present study focused on species-level taxonomic resolution, the evaluation framework established here provides a foundation for extension to functional gene and pathway recovery, particularly in whole-metagenome sequencing (WMS) datasets. Functional profiling introduces additional layers of complexity beyond taxonomic assignment, including gene copy number variation, horizontal gene transfer events, and assembly-dependent biases that differentially affect pathway reconstruction tools such as HUMAnN or MetaCyc-based pipelines. Investigating how sequencing input mass and depth influence not only community composition but also the accuracy and completeness of metabolic pathway inference would substantially broaden the translational relevance of this framework especially for studies linking microbial function to host phenotype or environmental processes. We therefore identify functional resolution benchmarking as a priority direction for future work.

Conclusions

We quantitatively evaluated taxonomic resolution at the species level across various sequencing conditions using defined MCs, providing a basis for selecting optimal configurations tailored to specific analytical goals. By applying an adjusted F1-score, we quantified accuracy differences across input/output combinations and sequencing platforms. We identified the 10 ng DNA input and 10 Gb output condition as the most consistently reliable. We observed species-specific differences in detection patterns, with certain taxa detectable only under specific conditions, highlighting the need for strategic selection based on target species and study objectives. These findings offer practical guidance for maximizing sequencing efficiency under realistic constraints.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

MC

Mock community

KMC

Korea domestically developed whole-cell-based mock community

16S rRNA sequencing

16S ribosomal RNA gene sequencing

16F sequencing

Full-length 16S rRNA gene sequencing

16P sequencing

16S rRNA V3–V4 region sequencing

WMS sequencing

Whole metagenome shotgun sequencing

ASV

Amplicon sequence variant

KMCG

Whole-cell mock community from the gut

KMCO

Whole-cell mock community from the oral

KMCS

Whole-cell mock community from the skin

KMCV

Whole-cell mock community from the vaginal

MCG

DNA-based mock community from the gut

MCO

DNA-based mock community from the oral

MCS

DNA-based mock community from the skin

MCV

DNA-based mock community from the vaginal

Gb

Gigabases

TP

True positives

FP

False positives

FN

False negatives

Author contributions

S.H.L and H.A.L contributed to the conceptualisation of the study, and were responsible for writing the original draft and performing the formal analysis. They also revised the manuscript. H.J.K and J.W.K was involved in performing the experimental work and participated in reviewing and refining the manuscript. K.J.L. contributed to the acquisition of funding and the supervision of the study. All authors have read and approved the final manuscript.

Funding

This research was supported by the National Institute of Health (NIH) research project (project No. 2023-NI020–01, 2023-NI-019-03).

Data availability

The datasets generated and analysed during the current study are available in the NCBI repository, accession number PRJNA1464019. https://www.ncbi.nlm.nih.gov/bioproject/PRJNA1464019.

Declarations

Competing interests

The authors declare no competing interests.

Ethical approval

Not applicable.

Consent to participate

Not applicable.