Integrative analysis of microbial 16S gene and shotgun metagenomic sequencing data improves statistical efficiency in testing differential abundance

Abstract

The most widely used technologies for profiling microbial communities are 16S marker-gene sequencing and shotgun metagenomic sequencing. Surprisingly, many microbiome studies have performed both experiments on the same cohort of samples. The two sequencing datasets often reveal consistent patterns of microbial signatures, suggesting that an integrative analysis of both datasets could enhance the testing power for these signatures. However, differential experimental biases, partially overlapping samples, and uneven library sizes pose tremendous challenges when combining the two datasets. In this article, we introduce the first method of this kind, named Com-2seq, that combines the two datasets for testing differential abundance at the genus level as well as the community level while overcoming these difficulties. Our simulation studies demonstrate that Com-2seq substantially enhances statistical efficiency over analysis of a single dataset and outperforms two ad hoc approaches to integrative analysis. In analysis of real microbiome data, Com-2seq uncovered scientifically plausible findings, namely, the association of Butyrivibrio, Gemella and Ignavigranum with prediabetes status, which would have been missed by analyzing a single dataset. Butyrivibrio failed to reach the significance level in the analysis of each dataset despite showing a consistent trend; Gemella and Ignavigranum failed to produce adequate data in the 16S experiment.

1. Introduction

Thanks to technological advances in high-throughput sequencing, microbiome research has proliferated in the past decade and revealed important relationships between human microbiome and many diseases and conditions such as inflammatory bowl diseases [1], obesity and type II diabetes [2], and even cancers [3]. The most widely used technologies for profiling microbial communities are 16S marker-gene sequencing and shotgun metagenomic sequencing (SMS) [4]. The 16S method employs primers that target at highly variable regions of the 16S ribosomal RNA gene, which are then PCR amplified and sequenced. This approach is well-tested, fast and cost effective, and provides a low-resolution view of a microbial community, typically at the genus level. On the other hand, SMS extracts all microbial genomes within a sample, which are then fragmented and sequenced. This technique offers detailed genomic information, including higher taxonomic resolution and additional functional capabilities, but it is 10–30 times more expensive to perform a sequencing experiment and even more challenging to conduct bioinformatics analysis. The 16S data are routinely processed into a taxa count table by the bioinformatics platform QIIME2 [5]. While these is less consensus, the SMS data can be classified into taxonomies using tools such as MetaPhlAn [6–8], Kraken [9, 10], or more recently, Woltka [11]. With either sequencing method, the resulting taxa count data are known to be compositional (i.e., providing information on the relative abundances of taxa), sparse (containing 50–90% zeros), high-dimensional (involving 50–10000 taxa), and overdispersed.

Both the 16S and SMS methods introduce experimental bias at every step of the experiment (i.e., DNA extraction, PCR amplification, amplicon or metagenomic sequencing, and bioinformatics processing), as each step preferentially measures certain taxa over others [12], leading to a systematic distortion of the measured taxon abundances from their actual values. This bias is significantly different between 16S and SMS due to differences in their processes (e.g., the presence or absence of PCR amplification) and even in the protocols used for the same steps [13]. Consequently, each experiment may underrepresent or completely miss some taxa [14]. As these taxa may be complementary, combining data from both experiments could improve the coverage of complex microbial communities.

Surprisingly, many microbiome studies have performed both the 16S and SMS experiments on the same cohort of samples. On Qiita, the largest open-source microbiome study management platform (https://qiita.ucsd.edu) [15], 26 out of all 698 studies (as of June 2023) possess datasets from both experiments. This number is likely an underestimation, as many studies might have deposited only the dataset used in their publications to Qiita. In addition, many major national and international large-scale studies have collected both 16S and SMS data on the same samples [16] (Table S1). These studies include the Phase-I Human Microbiome Project (HMP) [17], the Phase-II Integrative HMP (iHMP) [18], the Finnish National FINRISK study [19], the Dutch LifeLines-DEEP (LLD) cohort study [20], The Environmental Determinants of Diabetes in the Young (TEDDY) study [21], and the Earth Microbiome Project (EMP) [22]. It is expected that any dataset derived from these studies will include both types of data, underscoring the relevance of Com-2seq for integrative analysis. There are primarily two scenarios leading to the availability of both datasets. In one, studies initially used 16S and subsequently employed SMS to delve into higher-resolution taxa or biological functions. In the other scenario, studies initially conducted SMS but later added 16S due to its low cost. The taxonomic profiles from 16S and SMS for the same samples often reveal consistent microbial signatures [14, 23–29], suggesting that an integrative analysis of both taxonomic profiles could enhance the testing power for these signatures.

In Table S2, we summarized the 16S and SMS datasets for the 26 studies sourced from Qiita, including their overlapping samples and taxa (at the genus level). Although the overlaps in samples and genera are often incomplete, they remain substantial for most studies, as illustrated in Figure 1 (left). The library sizes (i.e., depths) yielded by the two experiments may vary significantly, with differences ranging from 1.4 to 1500 fold. In Figure 2, we contrasted the relative abundances of genera from 16S and SMS, for each overlapping sample in the ORIGINS study. There is a notable consistency between the 16S and SMS data at many taxa, such as the most abundant genus Prevotella. However, systematic departures from the 45° line at some taxa, like the second most abundant genus Haemophilus, highlight the differential experimental biases of the two experiments. Some genera, including Gemella and Geobacillus, show a marked discrepancy, appearing almost absent in one dataset but reasonably abundant in the other. Figure 3 for the Dietary study exhibits similar patterns but with a higher degree of overdispersion (i.e., a larger variation of data from the fitted line) compared to Figure 2.

Figure 1:

Illustration of the data structure (left) and analytical strategies (right). The shaded area represents the 16S and SMS data for the overlapping samples and taxa. New EE is our Equation (1) or bias-corrected Equation (S3). LOCOM EE is the LOCOM estimating equation [30].

Figure 2:

Scatter plots of genus relative abundances (RA) obtained from the 16S and SMS experiments, for the top 1–15 and 36–50 most abundant genera (ordered by decreasing abundance) in the ORIGINS study and for each overlapping sample. The $\rho$ value is the Pearson correlation coefficient. The red line is the 45° reference line. The black line depicts a fitted linear regression.

Figure 3:

Scatter plots of genus relative abundances obtained from 16S and SMS, for the top 1–15 and 36–50 most abundant genera in the Dietary study and for each overlapping sample. See more information in the caption of Figure 2.

Currently, researchers either discard one of the two datasets completely or use different datasets for different research objectives. To our knowledge, there currently exists no statistical method designed for the integrative analysis of the two sequencing datasets. To address this gap, we introduce a new method called Com-2seq. Our method is built on the LOCOM model [30], which utilizes logistic regression to test for differential abundance of taxa and was crafted to be robust against experimental bias and to effectively handle the complexities associated with taxa count data, such as compositionality and sparsity. Com-2seq is designed to combine data from 16S and SMS to test for differential abundance at both the genus and community levels, while overcoming the challenges posed by differential experimental biases, partially overlapping samples, and uneven library sizes. We benchmark Com-2seq against two ad hoc approaches and conduct extensive simulation studies to evaluate the performance of all methods. Finally, we apply these methods to analyze the 16S and SMS data from the ORIGINS and Dietary studies.

2. Methods

2.1. Com-2seq

Let $A_{ik}=1$ indicate that sample $i$ is available in data source $k$, where $k=1$ represents 16S and $k=2$ represents SMS. For samples in the overlapping subset, both $A_{i1}$ and $A_{i2}$ are equal to 1. In contrast, for non-overlapping samples, one of $A_{i1}$ or $A_{i2}$ will be 1, while the other will be 0. Let $Y_{ik,j}$ be the read count of taxon $j(j=1,\dots ,J)$ in sample $i(i=1,\dots ,n)$ obtained from data source $k$. Let $Z_i$ be a vector of covariates including the trait of interest that we wish to test and a group of potential confounders that we wish to adjust for, but excluding the intercept. We define $p_{ik,j}$ to be the expected value of the observed relative abundance of taxon $j$ in sample $i$ and data source $k$ and relate it to the underlying, true relative abundance ${\pi }_{i,j}$, which is unrelated to any data source, by $\log \left(p_{ik,j}\right)={\gamma }_{k,j}+\log \left({\pi }_{i,j}\right)+{\alpha }_{ik}$, where ${\gamma }_{k,j}$ is a taxon- and source-specific bias factor that describes how the observed relative abundance is distorted by the bias and ${\alpha }_{ik}$ is the normalization factor that ensures the composition constraint ${\sum }_{j=1}^J{p}_{ik,j}=1$. Following [31], we assume $\log \left({\pi }_{i,j}\right)=\log \left({\pi }_j^0\right)+Z_i^{\text{T}}{\beta }_j$, where ${\beta }_j$ contains the effect sizes of all covariates and is our parameter of interest, and ${\pi }_j^0$ contains the baseline relative abundances. This formulation of $p_{ik,j}$ implies a polychotomous logistic regression for the full $n\times J$ taxa count table, satisfying $\log \left(p_{ik,j}/p_{ik,J}\right)={\eta }_{k,j}+Z_i^{\text{T}}\left({\beta }_j-{\beta }_J\right)$ for $j=1,\dots J-1$, where $J$ is chosen as the reference taxon without loss of generality. The intercept ${\eta }_{k,j}=\left({\gamma }_{k,j}-{\gamma }_{k,J}\right)+\left[\log \left({\pi }_j^0\right)-\log \left({\pi }_J^0\right)\right]$ is treated as a free parameter for each $k$ and $j$. Since our interest lies in ${\beta }_j$ only, there is no need to distinguish between ${\gamma }_{k,j}$ and ${\pi }_j^0$ within ${\eta }_{k,j}$. The polychotomous logistic regression can be numerically challenging, as the analysis of each taxon may require estimating all the ${\beta }_j$ parameters. To simplify this, we follow the approach of Begg and Gray (28) using separate or individualized logistic regressions. Each logistic regression analyzes data from only two taxa at a time. Rather than considering all possible pairs of taxa, we select one taxon (taxon $J$ again) with the highest mean relative abundance across both data sources to serve as the reference taxon and compare each of the other taxa to this reference using logistic regressions. Importantly, the reference taxon does not need to be a null taxon, unassociated with the trait, as we will demonstrate later. Given that the most abundant taxa are usually well captured by both 16S and SMS experiments, our selection often leads to a reference taxon that is among the most abundant in each data source. When comparing taxon $j$ to the reference taxon $J$, we define a new relative abundance between the two taxa as ${\mu }_{ik,j}=p_{ik,j}/\left(p_{ik,j}+p_{ik,J}\right)$ to obtain a standard logistic regression

$$\log \left\{{\mu }_{ik,j}/\left(1-{\mu }_{ik,j}\right)\right\}={\eta }_{k,j}+Z_i^{\text{T}}\left({\beta }_j-{\beta }_J\right),1\le j\le J-1.$$

To ensure identifiability of ${\beta }_j\text{s}$, we set ${\beta }_J=0$. In this approach, the analysis of taxon $j$ depends only on ${\beta }_j$, which represents the odds ratio comparing taxon $j$ to the reference $J$.

To combine data from both sources for inference on the ${\beta }_j\text{s}$, we solve the following estimating equation (EE) for ${\beta }_j$ for each $j$:

$${\mathbb{U}}_j\left({\eta }_{1,j},{\eta }_{2,j},{\beta }_j\right)=\sum _{i=1}^n\sum _{k=1}^2\mathbb{I}\left(A_{ik}=1\right){\omega }_{ik,j}\left(\frac{Y_{ik,j}}{Y_{ik,j}+Y_{ik,J}}-{\mu }_{ik,j}\right)\left[\begin{array}{c}\mathbb{I}(k=1)\\ \mathbb{I}(k=2)\\ Z_i\end{array}\right]=0,$$ (1)

where ${\omega }_{ik,j}$ represents the weight for the (centered) relative abundance of taxon $j$ in sample $i$ from data source $k$ that we will discuss later. When taxon $j$ has sparse count data, it may occur that all (or nearly all) counts for that taxon are zero in one group (e.g., the case or control group), which is referred to as separation in the literature on logistic regression [32]. In this case, we solve the Firth-bias-corrected estimating equation derived in the Supplementary Materials instead. The solution is denoted by ${\widehat{\beta }}_j$. Note that the EE of LOCOM is a special case of Equation (1) when ${\omega }_{ik,j}$ is set to $Y_{ik,j}+Y_{ik,J}$ for one data source and 0 for the other. In the generalized estimating equation (GEE) framework, Equation (1) corresponds to a GEE with an independence correlation structure, which has been shown to be highly efficient and robust for estimating coefficients in logistic regressions for paired data compared to using an exchangeable correlation structure [33]. This is particularly true when the correlation between the two data sources is not high and the sample size is not large. In the Supplementary Materials, we derived the estimating equation based on an exchangeable correlation structure, referred to as Com-2seq-exch. We will compare the performance of Com-2seq and Com-2seq-exch, which differ in their correlation structures, in our simulation studies.

We explore two options for the weights. When applying the “count” weights ${\omega }_{ik,j}=Y_{ik,j}+Y_{ik,J}$, the relative abundances in Equation (1) with larger total counts receive greater weighting. These weights are optimal in scenarios with minimal overdispersion in the count data, as Equation (1) under these weights becomes the score equation for binomial count data (without overdispersion). Additionally, these weights are suitable for taxa when the count data from one data source are very sparse (i.e., predominantly zeros) while the data from the other source are not. However, since taxa count data are often overdispersed, the amount of information tends to plateau even with moderate coverage of reads. In such scenarios, it becomes sensible to use “equal” weights ${\omega }_{ik,j}=1$. These weights ensure that the relative abundance data from the two sources are treated equally, irrespective of their total counts. These weights are particularly important when the library sizes of SMS are orders of magnitude (e.g., 10 times) larger than those of 16S, preventing the SMS data from dominating the results. Since it is not known a priori which weighting scheme is more effective for each taxon, Com-2seq applies Equation (1) using both sets of weights and then constructs an omnibus test that combines their results, as detailed below. When a taxon is observed in only one data source, Com-2seq uses weights ${\omega }_{ik,j}=Y_{ik,j}+Y_{ik,J}$ for that source and sets ${\omega }_{ik,j}=0$ for the other, simplifying Equation (1) to the LOCOM EE (Figure 1, right). In cases where only relative abundance data are available, instead of count data, Com-2seq applies equal weights ${\omega }_{ik,j}=1$ across all data.

Let ${\beta }_{j,1}$ be the component of ${\beta }_j$ that corresponds to the trait of interest. To test the association between taxon $j$ and the trait, it is insufficient to simply test ${\beta }_{j,1}=0$, unless the reference taxon is, by chance, a null taxon. Given the lack of a priori knowledge about the reference taxon, we seek an approach that does not rely on such knowledge and also allows for testing the reference taxon itself. Following LOCOM and other compositional methods, we assume that more than half of the taxa are null. Under this assumption, we expect that $\text{media}{\text{n}}_{j^{\prime }=1,\dots ,J\left\{{\beta }_{j^{\prime },1}\right\}}$ corresponds to ${\beta }_{j^{*},1}$ for some null taxon $j^{*}$. Therefore, testing

$$H_{j0}:{\beta }_{j,1}-\text{media}{\text{n}}_{j^{\prime }=1,\dots ,J}\left\{{\beta }_{j^{\prime },1}\right\}=0$$

is equivalent to testing whether taxon $j$ is a null taxon. Consequently, we consider the test statistic ${\widehat{\beta }}_{j,1}–\text{media}{\text{n}}_{j^{\prime }=1,\dots ,J}\left\{{\widehat{\beta }}_{j^{\prime },1}\right\}$. Notably, in the simplest case testing a binary trait without additional covariates, this test statistic is invariant to the choice of reference taxon $J$.

To avoid making distributional assumptions with sparse microbiome data, Com-2seq employs permutation-based inference. Specifically, Potter’s method [34] is used to generate null replicates of data, while Sandve’s sequential stopping rule [35] is applied to terminate the permutation procedure efficiently. This procedure yields two-sided permutation $p$-values for each taxon, along with $p$-values adjusted for multiple testing (q-values). For more details on the permutation procedure, please refer to the LOCOM paper [30]. Care must be taken to preserve the sample structure when generating permutation replicates. The permutation procedure should be stratified, restricting the shuffling of trait residuals within specific strata: one for samples with data from both sources, one for samples with data exclusively from 16S, and another for those with data solely from SMS (Figure 1, right). These permutation replicates enable us to construct an omnibus test for each taxon, leveraging the benefits of both count and equal weights. Essentially, we use the minimum of the $p$-values obtained using both weight sets as the final test statistic and find the corresponding minima from the permutation replicates to simulate the null distribution [36].

Even if no individual taxon reaches the significance level, there remains interest in evaluating whether the overall microbial community is associated with the change of trait. To test this community-level (i.e., global) hypothesis, i.e., $H_{j0}$ holds for all taxa in the community, we employ the Harmonic Mean (HM) of the taxon-specific $p$-values $\left(p_j\right)$ as the test statistic: $J/\left({\sum }_{j=1}^J{p}_j^{-1}\right)$. We assess its significance and calculate the global $p$-value using the previously generated permutation replicates. For details on how to construct the statistics from permutation replicates, please refer to the LOCOM paper.

Finally, we implement a filter to exclude extremely rare taxa that may compromise the validity of Com-2seq. Recall that the filter in LOCOM excludes rare taxa present in fewer than 20% of samples. In Com-2seq, a sample with a non-zero count from either data source makes a contribute to the third equation of (1), which is most relevant to ${\beta }_j$. Therefore, we retain a taxon if more than 20% of the samples have non-zero counts from either data source. Additionally, we retain a taxon if it passes the LOCOM filter applied to either 16S or SMS data, given the filter’s effectiveness even in scenarios where one data source consists entirely of zeros. The complete Com-2seq algorithm is provided in the Supplementary Materials.

2.2. Ad hoc approaches: Com-count and Com-p

To benchmark our new method, we consider two ad hoc approaches: one initially pools the taxon counts of the two datasets and then applies LOCOM to the pooled data, referred to as Com-count; the other applies LOCOM separately to the two datasets and subsequently combines the resulting two $p$-values at each taxon using the Cauchy $p$-value combination method [37], referred to as Com-p.

Specifically, Com-count first creates an “augmented” table consisting of the union of samples and the union of taxa across the two data sources. It then pools (i.e., sums) the read counts for cells that have both 16S and SMS data, copies the read counts for cells with data from only one source, and fills in zeros for cells where no data exists (i.e., where samples are found in only one data source and taxa are present exclusively in the other, resulting in empty cells). LOCOM is then applied to this augmented table. However, the zero-filling strategy may introduce spurious associations, especially if the samples filled with zeros have a different trait distribution compared to those with observed, potentially non-zero counts. Moreover, the results of Com-count are often influenced more by the data from the source with larger library sizes. Finally, the LOCOM filter applied to the “augmented” table is more stringent than the Com-2seq filter mentioned earlier.

Com-p combines the two LOCOM $p$-values into a single $p$-value for taxa with data from both sources using the Cauchy method. These $p$-values, along with the LOCOM $p$-values for taxa with data from only one source, are then corrected for multiple testing by the Benjamini-Hochberg method [38]. They are also aggregated into a single global $p$-value (again using the Cauchy method) for testing the global hypothesis. We choose the Cauchy method because the two $p$-values from the same taxon are expected to be correlated due to overlapping samples, and the $p$-values across taxa also tend to be correlated due to inter-taxon interactions. We prefer Cauchy over HM for the Com-p approach because HM resulted in an inflated error rate in our simulations (results not shown). It is worth noting that while both the Cauchy and HM methods in Com-p are based on asymptotic theories, the HM-combined global $p$-value in Com-2seq relies on permutation. However, Com-p is inherently unable to produce an overall $p$-value that is more significant than the most significant individual $p$-value, while Com-2seq can achieve this by combining the two datasets at the more appropriate taxon-count and relative-abundance levels. Additionally, Com-p combines $p$-values without considering directions of association in the two data sources, while Com-2seq can enhance signals if they are consistent across the two sources and disregard contradictory ones. Furthermore, taxa that fail the LOCOM filter applied to each taxa count table are completely overlooked by Com-p, whereas they still have a good chance of passing the Com-2seq filter (which essentially relies on pooled data), especially when the sample overlap is substantial.

3. Results

3.1. Simulation settings

Our simulations are based on data on 856 taxa of the upper-respiratory-tract (URT) microbiome by Charlson et al. [39]. We fixed the sample size to 100 unless otherwise specified and considered a binary trait $T_i$ throughout the simulations. In some cases, we also simulated a continuous confounder $C_i$ by drawing values from $U[-1,1]$ for samples with $T_i=0$ and from $U[0,2]$ for those with $T_i=1$. We used the two sets of associated taxa employed in [30], referred to as M1 and M2, which are a random sample of 20 taxa with mean relative abundances greater than 0.005 and the five most abundant taxa, respectively. Additionally, we considered a set of rare associated taxa by randomly sampling 50 taxa with mean relative abundances between 0.0005 and 0.001; we refer to this set as M3. When a confounder was present, we randomly sampled 5 taxa with mean relative abundances greater than 0.005 to be associated with the confounder.

We assumed that the 856 taxa constitute the complete set of underlying taxa in the community and generated bias factors ${\gamma }_{1,j}$ and ${\gamma }_{2,j}$ for taxon $j$ in the 16S and SMS experiments, respectively. We set ${\gamma }_{1,j}$ and ${\gamma }_{2,j}$ to the small value of −5 to create absence of some taxa in each data source: in M1 (M2 and M3), we selected two sets of five (two and five) non-overlapping taxa among the associated taxa to be absent in 16S and SMS, respectively; we also sampled 20% of null taxa to be absent in each data source. We set ${\gamma }_{1,j}$ and ${\gamma }_{2,j}$ to 1 for the most abundant taxon, reflecting the efficiency of both sequencing methods in capturing this taxon. For all other taxa, we independently drew the ${\gamma }_{1,j}$ and ${\gamma }_{2,j}$ values from $N\left(0,{0.5}^2\right)$.

We simulated read count data for the 856 taxa in the two data sources, incorporating the effects of the trait and confounder as well as the influences of bias factors. First, we sampled the baseline relative abundances $\left({\pi }_{i,1}^{(0)},{\pi }_{i,2}^{(0)},\dots ,{\pi }_{i,J}^{(0)}\right)$ of all taxa for each sample from $Dirichlet(\overline{\pi },\theta )$, where $\overline{\pi }$ contains the mean relative abundances and $\theta$ is the overdispersion parameter estimated from fitting the Dirichlet-Multinomial (DM) model to the URT data. The overdispersion parameter $\theta$ controls the sample heterogeneity in baseline relative abundances, not including the variability in read count data. Thus, we set $\theta$ to 0.01, which is half of the overall overdispersion (0.02) estimated from the URT data. Then, we formed the expected value of the observed relative abundances obtained from the kth data source, $p_{ik,j}$, by spiking in the associated taxa and confounder-associated taxa, applying bias factors on all taxa, and then normalizing the relative abundances to sum to 1, resulting in the following equation:

$$p_{ik,j}=\frac{\exp \left({\gamma }_{k,j}+{\beta }_{1,j}{T}_i+{\beta }_{2,j}{C}_i\right){\pi }_{ij}^{(0)}}{{\sum }_{j^{\prime }=1}^J\exp \left({\gamma }_{k,j^{\prime }}+{\beta }_{1,j^{\prime }}{T}_i+{\beta }_{2,j^{\prime }}{C}_i\right){\pi }_{i{j}^{\prime }}^{(0)}}.$$ (2)

Here, ${\beta }_{1,j}=0$ for null taxa, and ${\beta }_{2,j}=0$ for confounder-independent taxa. For simplicity, we set ${\beta }_{1,j}=\beta$ for all associated taxa, referred to as the effect size, and we fixed ${\beta }_{2,j}=\log (1.5)$ for all confounder-associated taxa. Subsequently, we generated the read count data for sample $i$ in data source $k$ from the DM distribution with mean $\left(p_{ik,1},p_{ik,2},\dots ,p_{ik,J}\right)$, overdispersion parameter ${\tau }_k$, and library size drawn from $N\left({\nu }_1,{\left({\nu }_1/3\right)}^2\right)$ and $N\left({\nu }_2,{\left({\nu }_2/3\right)}^2\right)$ for 16S and SMS, respectively, with left truncation at 2,000. We fixed ${\nu }_1=10,000$ and varied ${\nu }_2$ to achieve the ratio of depth at ${\nu }_1:{\nu }_2=1:1$ or $1:10$. The overdispersion parameter ${\tau }_k$ controls the deviation of observed relative abundances from their expected values $p_{ik,j}$ in the kth data source. We set ${\tau }_1={\tau }_2=\tau$ without loss of generality and varied $\tau$ between 0.01 and 0.001, corresponding to large and small deviations as observed in the Dietary and ORIGINS data, respectively.

In cases where samples with 16S and SMS data completely overlap, we generated taxa count data from both sources for all 100 samples. In cases where these samples partially overlap, we generated data from both sources for 40 samples (15 cases and 25 controls), from 16S only for 40 samples (30 cases and 10 controls), and from SMS only for 20 samples (5 cases and 15 controls), resulting in a total of 100 samples, with 50 cases and 50 controls, and varying case-control ratios across the three strata of samples.

We applied Com-2seq (assuming the omnibus test if not stated otherwise), Com-count, and Com-p to conduct integrative analysis of 16S and SMS data, using the LOCOM results from analyzing individual datasets as benchmarks. We evaluated the sensitivity and empirical false discovery rate (FDR) of each method for testing individual taxa at the nominal FDR level of 20% (chosen to be relatively high due to the small numbers of associated taxa in both M1 and M2), and the type I error and power of each method for testing the global association at the nominal level of 0.05. The type I error results were based on 10,000 replicates of simulated data, while all other results were derived from 1,000 replicates.

To validate that the simulated data captured the essential characteristics of the ORIGINS data (Figure 2) and the Dietary data (Figure 3), we present scatter plots for the simulated data based on the sample size (n = 152) of the ORIGINS study and $\tau =0.001$ in Figure S1, and scatter plots for the simulated data based on the sample size (n = 76) of the Dietary study and $\tau =0.01$ in Figure S2. We observe that the deviations of individual data points from the fitted line in the simulated data with $\tau =0.001$ resemble those of the ORIGIN data, while the simulated data with $\tau =0.01$ resemble those of the Diet data. Indeed, the 16S-SMS data correlations, estimated by Com-2seq-exch, exhibit comparable distributions between the simulated and real data (Figure S3).

3.2. Simulation results

The results for the complete-overlap case, based on the four combinations of overdispersion value and ratio of depth, are shown in Figures 4 and S4–S6, respectively. Across all scenarios, all global tests maintained type I error control and all taxon-level tests maintained FDR control, both at their nominal levels. All integrative analyses consistently improved statistical efficiency over analyses conducted on individual datasets, at both the taxon and global levels. The increase in sensitivity of detecting associated taxa is substantial, as a single sequencing experiment may overlook certain associated taxa, whereas the combination of two experiments provided better coverage. The boost in power of the global test is less pronounced, as the HM and Cauchy statistics underlying the global tests were primarily influenced b y a few smallest individual $p$-values rather than the total number of associated taxa. Among the three integrative approaches, Com-2seq consistently yielded the highest or nearly highest global power and sensitivity. Additionally, we confirmed that within the Com-2seq framework, the omnibus test consistently achieved optimal performance among tests utilizing count or equal weights (Figures S7 and S8).

Figure 4:

Type I error rate (first row, when $\beta =0$) and power (second row) for testing the global association at the nominal level of 0.05 (gray dashed line), and sensitivity (third row) and empirical FDR (fourth row) for testing individual taxa at the nominal FDR level of 0.2 (gray dashed line), based on data simulated with completely overlapping samples, overdispersion of $\tau =0.01$, and depth ratio of 1:10.

Similar patterns of results are observed in the partial-overlap case (Figures 5, S9, and S10). One notable difference from the complete-overlap case is that, Com-count failed to control the type I error, as its zero-filling strategy tended to generate spurious associations. Another difference lies in the permutation scheme utilized in Com-2seq, which was based on three strata of samples in the partial-overlap case, as illustrated in Figure 1 (right). This permutation scheme was the only one that resulted in correct type I error control, in contrast to two alternative schemes: one that pools samples with data from only one experiment into one stratum (totaling two strata), and another that pools all samples together (resulting in one stratum). Finally, the same patterns of results were observed when a confounder was simulated (Figure S11). It was confirmed that the confounding effect was significant, leading to markedly inflated type I error if left uncontrolled. However, all methods exhibited proper type I error control after adjusting for the confounder.

Figure 5:

Results for data simulated with partially overlapping samples. Com-2seq annotated with “(3)”, “(2)” and “(1)” used permutation schemes that were based on three strata (the recommended one), two strata, and one stratum, respectively. See more information in the caption of Figure 4.

The results of Com-2seq and Com-2seq-exch are presented in Figure S12. While both methods effectively controlled the FDR, Com-2seq consistently exhibited superior sensitivity across all scenarios, especially when the associated taxa had relatively low abundance, as seen in M1 and M3. Com-2seq maintained its superior sensitivity across different ranges of 16S-SMS data correlations (Figure S13), and as $\tau$ was changed from 0.01 to 0.001 (Figure S14), resulting in uniformly higher correlations (Figure S3). Additionally, Figure S15 confirmed that, although the coefficient estimates from both methods were similar, Com-2seq produced estimates with consistently smaller variances compared to Com-2seq-exch. Moreover, Com-2seq converged for all taxa in most replicates, whereas Com-2seq-exch failed to converge for approximately four taxa in each replicate. Notably, Com-2seq was also twice as fast as Com-2seq-exch.

3.3. Analysis of ORIGINS data

We analyzed data generated from the Oral Infections, Glucose Intolerance, and Insulin Resistance Study (ORIGINS) [40] to explore the association between periodontal bacteria and prediabetes status (yes/no) among diabetes-free adults. Each participant contributed one subgingival plaque sample, which was sequenced using either 16S, SMS, or both. Taxa count data with study ID 11808 were obtained from Qiita. The 16S data comprised 271 samples with associated metadata and adequate library sizes (> 5,000), resulting in 234 genera after quality control (QC) procedures (which excluded genera found in fewer than 5 samples). The SMS data included 183 samples with associated metadata and adequate library sizes and resulted in 756 genera after QC. In total, there were 302 distinct samples (56 cases and 246 controls) and 864 distinct genera, among which 152 samples (47 cases and 105 controls) had both 16S and SMS data for 125 genera. The mean library sizes of the 16S and SMS data were 26,950 and 176,321, respectively, resulting in a depth ratio of 1:6.5. In analyzing the association between the periodontal microbiome and prediabetes status, we did not adjust for any covariates, as none of the established metabolic risk factors (i.e., age, sex, race, education, smoking status, body mass index and baseline glucose levels) were significantly associated with the periodontal microbiome (Table 1 of [40]) and are therefore unlikely to act as strong confounders.

We began by analyzing data from the overlapping samples and genera, with the main purpose of comparing methods. We applied Com-2seq, Com-count, and Com-p for integrative analysis of the 16S and SMS data, along with LOCOM to analyze each dataset separately. The results of their global test $p$-values and the detected differentially abundant genera (at the nominal FDR level of 10%) are summarized in Table 1. Com-2seq and Com-count yielded significant global $p$-values (at the nominal level of 0.05) and detected 7 and 3 genera, respectively. In contrast, analyzing either dataset alone or performing an integrative analysis by combining $p$-values (Com-p) resulted in non-significant global $p$-values and detected zero or at most one genus.

Table 1:

Global test $p$-value and detected differentially abundant genera in the analysis of the real datasets

Dataset	Method	$J$	Global $p$-value	Detected genera
ORIGIN data for overlapping samples and genera	Com-2seq	98	0.0345	Pseudoalteromonas, Actinomyces Capnocytophaga, Lonepinella Campylobacter, Gemella, Kocuria
	Com-count	98	0.0226	Campylobacter,Gemella, Butyrivibrio
	Com-p	96	0.106	None
	16S	54	0.0689	Campylobacter
	SMS	90	0.170	None
Full ORIGIN dataset	Com-2seq	440	0.0340	Ignavigranum, Gemella, Butyrivibrio
	Com-p	440	0.245	None
	16S	85	0.0682	Chelonobacter
	SMS	403	0.0866	None
Dietary data for overlapping samples and genera	Com-2seq	195	0.0001	54
	Com-count	195	0.0006	27
	Com-p	191	0.00213	36
	16S	61	0.0001	24
	SMS	189	0.0002	23
Full Dietary dataset	Com-2seq	1062	0.0003	353
	Com-p	1062	0.00196	195
	16S	122	0.0001	57
	SMS	1003	0.0003	154

Note: $J$ represents the number of genera that passed the filter for rare taxa and were included in the analysis. Com-count is invalid for analyzing data from partially overlapping samples and was thus not applied to the analysis of the full dataset. The nominal FDR level is 10%.

More details on the detected genera are provided in Table 2 and Figure 6. Actinomyces, Campylobacter, and Capnocytophage were effectively captured by both sequencing methods and exhibited consistent prediabetes-related trends in both datasets (Figure 6, second and third columns). Their relative abundances across datasets showed strong concordance, with Pearson’s correlation coefficients exceeding 0.6 (Figure 6, first column). This concordance amplified the trend in the averaged relative abundances across datasets (Figure 6, last column), especially for Actinomyces and Capnocytophage, whose individual dataset $p$-values were not particularly small. Gemella, Kocuria, Pseudoalteromonas, and Lonepinella were present in too few samples in the 16S data and were therefore excluded from the 16S data analysis by the LOCOM filter. However, when their 16S data were integrated with their SMS data using Com-2seq, the 16S data reinforced the trend observed in the SMS data. Although Butyrivibrio exhibited the same prediabetes-related trend in both datasets, the concordance between the two datasets was only moderate (Pearson’s correlation coefficient of 0.47). As a result, the integrative analysis did not significantly improve power over the individual datasets and failed to meet the significance threshold for controlling the FDR at 10%. However, it would have been considered significant at an FDR level of 20%.

Table 2:

$P$-value and adjusted $p$-value for the detected genera in the analysis of the ORIGINS data for overlapping samples and genera

Method	Gemella	Actinomyces	Campylobacter	Kocuria	Pseudoalteromonas	Capnocytophaga	Lonepinella	Butyrivibrio
	$p$-value
Com-2seq	0.000947	0.00132	0.00137	0.00358	0.00363	0.00389	0.00553	0.0175
Com-count	0.000500	0.374	0.00090	0.0133	0.0501	0.248	0.0165	0.0005
Com-p	0.00491	0.201	0.00257	0.0647	0.183	0.347	0.0852	0.0300
16S	NA	0.131	0.00172	NA	NA	0.794	NA	0.0157
SMS	0.00491	0.368	0.00509	0.0647	0.183	0.127	0.0852	0.261
	Adjusted $p$-value
Com-2seq	0.0447	0.0447	0.0447	0.0636	0.0636	0.0636	0.0774	0.1900
Com-count	0.0245	0.7200	0.0294	0.2020	0.3510	0.6570	0.2020	0.0245
Com-p	0.236	0.682	0.236	0.415	0.652	0.818	0.481	0.288
16S	NA	0.634	0.093	NA	NA	0.967	NA	0.212
SMS	0.229	0.862	0.229	0.531	0.660	0.660	0.565	0.730

Note: “NA” means that the genus failed to pass the LOCOM filter.

Figure 6:

More details on the detected genera in the analysis of the ORIGINS data for overlapping samples and genera. In the first column, the scatter plots are the same as those in Figure 2. The second and third columns display observed relative abundances (RA) from 16S and SMS, respectively, including LOCOM $p$-values from analyzing the 16S and SMS data separately. The last column shows the averages of 16S and SMS relative abundances, along with the Com-2seq $p$-values.

We proceeded to analyze the full dataset for scientific discovery and summarized the main results in Table 1. In this case, Com-count is invalid (as demonstrated in simulation results) and was therefore not applied. Once again, Com-2seq yielded a significant global $p$-value, while the $p$-values of the other methods failed to reach the significance level. Com-2seq detected three genera, Butyrivibrio, Gemella, and Ignavigranum. In contrast, the analysis of the 16S data alone detected a different genus, Chelonobacter, while the analysis of the SMS data alone and the integrative analysis by Com-p failed to detect any genus. Recall that in the previous analysis of overlapping samples and genera, Butyrivibrio was nearly significant, Gemella was significant, and Ignavigranum was excluded due to its complete absence in the 16S data. Meanwhile, the genera that were significant in the previous analysis but not in this one still produced relatively small $p$-values (< 0.1) in the current analysis.

More details on the detected genera from the full dataset analysis are presented in Table 3 and Figure 7. Due to the presence of many non-overlapping samples, we are unable to display the paired data and averaged relative abundances as shown in Figure 6. For Butyrivibrio, data from non-overlapping samples corroborated the trend previously observed with overlapping samples, and Com-2seq deemed the overall signal highly significant. Indeed, Butyrivibrio is a key producer of butyrate, a beneficial short-chain fatty acid. Butyrate is the major end product of bacterial fermentation of dietary fiber in the large intestine and plays a crucial role in improving insulin sensitivity [41]. The negative association of Butyrivibrio with prediabetes status, as shown in Figure 7, aligns with its known beneficial effects. Gemella was highly significant in both analyses. This finding is also plausible, as Gemella has been linked to various infections, including those affecting heart valves [42], brain membranes [43], and bloodstreams [44]. The positive association of Gemella with prediabetes status, as shown in Figure 7, is consistent with its known adverse effects. Ignavigranum was completely missed by 16S sequencing, and its detection by Com-2seq was solely driven by its differential abundance revealed by SMS. Although little is known specifically about Ignavigranum in human health, other members of the Synergistetes phylum have been identified in the human oral cavity [45], gastrointestinal tract [46], and even in infections [47], suggesting possible roles in dysbiosis or disease under certain conditions. Chelonobacter exhibited significant differential abundance in the 16S data, but this signal was not replicated in the SMS data, resulting in a non-significant overall result by Com-2seq. Notably, Chelonobacter is a genus of bacteria primarily found in turtles rather than humans.

Table 3:

$P$-value and adjusted $p$-value for the detected genera in the analysis of the full ORIGINS dataset

Method	Butyrivibrio	Gemella	Ignavigranum	Chelonobacter
	$p$-value
Com-2seq	0.00042	0.00052	0.00014	0.189
Com-p	0.120	0.00058	0.00062	0.0015
16S	0.0711	NA	NA	0.00075
SMS	0.313	0.00058	0.00062	0.727
	Adjusted $p$-value
Com-2seq	0.0763	0.0763	0.0616	0.617
Com-p	0.499	0.136	0.136	0.191
16S	0.465	NA	NA	0.0637
SMS	0.741	0.125	0.125	0.945

Note: See Note in Table 2.

Figure 7:

Observed relative abundances (RA) by prediabetes status for the detected genera in the analysis of the full ORIGINS dataset. The observed relative abundances were calculated based on all genera across all samples in the 16S or SMS data. The $p$-values are from the analysis of individual 16S and SMS datasets, as also listed in Table 3.

3.4. Analysis of Dietary data

We analyzed the Dietary data from Amato et al. [48] which was also used to motivate our research problem and inform simulation parameters. Compared to the ORIGINS data, this dataset exhibits a higher degree of overdispersion and includes an important confounder that requires adjustment. Despite these differences, we reached similar conclusions regarding the methods as we did with the ORIGINS data. Therefore, we present these results concisely.

The goal of the Dietary study was to test the influence of host dietary niche (folivore vs. non-folivore) on the gut microbiome of wild non-human primates. One fecal sample was collected for each animal and sequenced by either 16S, SMS, or both. We downloaded the 16S and SMS taxa count data with study ID 11212 from Qiita, the statistics of which are listed in Table S2. In total, there are 172 distinct samples (94 folivore and 78 non-folivore) and 2062 distinct genera, among which 76 samples (40 folivore and 36 non-folivore) have both 16S and SMS data for 236 genera. The ratio of 16S to SMS mean library sizes is 1:9.8.

Host phylogeny, categorized into apes, lemurs, new world monkeys, and old world monkeys, is expected to be a significant confounder in the relationship between the host dietary niche and gut microbiome. Indeed, we found that host phylogeny is strongly associated with both the dietary niche (Chi-squared $p$-value < 10⁻⁶) and gut microbiome (Com-2seq global $p$-value < 10⁻⁶, with more than 50% of genera associated with host phylogeny). Therefore, we included host phylogeny as a covariate in our analysis to adjust for its effects.

Table 1 (lower panel) shows that all analysis methods yielded highly significant global $p$-values and detected a large number of differentially abundant genera at the nominal FDR level of 10%, both in the analysis of data for overlapping samples and genera and in the analysis of the full dataset. In both analyses, Com-2seq detected the most genera, far exceeding the number of detections by analyzing the 16S or SMS data alone or by using Com-p. Figure 8 displays the Venn diagram of the detected genera by different methods when analyzing the full dataset. Com-2seq detected 104 novel genera that were missed by analyzing the 16S and SMS data separately, whereas Com-p detected only 8 such genera.

Figure 8:

Venn diagram for the detected genera in the analysis of the full Dietary dataset.

4. Discussion

We have introduced a novel method, Com-2seq, which represents the first method for integrative analysis of 16S and SMS data. Com-2seq is specifically designed to combine these datasets in a way that properly accounts for their differences in experimental bias, sequenced samples, and library sizes. We have demonstrated that Com-2seq significantly enhances statistical efficiency over analysis of either of the two datasets and outperforms the taxa-count-pooling and $p$-value-combination approaches. Inference with Com-2seq relies on a permutation procedure that preserves sample structure, rendering it suitable for partially overlapping samples and small sample sizes. Moreover, Com-2seq allows for testing traits of various types, including binary, continuous, or multivariate traits (e.g., a categorical trait with more than two levels or a set of multiple measurements), and supports adjustment of confounding covariates.

Com-2seq is capably of utilizing all available data from both data resources, including data from both overlapping and non-overlapping samples. If no overlapping samples exist, Equation (1) becomes a sum of two LOCOM estimating equations based on independent samples sharing the parameter ${\beta }_j$. In such cases, Com-2seq reduces to a meta-analysis of independent studies, which in itself represents a novel approach.

As demonstrated in the analysis of the ORIGINS data, Com-2seq tends to favor taxa with high relative abundance correlations between the two datasets. This is expected, as measurements from taxa that are more highly correlated across datasets are inherently more reliable compared to those with low, near-zero, or negative correlations.

In our simulation studies, we adopted certain simplified settings. For example, we set the coefficients for all associated taxa to be the equal. In addition, we selected small numbers of taxa, specifically 20, 5, and 50 out of 856 under M1, M2, and M2, respectively, to be associated with the trait. In the LOCOM paper [30], we demonstrated that the results of LOCOM with heterogeneous coefficients (Figure S10 of [30]) followed similar patterns to those with homogeneous coefficients (Figure 2 of [30]. Furthermore, we showed that even when 500 out of 856 taxa were associated with the trait, LOCOM maintained robust performance (Figure 6 of [30]). Given the strong parallels between LOCOM and Com-2seq, we expect these conclusions to hold true for Com-2seq when relaxing the simplified settings to more complex scenarios.

As one reviewer noted, the approach of averaging relative abundances could be an interesting baseline for comparison. In fact, Com-2seq using equal weights $\left({\omega }_{ik,j}=1\right)$ follows the same principle as averaging relative abundances but offers superior performance. Unlike the naive averaging approach, which assumes a common underlying relative abundance for each pair of observed values and therefore no experimental bias, Com-2seq accounts for potential biases introduced by the two sequencing methods. Additionally, even if the average relative abundances are obtained, it remains unclear how they could be used for testing differential abundance, as current compositional methods typically rely on read count data. Com-2seq is the only method capable of handling relative abundance data for this purpose.

Since the development of LOCOM, there have been recent advancements in differential abundance methods. Among these, ANCOM-BC2 [49] and LinDA [50] have emerged as the most commonly used methods. We included both in our simulation to provide a more comprehensive assessment of Com-2seq’s performance compared to the analysis of individual datasets using various methods. As shown in Figure S16, ANCOM-BC2 exhibits a significant inflation of FDR, though it has low sensitivity. LinDA, while still prone to some FDR inflation, shows sensitivity comparable to that of LOCOM at best.

We have integrated Com-2seq into our R package LOCOM, accessible on GitHub at https://github.com/yijuanhu/LOCOM. Com-2seq performs best when given count data. However, some bioinformatics programs for processing shotgun metagenomic data, such as Kraken, output relative abundance data only. In this case, Com-2seq can still be used, although its performance may be slightly worse than if count data were available, as only equal weights are used. Our Com-2seq program automatically adjusts to handle only relative abundance data.