Multi-group Analysis of Compositions of Microbiomes with Covariate Adjustments and Repeated Measures

Abstract

Microbiome differential abundance analysis methods for a pair of groups are well established in the literature. However, many microbiome studies involve multiple groups, sometimes even ordered groups, such as stages of a disease, and require different types of comparisons. Standard pairwise comparisons are not only inefficient in terms of power and false discovery rates, but they may not address the scientific question of interest. In this paper, we propose a general framework for performing a wide range of multi-group analyses with covariate adjustments and repeated measures. We demonstrate the effectiveness of our methodology through two real data sets. The first example explores the effects of aridity on the soil microbiome, and the second example investigates the effects of surgical interventions on the microbiome of IBD patients.

Introduction

Increasingly researchers are recognizing the role of the microbiome in human health and diseases^{1, 2}. Accordingly, it has become a common practice for researchers to investigate differences in microbial compositions and to identify differentially abundant taxa among two or more study or experimental groups. Most existing methods for differential abundance (DA) analysis, such as ANCOM³, ANCOM-BC⁴, LOCOM⁵, etc., are designed for discovering differentially abundant taxa between two groups. A variety of strategies are used in the literature for performing multi-group differential abundance analysis. For example, some researchers use the above methods to analyze two groups at a time using a false discovery rate (FDR) threshold within each pairwise comparison and pool the results from all such pairwise comparisons to interpret the data. Such a strategy does not account for the fact that multiple tests and multiple pairwise comparisons are being performed when controlling for false discovery rates. Some researchers perform global beta diversity analyses for multiple groups and follow up with pairwise differential abundance analyses for groups that appear to be different. While all such approaches are intuitive and reasonable, they may not be designed to answer the specific scientific question of interest. Secondly, the statistical properties of such procedures are not well understood. For example, they may not control the overall FDR for multiple tests and multiple pairwise comparisons.

It is well recognized in gene expression studies that when there are more than two groups, the definitions of FDR depend upon the study design and the hypotheses of interest. Standard procedures, such as the BH procedure⁶, are designed for testing multiple hypotheses between two groups. When there are multiple groups, the standard concept of FDR, and methods controlling the corresponding error rates, need to be modified according to the study design and type of analyses to be performed^7–9. A variety of multi-group studies are performed by researchers, such as: (1) Multiple pairwise comparisons: A dietitian may be interested in making all pairwise comparisons of the gut microbial compositions among subjects receiving diets $D_1,D_2$ or $D_3$. Furthermore, for each pairwise comparison, $D_1$ vs. $D_2,D_1$ vs. $D_3$, and $D_2$ vs. $D_3$, the goal is often to identify taxa whose abundance increased (or decreased). (2) Multiple pairwise comparisons against a specific reference group: Same as in scenario (1), but the investigator is only interested in a subset of pairwise comparisons. For example, compare each group with a reference group. Suppose $D_1$ is the standard diet, and the researcher may be interested in identifying taxa whose abundance may have increased (or decreased) for subjects receiving a new diet $D_2$, compared to $D_1$, and similarly new diet $D_3$, compared to $D_1$. (3) Pattern analysis over ordered study groups: In some instances, an investigator may be interested in discovering trends or patterns in abundances of taxa over ordered groups, such as the health of subjects, changes in climate, doses of a drug, and others. For instance, during normal pregnancy, women not only undergo major physiological and hormonal changes, but they also experience significant changes in their gut and vaginal microbiome¹⁰. These changes are necessary for maternal metabolism, immune response, and hormonal changes to support pregnancy and to provide healthy flora for babies at birth^{11, 12}. In fact, the alpha diversity of the microbiome dramatically decreases temporally during pregnancy¹³. During the first trimester of a healthy pregnancy, the gut microbiota resembles that of a healthy non-pregnant woman¹³. However, as the pregnancy progresses from the first to the third trimester, there is a reduction in the relative abundance of anti-inflammatory butyrate-producing bacteria and an increase in the pro-inflammatory phylum Proteobacteria, Bifidobacterium spp. in the phylum Actinobacteria, and lactic acid-producing bacteria in preparation for energy demands of lactation¹². Thus, in many scientific investigations, researchers may be interested in studying changes in the microbiome over an ordered set of conditions. The patterns of microbial abundance may not always be monotonic. They may display other shapes, such as an umbrella or an inverted umbrella with the location of the peak or trough unknown a priori. Additionally, depending upon the scientific question of interest, such as microbial changes during pregnancy, repeated measures are taken on the same subject. Although the pattern analyses mentioned here could be accomplished by conducting a sequence of pairwise tests over adjacent ordered groups, such a strategy may have lower power than a test designed for pattern analysis, as will be demonstrated in the analysis of soil aridity data described later in this paper.

The focus of this paper is to develop methodologies for performing multi-group differential abundance analyses for studies such as the above-noted ones. To the best of our knowledge, there does not exist a formal methodology for performing such analyses, with the exception of ANCOM-II¹⁴. While ANCOM-II considered some of the above testing problems, it does not develop a formal framework for bias correction and is heuristic. The more recent methodology LinDA¹⁵, which uses a model similar to the one developed in ANCOM-II, does not address the above multi-group testing problems. Thus, there is a major gap in the literature for analyzing multi-group microbiome studies, which will be filled by the methodology developed in this paper called ANCOM-BC2.

Before developing ANCOM-BC2, we first make some modifications to ANCOM-BC⁴ for testing multiple hypotheses regarding two groups which would result in a better control of FDR. This framework will then be extended to the multi-group testing problem considered in this paper. Although the ANCOM-BC methodology accounted for sample-specific bias, we now also account for taxon-specific bias. This is important because sequencing efficiencies can vary across taxa, leading to a taxon-specific bias when some taxa are preferentially measured over others during the sequencing experimental workflow. For example, gram-positive bacteria have stronger cell walls than gram-negative bacteria, making them harder to extract during the data preparation step. As a result, gram-positive bacteria may be underrepresented in the observed abundances, leading to biased results if not properly accounted for in the analysis¹⁶. Also, it is well-known in the analysis of other omics data that small signals are often associated with small variances. Consequently, in such cases, the value of the test statistics is inflated which potentially results in a highly significant p-value, even though the effect sizes are trivially small. Inspired by Significance Analysis of Microarrays (SAM)¹⁷ methodology, we regularize the variance to avoid inflated values for the test statistics and hence moderate the p-values to avoid false positives and false discoveries. Lastly, zeros are a common problem for many DA methods, including ANCOM-BC, that work with log-abundance data. Often such methods use pseudo-counts to deal with zero before taking logarithms. However, the choice of pseudo-count can impact the results of rare taxa containing excess zeros, potentially leading to an increase in false discoveries^{14, 18, 19} and inflate FDR. To mitigate this issue, we conduct a sensitivity analysis for pseudo-count addition and assign a sensitivity score to each taxon, indicating the likelihood of a false positive result for a particular taxon that is declared significant. A larger sensitivity score indicates a higher risk of a false positive, thus providing a tool for a researcher when interpreting the results.

Using constrained statistical inference-based methods⁷ and mixed directional false discovery rate (mdFDR) methods for multiple pairwise comparisons^{8, 9}, along with the above-noted modifications to ANCOM-BC, we develop ANCOM-BC2 for multi-group microbiome studies in this paper. ANCOM-BC2 allows modeling covariates as well as repeated measures. Details of the method are described in the Methods Section. The performance of ANCOM-BC2 is evaluated using extensive simulation studies under a variety of settings. The results are described in the Results Section. The study designs for the simulation experiments are provided in the Supplementary Methods. ANCOM-BC2 is illustrated using two publicly available data, namely, soil microbiome data and irritable bowel disease data.

Results

Simulations: Continuous and Binary Exposures

In light of practical applications, we conducted simulation studies under two common scenarios, when the exposure variable is continuous or it is binary while adjusting for covariates in the model. To benchmark the performance of ANCOM-BC2, we compared it to its predecessor, ANCOM-BC⁴, which was originally developed for binary covariates, as well as two other state-of-the-art DA methods: 1) LinDA¹⁵, which is based on linear regression and employs centered log-ratio transformed abundance data, and 2) LOCOM⁵, a logistic regression-based approach that obviates the need for pseudo-counts and utilizes permutation methods to address overdispersion and small sample sizes.

We utilized a subset of data from the Quantitative Microbiome Project (QMP)²⁰ consisting of 106 samples and 91 operational taxonomic units (OTUs) to generate simulated data. By using real data as a template, we ensured that the data-generating process did not favor our methods, enabling a fair comparison across all methods. We applied the effects of the exposure and adjusting covariates to the log abundance data based on the QMP data template. The log fold-changes (based on the natural log) of these variables ranged from −2 to 2, which corresponds to a fold-change between 0.14 to 7.4 in the original scale. To evaluate the robustness of differential abundance (DA) methods, we conducted a comprehensive simulation study that incorporated two sources of bias into the synthetic data. To account for sequencing efficiency differences, we applied a feature-specific bias that was sampled from a uniform distribution $\left(U\right[0.1,1\left]\right)$ and applied to each taxon. Additionally, we included a sample-specific bias that ensured the presence of rare taxa (with more than 50% zeros across samples) to test the robustness of the DA methods to pseudo-count addition. The sample-specific bias was also highly correlated with the exposure of interest to assess the methods’ performance against batch effects, which are a concern in large-scale omics studies. To test the robustness of the DA methods regarding the violation of the assumption that most taxa are not differentially abundant in most DA methods, we considered four different cases of the underlying true proportion of DA taxa, ranging from 10% to 90%. For ANCOM-BC2, we used a sensitivity analysis to pseudo-count addition, setting the cutoff value at one, which means any taxon with a sensitivity score greater than one would be automatically declared not significant. To control the false discovery rate due to multiple testing, we utilized the Holm-Bonferroni method²¹ instead of the Benjamini-Hochberg (BH) procedure⁶, as it does not rely on assumptions about the dependence structure among the underlying p-values and is known to be more robust when dealing with inaccurate p-values²². Supplementary Figures 1 and 2 provide additional details on the simulation study design and the results obtained. More information is available in the Supplementary Methods.

Figure 1a displays the simulation results for DA analysis with respect to a continuous exposure. ANCOM-BC2 outperformed all other methods by maintaining a FDR control under the nominal level of 0.05, even with highly dynamic microbial environments, where up to 90% of taxa were differentially abundant. Also, as expected, ANCOM-BC2 demonstrated increased power with larger sample sizes, achieving comparable power to the other methods when the sample size was relatively large (> 50). In contrast, all competing methods had substantially higher FDR than ANCOM-BC2. LOCOM had severely inflated FDR and reduced power with limited sample sizes, reaching up to 50% FDR and as low as 20% power in simulations with 10 samples and 10% of taxa being differentially abundant. ANCOM-BC and LinDA had the highest power among all methods, but both methods exhibited FDR exceeding the nominal level in most simulation scenarios. This increase in FDR correlated with the increase in sample size, suggesting a systematic bias in their test statistics due to the addition of pseudo-counts. Proper sensitivity analysis regarding pseudo-count addition is necessary to prevent random false positives and FDR inflation for rare taxa in both methods.

Figure 1.

Comparisons of FDR (mdFDR) and power in identifying DA taxa in (a) continuous, (b) binary, or (c) categorical exposure. Synthetic data were generated based on the QMP data²⁰. The X-axis shows the sample size (or sample size per group for the categorical covariate), and the Y-axis shows the FDR (mdFDR) or power. The proportion of true DA taxa is indicated in the panel title. Results are represented by the average of the corresponding measure (FDR, mdFDR, or power) ± standard errors (shown as error bars) across 100 simulation runs for each setting. The results demonstrate that ANCOM-BC2 outperformed all competing methods in terms of uniformly small FDR (mdFDR) and comparable power.

Simulation results for differential abundance analysis (DA) with respect to a binary exposure are shown in Figure 1b. The results are consistent with those in Figure 1a. The competing methods had substantially inflated FDR as compared to ANCOM-BC2, and their FDRs increased monotonically with the increase in sample size. ANCOM-BC2 outperformed all other methods in terms of maintaining a uniformly small FDR and reasonable power. Note that given the striking differences in FDR, it may not be reasonable to compare powers.

Simulations: Multiple Groups

The simulation settings for multi-group comparisons were consistent with those outlined in the previous section. Similarly, we conducted a sensitivity analysis for ANCOM-BC2 with a cutoff value of one for pseudo-count addition and controlled the FDR due to multiple testing using the Holm-Bonferroni method²¹.

Multiple pairwise comparisons against a reference group.

In this simulation study, we evaluated the performance of ANCOM-BC2 in multiple comparisons against a reference group, where the exposure variable consisted of three groups and adjusting covariates were present. We compared ANCOM-BC2 against ANCOM-BC and LinDA. We did not include LOCOM because, currently, it is not designed for multiple groups. Figure 1c illustrates that ANCOM-BC2 effectively controlled the mdFDR^{8, 9}, the combination of both Type I and the directional errors in the FDR framework (see the Methods section for details), below the nominal level of 0.05 and maintained a power greater than 0.8 when the sample size per group exceeded 30. On the other hand, neither ANCOM-BC nor LinDA controlled the FDR, highlighting the advantage of ANCOM-BC2 over existing methods in controlling mdFDR in multi-group comparisons.

Multiple pairwise comparisons.

In this simulation study, we evaluated the performance of ANCOM-BC2 for all possible pairwise comparisons, rather than focusing on comparisons against a specific group. As previously mentioned, the competing methods considered in this paper are not equipped to handle pairwise comparisons in their existing forms, which led to their exclusion from this simulation study. Figure 2a demonstrates that, on average, ANCOM-BC2 effectively controlled the mdFDR well below the nominal level of 0.05 while maintaining substantial power $(>0.8)$ when the sample size per group was not exceedingly small $(\mathrm{n}>30)$.

Figure 2.

Comparisons of FDR (mdFDR) and power in identifying DA taxa in (a) multiple pairwise comparisons, and (b) pattern analysis. Synthetic data were generated based on the QMP data²⁰. The X-axis shows the sample size per group, and the Y-axis shows the FDR (mdFDR) or power. The proportion of true DA taxa is indicated in the panel title. The box plot shows the distribution of the corresponding measure (FDR or power) across 100 simulation runs for each setting. Each box represents the interquartile range (IQR) of the data, with the horizontal line in the box representing the median. The whiskers extend to the furthest data point which is within 1.5 times the IQR. Any data points beyond this range are shown as individual points. Additionally, the data points are jittered to avoid overlap and provide a better visualization of the data. The results demonstrate that ANCOM-BC2 controlled FDR (mdFDR) while maintaining adequate power.

Pattern analysis.

Similar to pairwise comparisons, pattern analysis stands out as an important feature of ANCOM-BC2. In this simulation study, we benchmarked the scenario of a monotonically increasing pattern, where the log fold-change between the second group and the reference group, denoted as $\delta$, ranged from 0.5 to 2, and the third group had the log fold-change equal to $\delta +1$ with respect to the reference group. The “discovery” in the pattern analysis was defined as the identification of a taxon that was monotonically increasing across the three groups. As shown in Figure 2b, ANCOM-BC2 was successful in controlling the false discovery rate (FDR) in this simulation setting and maintained high power $(>0.8)$ in most cases. However, under the most extreme scenario where 90% of taxa were truly differentially abundant, ANCOM-BC2 experienced a loss of power. It is important to note that in this simulation study, ANCOM-BC2 pattern analysis controlled the FDR in a very conservative manner, with an FDR equal to 0. Increasing the cutoff value for the sensitivity analysis of pseudo-count addition or using a less stringent FDR controlling method instead of the Holm-Bonferroni method could improve the power and the FDR/power tradeoff (data not shown).

Simulations: Correlated Samples

We also performed benchmarks comparing ANCOM-BC2 to LinDA with respect to correlated microbiome data. Note that ANCOM-BC and LOCOM were not included since neither of them is designed for correlated experimental groups, such as repeated measures. We considered two scenarios in this simulation study. The first scenario contained only a random intercept effect in the data, while the second had both random intercept and random slope effects. Both random effects had variances equal to 1, and a correlation coefficient of 0.5 was set if both were present. The exposure variable had three levels, i.e., there were three experimental groups. The simulation study included a continuous covariate. The remaining simulation settings were the same as indicated in the previous section, and we provide further details in the Supplementary Methods. The results were similar in both scenarios (Fig. 3). ANCOM-BC2 controlled the FDR well below the nominal level of 0.05 in all simulation settings while maintaining adequate power as the sample size increased. Meanwhile, although LinDA had uniformly larger power than ANCOM-BC2, it suffered inflated FDR, particularly when the sample size was large (e.g., the FDR could reach 70% when the sample size per group was 100).

Figure 3.

Comparisons of FDR and power in identifying DA taxa in (a) a random intercept model, and (b) a random coefficients model. Synthetic data were generated based on the QMP data²⁰. The X-axis shows the sample size per group, and the Y-axis shows the FDR or power. The proportion of true DA taxa is indicated in the panel title. Results are represented by the average of the corresponding measure (FDR or power) ± standard errors (shown as error bars) across 100 simulation runs for each setting. The results demonstrate that ANCOM-BC2 outperformed LinDA in terms of uniformly small FDR and comparable power.

Illustration: Soil Microbiome and Aridity

The threat of climate change not only threatens the dryland ecosystems with increasing temperatures and aridity but there is a concern that some of the microbes native to such conditions may be trans-locating to other environments with the expansion of deserts and desert ecologies. There is growing literature on desert microbiome²³. The translocation of such bacteria may affect vegetation, plant life, and human health. Most importantly, there are concerns regarding the spread of antimicrobial or antibacterial resistance-causing microbes, which may make infections harder to treat and increase the risk of spreading diseases. Accordingly, there is interest in identifying microbial species that are native to extremely hot conditions, such as hyper-arid soils.

Recently, Neilson et al.² investigated the differences in soil microbiomes according to soil aridity in the Atacama Desert in Chile. They classified soil into three ordered categories based on aridity, namely, arid, margin, and hyper-arid, and sequenced data from 63 sample pits from 18 sites in the desert. Since they did not conduct formal differential abundance analyses of those data, we reanalyzed those data using the ANCOM-BC2 methodology. To begin with, we conducted a trend analysis of species richness with respect to the ordered aridity categories (arid to hyper-arid) (Fig. 4a). The trend test using the ORIOGEN²⁴ software yielded a p-value $<0.0001$, suggesting a significant loss of species richness with the increase in aridity. This finding regarding the species richness is consistent with the original publication of Neilson et al.², except that we are able to provide a p-value for trend using ORIOGEN.

Figure 4.

Differential abundance analysis for the impact of aridity in desert soil samples. (a) Violin plot of the relationship between aridity and microbial richness. The black dot in the middle of each violin represents the median value, while the black bar represents the interquartile range (IQR). The width of each violin reflects the density of data points at each value of microbial richness. Individual data points are also shown as jittered dots. The ORIOGEN software was utilized to conduct a trend test, which revealed a significant reduction in species richness with increasing aridity with a p-value $<0.0001$. (b) Heatmap of ANCOM-BC2 pattern analysis with respect to aridity. Monotonic increasing and decreasing trends were evaluated across ordered soil categories, with arid soil as the reference category. The X-axis represents soil categories, while the Y-axis displays significant genera identified by ANCOM-BC2 pattern analysis. Each cell is color-coded with blue representing reduced abundance and red representing increased abundance, with the numbers on each cell indicating the log fold-change relative to the reference group (arid group). The Holm-Bonferroni method was utilized to correct for multiple testing, and genera colored in brown on the Y-axis were found to be significant after correcting for multiple testing.

Next, we performed a pattern analysis to discover patterns in microbial taxa abundances over the ordered soil categories using the arid soil as the reference category. We discovered Thermobaculum, Geodermatophilus, and Rubrobacter to increase in mean abundance with the aridity of the soil $(\mathrm{p}<0.05)$, and among these, Thermobaculum had a significant trend after correcting for multiple testing with a multiple testing adjusted p-value $<0.0001$ (Fig. 4b). Thermobaculum spp. is well-known to exist in temperatures as high as 90 degrees Celsius²⁵, and it is documented to contain antibacterial or antimicrobial-resistant genes^{26, 27}. Similarly, the two Actinobacteria genera, Geodermatophilus and Rubrobacter, are also known to be antibacterial-resistant genera. For a review of these taxa, one may refer to Montero-Calasanz et al.²⁸ and Li et al.²⁹, respectively. Thus, using our proposed methodology, we discover genera that have an increase in abundance with respect to aridity and which may have antibacterial resistance properties.

Not surprisingly, nitrogen-cycling microbes are significantly diminished by increasing aridity in desert soils. For example, the presence/absence results (Supplementary Table 1) show that Nitrobacter, a common contributor to nitrification, along with putative broadly distributed N2 fixers Sinorhizobium, Rhizobium, and Azospirillum, were not detected in hyper-arid samples. The results of the ANCOM-BC2 pattern analysis also show that increasing aridity correlates with significant reductions in the abundance of taxa typically associated with fertile soils (Fig. 4b). Genera such as Ca. Nitrososphaera, which are chemolithoautotrophs and have essential biogeochemical roles as nitrifying organisms³⁰; Paenibacillus, which contains many species that promote crop growth through nitrogen fixation, phosphate solubilization, production of the phytohormone indole-3-acetic acid (IAA), and release of siderophores that enable iron acquisition³¹; and Pseudonocardia, which has been reported to achieve associative nitrogen fixation and protect their hosts against soil-borne pathogenic infection³², monotonically decreased with aridity.

ANCOM-BC2 sensitivity analysis revealed that none of the significant taxa exceeded the threshold of 1, indicating that these discoveries are likely true positives (Supplementary Fig. 3).

Illustration: Gut Microbial Composition of IBD Patients after Surgery

Numerous studies have delved into the role of the microbiome in inflammatory bowel disease (IBD); however, only a few have specifically addressed the impact of surgery on the gut microbiome and its subsequent consequences. We employ the ANCOM-BC2 to analyze a longitudinal dataset obtained from Fang et al.³³ to investigate the changes in the gut microbiome following gastrointestinal surgery in IBD patients. The dataset utilized in this study consists of 322 stool samples collected from 125 patients. Of these, 46 patients were diagnosed with ulcerative colitis (UC) and 79 with Crohn’s disease (CD). Stool samples were obtained from each subject at approximately 6-month intervals, beginning at the baseline time point. Specifically, 21 patients provided one sample, 38 patients provided two samples, 41 patients provided three samples, 23 patients provided four samples, and 2 patients provided five samples. Of the total patient population, 87 (70.0%) had no history of intestinal surgery, while 22 CD patients had undergone ileocolonic resection, and 13 CD patients and 3 UC patients had undergone different types of colectomies. These surgeries had occurred prior to the collection of the baseline stool sample. For the purposes of this study, we focused on comparing the microbial compositions between patients who had not undergone gastrointestinal surgery, those who had undergone ileocolonic resection, and those who had undergone colectomies. We adjusted the ANCOM-BC2 model for IBD disease type (UC vs. CD) and two potential confounders, namely disease state (inactive vs. active) and antibiotic use (absent vs. present). The results are depicted in Fig. 5.

Figure 5.

Heatmap of ANCOM-BC2 pairwise analysis for the effect of surgical resection in a cohort of patients with IBD. The analysis included multiple pairwise comparisons among the three groups: ileocolonic resection, colectomy, and no intestinal surgery, while controlling the overall mdFDR at 0.05. The X-axis represents the specific comparisons made between ileocolonic resection vs. no intestinal surgery, colectomy vs. no intestinal surgery, and ileocolonic resection vs. colectomy. The Y-axis displays the significant species identified by ANCOM-BC2. Each cell in the heatmap is color-coded with blue representing reduced abundance and red representing increased abundance, and the numbers on each cell indicate the log fold-change. The Holm-Bonferroni method was utilized to correct for multiple testing.

We performed multiple pairwise comparisons among the three groups controlling the overall mdFDR at 0.05 using ANCOM-BC2. Ileocoloic section is the surgical removal of the diseased section of the ileum, which is the junction area between the small and last intestines. In contrast, colectomy is the surgical removal of most or all of the large intestine. Interestingly, our analysis revealed that almost no microbial species were differentially abundant between the two surgical groups of patients, except for H. parainfluenzae and C. perfringens, which are more abundant in the colectomy group, and conversely, C. aldense, and B. producta are more abundant only in the ileocolonic section group.

We saw reduced abundances of several native gut bacterial species in patients who underwent either ileocolonic resection or colectomy compared to patients with no intestinal surgery. Specifically, These species include Bacteroides spp. (ovatus and uniformis), Butyricicoccus pullicaecorum, Dorea (formicigenerans and longicatena), Faecalibacterium prausnitzii, Roseburia spp. (faecis and inulinivorans), and Ruminococcus torques. Most of these bacterial species are involved in the production of short-chain fatty acids (SCFAs), such as butyrate, propionate, and acetate^34–40. SCFAs play essential roles in maintaining gut health, supporting gut barrier function, possessing anti-inflammatory properties, and providing energy sources for colonocytes. Thus, although the surgical intervention was necessary for these patients, the surgery may have impacted the host immune response and health of these patients. The IBD patients who underwent any of these surgeries may require probiotics or other forms of microbial supplementation.

We also conducted an ANCOM-BC2 sensitivity analysis, revealing that none of the significant taxa exceeded a threshold of 1. This suggests that the observed findings are likely true positives (Supplementary Fig. 4).

Discussion

In this article, we introduced a general framework called ANCOM-BC2 for performing differential abundance analysis when the exposure variable is continuous, binary, or (ordered) categorical. The proposed methodology allows for adjusting for covariates and repeated measures (longitudinal measures) while controlling for FDR or mdFDR when the exposure variable has more than two groups, and the researcher is interested in inferring if the abundance of a taxon increased or decreased for each pairwise comparison. Furthermore, using the theory of constrained statistical inference, ANCOM-BC2 allows researchers to infer patterns in microbial abundance over ordered categories of exposure variables. For example, it allows a researcher to test if a particular microbe increased (or decreased) in abundance over ordered disease categories (very healthy to least healthy). This is a unique feature of ANCOM-BC2. Lastly, ANCOM-BC2 overcomes some of the limitations of ANCOM-BC for controlling the FDR while maintaining high power, especially with the presence of rare taxa with excess zeros.

The power of ANCOM-BC2’s pattern analysis was demonstrated in the soil microbiome data analyzed in this paper. When standard pairwise analyses were performed, we only discovered Paenibacillus that was significantly differentially abundant across different groups (data not shown). However, using the pattern analysis, we discovered several taxa display increasing or decreasing trends over the ordered soil aridity groups. This is because, unlike pairwise comparisons, pattern analysis uses constrained inference methods, which “borrow” information from neighboring ordered groups. Thus, increasing the effective sample size and the power^{7, 41, 42}.

Intrinsically, the ileocolonic section and colectomy are surgically removing different regions of the intestines, and yet based on our analysis of the IBD data, there were no significant differences in the abundance of the majority of the gut bacteria. Furthermore, the two groups of patients have similarly reduced abundances of these bacteria relative to those who did not undergo either of the two surgeries. Based on these findings, it may be reasonable to hypothesize that most species of gut microbiota are spatially uniformly distributed in the ileum and large intestines.

Methods

Notation

The notations described in ANCOM-BC2 methodology are summarized in Table 1.

Table 1.

Summary of notations

Notation	Description
$i$	Sample index, $i=1,2,\dots ,n$.
$j$	Tax on index, $j=1,2,\dots ,d$.
$k$	Index of fixed effects, $k=1,2,\dots ,p$.
$l$	Index of Random effects, $l=1,2,\dots ,q$.
$x_{ik}$	The $k^{th}$ fixed effect of interest for the $i^{th}$ sample.
$z_{il}$	The $l^{th}$ random effect of interest for the $i^{th}$ sample.
$A_{ij}$ ‡	Unobserved abundance of $j^{th}$ taxon in a unit volume of ecosystem of $i^{th}$ sample.
$O_{ij}$ ‡	Observed abundance of $j^{th}$ taxon in a random specimen taken from a unit volume of ecosystem of $i^{th}$ sample.
$E_{ij}$ ‡	Random error for taxon $j$ in sample $i$.
$S_i$ †	Sample-specific sampling fraction.
$C_j$ †	Taxon-specific sequencing efficiency.
$a_{ij}$ ‡	log $A_{ij}$.
$o_{ij}$ ‡	log $O_{ij}$.
$e_{ij}$ ‡	Random error for taxon $j$ in sample $i$ in log scale.
$s_i$ †	Sample-specific sampling fraction in log scale.
$c_j$ †	Taxon-specific sequencing efficiency in log scale.

^†Parameter;

^‡Random variable.

ANCOM-BC2 for fixed effects models

Model assumptions

Assumption 1 (Multiplicative model for observed abundances):

$$O_{ij}=S_i{C}_j{A}_{ij}{E}_{ij}.$$

Assumption 1 indicates that, in expectation, the observed abundance of a taxon in a random sample is in constant proportion to the abundance in a unit volume of the ecosystem of the sample. This proportion can be decomposed into two parts: (1) sample-specific sampling fraction, and (2) taxon-specific sequencing efficiency.

According to Assumption 1, for non-zero observed abundance, the above multiplicative model can be transformed into an additive model by log transformation

$$o_{ij}=s_i+c_j+a_{ij}+e_{ij}.$$

Assumption 2 (Linear model for log abundances): For each taxon $j,a_{ij},i=1,\dots ,n$ are independently distributed, and

$$a_{ij}=b_j^T{x}_i+e_{ij}^{(a)},$$

where

1. ${\mathbf{x}}_{\mathbf{i}}={\left(1,x_{i1},x_{i2},\dots ,x_{ip}\right)}^T$ are the covariates of interest (including the intercept) for the $i^{\mathit{th}}$ sample,

2. ${\mathbf{b}}_{\mathbf{j}}={\left(b_{j0},b_{j1},b_{j2},\dots ,b_{jp}\right)}^T$ are the corresponding coefficients for ${\mathbf{x}}_{\mathbf{i}}$.

3. $e_{ij}^{\left(a\right)},i=1,\dots ,n$ are independently distributed random errors for log abundances with $E(e_{ij}^{\left(a\right)})=0,\mathit{Var}(e_{ij}^{\left(a\right)})={\sigma }_{jj}^{\left(a\right)}$.

Assumption 3 (Independent random error for log observed abundances): Assume there are random errors, $e_{ij}^{\left(o\right)},i=1,\dots ,n,j=1,\dots ,d$, for log observed abundances $o_{ij}$, which are independently distributed with heteroskedasticity:

$$E(e_{ij}^{(o)})=0,\mathit{Var}(e_{ij}^{(o)})={\sigma }_{ij}^{(o)},e_{ij}^{(o)}⫫e_{ij}^{(a)}.$$

Regression framework

Based on the Assumptions 2 and 3, $o_{ij}$ can be modeled as:

$$o_{ij}=s_i+c_j+{b_j}^T{x}_i+e_{ij}^{(a)}+e_{ij}^{(o)}:=s_i+c_j+{b_j}^T{x}_i+e_{ij},$$ (1)

with

$$E(o_{ij})=s_i+c_j+{b_j}^T{x}_i,\mathit{Var}(o_{ij})=\mathit{Var}(e_{ij})={\sigma }_{jj}^{(a)}+{\sigma }_{ij}^{(o)}:={\sigma }_{ij}^{(t)}.$$

where ${\sigma }_{ij}^{\left(t\right)}$ denotes the total variance.

Model (1) can also be written in a vector notation as follows:

$$o_j=s+c_j\mathbf{1}+X{b}_j+e_j,$$ (2)

with

$$\begin{gathered} E(e_j)={\left(0,\dots ,0\right)}^T, \\ E(o_j)=s+c_j\mathbf{1}+X{b}_j, \\ \mathit{\text{Cov}}(o_j)=\left[\begin{array}{cccc}{\sigma }_{1j}^{\left(t\right)}& 0& \dots & 0\\ 0& {\sigma }_{2j}^{\left(t\right)}& \dots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & {\sigma }_{nj}^{\left(t\right)}\end{array}\right]. \end{gathered}$$

where

1. $\mathbf{1}=(\mathrm{1,1},\dots ,1{)}^T$,

2. ${\mathbf{o}}_{\mathbf{j}}={\left(o_{1j},o_{2j},\dots ,o_{nj}\right)}^T$,

3. $\mathbf{s}={\left(s_1,s_2,\dots ,s_n\right)}^T$,

4. ${\mathbf{b}}_{\mathbf{j}}={\left(b_{j0},b_{j1},b_{j2},\dots ,b_{jp}\right)}^T$,

5. ${\mathbf{e}}_{\mathbf{j}}={\left(e_{1j},e_{2j},\dots ,e_{nj}\right)}^T$,

6. $\mathbf{X}=\left[\begin{array}{ccccc}1& x_{11}& x_{12}& \dots & x_{1p}\\ 1& x_{21}& x_{22}& \dots & x_{2p}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 1& x_{n1}& x_{n2}& \dots & x_{np}\end{array}\right]$.

It is important to note that within each sample $i$, for taxa $l\ne m,o_{il}$ and $o_{im}$ are not necessarily independent due to correlations between $a_{il}$ and $a_{im}$. Thus vectors ${\mathbf{o}}_{\mathbf{l}}$ and ${\mathbf{o}}_{\mathbf{m}}$ are not independent random vectors.

Remove the effect of taxon-specific sequencing efficiency

To eliminate the effect of $c_j$, we first center the log observed abundances across samples, i.e.

$$\begin{gathered} y_{ij}:=o_{ij}-{\overline{o}}_{\cdot j}=\left(s_i-\overline{s}\right)+{b_j}^T\left(x_i-\overline{x}\right)+\left(e_{ij}-{\overline{e}}_{.j}\right), \\ :={\theta }_i+{\beta }_j{ }^T{x}_i+{\epsilon }_{ij}, \end{gathered}$$ (3)

where

1. ${\beta }_{jk}=b_{jk}$ for $k=1,\dots ,p$, and ${\beta }_{j0}={\sum }_{k=1}^p{b}_{jk}{\bar{x}}_{\cdot k}$,

2. $\mathit{Var}({\epsilon }_{ij})=\frac{(n-1{)}^2}{n^2}{\sigma }_{ij}^{\left(t\right)}+\frac{1}{n^2}{\sum }_{i^{\prime }\ne i}{\sigma }_{i^{\prime }j}^{\left(t\right)}:={\sigma }_{ij}$.

Algorithm 1.

Iterative MLE

Initialize:

For $j=1,\dots ,d$

$\theta \leftarrow \mathbf{0}$

$y_j^{(\mathbf{\text{crt}})}\leftarrow y_j-\theta =y_j$

${\beta }_j\leftarrow {\left(X^{T}X\right)}^{-1}{X}^T{y}_j^{(crt)}={\left(X^{T}X\right)}^{-1}{X}^T{y}_j$

while not converge do

$\theta \leftarrow \frac{1}{d}{\sum }_{j=1}^d\left(y_j-X{\beta }_j\right)$

$y_j^{(crt)}\leftarrow y_j-\theta$

${\beta }_j\leftarrow {\left(X^{T}X\right)}^{-1}{X}^T{y}_j^{(crt)}$

end while

Estimation of sample-specific bias

As can be seen from (3), ${\beta }_j$ are not identifiable without determining the nuisance parameter ${\theta }_i$. We define bias-corrected log abundance $y_{ij}^{\left(crt\right)}=y_{ij}-{\theta }_i$, then the ordinary least squares (OLS) estimators of ${\theta }_i$ and ${\beta }_j$ can be obtained by iteratively solving the following equations. For ease of exposition, the algorithm is described in the vector form, i.e. ${\mathbf{y}}_{\mathbf{j}}={(y_{1j},y_{2j},\dots ,y_{nj})}^T,\theta ={({\theta }_1,{\theta }_2,\dots ,{\theta }_n)}^T$, etc.

Upon convergence,

$${\theta }^{\ast }=\frac{1}{d}\sum _{j=1}^d(y_j-X{\beta }_j^{\ast }),y_j^{{(crt)}^{\ast }}=y_j-{\theta }^{\ast },{\beta }_j^{\ast }={\left(X^{T}X\right)}^{-1}{X}^T{y}_j^{{(crt)}^{\ast }}.$$ (4)

Therefore

$$\begin{gathered} {\theta }^{\ast }=\frac{1}{d}\sum _{j=1}^d(y_j-X{\beta }_j^{\ast })=\frac{1}{d}\sum _{j=1}^d(y_j-P{y}_{\text{j}}^{{(\mathbf{\text{crt}})}^{\ast }}) \\ =\frac{1}{d}\sum _{j=1}^d(y_j-P{y}_j+P{\theta }^{\ast })=\frac{1}{d}\sum _{j=1}^d\left[y_j^{(crt)}+\theta -P(y_j^{(crt)}+\theta )+P{\theta }^{\ast }\right] \\ =(I-P)\theta +P{\theta }^{\ast }+\frac{1}{d}\sum _{j=1}^d(I-P)y_j^{(crt)} \\ =(I-P)\theta +P{\theta }^{\ast }+\frac{1}{d}\sum _{j=1}^d{\epsilon }_j, \end{gathered}$$ (5)

where

1. $\mathbf{P}=\mathbf{X}{\left({\mathbf{X}}^T\mathbf{X}\right)}^{-1}{\mathbf{X}}^T$ is the projection matrix onto $𝒞\left(\mathbf{X}\right)$, the column space of $\mathbf{X}$,

2. ${\epsilon }_j=(\mathbf{I}-\mathbf{P}){\mathbf{y}}_{\mathbf{j}}^{\left(\mathbf{c}\mathbf{r}\mathbf{t}\right)}$ with $E({\epsilon }_j)=\mathbf{0}$.

Rearranging (5), we see that

$$(I-P){\theta }^{\ast }=(I-P)\theta +\frac{1}{d}\sum _{j=1}^d{\epsilon }_j.$$

Taking expectations on both sides leads to

$$(I-P)\left[E\left({\theta }^{\ast }\right)-\theta \right]=\mathbf{0}.$$

As $I-P$ is an orthogonal projector onto $𝒞\left(\mathbf{X}\right)$, the above equation holds as long as either of the following is valid:

1. $E({\theta }^{\ast })-\theta =\mathbf{0}$,

2. $E({\theta }^{\ast })-\theta \in 𝒞\left(\mathbf{X}\right)$.

It is sufficient to consider (2) because (1) is the trivial case. If (1) were true then from (4) we deduce that there is no sample-specific effect and that $E({{\beta }_j}^{\ast })={\beta }_j$. Suppose (2) is true, then there exists a vector $\delta \ne \mathbf{0}\in {ℝ}^p$, such that

$$E\left({\theta }^{\ast }\right)=\theta -X\delta .$$ (6)

Then by combining with equation (4), we have

$$E({{\beta }_j}^{\ast })=\delta +{\beta }_j.$$ (7)

We shall denote ${\theta }^{\ast }$ and ${{\beta }_j}^{\ast }$ obtained from the above iterative algorithm as preliminary estimators of $\theta$ and ${\beta }_j$, respectively. Without loss of generality, throughout this paper we assume ${\mathbf{X}}^T\mathbf{X}$ is a full rank matrix. If it is not a full rank matrix, then we may use any generalized inverse of ${\mathbf{X}}^T\mathbf{X}$ because $\mathbf{X}{{\beta }_j}^{\ast }$ in (5) is invariant of the choice of generalized inverse ${\left({\mathbf{X}}^T\mathbf{X}\right)}^g$ used in ${{\beta }_j}^{\ast }={\left({\mathbf{X}}^T\mathbf{X}\right)}^g{\mathbf{X}}^T{\mathbf{y}}_{\mathbf{j}}{ }^{\left(\mathbf{crt}\right)}$. Thus the preliminary estimator ${\theta }^{\ast }$ provided above is invariant of the choice of generalized inverse used in deriving ${{\beta }_j}^{\ast }$. Furthermore, throughout this paper we are interested in testing hypothesis regarding linearly estimable parameters $\mathbf{A}{\beta }_j$, i.e. $𝒞\left({\mathbf{A}}^T\right)\subset 𝒞\left({\mathbf{X}}^T\right)$⁴³. Consequently, the estimator $\mathbf{A}{{\beta }_j}^{\ast }$ is invariant of the generalized inverse used in the estimation of ${{\beta }_j}^{\ast }$. Hence throughout this text, for simplicity of exposition, we shall assume ${\mathbf{X}}^T\mathbf{X}$ is of full rank.

For each taxon $j=1,\dots ,d$, by (7), ${{\beta }_j}^{\ast }$ is a biased estimator if $\delta \ne \mathbf{0}$. Suppose we wish test the following hypothesis

$$\begin{gathered} H_0:A{\beta }_j=A{{\beta }_j}^0, \\ {H}_1:A{\beta }_j\ne A{{\beta }_j}^0. \end{gathered}$$

Under the null hypothesis, $E(\mathbf{A}{{\beta }_j}^{\ast })-\mathbf{A}{{\beta }_j}^0=\mathbf{A}\delta \ne \mathbf{0}$ and hence biased. The next step is to estimate this bias $\delta$ and accordingly modify the estimator $\mathbf{A}{{\beta }_j}^{\ast }$ so that the resulting estimator is asymptotically centered at $\mathbf{A}{{\beta }_j}^0$ under the null hypothesis and hence the test statistic is asymptotically centered at zero.

First we make the following observations. Since $E({{\beta }_j}^{\ast })=\delta +{\beta }_j$, we note that as $n\to \mathrm{\infty }$, for finite dimension $d$,

$${{\sum }_j}^{-\frac{1}{2}}({{\beta }_j}^{\ast }-\left(\delta +{\beta }_j\right)){\to }_d{N}_p(\mathbf{0},I),$$ (8)

where

$${\sum }_j=\underset{n\to \infty }{\text{lim}}{(X^{T}X)}^{-1}(\sum _{i=1}^n{\sigma }_{ij}^2{x}_i{x_i}^T){(X^{T}X)}^{-1}.$$ (9)

Since

$$E({\theta }^{\ast }+X{{\beta }_j}^{\ast })=\theta -X\delta +X(\delta +{\beta }_j)=\theta +X{\beta }_j,$$

i.e. ${\theta }^{\ast }+\mathbf{X}{{\beta }_j}^{\ast }$ is an unbiased estimator of $\theta +\mathbf{X}{\beta }_j$, hence a possible estimator of ${\text{Σ}}_j$ is given by

$${\widehat{\text{Σ}}}_j={(X^{T}X)}^{-1}(\sum _{i=1}^n{(y_{ij}-{\theta }_i^{\ast }-{{\beta }_j}^{\ast T}{x}_i)}^2{x}_i{x_i}^T){(X^{T}X)}^{-1}.$$ (10)

Under some mild regularity conditions⁴⁴, with finite $d$, we have the following consistency result

$$n({\widehat{\text{Σ}}}_j-{\text{Σ}}_j){\to }_p\mathbf{0},\text{as}n\to \infty .$$ (11)

Therefore, replacing ${\mathrm{\Sigma }}_j$ with ${\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}}_j$ in (8) and appealing to Slutsky’s theorem, we have

$${\widehat{\text{Σ}}}_j^{-\frac{1}{2}}({\beta }_j^{\ast }-(\delta +{\beta }_j)){\to }_d{N}_p(\mathbf{0},I),\text{as}n\to \infty .$$

By (9) and (11), under some mild regularity conditions⁴⁴, for finite $d$, we obtain

$${\widehat{\text{Σ}}}_j{\to }_p\mathbf{0},\text{as}n\to \infty .$$

Consequently,

$${{\beta }_j}^{\ast }{\to }_p\delta +{\beta }_j,\text{as}n\to \infty .$$ (12)

The above observation regarding the convergence of ${{\beta }_j}^{\ast }$ plays a critical role in the following. Since the sampling fraction is constant for all taxa within a sample, we pool information across taxa within each sample when estimating $\delta$. We model each taxon abundance using the following Gaussian mixture model. For the $j^{\mathit{\text{th}}}$ taxon and the $k^{\mathit{\text{th}}}$ covariate, let $C_0$ denote the set of taxa that are not differentially abundant with respect to $x_{ik}$, i.e. $C_0=\{j\in (1,2,\dots ,d):{\beta }_{jk}=0\},C_1$ denote the set of taxa whose abundance decreases with $x_{ik}$, i.e. $C_1=\{j\in (1,2,\dots ,d):{\beta }_{jk}<0\}$, and let $C_2$ denote the set of taxa whose abundance increases with $x_{ik}$, i.e. $C_2=\{j\in (1,2,\dots ,d):{\beta }_{jk}>0\}$. Let ${\pi }_r$ denote the probability that a taxon belongs to set $C_r,r=0,1,2$. For simplicity of estimation of parameters, similar to GEE, we shall assume that ${\beta }_{jk}^{\ast },j=1,2,\dots ,d$, are independently distributed. As commonly done in the analyses of various ’omics data, we ignore the underlying correlation structure when estimating $\delta$. Thus, we model the distribution of ${\beta }_{jk}^{\ast }$ by Gaussian mixture model as follows:

$$f({\beta }_{jk}^{\ast })={\pi }_0\varphi \left(\frac{{\beta }_{jk}^{\ast }-{\delta }_k}{v_{j0}}\right)+{\pi }_1\varphi \left(\frac{{\beta }_{jk}^{\ast }-\left({\delta }_k+l_1\right)}{v_{j1}}\right)+{\pi }_2\varphi \left(\frac{{\beta }_{jk}^{\ast }-\left({\delta }_k+l_2\right)}{v_{j2}}\right),$$ (13)

where

1. $\varphi$ is the standard normal density function,

2. ${\delta }_k,{\delta }_k+l_1$, and ${\delta }_k+l_2$ are means for ${\beta }_{jk}^{\ast }|C_0,{\beta }_{jk}^{\ast }|C_1$, and ${\beta }_{jk}^{\ast }\mid C_2$, respectively. $l_1<0,l_2>0$,

3. $v_{j0},v_{j1}$, and $v_{j2}$ are variances of ${\beta }_{jk}^{\ast }|C_0,{\beta }_{jk}^{\ast }|C_1$, and ${\beta }_{jk}^{\ast }\mid C_2$, respectively.

Note that instead of fitting a multivariate Gaussian mixture model for all covariates together, we choose to fit a univariate Gaussian mixture model repeatedly for every single covariate. This repetition is simply because the sets of taxa $\left\{C_0,C_1,C_2\right\}$ are not necessarily the same for different covariates. Also, note that for a categorical covariate of $s$ levels, it contains $s$ coefficients, e.g. ${\beta }_{j1},\dots ,{\beta }_{js}$, and we shall fit the Gaussian mixture model for these $s$ coefficients separately.

For computational simplicity, we assume that $v_{j1}>v_{j0},v_{j2}>v_{j0}$. Thus, Without loss of generality for ${\kappa }_1,{\kappa }_2>0$, let $v_{j1}=v_{j0}+{\kappa }_1$ and $v_{j2}=v_{j0}+{\kappa }_2$. While this assumption is not a requirement for our method, it is reasonable to assume that variability among differentially abundant taxa is larger than that among the null taxa. By making this assumption, we simplify the computation.

Assuming samples are independent, we begin by first estimating $v_{j0}^2=\mathit{Var}({\beta }_{jk}^{\ast })$. Note that $v_{j0}^2$ is the function of heteroscedastic variances, a consistent estimator of $v_{j0}^2$, which we refer to as ${\hat{v}}_{j0}^2$, is the $k^{\mathit{\text{th}}}$ diagonal element of ${\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}}_j$ stated in (10). In all future calculations, we plug in ${\hat{v}}_{j0}^2$ for $v_{j0}^2$. This is similar in spirit to many statistical procedures involving nuisance parameters. The following lemma is useful in the sequel.

Lemma 1 (Introducing the latent variable in calculating log-likelihood⁴⁵):

$$\text{log}f(x\mid \theta )=E_{f(z\mid x,\theta )}[\text{log}f(z\mid \theta )+\text{log}f(x\mid z,\theta )].$$

Let $\mathrm{\Theta }={\left({\delta }_k,{\pi }_1,{\pi }_2,{\pi }_3,l_1,l_2,{\kappa }_1,{\kappa }_2\right)}^T$ denote the set of unknown parameters, then for each taxon the log-likelihood can be reformulated using Lemma 1, as follows:

$$\text{Θ}\leftarrow \underset{\text{Θ}}{\text{arg max}}\sum _{j=1}^d\sum _{r=0}^2{p}_{r,j}\left[\text{log}\mathit{Pr}\left(j\in C_r\right)+\text{log}f\left({\beta }_{jk}\mid j\in C_r\right)\right].$$ (14)

Then the E-M algorithm is described as follows:

• E-step: Compute conditional probabilities of latent variables. Define $p_{r,j}=\mathit{Pr}(j\in C_r\mid {\beta }_{jk},\mathrm{\Theta })=\frac{{\pi }_r\varphi (\frac{{\beta }_{jk}-\left({\delta }_k+l_r\right)}{v_j})}{{\sum }_r{\pi }_r\varphi (\frac{{\beta }_{jk}-\left({\delta }_k+l_r\right)}{v_{jr}})},r=0,1,2;j=1,\dots ,d$, which are conditional probabilities representing the probability that an observed value follows each distribution. Note that $l_0=0$.

• M-step: Maximize the likelihood function with respect to the parameters, given the conditional probabilities.

We shall denote the resulting estimator of ${\delta }_k$ upon convergence of the algorithm by ${\hat{\delta }}_k^{EM}$.

As stated in Lin and Peddada⁴, compared to ${\hat{v}}_{j0}^2$, the variance and covariance contributed by ${\hat{\delta }}_k^{EM}$ is negligible when the number of non-differentially abundant taxa is large, such as when analyzing the microbiome data at the OTU/ASV or species level of the phylogenetic tree.

The above procedure is applied to every ${\beta }_{jk},k=1,\dots ,p$, eventually, we obtain the estimator of $\delta$ as

$${\hat{\delta }}^{EM}={({\hat{\delta }}_1^{EM},{\hat{\delta }}_2^{EM},\dots ,{\hat{\delta }}_p^{EM})}^T.$$ (15)

Therefore, the final estimator of ${\beta }_j$ is defined as

$${\hat{\beta }}_j={{\beta }_j}^{\ast }-{\hat{\delta }}^{EM},$$ (16)

with

$${\hat{\beta }}_j{\to }_p{\beta }_j,\text{as}n\to \infty ,$$ (17)

given that ${\hat{\delta }}^{EM}$ is a good approximation of $\delta$.

The estimation procedure is summarized in Algorithm 2.

Algorithm 2.

E-M algorithm

1:	Input:
	${{\beta }_j}^{\ast },{\sum }_j,j=1,\dots ,d$
2:	procedure $E-M({{\beta }_j}^{\ast },{\sum }_j)$
3:	return ${\widehat{\delta }}_k^{EM},k=1,\dots ,p$
4:	end procedure
5:	for $k=1,\dots ,p$ do
6:	${\widehat{\beta }}_{jk}\leftarrow {\beta }_{jk}^{\ast }-{\widehat{\delta }}_k^{EM}$
7:	end for

For taxon $j$, we now describe our methodology for testing the following hypotheses

$$\begin{gathered} H_0:A{\beta }_j=A{{\beta }_j}^0, \\ {H}_1:A{\beta }_j\ne A{{\beta }_j}^0. \end{gathered}$$

From Slutsky’s theorem, as $n\to \mathrm{\infty }$, the following test statistic is approximately central chi-square distributed under the null hypothesis

$$\begin{gathered} W_j={(A{\hat{\beta }}_j-A{{\beta }_j}^0)}^T{(A{\widehat{\text{Σ}}}_j{A}^T)}^{-1}(A{\hat{\beta }}_j-A{{\beta }_j}^0) \\ ={(A{{\beta }_j}^{\ast }-A{\hat{\delta }}^{EM}-A{{\beta }_j}^0)}^T{(A{\widehat{\text{Σ}}}_j{A}^T)}^{-1}(A{{\beta }_j}^{\ast }-A{\hat{\delta }}^{EM}-A{{\beta }_j}^0) \\ {\to }_d{\chi }_q^2, \end{gathered}$$

where $q=\mathit{rank}\left(A\right)$.

To control the FDR due to multiple testing, we recommend applying Holm-Bonferroni method²¹ instead of Benjamini-Hochberg (BH) procedure⁶ because the Holm-Bonferroni method does not require any assumptions regarding the dependence structure in the underlying p-values, and is also known to be a better method to control FDR when p-values are not accurate²².

Sampling-specific biases estimation

After obtaining ${\hat{\delta }}^{EM}$, the estimator of sample-specific biases $\theta$ is defined as follows:

$$\hat{\theta }=\frac{1}{d}\sum _{j=1}^d(y_j-X{\hat{\beta }}_j).$$ (18)

Let ${\mathrm{\Sigma }}^{\left(i\right)}={[{\sigma }_{lm}^{\left(i\right)}]}_{l,m=1,\dots ,d}$ denote the $d\times d$ covariance matrix of ${\epsilon }^{\left(i\right)}={({\epsilon }_1^{\left(i\right)},{\epsilon }_2^{\left(i\right)},\dots ,{\epsilon }_d^{\left(i\right)})}^T$, where ${\sigma }_{lm}^{\left(i\right)}$ is the $(l,m{)}^{th}$ element of ${\mathrm{\Sigma }}^{\left(i\right)}$ and ${\sigma }_{jj}^{\left(i\right)}$ is the $j^{th}$ diagonal element of ${\mathrm{\Sigma }}^{\left(i\right)}$. Furthermore, suppose

Assumption 4 (Sparse correlations among taxa):

$$\begin{gathered} {\sigma }_{jj}^{(i)}<K<\infty , \\ \frac{{\sum }_{l\ne m}^d{\sigma }_{lm}^{(i)}}{d^2}=o(1). \end{gathered}$$

From Assumption 4, we have

$$0\le {\mathbf{1}}^T{\text{Σ}}^{(i)}\mathbf{1}=\sum _{l=1}^d\sum _{m=1}^d{\sigma }_{lm}^{(i)}=\sum _{j=1}^d{\sigma }_{jj}^{(i)}+\sum _{l\ne m}^d{\sigma }_{lm}^{(i)}\le dK+\sum _{l\ne m}^d{\sigma }_{lm}^{(i)}.$$

Hence

$$0\le \frac{{\mathbf{1}}^T{\text{Σ}}^{(i)}\mathbf{1}}{d^2}\le \frac{K}{d}+\frac{{\sum }_{l\ne m}^d{\sigma }_{lm}^{(i)}}{d^2}=o(1).$$

Thus, for each taxon $j=1,2,\dots ,d$, we have

$$\frac{1}{d}\sum _{j=1}^d(y_j-(\theta +X{\beta }_j)){\to }_{p}0,\text{as}d\to \infty .$$ (19)

Therefore, according to (17) and (19), as both $n,d\to \mathrm{\infty }$,

$$\hat{\theta }\to \theta .$$ (20)

Remark 1. Regularization of variance: To avoid the spurious detection of significance due to extremely small standard errors, particularly for rare taxa, we incorporated a small positive constant in the denominator of the ANCOM-BC2 test statistic for each taxon. This approach was inspired by the Significance Analysis of Microarrays (SAM) methodology¹⁷. Specifically, the regularization factor was set as the 5th percentile of the distribution of standard errors for each fixed effect, unless otherwise specified.

Remark 2. Sensitivity analysis for the pseudo-count addition: To mitigate the risk of inflated false positive rates resulting from the choice of pseudo-count in ANCOM-BC2, we conducted a sensitivity analysis to assess the impact of varying pseudo-count values on differential abundance results. This is particularly important, as several studies have shown that the choice of pseudo-count can significantly influence the results of differential abundance analysis methods^{18, 19}. Specifically, we surveyed a range of pseudo-count values from 0 to 1 with 0.01 increments, and a sensitivity score was assigned to each taxon to evaluate its sensitivity to the choice of pseudo-count. The sensitivity score was defined as the standard deviation of the resulting negative log p-values, with a larger score indicating greater sensitivity to the pseudo-count choice. Consequently, a significant result for a taxon with a high sensitivity score is likely to be a false positive.

Multi-group comparison

In some applications, for a given taxon, researchers are interested in drawing inferences regarding differential abundance among different pairs of experimental groups. We refer to this kind of problem as a multi-group comparison problem, and extra caution needs to be exercised to correct p-values due to multiple comparisons. For simplicity, we drop the subscript $j$ (taxon index) in the following discussions.

Global test

For a given taxon and a total of $g+1$ experimental groups (including the reference group), researchers may want to test whether there exists at least one group that is significantly different from others. For ease of exposition, we split the covariates $\mathbf{X}$ into two parts, where ${\mathbf{X}}_1$ stands for the group assignment, and ${\mathbf{X}}_2$ denotes the remaining covariates. Note that the difference of group effects against the reference group is estimable, while the individual group effect is not. For simplicity, in the discussions of multi-group comparisons among group 0 to group $g$, we assume group 0 is the reference group. We use ${\beta }_k,k=1,\dots ,g$ to denote the group effect, but notice that it actually estimates ${\beta }_k-{\beta }_0$. We rewrite the model stated in (3) as

$$y=\theta +X_1\beta +X_2\gamma +\epsilon ,$$ (21)

where

1. $\theta$ is the sample-specific bias,

2. $\beta$ is the vector of group effects (as compared to group 0) of the order $g\times 1$,

3. ${\mathbf{X}}_1$ is the design matrix of the order $n\times g$ consisting of 0s and 1s,

4. ${\mathbf{X}}_2$ is the known matrix of other covariates (including the intercept) of the order $n\times (p-g+1)$ with the corresponding regression parameter vector $\gamma$ of the order $(p-g+1)\times 1$.

The global test intends to test

$$\begin{gathered} H_0:{\cap }_{k\in \{1,\dots ,g\}}{\beta }_k=0, \\ {H}_1:{\cup }_{k\in \{1,\dots ,g\}}{\beta }_k\ne 0, \end{gathered}$$

which can be reformulated as

$$\begin{gathered} H_0:A\beta =\mathbf{0}, \\ {H}_1:A\beta \ne \mathbf{0}, \end{gathered}$$

where

$$A=I_g=\left[\begin{array}{ccccc}1& 0& 0& \dots & 0\\ 0& 1& 0& \dots & 0\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& \dots & 0& 1\end{array}\right]$$

with the test statistic

$$W={(A\hat{\beta })}^T{(A{\widehat{\text{Σ}}}^{(g)}{A}^T)}^{-1}(A\hat{\beta }){\to }_d{\chi }_g^2,\text{as}n\to \infty ,$$

where ${\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}}^{\left(g\right)}$ is the corresponding sub-matrix of $\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}$ defined in equation (10).

Similarly, to control the FDR due to multiple testing, we recommend applying Holm-Bonferroni method²¹ instead of Benjamini-Hochberg (BH) procedure⁶ due to the underlying complex dependence structure between taxa.

Example 0.1. Suppose there are 3 groups, namely, groups 0 (reference), 1, and 2, and no other covariates. For each sample $i,i=1,\dots ,n$, we have:

$$y_i={\theta }_i+\mu +{\beta }_{1}I\{\mathit{\text{group}}=1\}+{\beta }_{2}I\{\mathit{\text{group}}=2\}+{\epsilon }_i.$$

To test whether there is at least one group among 0, 1, and 2, that is significantly different from others, we would like to test:

$$\begin{gathered} H_0:{\beta }_1={\beta }_2=0, \\ {H}_1:{\beta }_1\ne 0\cup {\beta }_2\ne 0, \end{gathered}$$

which is the same as testing:

$$\begin{gathered} H_0:A\beta =0, \\ {H}_1:A\beta \ne \mathbf{0}, \end{gathered}$$

where $\mathbf{A}=\left[\begin{array}{ll}1& 0\\ 0& 1\end{array}\right]$, and $\beta ={\left({\beta }_1,{\beta }_2\right)}^T$.

Multiple pairwise comparisons

If we are interested in knowing whether the abundance increased or decreased between various pairs of groups, then it amounts to testing the following hypotheses:

$$\begin{gathered} H_{0,k,k^{\prime }}:{\beta }_k={\beta }_{k^{\prime }} \\ {H}_{1,k,k^{\prime }}:\{{\beta }_k<{\beta }_{k^{\prime }}\}{\cup }^{\text{}}\{{\beta }_k>{\beta }_{k^{\prime }}\}, \end{gathered}$$

where $k\ne k^{\prime }\in \{1,\dots ,g\}$. Denote the test statistic for a given pairwise comparison as

$$W_{k{k}^{\prime }}=\frac{{\hat{\beta }}_k-{\hat{\beta }}_{k^{\prime }}}{\sqrt{\widehat{\mathit{\text{Var}}}({\hat{\beta }}_k)+\widehat{\mathit{\text{Var}}}({\hat{\beta }}_{k^{\prime }})}}{\to }_{d}N(0,1),\text{as}n\to \infty ,$$

where $\widehat{\mathit{Var}}({\hat{\beta }}_k),\widehat{\mathit{Var}}({\hat{\beta }}_{k^{\prime }})$ are the $k^{\mathit{\text{th}}}$ and $k^{\prime \mathit{th}}$ diagonal elements of ${\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}}^{\left(g\right)}$, respectively. Thus, the raw p-value for comparing group $k$ and group $k^{\prime }$ is defined as:

$$p_{k{k}^{\prime }}=2\left[1-\varphi \left(\left|W_{k{k}^{\prime }}\right|\right)\right].$$

For comparing with the reference group (group 0), the hypotheses become:

$$\begin{gathered} H_{0,k}:{\beta }_k=0 \\ {H}_{1,k}:\left\{{\beta }_k<0\right\}\cup \left\{{\beta }_k>0\right\}. \end{gathered}$$

We also replace ${\hat{\beta }}_{k^{\prime }}$ and $\widehat{\mathit{Var}}({\hat{\beta }}_{k^{\prime }})$ with 0s in the test statistic.

Note that the null and alternative hypotheses for the global test are denoted as $H_0$ and $H_1$, a Type I error might occur due to wrongly rejecting $H_0$ or correctly rejecting $H_0$ but wrongly rejecting $H_{0,k,k^{\prime }}$. A directional error might occur due to correctly rejecting $H_0$ but wrong assignment of the direction between ${\beta }_k$ and ${\beta }_{k^{\prime }}$ while correctly rejecting $H_{0,k,k^{\prime }}$. In this case, we need to control the error rate combining both Type I and the directional errors in the FDR framework, which is referred to as mixed directional FDR (mdFDR)^{8, 9}.

Definition 1 (mdFDR): Let $V\left(j\right)$ denote the indicator function of at least one Type I error or directional error committed, i.e.

$$V(j)=\{\begin{array}{ll}1\hfill & \text{if Type I or directional error occurs}\hfill \\ 0\hfill & \text{otherwise}.\hfill \end{array}$$

Then, mdFDR is defined as the expected proportion of Type I and directional errors among all discovered taxa.

$$mdFDR=E\left(\frac{{\sum }_{j=1}^{d}V(j)}{\text{max}(R,1)}\right),$$

where $R$ denote the number of taxa discovered.

To control the mdFDR for all pairwise tests, we adopt the general mdFDR controlling procedure⁹, and do the following:

1. Apply global test method stated above to obtain the p-value for each taxon. We denote these p-values as screening p-values. Apply BH procedure to identify taxa that are differentially abundant in at least one pairwise comparison. Let $R$ denote the number of taxa discovered.

2. For each taxon discovered in step 1, apply any mixed directional family wise error (mdFWER) controlling procedure, such as Holm-Bonferroni (default), Hochberg, etc., to the pairwise p-values $(p_{k{k}^{\prime }})$ at level $R\alpha /d$.

3. For a given taxon discovered in step 1, if a pairwise hypothesis is rejected in step 2, then we declare ${\beta }_k<{\beta }_{k^{\prime }}$ or ${\beta }_k>{\beta }_{k^{\prime }}$ according to $W_{k{k}^{\prime }}<0\mathrm{or}>0$.

It has been proved that under the assumption of independence of p-values obtained from the global test, the mdFDR of the above procedure is strongly controlled at level $\alpha$⁹.

Example 0.2. Suppose there are 3 groups, namely, groups 0 (reference), 1, and 2, and no other covariates. For each sample $i,i=1,\dots ,n$, we have:

$$y_i={\theta }_i+\mu +{\beta }_{1}I\{\mathit{group}=1\}+{\beta }_{2}I\{\mathit{group}=2\}+{\epsilon }_i.$$

To test whether the taxon is differentially abundant between group 1 and 0 (reference), we would like to test:

$$\begin{gathered} H_0:{\beta }_1=0, \\ {H}_1:\left\{{\beta }_1<0\right\}\cup \left\{{\beta }_1>0\right\}, \end{gathered}$$

with the test statistic:

$$W_{10}=\frac{{\hat{\beta }}_1}{\sqrt{\widehat{\mathit{\text{Var}}}({\hat{\beta }}_1)}}.$$

Additionally, if we want to test whether the taxon is differentially abundant between group 1 and 2:

$$\begin{gathered} H_0:{\beta }_1={\beta }_2, \\ {H}_1:\left\{{\beta }_1<{\beta }_2\right\}\cup \left\{{\beta }_1>{\beta }_2\right\}. \end{gathered}$$

The test statistic is:

$$W_{12}=\frac{{\hat{\beta }}_1-{\hat{\beta }}_2}{\sqrt{\widehat{\mathit{\text{Var}}}({\hat{\beta }}_1)+\widehat{\mathit{\text{Var}}}({\hat{\beta }}_2)}}.$$

Test against a specific group

Often researchers are interested in knowing whether the abundance increased or decreased in an ecosystem relative a pre-specified group, say the control group. Again, assume group 0 is the reference group and ${\beta }_0=0$, then one may be interested in testing the following hypotheses:

$$\begin{gathered} H_{0,k}:{\beta }_k=0, \\ {H}_{1,k}:\left\{{\beta }_k<0\right\}{\cup }^{\text{}}\left\{{\beta }_k>0\right\}, \end{gathered}$$

where $k\in \{1,\dots ,g\}$.

As before, the pairwise test statistic is defined as follows:

$$W_k=\frac{{\hat{\beta }}_k}{\sqrt{\widehat{\mathit{\text{Var}}}({\hat{\beta }}_k)}}{\to }_{d}N(0,1),\text{as}n\to \infty ,$$

where $\widehat{\mathit{Var}}({\hat{\beta }}_k)$ is the $k^{th}$ diagonal elements of ${\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}}^{\left(g\right)}$. Thus, the raw p-value for comparing group $k$ and group 1 is defined as

$$p_k=2\left[1-\varphi \left(\left|W_k\right|\right)\right].$$

Likewise, we apply the mdFDR controlling procedure for all pairwise tests. To improve power, we modify the global test mentioned earlier to a Dunnet-based^46–48 test as described below:

1. The test statistic $W=\underset{k\in \{1,\dots ,g\}}{\max }\left|W_k\right|$,

2. Generate $W_k^{\left(b\right)}\sim N(0,1),k=1,\dots ,g$.

3. Compute $W^{\left(b\right)}=\underset{k\in \{1,\dots ,g\}}{\max }|W_k^{\left(b\right)}|$.

4. Repeat the above steps $B$ times, we get the null distribution of $W$.

The screening p-value is calculated as:

$$p=\frac{1}{B}\sum _{b=1}^{B}I(W^{(b)}>W).$$

Pattern analysis

When the experimental groups are ordered naturally, such as doses of exposure or duration of exposure or stages of a disease, etc., for a given taxon, researchers may be interested in testing whether the abundance of the taxon is changing with the ordered experimental groups according to some specific pattern. Thus, the null and alternative hypotheses one wants to test become (assume group 0 is the reference group):

$$\begin{gathered} H_0:{\beta }_1={\beta }_2=\dots ={\beta }_g=0, \\ {H}_1:\beta ={\left({\beta }_1,\dots ,{\beta }_g\right)}^T\in ℂ, \end{gathered}$$

where $ℂ$ is one or a collection of patterns. Examples of patterns are given below.

Example 0.3 (Simple order).

$${ℂ}_1=\{0\le {\beta }_1\le {\beta }_2\le \dots \le {\beta }_g\}\text{with at least one strict inequality.}$$ (22)

Example 0.4 (Tree order).

$${ℂ}_2=\{{\beta }_k\ge 0,k=1,\dots ,g\}\text{with at least one strict inequality.}$$ (23)

Example 0.5 (Umbrella order).

$${ℂ}_4=\{0\le {\beta }_1\le \dots \le {\beta }_{k-1}\le {\beta }_k\ge {\beta }_{k+1}\dots \ge {\beta }_g\}\text{with at least one strict inequality.}$$ (24)

Estimation of $\beta$ under a certain pattern (constraint) can be obtained by solving the following convex optimization problem⁴⁹:

$${\hat{\beta }}^{\mathit{\text{opt}}}=\underset{\beta \in ℂ}{\text{arg min}}{(\hat{\beta }-\beta )}^T{\widehat{\text{Σ}}}^{{(g)}^{-1}}(\hat{\beta }-\beta ),$$ (25)

where ${\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}}^{\left(g\right)}$ is the corresponding sub-matrix of $\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}$ defined in (10). The solution to (25) can be numerically obtained by using a suitable convex optimization algorithm, such as CVRX⁵⁰.

Example 0.6. Suppose there are 3 groups, namely, groups 0 (reference), 1, and 2, and no other covariates. For each sample $i,i=1,\dots ,n$, we have:

$$y_i={\theta }_i+\mu +{\beta }_{1}I\{\mathit{\text{group}}=1\}+{\beta }_{2}I\{\mathit{\text{group}}=2\}+{\epsilon }_i.$$

To test whether the group effect is monotonically increasing, we would like to test:

$$\begin{gathered} H_0:{\beta }_1={\beta }_2=0, \\ {H}_1:\beta \in ℂ=\left\{0\le {\beta }_1\le {\beta }_2\right\},\text{with at least one strict inequality.} \end{gathered}$$

The estimation of $\beta$ under $ℂ$ can be obtained by solving:

$$\begin{gathered} {\hat{\beta }}^{\mathit{\text{opt}}}=\underset{\beta \in {ℝ}^2}{\text{arg min}}{(\hat{\beta }-\beta )}^T{\widehat{\text{Σ}}}^{{\left(g\right)}^{-1}}(\hat{\beta }-\beta ), \\ \text{s.t.}A\beta \ge 0, \end{gathered}$$

where $\mathbf{A}=\left[\begin{array}{cc}1& 0\\ -1& 1\end{array}\right]$, and $\beta ={({\beta }_1,{\beta }_2)}^T$.

Once the constrained estimator is obtained, there exist a variety of options to test the above hypotheses. For example, one may consider William’s type of statistic⁵¹. We adopt the following definitions from Peddada et al. (2002)⁷ to facilitate the construction of the test statistic.

Definition 2 (Linked parameters): Two parameters in a given pattern are said to be linked if the inequality between them is specified a priori.

Definition 3 (Nodal parameter): For a given pattern, a parameter is said to be nodal if it is linked with every other parameter in the profile.

For example, every parameter is a nodal parameter in ${ℂ}_1$; no nodal parameter in ${ℂ}_2$; and ${\beta }_k$ is the only nodal parameter in ${ℂ}_3$.

Definition 4 (Norm of maximum difference): Define the norm $l_{\mathrm{\infty }}\left(ℂ\right)$ of pattern $ℂ$ as the maximum difference between the estimates of two linked parameters.

For example, $l_{\mathrm{\infty }}({ℂ}_3)=\max \{{\hat{\beta }}_k,{\hat{\beta }}_k-{\hat{\beta }}_g\}$.

Given a collection of potential patterns, ${ℂ}_1,{ℂ}_2,\dots ,{ℂ}_T$, the William’s type of test statistic is defined as:

$$\begin{gathered} W=\text{max}\{l_{\infty }\left({ℂ}_t\right),t=1,\dots ,T\}, \\ \text{with}{t}^{opt}=\text{arg max}\{l_{\infty }\left({ℂ}_t\right),t=1,\dots ,T\}, \end{gathered}$$

where $t^{\mathit{\text{opt}}}$ is regarded as the optimal pattern for the microbial abundance of a specific taxon.

Under null hypothesis, the expectations for ${\hat{\beta }}_k,k=1,\dots ,g$ are 0s; thus, we can construct the null distribution of $W$ as follows:

1. Generate ${\hat{\beta }}_k^{\left(b\right)}\sim \sqrt{\widehat{\mathit{Var}}({\hat{\beta }}_k)}N(0,1),l=1,\dots ,g$.

2. Obtain constrained regression estimators for ${\hat{\beta }}_k^{opt,\left(b\right)}$ using the convex optimization problem described above.

3. Compute $W^{\left(b\right)}=\mathrm{m}\mathrm{a}\mathrm{x}\left\{l_{\mathrm{\infty }}\left({ℂ}_t\right),t=1,\dots ,T\right\}$ using the simulated data under pre-specified patterns.

4. Repeat the above steps $B$ times, and we get the null distribution of $W$.

The raw p-value is calculated as

$$p=\frac{1}{B}\sum _{b=1}^{B}I(W^{(b)}>W).$$

We then apply the Holm-Bonferroni correction or BH procedure on raw p-values to control the FDR.

ANCOM-BC2 for mixed effects models

Similar to the fixed effects model stated in (3), for each taxon $j,j=1,\dots ,d$, and each sample $i,i=1,\dots ,n$, suppose each sample has $n_i$ observations and ${\sum }_i{n}_i=n$. The offset-based mixed effects log-linear model is set up as

$$y_{ij}={\theta }_i{\mathbf{1}}_{n_i}+X_i{\beta }_j+Z_i{\alpha }_i+{\epsilon }_{ij},$$ (26)

where

1. ${\mathbf{y}}_{\mathbf{i}\mathbf{j}}$ is the $n_i$-vector centered observed abundances,

2. ${\mathbf{1}}_{{\mathbf{n}}_{\mathbf{i}}}=(1,\dots ,1{)}^T\in {ℝ}^{n_i}$ is a vector of 1s,

3. ${\mathbf{X}}_{\mathbf{i}}$ is the $n_i\times p$ design matrix for fixed effects,

4. ${\beta }_j$ is the $p$-vector of fixed effects regression coefficients to be estimated,

5. ${\mathbf{Z}}_{\mathbf{i}}$ is the $n_i\times q$ design matrix for the random effects,

6. ${\alpha }_i$ is the $q$-vector random effects,

7. ${\epsilon }_{ij}$ is the $n_i$-vector residuals.

The following distributional assumptions are made

$$\begin{gathered} {\alpha }_i\sim N(\mathbf{0},D_{q\times q}), \\ {\epsilon }_{ij}\sim N(\mathbf{0},{\sigma }_j^2{I}_{n_i}), \\ {\alpha }_i⫫{\epsilon }_{ij}\text{for}i=1,\dots ,n. \end{gathered}$$

Thus, for each taxon $j,j=1,\dots ,d$, and each sample $i,i=1,\dots ,n$, we have

$$y_{ij}\sim N({\theta }_i{\mathbf{1}}_{n_i}+X_i{\beta }_j,H_{ij}(\tau )),$$

where ${\mathbf{H}}_{\mathbf{i}\mathbf{j}}(\tau )={\mathbf{Z}}_{\mathbf{i}}\mathbf{D}{\mathbf{Z}}_{\mathbf{i}}^T+{\sigma }_j^2{\mathbf{I}}_{\mathbf{n}\mathbf{i}}$ (or ${\mathbf{H}}_{\mathbf{i}\mathbf{j}}$ for short) denotes a general covariance matrix parametrized by $\tau$.

Stack up observations across samples, we have:

$$y_j=\theta +X{\beta }_j+Z\alpha +{\epsilon }_j,$$ (27)

where

$$\begin{array}{l}{y}_j=\left[\begin{array}{c}{y}_{\mathbf{1}j}\\ y_{\mathbf{2}j}\\ ⋮\\ y_{nj}\end{array}\right],\theta =\left[\begin{array}{c}{\theta }_1{\mathbf{1}}_{n_{\mathbf{1}}}\\ {\theta }_2{\mathbf{1}}_{n_{\mathbf{2}}}\\ ⋮\\ {\theta }_n{\mathbf{1}}_{n_n}\end{array}\right],X=\left[\begin{array}{c}{X}_{\mathbf{1}}\\ X_{\mathbf{2}}\\ ⋮\\ X_n\end{array}\right],{\beta }_j=\left[\begin{array}{c}{\beta }_{j1}\\ {\beta }_{j2}\\ ⋮\\ {\beta }_{jp}\end{array}\right],\\ Z=\left[\begin{array}{cccc}{Z}_{\mathbf{1}}& \mathbf{0}& \dots & \mathbf{0}\\ \mathbf{0}& Z_{\mathbf{1}}& \mathbf{0}& \mathbf{0}\\ ⋮& ⋮& \ddots & ⋮\\ \mathbf{0}& \mathbf{0}& \dots & Z_{\mathbf{1}}\end{array}\right],\alpha =\left[\begin{array}{c}{\epsilon }_{1j}\\ {\alpha }_2\\ ⋮\\ {\epsilon }_{2j}\\ ⋮\\ {\alpha }_n\end{array}\right],{\epsilon }_j=\left[\begin{array}{c}{\epsilon }_{1j}\\ {\epsilon }_{2j}\\ ⋮\\ {\epsilon }_{nj}\end{array}\right].\end{array}$$

That is,

$$y_j\sim N\left(\theta +X{\beta }_j,H_j(\tau )=\left[\begin{array}{cccc}{H}_{\mathbf{1}j}(\tau )& \mathbf{0}& \dots & \mathbf{0}\\ \mathbf{0}& H_{\mathbf{2}j}(\tau )& \mathbf{0}& \mathbf{0}\\ ⋮& ⋮& \ddots & ⋮\\ \mathbf{0}& \mathbf{0}& \dots & H_{nj}(\tau )\end{array}\right]\right),$$

where ${\mathbf{H}}_{\mathbf{j}}\left(\tau \right)$ (or ${\mathbf{H}}_{\mathbf{j}}$ for short) is a block diagonal matrix.

Similarly, we estimate $\theta$ and ${\beta }_j$ iteratively to obtain the corresponding preliminary estimators. As compared to Algorithm 1, the Maximum Likelihood (ML) is replaced with Restricted Maximum Likelihood (ReML)^{52, 53}.

Algorithm 3.

Iterative ReMLE

1:	Initialize:
	For $j=1,\dots ,d$
	$\theta \leftarrow 0$
	$y_j^{(\mathbf{\text{crt}})}\leftarrow y_j-\theta =y_j$
	${\beta }_j\leftarrow \text{ReML}(y_j^{(\mathbf{crt})})=\mathbf{\text{ReML}}(y_j)$
2:	while not converge do
3:	$\theta \leftarrow \frac{1}{d}{\sum }_{j=1}^d\left(y_j-X{\beta }_j\right)$
4:	${\mathbf{y}}_{\mathbf{\text{j}}}^{\left(\mathbf{\text{crt}}\right)}{\mathbf{y}}_{\mathbf{\text{j}}}-\theta$
5:	${\beta }_j\leftarrow \text{ReML}({\mathbf{y}}_{\mathbf{\text{j}}}^{\left(\mathbf{\text{crt}}\right)})$
6:	end while

Note that the estimators for regression coefficients ${\beta }_j$ and variance components $\tau$ are obtained iteratively by maximizing the following log-likelihood function:

$$ℒ(\tau \mid y_j)=-\sum _{i=1}^n\text{log}|H_{ij}|-\sum _{i=1}^n\text{log}|{X_i}^T{H_{ij}}^{-1}{X}_i|-\sum _{i=1}^n{(y_{ij}-X_i{\beta }_j)}^T{H_{ij}}^{-1}(y_{ij}-X_i{\beta }_j),$$ (28)

where ${\beta }_j\leftarrow {({\mathbf{X}}^T{{\mathbf{H}}_{\mathbf{j}}}^{-1}\mathbf{X})}^{-1}{\mathbf{X}}^T{{\mathbf{H}}_{\mathbf{j}}}^{-1}{\mathbf{y}}_{\mathbf{j}}$. As close-form solutions of (28) do not exist, Newton-Raphson method is usually employed⁵⁴.

Suppose on convergence, $\theta \leftarrow {\theta }^{\ast },{\mathbf{y}}_{\mathbf{j}}^{(\mathbf{c}\mathbf{r}\mathbf{t})}\leftarrow {\mathbf{y}}_{\mathbf{j}}^{{(\mathbf{c}\mathbf{r}\mathbf{t})}^{\ast }},\mathbf{H}\leftarrow {\mathbf{H}}^{\ast },{\beta }_j\leftarrow {\beta }_j^{\ast }$, we have

$$\begin{gathered} {\theta }^{\ast }=\frac{1}{d}\sum _{j=1}^d(y_j-X{{\beta }_j}^{\ast }), \\ {y}_j^{{(crt)}^{\ast }}=y_j-{\theta }^{\ast }, \\ {{\beta }_j}^{\ast }={(X^T{H}_j^{\ast -1}X)}^{-1}{X}^T{H}_j^{\ast -1}{y}_j^{{(crt)}^{\ast }}. \end{gathered}$$

It is easy to show that there exists a vector $\delta \in {ℝ}^p$, such that

$$\begin{gathered} E({\theta }^{\ast })=\theta -X\delta , \\ E({{\beta }_j}^{\ast })=\delta +{\beta }_j. \end{gathered}$$

i.e., ${{\beta }_j}^{\ast }$ is a biased estimator for ${\beta }_j$.

Similar to the case of fixed effects model, we fit the Gaussian mixture model to each ${\beta }_{jk},k=1,\dots ,p$ separately, to correct the bias $\delta$, and final estimators for ${\beta }_j$ and $\theta$ are given by

$$\begin{gathered} {\hat{\beta }}_j={\beta }_j^{\ast }-{\hat{\delta }}^{EM}, \\ \hat{\theta }=\frac{1}{d}\sum _{j=1}^d(y_j-X{\hat{\beta }}_j). \end{gathered}$$

Thus, for taxon $j$, the Wald statistic for the following hypotheses

$$\begin{gathered} H_0:A{\beta }_j=A{{\beta }_j}^0, \\ {H}_1:A{\beta }_j\ne A{{\beta }_j}^0, \end{gathered}$$

is given by

$$\begin{gathered} W_j={(A{\hat{\beta }}_j-A{{\beta }_j}^0)}^T{(A{\widehat{\text{Σ}}}_j{A}^T)}^{-1}(A{\hat{\beta }}_j-A{{\beta }_j}^0) \\ ={(A{{\beta }_j}^{\ast }-A{\hat{\delta }}^{EM}-A{{\beta }_j}^0)}^T{(A{\widehat{\text{Σ}}}_j{A}^T)}^{-1}(A{{\beta }_j}^{\ast }-A{\hat{\delta }}^{EM}-A{{\beta }_j}^0) \\ {\to }_d{\chi }_q^2, \end{gathered}$$

where

1. ${\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}}_j={\left({\mathbf{X}}^T{{\mathbf{H}}_{\mathbf{j}}}^{\ast -1}\mathbf{X}\right)}^{-1}$,

2. $q=\mathit{rank}(\mathbf{A})$.

To draw statistical inference under inequality constraints in linear mixed effects model, we use William’s type test statistic as specified in the last section, and adopt the Constrainted Linear Mixed Effects (CLME) framework⁴² into ANCOM-BC. The procedure is summarized as follows:

1. Obtain ${\hat{\beta }}_j$, the estimate of ${\beta }_j$ under the null hypothesis,

2. Compute the observed values of residuals and random effects: ${\hat{\epsilon }}_j=(\mathbf{I}-\mathbf{X}{({\mathbf{X}}^T{\hat{\mathbf{H}}}_{\mathbf{j}}^{-1}\mathbf{X})}^{-1}{\mathbf{X}}^T{\hat{\mathbf{H}}}_{\mathbf{j}}^{-1}){\mathbf{y}}_{\mathbf{j}}^{(\mathbf{c}\mathbf{r}\mathbf{t}{)}^{\ast }}$, and $\hat{\alpha }=\hat{\theta }{\mathbf{Z}}^T{\hat{\mathbf{H}}}_{\mathbf{j}}^{-1}{\hat{\epsilon }}_j$,

3. Standardize the observed values of the random effects and residuals. Define $\hat{\mathbf{a}}=SE(\hat{\alpha }{)}^{-1}\hat{\alpha }$, and $\hat{\mathbf{e}}=SE(\hat{\epsilon }{)}^{-1}\hat{\epsilon }$, where $SE(\cdot )$ denotes the standard error,

4. Obtain bootstrap samples. Let ${\mathbf{a}}^{\left(b\right)}$ and ${\mathbf{e}}^{\left(b\right)}$ denote the bootstrap samples of $\mathbf{a}$ and $\mathbf{e}$, respectively. Then define ${\hat{\alpha }}^{\left(b\right)}=SE(\hat{\alpha }){\mathbf{a}}^{\left(b\right)}$ and ${\hat{\epsilon }}^{\left(b\right)}=SE(\hat{\epsilon }){\mathbf{e}}^{\left(b\right)}$. Finally construct the final bootstrap sample as: ${\mathbf{y}}_{\mathbf{j}}{ }^{\left(b\right)}=\hat{\theta }+\mathbf{X}{\hat{\beta }}_j+\mathbf{Z}{\hat{\alpha }}^{\left(b\right)}+{\stackrel{\mathit{ˆ}}{\mathit{\epsilon }}}^{\left(b\right)}$,

5. Repeat $B$ times $(b=1,\dots ,B)$, we construct the null distribution for the test statistic.

6. Compute raw p-values,

7. Apply the Holm-Bonferroni correction (default) or BH procedure on raw p-values to control the FDR.

Data availability

Simulation data can be found on the corresponding GitHub repository⁵⁵. The soil microbiome data for aridity can be found in Qiita: https://qiita.ucsd.edu/study/description/10360. The gut microbiome data for IBD can be found in Qiita: https://qiita.ucsd.edu/study/description/11546.