benchdamic: benchmarking of differential abundance methods for microbiome data

Abstract

Summary

Recently, an increasing number of methodological approaches have been proposed to tackle the complexity of metagenomics and microbiome data. In this scenario, reproducibility and replicability have become two critical issues, and the development of computational frameworks for the comparative evaluations of such methods is of utmost importance. Here, we present benchdamic, a Bioconductor package to benchmark methods for the identification of differentially abundant taxa.

Availability and implementation

benchdamic is available as an open-source R package through the Bioconductor project at https://bioconductor.org/packages/benchdamic/.

Supplementary information

Supplementary data are available at Bioinformatics online.

1 Introduction

Differential abundance (DA) analysis identifies significant differences in the microbial community composition between groups of samples, providing new insights into the composition of microbial communities and on their associations with the environment. Although many approaches have been proposed for DA analysis, it is widely recognized that the best method (i.e. a method with performances uniformly better than all the others) does not exist and that a careful exploratory data analysis is necessary to address methodological choices (Calgaro et al., 2020; Hawinkel et al., 2019; Nearing et al., 2022; Thorsen et al., 2016; Weiss et al., 2017).

Building on our previous work (Calgaro et al., 2020), we present the benchdamic R/Bioconductor package, which provides a computational framework to guide researchers in the selection of the method that best fits their data.

The structure of benchdamic can be summarized into four main parts (Fig. 1). Each section is developed to answer specific questions when comparing samples from different experimental groups, namely (i) the ability for a given statistical distribution to successfully fit the input data, with particular focus on sparsity and their count nature; (ii) the ability of the DA methods to control the type I error; (iii) the concordance among methods; and (iv) the accuracy of the findings based on a priori biological knowledge. Altogether, benchdamic is a flexible and customizable framework that can be used for the benchmarking of new and existing DA methods.

Fig. 1.

Graphical abstract. Each box on the right represents a step of the analysis where information about the research question, type of input data, working functions and outputs are reported

2 Implementation

benchdamic builds on existing R/Bioconductor infrastructure packages: the primary input of benchdamic’s main functions is a phyloseq or a TreeSummarizedExperiment object (Huang et al., 2021; McMurdie and Holmes, 2013). Ready-to-use normalization and DA methods included in benchdamic are based on the edgeR (Robinson et al., 2010), DESeq2 (Love et al., 2014), limma-voom (Law et al., 2014; Phipson et al., 2016; Ritchie et al., 2015), metagenomeSeq (Paulson et al., 2013), ALDEx2 (Fernandes et al., 2014, 2013), corncob (Martin et al., 2020), MAST (Finak et al., 2015), Seurat (Butler et al., 2018), dearseq (Gauthier et al., 2020), NOISeq (Tarazona et al., 2015), ANCOMBC (Lin and Peddada, 2020; Mandal et al., 2015) and zinbwave (Risso et al., 2018; Van den Berge et al., 2018) packages. Combinations of parameters are possible as well as the inclusion of custom methods (see Supplementary material Section S3).

In the following sections, we briefly outline the main functionality of the package. See Calgaro et al. (2020) for technical details on how these metrics are computed.

2.1 Goodness of fit

DA statistical models are based on different statistical distributions. Five different distributions are available in benchdamic for testing the goodness of fit on user-provided data: negative binomial, zero-inflated negative binomial, zero-inflated Gaussian, truncated Gaussian and Dirichlet-multinomial (see Supplementary material Section S2). Goodness of fit is measured by the ability of each method to correctly estimate the average counts and the probability of observing a zero.

2.2 Type I error control

To investigate the Type I error rate control of each DA method (i.e. the probability of the statistical test to call a feature DA when it is not) mock datasets with no true DA are generated starting from the user-provided data (see Supplementary material Section S4).

Briefly, the dataset is split into two random subsets and DA analysis, based on a chosen list of methods, is performed. The process is repeated N times (N ≥ 1000 suggested). The performances of each method are then summarized and graphically represented considering the false positive rate, false discovery rate, and departure from uniformity for the P-values distribution.

2.3 Concordance

benchdamic can be used to measure the between-method concordance (BMC), in which a DA method is compared to other methods in the same dataset, and the within-method concordance (WMC), in which a method is compared to itself in two random subsets of the same dataset (Supplementary material Section S5). Firstly, the dataset is randomly divided in half to obtain two subsets (Subset1 and Subset2) with samples from two or more biological groups, then DA analysis is performed between two groups, independently on each subset. The process is repeated N times (N ≥ 100 suggested) and average WMC and BMC metrics are computed and summarized using a heatmap representation.

2.4 Enrichment

Enrichment analysis can provide an alternative way of ranking methods in terms of their ability to identify, as significantly different, taxa that are known to be differentially abundant between two groups. DA analysis needs to be performed on a dataset where some a priori knowledge is available (Supplementary material Section S6). Given the direction of the DA features (over- or under-abundant) and the expected group in which the features should be differentially abundant according to the prior knowledge, several contingency tables are created for each DA method. A Fisher exact test is then performed to test the enrichment and the DA features identified by more than one method are highlighted. Additionally, the users will be able to rank the methods based on the difference between the total number of true positives and false positives for several thresholds (based on P-values, adjusted P-values or other statistics). The same approach can also be used to perform power analysis using simulated data (Supplementary material Section S6.8).

3 Conclusions

The benchdamic R/Bioconductor package aims to be a support tool for the identification of DA microbial taxa and the benchmarking of new methods. We envision two main uses of our package: (i) for practitioners interested in performing DA analysis on a new dataset, benchdamic can be used to identify the best DA methods among those already in the literature; (ii) for method developers interested in benchmarking their new approach, benchdamic can be used as an impartial tool to evaluate the relative merits of the new method compared to what is already available.

benchdamic is available as an open-source package through the Bioconductor project. The package includes a vignette with a detailed tutorial.

The future of benchdamic is oriented to the addition of new aspects of analysis e.g. new normalization methods and new DA approaches.