redbiom: a Rapid Sample Discovery and Feature Characterization System

Although analyses that combine many microbiomes at the whole-community level have become routine, searching rapidly for microbiomes that contain a particular sequence has remained difficult. The software we present here, redbiom, dramatically accelerates this process, allowing samples that contain microbiome features to be rapidly identified. This is especially useful when taxonomic annotation is limited, allowing users to identify environments in which unannotated microbes of interest were previously observed. This approach also allows environmental or clinical factors that correlate with specific features, or vice versa, to be identified rapidly, even at a scale of billions of sequences in hundreds of thousands of samples. The software is integrated with existing analysis tools to enable fast, large-scale microbiome searches and discovery of new microbiome relationships.

ABSTRACT

Meta-analyses at the whole-community level have been important in microbiome studies, revealing profound features that structure Earth’s microbial communities, such as the unique differentiation of microbes from the mammalian gut relative to free-living microbial communities, the separation of microbiomes in saline and nonsaline environments, and the role of pH in driving soil microbial compositions. However, our ability to identify the specific features of a microbiome that differentiate these community-level patterns have lagged behind, especially as ever-cheaper DNA sequencing has yielded increasingly large data sets. One critical gap is the ability to search for samples that contain specific features (for example, sub-operational taxonomic units [sOTUs] identified by high-resolution statistical methods for removing amplicon sequencing errors). Here we introduce redbiom, a microbiome caching layer, which allows users to rapidly query samples that contain a given feature, retrieve sample data and metadata, and search for samples that match specified metadata values or ranges (e.g., all samples with a pH of >7), implemented using an in-memory NoSQL database called Redis. By default, redbiom allows public anonymous sample access for over 100,000 publicly available samples in the Qiita database. At over 100,000 samples, the caching server requires only 35 GB of resident memory. We highlight how redbiom enables a new type of characterization of microbiome samples and provide tutorials for using redbiom with QIIME 2. redbiom is open source under the BSD license, hosted on GitHub, and can be deployed independently of Qiita to enable search of proprietary or clinically restricted microbiome databases.

IMPORTANCE Although analyses that combine many microbiomes at the whole-community level have become routine, searching rapidly for microbiomes that contain a particular sequence has remained difficult. The software we present here, redbiom, dramatically accelerates this process, allowing samples that contain microbiome features to be rapidly identified. This is especially useful when taxonomic annotation is limited, allowing users to identify environments in which unannotated microbes of interest were previously observed. This approach also allows environmental or clinical factors that correlate with specific features, or vice versa, to be identified rapidly, even at a scale of billions of sequences in hundreds of thousands of samples. The software is integrated with existing analysis tools to enable fast, large-scale microbiome searches and discovery of new microbiome relationships.

OBSERVATION

Data reuse has posed a significant challenge in the microbiome field, especially because of technical variation among studies (1). Analyses at the whole-community level, typically using principal-coordinate analysis (PCoA) or similar dimensionality reduction techniques, have nevertheless revealed many large-scale patterns relating microbiomes to one another (2–4), especially when standardized techniques are used either within one study or across many studies in a consortium effort using common protocols (5). In particular, resources such as Qiita (6) were developed to facilitate reuse of data and now house amplicon data from hundreds of thousands of microbiomes with associated metadata (per-sample, per-individual, and/or per-site information related to each sample) in the standardized MIxS format introduced by the Genomic Standards Consortium (7).

There is a need to search for samples that contain particular microbial taxa and for taxa that explain differences among samples. These tasks are especially important for revealing which specific microbes are associated with particular environmental or clinical metadata. Performing the search directly at the sequence level is possible, but typically incurs substantial computational effort, especially as improvements in sequencing technology yield ever-larger data sets. To address this need, we developed redbiom, which enables rapid discovery and retrieval of sample data into BIOM tables (8) for immediate integration for meta-analysis.

Figure 1 outlines the redbiom data model. At its core, redbiom is a structured data model built off Redis, a key-value in memory NoSQL database. Sample data are stored in sparse vectors allowing hundreds of thousands of samples with multiple different processing to be represented in under 40 GB (underlying sequence data total of 45 TB). Identifiers are remapped into a unique integer space to minimize memory utilization and to leverage Redis ziplist optimizations. Data are partitioned by sequencing and bioinformatics protocol to minimize technical biases. These partitions, called “contexts,” allow for identifying samples processed in one way (e.g., Deblur [9]) and obtaining data from another (e.g., closed reference operational taxonomic unit [OTU] picking). Sample and preparation information are indexed efficiently and allow retrieval of a specific variable for a given sample. These variables are additionally indexed by applying Porter stemming (10) to all unique strings such that each stem forms a key that is associated with a set containing the samples where that stem was observed. The combination of indexing strategies allows users to generally search for samples (e.g., all samples with the stem of antibiotics) or to constrain the searches to specific variables and values (e.g., all samples with “soil” in the description field and a pH of <7).

FIG 1.

The redbiom data model is a key-value store built on top of Redis. By storing features and sample identifiers as keys, it is possible to rapidly query the resource for information on those entities. Similarly, by indexing the sample metadata, queries can be performed against variables of interest (e.g., pH) in order to identify sample identifiers of interest, which can then be used to extract a feature table for downstream analysis. (A) A “set” command associates a key with a value: in this case, a feature identifier is associated with the samples the feature was observed in. A “get” command can then be issued using the feature identifier as the key to obtain the associated values (i.e., the samples). (B) Feature counts (e.g., a vector from an OTU table) are associated with a composite key that describes the processing context and the sample identifier. The processing context, in this case “deblur,” denotes a bioinformatic procedure applied. For Qiita, the context names also include molecular preparation details. The expectation is the data within a context should be comparable. The sample data themselves are encoded in a sparse vector format with the feature identifiers remapped into unique integers to improve compression and reduce data redundancy. (C) The Porter stem of the word “Antibiotics.” (D) The association of metadata word stems with sample identifiers. Redis natively supports classic set operations, which can be applied to keys to obtain, for example, the intersection of sample identifiers represented by two keys.

redbiom enables a new paradigm for microbiome analysis and data mining. With indexed exact sequences, it is possible to perform a maximal-precision search of deposited studies to test for replication (as noted in reference 11 [example below]). This is in contrast to manually identifying studies and processing and searching existing raw data or the more frequent strategy of relying on imprecise taxon names mentioned in manuscripts (e.g., hunting for Clostridium sp. enrichment in human fecal studies). As redbiom indexes sample metadata and taxonomic information (when available), it also readily allows users to identify samples of interest for comparative purposes: e.g., “How do my samples compare to the Earth Microbiome Project soil samples?” By partitioning technical parameters, it is possible to identify samples in one context and extract from another (e.g., selecting samples with closed reference OTUs based on the presence of specific 16S Deblur sub-operational taxonomic units [sOTUs]).

To test the search capability, we obtained sOTUs from a novel differential abundance method (12) in which five sOTUs were observed to strongly associate with high-pH soils and five with low-pH soils (see Table S1 in the supplemental material), in a reanalysis of a study by Ramirez et al. (13). We sought to determine whether the pH association of these sOTUs replicated across studies. Each sOTU was searched against 137,678 samples using redbiom, resulting in a total of 560 unique samples from 20 different Qiita studies (see the observed studies in Table S2 and the bash script for search in Text S1 in the supplemental material); a sample was only pulled out of Qiita if it contained any of the five high-pH or five low-pH sOTUs of interest. We did not calculate the prevalence of an sOTU because the interpretation may be misleading given inherent biases in which studies are represented in Qiita, different depths of sampling among different studies, etc. The search for samples, extraction of Deblur-processed data in BIOM format, and retrieval of sample metadata was performed per feature and took an average of 20 s. The pH of the observed samples was significantly different depending on the source feature set (Fig. 2A; Mann-Whitney U statistic = 7,280, P < 9.95 × 10⁻⁶⁵). We then rarefied the samples to 1,000 sequences per sample and performed UniFrac (14) and principal-coordinate analysis on the collected samples, observing pH as a driver of community composition (Fig. 2B, unweighted Unifrac, Pearson’s r = 0.552, P < 6.61 × 10⁻⁴⁶; Fig. 2C, weighted UniFrac, r = 0.562, P < 6.8 × 10⁻⁴⁸). Visualization of the coordinates shows a visual pH gradient despite some study grouping (Fig. 2D to G), which is expected given the design of some studies (e.g., the Cannabis soil microbiome [15]). The analysis indicates that pH is a driver of overall community structure across multiple projects from a variety of institutions with markedly different research questions, soils, and locations.

FIG 2.

Feature search example. Differential sOTUs from a reanalysis of the study by Ramirez et al. (13) by Morton et al. (unpublished), characterized as associating with a low- or high-pH soil, were obtained. Features were trimmed to 90 nucleotides (nt) to maximize overlap of the Earth Microbiome Project and were searched using redbiom against the Deblur 16S V4 90-nt context with the following sample constraints: “where empo_3=='Soil (non-saline)' and ph > 0.” All samples from Ramirez et al. were removed to create a sample set independent from the observation source: 560 samples remained for assessment following constraints and filtering. (A) Box-whisker plot of the pH values reported in the sample information (Mann-Whitney U statistic = 7,280, P < 9.95 × 10⁻⁶⁵). (B and C) Regressions of the reported pH values against the first principal coordinate (PC1) from unweighted (B) and weighted (C) UniFrac analysis (Pearson r = 0.552, P < 6.61 × 10⁻⁴⁶, and r = 0.562, P < 6.8 × 10⁻⁴⁸, respectively). (D to G) Principal-coordinate plots of unweighted (D) and weighted (E) UniFrac of the observed samples colored by pH and unweighted (F) and weighted (G) UniFrac colored by the Qiita study identifier. (See Table S2 for additional study information.)

TEXT S1

A BASH script that performs the search in support of Fig. 2. Download Text S1, TXT file, 0.1 MB ^{(1.7KB, txt)} .

Copyright © 2019 McDonald et al.

This content is distributed under the terms of the Creative Commons Attribution 4.0 International license.

TABLE S1

The features searched for in the meta-analysis in Fig. 2. Download Table S1, XLSX file, 0.1 MB ^{(37.4KB, xlsx)} .

Copyright © 2019 McDonald et al.

This content is distributed under the terms of the Creative Commons Attribution 4.0 International license.

TABLE S2

Studies used in the meta-analysis presented in Fig. 2. Download Table S2, XLSX file, 0.1 MB ^{(20.5KB, xlsx)} .

Copyright © 2019 McDonald et al.

This content is distributed under the terms of the Creative Commons Attribution 4.0 International license.

redbiom provides a critical part of the Earth Microbiome Project (5) infrastructure, underpinning the popular Trading Cards, with a default database that is regularly updated as new data are made public in Qiita (6). Additionally, redbiom allows queries across processing partitions, allowing users to operate across technical parameters if needed (e.g., to identify samples by Deblur and retrieve closed reference OTUs), as well as searching for samples by taxonomy when taxonomic information is present. These issues and others are explored in detail in a community tutorial for using redbiom with QIIME 2 (16), which together with the forum, the BSD open source license, and compatibility with microbiome standards will promote a broad user community. Finally, we note that the data model on which redbiom depends is general, allowing storage of gene expression and metabolomics data, and we expect that redbiom will provide a key underpinning for future multiomics microbiome studies as these capacities expand in the field.