Are microbiome studies ready for hypothesis-driven research?

Abstract

Hypothesis-driven research has led to many scientific advances, but hypotheses cannot be tested in isolation: rather, they require a framework of aggregated scientific knowledge to allow questions to be posed meaningfully. This framework is largely still lacking in microbiome studies, and the only way to create it is by discovery-, tool-, and standards-driven research projects. Here we illustrate these issues using several such non-hypothesis-driven projects from our own laboratories, including spatial mapping, the American Gut Project, the Earth Microbiome Project (which is an umbrella project integrating many smaller hypothesis-driven projects), and the knowledgebase-driven tools GNPS and Qiita. We argue that an investment of community resources in infrastructure tasks, and in the controls and standards that underpin them, will greatly enhance the investment in hypothesis-driven research programs.

Introduction

Microbiome research is making dramatic progress, with thousands of papers now published each year linking specific microbes and/or host-microbe co-metabolites to specific diseases, physiological properties, or environmental parameters. Much of this research is performed in a traditional, hypothesis-driven way, or at least presented as a rational reconstruction that fits this model, much as Darwin re-wrote much of his discovery-driven work as hypothesis driven to increase its respectability under the influence of contemporary philosophers of science such as William Whewell (1). However, it should be noted that hypothesis-driven science was not always so respectable -- Isaac Newton famously wrote “Hypotheses non fingo”, or “I feign no hypotheses”, in an essay appended to the second edition of the Principia (2) -- so the tradition of modifying how science is framed to meet respectability criteria dates back at least 300 years. What can be framed as a testable hypothesis suffers important limitations based on what we can measure and what we already know.

Ten years ago Chris Anderson, editor of Wired magazine, set off an international debate with his article “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete” (3). The idea was that with enough data, hypotheses will emerge (“Let the data speak for itself”) has become widely discussed in the rapidly growing data science profession. A thoughtful review of this topic was written in EMBO Reports in 2015-”Could Big Data be the end of theory in science? A few remarks on the epistemology of data-driven science” (4). As the author points out:

“Francis Bacon, the “father of the scientific method” himself, in his Novum Organum (1620), argued that scientific knowledge should not be based on preconceived notions but on experimental data. Deductive reasoning, he argued, is eventually limited because setting a premise in advance of an experiment would constrain the reasoning so as to match that premise. Instead, he advocated a bottom-up approach: In contrast to deductive reasoning, which has dominated science since Aristotle, inductive reasoning should be based on facts to generalize their meaning, drawing inferences from observations and data.”

We recently reviewed experimental design considerations for traditional hypothesis-driven microbiome studies elsewhere (5, 6), and do not discuss these issues further in this review. Here we describe the danger of jumping too soon into hypothesis testing, and describe the need for four major categories of non-hypothesis-driven research: better spatial and abstract maps, better tools, and better standards. Given space constraints, we illustrate these primarily using the American Gut Project (7), the Earth Microbiome Project (8), and tools we developed in our laboratories.

The challenge of unknown unknowns

In microbiome research, a recurring challenge has been that factors intuitively suspected to drive differences in the microbiome are less important than other, more surprising factors. For example, sex has a small impact on microbiomes across the human body (9, 10) and has a much weaker effect than many other variables such as age (even within adults), or the time of year the sample was collected(11, 12). However, sex is far more frequently reported than time of year. Similarly, although long-term dietary habits are correlated with the overall composition of the human microbiome within and between populations (7, 13–16), and dietary changes over months can lead to changes in overall microbiome composition larger than the differences between arbitrarily chosen individuals (17, 18), but short-term changes have transient effects smaller than typical differences between individuals (14, 19). However, many studies focus on short-term rather than long-term diet. Perhaps even more surprisingly, factors such as temperature and pH have much smaller impacts on environmental microbiomes than salinity(8, 20), and even the saline vs. non-saline difference is much smaller than the host-associated vs free-living difference (8, 21). Samples from different parts of the same person’s body differ more from one another in their overall microbial communities than radically different free-living microbial communities, such as soils versus oceans (8). Differences of this magnitude can also occur within the gut of a single person, with sufficiently large perturbation (7).

Because factors of large effect are often unknown and unreported, studies testing hypotheses concerning intuitively obvious factors of small effect are often subject to important confounding variables, that, when uncovered, prompt complete reinterpretation of the study. For example, suppose an investigator is unaware that cage effects are important in the microbiome (22), and profiles microbiomes in two cages each of two different genotypes of mice. The results will likely be driven by which cages happens to resemble each other more closely. If the variable of cage is not measured, or not tested in an unsupervised model, this important confounding variable will likely remain undiscovered, and the interpretation of the experiment entirely incorrect

Similarly, a frequent practice is to discard unannotated microbes or unannotated molecules, focusing on the subset of microbes or molecules that can be matched to an existing database. Because databases of both microbes and molecules are heavily biased (microbes, by studies of known pathogens that come from only a few taxonomic groups, and molecules, by commercially available compounds), the entities that best discriminate among classes of samples may be lost in the analysis: often, only 60% of sequences and 2% of molecular features from an untargeted metabolomics experiment can be annotated by existing references (23, 24). However, a rational reconstruction of why the annotatable microbes or molecules are plausibly connected to a phenotype of interest can frequently be developed, especially given the characteristics of these highly multivariate datasets that can lead to high false discovery rates when the true number of implicitly tested hypotheses is considered (25).

The need for spatial maps

An important metaphor in science and information visualization is the idea of the map. As data volumes increase, it is frequent that the main research activity in a field moves from tests of hypotheses of differences in individual variables among sites, to tests of these hypotheses with replicates at each site, to spatially or temporally explicit sampling, to detailed spatial maps that reveal otherwise unsuspected patterns. This progression has occurred in 16S rRNA ampliconbased microbiome studies over the past decade (8, 26), and increasingly characterizes mass spectrometry-based metabolome studies over the past four years (27–32).

The value of spatial maps is so self-evident that the results are often cursed by obviousness. For example, the finding that metabolomes cluster by individual, as revealed by principal coordinates analysis (PCoA), is interesting (Fig. 1A). However, the finding that a given molecule such as lauryl sulphate (m/z 355.219) is distributed across the body of one of the two individuals, but is absent from the other individual is obvious (Fig. 1B), especially when subject A, who is male, reports using the skin care product Nivea for Men, the source of the molecule(28). Similarly, the finding that samples from four individuals differ significantly in levels of specific purines between and within subjects might well prompt further investigation. However, a spatial map with dense sampling of the same individuals (Fig. 1C) makes it obvious that the molecule is something that is touched and consumed, and sometimes spilled, allowing one to guess that it is caffeine; similarly, the spatial map reveals that one person likely spends time in the ocean based on the distribution of Synechococcus spp. (Fig. 1D) (30).

Figure 1.

Spatial analysis based on metabolomics of skin samples and a human habitat. A) Principal coordinates analysis (Hellinger distance) of metabolomics data of skin swabs obtained from several hundreds locations on the human body of four volunteers. B) The detection of lauryl sulfate (m/z 355.219) from the shampoo Nivea for Men on a male volunteer. C) The distribution of caffeine (m/z 195.088) on four individuals and office environment. D) The distribution of Synechococcus spp. on within that same office environment.

We have no idea where most microbes and molecules occur in and on the human body, in natural environments, or in human-constructed and human-impacted environments. Spatial maps could make many of these distributional patterns obvious, just as John Snow’s map of cholera instantly led to the hypothesis that this disease was water-borne and stemmed from the Broad Street pump, reinforced by the map’s revelation that the block that drank alcohol rather than pump water was spared from cholera (33). In an analogous manner, systematically collected maps of microbes and molecules across different spatial scales will fundamentally transform the types of questions that can be asked of microbiome and metabolomics data.

The need for abstract maps

Despite the intuitive appeal of spatial maps, the value of abstract maps, including ordinations such as principal coordinates analysis (PCoA), non-metric multidimensional scaling (NMDS), tdistributed stochastic neighbor embedding (t-SNE), and network diagrams from object similarity (sequence or spectrum) or co-occurrence, is also considerable. The correct data frame and distance metric often immediately reveal the key result, without a specific hypothesis in mind. Consider the starting and ending time point of a fecal transplantation series (34) (Fig. 2A), where the different clusters are obvious, but the direction and meaning of this difference are not. Placing these samples in the context of the Human Microbiome Project data (10) reveals immediately that the difference between start and endpoint is much greater than the difference between healthy and diseased samples, and adding intermediate timepoints shows that the transition occurs rapidly. The map thus enables new hypotheses about how to move individuals along a desired trajectory in the abstract space. A major driving force behind both the Earth Microbiome Project (8) and the American Gut Project (7) has been to build out these abstract maps for additional sample types and populations.

Figure 2.

Untangling the meaning of complex microbial interactions through meta-analyses. (A) Principal coordinates analysis (unweighted UniFrac) of Clostridium difficile Infection subjects, before and after a fecal transplant, along with the fecal donor and 10 untreated subjects (34). (B) Principal coordinates analysis (unweighted UniFrac) of the Human Microbiome Project (HMP) (10) combined with the data in panel A, the longitudinal samples for subjects 1–4 are connected as lines displaying the temporal variability and the shift from a disjointed untreated state of the patients vs. the healthy frame of the HMP. A high-resolution version can be found athttps://www.dropbox.com/sh/paq9sdiqvzp5mog/AABTUuRkZlHzlPbsN0riWIICa?dl=0.

An important question in building maps is often whether, given a fixed sequencing budget, it is better to have more points (samples), or more accurate or detailed characterization of each point. In our experience, for amplicon sequencing, having more samples outweighs the value of having more sequences per sample, down to surprisingly low thresholds. For example, Fig. 3 shows the Earth Microbiome Project dataset (8) sampled at 500,000 sequences per sample, 1000 sequences per sample, and just 200 sequences per sample. The overall patterns, e.g. the host/non-host split and the saline/non-saline split, are much clearer with more samples than with more precision about the location of each sample in PCoA space. Multinomial sampling considerations make it immediately clear why this is true: with 100 sequences per sample, the standard error in inferring the proportion of a taxon at 5% frequency is ~√(100*.95*.05) or 2.18%, or nearly 50% error in proportion; the standard error at a taxon at 1% frequency is ~√(100*.99*.01) or 0.99%, essentially 100% error. Consequently, even low-abundance taxa are sampled accurately enough to place a sample in a map with surprisingly few sequences. Logically, this must be true, or all ordination diagrams in microbial ecology before nextgeneration sequencing would have been useless.

Figure 3.

Broader sampling improves abstract maps of the microbial world in the Earth Microbiome Project, even with low resolution. All panels show principal coordinates analysis of unweighted UniFrac distances between samples. (A) Samples rarefied to 500,000 sequences, showing only those exceeding this threshold sampling depth. (B) Samples rarefied to 1000 sequences. (C) Samples rarefied to 200 sequences. Even with few observations per sample, the overall relationships among sample types are preserved; in contrast, the overall pattern is lost with too few samples no matter how exquisitely characterized.

The need for improved tools

Amplicon studies have been greatly enabled by improvements in processing pipelines, distance metrics, and reference databases, which we have recently reviewed elsewhere (35), greatly enabling hypothesis-driven studies about relative abundance of particular microbial taxa and their placement on abstract maps such as those produced by the Earth Microbiome Project (8) and the American Gut Project (7). As we extend these projects to other data types, notably shotgun metagenomics and metabolomics, we face new challenges that can best be solved by new tools. However, tool production is fundamentally itself more of an engineering than a hypothesis-driven activity, especially when the main advances are in user interfaces (32, 36) or in software engineering (37).

Although most 16S rRNA fragments can now be identified, in shotgun metagenomics only a small fraction of the sequences can typically be associated with known taxonomy or function. Genome assembly is especially valuable in identifying biosynthetic pathways, allowing taxonomic resolution at the species or strain level, and generating high-resolution single nucleotide polymorphism (SNP) profiles to characterize novel strains and confirm functional variants (38). Consequently, methods that can identify genetic variation from lower-coverage data, and that can estimate features of interest from less data or with efficient target capture, are needed for improved sample throughput. Shotgun metagenomics also often requires host DNA depletion because total DNA extracts from biological specimens can be dominated by host DNA (39).

Metabolomics poses different challenges (40). According to NHGRI, the cost per megabase of raw genome DNA sequence reduced in cost by almost six orders of magnitude since 2001(41), but mass spectrometry only decreased in cost by only two orders of magnitude during this period (40). However, the main limitation in metabolomics is the enormous chemical diversity, which hinders molecular identification and impacts the choice of extraction solvents, separation methods, instrumentation, and data analysis approaches. Because the multiplexing strategies that are successful in both amplicon- and shotgun-based sequencing approaches are not available in mass spectrometry, instrument time is directly proportional to the number of samples and limiting for large-scale projects. As with sequencing a decade ago, most molecular features found in a sample are unidentified, and many are likely technical artifacts, e.g. adducts formed in the gas phase, solvent artifacts (42) and multimers of the same compound (40). Better methods and incentives for aggregating community knowledge (24) (e.g. retention of knowledge of the large number of manual annotations performed by the community) and for automatically assigning unknown mass peaks and fragmentation spectra to molecules and have an estimation of error rates (43), as opposed to heuristics subject to personal interpretation rules (44), are urgently needed. Global Natural Products Social molecular networking (GNPS) (24) offers alternative solutions for computational mass spectrometry infrastructure. Spectral datasets can be publicly deposited with a unique identifier and transformed to “living data”, as they will be continuously searched against reference libraries to update users on new identifications. Furthermore, annotations can also be made by the scientific community within GNPS and propagated to all other data sets in the public domain with notifying subscribers on new annotations. Other expanded capabilities include automated species metabolome references (45) and the Molecular Explorer (24) for cross-searching annotated MS/MS spectra between datasets. Connections between several datasets, within the same knowledge base or between different spectral repositories such as Metabolights (46) and Metabolomics Workbench (47), can be made to highlight annotated compounds found in several data sets Such analysis is trivial in sequencing but still novel in mass spectrometry.

Integration of taxonomic, genomic and metabolomic data remains an important unsolved challenge. Although genome mining can successfully identify the sources of individual natural products (48), matching an overall taxonomic or functional microbiome profile to a molecular profile remains difficult because of procedural and analytical differences in data acquisition. In particular, the likelihood of time lags in chemical production or in genomic response to environmental changes, which may appear on different timescales, reduce the power of correlation approaches based on cross-sectional data (49). In cases where microbial and molecular composition is driven by a dominant effect (e.g. a dataset composed of soil and fecal samples will divide into two clusters driven by the difference between soil and feces), the molecular and metagenomic datasets will appear concordant by Procrustes analysis (50), but this is an artifact of the approach. An integrated systems biology approach that maps all data layers onto common pathways is likely needed, but cannot be performed today because most genes, pathways, and molecules are unknown and because even known system components lack coherent ontological conventions across databases.

The Need for Standards

Another branch of non-hypothesis-driven research critically important for framing precise hypotheses is standards development. In microbiome science these broadly take three tracks: analytical standards for determining the accuracy and fidelity of readouts, procedural standards for sample collection and handling, and annotation standards for integrating results across studies.

The lack of agreed-on standards stems from the origin of microbiome science in the discipline of ecology, where fundamental questions revolved around finding new kinds of organisms to fill out the phylogenetic tree of life (51), and finding statistically significant differences in microbial diversity or composition among samples within an individual study. Because the goal was to test whether any difference existed in the microbiome as a function of disease, physiological, or environmental state, biases (including missing taxa, or missing classes of molecules) were unimportant if a difference could be discovered. However, this situation diverges radically from the present, where physicians and engineers expect to be able to measure the correct, absolute abundance of all microbes or molecules in a given sample simultaneously. The realities of nucleic acid or organic extraction, detection methods for sequences and molecules, and downstream data processing do not support this important goal. Without consistent and welldefined measurements underpinned by a mechanistic causal model of error and bias, the state of microbiome-based predictions could be characterized as more like astrology than like astronomy, as pre-science rather than science.

To improve sample readout, we need known reference standards that can be spiked into samples at different stages, from original specimen to extracted DNA or compounds, that are agreed on, widely used, and have inexhaustible supply. Previous efforts, such as the HMP standards, have been limited by insufficient availability of materials, taxonomic complexity, or both. KatharoSeq in particular (52) benefits from having different spike-in standards at the level of primary sample and DNA, allowing different sources of contamination to be tracked down. Comparable development in mass spectrometry would be of tremendous value.

Sample collection and storage can bias specimen readout (53–55), but for most sample types the implications of different forms of degradation are unknown. Consequently, the conservative recommendation is always to expensively collect pristine samples (e.g. flash-frozen in liquid nitrogen), even though more practical methods would often suffice. For a few sample types, such as amplicon processing of stool, considerable data is now available on a range of conditions (55–58), and researchers can make informed decisions about which methods to use, as we did for citizen-scientists in the American Gut Project after exploring the limits of what can and cannot be usefully obtained for amplicon collection from stool shipped at room temperature (7). However, we know much less about the implications of sample degradation for most other types of biospecimens, and for the implications for reading out different molecular fractions with mass spectrometry (although see (59)).

Finally, integrating samples from different studies remains challenging because of differences in annotation (often called “metadata”). For example, different studies may refer to “stool”, “feces”, “gut”, or other synonyms, or rely on different units of measurement. Efforts such as the Genomic Standards Consortium MIxS family of standards (60), the Earth Microbiome Project Ontology (EMPO) (8), and other annotation schemes assist considerably in these tasks, but have been applied to relatively few datasets to date. The potential for natural language processing (NLP) and/or data-based methods for automatically applying annotations is considerable. These strategies were successful in Qiita for inferring EMPO annotations for tens of thousands of samples from the researcher-reported “sample_type.” However, further development is needed to enable researchers to “discover” variables and controlled vocabularies that can be generally applied.

Conclusions

Although hypothesis-driven science has immense value, it depends to a considerable degree on a framework of maps, tools, and standards whose own development often does not fit meaningfully into a hypothesis-driven framework. However, without these developments, hypotheses more explicit than “differences in the microbiome” or “elevation or depletion of specific pre-defineid taxa or molecules” cannot be tested, and completely new ideas about how to read out or control the microbiome will not be developed.

Extraordinary advances in data collection technologies leave us in a world where we regularly make make millions of observations of organisms about which we know virtually nothing -- as exemplified by the recent ‘discovery’ of the most abundant phage in the human gut via metagenome mining (61). To bring about a future of precision medicine and precision ecological remediation, where we can specify precise microbiome changes and bring them about through defined interventions, a vast amount of non-hypothesis-driven research, often dismissed as “technical work” or “fishing expeditions”, remains to be done.