Modern microbiology: embracing complexity through integration across scales

Abstract

Microbes were the only form of life on Earth for most of its history, and they still account for the vast majority of life’s diversity. They convert rocks to soil, produce much of the oxygen we breathe, remediate our sewage, and sustain agriculture. Microbes are vital to planetary health as they maintain biogeochemical cycles that produce and consume major greenhouse gasses and support large food webs. Modern microbiologists analyze nucleic acids, proteins, and metabolites, leverage sophisticated genetic tools, software and bioinformatic algorithms and process and integrate complex and heterogeneous datasets so that microbial systems may be harnessed to address contemporary challenges in health, the environment and basic science. Here we consider an inevitably incomplete list of emergent themes in our discipline and highlight those that we recognize as the archetypes of its modern era that aim to address the most pressing problems of the 21st century.

Introduction

Our planet faces formidable challenges that span from climate change, environmental pollution, depletion of fertile soils, and loss of biodiversity to feeding and caring for the ever-growing human populations in a sustainable fashion. Microbes, whose evolution has been intimately interconnected with changes in Earth’s physicochemical environments over biological time, remain an integral part of these interlinked crises. With its aim to understand microbial life, and given the potential to harness microbes to address these huge problems, the field of microbiology is emerging as arguably the most important science of the mid-21st century.

Microbiology has made remarkable progress throughout its relatively brief few hundred-year history. The advances seem far from reaching a plateau, as we have likely just begun to scratch the surface of major biotechnological and medical advances that are enabled by studies of microbial life. Today, microbiologists aim to elucidate and ultimately address a wide range of processes that include greenhouse gas emissions and soil carbon sinks to wastewater management, protection of clean water resources, detoxification of heavy-metal and organic pollutants, maintenance of soil fertility and rhizosphere nutrition, the emergence and mitigation of plant and animal pathogens, antibiotic resistance, pandemics. Investigations to address such diverse questions often rely on a broad range of data producing and processing technologies, which demand increasingly higher levels of integration and synthesis for unified insights.

Microbiology draws its power from relatively well-established technologies such as molecular sequencers, mass spectrometers, high-throughput cultivation, genetic manipulation and a large arsenal of ‘omics strategies, statistics, and modeling, as well as technologies that are rapidly emerging, such as artificial intelligence, machine learning, microbiome-based genome editing, and more. As the contemporary questions of microbiology demand the orchestration of diverse data generation and analysis strategies that yield increasingly large datasets, integrative approaches that span across scales of complexity emerges as one of the hallmarks of modern microbiology. Here we take a look at the past and contemplate the future of our rapidly transforming discipline by starting with a discussion on some of the major themes that underpin the new data-intensive epoch of microbiology.

Modern microbiology: from individual microbes to communities

It is a daunting task to precisely define what modern microbiology is, but a useful one to reminisce how we got to where we are, and to recognize new motivations, opportunities, and challenges that influence today’s microbiologists. If we could explain how we study microbes today and what we know about them to Antonie van Leeuwenhoek, who saw microbes the first time in 1670s, to Ignaz Semmelweis, who realized washing hands reduces patient mortality in hospitals in 1840s, or to Fanny Hesse, who established agar to grow microbes in laboratory settings in 1880s – what would they find to be the most fascinating advance microbiology has achieved compared to its early days? We indeed would have had a myriad of surprises to show them proudly, as we now can isolate microbes in nanodroplets¹, image them in situ with colorful probes² and in aqua in three dimensions³, survey metabolites they exchange⁴, visualize and model their metabolic activities both at the level of individual elements⁵ or at planetary scales⁶, pull them around from their surface proteins⁷, stick them into little wells⁸, and even modify⁹, transform¹⁰, or synthesize¹¹ their genetic repertoire, or transfer their genes into the genomes of plants¹² and animals¹³. But apart from the permanently central role of technology in microbiology, there is one conceptual fracture in our otherwise continuous progress towards a better understanding of microbial life which can serve as a line between the still alive and well classical roots of microbiology and its aspiring modern times: our expanding focus from individual organisms in cultures to complex communities in natural habitats, with the admission that ‘a pure culture is not enough’¹⁴.

The shift from individuals to communities in microbiology was enabled by a series of technological and conceptual advances: it was foreshadowed by the theory of evolution, and guided by many breakthroughs such as the discovery of the structure of DNA¹⁵, the recognition of the utility of the molecular information encoded in genomes as a means to document evolutionary history¹⁶, and the arrival of sequencing¹⁷. One of the latest of the most defining conceptual advances that culminated the thread of progress into a single evolutionary framework that connected all cellular life was the work led by Carl Woese, which offered a first glimpse of the tree of life, revealed by comparison of ribosomal RNA gene sequences in 1977¹⁸. The translation of this transformative idea to practical applications in microbiology started to emerge through the pioneering work by Norman Pace and colleagues that described rapid sequencing of ribosomal RNA (rRNA) gene fragments¹⁹ and cloning methods that enabled gene recovery from environmental samples²⁰, which gained further traction with simpler molecular strategies that require lower input biomass²¹. In the following years, PCR-amplification of 16S and 18S rRNA gene fragments became an increasingly popular method to survey microbial diversity in naturally occurring habitats^22,23. Coinciding with increasing availability and affordability of short read sequencing of early 2000s, a plethora of amplicon studies revealed the richness and dynamism of microbial communities from marine²⁴ and terrestrial systems²⁵ to host associated habitats²⁶, and led to a broad recognition and awareness of Earth’s flourishing microbiomes and their connection to the rest of life²⁷. The transition from individuals to whole microbiomes motivated a societal level adjustment of our perspective: microbes were no longer a band of bad actors in the public mind that typically caused diseases, but the architects of life, the wardens of planetary health, and the fundamental forces that make our ecosystems tick.

Surveys of microbial diversity through rRNA gene amplicons was a wildfire in microbiology and resulted in over 85,000 studies in the past 20 years. Yet, taxonomic insights offer limited utility to understand functional drivers of biological systems, a pinnacle desire that brings together many corners of microbiology. Despite the inspiration to better understand microbial systems, the integrative power of amplicon surveys remained severely constrained. Understanding of microbes in their environmental context required genomes, which first became available for isolates that were generated by indispensable yet labor intensive isolation efforts²⁸. However, isolation is inherently biased towards microbes that can grow alone and for which suitable growth conditions and substrates are known, and has not delivered even a single representative of the majority of currently known lineages²⁹.

The realization of the need for direct access to the genetic makeup of environmental populations intensified during the late 1990s. In pursuit of natural products potentially available from the untapped diversity of microbes in the environment, Handelsman and colleagues introduced a cloning-based approach to access genes and pathways of interest from soil ‘metagenome’³⁰. The early approaches to metagenomics, which promoted the idea of cultivation-independent surveys of environmental populations, relied upon “isolating DNA from an environmental sample, cloning the DNA into a suitable vector, transforming the clones into a host bacterium, and screening the resulting transformants”¹⁴. The arduous beginnings of metagenomics offered unique insights into environmental microbes^31,32, and demonstrated the feasibility of culture-independent surveys of natural populations. An alternative approach to metagenomics relied upon the sequencing of randomly chosen short DNA fragments³³. Combined with the advances in sequencing chemistry and molecular approaches, this ‘shotgun sequencing’ approach replaced cloning-based strategies, and ultimately enabled the reconstruction of genomes directly from natural ecosystems.

The core idea behind reconstructing genomes from environmental metagenomes, which was laid out in the first example of its kind, stood on a relatively simple principle that relied on a few imperfect but computationally tractable steps³⁴: assembling metagenomic short reads into longer contiguous DNA segments (contigs), estimating the abundance of contigs in metagenomes, and partitioning contigs into separate genome bins by exploiting differences in GC content and abundance of each discrete population. The power of genome-resolved metagenomics came from the fact that no aspect of it required any prior knowledge about organisms in a given environment. Its simplicity allowed this strategy to be encapsulated in automated workflows, from which emerged a plethora of genomes, not only from taxa microbiology knew about but also from those we had not recognized before, and not only from the primary domains of life but also from viruses, plasmids, and other exciting biological entities. Over time, the size of sequencing datasets increased, along with the power of bioinformatics tools, enabling the transition of genome-resolved surveys from relatively low complexity systems to the microbiologically most complex ecosystems on Earth, such as oceans and soils. Metagenome-assembled genomes (MAGs), along with single-amplified genomes (SAGs) and those that emerged from improved isolation efforts, opened the floodgates of genomic data pouring in from all environments. Today, the rapidly maturing evolutionary framework around genomes offers a playground for all corners of our discipline to integrate heterogeneous data types that slice through microbial systems in complementary ways through analytical, computational, and molecular tools to address fundamental questions of modern microbiology (Figure 1).

Figure 1.

The measurement tools, data types, and analysis strategies commonly used in modern microbiology to study naturally occurring microbial life in a wide range of complex habitats through rapidly emerging genomic blueprints of the diversity of life.

From communities back to individual microbes

Cultivation, the most fundamental methodological advance that shaped classical microbiology by enabling molecular insights with broad applications through manipulation of representative microbial organisms, continues to be the bedrock of contemporary studies of microbial physiology in the post-omics era³⁵. The ability to reconstruct microbial genomes directly from the environment represents an effective means to improve cultivation efforts as it can provide clues regarding metabolic requirements of little known organisms, and guide their directed isolation³⁶. The first demonstration of this idea was the isolation of Leptospirillum ferrodiazotrophum³⁷ from a relatively simple subsurface acid mine drainage biofilm following the observation that it was the only organism in its community capable of nitrogen fixation. The first successful isolation of Coxiella burnetii³⁸, the causative agent of human Q fever³⁹, represents another example of how partial insights into genomic features of target organisms can help define complex media that support the axenic growth of even obligate intracellular organisms. Peering into the metabolic make up of environmental communities through metagenome-assembled genomes continues to be a powerful tool to support cultivation efforts even from complex mixtures of microbes. For instance, Pope et al. were able to isolate a representative of the family Succinivibrionaceae, so called WG-1, that dampens methane emissions from digestive activities of macropods using a composite genome reconstructed directly from a metagenome. Metabolic insights gained from this genome supported the development of a medium with specific carbohydrate and nitrogen sources as well as antibiotics to achieve an axenic culture⁴⁰. Others applied similar principles to grow microbes from other environments, including human gut⁴¹ and even extreme subsurface environments⁴². Prediction of the episymbiotic lifestyles of Candidate Phyla Radiation (CPR) bacteria from environmentally-derived genomes⁴³ led to the co-isolation of these tiny bacteria with their hosts⁴⁴, and opened the way for microscopic⁴⁵, metabolic⁴⁶, transcriptomic⁴⁷ definitions of the impact of these symbioses, and their interactions with their hosts⁴⁸. An exciting workflow that integrated the use of environmental genomes, host immunity, and fluorescent labeling for targeted cultivation of microbes was demonstrated by Cross et al., who cultivated multiple CPR bacteria, including the elusive Absconditabacteria (SR1)⁷. Cross et al first surveyed single-amplified and metagenome-assembled genomes of target bacteria to identify membrane proteins that could serve as immunogens, then predicted extracellular domains in these candidate genes with homology to previously recognized immunogens, synthesized matching peptides, introduced them into rabbits, recovered antibodies generated by the rabbit immune system, used these antibodies to ‘label’ bacterial cells in a human saliva sample, and were able to sort labeled cells into cultures⁷.

In addition to culture-independent insights that improve cultivation efforts and new approaches that aim to preserve natural growth conditions of environmental microbes^49,50 or their interactions⁵¹, modern approaches to cultivation include high-throughput strategies that exploit robotics and microfluidics. Recent studies demonstrate the utility of microfluidics to generate thousands of nanoliter droplets on petri dishes with chemical gradients⁵², and the application of robotics and machine learning to process thousands of colonies per hour⁵³. One critical application of microfluidics is targeted cultivation of microorganisms based on any genetic feature that is learned from metagenomes. Using this strategy, Ma et al. were able to identify a microcolony that carried their target gene sequence among 500 microcolonies they grew on a multiplexed microfluidic device, and recovered the first isolate of a previously unidentified genus of the Ruminococcaceae family⁵⁴ that has been among the most abundant and prevalent taxa in the human gut with no cultured representatives until then⁵⁵. Another exciting application of microfluidics is the untargeted, high-throughput cultivation of microorganisms in droplets. The isolation of individual bacterial cells in droplets reduces the loss of microbial colonies in cultivation efforts due to resource competition, differences in growth rates, or dilution of quorum sensing molecules⁵⁶. In an application of this approach, Watterson et al. populated millions of picoliter droplets created from distinct culture media with individual microbial cells from a single sample¹. Droplets they sorted based on colony density included many of the rare organisms conventional cultivation strategies missed in the same sample, and recovered antibiotic resistant strains that conventional plate-based assessments of antibiotic resistance were unable to detect¹.

Dramatic expansion of the tree of life

Our newly found power to sequence genes from environmental samples had far reaching implications on our understanding of the diversity and evolution of life through phylogenomics and resulted in remarkable changes to the structure of the tree of life. The sequencing of marker genes such as rRNAs had revealed many, but not all, branches of life, but inconsistencies in predicted physiology in major branches and unexpected gene inventories only became evident upon recovery of genomes. Notable features of the revised and genome-informed tree of life include the Asgard archaea with their inventories of eukaryote-associated genes⁵⁷, making them of great interest from the perspective of the origin of Eukaryotes⁵⁸. Other major groups of particular interest include the possibly not monophyletic DPANN archaea and the seemingly monophyletic CPR, which display consistently small genomes and limited biosynthetic capacities. This led to the prediction that both the bacterial and archaeal domains of life feature large groups of microbes that they generally depend on host microorganisms for basic building blocks such as lipids and nucleotides⁵⁹, and recent work has clarified the partitioning of host-derived lipids into some episymbionts⁶⁰. Inclusion of sequences from metagenomes as well as isolates brought into focus the dramatic prevalence of branches of the tree that remain without any isolated representatives²⁹. Further refinements have begun to clarify relationships between deep branches, for example the Chloroflexi and CPR, and the distribution of traits⁶¹. The revised views of the bacteria and archaeal domains as well as the whole Tree of Life contrast dramatically with even contemporary views of the diversity of life that are highly eukaryote-centric and in some cases, don’t even include non-eukaryotic organisms. These renderings enabled a new appreciation of our place in biological evolution and biodiversity.

The value of complete and finished genomes is hard to over emphasize. In addition to providing a quality assured inventory of genes and metabolic potential without contamination by sequences from other sources, finished genomes ensure full genomic context for any genetic feature and define the overall genome architecture, including exact genome size and replichore structure. However, one of the hallmarks of the contemporary catalogs of metagenome-assembled genomes is the low completion and occasional contamination issues due to the highly fragmented nature of metagenomic assemblies⁶². While some of these concerns can be addressed by manual^62,63 or automated genome curation and bin refinement approaches⁶⁴, long-read sequencing technologies are already enabling the reconstruction of complete genomes without binning, and this will soon be possible at scale. Products of long-read sequencing can be error prone, and efforts to automate error correction and to improve the accuracy of long-reads are underway (e.g.,^65–67). The number of assembly algorithms that can report circularized sequences from metagenomes is also increasing^68–70. However, assembled long-reads are not without issues, and the resulting sequences can suffer from false circularization due to artifactual repeated sequences and misassemblies that are not supported by the original reads. High-throughput methods for detection and correction of assembly errors will ultimately bring microbiology closer to the ideal of routinely rendering accurate and complete genomes and increase the utility of genomes from environments for downstream applications in microbiology.

Expanding and exploring the protein universe

Metagenomic sequencing can uncover new protein families and whole proteome comparisons (including proteins with unknown functions) can establish relatedness amongst large groups of organisms⁷¹, including those that lack universal genes, such as extrachromosomal elements⁷². Despite the lack of recognizable functional domains, the sequences themselves can provide information about physicochemical properties of the protein (e.g., hydrophobic domains or positively charged regions that may be involved in nucleic acid binding) and uncover metal and ligand binding sites. A new generation of computational methods such as AlphaFold⁷³, RoseTTAFold⁷⁴, and ESMFold⁷⁵ enable prediction of the three-dimensional structures of proteins, thus can provide functional clues that are not apparent based on the sequence alone. Often, predictions leverage templates in the form of the structures of biochemically validated proteins accessed via large protein databases such as the Protein Data Bank (PDB)⁷⁶. However, it is possible to improve these predictions by supplementing the underlying multi-sequence protein alignments using the ever expanding database of amino acid sequences derived from metagenomic studies⁷⁷. In fact, using sequences independent of templates it is possible to discover new protein folds and vastly increase access to three dimensional structures for proteins with no related solved structures or sequence similarity to known proteins⁷⁸. These approaches, in combination with ecological and other information, have the potential to advance understanding the protein inventories of enigmatic entities such as viruses⁷⁹ and aid in the development of biotechnology tools⁸⁰. Exciting new bilingual models that translate information between structural and sequence space⁸¹, and computational tools that increase the accessibility of protein structure calculations⁸², and searchability of existing structures^83,84 in large databases with millions of predictions⁷⁵ help locating variants of proteins of interest with potentially different size or kinetics. It may be possible to revolutionize drug development given methods that predict molecular interactions and discover small molecules or antibodies that bind to proteins identified as drug targets⁸⁵. Indeed, computational approaches continue to improve at a rapid pace to predict a broad range of biomolecular interactions⁷³ and the stoichiometry of large protein complexes⁸⁶. Altogether, the ongoing revolution in protein structure space and new opportunities to integrate structural insights with established ‘omics data types and analysis strategies will undoubtedly influence every corner of microbiology.

Going beyond the tree of life – (giant) viruses, (giant) plasmids, and other extrachromosomal entities

With their astronomical numbers and complexity, viruses represent a force of nature that is implicated in all aspects of life, starting from its very origins^87,88. The immense impact of viruses on life continue through metabolic reprogramming of their hosts^89,90, biogeochemical and ecological processes they influence in complex habitats⁹¹, and even the modifications of the reproductive behavior of animals⁹².

Contemporary virus research spans from their characterization and organization into cohesive units to linking them to particular hosts or ecosystems and investigating their biotechnological applications. The emergence of computational tools to recognize viral sequences in metagenomic assemblies^93–95 resulted in the recovery of tens of thousands of new viral genomes from the human gut⁹⁶, oceans⁹⁷, and soils⁹⁸. The increasing access to metatranscriptomics further supports the recovery of RNA viruses⁹⁹, and enables spatiotemporal tracking of eukaryotic viruses of public health significance and their variants¹⁰⁰. Microbiologists were surprised when ‘giant eukaryotic viruses’, whose genome sizes surpass those of some bacteria¹⁰¹, were discovered¹⁰². Genome-resolved metagenomics had a remarkable impact also on our rapidly emerging understanding of the diversity and biogeography of giant viruses¹⁰³, their complex metabolic capabilities¹⁰⁴, host reprogramming¹⁰⁵, and impact on host genome evolution¹⁰⁶. Despite the progress towards a global taxonomic framework for viruses^107,108, the luxury to be represented in a universal tree of life continues to be limited to cellular organisms. However, at least at the level of individual studies, the integrated use of metagenomic assemblies or genome-resolved metagenomics, phylogenomics, and protein structure prediction offers promising insights, not only to achieve a more complete representation of viral clades but also fill the gaps between disconnected realms of viruses. For instance, in a recent study Gaïa et al. used phylogeny-guided genome-resolved metagenomics using DNA-dependent RNA polymerase subunits to describe a large clade of viruses, and by combining functional features and predicted protein structures they were able to reveal a first association between giant viruses that infect unicellular marine eukaryotes and herpesviruses that infect mammals¹⁰⁹. In addition to their prominent place in basic science, viruses continue to be a powerful driver of biotechnology for innumerable reasons, including therapeutic opportunities¹¹⁰, the potential of their proteins as small genome editors¹¹¹ and their ability to deliver DNA (including editing tools) into the cells in which they replicate^112–114. They also force us to question the classical definitions of what is a bona fide living entity¹¹⁵ with opposing views¹¹⁶, yet continuously elucidate the conceptual frameworks that can offer formal organizations of the realms of biology¹¹⁷.

Plasmids, which were first defined by Joshua Lederberg in 1952¹¹⁸ and have served as a workhorse of molecular biology and genetics¹¹⁹, represent another exciting corner of microbiology. The significance of plasmids parallels that of viruses: plasmids also impact the ecology and evolution of their hosts¹²⁰, often find themselves under the spotlight of public health concerns such as global antibiotic resistance¹²¹ largely due to their contribution to the proliferation of such genetic factors¹²², encode genes that are associated with a wide range of metabolic activities¹²³, and suffer from a similar exclusion from trees of life. It is conceivable that these similarities may simply be an echo of deeper evolutionary connections that are no longer present in contemporary sequence space. Indeed, recent characterizations of ‘virus-plasmids’ that can act both as viruses and plasmids¹²⁴, observations of plasmids that propagate through virus-like particles¹²⁵, or phylogenetic analyses that attribute the emergence of DNA viruses to recombination events between RNA viruses and plasmids¹²⁶ reinforce the likely presence of deeper evolutionary links and hint that there is likely much more to discover in this junction. Also similar to viruses, the diversity of naturally occurring plasmids has been grossly underestimated. While metagenomic sequencing and assemblies access naturally occurring plasmids, recognizing plasmids in sequence collections remains a challenge. Some plasmids, even for some of the most rigorously studied microbial taxa such as Wolbachia, remained undetected for decades¹²⁷. But our ability to recognize plasmids is improving thanks to computational strategies that exploit k-mer profiles learned from reference plasmids¹²⁸, conserved plasmid functions¹²⁹, circularity of sequences in metagenomes¹³⁰, or deep learning approaches that use combinations of these features¹³¹. Today we know that even for relatively well studied ecosystems, plasmids found exclusively in metagenomes exceed the number of previously known plasmids multiple folds¹³². An interesting insight comes from a recent study that uses 68,000 non-redundant plasmids that were identified from human gut metagenomes and shows that plasmid distribution patterns is not correlated with bacterial taxonomy¹²³, which reinforces the critical importance of treating them as semi-independent entities that represent an adaptive force in microbial systems¹³³, similar to viruses. Yet organizing plasmids into evolutionarily cohesive units is perhaps an even bigger challenge compared to their identification. Unlike viruses, where relatively stable genes within distinct viral realms can yield taxonomic frameworks to shed some light on evolutionary routes of individual viral genomes¹³⁴, attempts to find cohesive units in diverse sets of plasmids have been largely limited to partitioning pairwise sequence similarity networks^135,136. A recent attempt to identify evolutionarily cohesive ‘plasmid systems’ in such networks through a containment-aware partitioning approach demonstrated a means to identify plasmid backbones and cargo genes, and partition plasmid genes into housekeeping genes and those that respond to particular environmental conditions de novo¹²³. Plasmids are a pillar of biopharmaceutical advances¹¹⁹, and naturally occurring plasmids represent a new asset. But the world of plasmids is chaotic: some can be so extremely essential and integrated into the host cellular physiology that they compelled researchers to call them ‘chromids’¹³⁷, while others can look so small and irrelevant even though they can be some of the most numerous genetic elements in natural habitats¹³⁸. With their unruly nature, plasmids remain an intimidating and exciting dimension of microbiology.

A significant challenge that is generally true for most extrachromosomal elements that are not typically found integrated into host genomes is the difficulty of linking them to their host organisms, which can be a significant burden in complex environments. By preserving the unity of naturally occurring genetic elements within cells that are typically lost in bulk sequencing efforts, single-cell genomics approaches offer somewhat straightforward means to capture virus-host¹³⁹ and plasmid-host¹⁴⁰ associations with high confidence. In some cases it is possible to leverage the fact that bacteria and archaea incorporate snippets of their extrachromosomal element genomes into their CRISPR loci (spacers) so that invasive elements can be recognized by the protein-RNA-based surveillance system. Extrachromosomal elements whose genomes match to CRISPR spacer sequences, especially from the same or related samples, are likely the targets of that microbial immune system, thus indicating ECE-host relationships¹⁴¹. Because CRISPR loci are often very fast evolving, the effectiveness of this method can be improved when spacer sequences are recovered directly from short metagenomic reads rather than from the assembled consensus sequence¹⁴¹. Another solution to identify hosts for extrachromosomal entities in complex samples comes from Hi-C¹⁴², a molecular strategy that can physically link genetic entities that are adjacent to one another prior to sequencing, and can reveal associations between host genomes and extrachromosomal entities, including plasmids¹⁴³ and viruses^144,145. Finally, heterogeneity in population genomic data can identify fast evolutionary modes and uncover the exact integrations sites for prophages and proviruses, simultaneously establishing host association for these elements^146,147.

Overall, our insights into extrachromosomal elements are improving at a rapid pace, and cover more than historically well-recognized ones such as viruses and plasmids. The recent description of ‘Borgs’, which can have over 1 Mbp linear genomes, contain large inventories of metabolically relevant genes, and are not simply classifiable as plasmids or viruses¹⁴⁸, ‘obelisks’, which are ~1 kbp viroid-like extrachromosomal circular RNAs that are prevalent in gut metagenomes and metatranscriptomes¹⁴⁹, the recent expansion of ‘viroid-like circular RNAs’¹⁵⁰, the ‘phage-inducible chromosomal island-like elements’ that demonstrate the extent of parasitism among mobile genetic elements¹⁵¹ and the overall prevalence of poorly characterized phage satellites across bacterial genomes¹⁵² indicate that there is more to be learned about these mysterious entities.

Discovery and validation of new genetic codes

Metagenomic sequencing has brought to light unexpected phenomena that present new dimensions of modern microbiology. One of these relates to the genetic code and its variability. The genetic code was long considered immutable and, indeed, it is widely conserved in bacteria and archaea. However, partial sequencing of a ribosomal gene cluster in the Mycoplasma capricolum genome revealed that, as in some mitochondrial genomes, TGA (normally a stop codon) is read as tryptophan (genetic code 4)¹⁵³. Subsequently, metagenomics-based analyses indicated that Gracilibacteria and Absconditabacteria, sibling lineages within the CPR, have reassigned the TGA codon⁴³. Translation as glycine¹⁵⁴, confirmed using metaproteomics¹⁵⁵, ushered in a third bacterial genetic code (Code 25). These code variations appear to have arisen in single, rare and ancient evolutionary events. To date, no analogous genome-wide codon reassignments have been reported in archaea, although this domain of life remains under-explored relative to bacteria and eukaryotes. Recently, a new computational approach, ‘Codetta’, was developed to identify new instances of codon reassignment in the now vast genomic sequence databases¹⁵⁶. Analyses revealed the first evidence for changes in the amino acid encoded by sense codon changes in bacteria. In combination, these studies broaden and deepen our understanding of the modes and frequency of genetic code evolution in prokaryotes.

The origin of the relatively (but with exceptions) stable modern genetic code has been investigated recently by Žagrović and colleagues, who found that, although mRNAs and their cognate proteins are different types of polymers, mRNAs and the proteins they encode display complementarity¹⁵⁷. This is particularly evident if the molecules are unstructured (disordered), as would have been the case during evolution of the primordial genetic code. In other work potentially relevant to understanding of the origin of the genetic code, analysis of large sequence databases from across the tree of life and confirmed that frame shifted proteins have similar physicochemical properties (e.g., hydrophobicity, nucleobase affinity and intrinsic disorder) to the original, non-frame shifted proteins, despite essentially no primary sequence similarity¹⁵⁸. It has long been known that similar codons encode amino acids with similar physicochemical properties and that the second codon position is most important for determining the properties of the amino acid encoded¹⁵⁹. Thus, frame shifted codons, which share two of three nucleotides, tend to incorporate related amino acids. Further insights into the evolution of genetic code come from integrated analyses of genomes and environmental selective pressures quantified through metagenomes. One such investigation by Shenhav and Zeevi shows that the genetic code optimizes codon assignments that minimize the impact of DNA mutations to dramatically change nutrient requirements of protein synthesis, revealing an overarching principle across the tree of life that enables relatively less risky exploration of the mutational landscape¹⁶⁰.

Findings related to genetic codes of bacteriophages (phages) challenge the expectation that phages are unlikely to use a genetic code that differs from that of their hosts and the perception that genetic code changes are rare evolutionary events. In 2008, Shackelton and Holmes wrote “… it is extremely unlikely that viruses of hosts utilizing the universal genetic code would emerge, via cross-species transmission, in hosts utilizing alternative codes, and vice versa”¹⁶¹. Consistent with this prediction, these authors confirmed that alternatively coded phages replicate in hosts that also use an alternative genetic code. However, Ivanova et al. reported that some phage genomes do in fact use a genetic code in which one of the three normal stop codons is read as an amino acid yet they replicate in hosts that use the standard genetic code¹⁶². Recently, the reverse was suggested: standard code phages can replicate in bacteria that use a modified genetic code, specifically, Gracilibacteria and Absconditabacteria, whose genomes that encode TGA as glycine¹⁶³. Studies now show that code mismatch between phages and hosts is a relatively common phenomenon. For example, diverse alternatively coded phages replicate in Bacteroidetes and Firmicutes, bacteria that are particularly prevalent in human and animal microbiomes¹⁶⁴. Combined taxonomic and genetic code analysis of a metagenomically-derived database of alternatively coded phages revealed that these alternatively coded phages can be closely related to phages that use the standard code. In the most surprising case, one phage lineage features standard code, TGA-reassigned and TAG reassigned genomes. Thus, contrary to evidence from prokaryote genomes, results for phages suggest that the barrier to code change is relatively low, thus codon reassignment can arise independently in different lineages, and over relatively short evolutionary timescales¹⁶⁴.

Indications that code shifts in phages are relatively facile raises the questions of the evolutionary path to code change and the machinery required for different codes to be interpreted by the ribosome during phage infection. Acquisition of a suppressor tRNA that translates a normal stop codon as an amino acid could occur via lateral gene transfer or, more likely, via mutation of the anticodon of a tRNA (possibly preceded by duplication of the tRNA¹⁶⁵. Interestingly, to date, analyses of metagenome-derived phage genomes suggest charging of suppressor tRNAs by one of only three amino acids, suggesting convergent paths to these codon reassignments. The advantages of code mismatch remain unclear, but may prevent premature production of structural proteins and proteins involved in host cell lysis¹⁶⁴.

Another form of genetic code variation involves the reassignment of normal stop codons to non-standard amino acids. The first recognized example of this was the translation of TGA codon to selenocysteine in specific proteins, as signaled by a SECIS element, a specific structural mRNA feature¹⁶⁶. This can substantially modify the properties (e.g., stronger nucleophilic ability and lower reduction potential) of proteins relative to the cysteine-containing homologs¹⁶⁷. More recently, reassignment of the TAG stop to the 22nd amino acid, pyrrolysine, was discovered, and, to date, found to be prevalent in methylamine methyltransferases and a few other proteins¹⁶⁸.

Mapping biogeography to genes and genomes

Genomes have been reconstructed directly from metagenomes sampled from every major ecosystem, including microbiomes of humans, animals, insects, plants, soils, lakes, aquifers and oceans. The large influx of genomes had an immediate impact on life sciences through broader access to predicted genes, functions, and metabolic potential of naturally occurring organisms. Surveys of traits that define the biogeographical patterns and niche partitioning of plants and animals go as far back as the eighteenth century in biology. However, a broad appreciation of the fact that microbes also display non-uniform biogeographical distribution patterns as a function of environmental gradients took time to come into focus¹⁶⁹. Some of the most defining work emerged from the studies of microbial mats in the Yellowstone National Park, United States. Initially using denaturing gradient gel electrophoresis to separate 16S rRNA gene fragments amplified from the environment, Ward and colleagues showed the astonishingly clear patterns of microbial population biogeography within one-millimeter slices of Mushroom Spring cyanobacterial mats¹⁷⁰. These observations at the micron scale were confirmed by others at the scale of meters, when Moore et al. showed that physiologically and phylogenetically distinct Prochlorococcus populations distribute across the water column as a function of light and temperature¹⁷¹, and across continents, or when Cho and Tiedje showed that genotypic differences between Pseudomonas populations correlated with geographic distances between soil samples from which they originated¹⁷². With the recognition of the impact of microbial traits, their ecology, and impact on biogeochemical systems¹⁷³, the next challenge was to go from phylogenetic indicators of biogeography to traits that drive biogeographical partitioning of microbes¹⁷⁴. The initial gene-centric analyses of short metagenomic reads¹⁷⁵ and longer genomic fragments recovered directly from environmental samples¹⁷⁶ quickly revealed non-uniform distribution patterns for genetic features that were associated with environment-specific metabolic requirements. The ability to characterize biogeography of microbial populations along with their genes was a step towards one of the holy grails of microbiology with significant biotechnological and biomedical implications: elucidating genetic determinants of fitness encoded in genomes, which benefited from ecology-enabled comparative genomics approaches that soon followed.

Characterizing genetic heterogeneity in microbial populations

The first comparison of complete genomes of two closely related Helicobacter pylori strains revealed a large extent of gene conservancy and systematic differences¹⁷⁷ and the existence of ‘core’ and ‘flexible’ gene pools¹⁷⁸. Research in the early 2000s took advantage of the increasing access to microbial genomes to reveal the utility of comparative genomics to gain insights into the likely determinants of microbial colonization and virulence¹⁷⁹, or develop vaccines¹⁸⁰. The idea to systematically study differentially occurring genetic features of closely related microbial populations came into fruition with the recognition of the microbial pangenomes¹⁸¹ and evolutionary processes that shape them^182,183. By partitioning the gene pool of a set of genomes, pangenomics offers insights into genes that are found in the vast majority of genomes as well as those that differentially occur between them, which is an immense power to link gene function to population ecology.

Genomes can tell very little about their biogeography, and even less about how well their gene content represents the gene content of environmental populations to which they belong. Pangenomics combined with shotgun metagenomes can uncover not only the biogeography of individual populations, but also gene conservancy across habitats¹⁸⁴. An application of this idea by Coleman and Chisholm to a Prochlorococcus pangenome and two marine metagenomes reveal that the scarcity of phosphorus in the North Atlantic Ocean resulted in a strict maintenance of accessory phosphorus acquisition genes for Prochlorococcus populations in the North Atlantic while their close relatives in the Pacific ocean had largely lost them¹⁸⁵. This integrated approach to pangenomics and metagenomics to identify selective forces that shape gene variation across populations and environments, also termed as metapangenomics¹⁸⁶, has revealed differential enrichment of near-identical Haemophilus parainfluenzae populations that are millimeters apart from each other in human oral cavity, where populations enriched in tongue commonly encoded sodium-dependent oxaloacetate decarboxylase enzymes, and those enriched in dental plaque completely lacked them¹⁸⁷, and differential distribution of transporters and enzymes within marine SAR324 populations to balance autotrophic and heterotrophic lifestyles as a function of water depth¹⁸⁸. The metapangenomic approach can also be used to determine whether critical genes validated experimentally in cultures are conserved in environmental populations¹⁸⁹.

The inclusion of ecological patterns into comparative genomics reduces the number of targets in our search for genes that may be particularly important under particular environmental conditions. However, difficulty with de novo prediction of gene function poses challenges. Efforts to shed light on the ecology of unknown genes^190,191, profile hidden Markov model search algorithms¹⁹², application of deep learning to distant homology searches¹⁹³, algorithms that enable rapid mining for protein structures⁸³, and bringing in graph theory into pangenomes to infer synteny of genes^194,195 will improve insights into microbial gene pool.

Disentangling targets of long-term evolutionary processes

Even the genes that are shared across all members of a population will vary across individuals in subtle ways. Since the early descriptions of within-population trait variations in animals and plants by Darwin and Mendel, population genetics, studies of genetic variation among individual members of a population, has had a remarkable history in biology revealing molecular principles of evolution¹⁹⁶, and rapidly gained attention in microbiology with the increasing number of genomes and metagenomes. Even with limited access to sequencing, it was becoming clear during early 1990s that microbial organisms were not strictly clonal¹⁹⁷, and this signal was essential to study fundamental principles of genetic innovation and divergence¹⁹⁸ as the microbial world is a true playground for evolution¹⁹⁹, which enables them to respond to environmental change within seasonal timescales²⁰⁰. Metagenomic datasets capture genomic heterogeneity of closely related populations and provide the data needed to identify genes under neutral, adaptive, or purifying forces. Indeed, the initial culture-independent observations of fine-scale population structures through microdiversity patterns found in clone libraries from marine systems²⁰¹ foreshadowed the monumental impact of shotgun metagenomics to study genome-wide spatiotemporal patterns of evolution. Identifying targets of distinct evolutionary processes in complex systems is an intimidating task, given that even in the most well-controlled laboratory systems conceivable, such as the long-term evolution experiment that exceeds 75,000 generations for 12 Escherichia coli populations²⁰², presents us with overwhelming complexity²⁰³. Read recruitment from shotgun metagenomes using assembled contigs or reference genomes enables the characterization of genetic variants with access to exact allele frequencies at a single-nucleotide resolution²⁰⁴. Early studies that took advantage of shotgun metagenomics to study population genetics in relatively simple microbial systems observed the outcomes of transposase activity and purifying selection¹⁴⁶ and quantified genetic variability patterns throughout entire genomes of environmental populations²⁰⁵. In addition to increasing depth of metagenomic sequencing efforts, large-scale metagenomic surveys that yield public datasets with high-legacy^206,207 have been improving the application of microbial population genetics. More recent research efforts have elucidated variable rates of evolution for genes within individual microbial populations of the human gut²⁰⁸, patterns of gene- and genome-wide selective sweeps in lakes²⁰⁹, fate of subpopulations after human fecal microbiota transplantation procedures²¹⁰, associations between microbial population structures and human biogeography²¹¹, location-specific purifying forces in deep-sea hydrothermal vents²¹², the impact of large-scale oceanic current temperatures on population structures of marine organisms²¹³, the impact of anatomy on population structures on human skin²¹⁴, and more^215,216.

Many of the studies to date used subtle genetic variation in natural populations to infer to what extent divergence within microbial populations predict or correlate with other environmental variables, such as differences in biogeography, various ecological measurements, or health and disease states of organisms colonized by microbes. However, while such statistical inferences that treat every variant as equal are useful for broad ecological insights, they are not suitable to resolve the impact of individual variants on function, or the evolutionary processes that maintain them. Increasing the utility of environmental surveys of genetic variation to advance biotechnological applications will undoubtedly benefit from consideration of protein structures²¹⁷, given the sequence-structure-function paradigm²¹⁸ and our renewed understanding of the direct relationships between protein sequence and function²¹⁹. The well-understood importance of protein structures to interpret genomic variants²²⁰ was recognized in the context of population genetics over two decades ago²²¹. However, these ideals were delayed by challenges associated with predicting protein structures from amino acid sequences until recently²²². Especially with the introduction of a new generation of deep learning frameworks that enable accurate predictions of protein structures, careful interpretations of variants observed in metagenomes is becoming more accessible through integrative ‘omics platforms²²³ for structure-guided mining of metagenomes²²⁴. A recent study by Kiefl et al. that linked protein structures and microbial population genetics demonstrated the non-uniform distribution of non-synonymous variants with respect to their relative solvent accessibility and distance to ligands, and the rapidly decreasing rates of non-synonymous variants around ligand-binding sites of key metabolic genes as a function of bioavailable nitrogen concentrations in marine systems²²⁵. In another study, Han and Peng et al. observed variable selective pressures across the genomic features of organohalide reducing taxa that occupy deep sea cold seeps, and demonstrated the presence of stronger purifying selective forces acting on buried residues of metabolically critical reductive dehalogenases in the environment²²⁶.

Elucidating how microbes respond to rapid environmental change

Environments change constantly, and survival depends on the ability to keep up. Of all organisms, microbes are unusual in that their evolutionary rates may be fast enough to respond to steady and relatively slow environmental shifts that take place in timescales of decades or longer. However, the conventional means of genome evolution is not suitable to respond to rapid changes that may occur within hours, days, or weeks. Thus, life has evolved many means to dynamically respond to ever shifting fitness landscapes, some of which are just coming to light.

Optimization of transcription and mechanisms behind transcriptional regulation is arguably one of the most well understood means to respond to rapidly changing environmental conditions. Technological advances brought transcriptomics to fruition starting around the 1970s, and continued at a rapid pace with the increasing access to sequencing technologies²²⁷. The first whole-genome characterization of the transcriptomic landscape of a microbe demonstrated that Listeria monocytogenes had remarkable transcriptional changes as a function of the environment in which it occurred²²⁸. With massive advances in single-cell transcriptomics, recent studies demonstrate responses such as spatial expression profiles in biofilms²²⁹, heterogeneous responses to antibiotics exposure²³⁰, and transcriptional responses to phage infection²³¹ at the level of individual cells across isogenic populations. Metatranscriptomics, the study of RNA molecules directly recovered from complex samples, was applied first to marine microbial populations²³² and had challenging beginnings due to the the extremely low numbers of messenger RNAs per cell, short half-life of RNA molecules, lack of poly(A) tails that increase signal-to-noise ratio in RNA libraries, and challenges associated with the quantification of RNA sequencing results²³³. Nevertheless, contemporary large-scale metatranscriptomics surveys have reached a level of maturity to track microbial responses to warming oceans²³⁴, and elucidate disease-specific microbial responses in the human gut²³⁵.

Changing patterns of expression via transcriptional regulation is only one aspect of the genetic toolkit that rapidly optimizes microbial responses. Analyses of genomes and metagenomes revealed the pervasiveness of ‘phase variation’ in bacteria through inversion of regions in intergenic DNA to prevent the expression of costly genes in environments that don’t require them²³⁶. Jiang and Hall et al. demonstrated that the same microbial populations that were engrafted to multiple individuals through fecal microbiota transplantation can acquire rapid and heritable changes in their promoter orientations within days, highlighting the speed by which microbes can optimize their gene use to respond to individual-specific selective forces without changing their gene content²³⁶. Another intriguing mechanism that enables microbes to change the efficiency of their existing genes is implemented by diversity generating retroelements (DGRs)²³⁷. DGRs are a family of genetic elements that generate hypervariability in target genes based on a template sequence where every adenosine is replaced with a random base prior to mutagenic homing thanks to an highly error-prone retron-type reverse transcriptase²³⁸. DGRs, which can generate billions of sequence variants in a single population within a few generations, were first discovered in a phage that infects Bordetella²³⁹, yet they are widespread in bacteria and archaea²⁴⁰, and the analyses of metagenomes show that they target a wide variety of functions, including those that are involved in protein binding, carbohydrate binding, and cell adhesion²⁴¹. Within-population variants of DGR targets recovered from metagenomes can yield strong biogeographical partitioning²⁴², suggesting that selective forces optimize the pool of random variants to match the fitness requirements imposed by environmental conditions.

Translational regulation is yet another layer of biology that enables cells from all domains of life to mount rapid responses to environmental change, this time without changing their genomes or patterns of transcription, but by changing what is prioritized by the translational machinery. One of the essential components of the protein synthesis, transfer RNAs (tRNAs), play a significant role in this process through their relative abundances in the cell and chemical modifications that influence their interactions with the ribosome^243,244. Cells can actively change their tRNA relative abundances to switch their metabolic state and prioritize the translation of immediate needs to respond to environmental conditions²⁴⁵. Phages and plasmids can also influence the tRNA pool as they carry large inventories of tRNAs²⁴⁶ and tRNA modification enzymes¹²³ that are most likely driven by adaptive processes²⁴⁷ that enable direct influence on translation²⁴⁸. Characterization of tRNA transcripts and their chemical modifications to study the dynamics of translational regulation is well-established for cell populations that belong to the same organism, yet expanding these surveys into naturally occurring complex habitats is complicated by immense molecular and analytical challenges. For instance, due to their rigid secondary structures and chemical modifications, high-throughput sequencing of tRNA transcripts has been notoriously difficult²⁴⁹. But, improving molecular protocols promise broader access to tRNA sequencing²⁵⁰ and new opportunities to study targets and implications of translational regulation also in complex systems. For instance, an application of tRNA sequencing to animal gut microbial populations revealed differences in taxon-specific tRNA chemical modification profiles as a function of diet, where highly modified tRNAs in high-fat diet condition resolved to codons that were highly enriched in highly expressed proteins compared to low-fat diet condition²⁵¹, and demonstrated the feasibility of this approach to track rapid changes in translational priorities and their genetic targets. Another promising approach to study translational responses of environmental microbial populations is ribosome profiling – a strategy that gives access to ribosome-protected RNA fragments that are being translated at a given time point²⁵². An application of ribosome profiling to human gut microbial populations identified significant differences in transcripts and what was actually translated²⁵³, highlighting the need for highly-resolved strategies that target critical components of translation to gain more complete insights into microbial responses to environmental conditions.

Insights into in situ microbial activity

From the perspective of ecological studies that seek to understand in situ processes it is important to distinguish active versus inactive microbial community members, an ideal which has become more accessible in recent years. For example, bioorthogonal non-canonical amino acid tagging (BONCAT) can visualize translationally active cells within complex environmental samples²⁵⁴. Another innovative approach uses fluorescence in situ hybridization (FISH) and nanoscale secondary ion mass spectrometry (FISH–nanoSIMS) in combination with ¹⁵N stable isotope probing to link metabolic activity of cells to their location in complex consortia^255,256. From the perspective of metagenomic studies, an important motivation for use of methods that can identify organisms that are active in situ is the recognition that a substantial fraction of DNA (e.g., in soil) is from dead organisms or is extracellular²⁵⁷. This may in part be a reflection of the fact that soils are incredibly biodiverse and display rapid shifts in community composition over the seasons. A key approach to identify the subset of organisms that are active at any time is to leverage isotopes, which are selectively incorporated into proteins²⁵⁸ or DNA²⁵⁹ of actively growing organisms. Typically, stable isotope amendments are applied to samples of interest shortly before extraction of proteins or DNA. In the case of proteins, labeled vs. unlabeled proteins (e.g., containing ¹⁵N amended in the form of ammonia) are distinguished using highly accurate mass spectrometry measurements of peptides and their fragmentation productions²⁶⁰. In the case of DNA, fractions are separated by weight and different fractions sequenced independently. Implementation examples include growth of plants in a ¹³CO₂ atmosphere and tracking of ¹³C into the rhizosphere microbiome or addition of isotopically heavy water that is tracked into organisms that incorporate oxygen into de novo synthesized DNA. In the original implementation, referred to as qSIP (quantitative stable isotope probing), the heavy fraction (after consideration of the effect of genome GC content) is linked to actively growing organisms²⁵⁹. For example, one study used H₂¹⁸O addition and quantitative PCR of 16S rRNA genes to track growth and turnover of bacteria and fungi and thus quantified these contributions to the CO₂ flux from soil following the water addition to dry grassland soil²⁶¹. Subsequently, DNA labeling was tracked into whole metagenomes, enabling stable isotope-informed genome-resolved metagenomics²⁶². Despite the potential for artifacts when, for example, field-collected soil samples are necessarily incubated with heavy water in the laboratory, stable isotope labeling is emerging as an important method for the study of microbial ecosystem dynamics.

Emerging opportunities enabled by genome editing of communities

Most current biological knowledge was generated via experiments in which a specific microbe was grown as an isolate (i.e., in a pure culture) and a certain gene was knocked out or added experimentally to determine its function, and much of it involved just one organism, E. coli. However, the tree of life is vast, and the suite of isolates available for traditional genetic manipulation is extremely limited. Even after functional assignments that leverage structural similarity, there are vast numbers of proteins for which functions cannot be predicted.

Why do microbes grow as members of communities, rather than evolve into multicellular organisms such as trees or insects? The benefits may relate to adaptability to changing environmental conditions and likely comes from flexibility in community membership and efficiencies gained via cooperative activities. As a consequence, microbes often depend on other organisms for key resources and devote genetic potential to communication and competition. These facets of microbial activity are understandably understudied because the majority of genetic and biochemical research conducted to date has been performed on organisms growing alone. In the absence of new methods that enable genetic manipulations of organisms without isolation, this is an essentially insurmountable problem. Understanding of the genes and pathways via which microbes interact with each other (and macroorganisms) requires the ability to perform “gold standard” genetic manipulation without the requirement for isolation. In other words, genome editing of microbes while they exist in the context of their communities. With this approach, not only genes involved in interaction can be interrogated but also organisms that cannot be grown in isolation become tractable for experimental manipulation.

If one is to edit a microbiome using CRISPR-Cas methods, there are several key requirements. First, stable, reproducible laboratory-based microbiomes are needed for method development and testing. Next, it is vital that the sequence that is targeted for modification is known. For this reason, genome-resolved metagenomics is foundational. The third challenge is to find a means by which the genome editing tools can be delivered into the targeted cells, and this is likely to be microbe-specific (e.g., conjugation, natural competence). To address this challenge, Rubin and others developed ET-seq and to enable organism- and locus-specific genetic manipulation, a DNA-editing all-in-one RNA-guided CRISPR–Cas transposase (DART)²⁶³. Although direct editing of a human infant microbiome member was achieved, the editing efficiency was low and occurred in well studied bacteria. Future delivery improvements are needed to target diverse organisms from little studied branches of the tree of life before this potentially revolutionary approach can reach its potential.

Genetic manipulation in community context has been explored in the mouse microbiome, ultimately, with the goal of engineering communities in situ. The first step recovered engineered strains for reintroduction into the gut to alter overall community function. To do this, Ronda et al. developed Metagenomic Alteration of Gut microbiome by In situ Conjugation (MAGIC)²⁶⁴. The method harnesses horizontal gene transfer to genetically modify gut bacteria. A vector introduces green fluorescent protein labeling to recipient cells that can then be cell sorted, or introduces antibiotic resistance that can be selected for. Libraries included vectors bearing replication elements suitable for different gut bacteria. Among the challenges identified was long-term stability of the engineered genetic constructs, an issue that was somewhat addressed by isolation and engineering of host-derived strains that could mediate further transfer of engineered functions in the microbiome.

A recent study reported the important accomplishment of genetic editing of a population of episymbiotic Saccharibacteria in the context of a co-culture with host Actinobacteria²⁶⁵. The experiments harnessed the natural competence of the Saccharibacteria to insert heterologous sequences and targeted gene deletions. This enabled identification of novel Saccharibacterial genes as important for the growth of these cell surface-attached symbionts on their Actinobacteria host cells.

Ultimately, the ability to genetically modify strains of interest in microbiome context will make it possible to assign functions to a vast spectrum of novel genes and enable precise tuning of microbiome composition and function. By microbiome manipulation, a vast range of desirable outcomes of societal and environmental importance may be achieved.

A shifting mindset to address the challenges of our modern era

A hallmark of modern microbiology is the plethora of new kinds of data generated from inconceivably intricate microbial systems as a whole with brand new molecular and computational strategies. Taking full advantage of our observational powers demands us to focus on how to improve many of the well-established practices in microbiology; from how we responsibly disseminate data for transparency and reproducibility, or how we train life scientists to better exploit new data streams and computational resources, to how we encourage, support, and promote team science efforts to solve our immense need for integration when we mostly credit the first name and/or the last name that appear in final publications. Apart from such tangible issues that can be solved through tangible actions, some aspects of modern microbiology can also benefit from discussions regarding how the classical determinants of good scientific practice apply to this new era in which our observational powers far exceeds our ability to interpret the data they yield. For instance, how critical is it to lead contemporary research in microbiology with the expectation of testable hypotheses as the first and foremost requirement to benchmark or implement new ideas? And how should we recognize, prioritize, or promote the synthesis of research products from laboratory experiments or data-driven explorations?

We believe it is important to recognize that hypothesis-driven research is inevitably limited by the ability of the human mind to imagine facets of microbiology that may be far from what we know today. Indeed, from the discovery of penicillin to insulin, from radioactivity to X-rays, history of science has a long list of breakthroughs that emerged as unintended consequences of exploratory research and serendipity. For example, although the CRISPR-Cas system could theoretically be conceived of de novo, its accidental discovery through microbial genome sequencing provided a new window on virus-host interactions and ultimately opened the path for a revolution in biotechnology. Sequences from bacterial host CRISPR loci and coexisting viruses revealed infecting bacteriophage genomes as the source of the CRISPR spacer sequences^266–268, but biochemical experiments were essential for uncovering mechanisms and thus enabling adaptation for targeted editing of eukaryotes²⁶⁹, with almost unimaginable impacts in medicine, agriculture, and basic science. This and other examples underline the opportunities that arise from exploratory research, and its confluence with laboratory experimentation, once the hypotheses reveal themselves from exploration.

In a world of unknowns, rigorous exploratory research provides the basis for hypothesis-driven research. Thus, given the huge scale and the microbial underpinnings of many of the problems of the modern era, and the hope and potential for microbiome-based solutions, we advocate increasingly deploying the methods of microbiology in the context of ecosystems, whether they be the ocean, agricultural land, wetlands, municipal landfills, or others, and explore open-mindedly, and dare we say it, without asking for or testing any strict hypotheses. Only by developing a concrete understanding of how these ecosystems seem to work, and only after conducting targeted, field-scale experiments within them, will it be possible to tackle questions of utmost importance for the wellbeing of us and our planet, such as “how do we store more carbon in soil?”, “how do we cut trace gas emissions from rice paddies?”, “what can we do to reduce methane releases from cattle?”, “how do we improve the quality of marginal drinking water, accelerate the breakdown of ocean plastic, reduce salinization of agricultural soils, and use the earth’s subsurface to store CO2?”, “can food biotechnologies be improved to make, for example, vegan meat alternatives more palatable?”, “how do manipulation of diet, probiotics or environmental exposures alter development of human microbiomes and can changes in these factors address the ever growing list of microbiome-related diseases?”. For example, given that around 40% of the land surface is already highly manipulated in support of food production²⁷⁰, agricultural systems are a logical target to learn how microbiome functionality can be modulated to reduce greenhouse gas emissions²⁷¹. Yet our ability to implement investigations of such magnitude in terrestrial, marine, or host-associated habitats requires a mindset that embraces natural complexity and remains open to experiments in such systems at large scale, with integrated analyses of readouts ranging from microbiome composition to biogeochemical fluxes to crop productivity or health status.

These considerations are particularly relevant to highly critical ecosystem-level questions such as “how will future climate scenarios alter labile and recalcitrant dissolved organic carbon pools in warming oceans and impact marine food webs and the atmosphere?”, “how nitrous oxide emissions from grassland soil will be altered by shifts in rainfall amount and timing or temperature driven by climate change?”, or “how does soil recover after severe wildfires and how the patchiness of burnt areas impacts the reestablishment of healthy soil microbiomes?”. For questions at this magnitude, harnessing natural events and variation over temporal and geographical gradients may have bigger potential to improve our understanding of key processes and guide modeling efforts to predict ecosystem behavior compared to artificial or small-scale controlled disruptions to test hypotheses. In other words, sometimes there are reasons to let nature do the experiment.

Our review highlights the motivation, progress, and potential of the data-driven epoch of microbiology to tackle systems of great complexity with vast scales of unknowns. A critical admission in our search for solutions to the most pressing needs of our planet is that the complexity of microbial systems that modern microbiology aims to elucidate may not always be compatible with the desire for hypothesis-driven research. Thus, from the level of PhD committees to funding agencies, a strict requirement for ‘hypotheses’ should not be a critical determinant of the fate of research proposals. While we should not hold data products to a lesser degree of scrutiny, we also should not trivialize data-driven investigations at the level of grant panels or editorial decisions. In the quest of understanding how nature works, it is important to embrace complexity while looking for means to reveal its driving principles.