Skip to main content
eLife logoLink to eLife
. 2017 Dec 27;6:e31035. doi: 10.7554/eLife.31035

Phenotype inference in an Escherichia coli strain panel

Marco Galardini 1, Alexandra Koumoutsi 2, Lucia Herrera-Dominguez 2, Juan Antonio Cordero Varela 1, Anja Telzerow 2, Omar Wagih 1, Morgane Wartel 2, Olivier Clermont 3,4, Erick Denamur 3,4,5, Athanasios Typas 2,, Pedro Beltrao 1,
Editor: Naama Barkai6
PMCID: PMC5745082  PMID: 29280730

Abstract

Understanding how genetic variation contributes to phenotypic differences is a fundamental question in biology. Combining high-throughput gene function assays with mechanistic models of the impact of genetic variants is a promising alternative to genome-wide association studies. Here we have assembled a large panel of 696 Escherichia coli strains, which we have genotyped and measured their phenotypic profile across 214 growth conditions. We integrated variant effect predictors to derive gene-level probabilities of loss of function for every gene across all strains. Finally, we combined these probabilities with information on conditional gene essentiality in the reference K-12 strain to compute the growth defects of each strain. Not only could we reliably predict these defects in up to 38% of tested conditions, but we could also directly identify the causal variants that were validated through complementation assays. Our work demonstrates the power of forward predictive models and the possibility of precision genetic interventions.

Research organism: E. coli

Introduction

Understanding the genetic and molecular basis of phenotypic differences among individuals is a long-standing problem in biology. Genetic variants responsible for observed phenotypes are commonly discovered through statistical approaches, collectively termed Genome-Wide Association Studies (GWAS, Bush and Moore, 2012), which have dominated research in this field for the past decade. While such approaches are powerful in elucidating trait heritability and disease associations (Yang et al., 2010; Welter et al., 2014), they often fall short in pinpointing causal variants, either due to lack of power to resolve variants in linkage disequilibrium (Edwards et al., 2013) or for the dispersion of weak association signals across a large number of regions across the genome (Boyle et al., 2017). Furthermore, by definition GWAS studies are unable to assess the impact of previously unseen or rare variants, which can often have a large effect on phenotype (Bodmer and Bonilla, 2008). Therefore, the development of mechanistic models that address the impact of genetic variation on the phenotype can bypass the current bottlenecks of GWAS studies (Lehner, 2013).

In principle, phenotypes could be inferred from the genome sequence of an individual by combining molecular variant effect predictions with prior knowledge on the contribution of individual genes to a phenotype of interest. Such knowledge is now readily available by chemical genetics approaches, in which genome-wide knock-out (KO) libraries of different organisms are profiled across multiple growth conditions (Kamath et al., 2003; Dietzl et al., 2007; Hillenmeyer et al., 2008; Nichols et al., 2011; Price et al., 2016). One output of such screens are genes whose function is required for growth in a given condition (also termed ‘conditionally essential genes’). Variants negatively affecting the function of those genes are likely to be associated with individuals displaying a significant growth defect in that same condition. This approach would then effectively be a way to prioritize variants with respect to their impact on the growth on a particular condition. The impact of variants on gene function can be reliably inferred using different computational approaches (Thusberg et al., 2011; Kulshreshtha et al., 2016), which can offer mechanistic insights into how such variants impact on a protein’s structure and function. As rare or previously unseen variants can also be used by this approach, there lies the possibility to deliver predictions of phenotypes at the level of the individual, with no prior model training. Combining variant effector predictors with gene conditional essentiality has been previously tested with some success in the budding yeast Saccharomyces cerevisiae, but only on a limited number of individuals and conditions (15 and 20, respectively, Jelier et al., 2011). Given the continuous increase in genome-wide gene functional studies, there is an opportunity to apply such an approach more broadly.

In contrast to eukaryotes, diversity within the same bacterial species can result in two individuals differing by as much as half of their genomic content. Escherichia coli is not only one of the most studied organisms to date (Blount, 2015), but also one of the most genetically diverse bacterial species (Lukjancenko et al., 2010; Tenaillon et al., 2010). Individuals of this species (strains), exhibit a diverse range of genetic diversity, from highly homologous regions to large differences in gene content, collectively termed ‘pan-genome’ (Medini et al., 2005). Phenotypic variability is therefore likely to arise from a combination of single nucleotide variants (SNVs) and changes in gene content. Since gene conditional essentiality has been heavily profiled for the reference E. coli strain (K-12, Nichols et al., 2011; Price et al., 2016), here we set out to systematically test the applicability of such genotype-to-phenotype predictive models for this species. We reasoned that it would also test the limits of the underlying assumption of this model, which is that the effect of the loss of function of a gene is independent of the genetic background (Dowell et al., 2010).

We therefore collected a large and diverse panel of E. coli strains (N = 894), for which we measured growth across 214 conditions, as well as obtained the genomic sequences for the majority of the strains (N = 696). For each gene in each sequenced strain we calculated a ‘gene disruption score’ by evaluating the impact of non-synonymous variants and gene loss. We then applied a model that combines the gene disruption scores with the prior knowledge on conditional gene essentiality to predict conditional growth defects across strains. The model yielded significant predictive power for 38% of conditions having a minimum number of strains with growth defects (at least 5% of tested strains). We independently validated a number of causal variants with complementation assays, demonstrating the feasibility of precision genetic interventions. Since our predictions did not apply equally well for all conditions, we conclude that the set of conditionally essential genes has diverged substantially across strains. Overall, we anticipate that the E. coli reference panel presented here will become a community resource to address the multiple facets of the genotype to phenotype research, ultimately enabling the development of biotechnological and personalized medical applications.

Results

The phenotypic landscape of the E. coli collection

We have assembled a large genetic reference panel of 894 E. coli strains able to capture the genetic and phenotypic diversity of the species. These are broadly divided into natural isolates (N = 527) and strains derived from evolution experiments (N = 367). Out of these, 321 had available genome sequences, and we obtained the genomes of additional 375, reaching a total of 696 strains. The full list of strains, including name, collection of origin and links to relevant references is provided in Supplementary file 1 and online at https://evocellnet.github.io/ecoref.

To test our capacity to develop genotype-to-phenotype predictive models we measured the fitness of this strain collection on a large variety of conditions (N = 214). We used high-density colony arrays and measured colony size as a proxy for fitness, similar to what has been applied before for the E. coli K-12 knockout (KO) library (Nichols et al., 2011). We used the deviation of each strain’s colony size from itself across all conditions and all other strains in the same condition as our final phenotypic measure (Collins et al., 2006). Thereby we obtained a list of conditions for which we know whether a tested strain has grown significantly less than the expectation (Figure 1A and Materials and methods). Both biological replicates (Figure 1B) and strains present in two distinct plates (Figure 1—figure supplement 1) were positively correlated (Pearson’s r 0.693 and 0.648, respectively), indicating that we measured phenotypes with high confidence. Clustering of conditions based on phenotypic profiles across all strains was consistent with the conditions macro-categories (e.g. stresses versus nutrient sources) and with the number of sensitive strains (Figure 1C). The correlation of phenotypic profiles also clustered drugs with same mode of action (MoA), as shown by comparing the clusters’ purity against those of random clusters (Figure 1D). The full phenotypic matrix for each strain across all conditions contains 114,004 single measurements (Supplementary file 1).

Figure 1. The phenotypic landscape of the E.

coli strain collection. (A) Phenotypic screening experimental design and data analysis. (B) Phenotypic measurements replicability, as measured by pairwise comparing the S-scores of all three biological replicates. (C) Hierarchical clustering of condition correlation profiles; the threshold is defined as the furthest distance at which the minimum average Pearson’s correlation inside each cluster is above 0.3. The two colored bands on top indicate the number of strains with growth defects for each condition and its category, showing consistent clustering. Gray-colored cells in the matrix represent missing values due to poor overlap of strains tested in the two conditions. (D) Clusters purity (computed for drug targets) for each hierarchical distance threshold, against that of random clusters (100 repetitions) shows that drugs with similar target tend to cluster together. (E) Pearson’s correlation between phylogenetic and phenotypic distances, based on phylogenetic independent contrasts (see Materials and methods). (F) Core genome SNP tree for all strains in the collection. Grey shades in the inner ring indicate the number of conditions tested for each strain, red shades in the outer ring indicate the proportion of tested conditions in which the strain shows a significant growth defect. The black arrow indicates the reference strain.

Figure 1.

Figure 1—figure supplement 1. Phenotypic measurements reproducibility and correlation with genetic distance.

Figure 1—figure supplement 1.

(A–C) S-scores for each replicate are compared against each other (D) S-scores comparison between strains present in multiple 1536 plates.

If all genetic variants were to contribute to phenotypic divergence between strains we would expect to observe a strong positive correlation between those two metrics. This was not the case even when taking into account the phylogenetic dependencies between strains (r = 0.07, see Materials and methods). This finding reinforces the idea that most variants across these strains have a neutral impact on their phenotypes. Of particular interest are those strains derived from evolution experiments, such as the members of the LTEE collection (Tenaillon et al., 2016). While most strains (189 out of 266 tested) grew as expected in all tested conditions, a significant fraction (N = 77) exhibited at least one conditional growth defect phenotype. Again, the phylogenetic distance from the parental strains (REL606 and REL607) is not correlated with the proportion of growth phenotypes (Pearson’s r: 0.08), even though hypermutators exhibit a slightly higher number of phenotypes (Cohen’s d: 0.513). These results clearly underline that simple metrics of phylogenetic similarity are not predictive of differences in phenotype. Instead, few DNA variants are sufficient to cause clear phenotypic differences, indicating the importance of statistical or predictive strategies to prioritize those variants.

In toto, we have assembled an E. coli reference strain panel, which we have broadly phenotyped. This phenotyping resource recapitulates known biology, surveying a rich phenotypic space and providing insights into the evolution of phenotypes within the species.

Gene level predictions of variant effects

As most DNA variants are likely to be neutral in their impact on gene function, variant effect predictors are required to build phenotype predictive models for each strain. For this we derived structural models and protein alignments covering 60.2% and 94.7% of the E. coli K-12 proteome, respectively (or 50.9% and 95.9% of all protein residues) and used them to compute the impact of all possible nonsynonymous variants (see Materials and methods, available at http://mutfunc.com). The impact of multiple non-synonymous variants within a gene was then combined using a probabilistic approach (Jelier et al., 2011), into a single likelihood measure of gene disruption (here termed ‘gene disruption score’, Figure 2A and Materials and methods). We also assumed that reference genes missing completely from a strain would have a high gene disruption score.

Figure 2. The ‘gene disruption score’, a gene-level prediction of the impact of genetic variants.

(A) Schematic representation of how all substitutions affecting a particular gene are combined to compute the gene disruption score. (B) Average gene disruption score across all strains for four categories of genes conserved in all strains (‘core genome’). Conserved genes are defined as those genes found in more (or less) than 95% of the bacterial species present in the eggNOG orthology database (Huerta-Cepas et al., 2016a). Statistically significant differences (Cohen’s d value >0.3) are reported. (C) Gene-gene correlation profile of gene disruption across all strains shows clusters of potentially functionally related proteins. (D–E) The gene disruption profiles as a predictor of genes function. (D) Pairwise correlation of gene disruption scores inside each annotation set (colored boxes) and inside a random set of genes of the same size (grey boxes). (E) Gene-gene correlation profile of gene disruption scores in protein complexes; shown here a subset with high disruption score correlation.

Figure 2.

Figure 2—figure supplement 1. Additional properties of the gene disruption score.

Figure 2—figure supplement 1.

(A) Correlation between phylogenetic distance from the reference strain and average disruption across all reference genes. (B) Significant reduction in correlation after correction of the disruption score (see Materials and methods). (C–E) Prediction of gene functional associations using disruption score profiles. (C) ROC curve using disruption score profiles across all genes. (D–E) Higher predictive power of the gene disruption profile when restricted to accessory genes, including (D) or not (E) the information about gene presence absence.

The gene disruption score can be considered as a relevant measure of the impact of mutations on gene function. Essential and phylogenetically conserved genes show lower average gene disruption scores across all strains, as compared to less conserved or random ones (Figure 2B). We also observed that genes that are predicted to lose their function together across all strains (Figure 2C) tend to be functionally associated (Figure 2D and E), in particular for genes belonging to the accessory genome (i.e. genes not present in all strains, Figure 2—figure supplement 1). These observations suggest that the gene disruption score is a biologically relevant measure of the impact of mutations at the gene level and that it could be used for growth phenotype predictions.

Predictive models of conditional growth defects

We combined the gene disruption scores for each gene in each strain with the genes’ conditional essentiality to obtain conditional growth predictive models. For each condition, we evaluated the impact of those variants affecting the genes that are essential for that same condition in the reference K-12 strain. We selected 148 conditions in which at least one strain displayed a growth defect and the E. coli K-12 KO collection had also been tested for (Nichols et al., 2011 and Herrera-Dominguez et al., unpublished). For those, we computed a conditional score that would rank the strains according to their predicted growth level in the tested condition, from normal growth to the most defective growth phenotype (see Materials and methods). Briefly, if a strain has many detrimental variants in genes important for growth in a given condition, our predictive score will rank that strain as being highly sensitive in that condition. We then compared this predicted ranking with the experimental fitness measurements. The Area Under the Curve of a Precision-Recall curve (PR-AUC) was used for assessing our predictive power (Figure 3A and Materials and methods). We note that this predictive score is condition specific and no parameter fitting or training was used.

Figure 3. Prediction of growth-defect phenotypes in the E.coli strain collection.

(A) Schematic representation of the computation of the prediction score and its evaluation; for each condition the predicted score is computed using the disruption score of the conditionally essential genes. The score is then evaluated against the actual phenotypes through a Precision-Recall curve. (B) Higher predictive power for conditions with higher proportion of growth phenotypes. For each condition set, the PR-AUC value for each condition is reported, together with the median and mean value (C) Significance of the PR-AUC value reported for the condition ‘Clindamycin 3 μg/ml’, against the distribution of three randomization strategies. ‘Shuffled strains’ indicates a prediction in which the actual strains’ phenotypes have been shuffled; ‘shuffled sets’ indicates a prediction where the conditionally essential genes of a different condition have been used, and ‘random set’ indicates a prediction where a random gene set has been used as conditionally essential genes. For all three randomizations we report a significant difference between the actual prediction and the distribution of the randomizations (q-values of 1E-30, 0.05 and 1E-22, respectively). See Figure 3—figure supplement 2 for the other conditions. (D) Genome-wide gene associations are in agreement with the predictive score; the enrichment of conditionally essential genes in the results of the gene association analysis is significantly higher in conditions with higher PR-AUC.

Figure 3.

Figure 3—figure supplement 1. Detailed view of the predicted growth score and its properties.

Figure 3—figure supplement 1.

(A) Influence of the different predictors of the impact of mutations on each condition PR-AUC. ‘SNPs’ indicates single nucleotide variants only, ‘Accessory genome’ gene presence-absence patterns, ‘All predictors’ the combination of both, plus stop codon substitutions. (B–C) Proportion of well-predicted conditions (PR-AUC >= 0.1 and Pearson’s FDR-corrected p-value<=0.01) over total conditions with at least 1% and 5% sick strains. Marker’s size is proportional to the log10 of the FDR-corrected p-value. (E) Conditions with higher predictive power (measured as PR-AUC) also have an enrichment of sick strains at the top of the predicted score, as measured by the Gene Set Enrichment Analysis (GSEA); sick strains are used as ‘gene sets’. A pseudocount of 10−4 has been added to the GSEA p-values. (F) Prediction performance improves when using the weighting scheme to account for conservation of gene essentiality, especially for well-predicted conditions.
Figure 3—figure supplement 2. Significance of the PR-AUC value reported for all conditions with at least 5% of the tested strains showing a sick phenotype, against the distribution of three randomization strategies.

Figure 3—figure supplement 2.

‘Shuffled strains’ indicates a prediction in which the actual strains’ phenotypes has been shuffled; ‘shuffled sets’ indicates a prediction where the conditionally essential genes of a different condition have been used, and ‘random set’ indicates a prediction where a random gene set has been used as conditionally essential genes. Conditions are ordered by the significance of the difference between the actual prediction and the ‘Random genes’ bootstrap. The proportion of conditions where the actual prediction is significantly different than the randomizations (q-value <0.05) are 52%, 21% and 33.3% for the ‘shuffled strains’, ‘shuffled sets’ and ‘random genes’ randomizations, respectively.

Our predictive score is able to discriminate strains with normal growth from ones with growth defects with significantly higher power than randomized scores. Both the predicted impact of single nucleotide variants and gene presence/absence patterns contribute to the predictive power of the model (Figure 3—figure supplement 1). The predictive power increases for conditions in which more strains displayed growth defects (Figure 3B). We verified the significance of our predictions through the comparison against three randomization strategies: shuffled strains, shuffled gene sets and random gene sets (see Materials and methods). For all three strategies we observed a clear dependency between the PR-AUC value and the significance of our predictions (Figure 3C and Figure 3—figure supplement 2). Since some conditions are similar and share a significant fraction of their conditionally essential genes, we sometimes observe a skew in the performance of the ‘shuffled set’ randomizations (Figure 3C). We correctly predicted 20% of conditions that have at least 1% of poorly growing strains, with higher predictive capacity for conditions with larger number of strains with growth defects, reaching 38% for all conditions with more than 5% of poorly growing strains (Figure 3—figure supplement 1). Weighting the contribution of the conditionally essential genes to each condition also improved the predictive power, particularly for well-predicted conditions (Figure 3—figure supplement 1, Materials and methods).

To independently validate our predictive models, we carried out a GWAS analysis based on genes presence/absence and the growth phenotypes. Consistent with the validity of our models, we found that for conditions we predict with higher confidence (PR-AUC >= 0.1), there is a significant overlap between the K-12 genes predicted to be essential in the condition and the genes associated with poor growth by the association analysis (Figure 3D, Fisher’s exact test, p-value 0.005).

We further examined two well predicted conditions (PR-AUC >0.35), pseudomonic acid 2 μg/ml (the antibiotic mupirocin) and minimal media with the addition of amino acid mix, to inspect our predictive model (Figure 4). Both conditions showed an enrichment of strains with growth defects at high predicted scores (GSEA p-values of 0.001 and <10−6, respectively), which is a common property of conditions with higher PR-AUC (Figure 3—figure supplement 1). The reference K-12 strain harbors many conditionally essential genes in minimal media (N = 181), providing an example for which growth phenotypes are well predicted from deleterious effects across a large number of genes in different strains. In contrast, pseudomonic acid is a condition with few (N = 10) conditionally essential genes, thus making it easier to evaluate the contribution of single genes to the phenotype. Of the 25 strains with highest predicted score in pseudomonic acid, 13 have been misclassified by the model. Seven misclassified strains had high disruption scores in either lpcA or rfaE, two genes involved in the biosynthesis of a lipopolysaccharide (LPS) precursor (L-glycero-β-D-manno-heptose). Both, when deleted in K-12, cause a strong growth defect under pseudomonic acid. Only 1 of the correctly predicted strains had a mutation in one of those two genes (IAI36), suggesting that this pathway might be conditionally essential only in K-12. Another example of incorrect predictions involves two strains (ECOR-27 and ECOR-58) that share very similar disruption score profiles for the conditionally essential genes in pseudomonic acid (Figure 4C), but only ECOR-27 had a growth defect in this condition. Both strains harbor a single nonsynonymous mutation in acrB (E414G for ECOR-27 and I466T for ECOR-58), which, in both cases, is predicted as highly deleterious. Changes in conditional essentiality or epistatic effects are possible explanations for this misclassification, and therefore mapping and incorporating this information in our models could significantly benefit predictions in the future.

Figure 4. Detailed example on the computation and evaluation of the predicted score on two conditions.

Figure 4.

(A) Precision-Recall curve for the two example conditions. Dashed lines represent the average Precision-Recall curve for the same predictions carried out using 10’000 random gene sets of the same size of the actual conditionally essential genes for both conditions. Vertical lines represent the mean absolute deviation for the precision across the randomizations. The two randomization sets are significantly different than the actual predictions (q-values of 1E-33 and 0.02, respectively). (B) Receiver operating characteristic curve for the two example conditions. Dashed lines represent the same randomizations as in (A). (C–D) For each condition, the gene disruption score for the conditionally essential gene across all strains is reported, together with the resulting predicted conditional score and actual binary phenotypes (pale red: healthy and red: growth defect). Strains are sorted according to the predicted conditional score, while genes are sorted according to their weight in the predictive model; only the top 10 conditionally essential genes are shown. The inset reports the disruption score, predicted score and actual phenotypes for the top 25 strains.

Experimental validation of causal variants

Since we use mechanistic models in our phenotype predictions to calculate the impact of non-synonymous genetic variants, we can then also directly pinpoint the causal variants and implement genetic therapies to correct growth phenotypes. We tested this by ranking the mutated conditionally essential genes in each condition according to their predicted ability to rescue growth phenotypes (Figure 5A and Materials and methods). Several genes were predicted to restore many growth phenotypes (Figure 5B), including the genes forming the AcrAB-TolC multidrug efflux pump, which were predicted to restore growth in up to ~1000 condition-strain pairs (1012 acrB, 494 acrA and 517 tolC, respectively), reflecting the importance of this efflux system in drug resistance (Li et al., 2015).

Figure 5. Experimental confirmation of the predicted phenotype-causing genotypes.

(A) Schematic representation of the experimental approach. (B) Distribution of the number of predicted restored phenotypes per gene; red stripes indicate the genes that were experimentally tested. (C) Growth change between the target strain with the empty plasmid against the one expressing the reference copy of the target gene (complementation). Strains for which a change in phenotype is predicted are compared to those where no change is predicted. (D–F) Detailed representation of the results of the complementation experiment in three conditions. Mean colony area and the 95% confidence intervals are reported. Significant differences (t-test p-value<0.01) between colony area of the strains with the empty and complemented plasmids are reported. ‘Other complementation’ reports strains expressing a different gene than the focal one, indicated between parenthesis. (G) Cartoon representation of AcrB 3D structure (PDB entry: 2dr6). Only one of the three monomers is highlighted in blue; colored spheres represent known non-synonymous variants in the reported strains.

Figure 5.

Figure 5—figure supplement 1. Detailed view of the experimental confirmation of the predicted phenotype-causing genotypes.

Figure 5—figure supplement 1.

(A) Replicability in colony size (measured in pixels) after outer frame and spatial correction between the two replicates. (B) Overview of the results of the complementation experiments. Mean colony area and the 95% confidence intervals are reported. Significant differences (t-test p-value<0.01) between colony area of the strains with the empty and complemented plasmids are reported. Conditions CHIR090.MECILLINAM, MECILLINNAM and OXACILLIN.5UM have been excluded from further analysis, as the deletion strain from the KEIO collection (positive control) does not show a sick phenotype with an empty plasmid.

We selected eight genes to experimentally verify our predictions, including genes involved in either drug resistance (acrAB, soxSR and slt) or auxotrophic growth (glnD and proAB). We then selected conditions for which the introduction of the reference allele in the target strains was predicted to restore growth, and controls where no improvement in phenotype was expected (Materials and methods). We also introduced the plasmid expressing the reference allele in the reference strain as a negative control and in the corresponding deletion strain from the KEIO collection (Baba et al., 2006) as a positive control. Overall, we observed a high correlation between replicates of this experiment (Pearson’s r: 0.94, Figure 5—figure supplement 1). In total, we tested 64 strain-condition combinations, detecting a significant difference (p-value 8.4E-11, two-sided t-test) in colony size change between strain-condition pairs in which we had predicted growth will be restored (N = 14) versus ones we had not (N = 50) (Figure 5C and Materials and methods). This validated the effectiveness of our predictive models and more generally confirmed that genetic variants can be used to predict causal mutations and prioritise strategies for reverting/modifying phenotypes.

We investigated three conditions in more detail: pseudomonic acid 2 μg/ml with an acrB complementation, pyocyanin 10 μg/ml (a toxic secondary metabolite produced by Pseudomonas aeruginosa) with a soxSR complementation and the L-glutamine amino acid as nitrogen source with a proAB complementation (Figure 5D–F and Figure 5—figure supplement 1 for all conditions and strains). In all cases, strains harboring the gene(s) predicted to restore the phenotypes grew significantly better than strains carrying the empty plasmid (t-test p-value<0.01). In contrast, none of the strains harboring a different complementation gene showed a significant increase in colony size. In one case we detected an unexpected increase in colony size for one of the strains where we hadn’t predicted any change (strain IAI39 expressing AcrB, Figure 5D). We hypothesize that the original growth phenotype is either due to an incorrectly classified variant in acrB or due to another variant that acts on the expression of this efflux pump.

Of the five strains expressing the reference allele of acrB tested in pseudomonic acid, three harbored nonsynonymous variants in their chromosomal copy. One of them (IAI30) was predicted to increase its colony size after the complementation, whereas the other two (ECOR-42 and ECOR-28) were not predicted to be affected by the complementation (Figure 5D). We mapped those nonsynonymous variants to the three-dimensional structure of AcrB (Murakami et al., 2006) to inspect their potential impact on the protein function (Figure 5G). Both ECOR-42 and ECOR-28 harbor a single deleterious nonsynonymous variant (SIFT, p-value~0.01) in the transmembrane domain of the protein (A915D and T1013I, respectively). Strain IAI30 on the other hand carries two nonsynonymous variants: E567V and H596N, both located in the AcrB PC1 subdomain, which is involved in the entry of the ligand that will be then extruded by the efflux pump (Murakami et al., 2006; Seeger et al., 2006). Since the E567V variant is predicted to be highly deleterious (SIFT, p-value~0.001), we presume that it impairs the fundamental drug uptake function of the AcrAB-TolC multidrug efflux pump, while the variants in the other two strains might not be significantly affecting the pump’s function. This example shows how mechanistic interpretations of the impact of genetic variants can direct insights into the emergence of fitness defects and suggest potential gene therapeutic strategies, down to the level of the single genetic variant.

Discussion

As part of this study, we have assembled and phenotyped a large, phylogenetically diverse E. coli resource strain collection. We observed little to no correlation between genotypic and phenotypic distance across strains. This demonstrates the need of assessing the impact of genetic variability as most variants are likely to be neutral. We combined these mechanistic interpretations of the impact of non-synonymous variants in the genomes of all these strains with prior knowledge on E. coli K-12 gene conditional essentiality to predict conditional growth defects of strains in the collection. This resulted in reliable predictions for up to 38% of the tested growth conditions. Our mechanistic-based model directly pinpoints the causal variants and allows us to design successful genetic therapies, something that is often the bottleneck in traditional GWAS approaches. Moreover, our approach opens up the door for predicting strain functional capacities in less studied microbes, such as those present in the human microbiome, with sole requirements being the strain genome sequence and knowledge of gene function in a model strain.

Despite the significant predictive power, we still could not reliably predict the strains phenotypic response in a number of conditions. Epistatic interactions, the impact of synonymous variants, variants in non-coding regions and the large accessory genome are four possible factors influencing phenotypic diversity. Models that account for these factors could therefore be required for more accurate predictive models. As compared to the previous application of this method to S. cerevisiae strains (Jelier et al., 2011), we observed a lower overall accuracy. The larger and more diverse strains cohort, the higher diversity in conditions tested, including several concentrations for each chemical, and the larger genetic diversity in bacteria are potential causes for the differences we have observed. Most importantly, we postulate that conditional essentiality may not be conserved across strains and therefore that it interacts with each specific genetic background. Previous work in two closely related S. cerevisiae strains and five human cancer cell lines has shown that gene-specific phenotypes can diverge considerably (Ryan et al., 2012; Hart et al., 2015). Given their higher genetic variability, this genetic background dependence of gene-specific phenotypes is expected to be stronger in E. coli and other bacteria, even though this remains to be tested. Despite these limitations we believe that we have demonstrated how gene function assays and genetic variant prioritization can be leveraged to deliver growth predictions and genetic intervention strategies. Future iterations based on a more complete understanding on conservation of conditional essentiality should improve these predictions.

Finally, we propose this strain collection as a valuable community resource to develop and test genotype-to-phenotype models, following the example of previous successful genetic reference panels (Ayroles et al., 2009; Liti et al., 2009; Bennett et al., 2010; Atwell et al., 2010; 1001 Genomes Consortium, 2016; Weinstein et al., 2013; 1000 Genomes Project Consortium et al., 2015). Any additional molecular and phenotypic measurement on the collection members will amplify the benefit for the entire research community, moving us closer to understanding the basic biological question of how genetic variation translates to differences among individuals.

Materials and methods

Genome sequencing, assembly and annotation

Strains whose genome sequence were not yet available were sequenced using various Illumina paired-end platforms (Supplementary file 1). The resulting sequencing reads were quality checked using FastQC version 0.11.3, and contaminating sequencing adapters were removed using seq_crumbs version 0.1.9. Reads were assembled with Spades (Bankevich et al., 2012) version 3.5.0, using different k-mer sizes according to reads length and with the ‘careful’ option to reduce assembly errors; contigs below 200 base pairs were excluded. Resulting assembled contigs were annotated for coding genes, ribosomal RNAs and tRNAs using Prokka (Seemann, 2014) version 1.11. Strains not belonging to the E. coli species were excluded from subsequent analysis after being highlighted by Kraken (Wood and Salzberg, 2014) version 0.10.5. When available, typing information was used to spot incorrect genome sequences due to culture contaminations or other factors; known strains typing was compared to the ones predicted from the genome sequence using mlst version 2.8. Strain names were amended when possible (see the ‘Notes’ column in Supplementary file 1). The ECOR collection was carefully checked for inconsistencies, as it is well known that different ‘versions’ of this collection are circulating in the scientific community (Johnson et al., 2001; Clermont et al., 2015). The genome sequences were further checked for duplicated genomes: strains with highly similar genomes but highly divergent phenotypes (phenotypes S-score correlation below 0.6) were flagged and the genome belonging to the most likely incorrect strain was removed. If the conflict could not be resolved (i.e. using known typing information) both genomes were removed. Highly similar genomes were defined as those genomes whose distance was found to be below 0.001, as measured by mash (Ondov et al., 2016), version 1.1. Despite the draft status of the genomes presented in this study, we observed a similar core genome size as the one computed using a set of 376 complete E. coli genomes downloaded from NCBI’s RefSeq (data not shown).

SNP calling and annotation

Due to the variability in sequencing technologies or lack of the original reads for the already sequenced strains in the collection (N = 373), SNPs were called through a whole genome alignment between each strain and the genome of the reference individual (Escherichia coli str. K-12 substr. MG1655, RefSeq accession: NC_000913.3, strain collection identifier NT12001), using ParSNP (Treangen et al., 2014) version 1.2. Repeated regions in the reference genome were highlighted and masked through nucmer (Kurtz et al., 2004) version 3.1 and Bedtools (Quinlan and Hall, 2010) version 2.26.0. SNPs were then phased, and annotated using SnpEff (Cingolani et al., 2012) version 4.1g.

Pangenome analysis

Genes present in the reference individual but absent in each strain were highlighted by computing hierarchical orthologous group using OMA (Altenhoff et al., 2013) version 1.0.6. Each strain was previously re-annotated using Prokka (Seemann, 2014) version 1.11 to harmonize gene calls.

Phylogenetics

Strains phylogenetic tree was computed using a single ParSNP (Treangen et al., 2014) analysis, which generates a whole genome nucleotide alignment across all strains, which is then used as an input for FastTree (Price et al., 2010) version 2.1.7. The tree was visualized using the ete3 library (Huerta-Cepas et al., 2016a) version 3.0.0.

Phylogenetic distances between pairs of strains has been computed using phylogenetic independent contrasts, in order to account for phylogenetic interdependencies between strains (Felsenstein, 1985). In short, we have first restricted the phenotypic data to those strains and conditions with less than 10% missing data; we then imputed the rest using the average S-score for each strain, using the scikit-learn library (Pedregosa et al., 2011) version 0.17.1. For each condition we then computed the phylogenetic independent contrasts across the internal nodes of the strains’ tree, using the ape, geiger and phytools libraries (Paradis et al., 2004; Harmon et al., 2008Revell, 2012), versions 5.0, 2.0.6 and 0.6–20, respectively. The phenotypic distance between nodes was then computed using the euclidean distance between the phylogenetic independent contrasts across all conditions, using the nadist library version 0.1.0.

Computation of all possible mutations and their effect

The impact of all possible nonsynonymous substitutions on the reference individual has been precomputed to speed up the lookup process. Functional impact of nonsynonymous substitutions has been computed using SIFT (Ng and Henikoff, 2001) version 5.1.1, using uniref50 as a sequence search database. Structural impact of nonsynonymous substitutions has been computed using FoldX (Guerois et al., 2002) version 4; both 3-D structures present in the PDB database and homology models were used. Homology models were created using ModPipe (Pieper et al., 2014) version 2.2.0. Conversion from PDB to Uniprot residues coordinates was derived from the SIFTS (Velankar et al., 2013) database. PDB structures were ‘repaired’ to fix residues’ torsion angles and Van der Waals clashes before computing the impact of non-synonymous variants on structural stability. All the precomputed impacts of all possible nonsynonymous substitutions are available through the mutfunc database (http://mutfunc.com).

Computation of the disruption score

For each strain and each protein coding genes we have computed the overall impact of all nonsynonymous and nonsense substitutions, in a similar approach as the one used for Saccharomyces cerevisiae (Jelier et al., 2011). The output of each predictor (a deleterious probability for SIFT and a ΔΔG value for FoldX) has been converted to the probability of the substitution being neutral p(neutral). Mutations with known impact on the reference individual have been downloaded from Uniprot (UniProt Consortium, 2015, N = 3673) and used to derive such conversion; since only 580 mutations are reported to have a neutral impact, we added all observed nonsynonymous variants affecting known essential genes (as reported in the OGEE database, Chen et al., 2017) to the list of tolerated mutations. The distribution of the negative natural logarithm of the SIFT probability (plus a pseudocount equivalent to the lowest observed SIFT probability) for all the 6460 mutations was binned and the p(neutral) was computed as the proportion of tolerated mutations over the total number of mutations in each bin. A logistic regression curve was then fitted to the binned distribution to derive the conversion between the SIFT probability and p(neutral). For FoldX we used a similar approach, but using the computed ΔΔG value. The fitted logistic regression curves resulted in the following P(neutral)functions:

P(neutralSIFT)=1(1+e(0.625log(SIFT+1.527E4)+1.971))
P(neutralFoldX)=1(1+e(1.465FoldX+1.201))

The P(neutral) value attributed to nonsense mutations was assigned through heuristics: if the new stop codon was found within the last 16 residues of the protein it was given a P(neutral) value of 0.99, reflecting its unlikelihood of disrupting the function of the protein, 0.01 otherwise. Losing a start or a stop codon was given a P(neutral) value of 0.01, as they are very likely to impair protein function.

We gave a P(neutral) value of 0.01 to those genes that were found to be present in the reference individual but absent in the target strain, reflecting the fact that their function is most likely to be absent from the target strain.

We inferred the probability that each gene had its function affected by the ensemble of the substitutions in each strain by computing a disruption score, equivalent to the P(AF) (probability of the function being affected) used in S. cerevisiae study (Jelier et al., 2011).

P(AF)=1i=1kPi(neutral)

Where k is the ensemble of nonsynonymous and nonsense substitutions observed in each gene. When both FoldX and SIFT predictions were available for a given substitution, we used the SIFT prediction only. Variants with relatively high frequency (>=10%) in the strains collection were not considered, as well as those reference genes that are absent in a significant number of strains (>=10%), as we reasoned that they are unlikely to be deleterious given their high observed frequency. Given that many strains of the collection are closely related (e.g. strains derived from the LTEE collection), we clustered them based on phylogenetic distance before applying the filtering. We also didn’t consider variants and absent reference genes shared by all members of the LTEE strain collection, as those variants are present in the collection founder strain and therefore unlikely to affect the evolved clones phenotypes. Disruption scores for all proteins across all strains can be found in Supplementary file 5.

Use of the disruption score as a functional association predictor

We used the proteins pairwise Pearson’s correlation of disruption scores as a predictor of genes functional associations. We used four benchmarking sets: operons, as derived from the DOOR database (Mao et al., 2014), protein complexes and pathways derived from the EcoCyc database (Keseler et al., 2013), and protein-protein interactions derived from a recent yeast two-hybrid experiment (Rajagopala et al., 2014). The distribution of the disruption score correlation for each pair of related genes was compared against the same number of gene pairs randomly drawn from all reference genes. We also assessed the predictive power of the disruption score correlations by drawing a receiving operator characteristics curve (ROC) across each correlation threshold, using the scikit-learn library (Pedregosa et al., 2011) version 0.17.1.

Strains phenotyping

The strains phenotypes were measured in a similar way as the E. coli reverse genetic screen (Nichols et al., 2011). The strain collection was plated in three solid agar plates, each one containing 1536 single colonies, so that each strain was plated at least four times in each plate, each time with different neighboring strains. For each condition we prepared three replicates (each one using a different source plate to reduce batch effects) with the concentrations indicated in Supplementary file 2 and the addition of the Cosmos dye (CAS number 573-58-0) and Coomasie brilliant blue R-250 (CAS number 6104-59-2). The solution stains colonies when biofilm is being developed. Plates were stored in darkness at room temperature, unless otherwise required by the specific condition (i.e. higher temperature), and photographs of the plates were taken until colonies were found to be overgrowing into each other. Most of the conditions (N = 197) were tested at the same time and under the same laboratory conditions as the KEIO KO collection (Herrera-Dominguez et al., unpublished).

A series of colony parameters were extracted from each photograph, using Iris (Kritikos et al., 2017) version 0.9.4.71: colony size, opacity, roundness and color intensity. The most appropriate time point for each condition was determined by imposing a restriction on median colony size; between 1900 and 3600 pixels for the first two plates, and between 1300 and 3600 for the last plate, which contained the strains derived from evolution experiments, which tend to grow less than the natural isolates. The time points passing the first threshold were then sorted by the proportion of colonies with high roundness (>0.8), which is indicative of the overall quality of the plate, proportion of colonies over the minimum median colony size threshold, the spread of the colony size distribution (the lower the better), and mean colony size correlation across replicates.

A series of additional quality control measures were taken on the colony parameters. In order to remove systematic pinning defects, colonies appearing to be missing (colony size of zero pixels) in more than 66% of the tested plates were removed, unless all the internal replicates were found to be missing. Colonies with abnormal circularity were removed, as they were mostly due to incorrect colony recognition by the software: colonies with size below 1000 pixels and circularity below 0.5 and colonies with size above 1000 pixels and circularity below 0.3 were removed. Putative contaminations were spotted and removed through a variance jackknife approach: first, the size of two outermost rows and columns colonies was corrected to match the median of the rest of the plate, then for each strain, each of the four replicates inside the plate was tested whether it contributed to more than 90% of the total colony size variance. If so, the replicate was flagged as a contamination and removed. The same approach was repeated using colony circularity, with a variance threshold of 95%.

The final colony sizes were used as an input for the EMAP algorithm (Collins et al., 2006), with default parameters, in order to derive an S-score, which informs on the deviation of each strain from the expected growth in each condition. Final S-scores were quantile-normalized, and significant phenotypes were highlighted using a 5% FDR correction similar to the one used in the E. coli reverse genetic screen (Nichols et al., 2011), using the statsmodels library version 0.6.1. The phenotypic data is available in Supplementary file 4.

Computation of the conditional score and its assessment

Conditionally essential genes were derived for each condition of the E. coli reverse genetic screen overlapping with the conditions tested on the natural isolates collection. Mutants with a significant growth phenotype were considered to derive the list of conditionally essential genes. The conditional score for each strain, indicating the growth prediction, was computed as follows:

Ss,c=g=1l1EsWg,clog(1Ps,g(AF))

Where s and c represent the strain and condition, respectively, g each conditionally essential gene for condition c (with size l), and Esrepresenting a correction term for the disruption score, in order to remove the effect of phylogenetic distance (Figure 2—figure supplement 1).

Es=1ng=1nlog(1Ps,g(AF))

Where n represents all the reference genes. The term Wg,c is used to weight the contribution of each conditionally essential gene to the conditional score, and it is computed as follows:

Wg,c=log10(Fg,c)CgNc

Where Fg,c is the FDR-corrected p-value of gene g in condition c, Cg is the number of conditions in the chemical genomics screen where gene g shows a significant phenotype, and Nc the total number of tested conditions in the chemical genomics screen.

The conditional score was assessed by computing a Precision-Recall curve, whose area (PR-AUC) was used as a direct measure of the predictive power of the method; the growth phenotypes were considered true positives. The PR curve and its AUC were computed using the scikit-learn library (Pedregosa et al., 2011) version 0.17.1. Three randomization approaches were used to generate control conditional scores: one using strains shuffling, one using conditionally essential gene sets shuffling, and one using random conditionally essential gene sets. Each randomization strategy has been used to generate 10,000 randomized scores, which were scaled to the actual one before assessment.

The conditionally essential gene sets, the conditional score and the PR-AUC values are available in Supplementary file 5.

Association of accessory genes with phenotypes

Accessory genes from the strains collection pangenome were computed from the harmonized genome annotations made by Prokka (Seemann, 2014), using Roary (Page et al., 2015) version 3.6.1. The accessory genes were associated to each condition’s phenotypes using Scoary (Brynildsrud et al., 2016) version 1.4.0, with default parameters. Genes with corrected p-value (Benjamini-Hochberg) of association below 0.05 were considered significant. The enrichment of conditionally essential genes among the significant reference gene hits was assessed through a Fisher’s exact test, as implemented in the SciPy library, version 0.17.0.

Systematic in-silico complementation of conditionally essential genes

The potential to restore growth phenotypes through the introduction of reference alleles was predicted systematically in each strain by changing the disruption score to zero in each conditionally essential gene, and reporting the change in the conditional score : with respect to the maximum possible conditional score:

Ss,cmax=g=1l1EsWg,clog(1Pmax(AF))

Where Pmax(AF) is the maximum disruption score observed across all genes and strains. Any ΔSs,c higher than 1% of Ss,cmax was considered as potentially able to restore a growth phenotype.

Experimental complementation of predicted phenotype-causing genes

In order to experimentally verify our predictions, we introduced the reference (BW25113) gene in a low copy plasmid. For the slt gene, we used the available plasmid from the TransBac library (H. Dose and H. Mori, unpublished resource, Otsuka et al., 2015). For acrA, acrB and glnD, we used the available plasmid from the mobile plasmid library (Saka et al., 2005). We amplified acrAB, soxSR, proAB from BW25113 and ligated into pNTR-SD (the backbone plasmid for the mobile plasmid library). Deletions of acrAB, soxSR, proAB in the reference strain were made using the lambda red recombination approach (Datsenko and Wanner, 2000). The resulting seven plasmids and the two empty plasmid controls were introduced into BW25113 (negative control), in the deletion strains from the Keio collection or constructed by us (positive control) and in the targets strains. All resulting strains were pinned using a Singer Rotor robot in 10 different conditions, on two solid agar plates, so that each strain is pinned at least four times per plate. The plates were incubated at room temperature and multiple photographs were taken until colonies were found to be overgrowing into each other. Iris (Kritikos et al., 2017) version 0.9.7 was used to extract colony size from the pictures.

Code, data and strains collection availability

New genomic sequences have been deposited at the European Nucleotide Archive (ENA) with accession number PRJEB20550.

The source code used to perform the analysis reported here and generate the figures is available as Source code 1 and at the following URLs: https://github.com/mgalardini/screenings, https://github.com/mgalardini/pangenome_variation, https://github.com/mgalardini/ecopredict (copies archived at https://github.com/elifesciences-publications/screenings, https://github.com/elifesciences-publications/pangenome_variation and https://github.com/elifesciences-publications/ecopredict). Code is mostly based on the Python programming language, and using the following libraries: Numpy (Vanvan Derder Walt et al., 2011) version 1.10.4, SciPy version 0.17.0, Pandas (McKinney, 2010) version 0.18.0, Biopython (Cock et al., 2009) version 1.68, scikit-learn (Pedregosa et al., 2011) version 0.17.1, fastcluster (Müllner, 2013) version 1.1.20, statsmodels version 0.6.1, PyVCF version 0.6.8, ete3 (Huerta-Cepas et al., 2016a) version 3.0.0, Matplotlib (Hunter, 2007) version 1.5.1, Seaborn (Waskom et al., 2016) version 0.7.1 and svgutils version 0.2.0.

Genomic and phenotypic data, as well as relevant information on how to obtain the members of the strain collection is available at the following URL: https://evocellnet.github.io/ecoref.

Acknowledgements

We are particularly grateful to the various people providing us with many of the strains of this E. coli genetic reference panel, specifically (in alphabetical order): Alfredo G Torres, Catharina Svanborg, David Clarke, Erick Denamur, Ewa Bok and Pawel Pusz, Isabel Gordo and Lilia Perfeito, Jorg Weinreich and Peter Schierack, KC Huang, Lisa Nolan, Mark Goulian, Mathew Upton, Olin Silander, Richard Lenski, Scott Hultgren and Wanderley Dias da Silveira. We thank Amanda Miguel for helping in the phenotypic screen. We also thank the EMBL Gene Core, and especially Rajna Hercog and Vladimir Benes for the support in genome sequencing. We thank Ewan Birney, Oliver Stegle, KC Huang and Zam Iqbal for critical reading of the manuscript. This work was partially supported by the Sofja Kovaleskaja Award of the Alexander von Humboldt Foundation to ATy and a grant from the Fondation pour la Recherche Médicale (Equipe FRM 2016, DEQ20161136698) to ED.

Funding Statement

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Contributor Information

Athanasios Typas, Email: typas@embl.de.

Pedro Beltrao, Email: pbeltrao@ebi.ac.uk.

Naama Barkai, Weizmann Institute of Science, Israel.

Funding Information

This paper was supported by the following grants:

  • Alexander von Humboldt-Stiftung Sofja Kovaleskaja Award to Athanasios Typas.

  • Fondation pour la Recherche Médicale Equipe FRM 2016, DEQ20161136698 to Erick Denamur.

Additional information

Competing interests

No competing interests declared.

Author contributions

Conceptualization, Formal analysis, Investigation, Methodology, Writing—original draft, Project administration, Writing—review and editing.

Investigation.

Investigation.

Investigation.

Investigation.

Investigation.

Investigation.

Resources, Data curation, Validation.

Resources, Validation, Writing—review and editing.

Conceptualization, Supervision, Funding acquisition, Writing—original draft, Writing—review and editing.

Conceptualization, Supervision, Writing—original draft, Project administration, Writing—review and editing.

Additional files

Source code 1. code used to analyze the genetic and phenotypic data, predict phenotypes and draw all the manuscript figures.
elife-31035-code1.gz (10.6MB, gz)
DOI: 10.7554/eLife.31035.012
Supplementary file 1. detailed list of the members of the strain collection.
elife-31035-supp1.xlsx (79.8KB, xlsx)
DOI: 10.7554/eLife.31035.013
Supplementary file 2. detailed list of conditions on which the strain collection has been tested.
elife-31035-supp2.xlsx (16.1KB, xlsx)
DOI: 10.7554/eLife.31035.014
Supplementary file 3. categorization and mode of action (MoA) of each chemical used in the phenotypic screening.
elife-31035-supp3.xlsx (6.9KB, xlsx)
DOI: 10.7554/eLife.31035.015
Supplementary file 4. s-scores, FDR corrected p-values and binary growth defect matrix of the E. coli phenotypic screening (869 strains across 214 conditions).
elife-31035-supp4.zip (2.6MB, zip)
DOI: 10.7554/eLife.31035.016
Supplementary file 5: .disruption scores, conditional scores and prediction assessments.
elife-31035-supp5.zip (1.2MB, zip)
DOI: 10.7554/eLife.31035.017
Transparent reporting form
DOI: 10.7554/eLife.31035.018

References

  1. 1000 Genomes Project Consortium. Auton A, Brooks LD, Durbin RM, Garrison EP, Kang HM, Korbel JO, Marchini JL, McCarthy S, McVean GA, Abecasis GR. A global reference for human genetic variation. Nature. 2015;526:68–74. doi: 10.1038/nature15393. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. 1001 Genomes Consortium 1,135 Genomes reveal the global pattern of polymorphism in arabidopsis thaliana. Cell. 2016;166:481–491. doi: 10.1016/j.cell.2016.05.063. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Altenhoff AM, Gil M, Gonnet GH, Dessimoz C. Inferring hierarchical orthologous groups from orthologous gene pairs. PLoS One. 2013;8:e53786. doi: 10.1371/journal.pone.0053786. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Atwell S, Huang YS, Vilhjálmsson BJ, Willems G, Horton M, Li Y, Meng D, Platt A, Tarone AM, Hu TT, Jiang R, Muliyati NW, Zhang X, Amer MA, Baxter I, Brachi B, Chory J, Dean C, Debieu M, de Meaux J, Ecker JR, Faure N, Kniskern JM, Jones JD, Michael T, Nemri A, Roux F, Salt DE, Tang C, Todesco M, Traw MB, Weigel D, Marjoram P, Borevitz JO, Bergelson J, Nordborg M. Genome-wide association study of 107 phenotypes in Arabidopsis thaliana inbred lines. Nature. 2010;465:627–631. doi: 10.1038/nature08800. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Ayroles JF, Carbone MA, Stone EA, Jordan KW, Lyman RF, Magwire MM, Rollmann SM, Duncan LH, Lawrence F, Anholt RR, Mackay TF. Systems genetics of complex traits in Drosophila melanogaster. Nature Genetics. 2009;41:299–307. doi: 10.1038/ng.332. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Baba T, Ara T, Hasegawa M, Takai Y, Okumura Y, Baba M, Datsenko KA, Tomita M, Wanner BL, Mori H. Construction of Escherichia coli K-12 in-frame, single-gene knockout mutants: the Keio collection. Molecular Systems Biology. 2006;2:2006.0008. doi: 10.1038/msb4100050. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bankevich A, Nurk S, Antipov D, Gurevich AA, Dvorkin M, Kulikov AS, Lesin VM, Nikolenko SI, Pham S, Prjibelski AD, Pyshkin AV, Sirotkin AV, Vyahhi N, Tesler G, Alekseyev MA, Pevzner PA. SPAdes: a new genome assembly algorithm and its applications to single-cell sequencing. Journal of Computational Biology. 2012;19:455–477. doi: 10.1089/cmb.2012.0021. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Bennett BJ, Farber CR, Orozco L, Kang HM, Ghazalpour A, Siemers N, Neubauer M, Neuhaus I, Yordanova R, Guan B, Truong A, Yang WP, He A, Kayne P, Gargalovic P, Kirchgessner T, Pan C, Castellani LW, Kostem E, Furlotte N, Drake TA, Eskin E, Lusis AJ. A high-resolution association mapping panel for the dissection of complex traits in mice. Genome Research. 2010;20:281–290. doi: 10.1101/gr.099234.109. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Blount ZD. The unexhausted potential of E. coli. eLife. 2015;4:e05826. doi: 10.7554/eLife.05826. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Bodmer W, Bonilla C. Common and rare variants in multifactorial susceptibility to common diseases. Nature Genetics. 2008;40:695–701. doi: 10.1038/ng.f.136. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Boyle EA, Li YI, Pritchard JK. An expanded view of complex traits: from polygenic to omnigenic. Cell. 2017;169:1177–1186. doi: 10.1016/j.cell.2017.05.038. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Brynildsrud O, Bohlin J, Scheffer L, Eldholm V. Rapid scoring of genes in microbial pan-genome-wide association studies with Scoary. Genome Biology. 2016;17:238. doi: 10.1186/s13059-016-1108-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Bush WS, Moore JH. Chapter 11: Genome-wide association studies. PLoS Computational Biology. 2012;8:e1002822. doi: 10.1371/journal.pcbi.1002822. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Chen W-H, Lu G, Chen X, Zhao X-M, Bork P. OGEE v2: an update of the online gene essentiality database with special focus on differentially essential genes in human cancer cell lines. Nucleic Acids Research. 2017;45:D940–D944. doi: 10.1093/nar/gkw1013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Cingolani P, Platts A, Wang leL, Coon M, Nguyen T, Wang L, Land SJ, Lu X, Ruden DM. A program for annotating and predicting the effects of single nucleotide polymorphisms, SnpEff: SNPs in the genome of Drosophila melanogaster strain w1118; iso-2; iso-3. Fly. 2012;6:80–92. doi: 10.4161/fly.19695. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Clermont O, Gordon D, Denamur E. Guide to the various phylogenetic classification schemes for Escherichia coli and the correspondence among schemes. Microbiology. 2015;161:980–988. doi: 10.1099/mic.0.000063. [DOI] [PubMed] [Google Scholar]
  17. Cock PJ, Antao T, Chang JT, Chapman BA, Cox CJ, Dalke A, Friedberg I, Hamelryck T, Kauff F, Wilczynski B, de Hoon MJ. Biopython: freely available Python tools for computational molecular biology and bioinformatics. Bioinformatics. 2009;25:1422–1423. doi: 10.1093/bioinformatics/btp163. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Collins SR, Schuldiner M, Krogan NJ, Weissman JS. A strategy for extracting and analyzing large-scale quantitative epistatic interaction data. Genome Biology. 2006;7:R63. doi: 10.1186/gb-2006-7-7-r63. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Datsenko KA, Wanner BL. One-step inactivation of chromosomal genes in Escherichia coli K-12 using PCR products. PNAS. 2000;97:6640–6645. doi: 10.1073/pnas.120163297. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Dietzl G, Chen D, Schnorrer F, Su KC, Barinova Y, Fellner M, Gasser B, Kinsey K, Oppel S, Scheiblauer S, Couto A, Marra V, Keleman K, Dickson BJ. A genome-wide transgenic RNAi library for conditional gene inactivation in Drosophila. Nature. 2007;448:151–156. doi: 10.1038/nature05954. [DOI] [PubMed] [Google Scholar]
  21. Dowell RD, Ryan O, Jansen A, Cheung D, Agarwala S, Danford T, Bernstein DA, Rolfe PA, Heisler LE, Chin B, Nislow C, Giaever G, Phillips PC, Fink GR, Gifford DK, Boone C. Genotype to phenotype: a complex problem. Science. 2010;328:469. doi: 10.1126/science.1189015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Edwards SL, Beesley J, French JD, Dunning AM. Beyond GWASs: illuminating the dark road from association to function. The American Journal of Human Genetics. 2013;93:779–797. doi: 10.1016/j.ajhg.2013.10.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Felsenstein J. Phylogenies and the comparative method. The American Naturalist. 1985;125:1–15. doi: 10.1086/284325. [DOI] [PubMed] [Google Scholar]
  24. Guerois R, Nielsen JE, Serrano L. Predicting changes in the stability of proteins and protein complexes: a study of more than 1000 mutations. Journal of Molecular Biology. 2002;320:369–387. doi: 10.1016/S0022-2836(02)00442-4. [DOI] [PubMed] [Google Scholar]
  25. Harmon LJ, Weir JT, Brock CD, Glor RE, Challenger W. GEIGER: investigating evolutionary radiations. Bioinformatics. 2008;24:129–131. doi: 10.1093/bioinformatics/btm538. [DOI] [PubMed] [Google Scholar]
  26. Hart T, Chandrashekhar M, Aregger M, Steinhart Z, Brown KR, MacLeod G, Mis M, Zimmermann M, Fradet-Turcotte A, Sun S, Mero P, Dirks P, Sidhu S, Roth FP, Rissland OS, Durocher D, Angers S, Moffat J. High-resolution CRISPR screens reveal fitness genes and genotype-specific cancer liabilities. Cell. 2015;163:1515–1526. doi: 10.1016/j.cell.2015.11.015. [DOI] [PubMed] [Google Scholar]
  27. Hillenmeyer ME, Fung E, Wildenhain J, Pierce SE, Hoon S, Lee W, Proctor M, St Onge RP, Tyers M, Koller D, Altman RB, Davis RW, Nislow C, Giaever G. The chemical genomic portrait of yeast: uncovering a phenotype for all genes. Science. 2008;320:362–365. doi: 10.1126/science.1150021. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Huerta-Cepas J, Serra F, Bork P. ETE 3: reconstruction, analysis, and visualization of phylogenomic data. Molecular Biology and Evolution. 2016a;33:1635–1638. doi: 10.1093/molbev/msw046. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Huerta-Cepas J, Szklarczyk D, Forslund K, Cook H, Heller D, Walter MC, Rattei T, Mende DR, Sunagawa S, Kuhn M, Jensen LJ, von Mering C, Bork P. eggNOG 4.5: a hierarchical orthology framework with improved functional annotations for eukaryotic, prokaryotic and viral sequences. Nucleic Acids Research. 2016b;44:D286–D293. doi: 10.1093/nar/gkv1248. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Hunter JD. Matplotlib: A 2D Graphics Environment. Computing in Science & Engineering. IEEE Computer Society; 2007. pp. 90–95. [Google Scholar]
  31. Jelier R, Semple JI, Garcia-Verdugo R, Lehner B. Predicting phenotypic variation in yeast from individual genome sequences. Nature Genetics. 2011;43:1270–1274. doi: 10.1038/ng.1007. [DOI] [PubMed] [Google Scholar]
  32. Johnson JR, Delavari P, Stell AL, Prats G, Carlino U, Russo TA. Integrity of archival strain collections: The ECOR collection. ASM News-American Society for Microbiology. 2001;67:288–289. [Google Scholar]
  33. Kamath RS, Fraser AG, Dong Y, Poulin G, Durbin R, Gotta M, Kanapin A, Le Bot N, Moreno S, Sohrmann M, Welchman DP, Zipperlen P, Ahringer J. Systematic functional analysis of the Caenorhabditis elegans genome using RNAi. Nature. 2003;421:231–237. doi: 10.1038/nature01278. [DOI] [PubMed] [Google Scholar]
  34. Keseler IM, Mackie A, Peralta-Gil M, Santos-Zavaleta A, Gama-Castro S, Bonavides-Martínez C, Fulcher C, Huerta AM, Kothari A, Krummenacker M, Latendresse M, Muñiz-Rascado L, Ong Q, Paley S, Schröder I, Shearer AG, Subhraveti P, Travers M, Weerasinghe D, Weiss V, Collado-Vides J, Gunsalus RP, Paulsen I, Karp PD. EcoCyc: fusing model organism databases with systems biology. Nucleic Acids Research. 2013;41:D605–D612. doi: 10.1093/nar/gks1027. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Kritikos G, Banzhaf M, Herrera-Dominguez L, Koumoutsi A, Wartel M, Zietek M, Typas A. A tool named Iris for versatile high-throughput phenotyping in microorganisms. Nature Microbiology. 2017;2:17014. doi: 10.1038/nmicrobiol.2017.14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Kulshreshtha S, Chaudhary V, Goswami GK, Mathur N. Computational approaches for predicting mutant protein stability. Journal of Computer-Aided Molecular Design. 2016;30:401–412. doi: 10.1007/s10822-016-9914-3. [DOI] [PubMed] [Google Scholar]
  37. Kurtz S, Phillippy A, Delcher AL, Smoot M, Shumway M, Antonescu C, Salzberg SL. Versatile and open software for comparing large genomes. Genome Biology. 2004;5:R12. doi: 10.1186/gb-2004-5-2-r12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Lehner B. Genotype to phenotype: lessons from model organisms for human genetics. Nature Reviews Genetics. 2013;14:168–178. doi: 10.1038/nrg3404. [DOI] [PubMed] [Google Scholar]
  39. Li XZ, Plésiat P, Nikaido H. The challenge of efflux-mediated antibiotic resistance in Gram-negative bacteria. Clinical Microbiology Reviews. 2015;28:337–418. doi: 10.1128/CMR.00117-14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Liti G, Carter DM, Moses AM, Warringer J, Parts L, James SA, Davey RP, Roberts IN, Burt A, Koufopanou V, Tsai IJ, Bergman CM, Bensasson D, O'Kelly MJ, van Oudenaarden A, Barton DB, Bailes E, Nguyen AN, Jones M, Quail MA, Goodhead I, Sims S, Smith F, Blomberg A, Durbin R, Louis EJ. Population genomics of domestic and wild yeasts. Nature. 2009;458:337–341. doi: 10.1038/nature07743. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Lukjancenko O, Wassenaar TM, Ussery DW. Comparison of 61 sequenced Escherichia coli genomes. Microbial Ecology. 2010;60:708–720. doi: 10.1007/s00248-010-9717-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Mao X, Ma Q, Zhou C, Chen X, Zhang H, Yang J, Mao F, Lai W, Xu Y. DOOR 2.0: presenting operons and their functions through dynamic and integrated views. Nucleic Acids Research. 2014;42:D654–D659. doi: 10.1093/nar/gkt1048. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. McKinney W. Data Structures for Statistical Computing in PythonIn. Proceedings of the 9th Python in Science Conference; 2010. pp. 51–56. [Google Scholar]
  44. Medini D, Donati C, Tettelin H, Masignani V, Rappuoli R. The microbial pan-genome. Current Opinion in Genetics & Development. 2005;15:589–594. doi: 10.1016/j.gde.2005.09.006. [DOI] [PubMed] [Google Scholar]
  45. Murakami S, Nakashima R, Yamashita E, Matsumoto T, Yamaguchi A. Crystal structures of a multidrug transporter reveal a functionally rotating mechanism. Nature. 2006;443:173–179. doi: 10.1038/nature05076. [DOI] [PubMed] [Google Scholar]
  46. Müllner D. fastcluster: fast hierarchical, agglomerative clustering routines forrandpython. Journal of Statistical Software. 2013;53 doi: 10.18637/jss.v053.i09. [DOI] [Google Scholar]
  47. Ng PC, Henikoff S. Predicting deleterious amino acid substitutions. Genome Research. 2001;11:863–874. doi: 10.1101/gr.176601. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Nichols RJ, Sen S, Choo YJ, Beltrao P, Zietek M, Chaba R, Lee S, Kazmierczak KM, Lee KJ, Wong A, Shales M, Lovett S, Winkler ME, Krogan NJ, Typas A, Gross CA. Phenotypic landscape of a bacterial cell. Cell. 2011;144:143–156. doi: 10.1016/j.cell.2010.11.052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Ondov BD, Treangen TJ, Melsted P, Mallonee AB, Bergman NH, Koren S, Phillippy AM. Mash: fast genome and metagenome distance estimation using MinHash. Genome Biology. 2016;17:132. doi: 10.1186/s13059-016-0997-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Otsuka Y, Muto A, Takeuchi R, Okada C, Ishikawa M, Nakamura K, Yamamoto N, Dose H, Nakahigashi K, Tanishima S, Suharnan S, Nomura W, Nakayashiki T, Aref WG, Bochner BR, Conway T, Gribskov M, Kihara D, Rudd KE, Tohsato Y, Wanner BL, Mori H. GenoBase: comprehensive resource database of Escherichia coli K-12. Nucleic Acids Research. 2015;43:D606–D617. doi: 10.1093/nar/gku1164. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Page AJ, Cummins CA, Hunt M, Wong VK, Reuter S, Holden MT, Fookes M, Falush D, Keane JA, Parkhill J. Roary: rapid large-scale prokaryote pan genome analysis. Bioinformatics. 2015;31:3691–3693. doi: 10.1093/bioinformatics/btv421. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Paradis E, Claude J, Strimmer K. APE: Analyses of Phylogenetics and Evolution in R language. Bioinformatics. 2004;20:289–290. doi: 10.1093/bioinformatics/btg412. [DOI] [PubMed] [Google Scholar]
  53. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M. “Scikit-Learn: machine learning in python.”. Journal of Machine Learning Research : JMLR. 2011;12:2825–2830. [Google Scholar]
  54. Pieper U, Webb BM, Dong GQ, Schneidman-Duhovny D, Fan H, Kim SJ, Khuri N, Spill YG, Weinkam P, Hammel M, Tainer JA, Nilges M, Sali A. ModBase, a database of annotated comparative protein structure models and associated resources. Nucleic Acids Research. 2014;42:D336–D346. doi: 10.1093/nar/gkt1144. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Price MN, Dehal PS, Arkin AP. FastTree 2--approximately maximum-likelihood trees for large alignments. PLoS One. 2010;5:e9490. doi: 10.1371/journal.pone.0009490. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Price MN, Wetmore KM, Waters RJ, Callaghan M, Ray J. Deep annotation of protein function across diverse bacteria from mutant phenotypes. BioRxiv. 2016 doi: 10.1101/072470. [DOI]
  57. Quinlan AR, Hall IM. BEDTools: a flexible suite of utilities for comparing genomic features. Bioinformatics. 2010;26:841–842. doi: 10.1093/bioinformatics/btq033. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Rajagopala SV, Sikorski P, Kumar A, Mosca R, Vlasblom J, Arnold R, Franca-Koh J, Pakala SB, Phanse S, Ceol A, Häuser R, Siszler G, Wuchty S, Emili A, Babu M, Aloy P, Pieper R, Uetz P. The binary protein-protein interaction landscape of Escherichia coli. Nature Biotechnology. 2014;32:285–290. doi: 10.1038/nbt.2831. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Revell LJ. phytools: an R package for phylogenetic comparative biology (and other things) Methods in Ecology and Evolution. 2012;3:217–223. doi: 10.1111/j.2041-210X.2011.00169.x. [DOI] [Google Scholar]
  60. Ryan O, Shapiro RS, Kurat CF, Mayhew D, Baryshnikova A, Chin B, Lin ZY, Cox MJ, Vizeacoumar F, Cheung D, Bahr S, Tsui K, Tebbji F, Sellam A, Istel F, Schwarzmüller T, Reynolds TB, Kuchler K, Gifford DK, Whiteway M, Giaever G, Nislow C, Costanzo M, Gingras AC, Mitra RD, Andrews B, Fink GR, Cowen LE, Boone C. Global gene deletion analysis exploring yeast filamentous growth. Science. 2012;337:1353–1356. doi: 10.1126/science.1224339. [DOI] [PubMed] [Google Scholar]
  61. Saka K, Tadenuma M, Nakade S, Tanaka N, Sugawara H, Nishikawa K, Ichiyoshi N, Kitagawa M, Mori H, Ogasawara N, Nishimura A. A complete set of Escherichia coli open reading frames in mobile plasmids facilitating genetic studies. DNA Research. 2005;12:63–68. doi: 10.1093/dnares/12.1.63. [DOI] [PubMed] [Google Scholar]
  62. Seeger MA, Schiefner A, Eicher T, Verrey F, Diederichs K, Pos KM. Structural asymmetry of AcrB trimer suggests a peristaltic pump mechanism. Science. 2006;313:1295–1298. doi: 10.1126/science.1131542. [DOI] [PubMed] [Google Scholar]
  63. Seemann T. Prokka: rapid prokaryotic genome annotation. Bioinformatics. 2014;30:2068–2069. doi: 10.1093/bioinformatics/btu153. [DOI] [PubMed] [Google Scholar]
  64. Tenaillon O, Barrick JE, Ribeck N, Deatherage DE, Blanchard JL, Dasgupta A, Wu GC, Wielgoss S, Cruveiller S, Médigue C, Schneider D, Lenski RE. Tempo and mode of genome evolution in a 50,000-generation experiment. Nature. 2016;536:165–170. doi: 10.1038/nature18959. [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. Tenaillon O, Skurnik D, Picard B, Denamur E. The population genetics of commensal Escherichia coli. Nature Reviews Microbiology. 2010;8:207–217. doi: 10.1038/nrmicro2298. [DOI] [PubMed] [Google Scholar]
  66. Thusberg J, Olatubosun A, Vihinen M. Performance of mutation pathogenicity prediction methods on missense variants. Human Mutation. 2011;32:358–368. doi: 10.1002/humu.21445. [DOI] [PubMed] [Google Scholar]
  67. Treangen TJ, Ondov BD, Koren S, Phillippy AM. The Harvest suite for rapid core-genome alignment and visualization of thousands of intraspecific microbial genomes. Genome Biology. 2014;15:524. doi: 10.1186/s13059-014-0524-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  68. UniProt Consortium UniProt: a hub for protein information. Nucleic acids research. 2015;43:D204–212. doi: 10.1093/nar/gku989. [DOI] [PMC free article] [PubMed] [Google Scholar]
  69. van der Walt S, Colbert SC, Varoquaux G. The NumPy Array: A structure for efficient numerical computation. Computing in Science & Engineering. 2011;13:22–30. doi: 10.1109/MCSE.2011.37. [DOI] [Google Scholar]
  70. Velankar S, Dana JM, Jacobsen J, van Ginkel G, Gane PJ, Luo J, Oldfield TJ, O'Donovan C, Martin MJ, Kleywegt GJ. SIFTS: structure integration with function, taxonomy and sequences resource. Nucleic Acids Research. 2013;41:D483–D489. doi: 10.1093/nar/gks1258. [DOI] [PMC free article] [PubMed] [Google Scholar]
  71. Waskom M, Botvinnik O, Drewokane PH, David YH, Lukauskas S. v0.7.1Seaborn. 2016 doi: 10.5281/zenodo.54844. [DOI]
  72. Weinstein JN, Collisson EA, Mills GB, Shaw KR, Ozenberger BA, Ellrott K, Shmulevich I, Sander C, Stuart JM, Cancer Genome Atlas Research Network The cancer genome atlas pan-cancer analysis project. Nature genetics. 2013;45:1113–1120. doi: 10.1038/ng.2764. [DOI] [PMC free article] [PubMed] [Google Scholar]
  73. Welter D, MacArthur J, Morales J, Burdett T, Hall P, Junkins H, Klemm A, Flicek P, Manolio T, Hindorff L, Parkinson H. The NHGRI GWAS Catalog, a curated resource of SNP-trait associations. Nucleic Acids Research. 2014;42:D1001–D1006. doi: 10.1093/nar/gkt1229. [DOI] [PMC free article] [PubMed] [Google Scholar]
  74. Wood DE, Salzberg SL. Kraken: ultrafast metagenomic sequence classification using exact alignments. Genome Biology. 2014;15:R46. doi: 10.1186/gb-2014-15-3-r46. [DOI] [PMC free article] [PubMed] [Google Scholar]
  75. Yang J, Benyamin B, McEvoy BP, Gordon S, Henders AK, Nyholt DR, Madden PA, Heath AC, Martin NG, Montgomery GW, Goddard ME, Visscher PM. Common SNPs explain a large proportion of the heritability for human height. Nature Genetics. 2010;42:565–569. doi: 10.1038/ng.608. [DOI] [PMC free article] [PubMed] [Google Scholar]

Decision letter

Editor: Naama Barkai1

In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.

Thank you for submitting your article "Phenotype inference in an Escherichia coli strain panel" for consideration by eLife. Your article has been reviewed by two peer reviewers, and the evaluation has been overseen by Naama Barkai as the Senior Editor and Reviewing Editor. The reviewers have opted to remain anonymous.

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

As you will see below, the reviewers appreciated the extensive analysis and believe that the data will be a useful resource for the community. Please see below technical issues that were raised by the reviewers, which should be addressed in full, as you revise your submission for the Tools and Resources section.

Reviewer #1:

This paper does an extensive analysis and prediction of phenotypes caused by mutations in a large E. coli strain panel evaluated over several hundred conditions.

Overall, my concern with this paper applies similarly to many large-scale, high-throughput works I see: The authors have done so much, and are trying to present so many different results and approaches in one paper, that it's difficult to evaluate what exactly was achieved. This paper is either a monumental achievement or a collection of trivial results, and I find it difficult to distinguish between the two possibilities.

I am tentatively willing to give the authors the benefit of the doubt, but I have specific questions and concerns that would strengthen my confidence in the work.

- "We detected no strong positive correlation" this is a strange sentence, in particular if it then lists a significant negative correlation. I think the authors need to explain first why they would expect a positive correlation.

Also, with regards to these correlations, because of the phylogenetic dependency among strains they should be calculated on phylogenetic independent contrasts. Alternatively, it would be better to not report a p value than to report an incorrect one.

- Figure 3A: The precision-recall curves don't look particularly impressive. They rapidly decay to a precision below 0.5, at which point more cases are called incorrectly than correctly.

- Figure 3B: This figure is not properly explained. First, what are the small gray dots? Second, what are the three thick gray dots labeled as shuffled strains, shuffled sets, and random sets? Each randomization should provide a distribution which then is compared to one PR-AUC value. If the true PR-AUC value is rare under reshuffling, then it is significant. This relationship is not in any way visible in the figure.

- Figure 4: Again, the precision-recall curve looks pretty bad. I grant that there's some enrichment of phenotypes towards the top end of the predicted phenotype scale, but the prediction quality is not impressive and it's not clear to me whether a trivial predictor (e.g., one only evaluating whether an amino acid has been changed towards a highly biochemically dissimilar one) could do just as well.

- Section on "Experimental validation of causal variants": It is not entirely clear to me what has been achieved here. If a mutation knocks out an essential gene then clearly that defect can be rescued via complementation. So the achievement would have to be to predict that a gene is essential only under specific conditions. Is that what was done? And if so, how? Was essentiality simply observed, by noting that strains with mutations in certain genes wouldn't grow under specific conditions? Or was it somehow inferred from some other data?

- "with prior knowledge on gene essentiality" I'm not entirely convinced that gene essentiality matters all that much. A mutation in a non-essential gene can have a substantial fitness effect, and a mutation in an essential gene that doesn't fully disable that gene's function may have only a minor fitness effect.

- It is unclear to me whether the code underlying the mutfunc database has been provided or not. The database looks nice, but I'm generally wary of any bioinformatics tool that I can't download and run on my own machine. The authors provide several pipelines (https://github.com/mgalardini/screenings, https://github.com/mgalardini/pangenome_variation, https://github.com/mgalardini/ecopredict), but none seems to perform the task of the mutfunc database.

Reviewer #2:

We have read the study entitled "Phenotype inference in an Escherichia coli strain panel" by Galardini et al. This study performs phenotypic screens for 894 E. coli strains across 214 growth conditions. Most of the strains (696) have full genome sequence available. Gene-level functional predictions were correct in up to 38% of cases. This dataset will become an important and useful resource for the systems biology and microbiology community. We would appreciate of the authors addressed our minor comments below:

We commend the authors for putting all of their data in easily accessible format on github with links out to all of the available assemblies and phenotype measurements. This inclusion will ensure that the resource is easy to use for the research community. However, one question remains regarding the new annotations for each strain. Are these available anywhere? It was difficult to find them both on the web or in the authors’ supplemental files.

The authors previously used this approach on yeast with a median predictive performance of 0.76 (Jelier et al. 2011), but were only able to reliably predict 38% of the conditions tested in this instance. Could the authors comment on why their performance was lower?

Why did the authors use ParSNP to compute phylogeny rather than other methods (such as MLST, whole genome alignment, etc). The authors look for a correlation between phylogeny and genotype. Could the lack of correlation be due to assumptions made when calculating the phylogeny?

Why did the authors choose to incubate the most of the conditions at room temperature rather than the more common 37C? Might this significantly impact their results (i.e. would growth capabilities change if incubated at 37C?).

How do the authors define the "accessory genes/genome". Are these simply all genes that are not part of the core genome? Could the authors comment on the completeness of the genomic sequences in the data set?

Were SNPs only called for sequences in common with the reference E. coli K12 genome? Would that potentially explain some of the discrepancies between the model predictions and the experimental outcome?

Does the term "mechanistic" appropriately describe the method used? Especially when one considers disruption scores in many areas of a genome and the conditional essentiality of genes?

eLife. 2017 Dec 27;6:e31035. doi: 10.7554/eLife.31035.024

Author response


Reviewer #1:

This paper does an extensive analysis and prediction of phenotypes caused by mutations in a large E. coli strain panel evaluated over several hundred conditions.

Overall, my concern with this paper applies similarly to many large-scale, high-throughput works I see: The authors have done so much, and are trying to present so many different results and approaches in one paper, that it's difficult to evaluate what exactly was achieved. This paper is either a monumental achievement or a collection of trivial results, and I find it difficult to distinguish between the two possibilities.

We can understand the reviewer’s reluctance towards large-scale projects and of course we hope to convince him/her that this is not simply a large collection of data. We are in fact trying to ask a very fundamental biological question: given all that we currently know about E. coli biology, can we predict how a specific strain of E. coli will grow in a given condition based on its genome? It is not trivial at all to be able to take a genome sequence of an E. coli strain and determine in which conditions this strain is going to grow poorly in. By asking this question we are testing the limits of our knowledge of how DNA variation affects phenotypes. Working towards these predictions implies the capacity to encapsulate in computational models all of this understanding. We upfront admit that we are not yet fully capable of making such predictions, but we also think it is very important to test these limits and to challenge ourselves and the scientific community to improve on what we have achieved in this manuscript.

In particular, we have used the knowledge on E. coli gene function as defined by the chemical genomic screens done in one particular strain, the lab E. coli K-12. We then make the assumption that a gene that is essential for growth in a given condition in the K-12 strain will also be essential for growth in the same condition in any other strain of E. coli. Based on this knowledge we then compute the effects of non-synonymous variants from genomes of different strains, using folding and sequence conservation algorithms. This allows us to predict which strains will have growth defects in specific conditions. Our predictions are significantly better than random and we can show in the complementation assays that we are in fact predicting the genes that are causing the growth defects in the right conditions. Of course, there is much room for improvement and many ways to try to extend the predictions made here as we discuss in the manuscript. For example, we have not yet taken into account mutations that could alter the expression of the genes. As we and others incorporate such additional information the predictive models will become even more accurate. At the same time, there is also great opportunity for new biology from the deficiencies in the predictions. As we mentioned in the manuscript we see cases where loss of function of a gene is known to cause a condition specific growth defect in E. coli K-12 but the same is not observed in specific strains. We and others can now try to understand why this is the case.

Finally, in order to ask this fundamental question we also needed the genomes of many strains of E. coli and their growth phenotypes across a large number of conditions as well. This large-scale data gathering effort is in itself useful as a resource, but we would like to re-emphasize that the main focus of this paper is on the fundamental biological question we are asking.

I am tentatively willing to give the authors the benefit of the doubt, but I have specific questions and concerns that would strengthen my confidence in the work.

- "We detected no strong positive correlation" this is a strange sentence, in particular if it then lists a significant negative correlation. I think the authors need to explain first why they would expect a positive correlation.

Also, with regards to these correlations, because of the phylogenetic dependency among strains they should be calculated on phylogenetic independent contrasts. Alternatively, it would be better to not report a p value than to report an incorrect one.

We agree with the reviewer that we have failed in properly explaining what our expectations were with respect to this analysis. The message we wanted to convey was that if all genetic variation contributed equally to phenotypic variation we would observe a strong positive correlation between these two variables, which is not the case. The overall negative correlation presented in the main text is present merely because of the highly genetically similar strains coming from the LTEE collection (Long Term Evolution Experiment). Given this confounding factor and the suggestion to use phylogenetic independent contrasts to account for the strains’ dependencies, we have changed Figure 1E (now Figure 2E). As suggested by the reviewer we are now deriving the correlation between genetic and phenotypic distance using the phylogenetic independent contrasts computed along the core genome tree of the strains. We show that we did not observe a positive correlation between genetic and phenotypic distance (Pearson’s r: 0.07), confirming the assumption that not all genetic variation contributes equally to differentiate phenotypes. We have also updated the Materials and methods section (“Phylogenetics”) to indicate how this analysis has been performed. We have also removed the last two panels of Figure 1—figure supplement 1.

- Figure 3A: The precision-recall curves don't look particularly impressive. They rapidly decay to a precision below 0.5, at which point more cases are called incorrectly than correctly.

We thank the reviewer for raising this point. While the curves displayed in Figure 3A are merely part of the schematic used to explain the prediction and scoring strategies, they mimic some of the real curves we observed in our data, and therefore this comment is relevant and important. We address the concerns of the reviewer with respect to the drop in precision and general aspect of the curves in the comment about Figure 3B and Figure 4. In a nutshell, we would like to point out that the predictions are indeed significantly better than the various randomization strategies we have proposed.

- Figure 3B: This figure is not properly explained. First, what are the small gray dots? Second, what are the three thick gray dots labeled as shuffled strains, shuffled sets, and random sets? Each randomization should provide a distribution which then is compared to one PR-AUC value. If the true PR-AUC value is rare under reshuffling, then it is significant. This relationship is not in any way visible in the figure.

We agree with the reviewer that the visualization we chose doesn’t provide sufficient information and, most importantly, hides the relationship between the distribution of the actual predictions versus the three randomization strategies we have adopted. We therefore decided to remove the thick gray dots that corresponded to the median results of the randomizations (“shuffled strains”, “shuffled sets”, and “random sets”) and leave the small grey dots, which represent the PR-AUC value of each condition separately. To better illustrate the randomisations we have added a panel to Figure 3 (panel C), showing an example condition (Clindamycin 3 μg/ml). As suggested by the reviewer we now show the distribution of the randomizations and in a red line the result for the actual predictor for this condition. In the figure legend we have indicated the q-value related to the likelihood that the actual predictions has been drawn from the distribution of each randomization. Furthermore, we have added a new supplementary figure (Figure 3—figure supplement 2), which shows for each condition a similar plot as the one presented in Figure 3C. We have also updated the Results section to better explain this point. We believe that these changes will help the readers understanding why the predictions presented in this work are relevant.

- Figure 4: Again, the precision-recall curve looks pretty bad. I grant that there's some enrichment of phenotypes towards the top end of the predicted phenotype scale, but the prediction quality is not impressive and it's not clear to me whether a trivial predictor (e.g., one only evaluating whether an amino acid has been changed towards a highly biochemically dissimilar one) could do just as well.

The points raised by the reviewer are important. Indeed, the precision in the two example conditions drops at relatively low recall values. It is important to point out that the number of strains with growth defects in a given condition are a small fraction of the total and it is therefore a very unbalanced set of true and false positives. Achieving high precision is particularly challenging in such an unbalanced set. Nevertheless the rankings we predict are noticeably better than expected by random chance. Author response image 1 shows the comparison against the predicted score computed using random conditionally essential gene sets. In that respect the comments about Figure 3B allowed us to make this significance more evident.

Author response image 1.

Author response image 1.

Finally, we would like to point out how the choice of showing a precision-recall curve as the sole visual indicator of our prediction accuracy may be actually undermining the value of our approach. Author response image 1 shows two additional metrics for the two example conditions presented in Figure 4A (the “bootstrap” curves representing the average performance of 100 predictions using random gene sets):

As evident from the plots in Author response image 1, the predictions made in these two conditions are significantly better than using random gene sets. This is especially obvious in the ROC curve, where the randomizations are very similar to a random predictor (ROC AUC ~= 0.5), as opposed to very high ROC AUC for both conditions for the “real” predictions (~0.76). We have now updated Figure 4A to show how the predictions are significantly different than the “random sets” bootstraps. We have also added a new panel to Figure 4B to include the ROC curve.

While we recognize that our predictions are far from perfect, we believe they represent an important demonstration that is indeed possible to combine chemical genomics data from the reference K-12 strain with mutation data to infer conditional growth defects for other strains. In particular, leveraging the information on condition-dependent essentiality is key to the significance of our predictions; if we were to only use the predicted impact of non-synonymous variants we would not be able to deliver condition specific predictions. We have expanded our Discussion to point out this important conclusion of our study.

The reviewer asked if the same predictions could be achieved with a “trivial” variant predictor such as one that evaluates the effect of mutations based on amino-acids categories. Our predictions are a combination of variant effect predictions with gene function information from the K-12 reference strains. We assume that the reviewer is here asking about how much information is added by the interpretation of the variant effects only, keeping the gene function information unchanged. We have tested a more trivial variant effect predictor for the impact of nonsynonymous substitutions as suggested by the reviewer, and as can be seen from the three plots in Author response image 2, its predictive power is inferior to the one we propose.

Author response image 2.

Author response image 2.

In short, we have divided the 20 amino acids into 8 categories: hydrophobic (A, I, L, M, V), hydrophobic (aromatic) (F, W, Y), polar (N, Q, S, T), acidic (D, E), basic (R, H, K), cysteine (C), proline (P) and glycine (G). We then have assigned a P(neutral) value of 0.99 (tolerated) for amino acid substitutions within each category, and 0.01 (deleterious) otherwise. Again, all the other predictors (accessory genes and stop codons) have been kept the same as our proposed predictor, including the use of conditionally essential genes. The two top plots show the precision recall curves for 2 conditions comparing variant effect prediction approaches. The boxplots in the bottom figure summarize the results across all conditions that have at least 5% of strains with significant growth defects. We believe that this test demonstrates that there is added value in using variant effect predictors that take into account conservation and structural information as we are able to achieve higher precision. We have decided not to include this result in the main text, as introducing an additional predictor might generate confusion in the reader.

Finally, as indicated in the Discussion, we of course concede that these predictions have room for improvement, and suggest future directions on how to do so.

- Section on "Experimental validation of causal variants": It is not entirely clear to me what has been achieved here. If a mutation knocks out an essential gene then clearly that defect can be rescued via complementation. So the achievement would have to be to predict that a gene is essential only under specific conditions. Is that what was done? And if so, how? Was essentiality simply observed, by noting that strains with mutations in certain genes wouldn't grow under specific conditions? Or was it somehow inferred from some other data?

Looking at this and the subsequent comment by the reviewer, we suspect that we might have failed in properly explaining the approach we have taken and the terminology we are using. We therefore have updated the Introduction, Results and Discussion sections to make it easier to understand what exactly our approach consists of. In short, we have used the information on gene conditional essentiality derived from pre-existing chemical genomics screens from the E. coli lab strain K-12. We therefore know which genes are essential for growth in K-12 in a given condition, and we can therefore focus on interpreting genetic variation on those genes only, as they would be more likely to influence growth if a mutation has an impact on its function in another E. coli strain. This assumes that a given gene that is essential for growth in a condition in the E. coli strain K-12 will still be equally important for that same condition in any other strain of E. coli. As we described in the Discussion one of the potential reasons why we have achieved only moderate success could very well be because the genes essential for growth under specific conditions in the K-12 strain may not be essential for growth in those same conditions for other strains. While we have not proved this in this study, this would be an extremely important finding that would highlight the plasticity of gene function.

Going back to the experimental validation of our model, based on the mutations that we observed in the genome of a given strain we can then predict: 1) if a mutation will severely impact on the function of a giving gene; 2) if that gene is known to be essential for growth in a specific condition (in K-12). We can then predict a strain that has a damaging mutation in a gene essential to grow in a certain condition should also grow poorly in that condition. Purely from the genome sequence of a strain we can then rank all mutated genes as the most likely mutated gene that is causing a grown defect in a specific condition. These sets of complementation experiments show that we can in fact predict why a given strain of E. coli is growing poorly in a specific condition.

This is the reason why we believe we have demonstrated the long-term potential of our approach to pave the way for precision genetic intervention strategies. Our approach is specific and, to a certain extent, accurate. We believe that Figure 5C is an appropriate demonstration of this statement; in the experiments we have carried out we observe a clear difference between those complementations that were predicted to restore growth in specific conditions versus those that were not predicted to have an impact.

- "with prior knowledge on gene essentiality" I'm not entirely convinced that gene essentiality matters all that much. A mutation in a non-essential gene can have a substantial fitness effect, and a mutation in an essential gene that doesn't fully disable that gene's function may have only a minor fitness effect.

We believe that we have addressed this concern in the reply to the previous question. We indeed believe that conditional essentiality (i.e. essential for a given condition) matters, and we believe to have demonstrated it when showing how our predictions compare to the randomisations with shuffled and random gene sets (Figure 3C, 4A and B and Figure 3—figure supplement 2). Again, we thank the reviewer for pointing out how we have failed in making these points clear. We hope that our revised text improves how we explain this key concept.

- It is unclear to me whether the code underlying the mutfunc database has been provided or not. The database looks nice, but I'm generally wary of any bioinformatics tool that I can't download and run on my own machine. The authors provide several pipelines (https://github.com/mgalardini/screenings, https://github.com/mgalardini/pangenome_variation, https://github.com/mgalardini/ecopredict), but none seems to perform the task of the mutfunc database.

The mutfunc database contains all the precomputed non-synonymous substitutions and their predicted impact using a variety of computational approaches. As we described in the manuscript it includes predictions made with SIFT and FoldX which are tools that are available as standalones. A lot of work on our part went into running the calculations but the database itself is a front-end for all of the precomputed scores that we have obtained after many days of total compute time. No actual computation is done by the mutfunc server as all possible variants along the genome are already pre-computed. For these reasons there is no standalone tool as such that we can create. As the reviewer may imagine, the results of the pre-computed consequences of all possible variants along the genomes of the 3 species currently in mutfunc are also not something that can be easily provided as a standalone tool. The webservice we provide in mutfunc is not limited to searching small numbers of variants. Full genome variants can be queried with a very fast response time, since it is essentially looking up the pre-computed scores. There is also no limit in getting the outputs of large numbers of queries. We have added a few more details in the Materials and methods section (“Computation of all possible mutations and their effect”) regarding how we have run Sift and FoldX. If the reviewer found specific limitations from the webserver we can try to address them. We did try very hard to make all of the work associated with this publication available even before the work is published. To further facilitate the re-use of our results, we have now added the SIFT and FoldX scores for all the nonsynonymous variants observed in the strain collection to the Ecoref website, in the download section (https://evocellnet.github.io/ecoref/download/).

Reviewer 2:

[…] We commend the authors for putting all of their data in easily accessible format on github with links out to all of the available assemblies and phenotype measurements. This inclusion will ensure that the resource is easy to use for the research community. However, one question remains regarding the new annotations for each strain. Are these available anywhere? It was difficult to find them both on the web or in the authors’ supplemental files.

We also believe that making this resource easy to use for the community is a crucial point to our manuscript and long-term vision. We recognize that the gene annotations are important for the use of this resource and we therefore thank the reviewer to raise this point. We have released the annotated genomes in the ecoref website (“strains” section). The annotation for each strains are available in the standard and popular GFF3 format.

The authors previously used this approach on yeast with a median predictive performance of 0.76 (Jelier et al. 2011), but were only able to reliably predict 38% of the conditions tested in this instance. Could the authors comment on why their performance was lower?

If we understood the reviewer’s comment correctly, there might have been a misunderstanding. We are not the authors of the 2011 yeast paper (Jelier et al., 2011). As we highlighted in the Introduction, Jelier and coauthors have applied their predictive approach to a much smaller cohort of 15 strains across 20 conditions, most of which are related to metabolism. We therefore postulate that one potential reason for our lower performance when compared to the previous study is the larger variety in the type of conditions in our screen, but also in their strength. In fact some chemicals have been tested with multiple concentrations (e.g. amoxicillin), some of which might be suboptimal to elicit a strong response from the tested strains, making a prediction potentially more difficult. Another reason may be the larger genetic diversity in bacteria in general, and of our library in particular. We have added a short sentence in the discussion to explain the potential causes of these differences.

Why did the authors use ParSNP to compute phylogeny rather than other methods (such as MLST, whole genome alignment, etc). The authors look for a correlation between phylogeny and genotype. Could the lack of correlation be due to assumptions made when calculating the phylogeny?

We believe that we might have failed to properly explain the procedure we followed to obtain the strains’ phylogeny. As the reviewers suggests, a whole genome alignment is one of the best way to obtain a phylogeny. In fact, this is exactly what ParSNP does: it aligns the whole genome of the input genomes and it then feeds such alignment to FastTree (Price et al., 2010) which computes the final phylogeny. We have updated the methods section to explicitly state how the method works.

Why did the authors choose to incubate the most of the conditions at room temperature rather than the more common 37C? Might this significantly impact their results (i.e. would growth capabilities change if incubated at 37C?).

We did the strain collection phenotyping together with that of the KEIO gene-knockout collection – latter is part of a manuscript currently in preparation (Herrera-Dominguez et al.). We opted for room temperature because then we could simultaneously measure two readouts: growth and biofilm formation. E. coli forms biofilms only at 30C or below. We have still not analyzed the biofilm readout, but will do in the future.

How do the authors define the "accessory genes/genome". Are these simply all genes that are not part of the core genome? Could the authors comment on the completeness of the genomic sequences in the data set?

As suggested by the reviewer, we indeed consider those genes that are not present in the “core genome” (shared by all strains) as part of the so-called “accessory genome”. We are in fact following the definitions set up in the seminal pangenomics paper (Medini et al., 2005).

Regarding the completeness of the genome sequences, we believe that this is a fair point of discussion whenever draft genomes are involved. However, we are confident that the draft genomes we have collected/generated as part of this manuscript are of sufficient quality for nearly all genes to be correctly annotated. As a way of example we have compared the pangenome size rarefaction curves (computed using Roary, Page et al., 2015) between a cohort of 376 complete E. coli genomes vs. the genomes from our collection. In short, we have downloaded the complete genomes from NCBI’s refseq using the ncbi-genome-download script and annotated using prokka. We have then computed their pangeome using roary, using the same options that were used for the strains presented in our study. We have then plotted the pangenome size rarefaction curves, based on 10 shufflings of the strain’s order (Author response image 3).

Author response image 3.

Author response image 3.

The rarefaction curves are indeed very similar to each other; as expected the complete genomes set has a slightly higher number of total genes. This is expected given the completeness of these genomes. The difference with the strains presented in our study is however negligible. Furthermore, the core genome size of the RefSeq strains is composed of 2889 genes, versus 2815 for the genomes presented here, when considering 376 genomes. We have updated the methods section to report on the completeness of the presented genomes.

Were SNPs only called for sequences in common with the reference E. coli K12 genome? Would that potentially explain some of the discrepancies between the model predictions and the experimental outcome?

As the conditionally essential genes are derived from the reference strain (E. coli K12), we decided to call genetic variants with respect to this strain. As we have addressed in the discussion, we also believe that taking into account the impact of non-synonymous variants and the presence/absence of genes not present in the reference strain could improve our predictions. However, at the moment we cannot reliably assign a function to accessory genes in order to extend the predictive models in this way.

Does the term "mechanistic" appropriately describe the method used? Especially when one considers disruption scores in many areas of a genome and the conditional essentiality of genes?

We decided to adopt the term “mechanistic” as a contrast to more classical GWAS-like studies, which are based on statistical inference. Since the impact of nonsynonymous variants can be related to a protein’s functional constraints (i.e. SIFT) or structural stability (i.e. FoldX), we believe that the term “mechanistic” is appropriate. We have however added a short sentence in the Introduction explaining what we intend when using this term.

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    Source code 1. code used to analyze the genetic and phenotypic data, predict phenotypes and draw all the manuscript figures.
    elife-31035-code1.gz (10.6MB, gz)
    DOI: 10.7554/eLife.31035.012
    Supplementary file 1. detailed list of the members of the strain collection.
    elife-31035-supp1.xlsx (79.8KB, xlsx)
    DOI: 10.7554/eLife.31035.013
    Supplementary file 2. detailed list of conditions on which the strain collection has been tested.
    elife-31035-supp2.xlsx (16.1KB, xlsx)
    DOI: 10.7554/eLife.31035.014
    Supplementary file 3. categorization and mode of action (MoA) of each chemical used in the phenotypic screening.
    elife-31035-supp3.xlsx (6.9KB, xlsx)
    DOI: 10.7554/eLife.31035.015
    Supplementary file 4. s-scores, FDR corrected p-values and binary growth defect matrix of the E. coli phenotypic screening (869 strains across 214 conditions).
    elife-31035-supp4.zip (2.6MB, zip)
    DOI: 10.7554/eLife.31035.016
    Supplementary file 5: .disruption scores, conditional scores and prediction assessments.
    elife-31035-supp5.zip (1.2MB, zip)
    DOI: 10.7554/eLife.31035.017
    Transparent reporting form
    DOI: 10.7554/eLife.31035.018

    Articles from eLife are provided here courtesy of eLife Sciences Publications, Ltd

    RESOURCES