Homology-Aware Phylogenomics at Gigabase Scales

Abstract

Obstacles to inferring species trees from whole genome data sets range from algorithmic and data management challenges to the wholesale discordance in evolutionary history found in different parts of a genome. Recent work that builds trees directly from genomes by parsing them into sets of small -mer strings holds promise to streamline and simplify these efforts, but existing approaches do not account well for gene tree discordance. We describe a “seed and extend” protocol that finds nearly exact matching sets of orthologous -mers and extends them to construct data sets that can properly account for genomic heterogeneity. Exploiting an efficient suffix array data structure, sets of whole genomes can be parsed and converted into phylogenetic data matrices rapidly, with contiguous blocks of -mers from the same chromosome, gene, or scaffold concatenated as needed. Phylogenetic trees constructed from highly curated rice genome data and a diverse set of six other eukaryotic whole genome, transcriptome, and organellar genome data sets recovered trees nearly identical to published phylogenomic analyses, in a small fraction of the time, and requiring many fewer parameter choices. Our method’s ability to retain local homology information was demonstrated by using it to characterize gene tree discordance across the rice genome, and by its robustness to the high rate of interchromosomal gene transfer found in several rice species.

Construction of a phylogenetic tree from even a single gene is “hard” from the standpoint of algorithm theory (Felsenstein 2004), yet trees are now being inferred from entire transcriptomes (Wickett et al. 2014) or genomes (Neafsey et al. 2015) at a scale up to a million times larger than this—across taxa as diverse in scope as land plants (Wickett et al. 2014), viral epidemics (Worobey et al. 2014), and cancer tumors (Zhao et al. 2016). In addition to data set size, genomic data add complexities of annotation, orthology detection, and sequence alignment upstream of tree construction, and discordant gene trees caused by gene duplication, deep coalescence, and lateral transfer detected downstream of tree construction (Maddison et al. 1997; Fontaine et al. 2015; Liu et al. 2015; Nater et al. 2015). Phylogenomic analysis pipelines have accordingly become parameter-rich mash-ups of diverse algorithms and toolkits (Misof et al. 2014; Wickett et al. 2014; Neafsey et al. 2015; Prum et al. 2015; Zhao et al. 2016). Moreover, although some of these upstream components contribute substantial information about genomes, they can also introduce their own biases into phylogenetic inference proper. For example, Zwickl et al. (2014) highlighted annotation errors in rice phylogenomics that introduced “block shifts” into multiple sequence alignments of genes. These affected the overall frequency spectrum of gene trees and the final species tree reconstruction. Recent methods that avoid annotation, alignment, and even assembly, by recoding genomes as sets of short -mer strings, have shown promise to streamline and speed up inference and make its assumptions more robust and reproducible (Gardner and Hall 2013; Bertels et al. 2014; Chan et al. 2014; Leimeister and Morgenstern 2014; Fan et al. 2015; Haubold et al. 2015).

Using -mers sampled from sequences has been a mainstay of several core bioinformatic tools, especially alignment and database search, for many years (Gusfield 1997). Early attempts to use -mers for phylogenetic inference did not perform well (Hohl and Ragan 2007), which led to a wave of modifications to allow inexact -mer matching (Leimeister and Morgenstern 2014), to include genome coordinate information between matches (Haubold et al. 2015), and to correct -mer-based distances for multiple hits (Fan et al. 2015; Haubold et al. 2015). However, almost all these approaches estimate pairwise distances from numbers of shared -mers, and distance-based phylogenetic methods lose information about homology, especially positional homology, during data reduction, which may decrease statistical robustness in tree reconstruction (Huelsenbeck 1995). Even if the magnitude of this impact is small, a potentially more significant concern is that by reducing two genomes to a single pairwise distance, fine scale signal about discordant phylogenetic histories across the genome is discarded, which is inadvisable given the widespread occurrence of such discordance (Pollard et al. 2006; White et al. 2009; Zwickl et al. 2014; Nater et al. 2015).

To keep the speed and simplicity of -mer based approaches but retain information about positional homology, we combined and extended several well-tested ideas in new ways (Gardner and Hall 2013; Leimeister and Morgenstern 2014; Fan et al. 2015; Haubold et al. 2015) and leveraged recent improvements in engineering of a key data structure (Rajasekaran and Nicolae 2014). From a set of genomes, which may be at various stages of assembly, our algorithm builds short multiple sequence alignments, or “-mer blocks,” starting from approximately matching -mer “seeds” (Fig. 1) and adding adjacent short flanking sequences. Because a -mer block is typically too short to contain sufficient phylogenetic information for tree construction itself, it can be pooled by gene, contig, scaffold, chromosome, and so forth, into one or more concatenated alignments, or “supermatrices.” At the limit, all -mer blocks discovered across the genome can be grouped into one supermatrix, but it will often be more informative to construct sets of supermatrices in which smaller pools of -mer blocks reflect the local coordinates in the genome. This allows discovery of different gene tree histories across the genome, species tree inference based on sets of “gene trees” (Liu et al. 2015; Edwards et al. 2016), and other inferences about reticulation, hybridization, or introgression (Huson et al. 2010).

Figure 1.

Seed and extend strategy to construct -mer blocks. Step 1: a set of identical -mers is found in taxa A, B, and D. Step 2: this -mer block is extended in the taxon dimension by finding additional -mers that have at most mismatches (here ; the black x’s) relative to the -mers found in Step 1; here block extended to include taxon E. Step 3: the four-taxon -mer block is then extended in the genome dimension upstream and downstream of each -mer by a short distance, , nucleotides. Together these two extensions yield an ungapped alignment suitable for phylogenetic analysis.

The layout of the article is as follows. First, we describe the algorithm and its implementation details (reserving a full description of its relationship to existing methods for discussion). Next we describe simulation experiments designed to characterize the distribution of -mer blocks that may be expected in genome-scale data sets, to predict properties of the resulting phylogenetic data sets. To complement this, we also examine the genomic context of -mer blocks discovered in a well-annotated set of complete genome sequences for rice and its wild relatives in Oryza. Then, we examine the actual performance of the method in seven whole genome data sets at various levels of sequence divergence, by comparing results we obtain with published work for these taxa. This part is undertaken with supermatrix comparisons, which still provides a useful benchmark in the literature. However, we return to the discovery of gene tree discordance by examining collections of gene trees built from the Oryza data, and comparing the frequency distribution of discordant alternative trees inferred by our method compared to a more conventional approach using annotated and aligned genes. Finally, we consider limitations of the method and possible extensions.

MATERIALS AND METHODS

Overview of Algorithm for Finding Homologous k-mer Blocks

Our algorithm uses a “seed and extend” strategy to build sets of orthologous sequences in related species. The seed consists of a set of exact matching -mers, where is chosen large enough so that a -mer is unique in its own genome (to avoid paralogy), but small enough that it is present in at least taxa (Fig. 1). The seed is then extended in taxon space by using an algorithm that efficiently finds -mers in additional taxa that have at most mismatches relative to the -mers in the original block, where is small relative to . Together these -mers form an ungapped alignment. This is then extended in genome space by appending and prepending nucleotides upstream and downstream to it, where is of length on the order of . This final ungapped alignment is called a “-mer block.” By choosing relatively large, but keeping and small we limit inclusion of paralogous sequences, cap sequence divergence, and increase the reliability of the ungapped alignment as an approximation to full optimization-based multiple sequence alignment (Gusfield 1997).

Implementation

To scale this algorithm to gigabase genomes, running time and memory usage are both critical. To find the initial exact-matching -mer seeds, the sequences, each of length , are concatenated into a string, , of length . If reverse complements are included, these are simply concatenated to the end of this string, now with length 2. The string is stored as a memory-efficient-sorted suffix array containing the coordinates of all (or 2 suffixes. This data structure requires bytes/nucleotide. This array is constructed by a new, fast sorting method that is highly scalable (Rajasekaran and Nicolae 2014), having worst case run times of log and usually much better than this in practice. Once the suffix array is sorted, exact -mer matches form contiguous blocks in the array. The entire data set can then be processed merely by traversing the array from beginning to end, checking to see if -mers are (i) unique in their genomes, (ii) present in genomes, and (iii) not overlapping with any -mer blocks that have already been found.

The first extension to include additional taxa with up to mismatches is done by “filtration” (Pevzner and Waterman 1995). For any -mer that matches at most times with another -mer, there must exist at least one substring of length that matches exactly between the two. Thus, mismatches can be found by looking for exact matches of length in the suffix array, extending them the appropriate distance in the genome sequence and checking to see if indeed there are or fewer mismatches. Lookups in the suffix array for an arbitrary -mer are done efficiently with a binary search. -mers that satisfy the mismatch criteria are also checked for uniqueness and lack of overlap with other blocks. Extension to add the flanking sequences of each -mer is achieved trivially by fetching the coordinates of the -mers in the -mer block and referring back to the original stored genome sequence, .

Genomes may be input as a set of chromosomes, scaffolds, contigs, or other assembly units. The start and stop coordinates of each are maintained so that -mer blocks can later be pooled according to these assembly units if desired. Low complexity -mers are detected from the frequency spectrum of three-mer frequencies (Morgulis et al. 2006), and excluded to limit fruitless enumeration of matches in highly repetitive regions.

A software implementation hakmer (“omology-ware k-mers”) is free and open source C/C code (http://sourceforge.net/projects/hakmer/). Generic 64 bit C code for the suffix array library is also available (https://github.com/mariusmni/radixSA64).

Parameters Affecting -mer Block Discovery

Here we define several terms to let us quantify properties of the data sets produced by building -mer blocks. These properties depend on the sequence input and the parameter choices in hakmer, including -mer length, ; the maximum number of mismatches, ; minimum required taxonomic coverage, ; and width of flanking sequence, . The output, consisting of -mer blocks, can be characterized by three quantities. The first is data use efficiency, , which is the fraction of the bases in the input that are present in the -mer blocks in the output. The second is taxon coverage, , the number of elements out of the set of sequences possibly found among the -mer blocks that actually are present, since some taxa may not contain all -mers found in other taxa in a -mer block. It is guaranteed to be at least / but may be more. Taxon coverage is an important predictor of impacts of missing data on phylogenomic inference (Sanderson et al. 2010, 2011, 2015).

Low taxon coverage can induce “terraces” in phylogenetic inference, which are sets of trees with identical likelihood or parsimony scores (Sanderson et al. 2011, 2015). For a given collection of -mer blocks and its pattern of taxon coverage, it is possible to compute the size of the terrace in which any particular tree is imbedded. More generally, however, the probability that terraces will exist is a function of the coverage, the number of taxa, and number of -mer blocks. In particular, the number of blocks needed to ensure less than a 5% chance that terraces exist for any unrooted tree is given by

$$\begin{array}{c}{B}_{\mathrm{m}\mathrm{i}\mathrm{n}}=-\mathrm{l}\mathrm{o}\mathrm{g}((N-3)/0.05)/\mathrm{l}\mathrm{o}\mathrm{g}(1-{\rho }^4)\end{array}$$

(Sanderson et al. 2010, theorem 2). Based on this idea, we define a third quantity, the k-mer block differential , , which is positive when there are sufficient number of blocks discovered to avoid terraces and negative when terraces are more likely.

Expected Distributions of -mer Blocks: Simulations

To assay the impact of algorithm parameters on the distribution of -mer blocks discovered, genome sequences were simulated on a star phylogeny with 50 leaves using an HKY model in Seq-Gen 1.3.3 (Rambaut and Grassly 1997). Sequence lengths were set to 1 Mb, and sequence divergence between root and each leaf was set to range from 0.01 to 0.10 substitutions/site, with equal divergences for all edges. A star phylogeny is the worst case scenario for -mer block discovery since all sequences are maximally divergent from each other. These data sets were then run in hakmer across a range of parameter values to characterize patterns in their -mer block structure (Figs. 2–4).

Figure 2.

Effect of -mer length on number of -mer blocks discovered (closed shapes) or number of paralogs (open shapes) in simulations. Circles: sequence ; squares: sequence .

Figure 4.

Log block differential, , versus minimum taxon coverage, , in simulations, for two levels of sequence divergence, , and three levels of allowed mismatches, .

Distribution of -mer Blocks in Oryza Complete Genome Sequences

We used genome annotations for Oryza species (downloaded from the Gramene website: ftp://ftp.ensemblgenomes.org/pub/plants/release-27/gff3/) to identify the genomic context for each -mer block. Custom PERL scripts were used to match -mer coordinates with chromosome number and other features identified in the GFF files. In particular, we checked whether -mers were associated with genes and, more strictly, with coding sequences of protein-coding genes. We also tracked whether each sequence from a species was found on the same chromosome in different species or, alternatively, whether interchromosomal transfers had occurred.

Phylogenomic Analyses of Seven Genome-Scale Data Sets

To gauge downstream performance of our algorithm in reconstructing species trees from genome data, we constructed seven genome data sets spanning different time depths in the tree of life, different classes of genomic data, and input sizes ranging from 31 million to 9.4 billion base pairs and from 8 to 663 taxa (Table 1). These included two plant organellar genome data sets, a whole transcriptome data set for the angiosperm order Caryophyllales, and four whole genome data sets, three from angiosperms and one from Anopheles mosquitoes. In our first analyses, we focused on comparing global supermatrices built from -mers across the genome to results on supermatrix phylogenomic analyses in the published work. Our data sets were constructed to streamline data retrieval from GenBank and maximize taxon sampling, and our taxon samples, therefore, did not match exactly taxon sampling in the larger published analyses.

Table 1.

Genomic data sets used in this study (sorted by increasing data set size)

Clade	Data	Time depth (Ma)	No. of taxa	No. of bases (millions)	No. of sequences	Data source
Land plants	Mitochondrial genomes	450	93	31.3	93	GenBank
Angiosperms	Plastid genomes	139	663	98.3	663	GenBank
Caryophyllales	Transcriptomes	107	67	1388.2	1.61 million	Dryad
Oryza	Whole genomes	15	11	3731.9	20,071	Gramene rel. 27
Anopheles	Whole genomes	100	17	3871.0	148,163	GenBank
Fabaceae (Papilionoideae)	Whole genomes	60	8	5168.4	130,672	GenBank
Eudicots	Whole genomes	136	24	9402.8	298,907	Phytozome 10.3

That is, genomes, chromosomes, scaffolds, contigs, treated as disjoint assembly units; ages from Sanderson and Doyle (2001), Magallón et al. (2015), Tang et al. (2010), Neafsey et al. (2015), and Lavin et al. (2005).

Genome sequences were input without any preprocessing. For each data set, a variety of parameter sets were examined to identify an optimal set of run conditions that would minimize paralogy, maximize data use efficiency, and minimize terrace problems (Table 2). The reverse complement option was used in all data sets except for the plastid genome data sets, where the (usual) presence of a large inverted repeat was frequently bypassed by the algorithm’s protocol to avoid paralogy. Thus, ignoring reverse complements actually increased the amount of data extracted.

Table 2.

Selected -mer block algorithm parameter settings and resulting data set characteristics obtained for each genome data set

Clade		(index			-mer blocks	Alignment length
Land plant mitochondrial	93	24 (0)	2	10	2279	145,856	0.25
Angiosperm plastid	663	28 (5)	2	100	1711	116,348	0.51
Caryophyllales	67	28 (0)	2	15	684	46,512	0.58
Oryza	11	32 (0)	0	11	35,646	2,566,512	1.00
Anopheles	17	32 (0)	2	12	38,648	2,782,656	0.89
Fabaceae	8	32 (1)	2	4	33,609	2,419,848	0.82
Eudicots	24	28 (0)	2	5	31,897	2,168,996	0.36

Index indicates the numeral in the supermatrix file name corresponding to this run and these parameters (see Supplementary Materials available on Dryad); Undefined, because .

Phylogenetic Tree Construction

Sets of -mer blocks were concatenated as supermatrices, either across all blocks identified from the genome sequence data, or in some cases at more local scales within genomes, determined by genome coordinates. All phylogenetic trees were inferred using maximum likelihood implemented in RAxML v. 8.04 (Stamatakis 2014), with the default rapid hill climbing algorithm and a GTRGAMMA model used for data sets with taxa and the GTRCAT model used for the two genome data sets with taxa. Bootstrap estimates of support for clades were obtained using the RAxML -b option (full, slow bootstrap), except for the two taxon-rich data sets which used the faster -x bootstrap option (Supplementary Figs. 1–8 available on Dryad at http://datadryad.org/resource/doi:10.5061/dryad.96b0h).

Discovery and Characterization of Intra-genomic Discordance

We constructed sets of gene trees in the Oryza data set by concatenating neighboring -mer blocks into data sets of blocks each, where was 25, 100, or 1000. Bootstrap majority rule ML trees for the 1426 data sets for set were constructed using RAxML (raxmlHPC-AVX -m GTRGAMMA -x seed2 -# 100 -s alignmentFile -n outfilesname -p seed1). Each of these data sets comprised bp alignments. The discordance pattern among these gene trees was quantified and visualized as a consensus network in SplitsTree v. 4.10 (Kloepper and Huson 2008), with a threshold setting of 0.05 on split frequencies. This diagram captures the set of alternative gene tree splits that occur in at least 5% of all input gene trees.

To compare these results to a benchmark, we inferred the same kind of consensus network for the 6015 gene trees assembled for the same taxa in Stein J.C. et al. (submitted for publication). That set comprised alignments of whole gene regions (introns, exons, and flanking gene sequence), aligned with PRANK (Loytynoja and Goldman 2008), for which optimal trees were constructed using ML in GARLI (Zwickl 2006), with short length branches collapsed into polytomies. We compared both the graph structure of the consensus network and the split frequencies along sets of parallel edges in the network (Fig. 5).

Figure 5.

Similarity of Oryza gene tree discordance patterns across the whole genome inferred using our -mer block method versus conventional phylogenomic pipeline. Diagram is a consensus network of gene trees (Huson et al. 2010) (threshold 0.05), and has the same topology whether constructed from 1426 sets of 25 contiguous -mer blocks or from trees inferred from alignments of 6015 annotated rice genes (Stein J.C. et al., submitted for publication). Branch lengths are labeled with pairs of support values obtained in the -mer data set (left number) and phylogenomic data set (right), indicating the frequency of splits supported among the collection of gene trees.

Interchromosomal Transfers in Oryza

For the Oryza analysis, each -mer block was recoded as an ordered list of 11 integers, such that, for each, the first integer is the chromosome number for the block’s sequence in Leersia, the second the chromosome number in O. barthii, and so on in lexical order of the 11 taxon names: for example, 1-1-1-1-1-1-1-1-1-3-1 (chromosome 1 in Leersia, chromosome 1 in O. barthii, etc.). Then this list of lists was sorted according to chromosome, and then within chromosome by the start coordinate position of the Leersia-mer block sequence. Runs of consecutive identical integer patterns were identified for any pattern having more than one chromosome number: these represent potentially homologous sequence transferred between chromosomes. The length of these runs in the list was then translated into corresponding genome coordinates determined from the start positions of the -mer sequences in Leersia. Finally, the five longest of these runs was extracted for further analysis, which corresponded to runs spanning 250,000 nt in Leersia (Supplementary Table 1 available on Dryad). Phylogenetic trees were reconstructed using RAxML (as described above) for the concatenated sets of -mer blocks for each of the five runs of -mer blocks (Supplementary Fig. 8 available on Dryad).

RESULTS

Distribution of k-mer Blocks: Simulation Results

Simulation experiments showed that the length of -mers must be large enough to make matches within the same genome due to paralogy (or chance) low, but if was too large, few -mer blocks having at least taxa were found (Fig. 2). Data use efficiency, , was very high for low levels of sequence divergence (), but dropped off quickly above that (Fig. 3). However, the reduction in at higher sequence divergences was ameliorated by reducing , the minimum required taxon coverage, and increasing the number of mismatches allowed, (Fig. 3).

Figure 3.

Data use efficiency versus sequence divergence in simulations at differing levels of taxon coverage and mismatches. Squares: ; circles: ; triangles: . Open shapes: ; closed shapes: .

If is less than , however, there will likely be -mer blocks with data missing for some taxa, which can introduce terraces of trees with equal optimality scores (Sanderson et al. 2011, 2015). However, the probability of terraces emerging from our protocol, estimated from the log block differential, , is high only when there is a combination of high sequence divergence and high minimum taxon coverage. These conditions lead to a drop in the number of -mer blocks found, making terraces more likely for a given level of missing data. However, increasing data use efficiency, reducing , or allowing more mismatches reduces the probability of terraces (Fig. 4).

Distribution of -mer Blocks: Genomes of Oryza

Of the 35,646 -mer blocks having unique exact matches in all 10 species of Oryza (plus outgroup, Leersia), 95.5% of these mapped to assembled chromosomes in all 11 species. Of these, 76.6% of blocks were found on homologous chromosomes in all species, with the remaining 23.4% having a nonhomologous chromosome in at least one species. Most of the latter were in O. nivara and O. meridionalis. These putative interchromosomal transfers often involved contiguous runs of -mer blocks, with the largest spanning some 415 kb (as measured in Leersia). Furthermore, the intrachromosomal locations of -mer blocks were not random. Among the -mer blocks mapping to chromosomes, 88% of -mers proper were located within genes, and 71% of the -mer blocks consisted of -mers located in genes in all 11 species. Most of the latter (70%) were localized within genes to CDS (coding) regions, as opposed to UTRs or introns. On average only 34% of genome length is annotated as genic in these 11 species, so -mer blocks are clearly enriched in these more conserved regions.

Phylogenomic Analyses of Seven Genome-Scale Data Sets

Running times to construct data sets for further phylogenetic analysis ranged from a few minutes to a few hours for the largest input. The latter required substantial memory ( GB), but this is comparable to the memory required to assemble its genomes from raw sequence read data in the first place, but with trivial running times by comparison. Weeks or months of conventional informatics processing upstream of tree building were reduced to minutes to hours by sidestepping annotation and alignment.

Data use efficiency ranged from 40% in the plastid genome analysis to 0.1% in the transcriptome data, but the number of -mer blocks retained by the algorithm remained large, ranging from in the smaller organellar genomes and transcriptome data sets to in all the whole genome data sets (Table 2). In the largest, eudicot, data set 18.3 million nucleotides were present in the final concatenated “supermatrix” of -mer blocks (24 taxa 2.16 million bp, not counting missing data from partial taxon coverage), despite this representing just 0.2% of the original sequence data in these genomes.

Phylogenetic trees constructed by maximum likelihood methods using supermatrices of all -mer blocks in each data set recovered trees nearly identical to published trees in all seven data sets, including close agreement with bootstrap estimates of statistical significance of clades (Supplementary Figs. 1–8 available on Dryad). The few exceptions were revealing. For example, a few placements of taxa in the eudicot whole genome tree (Supplementary Fig. 7 available on Dryad) are at odds with widely cited relationships based largely on plastid genome data (Soltis et al. 2011), but these apparent oddities are actually quite consistent with recent phylogenomic studies using nuclear or mitochrondrial data (Sun et al. 2015). See Appendix for full discussion of each case study.

Discovery and Characterization of Intragenomic Discordance

A high level of intragenomic phylogenetic discordance in the Oryza data, not detectable in the supermatrix analysis of all data (Supplementary Fig. 4 available on Dryad), was easily detectable by pooling small contiguous sets of -mer blocks to build local phylogenies of different genomic regions. The spectrum of 1426 local trees, each built from a concatenation of 25 neighboring -mer blocks, was characterized using a consensus network (Fig. 5; Huson et al. 2010), which indicated that most of this conflict is within the species group closest to O. sativa, especially involving the South American species O.glumaepatula, which is likely to have undergone extensive introgression (Stein J.C. et al., submitted for publication). The consensus network structure matches exactly the network derived from an independent set of 6015 gene trees constructed using conventional phylogenomic methods for the same taxa from the same raw genome sequence data (Stein J.C. et al., submitted for publication), and the frequency distribution of conflicting splits was very similar between the two (Fig. 5).

A second example supports the overall robustness of the -mer method to structural changes in the genome. We followed up on the discovery of significant numbers of interchromosomal gene transfers by reconstructing phylogenetic trees for the five transfers having sequence lengths nt (Supplementary Table 1 available on Dryad). In each case, despite genome sequence having been translocated to another chromosome in at least one species, the trees obtained from the orthologous regions on these different chromosomes were consistent with trees from genomic regions not exhibiting these transfers, mirroring in particular the discordant placement of O. glumaepatula seen in Figure 5 (see also Supplementary Fig. 8 available on Dryad).

DISCUSSION

In this article, we developed and tested a method for quickly building phylogenetic data sets from whole genomes at large scale. The core of the method is the enumeration of -mer blocks, which are sets of nearly exact matching -mers and their flanking sequence. If chosen correctly, these have a high probability of orthology. The method can find tens of thousands of -mer blocks in whole genomes in minutes to hours of computing time, avoiding compute-expensive data processing steps of annotation and alignment, and controlling for limitations caused by sequence divergence and missing data. Genome sequence data can be input “as is” from assemblies at contig, scaffold, or whole chromosome scales, or any mixture thereof.

Our approach to testing the performance of this method was two-fold: first, we used simulation together with highly annotated genome test data from Oryza to discover the properties of sets of -mer blocks generated by our protocol; and second, we evaluated the performance of tree inference per se from these -mer blocks by analysis of seven diverse genomic data sets and comparison to published phylogenetic trees for the same taxa. A key finding of our simulations was that -mer block discovery will eventually be degraded at high levels of sequence divergence through a combination of finding fewer -mers blocks, and, in those that are found, having matches to fewer taxa (and hence more missing data). However, the simulations also indicated the extent to which these problems could be ameliorated by changing , increasing the number of allowed mismatches, , and increasing the tolerance for -mer blocks with missing taxa, but doing so in such a way that problems with phylogenetic terraces are avoided (Sanderson et al. 2011, 2015). Not surprisingly, when we examined where -mer blocks were found in a well-annotated test case of Oryza complete genome sequence, they are clearly concentrated in conserved coding regions, but they are not exclusively found there. Thus, to the extent that conventional annotation-based phylogenomic work tends to focus exclusively on protein-coding genes, this method is able to go beyond these regions to include other highly conserved parts of the genome.

Analyses of seven empirical studies suggested that the method works well over a range of phylogenomic scales at various time depths in the tree of life. To some extent this reflected trade-offs between average substitution rates in different kinds of genomes and the size of those genomes. Conserved plant mitochondrial and plastid genomes, for example, could be pushed to time depths of hundreds of millions of years, when combined with small mismatch allowances, , because their average substitution rates are so slow. Though relatively small genomes, these rates and parameter settings still led to discovery of enough -mer blocks to infer a high-quality tree. In whole nuclear genome data sets, the faster average rates might be anticipated to make up for the difficulty in finding -mer blocks at higher sequence divergences, and indeed this was true. There were no issues discovering vast numbers of -mer blocks in rice and Anopheles, the ages of which are only –15 Ma, but it was also still possible to find a reasonable number in angiosperms at time depths ranging from 50 to 100 Ma.

We judged performance by comparing simple supermatrix data sets obtained by concatenating all -mer blocks to results of supermatrix analyses published in the literature. Topologies and support values were remarkably consistent between published trees relying on conventional annotation/alignment data processing and those we obtained using -mers blocks. The very few areas of disagreement were almost all mirrored by alternative data sets and analyses found in the literature, especially those obtained from different genomes, such as plastid versus nuclear genome data sets for angiosperms (Soltis et al. 2011; Sun et al. 2015). Taxon samples were different between our trees and trees from the literature. We opted to sample the largest sets of whole genome sequences currently in GenBank, rather than the exact sample from the article in the literature. This made comparisons using something like a tree–tree distance problematic, as we would have to delete many taxa unique to one or the other tree. The extensive exposition in the Appendix is meant to leverage our experience with some of these phylogenetic problems and provides a hopefully more interpretable and falsifiable set of conclusions about performance.

However, the results of supermatrix analyses in phylogenomics studies may well typically only reflect a “first order” approximation to the species phylogeny that is most accurate in regions of the tree in which biological sources of discordance such as incomplete lineage sorting are absent. To obtain a better, “second order” approximation it is now clear that reconstruction of individual gene trees, by which we mean any appropriately local region in a genome, is necessary. Numerous methods of inferring species trees from such gene trees are now available (Liu et al. 2015), but rather than examining such methods explicitly in these data sets, we focused on characterizing the discordance in the gene trees proper, as this is ultimately a key determinant of the species tree. We took a similar tack in phylogenomic analysis of data for the chromosome 3 short arm in Oryza (Zwickl et al. 2014) to understand the sources of biases in these gene tree discordance patterns. Moreover, in both that study and our more recent whole genome phylogenomic analysis of Oryza using conventional annotation/alignment pipelines (Stein J.C. et al., submitted for publication), supermatrix methods and species tree inference using gene trees as input returned the same results.

Because rice genome data exhibit widespread gene tree discordance (Zou et al. 2008; Cranston et al. 2010; Zhang et al. 2014; Zwickl et al. 2014; Stein J.C. et al., submitted for publication), we used it to test the power of our -mer block method to retain critical information about homology in the context of discordance. We examined (i) intragenomic conflict in phylogenetic signal and (ii) interchromosomal transfers identified via the -mer blocks. The frequency distribution of gene trees obtained by Stein J.C. et al. (submitted for publication) using conventional methods, and by using pools of nearby -mer blocks, was nearly identical (Fig. 5), suggesting that the full arsenal of species tree inference methods now available could be used with -mer block data sets. The only caveat to this is that enough coordinate information must be available to allow pooling of nearby -mer blocks (see below in “Limitations, Extensions, and Prospects” section).

The potential of the -mer block method to grapple with data sets having complex discordant signals, however, was well illustrated by examining patterns of interchromosomal transfer in Oryza. A few species of Oryza exhibit large numbers of regions on certain chromosomes that evidently are homologous to regions on different chromosomes in the remaining species. This complex pattern of homology (or conceivably assembly errors in some cases) is evident in pairwise global alignments of all chromosomes across Oryza (see Gramene rel. 50 at http://www.gramene.org), but how to exploit it has not been so clear. Runs of -mer blocks having the same chromosomal distribution patterns were easy to find and their pooled phylogenetic histories were quite similar to the gene tree distributions found in gene trees not undergoing transfers. This indicates that -mer blocks can be used to directly uncover new sources of gene tree discordance within a genome, and yet are robust enough that the gene trees inferred from them correctly mimic those from regions of the genome not undergoing transfers.

Relationship of hakmer to Other Methods

Our method uses some techniques found in previous work, including especially the use of approximate -mer matching (though we use a different implementation), and the anchors together with flanking ungapped alignments, but it also differs from all of them in key ways. Most importantly, it builds ungapped multiple sequence alignments rather than distance matrices, thus preserving information about local homologies within the genome, and it effectively insures orthology between species by forcing exact or approximate -mer matches to be long enough to be single copy within a genome (Fig. 2).

Hakmer is similar to kSNP in certain respects (Gardner and Hall 2013; Gardner et al. 2015). Their method is aimed at SNP detection for phylogenetic reconstruction and other problems. It finds -mers (with odd) having exactly one mismatch that is flanked symmetrically by (1)/2 exact matches on either side, and uses the mismatch as phylogenetic information. In contrast, hakmer finds -mers with up to mismatches and then uses both the -mers and flanking regions of length in the multiple sequence alignment. Hakmer should, therefore, be useful to deeper phylogenetic depths, and less prone to paralogy problems, because hakmer allows to be set large enough to essentially guarantee a single match per genome, while also allowing more than just a single mismatch in the extended block, and including additional flanking regions for additional phylogenetic information. kSNP uses MUMmer’s suffix tree data structures (Kurtz et al. 2004) to find additional matches among genomes, which requires more memory than our suffix array to index positions of matches across all genomes without any speed improvement (Leimeister and Morgenstern 2014).

Chan et al. (2014) reviewed the performance of several pairwise distance methods, which use functions of the number of exact -mer matches between genomes of different species. They compared performance to what would be obtained from multiple sequence alignments using sequence data simulated under a wide variety of interesting evolutionary models. Fan et al. (2015) estimate a pairwise distance based on the number of shared exact matching -mers, but they also attempt to correct for multiple hits in the same -mer by reference to both a Poisson process model of base substitution and a model generating homoplasy in the -mers that depends on -mer frequencies and structure.

Not all pairwise distance methods require -mers of fixed length. Ulitsky et al. (2006) compute the pairwise distance between genomes as the average over all start positions in sequence 1 of the lengths of the longest exact matches in sequence 2. Leimeister and Morgenstern (2014) extended this method by allowing up to mismatches. They implemented this with an “extended” suffix array data structure. Haubold et al. (2015) compute pairwise distances by finding two anchor points between two genomes, where an anchor is a maximal exact match found in both genomes. They then use the ungapped alignment between anchors to compute a corrected distance measure.

Several methods use unassembled short read sequence data aligned to reference genomes to build alignments and then trees. Bertels et al. (2014) criticized potential biases of some such methods that align reads to a single reference genome, which they attempt to correct by identifying SNPs from reads mapped to several reference genomes simultaneously. These approaches require computationally relatively expensive short read aligners at their core, which limits scalability. Moreover, avoiding paralogous sequences in the same genome is fundamentally difficult in unassembled data, and Bertels et al. (2014) try to limit this by a weighting scheme based on how aligned reads map to multiple places in reference genomes. To achieve the level of performance seen in large genome data sets, we found it necessary to work with assembled genomes with hakmer.

Limitations, Extensions, and Prospects

Our method is designed to gain a rapid and accurate phylogenetic foothold in large and complex genomic data sets across taxa at moderate levels of sequence divergence. It is not ideal when sequence divergence is so large that there are few -mers shared by a substantial fraction of taxa and/or flanking regions are so diverged that they require formal multiple alignment algorithms. By the same token, in deep phylogenies there may well be issues with composition bias or saturation that require careful model fitting that may be degraded by having samples of short stretches of the genome rather than long aligned blocks. Nor is our method necessary when taxa are so closely related that very long stretches of exact matches are frequent between genomes (much longer than our -mers); these can be discovered by existing exact algorithms such as those in MUMmer (Kurtz et al. 2004), or by invoking very fast heuristics like BLAST (Altschul et al. 1997), which will perform well at such high levels of sequence identity.

Several possible strategies may increase the domain of reasonable problem instances for this method. Increasing , the mismatch limit, obviously detects more divergent -mers, and running times for algorithms for matching -mers with mismatches increase with sublinearly (Nicolae and Rajasekaran 2015; in fact at best at ), but in our implementation this is combined with binary searches of the whole suffix array, which adds considerable overhead. Increasing runs the risk of finding -mer blocks with highly divergent flanking sequences as well, necessitating gapped alignment protocols. It might be more productive to increase the length of the flanking regions, , directly, which adds phylogenetically variable sites roughly at a rate proportional to . One strategy to avoid the inevitable alignment problems that also increase in probability with would be to trigger a formal multiple sequence alignment algorithm if exceeds some threshold. Running time for this often scales as in practice, however, and would get quite expensive unless the number of taxa is kept low. Alternatively, the extension length could be chosen adaptively using techniques like those used in sequence database searches, such as extending the -mer block as an ungapped alignment until the running alignment score drops by some threshold amount from its peak value, as in BLAST (Altschul et al. 1997).

In general, sets of genomes are mixtures of less and more conserved regions, so our analyses based on -mer matches preferentially extracts a subset of these genomes, the properties of which might be biased with respect to phylogenetic inference. This sort of ascertainment bias seems much more likely to affect inferences derived from branch lengths or associated with rates, including coalescent-based species tree inference perhaps, than it does simple gene tree topology reconstruction (Costa et al. 2016), but this remains to be investigated for our method.

To fully exploit the power of species tree inference methods that allow gene tree discordance, we expect that pooling of nearby -mer blocks will be needed. This means in general that multiple -mer blocks must be available at the scaffold level in genome assemblies to make pooling worthwhile. If sequence divergence is high, there may be too few -mer blocks present on the average scaffold for pooling. This was not a problem in the Oryza or Anopheles data sets, which have large N50 values, but would have been more problematic in the other whole genome data sets having more heterogeneity in assembly quality, and certainly in the transcriptome data set, which is intrinsically limited in the length of its assembly unit by the size of a transcript. The same remedies for high sequence divergence described above can be helpful to reduce the average distance between blocks, by increasing the number of blocks discovered per unit coordinate distance.

In conclusion, our experience with this -mer based phylogenomic approach on real data sets suggests that it is a rapid and effective way to tease apart not just the primary phylogenetic signal one would obtain from more conventional supermatrix analyses but also more nuanced signals arising from discordant histories in the genomes. Though both simulations and empirical analyses suggest it ultimately is limited by sequence divergence, the phylogenetic scope of its applicability appears to be quite broad. In plant data sets, it provided useful results for nuclear genome data to depths of 50–100 Ma and for slower evolving plastid and mitochondrial genomes back to the origin of angiosperms and land plants at 140/450 Ma, respectively. The method offers dramatically increased speed, reproducibility, and simplicity of data analysis, but it is not without cost. Only further experience will demonstrate whether this kind of high-throughput phylogenomics can be applied sufficiently broadly in the tree of life (and with sufficient accuracy) to compete with more exhaustive conventional approaches.

SUPPLEMENTARY MATERIAL

Data available from the Dryad Digital Repository: http://datadryad.org/resource/doi:10.5061/dryad.96b0h.

FUNDING

This work was supported by the US National Science Foundation [PGRP-1026200 to M.J.S.; DEB-1353815 to M.J.S. and M.M.M.].

Appendix: Detailed Results of Phylogenomic Analyses of Seven Data Sets

Land plants: mitochondrial genomes.

Our bootstrap tree (Supplementary Fig. 1 available on Dryad) for 87 taxa has areas of strong and weak support. We can compare it to the 41-gene mitochondrial analysis of Liu et al. (2014), by pruning our tree to the taxa found in theirs. Only two differences emerged. In their tree, Oryza is one nearest neighbor interchange (NNI) closer to Triticum, and in their tree the hornworts are sister to vascular plants, whereas in ours hornworts are sister to mosses, albeit with weak support. The uncertain position of hornworts figured prominently in Liu et al.’s (2014) arguments for the need to compensate for convergent base compositional shifts causing conflicting phylogenetic signals in the deeper part of land plant phylogeny (cf. their figure 1). In an analysis of 17 chloroplast genes primarily for bryophytes, Chang and Graham (2014) agreed with the remaining relationships within the mosses indicated by our tree, although they also found the hornworts sister to vascular plants.

Within angiosperms, there are two areas of poor to medium support in our tree. The first is associated with the position of Vaccinium macrocarpon, which although is near its traditionally accepted place as sister to the rest of the asterids (Soltis et al. 2011) in our tree, it is in a polytomy. In the RAxML optimal tree (data not shown), it is found on a short branch within that clade, as sister to our representatives from Asterales (Helianthus and Daucus). A more surprising outlier is the aberrant position of the clubmoss Selaginella, which should be sister to the other clubmoss in our sample, Huperzia. Instead, in our bootstrap tree, it is in a polytomy with asterid and rosid angiosperms. More precisely, in the RAxML optimal tree (data not shown), it is nested within the genus Cucumis (cucumber). Bootstrap support in this entire area of the tree is low, suggesting that Selaginella is behaving as a “rogue taxon” (Aberer et al. 2013), which is also supported by the fact that it is present in very few -mer blocks, indicating low homology with other genomes in our data set. Because it hits with 99% identity (and 84% overlap) to the Selaginella chloroplast genome, it may be a scaffold mistakenly annotated as mitochondrial when it is in fact plastid (Banks et al. 2011). The cucumber sample (GenBank accession NC_016004.1) with which this problematic Selaginella genome grouped in the optimal tree is also unusual, as it is from a second mitochondrial chromosome found in that species. It also has few -mer blocks, indicating low homology to the rest of the data.

Like our tree based on nuclear genomes of eudicots (Supplementary Fig. 7 available on Dryad), our mitochondrial genome tree weakly suggests Vitis is outside asterids rosids and that the “Celastrales–Oxalidales–Malpighiales” (COM) clade (represented here only by Ricinus) is again closer to malvids than to rosids (see below for further discussion).

Angiosperms: plastid genomes.

Our tree (Supplementary Fig. 2 available on Dryad) for 663 taxa had strong support throughout. Major relationships along the backbone, as well as among the approximately 42 orders represented, were nearly identical to those in recently published phylogenies derived from multiple genomic compartments (Soltis et al. 2011) or from plastid genomes (Ruhfel et al. 2014) with fewer taxa. Exceptions include that our tree placed Ceratophyllum as sister to the monocots with weak support, instead of sister to eudicots (Moore et al. 2007; Soltis et al. 2011; Ruhfel et al. 2014), although support for that placement in other studies is also generally weak and sensitive to the data partitioning scheme (Moore et al. 2007; Ruhfel et al. 2014). Our only sample from Pandanales, the mycoheterotrophic Sciaphila densiflora, was on a very long branch in the RAxML optimal ML tree (data not shown) and placed in a polytomy with Asparagales and the commelinid clade. Sciaphila was also found on a long branch in the original analysis associated with the publication of its highly unusual reduced plastid genome, which is only 21 kbp in length (Lam et al. 2015). The only other significant departure from previous published phylogenies is for Cypripedium macranthos (Orchidaceae). Unlike the other two members of this genus represented in our tree, this taxon is grouped outside the orchids (but still within the Asparagales), with Eustrephus (Asparagaceae). A megablast search against GenBank’s nucleotide database shows equally good hits to members of Arecaceae, Bromeliaceae, and Asparagaceae with identity over its length, and it has been found on a long branch in previous trees (Luo et al. 2014). Our other plastid genomes from Cypripedium all hit to members of Orchidaceae in BLAST searches, indicating that the plastid genome for C. macranthos is an outlier. A handful of other genera were not monophyletic in our bootstrap tree, but in each case a few NNI moves would restore monophyly. Plastid genome evolution is so slow that it is rarely used for species-level relationships within genera; it is certainly possible that whole plastid genomes sampled via -mers are also not ideal at such a low taxonomic level.

Caryophyllales: transcriptomes.

Our bootstrap tree of 67 taxa (Supplementary Fig. 3 available on Dryad) was fully resolved and all but six clades were supported 90% (most 100%). This tree agreed with that of Yang et al. (2015: their figure S1) at every node but one, this involving a simple NNI within the genus involving Portulaca cryptopetala within Portulaca. Yang et al.’s analysis was based on 1122 putative orthologous loci identified from the set of all transcriptomes.

Oryza: whole genomes.

Our bootstrap tree (Supplementary Fig. 4 available on Dryad) for 10 species of Oryza and the outgroup Leersia is identical to that obtained by Stein J.C. et al. (submitted for publication) for the same species, including the 100% bootstrap support estimates at each node of the tree. Stein et al. used 6015 single copy orthologous genes across all 12 chromosomes. They obtained the same tree whether using a supermatrix of all loci, or the MP-EST species tree inference method (Liu et al. 2010), which uses the set of gene trees as input instead and optimizes across gene tree coalescent histories. Our tree also agrees with a considerably smaller analysis (Zwickl et al. 2014) of 473 genes obtained from the short arm of chromosome 3, except that in their tree O. rufipogon is switched from the sister group of O. sativa to be the sister group of O. nivara and O. indica. They obtained the same results with concatenation and species tree inference methods.

Anopheles: whole genomes.

Our bootstrap tree (Supplementary Fig. 5 available on Dryad) was fully resolved with 100% support at each node. We compared our tree with two recently published genome-scale phylogenomic studies of Anopheles: Neafsey et al. (2015), which has a broad sample across the genus with outgroups, and Fontaine et al. (2015), which focuses on the A. gambiae complex. Neafsey et al. (2015) identified 1085 “relaxed” single copy orthologs and estimated trees using concatenated amino acid sequences and a PROTGAMMAJTT model. Our tree agrees with theirs (their figures 1 and S4) except within the A. gambiae complex, where our tree is resolved but theirs is not. They did not report bootstrap values, but all of ours were 100%. Fontaine et al. (2015) inferred a complex phylogenetic history within the A. gambiae complex in which whole genome trees disagreed with inferences from the X chromosome alone (they argued the latter were more accurate). Our tree agrees with their whole genome tree (their figure S17), except in their tree A. melas is the sister group of A. merus. Their support values were 100% at each node. On the other hand, our tree was identical to their X chromosome tree with respect to these two species, but disagreed with respect to the position of A. gambiae within the complex (in accord with the disagreement they observed between autosomes and the X chromosome).

Fabaceae: whole genomes.

Our bootstrap tree (Supplementary Fig. 6 available on Dryad) was fully resolved with 100% support at each node. No whole genome trees have been published for Fabaceae, so we compared our tree with the three-gene taxon-rich analysis of the family presented in Bruneau et al. (2013). Our tree agrees with that tree at each node. Whether Lupinus or Arachis is the correct outgroup (or both are) is unclear, requiring further taxa to be sampled for resolution. Our tree also agrees with the large plastid-genome-based phylogenomic tree described above (Supplementary Fig. 2 available on Dryad). In that tree Lupinus and Arachis are sister groups, jointly acting as outgroups to the remaining six taxa in our whole genome tree.

Eudicots: whole genomes.

Our bootstrap tree for 24 taxa (Supplementary Fig. 7 available on Dryad) has only two nodes with less than 90% bootstrap support. Comparisons to previously published phylogenies suggest agreement within relatively closely related clades, such as Brassicaceae (Huang et al. 2016), but possible disagreement in the placement of three leaf taxa (Vitis, Cucumis, and Eucalyptus) as well as the position of the COM clade, represented here by Euphorbiaceae (Riccinus, Manihot), Salicaceae (Populus), and Linaceae (Linum). Soltis et al. (2011), in a combined analysis of 17 nuclear-ribosomal, mitochondrial, and chloroplast genes, found strong support for the placement of Vitaceae (Vitis) as sister to the Rosid clade (sister to all but Solanum and Mimulus in our analysis, after rooting on Aquilegia; one NNI move on our tree). However, placement of Vitis, and the support for that placement, varies across trees generated using nuclear nonribosomal data (Zhang et al. 2012), mitochondrial data (Zhu et al. 2007), and chloroplast data (Sun et al. 2015). In another discrepancy with the large 17-gene combined analysis (Soltis et al. 2011), we found support for Cucumis (Cucurbitaceae) to be phylogenetically closer to Fabales than to Rosales. This placement (again, one NNI move away on our tree) is consistent with the five-gene nuclear data set (Zhang et al. 2012) and some cpDNA analyses (Sun et al. 2015). Similarly, mitochondrial data (Zhu et al. 2007; Sun et al. 2015) and nuclear data (Zhang et al. 2012), reanalyzed in Sun et al. (2015), support our unexpected position of Myrtales (Eucalyptus) as sister to the rest of the rosids. Our most substantial discrepancy involves the COM clade, which has been considered part of the fabids (Soltis et al. 2011), but our analysis provides strong support for its placement closer to the malvids. However, Sun et al. (2015) reanalyzed several data sets to elucidate relationships among the fabid, malvid, and COM clades, and also found strong support from the mitochondrial and nuclear genomes for this placement. In all of these cases, the previously accepted phylogenetic placement was almost entirely derived from chloroplast data, whereas our results are consistent with at least some nuclear and mitochondrial genome analyses.