The need for speed

Short abstract

A new sequence-alignment algorithm that uses advanced data structures to help data analysis keep pace with data generation.

Abstract

DNA sequence data are being produced at an ever-increasing rate. The Bowtie sequence-alignment algorithm uses advanced data structures to help data analysis keep pace with data generation.

In this month's Genome Biology, Langmead and colleagues [1] present the Bowtie algorithm. Bowtie is designed to align large numbers of relatively short DNA sequencing reads to an entire reference genome. It does so by first taking the reference genome assembly and changing the order of the sequence using something called the Burrows-Wheeler Transform. Why is this useful? Speed is the best answer: Bowtie is more than 30 times faster than other published tools designed to do the same task. Let's step back and see why the need for speed in our analysis algorithms is greater now than at any time in the genomic age.

Over the past three years massively high-throughput sequencing, often called 'next-generation' sequencing, has developed from a few beta devices in key genome centers to a large installed base in research labs around the world. The success of sequencing machines such as Illumina/Solexa, ABI SOLiD and 454 FLX has facilitated the development of sequencing as a general-purpose experimental tool for many biological applications. The range of possible uses is rapidly establishing DNA sequencing as the microscope of modern biology.

The scale of data generation is amazing; for example, in the course of its pilot phase the 1000 Genomes Project [2] has already generated almost 2,000-fold total coverage of the human genome from 180 individual samples, an amount orders of magnitude larger than the original Human Genome Project. There is a very real chance that before 2012 the amount of data generated by worldwide DNA sequencing will exceed the expected 15 petabytes of data per year produced by CERN's Large Hadron Collider.

In the light of these spectacular developments in data-generation capacity, it should come as no surprise that the computational requirements for supporting large-scale genome sequencing are growing dramatically. A key question is whether bioinformaticians are up to the task. Fortunately, the sheer number of new algorithms - some, like Bowtie, are based on data structures and methods either newly introduced to biology or rediscovered in the light of challenges posed by next-generation sequence data - suggest that bioinformatics, if not yet entering a new golden age [3], is responding to the waves of data by building better surfboards rather than running for higher ground.

Alignment is one of the first and most fundamental problems for any sequencing-based project in which a reference genome assembly already exists for the species concerned. Today's resequencing and functional studies (Box 1) directly leverage the effort required to create high-quality finished and draft genome assemblies such as those available for the human and mouse genomes. For next-generation sequencing studies the collected DNA sequencing reads are almost completely meaningless until they are aligned. Even the knowledge of whether the experiment succeeded is unknown until the sequencing reads are aligned to the reference genome.

Box 1. Resequencing and functional studies.

A small sampling of recent work leveraging the developments in DNA sequencing technology.

How do we address this essential step in the analysis and get as quickly as possible to the point where we can start to make sense of the biology? Programs such as Bowtie dramatically accelerate the alignment step by storing the reference genome in a highly ordered manner that facilitates very rapid searching of sequence. The key technology in Bowtie is called the Burrows-Wheeler Transform (BWT), which was originally developed for data compression. It works by reordering the original genome sequence such that certain patterns within the sequence are made explicit and therefore simplifies compression of the sequence. Importantly, the BWT reordering is reversible, so we are always able to reconstruct the original sequence. In fact, those readers who have ever downloaded compressed files from the Internet have probably already benefited from the BWT, which is at the heart of the bzip2 data compression algorithm [4].

Once the BWT has been constructed for the given genome assembly it is indexed for optimal searching by creating an FM index, which is, roughly speaking, a compressed suffix array of the genome sequence. These existing techniques and novel modifications by Langmead et al. [1] to existing sequencing matching algorithms allow Bowtie to use the FM index to rapidly align both exactly matching DNA sequencing reads and those with mismatches caused by sequencing error or sequence polymorphism, all while maintaining a memory footprint low enough to run on many standard laptop computers.

The BWT and the FM index are not complete strangers to bioinformatics. Several groups have adopted the data structure to solve specific problems mostly related to comparing many short segments of the genome to the genome as a whole. Before massive resequencing datasets existed, a common application of this problem was microarray probe design [5,6]. In this case, one effective way to estimate cross-hybridization potential for a given array design is to do a brute-force comparison of all short DNA segments (that is, possible array probes) to the genome as a whole.

Even when there are hundreds of billions of short sequencing reads the problem of alignment remains relatively easy compared with the problem of de novo genome assembly from short sequencing reads (especially for mammalian-sized genomes). A key difference comes from how easy it is to distribute the required computational work over the nodes of the compute clusters that are commonly used for bioinformatics analysis.

For example, alignment is considered 'embarrassingly parallel', so named because of how easy it is to achieve parallelization. For the case of read alignment to the reference genome, the most common way to distribute the task across a compute cluster is to store the complete reference genome on each of the nodes of the cluster and then distribute the collection of reads equally across the nodes. The read alignments can be merged at the end of the process. De novo assembly requires that essentially all the information needed to solve the problem (that is, how sequencing reads are related to each other) is available to the assembly program. For short-read datasets and mammalian-sized genomes, this generally leads to extremely large memory requirements that grow with the genome size and number of sequencing reads or to software implementations based on complex message passing between compute nodes.

To achieve large-scale alignment parallelization one only needs to be able to store the entire reference genome in memory available at each compute node. Without the BWT and the data compression it provides, storing a search-optimized data structure such as a suffix array for the entire genome is not feasible on each of the compute nodes found in today's clusters (see [5] for a more detailed discussion of the memory requirements of a mammalian genome suffix array both before and after a BWT).

Bowtie is not the only alignment program designed for next-generation sequence data using an index based on the BWT, but it does appear to be the first reported in the literature. The creators of SOAP [7] have recently introduced SOAP2 [8] and the creators of MAQ [9] have produced BWA [10], both of which provide a significant improvement in speed over the hash-table-based implementations of SOAP and MAQ.

For applications such as ChIP-seq and for rapid confirmation that the sequencing experiment performed as expected, Bowtie is likely to be the most effective solution. For some other applications, including whole-genome, paired-end resequencing projects, it may not yet be the right choice. Although much faster, Bowtie is not as accurate as MAQ in the case of a real dataset aligned with Bowtie's default parameters [1]. Parameter choices can increase Bowtie's accuracy, but at the cost of speed. Bowtie is also currently missing some critical functionality (for example, the ability to align paired reads). This functionality will certainly be added soon - either by the Bowtie developers, who have already implemented preliminary support for pair-end alignment in the most up-to-date version available on the Bowtie website [11], or by someone else enabled by Bowtie's open-source license.

Bowtie is yet another example of a common story in bioinformatics. Whereas default alignment programs are provided by the instrument manufacturers, the wider scientific community has developed the programs now used by many, if not most, researchers. This is a testament to the software-development skills within the research community and the desire within that community to create tools that are easy to deploy and use within existing analysis pipelines. There can be no doubt that open data formats and the ability to tap into the widest segment of the community in the search for solutions is the best way forward for DNA sequence analysis.

For now, sequence-alignment algorithms based on the BWT allow us to keep pace with the sequencing machines for at least another year. In today's fast-moving world of sequence generation, this is indeed a dramatic development.