Fast construction of FM-index for long sequence reads

Abstract

Summary: We present a new method to incrementally construct the FM-index for both short and long sequence reads, up to the size of a genome. It is the first algorithm that can build the index while implicitly sorting the sequences in the reverse (complement) lexicographical order without a separate sorting step. The implementation is among the fastest for indexing short reads and the only one that practically works for reads of averaged kilobases in length.

Availability and implementation: https://github.com/lh3/ropebwt2

Contact: hengli@broadinstitute.org

1 INTRODUCTION

FM-index plays an important role in DNA sequence alignment, de novo assembly (Simpson and Durbin, 2012) and compression (Cox et al., 2012). Fast and lightweight construction of FM-index for a large dataset is the key to these applications. In this context, a few algorithms (Bauer et al., 2013; Liu et al., 2014) have been developed that substantially outperform earlier algorithms. However, they are only efficient for short reads. A fast and practical algorithm for long sequence reads is still lacking. This work aims to fill this gap.

2 METHODS

Let $\Sigma =\{\mathtt{A},\mathtt{C},\mathtt{G},\mathtt{T},\mathtt{N}\}$ be the alphabet of DNA with a lexicographical order $\mathtt{A}<\mathtt{C}<\mathtt{G}<\mathtt{T}<\mathtt{N}$. Each element in Σ is called a symbol and a sequence of symbols called a string over Σ. Given a string P, $\left|P\right|$ is its length and $P[i]$ the symbol at position i. A sentinel $ is smaller than all the other symbols. For simplicity, we let $P[-1]=P[|P|]=\$$. We also introduce $\stackrel{\sim }{P}$ as the reverse of P and $\overline{P}$ as the reverse complement of P.

Given a list of strings over Σ, ${(P_i)}_{0\le i<m}$, let T = P₀$₀ … P_{m − 1}$_{m − 1} with ${\$}_0<\cdots <{\text{\$}}_{m-1}<\mathtt{A}<\mathtt{C}<\mathtt{G}<\mathtt{T}<\mathtt{N}$. The suffix array of T is an integer array S such that S(i), $0\le i<\left|T\right|$, is the starting position of the i-th smallest suffix in the collection T. The Burrows-Wheeler Transform, or BWT, of T can be computed as $B[i]=T[S(i)-1]$. For the description of the algorithm, we segment B into $B=B_{\$}{B}_{\mathtt{A}}{B}_{\mathtt{C}}{B}_{\mathtt{G}}{B}_{\mathtt{T}}{B}_{\mathtt{N}}$, where $B_a[i]=B[i+C(a)]$ with $C(a)=\left|\right\{j:T[j]<a\left\}\right|$ being the array of accumulative counts. By the definition of suffix array and BWT, B_a consists of all the symbols with their next symbol in T being a.

The above defines BWT for an ordered list of strings. We next seek to define BWT for an unordered set of strings $\mathcal{C}$ by imposing an arbitrary sorting order on $\mathcal{C}$. We say list ${(P_i)}_i$ is in the reverse lexicographical order or RLO, if ${\stackrel{\sim }{P}}_i\le {\stackrel{\sim }{P}}_j$ for any i < j; say it is in the reverse-complement lexicographical order or RCLO, if ${\overline{P}}_i\le {\overline{P}}_j$ for any i < j. The RLO-BWT of $\mathcal{C}$, denoted by $B^{\text{RLO}}(\mathcal{C})$, is constructed by sorting strings in $\mathcal{C}$ in RLO and then applying the procedure in the previous paragraph on the sorted list. RCLO-BWT $B^{\text{RCLO}}(\mathcal{C})$ can be constructed in a similar way. In $B^{\text{RCLO}}({\left\{P_i\right\}}_i\cup {\left\{{\overline{P}}_j\right\}}_j)$, the k-th smallest sequence is the reverse complement of the k-th sequence in the FM-index. This property removes the necessity of keeping an extra array to link the rank and the position of a sequence in the FM-index, and thus helps to reduce the memory of some FM-index–based algorithms (Simpson and Durbin, 2012). For short reads, RLO/RCLO-BWT is also more compressible (Cox et al., 2012).

As a preparation, we further define two string operations: $\text{rank}(c,k;B)$ and $\text{insert}(c,k;B)$, where $\text{rank}(c,k;B)=\left|\right\{i<k:B[i]=c\left\}\right|$ gives the number of symbols c before the position k in B, and $\text{insert}(c,k;B)$ inserts symbol c after k symbols in B with all the symbols after position k shifted to make room for c. We implemented the two operations by representing each B_c in a B+-tree in memory, where a leaf keeps a run-length encoded string and an internal node keeps the count of each symbol in the leaves descended from the node.

Algorithm 1 appends a string to an existing index by inserting each of its symbol from the end of P. It was first described by Chan et al. (2004). Algorithm 2 constructs RLO/RCLO-BWT in a similar manner to Algorithm 1 except that it inserts $P[i]$ to $[l,u)$, the suffix array interval of P’s suffix starting at i + 1, and that BWT symbols in this interval are already sorted. This process implicitly applies a radix sort from the end of P, sorting it into the existing strings in the BWT in RLO/RCLO. Note that if we change line 1 to “$l\leftarrow u\leftarrow \left|\right\{i:B[i]=\$\left\}\right|$”, Algorithm 2 will be turned into Algorithm 1. Recall that the BCR algorithm (Bauer et al., 2013) is, to some extent, the multi-string version of Algorithm 1. Following similar reasoning, we can extend Algorithm 2 so as to insert multiple strings at the same time, which gives Algorithm 3. We use an array A(j) to keep the state of the j-th sequence after inserting its d-long suffix. At line 2, $A(j).c$ is the previously inserted symbol and $[A(j).l,A(j).u)$ is the interval to which the new symbol is inserted. In implementation, we may speed up the sorting mode by inserting multiple symbols at line 3.

When B is represented by a balanced tree structure, the time complexity of all three algorithms is $O(n\log n)$, where n is the total number of symbols in the input. However, we will see later that for short strings, Algorithm 3 is substantially faster than the first two algorithms, due to the locality of memory accesses, the possibility of cached B+-tree update and the parallelization of the ‘for’ loop at line 1. These techniques are more effective for a larger batch of shorter strings.

Disregarding RLO/RCLO, Algorithm 3 is similar to BCR except that BCR keeps B in monolithic arrays. As a result, the time complexity of BCR is O(nl), where l is the maximum length of reads, not scaling well to l.

3 RESULTS AND DISCUSSION

We implemented the algorithm in ropeBWT2 and evaluated its performance together with BEETL (http://bit.ly/beetlGH), the original on-disk implementation of BCR and BCRext, ropeBWT-BCR (https://github.com/lh3/ropebwt), an in-memory reimplementation of BCR by us, and NVBio (http://bit.ly/nvbioio), a GPU-based algorithm inspired by CX1 (Liu et al., 2014). Table 1 shows that for ∼100 bp reads, ropeBWT2 has comparable performance to others. For the ∼875 bp Venter dataset, NVBio aborted due to insufficient memory under various settings. We did not apply BCR because it is not designed for long reads of unequal lengths. Only ropeBWT2 works with this data set and the even longer moleculo reads.

Table 1.

Performance of BWT construction

Dataa	Algorithm	RCLO	Real	CPU%	RAMb (GB)	Comments
worm	nvbio	–	316 s	138	12.9	See notec
worm	ropebwt-bcr	–	480 s	223	2.2	-btORf
worm	Algorithm 3	Yes	506 s	250	10.5	-brRm10g
worm	Algorithm 3	No	647 s	249	11.8	-bRm10g
worm	beetl-bcr	–	965 s	259	1.8	RAM diskd
worm	beetl-bcr	–	2092 s	122	1.8	Networke
worm	Algorithm 1	–	5125 s	100	2.5	-bRm0
worm	beetl-bcrext	–	5900 s	48	0.1	Networke
12 878	ropebwt-bcr	–	3.3 h	210	39.3	-btORf
12 878	nvbio	–	4.1 h	471	63.8	See notef
12 878	Algorithm 3	Yes	5.0 h	261	34.0	-brRm10g
12 878	Algorithm 3	No	5.1 h	248	60.9	-bRm10g
12 878	beetl-bcr	–	11.2 h	131	31.6	Networke
Venter	Algorithm 3	Yes	1.4 h	274	22.2	-brRm10g
Venter	Algorithm 3	No	1.5 h	274	22.8	-bRm10g
mol	Algorithm 3	No	6.8 h	285	20.0	-bRm10g

^aDatasets—worm: 66M × 100 bp Caenorhabditis elegans reads from SRR065390; 12878: 1206M × 101 bp human reads for sample NA12878 (Depristo et al., 2011). Venter: 32M × 875 bp (in average) human reads by Sanger sequencing (Levy et al. 2007; http://bit.ly/levy2007); mol: 23M × 4026 bp (in average) human reads by Illumina’s Moleculo sequencing (http://bit.ly/mol12878).

^bHardware—CPU: 48 cores of Xeon E5-2697v2 at 2.70 GHz; GPU: one Nvidia Tesla K40; RAM: 128 GB; Storage: Isilon IQ 72000x and X400 over network. CPU time, wall-clock time and peak memory are measured by GNU time.

^cRun with option ‘-R -cpu-mem 4096 -gpu-mem 4096’. NVBio uses more CPU and GPU RAM than the specified.

^dResults and temporary files created on in-RAM virtual disk ‘/dev/shm’.

^eResults and temporary files created on Isilon’s network file system.

^fRun with option ‘-R -cpu-mem 48000 -gpu-mem 4096’.

Funding: NHGRI U54HG003037; NIH GM100233.

Conflict of Interest: none declared.