Faster run-length compressed suffix arrays

Abstract

We first review how we can store a run-length compressed suffix array (RLCSA) for a text $T$ of length $n$ over an alphabet of size $\sigma$ whose Burrows-Wheeler Transform (BWT) consists of $r$ runs in $O\left(r\text{log}\right(n/r)+r\text{log}\sigma +\sigma )$ bits such that later, given character $a$ and the suffix-array (SA) interval for $P$, we can find the SA interval for $aP$ in $O\left(\text{log}{r}_a+\text{log}\text{log}n\right)$ time, where $r_a$ is the number of runs of copies of $a$ in the BWT. We then show how to modify the RLCSA such that we find the SA interval for $aP$ in only $O\left(\text{log}{r}_a\right)$ time, without increasing its asymptotic space bound. Our key idea is applying a result by Nishimoto and Tabei (ICALP 2021) and then replacing rank queries on sparse bitvectors by a constant number of select queries. We also review two-level indexing and discuss how our faster RLCSA may be useful in improving it. Finally, we briefly discuss how two-level indexing may speed up a recent heuristic for finding maximal exact matches of a pattern with respect to an indexed text.

1. Introduction

Grossi and Vitter’s compressed suffix arrays (CSAs) [11] and Ferragina and Manzini’s FM-indexes [8] are sometimes treated as almost interchangeable, but their query-time bounds are quite different. With a CSA for a text $T$ of length $n$ over an alphabet of size $\sigma$, when given a character $a$ and the suffix-array (SA) interval for a pattern $P$ we can find the SA interval for $aP$ in $O\left(\text{log}{n}_a\right)$ time, where $n_a$ is the number of occurrences of $a$ in the text; with an FM-index we use $O\left(\text{log}\sigma \right)$ time. This difference carries over to run-length compressed suffix arrays (RLCSAs) [18, 24] and run-length compressed FM-indexes (RLFM-indexes) [10, 17], with both taking space proportional to the number $r$ of runs in the Burrows-Wheeler Transform (BWT) of the text but the former being generally faster for texts over large alphabets with relatively few runs of each character, and the latter being faster for texts over smaller alphabets.

In Section 2 we review (with some artistic license) CSAs, RLCSAs and interpolative coding and show how an RLCSA for $T$ takes $O\left(r\text{log}\right(n/r)+r\text{log}\sigma +\sigma )$ bits and allows us to find the SA interval for $aP$ from that of $P$ in $O\left(\text{log}{r}_a+\text{log}\text{log}n\right)$ time, where $r_a$ is the number of those runs in the BWT containing copies of $a$. In Section 3.2 we review a result by Nishimoto and Tabei [20] about splitting the runs in the BWT so that we can evaluate LF in constant time, without increasing the number of runs by more than a constant factor. In Section 3.3 we present our main result: how to modify the RLCSA from Section 2 such that finding the SA interval for $aP$ takes only $O\left(\text{log}{r}_a\right)$ time, without increasing the asymptotic space bound. In Section 4 we discuss two-level indexing, for which we build one index for the text and another for the parse of the text, and how our faster RLCSA may be more suitable for indexing parses than current options. Finally, in Section 5 we discuss a related heuristic for finding longest maximal exact matches (MEMs) of a pattern with respect to an indexed text.

2. Preliminaries

2.1. Compressed suffix arrays

The key idea behind compressed suffix arrays (CSAs) is to store $\Psi [0..n-1]$ compactly while supporting certain searches on it quickly, where $\Psi [0..n-1]$ is the permutation of $\{0,\dots ,n-1\}$ such that $\Psi \left[i\right]$ is the position of suffix-array entry $\left(\text{SA}\right[i]+1)$ mod $n$ in $\text{SA}[0..n-1]$ or, equivalently, the position in $L$ of $F\left[i\right]$. (This means $\Psi$ is the inverse of the LF mapping used in FM-indexes.) By the definition, $\Psi$ consists of at most $\sigma$ increasing intervals — one for each distinct character that occurs in the text, corresponding to the interval of suffixes starting with that character — and if we can support fast binary searches on these intervals then we can support fast pattern matching.

For example, consider the text

$$T=\mathtt{\text{CCTGGGCGAT\$CTTACACGAT\$GTTACCAGCT\$CTTACGCGCT\$CTGACGAATT\$CTTACGCGAT}}\mathtt{\#},$$

for which $\text{SA},\Psi ,F$ and $L$ are shown on the left in Figure 1. If we know SA[22..28] is the SA interval for CG (in the green rectangle) and we want the SA interval for GCG, then we can search in the increasing interval

$$\Psi [36..48]=6,9,14,15,16,23,24,28,29,30,42,46,63$$

for G (in the red rectangle, with $\Psi$ values between 22 and 28 shown as orange arrows and the others shown as black arrows) for the successor $\Psi \left[41\right]=23$ of 22 and the predecessor $\Psi \left[43\right]=28$ of 28. We thus learn that the SA interval for GCG is SA[41..43] (in the blue rectangle). Knowing this, we can continue backward stepping.

Figure 1.

For is shown in blue. (We are interested in the blocks marked with the 29th and 30th copies of 1 because we count from 0 in the $j$ column in Figure 1, so those blocks correspond to ${\Psi }^{\prime }\left[28\right]$ and ${\Psi }^{\prime }\left[29\right]$.) We can find this interval with 2 select₁ queries on $B_F$, which take constant time.

2.2. Run-length compressed suffix arrays revisited

Run-length compressed suffix array (RLCSA) were introduced in [24] for indexing highly repetitive collections. In this section we present an alternative, but functionally equivalent, description of RLCSAs which is more suitable for describing our improvements.

Definition 1. For a text $T[0..n-1]$, the array $L^{\prime }[0..r-1]$ stores the sequence of $r$ characters in the runs in the run-length encoding of $L$.

Definition 2. For a text $T[0..n-1]$, the array $F^{\prime }[0..r-1]$ stores the $r$ characters in $L^{\prime }$ in lexicographic order.

Definition 3. For a text $T[0..n-1]$, the array ${\Psi }^{\prime }[0..r-1]$ is the permutation of $\{0,\dots ,r-1\}$ such that ${\Psi }^{\prime }\left[i\right]$ is the position $F^{\prime }\left[i\right]$ in $L^{\prime }$.

In this paper we view a RLCSA as a data structure storing ${\Psi }^{\prime }[0..r-1]$ compactly while supporting certain searches on it quickly. By the definition of ${\Psi }^{\prime }$, it still consists of at most $\sigma$ increasing intervals — one for each distinct character that occurs in $T$, corresponding to the interval of suffixes starting with that character — and if we can still support fast binary searches on these intervals then we can still support fast pattern matching.

For example, consider

$$T=\mathtt{\text{CCTGGGCGAT\$CTTACACGAT\$GTTACCAGCT\$CTTACGCGCT\$CTGACGAATT\$CTTACGCGAT}}\mathtt{\#}$$

again, for which ${\Psi }^{\prime },F^{\prime }$ and $L^{\prime }$ are shown on the right in Figure 1. If we know the SA interval SA[22..28] for CG starts at offset 0 in the $L$ run of character $L^{\prime }\left[12\right]$ and ends at offset 1 in the $L$ run of character $L^{\prime }\left[15\right]$ (in the green rectangle) and we want the SA interval for GCG, then we can search in the increasing interval

$${\Psi }^{\prime }[25..33]=1,3,7,13,15,22,25,39$$

for G (in the red rectangle, with ${\Psi }^{\prime }$ values between 12 and 15 shown as orange arrows and the others shown as black arrows) for the successor ${\Psi }^{\prime }\left[28\right]=13$ of 12 and the predecessor ${\Psi }^{\prime }\left[29\right]=15$ of 15.

Because $L^{\prime }$ and $F^{\prime }$ do not have the predecessor-successor relationship of $L$ and $F$, we cannot deduce that the SA interval for GCG starts in the $L$ run of character $L^{\prime }\left[28\right]$ and ends in the $L$ run of character $L^{\prime }\left[29\right]$ (and, in fact, in this example it does not). Instead, we store two $n$-bit SD-bitvectors [21], $B_L$ and $B_F$, with $r$ copies of 1 each. The 1s in $B_L$ mark the starting positions of runs in $L$ and the 1s in $B_F$ mark the positions in $F$ of the marked characters in $L$. In our example

$$B_F=\mathtt{11100110111001111111111100010111011011100}\mathtt{101}\mathtt{0011110000010101111000}$$

$$B_L=\mathtt{10000011011101100101111101001001110011100}\mathtt{011}\mathtt{0111111111110001011110}.$$

The interval $B_F[41...43]$ in $B_F$ starting immediately before the bit with offset 0 in the block whose starting position is marked with the 29th copy of 1 and ending immediately before the bit with offset 1 in the block whose starting position is marked with the 30th copy of 1,

$$T=\mathtt{\text{CCTGGGCGAT\$CTTACACGAT\$GTTACCAGCT\$CTTACGCGCT\$CTGACGAATT\$CTTACGCGAT}}\mathtt{\#}$$

we show SA, $\Psi ,F$ and $L$ on the left and the ${\Psi }^{\prime },F^{\prime }$ and $L^{\prime }$ on the right. If we know SA[22..28] is the SA interval for CG (in the green rectangle on the left) and we want the SA interval for GCG, then we can search in the increasing interval

$$\Psi [36..48]=6,9,14,15,16,23,24,28,29,30,42,46,63$$

for G (in the red rectangle on the left, with $\Psi$ values between 22 and 28 shown as orange arrows and the others shown as black arrows) for the successor $\Psi \left[41\right]=23$ of 22 and the predecessor $\Psi \left[43\right]=28$ of 28. We thus learn that the SA interval for GCG is SA[41..43] (in the blue rectangle on the left).

On the other hand, if we know SA[22..28] starts at offset 0 in the $L$ run of character $L^{\prime }\left[12\right]$ — that is, at offset 0 in the 13th run, counting from 1 — and ends at offset 1 in the $L$ run of character $L^{\prime }\left[15\right]$ (in the green rectangle on the right), then we can search in the increasing interval

$${\Psi }^{\prime }[25..32]=1,3,7,13,15,22,25,39$$

for G (in the red rectangle, with ${\Psi }^{\prime }$ values between 12 and 15 shown as orange arrows and the others shown as black arrows) for the successor ${\Psi }^{\prime }\left[29\right]=13$ of 12 and the predecessor ${\Psi }^{\prime }\left[30\right]=15$ of 15 (in the blue rectangle on the right). We then use select and rank queries on two $n$-bit sparse vectors to find the SA interval for GCG, the $L$ runs containing that interval’s starting and ending positions, and those positions’ offsets in those runs.

The corresponding interval $B_L[41..43]$ in $B_L$ is also shown in blue, starting immediately before the bit with offset 3 in the block whose starting position is marked with the 22nd copy of 1 and ending immediately before the bit with offset 1 in the block whose starting position is marked with the 24th copy of 1. We can find the 2 indices 22 and 24 with 2 rank₁ queries on $B_L$, which take $O\left(\text{log}\text{log}n\right)$ time. This means the SA interval for GCG is SA[41..43] and it starts at offset 3 in the $L$ run of character $L^{\prime }\left[21\right]$ and ends at offset 1 in the $L$ run of character $L^{\prime }\left[23\right]$. Knowing this we can continue backward stepping.

The RLCSA in Sirén’s PhD thesis [24] for a text $T[0..n-1]$ with $r$ BWT runs takes $O\left(r\text{log}\right(n/r)+r\text{log}\sigma +\sigma \text{log}n)$ bits. Given a character $a$ and the SA interval for $P$, it can find the SA interval for $aP$ in $O\left(\text{log}n\right)$ time.

3. Faster RLCSAs

3.1. Searchable Interpolative coding

Suppose we are given an increasing list ${\ell }_1,\dots ,{\ell }_k$ of $k$ integers in the range $[0..n-1]$. To encode them with interpolative coding [19], we first write ${\ell }_{\lceil k/2\rceil }$ using $\lfloor \text{lg}(n-1)\rfloor +1$ bits (except that we write 0 using 1 bit). All the numbers ${\ell }_1,\dots ,{\ell }_{\lceil k/2\rceil -1}$ are in the range $\left[0..{\ell }_{\lceil k/2\rceil }-1\right]$, so we can encode them recursively. All the numbers ${\ell }_{\lceil k/2\rceil +1},\dots ,{\ell }_k$ are in the range $\left[{\ell }_{\lceil k/2\rceil }+1..n-1\right]$, so we can encode them recursively as ${\ell }_{\lceil k/2\rceil +1}-{\ell }_{\lceil k/2\rceil }-1,\dots ,{\ell }_k-{\ell }_{\lceil k/2\rceil }-1$. Each encoding has $O\left(\text{log}n\right)$ bits, so we can read them in $O\left(1\right)$ time.

For example, if $n=66,k=13$ and the list is 6, 9, 14, 15, 16, 23, 24, 28, 29, 30, 42, 46, 63, then we start by encoding ${\ell }_7=24$ using $\lfloor \text{lg}65\rfloor +1=7$ bits as 0011000. We then encode ${\ell }_3=14$ using $\lfloor \text{lg}23\rfloor +1=5$ bits as 01110. We then encode ${\ell }_1,{\ell }_2,{\ell }_5,{\ell }_4,{\ell }_6=6,9,16,15,23$ as 0110, 010, 0001, 0, 110, and ${\ell }_{10},{\ell }_8,{\ell }_9,{\ell }_{12},{\ell }_{11},{\ell }_{13}$ as 000101, 011, 0, 001111, 1011, 10000.

If we imagine the list stored as keys in a balanced binary search tree, as illustrated in Figure 2, we encode the keys according to a pre-order traversal: when we reach each key ${\ell }_i$, we know ${\ell }_i$ lies between the numbers shown to the left and right of ${\ell }_i$ and we encode ${\ell }_i$ using the maximum number of bits we would need for any key in that range. For example, when we reach 46 in a pre-order traversal of the tree in Figure 2, we know it lies between 31 and 65, so we encode it using $\lfloor \text{lg}(65-31)\rfloor +1=6$ bits as (46 − 31)₂ = 00111.

Figure 2.

A balanced binary search tree storing the $k=13$ keys from the increasing list 6, 9, 14, 15, 16, 23, 24, 28, 29, 30, 42, 46, 63 with each key in the range $[0..n-1=65]$. When we reach each key in a pre-order traversal or binary search, we know it lies between the two values show to its left and right, so we can encode it as the binary number shown below it, using a total of $O\left(k\text{log}\right(n/k)+k)$ bits. If we store a bitvector marking the start of each encoding as visited in an in-order traversal, as shown below the tree, then we can omit the leading 0s from the encodings and support binary search in time $O\left(\text{log}k\right)$ without changing our asymptotic space bound.

The binary search tree has height $\lfloor \text{lg}k\rfloor$ and the bottom level contains at most $k$ keys. By Jensen’s Inequality, we encode those keys using $O\left(k\text{log}\right(n/k)+k)$ bits. Similarly, there at most $k/2^h$ keys at height $h$ and we encode those keys using $O\left(\frac{k}{2^h}\text{log}\frac{n}{k/2^h}+\frac{k}{2^h}\right)=O\left(\frac{k}{2^h}\text{log}(n/k)+\frac{k(h+1)}{2^h}\right)$ bits. Since

$$\sum _{h=0}^{\lfloor \text{lg}k\rfloor }\frac{k(h+1)}{2^h}=O\left(k\right),$$

we use $O\left(k\text{log}\right(n/k)+k)$ bits in total.

In this paper we want to perform binary search on the list — the reader may have noticed that our example is $\Psi [36..48]$ from Figure 1 — so we want random access to the encodings of the numbers in it. We can store the encodings according to an in-order traversal instead of a pre-order traversal, and store an uncompressed bitvector with as many bits as there are in the concatenation of the encodings and 1s marking where the encodings start. Since the bitvector delimits the encodings, however, we can delete the leading 0s from each encoding before concatenating them and building the bitvector. The in-order encodings for our example are shown below the tree in Figure 2, with the leading 0s removed, and the bitvector is shown below them. Since the bitvector uses at most as many bits as the encodings, we still use $O\left(k\text{log}\right(n/k)+k)$ bits in total and we can perform binary search in $O\left(\text{log}k\right)$ time. This scheme is similar to Teuhola’s [25] and Claude, Nicholson and Seco’s [5].

To find the successor of 22 in the list, we start at the root knowing $n=66$ and $k=13$ and perform select₁(7) and select₁(8) queries on the bitvector to find the starting and ending positions of the encoding 0011000 of ${\ell }_7=24$ in the range [0..65]. Since 22 < 24, we then perform select₁(3) and select₁(4) queries to find the starting and ending positions of the encoding 01110 of ${\ell }_3=14$ in the range [0..23]. Since 22 > 14, we then perform select₁(5) and select₁(6) queries to find the starting and ending positions of the encoding 0001 of ${\ell }_5=16$ in the range [15..23]. Since 22 > 16, we then perform select₁(6) and select₁(7) queries to find the starting and ending positions of the encoding 110 of ${\ell }_6=23$ in the range [17..23]. Since 22 < 23, we know the successor of 22 in $L$ is 23. We can find the predecessor of 28 in $O\left(\text{log}k\right)$ time symmetrically.

If we apply interpolative coding with fast binary search to the increasing interval of $\Psi$ for a character $a$ in a text $T$ of length $n$, then we use $O\left(n_a\text{log}\left(n/n_a\right)+n_a\right)$ bits and can support binary search in $O\left(\text{log}{n}_a\right)$ time, where $n_a$ is the frequency of $a$ in $T$. If we do this for all the characters then we use $O\left(n\left(H_0\left(T\right)+1\right)\right)$ bits, where $H$ is the 0th-order empirical entropy of $T$. If we encode the increasing interval of ${\Psi }^{\prime }$ for $a$ with interpolative coding, then we use $O\left(r\text{log}\left(r/r_a\right)+r_a\right)$ bits and can support binary search in $O\left(\text{log}{r}_a\right)$ time, where $r_a$ is the number of runs of copies of $a$ in the BWT of $T$ (and, equivalently, in $L$). If we do this for all the characters then we use $O\left(r\left(H_0\left(L^{\prime }\right)+1\right)\right)$ bits, where $L^{\prime }$ is again the sequence of $r$ characters in the runs in the run-length encoding of $T$. To be able to find the increasing interval for $a$ in ${\Psi }^{\prime }$, we store an $r$-bit uncompressed bitvector with 1s marking the where the intervals start.

Theorem 4. We can store ${\Psi }^{\prime }$ for $T$ in $O\left(r\left(H_0\left(L^{\prime }\right)+1\right)\right)\subseteq O\left(r\text{log}\sigma \right)$ bits and support binary search in the increasing interval for a character a in $O\left(\text{log}{r}_a\right)$ time, where $r_a$ is the number of runs of copies of a in the BWT of $T$.

Using Theorem 4 in an RLCSA gives us the following corollary, with the $O\left(\text{log}\text{log}n\right)$ term in the query-time bound coming only from the $O\left(\text{log}\text{log}n\right)$ query-time bound for rank on SD-bitvectors. The $O\left(r\text{log}\right(n/r\left)\right)$ term in the space bound comes from the SD-bitvectors, the $O\left(r\text{log}\sigma \right)$ term comes from the interpolative coding (and includes an $r$-bit bitvector marking where the increasing intervals for the distinct characters start in $F^{\prime }$) and the $O\left(\sigma \right)$ term comes from a $\sigma$-bit uncompressed bitvector marking which distinct characters occur in $T$.

Corollary 5. We can store an RLCSA for $T$ in $O\left(r\text{log}\right(n/r)+r\text{log}\sigma +\sigma )$ bits such that, given character a and the $SA$ interval for $P$, we can find the $SA$ interval for $\mathit{\text{aP}}$ in $O\left(\text{log}{r}_a+\text{log}\text{log}n\right)$ time.

3.2. Splitting Theorem for RLCSAs

Nishimoto and Tabei [20] showed how we can split the runs in $L$ such that no block in $B_F$ overlaps more than a constant number of blocks in $B_L$ without increasing the number of runs by more than a constant factor, and then store LF in $O\left(r\text{log}n\right)$ bits and evaluate it in constant time. Brown, Gagie and Rossi [4] slightly generalized their key theorem:

Theorem 6 (Nishimoto and Tabei [20]; Brown, Gagie and Rossi [4]). Let $\pi$ be a permutation on $\{0,\dots ,n-1\}$,

$$P=\left\{0\right\}\cup \{i:0<i\le n-1,\pi (i)\ne \pi (i-1)+1\},$$

and $Q=\left\{\pi \right(i):i\in P\}$. For any integer $d\ge 2$, we can construct $P^{\prime }$ with $P\subseteq P^{\prime }\subseteq \{0,\dots ,n-1\}$ and $Q^{\prime }=\left\{\pi \left(i\right):i\in P^{\prime }\right\}$ such that

• if $q,q^{\prime }\in Q^{\prime }$ and $q$ is the predecessor of $q^{\prime }$ in $Q^{\prime }$, then $\left|\left[q,q^{\prime }\right)\cap P^{\prime }\right|<2d$,

• $-\left|P^{\prime }\right|\le \frac{d\left|P\right|}{d-1}.$

If $L\left[i\right]=L[i-1]$ then $\text{LF}\left(i\right)=\text{LF}(i-1)+1$, so

$$\left\{0\right\}\cup \{i:0<i\le n-1,\text{LF}(i)\ne \text{LF}(i-1)+1\}$$

has cardinality $r$. If $\text{LF}\left(i\right)=\text{LF}(i-1)+1$ then, since $\Psi$ and LF are inverse permutations, $\Psi \left[j\right]=\Psi [j-1]+1$ where $j=\text{LF}\left(i\right)$. Therefore,

$$\left\{0\right\}\cup \{j:0<j\le n-1,\Psi [j]\ne \Psi [j-1]+1\}$$

also has cardinality $r$ and applying Theorem 6 with $d=2$ to $\Psi$ splits the runs in the BWT such that no block in $B_F$ overlaps more than 3 blocks in $B_L$, without increasing the number of runs by more than a factor of 3/2. In fact, the number of runs increases by only 1, from 40 to 41, as shown below with the split block — corresponding to the first run of 6 copies of T in $L$ — in red:

$$B_F^{\prime }=\mathtt{1110011011100111111111110001011101101110010100111}\mathtt{100100}\mathtt{1010111100}$$

$$B_L^{\prime }=\mathtt{100100}\mathtt{110111011001011111010010011100111000110111111111110001011110}.$$

Suppose we apply Theorem 6 with $d=2$ to $\Psi$ and then store, for $0\le b<r$, the index of the block in $B_L$ containing $\text{LF}\left(i_b\right)$ and $\text{LF}\left(i_b\right)$’s offset in that block, where $i_b$ is the starting position of block $b$ in $B_L$. Nishimoto and Tabei called this the move table for LF (see also [4, 26]) and it takes a total of $O\left(r\text{log}n\right)$ bits. If we know $B_L\left[j\right]$ is in block $b$ in $B_L$ with offset $j-i_b$ then, since the block in $B_F$ to which LF maps block $b$ in $B_L$ now overlaps at most the block containing $B_L\left[\text{LF}\left(i_b\right)\right]$ and the next 2 blocks in $B_L$, we can find the index of the block in $B_L$ containing $B_L\left[\text{LF}\right(j\left)\right]=B_L\left[\text{LF}\left(i_b\right)\right]+j-i_b$ and $B_L\left[\text{LF}\right(j\left)\right]$’s offset in that block with at most 2 constant-time select queries on $B_L$. We could use at most 2 constant-time lookups instead if we have the starting positions of the blocks in $B_L$ stored explicitly in another $O\left(r\text{log}n\right)$ bits.

3.3. A faster RLCSA without rank queries

Recall that the $O\left(\text{log}\text{log}n\right)$ term in the query-time bound in Corollary 5 comes only from the use of rank queries on an SD-vector. Since rank and select queries can be combined to support predecessor queries and select queries on sparse bitvectors can easily be supported in constant time and space polynomial in the number of 1s, rank queries on compact sparse bitvectors inherit lower bounds from predecessor queries [3] — so they cannot be implemented in constant time. Therefore, to get rid of that $O\left(\text{log}\text{log}n\right)$ term, we must somehow avoid rank queries.

We could replace the rank queries with a move table, but that would result in an $O\left(r\text{log}n\right)$ term in our space bound. Instead, we introduce an uncompressed $2r$-bit bitvector $B_{FL}$ indicating how the starting positions of the blocks in $F$ and $L$ are interleaved. Specifically, we scan $B_F$ and $B_L$ simultaneously — assuming we have already applied Theorem 6 to them so that no block in $F$ overlaps more than 3 blocks in $L$ (so $r$ is a constant factor larger than it was before the application of the theorem) — and

• if we see 0s in both bitvectors in position $i$ then we write nothing;

• if we see a 1 in $B_F$ and a 0 in $B_L$ then we write a 1 (indicating that a block starts in $F$);

• if we see a 0 in $B_F$ and a 1 in $B_L$ then we write a 0 (indicating that a block starts in $L$);

• if we see 1s in both bitvectors then we write a 0 and then a 1 (indicating that blocks start in both $L$ and $F$).

This way, $B_{FL}$. ${\text{select}}_1\left(j\right)$ tells us which at most 3 blocks in $L$ — those corresponding to the 0 preceding the $j$th copy of 1 in $B_{FL}$ and possibly to the next 2 copies of 0 — could overlap block $j$ in $F$ (counting from 1). We can then find the starting positions of those blocks in $L$ using at most 3 select queries on $B_L$.

For our example, taking $B_F$ and $B_L$ to be $B_F^{\prime }$ and $B_L^{\prime }$ shown at the end of Section 3.2,

$$012345678901234567890123456789012345678901234567890123456789012345$$

$$B_F=\mathtt{11100110111001111111111100010111011011100}\mathtt{10100}\mathtt{11110010010101111000}$$

$$B_L=\mathtt{10010011011101100101111101001001110011}\mathtt{100011011}\mathtt{1111111110001011110},$$

(with the grey numbers only to show positions) we have

$$\mathtt{0123456789012345678901234567890123456789}$$

$$B_{FL}=\mathtt{0111010101010100101110110101010101010110}\dots$$

$$\mathtt{012345678901234}\mathtt{567890123456789012345678901}$$

$$\dots \mathtt{100110101}\mathtt{0}\mathtt{1}\mathtt{1}\mathtt{0}\mathtt{0}\mathtt{1}\mathtt{001010101000100011011010100}.$$

In position 38 we see 1s in both $B_F$ and $B_L$, so we write 01 in $B_{FL}$ (in positions 49 and 50, respectively); in positions 39 and 40 we see 0s in both in $B_F$ and $B_L$, so we write nothing; in position 41 we see a 1 in $B_F$ and a 0 in $B_L$, so we write a 1 in $B_{FL}$; in position 42 we see a 0 in $B_F$ and a 1 in $B_L$, so we write a 0 in $B_{FL}$; in position 43 we see 1s in both $B_F$ and $B_L$, so we write 01 in $B_{FL}$; in position 44 we see 0s in both $B_F$ and $B_L$, so we write nothing; and in position 45 we see a 0 in $B_F$ and a 1 in $B_L$, so we a 0 in $B_{FL}$ (in position 55). Admittedly, when $n=66$ and after applying Theorem 6 $2r=82$, it seems foolish to store a $2r$-bit uncompressed bitvector instead of simply storing $B_L$ uncompressed. This is due to the small size of our example, however; for massive and highly repetitive datasets, $r$ can easily be hundreds of times smaller than $n$.

Suppose we know the SA interval SA[41..43] for $aP$ starts at offset 0 in block 28 in $F$ and ends at offset 1 in block 29 in $F$ and we want to find which blocks contain its starting and ending positions in $L$ and the offsets of those positions. In Section 2, we performed 2 rank queries on $B_L$, but now we perform queries $B_{FL}$.select₁ (29) = 51 and $B_{FL}$.select₁ (30) = 54 (with arguments 29 and 30 instead of 28 and 29 because we mark with a 1 the starting of the first block in $F$, which we index with 0; the results 51 and 54 are indexed from 0 as well). Since the 29th and 30th copies of 1 are $B_{FL}\left[51\right]$ and $B_{FL}\left[54\right]$ (shown in red above), they are preceded by the 23rd and 25th copies of 0, respectively.

Because we applied Theorem 6, this means the 29th and 30th blocks in $F$ (shown in red in $B_F$ above) overlap the 23rd block in $L$ and possibly the 24th and 25th blocks (shown in blue in $B_L$), and the 25th block and possibly the 26th and 27th blocks (also shown in blue in $B_L$). Notice that, because we split the 34th block in $F$ but the first block in $L$ for Theorem 6, the block numbers we find in $F$ are the same as in Section 2 but the block numbers we find in $L$ will be incremented. Although in general we need 6 select queries on $B_L$, in this case we can use only $5—B_L$.select₁ (23),…, $B_L$.select₁ (27) — to find where these blocks begin in constant time, and determine which contain the starting and ending positions of the SA interval SA[41..43]: the 23rd and the 25th, respectively.

In short, we replace a rank query on SD-bitvector $B_L$ by queries on uncompressed bitvector $B_{FL}$ and constant-time select queries on $B_L$. This gives us the following theorem:

Theorem 7. We can store an RLCSA for $T$ in $O\left(r\text{log}\right(n/r)+r\text{log}\sigma +\sigma )$ bits such that, given character $a$ and the $SA$ interval for $P$, we can find the $SA$ interval for $\mathit{\text{aP}}$ in $O\left(\text{log}{r}_a\right)$ time, where $r_a$ is the number of runs of copies of $a$ in the BWT of $T$.

Instead of viewing $B_{FL}$ as replacing slow rank queries while using the overall same space, we can also view it (and $B_F$ and $B_L$) as replacing an $O\left(r\text{log}n\right)$-bit move table while using the same overall query time. Brown, Gagie and Rossi [4] implemented a similar approach to speeding up LF computations in an RLFM-index, but only alluded to it briefly in their paper — the path to Bitvector in their Figure 3 — and gave no analysis nor bounds. We conjecture that a similar approach can also be applied to reduce the size of fast move tables for $\varphi$ and ${\varphi }^{-1}$ [13], which return $\text{SA}[i-1]$ and $\text{SA}[i+1]$ when given $\text{SA}\left[i\right]$.

Figure 3.

A set of 50 similar toy genomes of length 50 each, with the first 49 separated by copies of $ and the last one terminated by #.

4. Two-level indexing

Corollary 5 and Theorem 7 suggest that RLCSAs should perform well compared to FM-indexes and RLFM-indexes when the BWT is over a fairly large alphabet and the number of runs of each character is fairly small; Ordóñez, Navarro and Brisaboa [23] have confirmed this experimentally. When indexing a highly repetitive text over a small alphabet, we can make RLCSAs more practical by storing a table of $k$-tuples that tells us in which range of ${\Psi }^{\prime }$ to search based on which character we are trying to match and which $k-1$ characters we have just matched. (This table can be represented with a bitvector to save space.) The table for our example from Figure 1 and $k=2$ is shown below:

#C	0	AG	8	CT	20..24	T#	33
$C	1..2	AT	9..12	GA	25..27	T$	33
$G	3	CA	13..14	GC	28..29	TA	34..35
AA	4	CC	15..16	GG	30..31	TG	36..37
AC	4..7	CG	17..19	GT	32	TT	38..39

This says that if we want the SA interval for GCG and we have just matched the C, then we should search in the range ${\Psi }^{\prime }[28..29]$. Notice that the largest range of ${\Psi }^{\prime }$ in which we search is now ${\Psi }^{\prime }[20..24]$ — of length 5 — when we are trying to match a C after just matching a T; without such a table, the largest range we search is ${\Psi }^{\prime }[13..24]$ — of length 12 — when trying to match a C.

There are interesting cases in which we want to index highly repetitive texts over large alphabets, however. For example, consider indexing a minimizer digest of a pangenome — considering minimizers as meta-characters from a large alphabet instead of tuples of characters from a small alphabet [1, 2, 7, 27] — or two-level indexing such a text. For two-level indexing we build one index for the text and another for a parse of the text; the alphabet of the parse is the dictionary of distinct phrases, which is usually large, but the parse itself is usually much smaller than the text and its BWT is usually still run-length compressible (albeit less than the BWT of the text) when the text is highly repetitive.

Something like two-level indexing was proposed by Deng, Hon, Köppl and Sadakane [6] but they did not use an index for the text and its absence made their implementation quite slow for all but very long patterns. Hong, Oliva, Köppl, Bannai, Boucher and Gagie [12] described another approach, which we will review here, but they used standard FM-indexes for the text and the parse instead of RLFM-indexes, so their two-level index was noticeably faster but hundreds of times larger than its competitors.

Consider the 50 similar toy genomes of length 50 each in Figure 3. Suppose we parse their concatenation similarly to rsync, by inserting a phrase break whenever we see a trigger string — ACA, ACG, CGC, CGG, GAC, GAG, GAT, GTG, GTT, TCG or TCT — or when we reach the TA# at the end of the text. (Considering g # = $ = 0, A = 1, C = 2, G = 3 and T = 4 and viewing triples as 3-digit numbers in base 5, the trigger strings are the triples in the concatenation whose values are congruent to 0 modulo 6.) If we replace each phrase in the parse by its lexicographic rank in the dictionary of distinct phrases, counting from 1, and terminate the sequence with a 0, we get the 562-number sequence shown in Figure 4. With a larger example, of course, we obtain longer phrases on average and better compression from the parsing.

Figure 4.

The 563-number sequence (20 numbers per line) over the alphabet {0, …, 90} we get from the concatenation of the toy genomes in Figure 3 by parsing, replacing each phrase by its rank in the dictionary (counting from 1) and appending a 0.

Run-length compression naturally works better on the BWT of the concatenation of the genomes than on the BWT of the parse, as shown in Figures 5 and 6. Again, with a larger example we would achieve better compression, also from the run-length compressed BWT (RLBWT) of the parse. Even this small example, however, gives some intuition how the dictionary of distinct phrases in the parse is usually large, but the parse is usually much smaller than the text and its BWT is usually still run-length compressible. In this case there are 90 distinct phrases in the dictionary, the parse is less than a quarter as long as the text, and the average run length in its RLBWT is slightly more than 2. An FM-index based on the RLBWT of the parse would generally use at least about ⌈lg 90⌉ = 7 rank queries on bitvectors for each backward step. The most common value in the runs, 19, occurs in only 16 runs, so we should spend at most about lg 16 = 4 steps in each binary search.

Figure 5.

The RLBWT of the concatenation of the toy genomes shown in Figure 3, consisting of 449 runs (20 runs per line).

Figure 6.

The RLBWT of the sequence shown in Figure 6, consisting of 226 runs (15 runs per line).

To search for a pattern, we start by backward stepping in the index for the text until we reach the left end of the rightmost trigger string in the pattern. We keep count of how often each trigger string occurs in the text and an $n$-bit sparse bitvector with 1s marking the lexicographic ranks of the lexicographically least suffixes starting with each trigger string. This way, when we reach the left end of the rightmost trigger string, with a rank query on that bitvector we can compute the lexicographic ranks of the suffixes starting with the suffix of the pattern we have processed so far among all the suffixes starting with trigger strings, and map from the index of the text into the index for the parse. The width of the BWT interval stays the same and may span several lexicographically consecutive phrases in the dictionary — all those starting with the suffix of the pattern we have processed so far — but it is possible to start a backward search in the index for the parse with a lexicographic range of phrases rather than with a single phrase.

When we reach the left end of the leftmost trigger string in the pattern, we can use the same bitvector to map back into the index for the text and match the remaining prefix of the pattern with that. While matching the pattern phrase by phrase against the index for the parse, we can either compare against phrases in the stored dictionary or just use Karp-Rabin hashes (allowing some probability of false-positive matches). We still have to parse the pattern, but that requires a single sequential pass, while FM-indexes in particular are known for poor memory locality. They key idea is that, ideally, we match most of the pattern phrase by phrase instead of character by character, reducing the number of cache misses.

We plan to reimplement two-level indexes for collections of similar genomes with RLFM-indexes for the collections themselves and CSAs, standard RLCSAs and our sped-up RLCSAs for the parses from Theorem 7 of those collections, and compare them experimentally. We also plan to try indexing minimizer digests with CSAs and RLCSAs.

5. Boyer-Moore-Li with two-level indexing

Olbrich, Büchler and Ohlebusch [22] recently showed how working with rsync-like parses of genomes instead of the genomes themselves can speed up multiple alignment. More specfically, they find and use as anchors finding maximal substrings (call multi-MUMs) of the parses that occur exactly once in each parse. In this section we speculate about how two-level indexing may similarly speed up searches for long maximal exact matches (MEMs). A MEM of a pattern $P[0..m-1]$ with respect to a text $T$ is a substring $P[i..j]$ of $P$ such that

• $P[i..j]$ occurs in $T$,

• $i=0$ or $P[i-1..j]$ does not occur in $T$,

• $j=m-1$ or $P[i..j+1]$ does not occur in $T$.

Finding long MEMs is an important task in bioinformatics and there are many tools for it.

Li [14] gave a practical algorithm, called forward-backward, for finding all the MEMs of $P$ with respect to $T$ using FM- or RLFM-indexes for $T$ and its reverse $T^{\text{rev}}$. Assume all the distinct characters in $P$ occur in $T$; otherwise, we split $P$ into maximal substrings consisting only of copies of characters occurring in $T$ and find the MEMs of those with respect to $T$. We first use the index for $T^{\text{rev}}$ to find the longest prefix $P\left[0..e_1\right]$ of $P$ that occurs in $T$, which is the leftmost MEM. If $e_1=m-1$ then we are done; otherwise, $P\left[e_1+1\right]$ is in the next MEM, so we use the index for $T$ to find the longest suffix $P\left[s_2..e_1+1\right]$ of $P\left[0..e_1+1\right]$ that occurs in $T$. The next MEM starts at $s_2$, so conceptually we recurse on $P\left[s_2..m-1\right]$. The total number of backward steps in the two indexes is proportional to the total length of all the MEMs.

Gagie [9] proposed a heuristic for speeding up forward-backward when we are interested only in MEMs of length at least $L$. We call this heuristic Boyer-Moore-Li, following a suggestion from Finlay Maguire [16]. Since any MEM of length at least $L$ starting in $P[0..L-1]$ includes $P[L-1]$, we first use the index for $T$ to find the longest suffix $P[s..L-1]$ of $P[0..L-1]$ that occurs in $T$. If $s=0$ then we fall back on forward-backward to find the leftmost MEM and the starting position of the next MEM. Otherwise, since we know there are no MEMs of length at least $L$ starting in $P[0..s-1]$, conceptually we recurse on $P[s..m-1]$. Li [15] tested Boyer-Moore-Li and found it practical enough that he incorporated it into his tool ropebwt3.

Suppose we build an rsync-like parse of $T[0..n-1]$ and two-level indexes for $T$ and $T^{\text{rev}}$ based on that parse and parse $P$ when we get it. With a naïve two-level version of Boyer-Moore-Li, we would simply use the two-level indexes in place of the normal FM- or RLFM-indexes for $T$ and $T^{\text{rev}}$. We conjecture, however, that we can do better in practice.

Let $P\left[k\right]$ be the last character of the last phrase that ends strictly before $P\left[L\right]$, let $P\left[j\right]$ be the first character of the first phrase such that $P[j..k]$ occurs in $T$, and let $P\left[i\right]$ be the second character of the phrase preceding the one containing $P\left[j\right]$. Notice we can find $i,j$ and $k$ by matching phrase by phrase using only the top level (for the parse) of the two-level index for $T$. If $i>0$ then we can immediately discard $P[0..i-1]$ and conceptually recurse on $P[i..m-1]$; otherwise, we proceed normally.

Of course, the value $i$ is at most the value $s$ found by regular Boyer-Moore-Li and could be much smaller, in which case discarding $P[0..i-1]$ benefits us much less than discarding $P[0..s-1]$. We hope this is usually not the case and we look forward to testing Boyer-Moore-Li with two-level indexing.