Fast and compact matching statistics analytics

Abstract

Motivation

Fast, lightweight methods for comparing the sequence of ever larger assembled genomes from ever growing databases are increasingly needed in the era of accurate long reads and pan-genome initiatives. Matching statistics is a popular method for computing whole-genome phylogenies and for detecting structural rearrangements between two genomes, since it is amenable to fast implementations that require a minimal setup of data structures. However, current implementations use a single core, take too much memory to represent the result, and do not provide efficient ways to analyze the output in order to explore local similarities between the sequences.

Results

We develop practical tools for computing matching statistics between large-scale strings, and for analyzing its values, faster and using less memory than the state-of-the-art. Specifically, we design a parallel algorithm for shared-memory machines that computes matching statistics 30 times faster with 48 cores in the cases that are most difficult to parallelize. We design a lossy compression scheme that shrinks the matching statistics array to a bitvector that takes from 0.8 to 0.2 bits per character, depending on the dataset and on the value of a threshold, and that achieves 0.04 bits per character in some variants. And we provide efficient implementations of range-maximum and range-sum queries that take a few tens of milliseconds while operating on our compact representations, and that allow computing key local statistics about the similarity between two strings. Our toolkit makes construction, storage and analysis of matching statistics arrays practical for multiple pairs of the largest genomes available today, possibly enabling new applications in comparative genomics.

Availability and implementation

Our C/C++ code is available at https://github.com/odenas/indexed_ms under GPL-3.0. The data underlying this article are available in NCBI Genome at https://www.ncbi.nlm.nih.gov/genome and in the International Genome Sample Resource (IGSR) at https://www.internationalgenome.org.

Supplementary information

Supplementary data are available at Bioinformatics online.

1 Introduction

Several large-scale projects are under way to assemble the genome of hundreds of new species, and comparing such genomes is crucial for understanding the genetic basis and origin of complex traits and related diseases (for a small sampler, see e.g. Feng et al., 2020; Formenti et al., 2020; Hecker and Hiller, 2020; Jebb et al., 2020; Rhie et al., 2020; Serres Armero et al., 2020; Teeling et al., 2018; Zhang et al., 2014). Efficient tools for comparing genome-scale sequences are thus becoming increasingly necessary. The matching statistics of a string S, called the query, with respect to another string T, called the text, is an array $\mathrm{M}{\mathrm{S}}_{S,T}[\mathrm{0..}|S|-1]$ such that $\mathrm{M}{\mathrm{S}}_{S,T}[i]$ is the length of the longest prefix of $S[i\mathrm{..}|S|-1]$ that occurs anywhere in T without errors. Since the match can occur anywhere in T, matching statistics is robust to large-scale rearrangements and horizontal transfers that are common in genomes, and the average matching statistics length over the whole sequence has been used for building consistent whole-genome phylogenies without alignment—and, unlike k-mer methods, without parameters (Cohen and Chor, 2012; Ulitsky et al., 2006). The effectiveness of matching statistics in alignment-free phylogenetics has even motivated variants that allow for a user-specified number of mismatches (for a small sampler see e.g. Apostolico et al., 2016; Leimeister and Morgenstern, 2014; Pizzi, 2016; Thankachan et al., 2016, 2017); and other, seemingly different, distances can be expressed in terms of matching statistics as well (Castiglione et al., 2019; Ehrenfeucht and Haussler, 1988; Ukkonen, 1992). For genomes from the same or from closely related species, matching statistics has been used for computing estimators of the number of substitutions per site, of the number of pairwise mismatches, and of the occurrence of recombination events (see e.g. Domazet-Lošo and Haubold, 2009; Haubold and Pfaffelhuber, 2012; Haubold et al., 2009, 2011, 2013); finally, the related notion of shortest unique substring, defined on a single sequence, has been employed for computing measures of genome repetitiveness (Haubold and Wiehe, 2006; Haubold et al., 2005), and it could be used as a parameter-free method for detecting segmental duplications (Pu et al., 2018). Since every position of the query S is assigned a match length, matching statistics can reveal ranges of locally high similarity (i.e. of large average matching statistics in the range) induced e.g. by horizontal gene transfer, or conversely ranges of locally low similarity induced by chromosomes of S missing from T, or by horizontal transfer events that affected S but not T (Domazet-Lošo and Haubold, 2011a, b; Haubold and Pfaffelhuber, 2012; Haubold and Wiehe, 2006; Haubold et al., 2011). See Supplementary Figures S6 and S7 for a concrete example. This idea has been recently applied to targeted Nanopore sequencing, using online matching statistics to eject from the pore a long DNA molecule that is not likely to belong to the species of interest, after having read just a short segment of the molecule (Ahmed et al., 2021).

Computing $\mathrm{M}{\mathrm{S}}_{S,T}$ is a classical problem in string processing, and in practice it involves building an index on a fixed T to answer a large number of queries S. Thus, solutions typically differ on the index they use, which can be the textbook suffix tree, the compressed suffix tree (Ohlebusch et al., 2010) or compressed suffix array, the colored longest common prefix array (Garofalo et al., 2018), a Burrows–Wheeler index combined with the suffix tree topology (Belazzougui and Cunial, 2014; Belazzougui et al., 2018), or the r-index combined with balanced grammars (Boucher et al., 2021). In the frequent case where T consists of one genome (or proteome), or of the concatenation of few similar genomes or of many dissimilar genomes, the Burrows–Wheeler transform of T does not compress well, and the best space-time tradeoffs are achieved by the implementation in Belazzougui et al. (2018) (see Boucher et al., 2021 for a runtime comparison, and see Supplementary Fig. S2 for a memory comparison). In this paper, we develop several practical tools for computing the matching statistics array between genome-scale strings, and for analyzing its values, faster and using less memory than the state-of-the-art.

Specifically, we design a practical variant of the algorithm by Belazzougui et al. (2018) that computes MS in parallel on a shared-memory machine, and that achieves approximately a 41-fold speedup of the core procedures and a 30-fold speedup of the entire program with 48 cores on the instances that are most difficult to parallelize. Our implementation takes around 12 min to compute the MS between the Homo sapiens and the Pan troglodytes genomes on a standard 48-core server. We also describe a theoretical variant with better asymptotic complexity, which takes $O(|S|\hspace{0.17em}\text{log}\hspace{0.17em}\sigma /t+(\text{log}\hspace{0.17em}\left|T\right|\hspace{0.17em}\text{log}\hspace{0.17em}\sigma )(\text{log}\hspace{0.17em}t+\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}t\hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}\left|T\right|))$ time and $2\left|T\right|\hspace{0.17em}\text{log}\hspace{0.17em}\sigma +O(n)$ bits of space when executed on t processors, where σ is the integer alphabet of S and T. To the best of our knowledge, no algorithm for computing matching statistics in parallel existed before.

Then, we implement fast range queries for computing the average and maximum matching statistic value inside a substring of S, taking advantage of the compact encoding of $\mathrm{M}{\mathrm{S}}_{S,T}$ introduced by Belazzougui and Cunial (2014): this encoding takes just $2\left|S\right|$ bits, and allows one to retrieve $\mathrm{MS}[i]$ in constant time for any i using just $o(|S|)$ more bits. In some cases, this bitvector is compressible, so our code can operate both on the plain encoding and on its compressed versions. Overall, we can answer queries over arbitrary ranges of the human genome in a few tens of milliseconds, taking just a few extra megabytes of space. No tool for fast range queries over a compact matching statistics encoding existed before.

Finally, we describe a lossy compression scheme that can reduce the size of our compact encoding to much less than $2\left|S\right|$ bits when S and T are dissimilar, by replacing small matching statistics values (that typically arise from random matches) with other, suitably chosen small values. In practice, this is most useful in applications that need the matching statistics array of every pair of genomes in a large dataset. The threshold of our lossy compression can be set according to some expected length of matches (see e.g. Haubold and Pfaffelhuber, 2012; Haubold et al., 2009, 2011), or it could be learnt from the distribution of match lengths itself, which usually peaks at noisy values (see e.g. Supplementary Fig. S4). Depending on the threshold, our scheme can shrink the encoding from 40% to 10% of its original size of 2 bits per character, and one of our variants achieves 2% for large thresholds. Another popular data structure in string indexing, the permuted longest common prefix array (Sadakane, 2007), has a similar bitvector encoding and shrinks at similar rates under our scheme in practice.

Our compression method bears some similarities to the lossless algorithm by Boffa et al. (2021), which builds an approximation of the select function on arbitrary bitvectors, and stores corrections: in our case, discarding the corrections would amount to replacing every matching statistics value (regardless of whether it is small or large) with another value (which could be either bigger or smaller) within a user-specified error. This might be undesirable for matching statistics, since there is often an expected length of random matches, and large values that carry information should better be kept intact for downstream analysis. The two lossy schemes are incomparable. In practice the one by Boffa et al. (2021) tends to produce smaller files, since it has more degrees of freedom; our methods manage to achieve compression rates of similar magnitude in several cases (see Supplementary Fig. S15). The lossless version of Boffa et al. (2021)expands our bitvectors for all settings (see Supplementary Fig. S3).

2 Preliminaries and notation

2.1 Strings and string indexes

Let $\Sigma =[\mathrm{1..}\sigma ]$ be an integer alphabet, and let $T\in {\Sigma }^{+}$ be a string. We call the reverse of T the string $\overline{T}$ obtained by reading T from right to left, and we denote by $f_T(W)$ the number of occurrences (or frequency) of string W in T. For reasons of space, we assume the reader to be already familiar with the notion of suffix tree $\mathrm{S}{\mathrm{T}}_T=(V,E)$ of T, which we do not define here. We just recall that every edge in E is labeled with a string of length possibly greater than one, and that a substring W of T can be extended to the right with at least two distinct characters iff $W=\ell (v)$ for some internal node v of the suffix tree, where $\ell (v)$ is the string label of node $v\in V$ obtained by concatenating the label of every edge in the path from the root to v. It is well-known that all the nodes in a suffix tree path have distinct frequencies, which decrease from top to bottom. If u is a node of the suffix tree of T, we use $f_T(u)$ as a shorthand for $f_T(\ell (u))$. We assume the reader to be familiar with the notion of suffix link connecting a node v with $\ell (v)=aW$ for some $a\in [\mathrm{1..}\sigma ]$, to a node w with $\ell (w)=W$. Here, we just recall that inverting the direction of all suffix links yields the so-called explicit Weiner links. Given an internal node v of $\mathrm{S}{\mathrm{T}}_T$ and a symbol $a\in [\mathrm{1..}\sigma ]$, it might happen that string $a\ell (v)$ occurs in T, but that it is not the label of any internal node: all such left extensions of internal nodes that end in the middle of an edge or in a leaf are called implicit Weiner links. An internal node of $\mathrm{S}{\mathrm{T}}_T$ can have several outgoing Weiner links, and every one of them is labeled with a distinct character.

We call suffix tree topology a data structure that supports operations on the shape of $\mathrm{S}{\mathrm{T}}_T$, like $\mathrm{parent}(v)$, which returns the parent of a node v; $\mathrm{lca}(u,v)$, which returns the lowest common ancestor of nodes u and v; $\mathrm{leftmostLeaf}(v)$ and $\mathrm{rightmostLeaf}(v)$, which compute the identifier of the leftmost (respectively, rightmost) leaf in the subtree rooted at node v; $\mathrm{selectLeaf}(i)$, which returns the identifier of the ith leaf in preorder traversal; $\mathrm{leafRank}(v)$, which computes the number of leaves that occur before leaf v in preorder traversal. It is known that the topology of an ordered tree with n nodes can be represented using $2n+o(n)$ bits as a sequence of 2n balanced parentheses, and that $2n+o(n)$ bits suffice to support every operation described above in constant time (Navarro and Sadakane, 2014; Sadakane and Navarro, 2010). We assume the reader to be familiar also with the Burrows–Wheeler transform of T (denoted $\mathrm{BW}{\mathrm{T}}_T$ in what follows). Here, we just recall that every suffix tree node corresponds to a compact lexicographic interval in the BWT, and that following a Weiner link in the suffix tree, i.e. extending a string $W=\ell (v)$ to the left with one character, corresponds to the well-known backward step from the BWT interval of W. We also mention the classical operations $\mathrm{rank}(T,a,i)$, which returns the number of occurrences of character a in string T up to position i, inclusive; and $\mathrm{select}(T,a,i)$, which returns the position of the ith occurrence of a in T. In what follows we omit subscripts that are clear from the context, and we use $\overline{\mathrm{ST}}$ and $\overline{\mathrm{BWT}}$ as shorthands for $\mathrm{S}{\mathrm{T}}_{\stackrel{ }{}}$_T¯ and $\mathrm{BW}{\mathrm{T}}_{\stackrel{ }{}}$_T¯ , respectively.

2.2 Matching statistics in small space

As mentioned, given a query string $S\in {\Sigma }^m$, we call matching statistics $\mathrm{M}{\mathrm{S}}_{S,T}[\mathrm{0..}m-1]$ an array such that $\mathrm{M}{\mathrm{S}}_{S,T}[i]$ is the length of the longest prefix of $S[i\mathrm{..}m-1]$ that occurs somewhere in T without errors. In this paper, we work with the compact representation of $\mathrm{M}{\mathrm{S}}_{S,T}$ as a bitvector $\mathrm{m}{\mathrm{s}}_{S,T}[\mathrm{0..2}|S|-1]$ of $2\left|S\right|$ bits, which is built by appending, for each $i\in [\mathrm{0..}|S|-1]$ in increasing order, $\mathrm{M}{\mathrm{S}}_{S,T}[i]-\mathrm{M}{\mathrm{S}}_{S,T}[i-1]+1$ zeros followed by a one (Belazzougui and Cunial, 2014). $\mathrm{M}{\mathrm{S}}_{S,T}[-1]$ is assumed to be one. Since the number of zeros before the ith one in $\mathrm{ms}$ equals $i+\mathrm{MS}[i]$, one can compute $\mathrm{MS}[i]$ for any $i\in [\mathrm{0..}|S|-1]$ using select operations on $\mathrm{ms}$: $\mathrm{MS}[i]=\mathrm{select}(\mathrm{ms},1,i)-2i$. As customary, $\mathrm{select}(T,a,i)$ is the position of the ith occurrence of character a in string T, starting from zero. $\mathrm{rank}(T,a,i)$ is the number of occurrences of a in T up to position i, inclusive and starting from zero. We also work with the algorithm by Belazzougui and Cunial (2014), which we summarize here for completeness. This offline algorithm computes $\mathrm{ms}$ using both a backward and a forward scan over S, and it needs in each scan just $\mathrm{BWT}$ with rank support, and the topology of $\mathrm{ST}$, or just $\overline{\mathrm{BWT}}$ with rank support, and the topology of $\overline{\mathrm{ST}}$. The two phases are connected via a bitvector $\mathrm{runs}[\mathrm{1..}|S|-1]$, such that $\mathrm{runs}[i]=1$ iff $\mathrm{MS}[i]=\mathrm{MS}[i-1]-1$, i.e. iff there is no zero between the ith and the $(i-1)$th ones in $\mathrm{ms}$.

First, we scan S from right to left, using $\mathrm{BWT}$ with rank support, and the suffix tree topology of T, to determine the runs of consecutive ones in $\mathrm{ms}$. Assume that we know the interval $[i\mathrm{..}j]$ in $\mathrm{BWT}$ that corresponds to substring $W=S[k\mathrm{..}k+\mathrm{MS}[k]-1]$, as well as the identifier of the proper locus v of W in the topology of $\mathrm{ST}$. We try to perform a backward step using character $a=S[k-1]$: if the resulting interval $[i\prime \mathrm{..}j\prime ]$ is non-empty, we set $\mathrm{runs}[k]=1$ and we reset $[i\mathrm{..}j]$ to $[i\prime \mathrm{..}j\prime ]$. Otherwise, we set $\mathrm{runs}[k]=0$, we replace v by $\mathrm{parent}(v)$ using the topology, we update $[i\mathrm{..}j]$ to the interval of the parent of v, and we try another backward step with character a from $[i\mathrm{..}j]$. We keep doing parent operations, followed by backward step trials, until the backward step succeeds or v is the root of $\mathrm{ST}$.

In the second phase, we scan S from left to right, and we build $\mathrm{ms}$ using $\overline{\mathrm{BWT}}$ with rank support, the suffix tree topology of $\overline{T}$, and bitvector $\mathrm{runs}$. Assume that we know the interval $[i\mathrm{..}j]$ in $\overline{\mathrm{BWT}}$ that corresponds to substring $W=S[k\mathrm{..}h-1]$ such that $\mathrm{MS}[k-1]=h-k$ but $\mathrm{MS}[k]\ge h-k$. We try to perform a backward step with character $S[h]$: if the backward step succeeds, we continue issuing backward steps with the following characters of S, until we reach a position $h^{*}$ in S such that a backward step with character $S[h^{*}]$ from the interval $[i^{*}\mathrm{..}{j}^{*}]$ of substring $W^{*}=S[k\mathrm{..}{h}^{*}-1]$ in $\overline{\mathrm{BWT}}$ fails. At this point we know that $\mathrm{MS}[k]=h^{*}-k$, so we append $h^{*}-k-\mathrm{MS}[k-1]+1=h^{*}-h+1$ zeros and a one to $\mathrm{ms}$. Moreover, we iteratively reset the current interval in $\overline{\mathrm{BWT}}$ to the interval of $\mathrm{parent}(v^{*})$, where $v^{*}$ is the proper locus of $W^{*}$ in $\overline{\mathrm{ST}}$, and we try another backward step with character $S[h^{*}]$, until we reach an interval $[i\prime \mathrm{..}j\prime ]$ for which the backward step succeeds. Let this interval correspond to substring $W\prime =S[k\prime \mathrm{..}{h}^{*}-1]$. Note that $\mathrm{MS}[k\prime ]>\mathrm{MS}[k\prime -1]-1$ and $\mathrm{MS}[x]=\mathrm{MS}[x-1]-1$ for all $x\in [k+\mathrm{1..}k\prime -1]$, so $k\prime$ is the position of the first zero to the right of position k in $\mathrm{runs}$, and we can append $k\prime -k-1$ ones to $\mathrm{ms}$. Finally, we repeat the whole process from substring $S[k\prime \mathrm{..}{h}^{*}]$ and its interval in $\overline{\mathrm{BWT}}$. This algorithm can be easily extended to compute the frequency of every $\mathrm{M}{\mathrm{S}}_{S,T}[i]$ in T: see Supplementary Section S2.

3 Computing matching statistics in parallel

It is natural to try and parallelize the construction of MS when query strings are long. In the case of proteomes, or of concatenations of several small genomes, reads or contigs, one could just split the query in chunks of approximately equal size along concatenation boundaries, and process each chunk in parallel. For the large, contiguous genome assemblies that are increasingly achievable with long reads, one could compute MS in parallel for each chromosome, but chromosomes might have widely different lengths and their number might be much smaller than the number of cores available. In this section, we describe algorithms for computing MS in parallel for long query strings, without assuming that they are the concatenation of shorter strings.

We work in the concurrent read, exclusive write (CREW) model of a parallel random-access machine, in which multiple processors are allowed to read from the same memory location at the same time, but only one processor is allowed to write to a memory location at any given time. Let $S^0,\dots ,S^{t-1}$ be a partition of S into t blocks of equal size, and let p_i be the first position of block Sⁱ. Once S and $\mathrm{runs}$ are loaded in memory, one can build $\mathrm{ms}$ in parallel with t threads, by computing $\mathrm{MS}[p_i,\dots ,p_{i+1}-1]$ independently for each i: this works since every thread can safely read the suffix of S to the right of its own block, as well as the corresponding positions of $\mathrm{runs}$. Recall, however, that the algorithm outputs a compact representation of array $\mathrm{MS}$, rather than array $\mathrm{MS}$ itself. Let $\mathrm{m}{\mathrm{s}}^i$ be the bitvector representation of $\mathrm{MS}[p_i,\dots ,p_{i+1}-1]$. Thread i computes $\mathrm{m}{\mathrm{s}}^i$ starting from position p_i of S, and it appends to the beginning of $\mathrm{m}{\mathrm{s}}^i$ a sequence of $\mathrm{MS}[p_i]$ zeros and a one; however, in the final bitvector $\mathrm{ms}$, such a sequence of bits should be replaced by a sequence of $\mathrm{MS}[p_i]-\mathrm{MS}[p_i-1]+1$ zeros and a one. We perform this correction in a final pass, in which a single thread concatenates all output bitvectors. Specifically, we use $\mathrm{MS}[|S^0|-1]=\left|\mathrm{m}{\mathrm{s}}^0\right|-2\left|S^0\right|+1$ to correct the first run of zeros of $\mathrm{m}{\mathrm{s}}^1$, and so on for the other blocks.

We compute bitvector $\mathrm{runs}$ with t parallel threads, as follows. Let $R^0,\dots ,R^{t-1}$ be the partition of $\mathrm{runs}$ induced by blocks $S^0,\dots ,S^{t-1}$. Thread i executes the algorithm for computing $\mathrm{runs}$ independently just inside blocks Sⁱ and Rⁱ, starting with filling the last bit of Rⁱ. Assume that thread i, while proceeding from right to left, sets bit b_i of Rⁱ to zero, and that it sets $R^i[b_i+\mathrm{1..}|R^i|-1]$ to all ones. All the bits that thread i sets in $R^i[\mathrm{0..}{b}_i]$ are correct, since they can be decided without looking at blocks $S^{i+1},\dots ,S^{t-1}$. However, to decide the value of bits $R^i[b_i+\mathrm{1..}|R^i|-1]$ one needs to look at the blocks that follow Sⁱ. We call marked the last block $R^{t-1}$, as well as any block Rⁱ that contains a zero after this phase. If Rⁱ is marked, let $W^i=S[p_i\mathrm{..}{p}_i+\mathrm{MS}[p_i]-1]$: then, thread i stores the BWT interval and the topology identifier of the locus of W_i in $\mathrm{S}{\mathrm{T}}_T$. In practice, we expect b_i to be close to $\left|R^i\right|-1$, and we expect most blocks to be marked. However, there could be an Rⁱ that contains no zero after this phase. Thus, we have to run a second phase in which, for every marked block Rⁱ, we start a thread that updates all the one-bits, in all blocks between $R^{i-1}$ and the rightmost marked block R^j before Rⁱ, including the suffix of R^j after its last zero. We perform this correction using the information stored in the previous phase. Note that this strategy might result both in using fewer than t threads (since we issue just one thread per marked block), and in linear time per thread, since the number of one-bits that a thread might have to update could be proportional to $\left|S\right|$.

These problems occur when S and T have long exact matches, and they become extreme when S = T. Thus, we experiment with the pairs of similar genomes and proteomes described in Supplementary Section S1. The construction of both $\mathrm{runs}$ and $\mathrm{ms}$ scales well on genomes and proteomes of similar species, although achieving the ideal speedup gets more difficult as the number of threads i ncreases (Fig. 1). Correcting the $\mathrm{runs}$ bitvector takes a negligible fraction of the total time for processing $\mathrm{runs}$, even for similar genomes, and it takes more time for proteomes than for genomes, probably because the proteomes of related species are more similar to one another than their genomes (see Supplementary Fig. S1). The fraction of time spent in correcting $\mathrm{runs}$ grows with the number of threads, probably because more threads imply shorter blocks, and shorter blocks are more likely to intersect with exact matches between S and T, or to be fully contained in them. Correcting the $\mathrm{ms}$ bitvector takes even less time than correcting $\mathrm{runs}$. Even running the algorithm on the very similar pairs of chromosome 1 from different human individuals shows the same trends (Supplementary Fig. S1). When S = T, correcting $\mathrm{runs}$ uses just one thread, since only the last block is marked, and it takes time proportional to $\left|S\right|(t-1)/t$ (Fig. 1); building $\mathrm{ms}$ uses all threads, but each one of them has to process the whole suffix of S that starts from its block, thus there is no speedup with respect to the sequential version. In the following section, we describe a way to achieve better asymptotic complexity even when S = T.

Fig. 1.

Scaling of our parallel implementation as the number of threads t increases. Line: ideal scaling $1/t$. Circles along the line: genomes of similar species, proteomes of similar species, pairs of human chromosome 1 from distinct random individuals. Circles far from the line: identical query and text (human chromosome 1 from two random individuals). Vertical axis: time of the parallel implementation divided by the time of the sequential implementation. Correction of $\mathrm{ms}$ is not shown since it is negligible. See Supplementary Figure S1 for a color version

3.1 Better asymptotic complexity

Another way of computing $\mathrm{runs}$ in parallel could be by performing a backward search from the end of every block Sⁱ using $\overline{\mathrm{BWT}}$, by mapping the resulting interval to the corresponding interval in $\mathrm{BWT}$, and by starting the computation of each block from such intervals. This naive approach has the disadvantage of requiring linear time per thread in the worst case to compute the initial BWT intervals of the blocks, and of needing to translate intervals from one BWT to the other. However, the general idea can be used to achieve better complexity, as follows:

Lemma 1.

Let T be a string on alphabet $[\mathrm{1..}\sigma ]$, and assume that we have a representation of $\mathrm{S}{\mathrm{A}}_T$that support suffix array and inverse suffix array queries in O(p) time, and a representation of $\mathrm{S}{\mathrm{T}}_T$that support $\mathrm{weinerLink}$and $\mathrm{parent}$queries in O(q) time. Given a query string S and t processors, we can compute $\mathrm{m}{\mathrm{s}}_{S,T}$in time $O(|S|·q/t+\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|·p\hspace{0.17em}\text{log}\hspace{0.17em}t)$using $O(|S|+t\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$bits of working space.

Proof. Without loss of generality, we assume that t is a power of two. To compute the $\mathrm{runs}$ bitvector, we proceed as follows. First, we split S into t blocks $S^0,\dots ,S^{t-1}$, and we build the BWT interval of every block Sⁱ in parallel, spending $O(|S|q/t)$ time overall. Then, we build the BWT interval of every disjoint superblock that consists of $2^j$ adjacent blocks, for all $j\in [\mathrm{0..}\hspace{0.17em}\text{log}\hspace{0.17em}t-1]$, in $\text{log}\hspace{0.17em}t$ phases. In phase j, we compute the BWT interval of every one of the $t/2^j$ superblocks in parallel, by merging the BWT intervals of the two smaller superblocks from the previous phase that compose it. Every such merge can be performed in $O(p\hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$ time using a data structure that takes $O(|S|)$ additional bits of space (Fischer et al., 2016), so all merging phases take $O(\text{log}\hspace{0.17em}t·p\hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$ time in total. Note that storing the BWT intervals of all superblocks from all phases takes just $O(t\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$ bits of working space. Then we compute, for every $i\in [\mathrm{0..}t-1]$, the largest $j\ge i$ such that $S^{i+1}\cdots S^j$ occurs in T [we call g(i) such a value of j in what follows]. This can be done by assigning a processor to every block Sⁱ, and by making the processor merge the BWT intervals of $O(\text{log}\hspace{0.17em}t)$ pairs of superblocks computed previously. This takes again $O(\text{log}\hspace{0.17em}t·p\hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$ time overall. Finally, for every Sⁱ in parallel, we try to extend the BWT interval of $S^{i+1}\cdots S^{g(i)}$ inside the next block $S^{g(i)+1}$, by performing $O(|S|/t)$ backward steps in overall $O(|S|q/t)$ time. We use the resulting intervals for computing the block of $\mathrm{runs}$ that corresponds to every block Sⁱ, independently and in parallel, in overall $O(|S|q/t)$ time. To compute the block of $\mathrm{ms}$ that corresponds to each Sⁱ, independently and in parallel, we first need to compute the interval in $\overline{\mathrm{BWT}}$ of the longest prefix of $S^i,\dots ,S^{t-1}$ that occurs in T: we compute all such intervals using the same superblock approach described above. □ □

Plugging into Lemma 1 some well-known suffix array representations, we can get $O(|S|\hspace{0.17em}\text{log}\hspace{0.17em}\sigma /t+\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|\hspace{0.17em}\text{log}\hspace{0.17em}|T|\hspace{0.17em}\text{log}\hspace{0.17em}\sigma \hspace{0.17em}\text{log}\hspace{0.17em}t)$ time and $2\left|T\right|\hspace{0.17em}\text{log}\hspace{0.17em}\sigma +O(n)$ bits of space for an integer alphabet of size σ, or $O(|S|/t+\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|\hspace{0.17em}\text{log}\hspace{0.17em}t)$ time and $O(|T|\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$ bits of space for an alphabet that is polynomial in $\left|T\right|$. We can further improve on the complexity of every step of Lemma 1, by using a parallel rather than a sequential algorithm for computing the BWT interval of VW, given the BWT intervals of V and of W. Specifically, we use the algorithm by Fischer et al. (2016), which runs in $O(p\hspace{0.17em}{\text{log}}_t\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$ time with t processors, taking again $O(|S|)$ bits of working space:

Lemma 2.

Given the assumptions of Lemma 1, we can compute$\mathrm{m}{\mathrm{s}}_{S,T}$in time $O(|S|·q/t+\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|·p\hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}t+p\hspace{0.17em}\text{log}\hspace{0.17em}t)$using $O(|S|+t\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$bits of working space.

Proof. To build the BWT interval of every superblock, we proceed as follows. In phase j we have to merge $t/2^j$ pairs of BWT intervals (one for each superblock of $2^j$ blocks), thus we can afford to allocate $2^j$ processors to each merge: it is easy to see that this yields $O(\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|·p\hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}t+p\hspace{0.17em}\text{log}\hspace{0.17em}t)$ time overall. Then, to compute g(i) for each i, we proceed as follows. If we had to solve the problem just for the blocks whose ID is a multiple of $\sqrt{t}$, we could allocate $\sqrt{t}$ processors to each task and be done in $O(\text{log}\hspace{0.17em}t·p\hspace{0.17em}{\text{log}}_{\sqrt{t}}\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$ time, which is $O(p\hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$. More generally, we could organize the computation in $O(\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}t)$ iterations: at iteration j we solve the problem for all remaining blocks whose ID is a multiple of $t^{2^{-j}}$ for increasing j (i.e. from larger to smaller offsets). Thus, every block i that we want to solve at iteration j lies between two blocks $i_x<i<i_{x+1}$ that we solved at iteration j − 1. Clearly there are $t^{2^{-j+1}}-1$ total blocks between $S^{i_x}$ and $S^{i_{x+1}}$ (excluded), thus in the current iteration we have to compute the solution for $t^{2^{-j}}-1$ blocks that lie between $S^{i_x}$ and $S^{i_{x+1}}$. Moreover, since we are dealing with matching statistics, $g(i_x)\le g(i)\le g(i_{x+1})$, and the sum of $g(i_{x+1})-g(i_x)$ over all x is at most t. It follows that, if we assigned $g(i_{x+1})-g(i_x)$ processors to compute each solution between $S^{i_x}$ and $S^{i_{x+1}}$, we would end up using $t·(t^{2^{-j}}-1)$ processors in total: since we have just t processors, we should thus assign $r=(g(i_{x+1})-g(i_x))/(t^{2^{-j}}-1)$ processors to each solution (actually, since at iteration j we compute up to $t/t^{2^{-j}}$ total solutions, we could afford to allocate $r=((g(i_{x+1})-g(i_x))/(t^{2^{-j}}-1)+t^{2^{-j}})/2$ processors per solution). Since $g(i_x)\le g(i)\le g(i_{x+1})$, we need to merge just $O(\text{log}\hspace{0.17em}(g(i_{x+1})-g(i_x)))$ pairs of superblocks to compute the solution for any Sⁱ, thus the total running time of one iteration is $O(\text{log}\hspace{0.17em}(g(i_{x+1})-g(i_x))·p\hspace{0.17em}{\text{log}}_r\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$ using the parallel algorithm by Fischer et al. (2016): it is easy to see that this is $O(p\hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$, thus we get the claimed bound over $O(\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}t)$ iterations. □ □

By plugging into Lemma 2 the same data structures as before, we can get $O(|S|\hspace{0.17em}\text{log}\hspace{0.17em}\sigma /t+\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|\hspace{0.17em}\text{log}\hspace{0.17em}|T|\hspace{0.17em}\text{log}\hspace{0.17em}\sigma \hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}t+\text{log}\hspace{0.17em}|T|\hspace{0.17em}\text{log}\hspace{0.17em}\sigma \hspace{0.17em}\text{log}\hspace{0.17em}t)$ time and $2\left|T\right|\hspace{0.17em}\text{log}\hspace{0.17em}\sigma +O(n)$ bits of space, or $O(|S|/t+\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}|T|\hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}t+\text{log}\hspace{0.17em}t)$ time and $O(|T|\hspace{0.17em}\text{log}\hspace{0.17em}|T|)$ bits of space, which is comparable to the complexity of prefix matching queries described by Fischer et al. (2016).

4 Compressing the matching statistics bitvector

Even though $\mathrm{m}{\mathrm{s}}_{S,T}$ takes just $2\left|S\right|$ bits, storing the bitvector of every pair of genomes in a large dataset for later analysis and querying might still require too much space overall. Real $\mathrm{ms}$ bitvectors, however, have several features that could be exploited for lossless compression. Specifically, if S and T are similar, they are likely to contain long maximal exact matches (MEMs), i.e. triplets $(i,j,\ell )$ such that $S[i\mathrm{..}i+\ell -1]=T[j\mathrm{..}j+\ell -1],\hspace{0.17em}S[i-1]\ne T[j-1]$ and $S[i+\ell ]\ne T[j+\ell ]$. In practice, MEMs tend to be surrounded in S by regions with short matches with T (see the example in Fig. 2), so $\mathrm{MS}[i-1]$ is likely to be short, and the run of $\ell -\mathrm{MS}[i-1]+1$ zeros induced by $S[i\mathrm{..}i+\ell -1]$ in $\mathrm{ms}$ is likely to be long; moreover, in practice $\mathrm{MS}[i\prime ]=\mathrm{MS}[i\prime -1]$ for all $i\prime \in [i+\mathrm{1..}i+\ell -k]$ for some small k (see again Fig. 2). Thus, every MEM is likely to induce a long run of zeros followed by a long run of ones in $\mathrm{ms}$, and if S and T share several long MEMs, run-length encoding (RLE) $\mathrm{ms}$ might save space. Another property of real $\mathrm{ms}$ bitvectors is that the length of a long run of zeros tends to be similar to the length of the following long run of ones, since $\ell -\mathrm{MS}[i-1]+1-\ell +k=k-\mathrm{MS}[i-1]+1$ is likely to be small (see Supplementary Figs S18 and S19). So, given a pair (z_i, o_i) representing a run of z_i zeros and the following run of o_i ones, and given an encoder δ, one might encode the pair as $\delta (z_i)\delta (o_i-z_i)$ if z_i is large, and as $\delta (z_i)\delta (o_i)$ otherwise.

Fig. 2.

Values of matching statistics (vertical axis) in a range of positions along human chromosome 1 (horizontal axis). Query: H.sapiens. Text: P.troglodytes. The right panel is a zoom-in of the left panel. See Supplementary Figure S6 for a bigger range. The PLCP array of a single genome (defined in Section 4) has a similar shape

RLE the bitvectors of pairs of genomes from human individuals using e.g. the $\mathrm{RLEVector}$ data structure by Sirén (2009) yields compression rates of about 20 (Fig. 3, right panel), and compressing the same bitvectors with the $\mathrm{rrr}\_\mathrm{vector}$ data structure from the SDSL library [Gog et al., 2014; which implements the RRR scheme by Raman et al. (2007)] yields compression rates of about 6 (see Supplementary Fig. S8). However, the bitvectors of pairs of genomes from different species are recalcitrant to compression, even when the species are related: RLE expands those files by a factor of 2 (Fig. 3, insert in the left panel), and RRR expands most of them slightly (by a factor of 1.1), and manages to compress just few pairs with rate 1.25 (Supplementary Fig. S8). The same happens with pairs of artificial strings with controlled mutation rate (see Supplementary Figs 16 and 17).

Fig. 3.

Ratio between the size of the $\mathrm{RLEVector}$ data structure by Sirén (2009) built on a permuted $\mathrm{ms}$ bitvector, and the size of the $\mathrm{bit}\_\mathrm{vector}$ data structure from SDSL built on the original $\mathrm{ms}$ bitvector, for the D, DL and ND lossy variants on pairs of genomes from different species and on pairs of genomes from human individuals, allowing and disallowing negative MS values. Size is measured on disk. The vertical axis in the middle panel shows negative powers of 10. Computation is exact for windows with up to 300 zeros and 300 ones, then it uses the first greedy strategy described in the text

In some applications, including genome comparison, short matches are considered noise by the user, and the precise length of a match can be discarded safely as long as we keep track that at that position the match was short. Given an array $\mathrm{M}{\mathrm{S}}_{S,T}$ and a user-defined threshold τ, let a thresholded matching statistics array $\mathrm{M}{\mathrm{S}}_{S,T,\tau }$ be such that $\mathrm{M}{\mathrm{S}}_{S,T,\tau }[i]=\mathrm{M}{\mathrm{S}}_{S,T}[i]$ if $\mathrm{M}{\mathrm{S}}_{S,T}[i]\ge \tau$, and $\mathrm{M}{\mathrm{S}}_{S,T,\tau }[i]$ equals an arbitrary (possibly negative) value smaller than τ otherwise. In some applications, τ might even change along S, e.g. when S is the concatenation of several genomes with different similarity to T. This notion is symmetrical to the one defined by Cunial et al. (2019), which discards instead long MS values in order to prune the suffix tree topologies and to make the data structures smaller. Given an encoder δ, we are interested in the $\mathrm{M}{\mathrm{S}}_{S,T,\tau }$ array whose $\mathrm{m}{\mathrm{s}}_{S,T,\tau }$ bitvector takes the smallest amount of space when encoded with δ. In what follows, we drop S and T from the subscripts whenever they are clear from the context.

Note that every $\mathrm{m}{\mathrm{s}}_{S,T,\tau }$ is a permutation of $\mathrm{m}{\mathrm{s}}_{S,T}$, since the two bitvectors must contain the same number of zeros and ones. Moreover, if $\mathrm{MS}[x]\ge \tau$ corresponds to the one-bit at position y in $\mathrm{ms}$, then every $\mathrm{m}{\mathrm{s}}_{\tau }$ must also have a one at position y, which corresponds to $\mathrm{M}{\mathrm{S}}_{\tau }[x]$ and is preceded by the same number of ones and zeros as position y in $\mathrm{ms}$ (this follows from the fact that $\mathrm{MS}[x]=\mathrm{select}(\mathrm{ms},x,1)-2x$). Let $x_0,\dots ,x_{k-1}$ be the sequence of all and only the positions of S whose MS value is at least τ, and let $y_0,\dots ,y_{k-1}$ be the sequence of the corresponding one-bits $y_i=\mathrm{select}(\mathrm{ms},x_i,1)$ in $\mathrm{ms}$. Clearly it can happen that $x_{i+1}=x_i+1$; if this does not happen, then $\mathrm{MS}[x_i]$ must be equal to τ, and $\mathrm{ms}[y_i+1]$ must be a one and $\mathrm{ms}[y_{i+1}-1]$ must be a zero, both in $\mathrm{ms}$ and in any $\mathrm{m}{\mathrm{s}}_{\tau }$. Thus, if we compress $\mathrm{m}{\mathrm{s}}_{\tau }$ by delta-coding the length of every run, we can build an $\mathrm{m}{\mathrm{s}}_{\tau }$ that is smallest after compression, by concatenating a permutation of every such interval $[y_i\mathrm{..}{y}_{i+1}-1]$ of $\mathrm{ms}$ that is smallest after compression, as well as of the non-empty intervals $[\mathrm{0..}{y}_0-1]$ and $[y_{k-1}+\mathrm{1..2}|S|]$ (and all such permutations can be computed in parallel).

Assume that we want to compute a smallest permutation of window $[y_i\mathrm{..}{y}_{i+1}-1]$, where $\mathrm{MS}[x_i]=\tau$ and every run is delta-coded in isolation. Clearly we could just replace the window with $1^p{0}^q$, where p (respectively, q) is total the number of ones (respectively, zeros) in the window; this could make some MS values negative, thus the resulting $\mathrm{m}{\mathrm{s}}_{\tau }$ might not be a valid MS bitvector, and before replacing $\mathrm{ms}$ with $\mathrm{m}{\mathrm{s}}_{\tau }$ one should make sure that any implementation that used $\mathrm{ms}$ handles negative values correctly. Building an MS bitvector without negative values is easy:

Lemma 3.

Given an interval $[y_i+\mathrm{1..}{y}_{i+1}]$of $\mathrm{ms}$, with z total zeros and o total ones, we can compute a smallest permutation with no negative value in $O(zo{\tau }^2)$time and words of space.

Proof. Every permutation of the interval can be represented as a sequence of pairs $(z_0,o_0),(z_1,o_1),\dots ,(z_k,o_k)$ for some $k\ge 0$, where z_i is the length of a run of zeros, o_i is the length of a run of ones, $z_0\ge 0,\hspace{0.17em}{z}_i>0$ for all i > 0, and $o_i>0$ for all $i\ge 0$. We work with the sequence of cumulative pairs $(Z_0,O_0),(Z_1,O_1),\dots ,(Z_k,O_k)$, where $Z_i=\sum _{j=0}^i{z}_i$ and $O_i=\sum _{j=0}^i{o}_i$. Given a pair (Z_i, O_i), we use $\mathrm{MS}(Z_i,O_i)$ as a shorthand for $\tau +Z_i-O_i$ (i.e. the MS value that corresponds to the last one-bit of the pair), and we say that the pair is valid iff it satisfies $Z_i<z,\hspace{0.17em}{O}_i<o$, and $\mathrm{MS}(Z_i,O_i)\in [\mathrm{0..}\tau -1]$. We draw a directed arc from every valid pair (Z_i, O_i) to every other valid pair (Z_j, O_j) such that Z_j > Z_i, O_j > O_i, and $\mathrm{MS}(Z_i,O_i)+Z_j-Z_i-1<\tau$ (this is the MS value of the first one-bit in the last run of ones in the pair), and we assign cost $\delta (Z_j-Z_i)+\delta (O_j-O_i)$ to the arc. Moreover, we add the invalid pair (z, o), we connect it to every valid pair $(Z_i,o-1)$, and we assign cost $\delta (z-Z_i)$ to the arc. A start pair (Z_i, O_i) is a valid pair with Z_i = 0, and it is assigned cost $\delta (O_i)$. A permutation of smallest size corresponds to a path in the resulting DAG $G=(V,E)$, from a start pair to pair (z, o), that minimizes the sum of the costs of its arcs plus the cost of the start pair. This can be derived by computing, for every node $v\in V$ that does not correspond to a start pair, quantity $f(v)=\min \{f(u)+c(u,v):(u,v)\in E\}$, using dynamic programming over the topologically-sorted DAG. □ □

One can easily modify this construction to enforce MS values in the permuted interval to be at least a positive number, rather than zero. To make compression faster in practice, we fix z and o to a large value and, for every τ used by the target application, we precompute and store a variant of the DAG that answers every possible query of length at most z + o: in addition to (z, o), this variant includes every pair (Z_i, O_i) with $\mathrm{MS}(Z_i,O_i)\ge \tau$, it connects it to all valid pairs $(Z_j,O_i-1)$ as described for (z, o), and it computes the min-cost path to every node. To permute a window with $z\prime$ zeros and $o\prime$ ones such that $z\prime \le z,\hspace{0.17em}o\prime \le o$, and $z\prime +o\prime \le z+o$, we go to node $(z\prime ,o\prime )$ in the DAG and we backtrack along an optimal precomputed path. If $(z\prime ,o\prime )$ does not belong to the DAG, we select a valid in-neighbor (Z_i, O_i) of $(z\prime ,o\prime )$ using a greedy strategy (e.g. the neighbor that maximizes $g_i=z\prime -Z_i+o\prime -O_i$ or $g_i/(\delta (z\prime -Z_i)+\delta (o\prime -O_i))$): if (Z_i, O_i) belongs to the DAG, we backtrack, otherwise we take another greedy step. In what follows, we label this approach ‘ND’.

As mentioned, in real MS bitvectors the length of a run of zeros and of the following run of ones tend to be similar: we can take this into account by setting the cost of an arc between (Z_i, O_i) and (Z_j, O_j) to $\delta (x)+\delta (g(y|x))$, where $x=Z_j-Z_i,\hspace{0.17em}y=O_j-O_i$ and $g(y|x)$ is the following map: since $y-x\ge 1-x$, we map all the negative values of y − x to the even integers up to $2(x-1)$ in increasing order of $|y-x|$, we map the positive values of y − x up to x − 2 to the odd integers $\ge 3$, and we map every remaining value of y − x to y. We use integer one to encode y = x. Recall that the interval of $\mathrm{ms}$ that we want to permute is $[y_i\mathrm{..}{y}_{i+1}-1]$, where y_i belongs to a (possibly long) run of one-bits, and $y_{i+1}$ is the first one-bit of a (possibly long) run. We might not want to alter the lengths of such runs of ones, so we might be interested in permuting just the subinterval $[p\mathrm{..}q]$ where p is the first zero after y_i and q is the last one before $y_{i+1}$ (if negative values of MS are allowed, the trivial scheme of writing all the ones at the beginning of $[p\mathrm{..}q]$ cannot be used, since it would alter the length of the run of y_i). We call this variant ‘D’ in what follows. Since in practice the correlation between the length of a run of zeros and the following run of ones is strong only for long runs, one might want to encode a run of x zeros ad the following run of y ones as $\delta (x)+\delta (g(y|x))$ only when $x\ge \tau$, and to encode it as $\delta (x)+\delta (y)$ otherwise. This would require permuting just $[p\mathrm{..}q]$, but in an optimal way with respect to the latter encoding. We call this variant ‘DL’ in what follows.

When S and T are dissimilar, run-length compressing the permuted $\mathrm{ms}$ bitvectors expands them for small values of τ when negative MS values are not allowed (Fig. 3, insert in the left panel). For τ = 32, RLE most of our $\mathrm{m}{\mathrm{s}}_{\tau }$ variants shrinks the bitvector to ∼40% of its original size, and increasing τ progressively brings its size down to 10% of the original. We do not detect any clear difference in performance between the variants, with D being significantly smaller in some but not all cases (Supplementary Fig. S11). A detailed analysis of how the permutation schemes compare when varying the similarity between query and text is provided in Supplementary Figures S16 and S17. For pairs of genomes from human individuals, RLE the original $\mathrm{ms}$ bitvector already brings its size down to ∼4.5% of the original, and increasing τ shrinks the bitvectors to 2% of the input (Fig. 3, right panel). Allowing for negative MS values compresses some pairs of genomes from different species already at τ = 16, and for $\tau \ge 32$ it shrinks the bitvector to ∼2% of the original (Fig. 3, center panel). Negative MS values do not give any significant gain for genomes of individuals (data not shown). Pairs of proteomes display similar trends, but this time RLE is able to compress some $\mathrm{ms}$ bitvectors, and τ = 8 is enough to compress most pairs (Supplementary Figs S9 and S10). Finally, we test our lossy compression on the permuted longest common prefix array (PLCP) of the genomes in our dataset, since this data structure is amenable to a compact encoding that is very similar to $\mathrm{ms}$ (Sadakane, 2007): we observe again a shrinkage from 40% to 10% of the original size when setting $\tau \ge 32$ (Supplementary Fig. S14).

Clearly, when very few MS values are above threshold, storing just those values might take less space than compressing the $\mathrm{ms}$ bitvector. We call $A_{\tau }$ a scheme that stores every MS value at least τ and its position in the minimum number of bits necessary to encode the respective numbers, and we call $B_{\tau }$ a scheme in which every MS value at least τ is stored in $\lceil \hspace{0.17em}\text{log}\hspace{0.17em}M\rceil$ bits and every position is stored in $\lceil \hspace{0.17em}\text{log}\hspace{0.17em}L\rceil$ bits, where M is the maximum observed MS value and L is the length of the query. Accessing MS from such structures might be much slower than using the $\mathrm{ms}$ bitvector with select queries. For $\tau \ge 32$, large values are rare enough in the MS arrays of genomes from different species that most compressed bitvectors take much more space than $A_{\tau }$ or $B_{\tau }$ (Supplementary Fig. S12). When negative values of MS are allowed; however, the permuted bitvectors of several pairs of genomes become smaller than or comparable to $A_{\tau }$ and $B_{\tau }$ (Supplementary Fig. S13), and for pairs of individuals, the permuted bitvectors are always two or three orders of magnitude smaller than $A_{\tau }$ and $B_{\tau }$, since 80% or more of all MS values are above threshold for every τ (Supplementary Fig. S12).

5 Querying the matching statistics bitvector

As mentioned, it is natural to formulate questions on the similarity between a substring of the query and the whole text in terms of matching statistics, and this approach has already been used in bioinformatics for detecting horizontal gene transfer and other structural variations between two genomes. In this section, we focus on two types of range query, which we implement on the $\mathrm{ms}$ bitvector: given an interval $[i\mathrm{..}j]$, we want to return either $\sum _{k=i}^j\mathrm{MS}[k]$ (e.g. to compute a local version of the score by Ulitsky et al. (2006)) or $\max \{\mathrm{MS}[k]:k\in [i\mathrm{..}j]\}$ (e.g. to detect the presence of significant matches).

We answer the max query using the standard approach of dividing the $\mathrm{ms}$ bitvector into blocks with a fixed number of bits, extracting for each block the maximum MS value that corresponds to a one-bit in the block, and building a range-maximum query (RMQ) data structure on such values. Given a range $\mathrm{MS}[i\mathrm{..}j]$ in the query, we find the block $i\prime$ of $\mathrm{ms}$ that contains the ith one-bit, the block $j\prime$ that contains the jth one-bit, and we query the RMQ data structure on the range of blocks $i\prime +\mathrm{1..}j\prime -1$ (if it is not empty): this returns the index of a block with largest value, thus we perform a linear scan of the returned block, as well as of the suffix of block $i\prime$ and of the prefix of block $j\prime$ (if any). We implement this approach using the $\mathrm{rmq}\_\mathrm{succinct}\_\mathrm{sct}$ and $\mathrm{bit}\_\mathrm{vector}$ data structures from the SDSL library (Gog et al., 2014). For ranges $i\mathrm{..}j$ approximately equal to two blocks or larger, this method allows answering a range-max query in the same time as scanning two full blocks and querying the RMQ (see Fig. 4); for shorter ranges, it takes the same time as a linear scan of the range. In practice, when the query is the human genome and the block size is, say, 1024 bits, we can answer arbitrary range-max queries in a few milliseconds using just two megabytes for the RMQ and 12 megabytes for the precomputed maximum of each block (if we do not want to compute it on the fly). Other space/time tradeoffs are possible, but we omit a detailed analysis for brevity.

Fig. 4.

Diagonal line: non-optimized scanning of the SDSL $\mathrm{bit}\_\mathrm{vector}$ data structure without an index. Plateaus: scanning combined with an RMQ index for different block size B. Every point is the average of 20 random queries of the same size. Vertical dashed lines: query sizes that are approximately equal to two blocks of the $\mathrm{ms}$ bitvector (labels: average size in bits of the query range when mapped to the bitvector). Dataset: H.sapiens and Mus musculus genomes. Similar trends appear for pairs of genomes from human individuals

After taking care of some details, this approach can be applied to compressed versions of the $\mathrm{ms}$ bitvector as well: our implementation supports RRR and RLE using the $\mathrm{rrr}\_\mathrm{vector}$ and $\mathrm{RLEVector}$ data structures, from SDSL and from the RLCSA code by Sirén (2009), respectively. During a scan, issuing one access operation for every bit is clearly suboptimal: instead, in the uncompressed and in the RRR-compressed $\mathrm{ms}$, we extract 64 bits at a time and we look up every byte in a precomputed table. This gives speedups between 2 and 8, depending on dataset and range size (see Supplementary Fig. S20). In the RLE-compressed $\mathrm{ms}$, we process one run at a time, and in very similar strings scanning becomes from a hundred to a thousand times faster than accessing every bit, or more. Overall, scanning the RLE-compressed bitvector of very similar strings processing one run at a time, is approximately 10 times faster than scanning the corresponding uncompressed bitvector processing 64 bits at a time (see Supplementary Fig. S21). Thus, if the target application is not interested in MS values below some threshold, one might swap the uncompressed bitvector with one of the permuted and RLE-compressed variants described in Section 4, and this might speed up range-max queries at no cost.

As customary, by recurring on the output of RMQ queries one can report all blocks in the range with maximum MS value, or all blocks with MS value at least τ, in linear time in the size of the output. Setting the block size to $\text{log}\hspace{0.17em}\left|S\right|$ gives an RMQ data structure of $O(|S|/\hspace{0.17em}\text{log}\hspace{0.17em}|S|)$ bits, and it allows replacing the linear scan of a block with a constant-time lookup from a table of $o(|S|)$ bits in which we store the relative location of a largest value inside each block. We use the RMQ to detect all blocks in the range that contain at least one large value, and we use lookups from another table of $o(|S|)$ bits (which stores offsets between one-bits with MS value at least τ) to report all locations with MS value at least τ in linear time on their number.

The optimized scanning can be applied to range-sum queries as well, with similar speedups (see Supplementary Fig. S21). To implement a range-sum query $[i\mathrm{..}j]$ over an arbitrary range, we just store the prefix sums that correspond to the last one-bit in every block, and we scan the two blocks that contain the one-bit that corresponds to i and the one-bit that corresponds to j. Scanning can also be used to implement other primitives in analytics, like plotting all MS values in a range or their histogram, computing the position k of a longest interval $[k\mathrm{..}k+\mathrm{MS}[k]-1]$ that contains $[i\mathrm{..}j]$, or finding all windows of fixed length k inside $[i\mathrm{..}j]$ with maximum sum.

Financial Support: none declared.

Conflict of Interest: none declared.