Enhancing Data Compression: Recent Innovations in LZ77 Algorithms

Abstract

The growing volume of genomic data, driven by advances in sequencing technologies, demands efficient data compression solutions. Traditional algorithms, such as Lempel-Ziv77 (LZ77), have been fundamental in offering lossless compression, yet they often fall short when applied to the highly repetitive structures typical of genomic sequences. This review explores the evolution of LZ77 and its adaptations for genomic data compression, highlighting specialized algorithms designed to handle redundancy in large-scale sequencing datasets efficiently. Innovations in this field have enhanced compression ratios and processing efficiencies leveraging intrinsic redundancy within genomic datasets. We critically examine a spectrum of LZ77-based algorithms, including newer adaptations for external and semi-external memory settings, and contrast their efficacy in managing large-scale genomic data. We conducted experiments to evaluate the performance of several algorithms, including KKP2, RLE-LZ, SE-KKP, BGone, and PFP-LZ77, on both real-world datasets from the Pizza&Chili repetitive corpus, Salmonella genomes, and human chromosome 19 genomes. These results underscore the trade-offs between time and memory consumption between algorithms. This article aims to provide a comprehensive guide on the current landscape and future directions of data compression technologies, equipping bioinformaticians and other practitioners with insight to tackle the escalating data challenges in genomics and beyond.

1. INTRODUCTION

In the era of increasingly larger datasets, the exponential growth of genomic data generated by advanced sequencing technologies poses significant challenges to data storage and management. The vast amounts of data produced are not only massive in scale, but also exhibit a high degree of redundancy, particularly within and between genomes of the same species (Lukens et al., 2004). This redundancy, while beneficial for data compression, demands sophisticated algorithms to efficiently leverage these repetitive structures for improved storage and retrieval processes.

To address these challenges of managing large and redundant datasets, bioinformatics has turned to compression algorithms that can effectively exploit repetitive structures in genomic data. One of the foundational methods in this area is LZ77- , which is a seminal lossless data compression algorithm developed by Abraham Lempel and Jacob Ziv. $\mathsf{LZ}$77 laid the foundation for numerous modern compression techniques. Published in 1977, $\mathsf{LZ}$77 employs a sliding-window mechanism to find repeated sequences within a fixed-size window, encoding data as a triple (distance, length, and next symbol). The distance in the tuple refers to how far back the encoder must look within the sliding window (the buffer of previously processed data) to find a matching substring. The length indicates how long the matching substring is, providing the decoder with information on how many characters to copy from the position identified by the distance. Finally, the next symbol is the character that follows the matched substring or the character itself when no match is found. This character represents the starting character of the new $\mathsf{LZ}$77 phrase, allowing us to continue processing beyond the matched part when we want to decode the $\mathsf{LZ}77$ sequence. This method allows $\mathsf{LZ}$77 to efficiently compress data by encoding repeated substrings using references to earlier parts of the string rather than storing them explicitly. For example, the string abcabcabcabc encodes the following series of tuples: (0, 0, a), (0, 0, b), (0, 0, c), (3, 3, a), (3, 3, a).

In this review, we trace the evolution of $\mathsf{LZ}$77 algorithms, focusing on innovations that make them suitable for large-scale genomic datasets. Designed for readers new to data compression, this work introduces methods based on $\mathsf{LZ}$77 that address the substantial redundancy and volume characteristic of genomic data, which often contains repetitive elements, especially within species. Efficient compression is crucial for managing the data produced by modern sequencing technologies, and $\mathsf{LZ}$77 has become a versatile tool in bioinformatics. For example, Wertenbroek et al. (2022) use $\mathsf{LZ}$77 to develop a genotype compression tool for large biobank data. Norri et al. (2021) used it to develop a method to identify structural variants from multiple alignments. The tools that use $\mathsf{LZ}$77 have been evaluated for their use for taxonomic identification in metagenomics (Silva and Almeida, 2024). It has also been used for read data compression sequencing (Kowalski and Grabowski, 2020; Castelli et al., 2024). With such varied applications, a wide range of $\mathsf{LZ}$77 algorithms warrants exploration. We also highlight advancements in hardware implementations of $\mathsf{LZ}$77, which address throughput challenges inherent to software-only approaches, enabling faster data processing and supporting real-time genomic analysis. Table 1 summarizes the representative $\mathsf{LZ}$77 algorithms discussed in this review, comparing their complexities in time and space.

Table 1.

Illustration the Time and Space Complexity for in Memory Construction Algorithm

Algorithm	Time complexity	Space complexity
$\mathsf{LZ}$ factorization by Peak Elimination (Ohlebusch and Gog, 2011)	$\mathsf{O}\left(n\right)$	136n bits
Ultra-Fast Lempel-Ziv77 ( $\mathsf{LZ}$77) factorization (Ohlebusch and Gog, 2011)	$\mathsf{O}\left(n\right)$	104n bits
$\mathsf{LZ}$77 factorization backward search (Ohlebusch and Gog, 2011)	$\mathsf{O}\left(n\log \right|\Sigma \left|\right)$	40n bits
Lazy LZ Construction (Kempa and Puglisi, 2013; Goto and Bannai, 2013)	$\mathsf{O}\left(n\right)$	$2n\log n$ bits
KKP1 (Kärkkäinen et al., 2013a)	$\mathsf{O}\left(n\right)$	40n bits
KKP2 (Kärkkäinen et al., 2013a)	$\mathsf{O}\left(n\right)$	45n bits
KKP3 (Kärkkäinen et al., 2013a)	$\mathsf{O}\left(n\right)$	95n bits
BGone (Goto and Bannai, 2014)	$\mathsf{O}\left(n\right)$	$n\log \hspace{0.17em}n$ + $\mathsf{O}\left(\right|\Sigma |\log \hspace{0.17em}n)$ bits
LZoneT (Liu et al., 2016)	$\mathsf{O}\left(n\right)$	$n\log \hspace{0.17em}n$ bits
LZoneSA (Liu et al., 2016)	$\mathsf{O}\left(n\right)$	$n\log \hspace{0.17em}n$ bits
RLE- $\mathsf{LZ}$ (Policriti and Prezza, 2018)	$\mathsf{O}\left(n\log r\right)$	$\mathsf{O}(r\log \hspace{0.17em}n)$ bits
PFP-LZ77 (Hong et al., 2023)	$\mathsf{O}\left(n\right)$	$\mathsf{O}(n\hspace{0.17em}\log \hspace{0.17em}n)$ bits

The symbol $\Sigma$ represents the alphabet, n corresponds to the length of the input, and r is equal to the number of maximal single-character runs in the BWT. For algorithms where experimental space usage was reported in the original articles, we present those measurements. Otherwise, we list the theoretical space bounds.

To assess the practical impact of these advances, we conducted a series of experiments to validate the performance of various $\mathsf{LZ}$77-based algorithms in the management of large-scale genomic data. We evaluated the performance of KKP2, RLE-LZ, PFP-LZ77, and SE-KKP, while KKP1 and KKP3 were excluded due to their inability to support large datasets. Although we reached out to the authors of the LZone algorithm, we were unable to access the source code, and there is currently no available implementation for the sublinear $\mathsf{LZ}$77 algorithm. SE-KKP was selected to represent external memory $\mathsf{LZ}$77 construction, given its demonstrated superior performance in external memory settings in a previous benchmark study. The experiments used real-world datasets from the Pizza&Chili corpus and Salmonella genomes obtained from the NCBI website.

Our results on the Pizza&Chili corpus revealed that all algorithms, except RLE-LZ, required less than 5 GB of memory and were completed in 20 minutes, with RLE-LZ consistently taking the longest time but consuming the least memory. SE-KKP consumed the most memory and time in various scenarios, while KKP2 and PFP-LZ77 performed similarly in terms of both time and memory. For Salmonella data sets, KKP2 was the fastest algorithm, and RLE-LZ, despite using the least memory, failed to complete within the 24-hour time limit for the largest data set. These findings underscore the trade-offs between time and memory efficiency in all algorithms tested.

Through a comprehensive examination of $\mathsf{LZ}$77 factorization algorithms, this article aims to provide a detailed guide on the current state of the art and future directions in this field. Our objective is to equip researchers and practitioners with the knowledge and tools necessary to tackle the challenges of genomic data management, paving the way for more efficient and effective use of genomic information in various biological and medical applications.

2. NOTATION AND PRELIMINARIES

2.1. Basic definitions

We denote a string S as a finite sequence of symbols $S=S[1‥n]=S\left[1\right]‥S\left[n\right]$ over an alphabet $\Sigma =\{c_1,‥,c_{\sigma }\}$ whose symbols have a total order. We denote by $\epsilon$ the empty string. We define the length of S as the number of characters and denote it by | S |. Given a string S, we denote the reverse of S as $\mathsf{rev}(S)$, that is, $\mathsf{rev}(S)=S\left[n\right]\dots S\left[1\right]$. Lastly, we define the rotation of a string as a transformation in which the characters of the string are shifted circularly such that some characters at the beginning of the string are moved to the end. Formally, for a string S of length n, the i-th rotation of S is the string $S_i=S[i+1]S[i+2]‥S\left[n\right]S\left[1\right]S\left[2\right]‥S\left[i\right]$ for 1 $\le i\le n$.

We denote by $S[i‥j]$ the substring $S\left[i\right]\dots S\left[j\right]$ of S starting at position i and ending at position j, with $S[i‥j]=\epsilon$ if $i>j$. For a string S and $1\le i\le n$, $S[1‥i]$ is called the i-th prefix of S, and $S[i‥n]$ is called the i-th suffix of S. We call a prefix $S[1‥i]$ of S a proper prefix if $1\le i<n$. Similarly, we call a suffix $S[i‥n]$ of S a proper suffix if $1<i\le n$.

We denote by < the lexicographic order: for two strings $S[1‥m]$ and $T[1‥n]$, $S<T$ if S is a proper prefix of T, or if there exists an index $1\le i\le \min (n,m)$ such that $S[1‥i-1]=T[1‥i-1]$ and $S\left[i\right]<T\left[i\right]$.

2.2. Rank and select

Given a string S of length n, a symbol $c\in \Sigma$, and an integer i, we define $S.\mathsf{ran}{\mathsf{k}}_c(i)$ as the rank, as the number of occurrences of c in $S[1‥i-1]$. We also define $S.\mathsf{selec}{\mathsf{t}}_c(i)$ as the position in S of the i-th occurrence of c in S if it exists and $n+1$ otherwise. For a bitvector $B[1‥n]$, that is, a string over $\Sigma =\{0,1\}$, to simplify the notation, we will refer to $B.\mathsf{ran}{\mathsf{k}}_1(i)$ and $B.\mathsf{selec}{\mathsf{t}}_1(i)$ respectively as $B.\mathsf{rank}(i)$ and $B.\mathsf{select}(i)$.

The operations $\mathsf{rank}$ and $\mathsf{select}$ efficiently count and locate repeated substrings, which accelerates LZ77’s process of replacing these substrings with references to their previous occurrences.

2.3. Burrows–Wheeler transform

The Burrows–Wheeler transform ( $\mathsf{BW}{\mathsf{T}}_S$) (Burrows and Wheeler, 1994) of a string S is a reversible permutation of the characters of the string S. This transform is obtained by sorting all rotations of a string S in lexicographic order and retaining the last character of each rotation. Let M be the matrix containing all sorted rotations of S, let F and L be the first and last columns of the matrix, then $\mathsf{BW}{\mathsf{T}}_S=\mathsf{L}$. Moreover, let $\mathsf{C}\left[c\right]$ be the number of suffixes starting with a character smaller than c, where c is in $[1,‥,|\Sigma \left|\right]$. We define $\mathsf{ran}{\mathsf{k}}_c(S,i)$ as the number of occurrences of character c in the substring $S[1‥i]$ and define LF-mapping (Burrows and Wheeler, 1994) as $\mathsf{LF}(i,c)=\mathsf{C}\left[c\right]+\mathsf{ran}{\mathsf{k}}_c(\mathsf{BW}{\mathsf{T}}_S,i)$ and $\mathsf{LF}(i)=\mathsf{LF}(i,\mathsf{BW}{\mathsf{T}}_S[i])$. LF-mapping can be used to count the occurrences of a pattern P in S. We refer to this technique as backward search. We note that when it is clear from the context, we remove the subscript in the notation, that is, $\mathsf{BW}{\mathsf{T}}_S$ is denoted as $\mathsf{BWT.}$

Given a string S and its BWT, a run in BWT is a maximal substring of equal characters in BWT. This leads to the definition of the run-length encoded Burrows–Wheeler Transform ( $\mathsf{RLBW}{\mathsf{T}}_S$) (Mäkinen and Navarro, 2004), which offers a more space-efficient variation by compressing sequences of repeated characters in the $\mathsf{BWT}$ sequence into a single run. Again, when it is clear from the context, we remove the subscript in the notation, i.e., $\mathsf{RLBW}{\mathsf{T}}_S$ is RLBWT. Hence, RLBWT is an equivalent representation of BWT, where each run is represented as the character of the run and its length. For example, given the $\mathsf{BWT}$ of string S as bd#bbbbddddbaaaaaaaaaaa, the sequence of characters beginning each run is described as $H=$ bd#bdba. Each character c in S has its run lengths encoded in a corresponding bitvector $B_c$, with a 1 marking the start of each run. In this case, the bitvectors are $B_{\mathit{all}}=11110001000110000000000$, $B_a=10000000000$, $B_{\#}=1$, $B_b=110001$, and $B_d=11000$. Hence, we note that $B_{\mathit{all}}$ denotes the location of the start of each run in S, and the character bit vectors denote the start and length of the runs for each character. The rank and access operations are efficiently translated into the rank, select, and access operations in H, $B_{\mathit{all}}$, and $B_c$ for all characters c. This encoding strategy reduced the storage of $\mathsf{BWT}$ to $\mathsf{O}(r)$, where r denotes the number of runs in the $\mathsf{BWT}$, since there exists a constant number of 1 bits for each run in this encoding.

2.4. Suffix array and inverse suffix array

The suffix array (Manber and Myers, 1993) ( $\mathsf{S}{\mathsf{A}}_S$) is a compact and efficient data structure that implicitly stores all the sorted suffixes of a string S of length n. More specifically, we note that the $\mathsf{S}{\mathsf{A}}_S$ for the string S is a permutation of $1,‥,n$ such that $S\left[\mathsf{S}{\mathsf{A}}_S\right[i]‥n]$ represents the i-th lexicographically smallest suffix of S. Relating this back to our definition of $\mathsf{BWT}$, this is the starting position of each of the rotations in the matrix M.

Complementing the $\mathsf{S}{\mathsf{A}}_S$ is the inverse suffix array, we denoted it as $\mathsf{IS}{\mathsf{A}}_S$. This serves as a mapping back to the lexicographical ranks of the suffixes, allowing one to determine the rank of each suffix in the lexicographical order efficiently. More formally, we find that $\mathsf{ISA}$ is defined so that $\mathsf{IS}{\mathsf{A}}_S\left[\mathsf{S}{\mathsf{A}}_S\right[i\left]\right]=i$ for all $i\in [1‥n]$. We remove the subscript from the notation when it is clear from the context, i.e., $\mathsf{S}{\mathsf{A}}_S$ is $\mathsf{SA}$ and $\mathsf{IS}{\mathsf{A}}_S$ is $\mathsf{ISA}$. The suffix array can help improve the efficiency of $\mathsf{LZ}$77 compression, as follows for the rapid identification of repeated substrings, which $\mathsf{LZ}$77 replaces with references to earlier positions in the string.

2.5. Longest common prefix array

Given two strings $S[1‥n]$ and $T[1‥m]$, we refer to the longest common prefix as the substring $S[1‥\mathcal{l}]=T[1‥\mathcal{l}]$ such that $\mathcal{l}=\max \{i\mid S[1‥i]=T[1‥i\left]\right\}$ and denote this by $\mathsf{lcp}(S,T)$. Next, we define the longest common prefix ( $\mathsf{LC}{\mathsf{P}}_S$) array as the data structure that stores the lengths of the longest common prefixes between consecutive suffixes in the $\mathsf{SA}$ of a string S. More specifically, $\mathsf{LC}{\mathsf{P}}_S\left[i\right]$ represents the length of the longest common prefix between the suffixes represented by $\mathsf{SA}\left[i\right]$ and $\mathsf{SA}[i-1]$ in S, i.e., $\mathsf{LC}{\mathsf{P}}_S\left[i\right]=\mathrm{lcp}(\mathsf{SA}[i],\mathsf{SA}[i-1])$. When the context is clear, we remove the subscript from the notation. The $\mathsf{LCP}$ can accelerate the process of $\mathsf{LZ}$77 by precomputing and storing the length of common prefixes between consecutive suffixes, optimizing the search for the longest match.

2.6. Longest common extension

The longest common extension ( $\mathsf{LCE}$) array measures the length of the longest common prefix between two suffixes of a string S. Given the indices i and j, $\mathsf{LCE}(i,j)$ is the length of the longest common prefix of the suffixes starting at position i and j in S.

2.7. Range minimum query

Given an array $\mathsf{A}[1‥n]$ of integers and an interval $1\le i\le j\le n$, a range minimum query (RMQ) of the interval [i, j] over the array $\mathsf{A}$, denoted by $\mathsf{RM}{\mathsf{Q}}_{\mathsf{A}}[i,j]$, returns the position k of the minimum value in $\mathsf{A}[i‥j]$. Formally, $\mathsf{RM}{\mathsf{Q}}_{\mathsf{A}}[i,j]=\text{argmin}(\mathsf{A}[k]\mid i\le k\le j)$. The $\mathsf{RMQ}$ is frequently used with $\mathsf{LCP}$ to answer questions such as: Given two suffixes $S\left[\mathsf{SA}\right[i]‥n]$ and $S[\mathsf{SA}[j]‥n]$ of the string $S[1‥n]$, find the value of $\mathsf{lcp}(S[\mathsf{SA}\left[i\right]‥n],S[\mathsf{SA}\left[j\right]‥n])$. This is achieved efficiently by performing an $\mathsf{RMQ}$ on the $\mathsf{LCP}$ array, as described by Kasai et al. (2001). This can be done using $\mathsf{RMQ}$ over $\mathsf{LCP}$. Hence, given two integers $1\le i\le j\le n$ we define $\mathrm{lc}{p}_{\mathsf{SA}}(i,j)$ as the longest common prefix between the i-th and j-th suffix in $\mathsf{SA}$ order, which is equivalent to computing the $\mathsf{LCE}$ of $\mathsf{SA}\left[i\right]$ and $\mathsf{SA}\left[j\right]$. Formally, the length of $\mathrm{lcp}(S[\mathsf{SA}\left[i\right]‥n],S[\mathsf{SA}[j]‥n])$ can be computed as $\mathsf{LCP}\left[\mathsf{RM}{\mathsf{Q}}_{\mathsf{LCP}}\right[i+1,j\left]\right]$ (Kasai et al., 2001).

2.8. $\Phi$ and $\Psi$

Given a string S[1, . . , n] and its $\mathsf{SA}$ and $\mathsf{ISA}$, the $\Phi$ and $\Psi$ are defined to facilitate efficient traversal of lexicographically adjacent suffixes. The $\Phi$ array maps each suffix to the lexicographically previous suffix, for a suffix starting at position i, $\Phi \left[i\right]$ is defined as $\mathsf{SA}\left[\mathsf{ISA}\right[i]-1]$ if $\mathsf{ISA}\left[i\right]>1$, and $\mathsf{SA}\left[n\right]$ otherwise (Kärkkäinen et al., 2009). The $\Phi \left[i\right]$ provides the position of the suffix that is lexicographically immediately smaller than the suffix $S[i‥n]$.

In contrast, $\Psi$ identifies the next lexicographic suffix. Formally, $\Psi \left[i\right]$ is defined as $\mathsf{SA}\left[\mathsf{ISA}\right[i]+1]$, with conditions $\Psi [\mathsf{SA}[n\left]\right]=0$ and $\Psi \left[0\right]=\mathsf{SA}\left[1\right]$ (Grossi and Vitter, 2005). This ensures that $\Psi \left[i\right]$ returns the position of the next lexicographic suffix following $S[i‥n]$ or 0 if no such suffix exists.

2.9. Previous smaller value and next smaller value queries

Previous smaller value (PSV) and next smaller value ( $\mathsf{NSV}$) queries are used frequently to calculate LZ77, as we will see within this survey. These queries concern themselves with an integer array $\mathsf{A}[1‥n]$ and a specific index $1\le i\le n$. The $\mathsf{PSV}$ of $\mathsf{A}$ at index i is defined as the maximum position preceding i within the array where there is a value smaller than $\mathsf{A}\left[i\right]$. Formally

$$\mathsf{P}\mathsf{S}{\mathsf{V}}_{\mathsf{A}}(i)\hspace{0.17em}=\hspace{0.17em}\{\begin{array}{ll}\max \{1\le j<i\mid \mathsf{A}[\hspace{0.17em}j]<\mathsf{A}[i]\}\hfill & \text{if }j\text{ exists,}\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}$$

Similarly, $\mathsf{NSV}$ of $\mathsf{A}$ in the index i is defined as the minimum position succeeding i within the array where there is a value smaller than $\mathsf{A}\left[i\right]$. Formally,

$$\mathsf{N}\mathsf{S}{\mathsf{V}}_{\mathsf{A}}(i)\hspace{0.17em}=\hspace{0.17em}\{\begin{array}{ll}\min \{i<j\le n|\mathsf{A}[\hspace{0.17em}j]<\mathsf{A}[i]\}\hfill & \text{if some such }j\text{ exists,}\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}$$

These queries find application in various algorithmic problems (Goto and Bannai, 2013; Kempa and Puglisi, 2013; Kärkkäinen et al., 2013a), particularly in tasks involving array traversal, range queries, and optimization problems. They are often used in algorithms to find the closest smaller element to the left or right for each element in an array.

Next, following the definitions provided by Kärkkäinen et al. (2013a), we introduce $\mathsf{NS}{\mathsf{V}}_{\mathsf{lex}}\left[i\right]$ and $\mathsf{PS}{\mathsf{V}}_{\mathsf{lex}}\left[i\right]$ which operate directly on $\mathsf{SA}$:

$$\mathsf{NS}{\mathsf{V}}_{\mathsf{lex}}\left[i\right]\hspace{0.17em}=\hspace{0.17em}\min \{j\in [i+1‥n\left]\right|\mathsf{SA}[\hspace{0.17em}j]<\mathsf{SA}\left[i\right]\}$$

and

$$\mathsf{PS}{\mathsf{V}}_{\mathsf{lex}}\left[i\right]\hspace{0.17em}=\hspace{0.17em}\max \{j\in [1‥i-1\left]\right|\mathsf{SA}[\hspace{0.17em}j]<\mathsf{SA}\left[i\right]\}.$$

Lastly, we define $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ and $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ as follows:

$$\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]\hspace{0.17em}=\hspace{0.17em}\mathsf{SA}[\mathsf{NS}{\mathsf{V}}_{\mathsf{lex}}(\mathsf{ISA}\left[i\right])],$$

and

$$\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right]\hspace{0.17em}=\hspace{0.17em}\mathsf{SA}[\mathsf{PS}{\mathsf{V}}_{\mathsf{lex}}(\mathsf{ISA}\left[i\right])],$$

assuming $\mathsf{SA}\left[0\right]$ = 0 for all $1\le i\le n$. These last equations are used frequently to calculate the $\mathsf{LZ}$77 factorization. We note that these text-based versions are essential for efficiently computing the $\mathsf{LZ}$77 factorization, as they enable quick identification of the longest match by locating the nearest lexicographic neighbors in the suffix array. Using $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ queries, the search for repeated substrings is optimized, avoiding redundant comparisons.

2.10. The FM-index

The FM-index (Ferragina and Manzini, 2000) is an index based on the BWT and the SA. As we have seen, BWT is capable of counting the number of occurrences of a pattern by itself. In order to locate all such occurrences, SA is required. We note that, in order to achieve compression, the FM-index does not store all the SA values, instead, it samples some of the values while still being able to compute the values that are not sampled through the LF-mapping. As a result, the FM-index of a string S is capable of computing the number of occurrences of a pattern P in $\mathsf{O}(|P|)$ time, and can compute the positions of all occ occurrences in $\mathsf{O}(|P|+\mathit{occ}\hspace{0.17em}{\log }^{ϵ}|S|)$ for $0<ϵ<1$ by storing the BWT plus $\mathsf{o}(n)$ bits of auxiliary information, where n is the length of S. That is, to find a pattern P in the FM-index, we can use the LF mapping to find the range [sp, ep] of the BWT that corresponds to all the suffixes of S that are prefixed by P. Once we obtain the range on the BWT, we land, in the last LF step, on one of the sampled positions of the SA, or we have to retrieve it. To do so, we continue performing steps of the backward search until we find a value of the SA that is sampled. From that sample, we can compute the value of SA that we were looking for. The FM-index is a compressed full-text substring index that leverages the BWT to efficiently search for an input string S. The index uses the BWT to rearrange the original string, grouping similar contexts together for improved compressibility. To navigate and reconstruct the string, the FM-index employs rank and select structures, which efficiently count character occurrences up to any position in the BWT. Instead of storing the entire suffix array, the FM-index keeps sampled positions, aiding in quick location jumps within the string. One of its fundamental features, the backward search, allows the index to efficiently find substrings by analyzing from the end of the pattern to its beginning.

2.11. LZ77

Given a string $S[1‥n]$, the $\mathsf{LZ}$77 (Ziv and Lempel, 1977) factoring of S is a decomposition of S into z phrases $S=Z_1{Z}_2\dots Z_z$. There are two types of phrases, literal phrases that are phrases representing the first occurrence of a character in S and repeat phrases that correspond to substrings that occurred previously in the string. The $\mathsf{LZ}$77 phrases can be constructed inductively as follows. Assume that we already computed the first $j-1$ phrases of the factorization, and let i be such that $S[1‥i-1]=Z_1\dots Z_{j-1}$. The next phrase $Z_j$ is a literal phrase if S[i] never occurred in $S[1‥i-1]$. Otherwise, the next phrase $Z_j$ is the longest prefix of $S[i‥n]$ that occurs in S at position $p_j<i$, i.e., $S[i‥i+{\mathcal{l}}_j-1]=S[p_j‥p_j+{\mathcal{l}}_j-1]$ with ${\mathcal{l}}_j$ denoting the length of the repeated phrase. The $\mathsf{LZ}$77 factorization can be compactly represented as a list of phrases $(p_1,{\mathcal{l}}_1)\dots (p_z,{\mathcal{l}}_z)$ where if $Z_j=(p_j,{\mathcal{l}}_j)$ is a literal phrase, we set ${\mathcal{l}}_j=0$ and $p_j$ equal to the character S[i] never seen before in the string.

While the $\mathsf{LZ}$77 factorization divides the string into nonoverlapping phrases, the concept of the longest previous factor (LPF) provides information about repetitions at every position in the string. For a position $1\le i\le n$, the pair $(p_i,{\mathcal{l}}_i)$ such that $p_i<i$ and $S[p_i‥p_i+{\mathcal{l}}_i]=S[i‥i+{\mathcal{l}}_i]$, with ${\mathcal{l}}_i$ maximized. This provides a compact way to capture the longest repeated substring starting at position i and occurring earlier in the string.

3. COMPUTING THE $\mathsf{LZ}$77 COMPRESSION

In-memory construction of $\mathsf{LZ}$77 involves managing a sliding window of data that efficiently encodes repeated substrings by referencing previous occurrences directly within memory. Here, we provide a concise overview of these in-memory construction algorithms, presented in chronological order.

3.1. Early methods of computing $\mathsf{LZ}$77

Ohlebusch and Gog (2011) presented two algorithms in 2011 for computing $\mathsf{LZ}$77 factorization. The first algorithm begins by computing the $\mathsf{SA}$ and $\mathsf{LCP}$ in linear time. Next, two additional arrays are computed: the longest previous substring ( $\mathsf{LPS}$) and the previous occurrence position of the first appearance of each character ( $\mathsf{PrevOcc}$). The $\mathsf{LPS}$ array stores the length of the longest prefix of the substring starting at position k that has occurred earlier in the string S, i.e.,

$$\mathsf{LPS}\left[k\right]\hspace{0.17em}=\hspace{0.17em}\max \{\mathcal{l}:0\le \mathcal{l}\le n-k+1\text{\hspace{0.17em}and\hspace{0.17em}}S[k‥k+\mathcal{l}-1\left]\text{\hspace{0.17em}is\hspace{0.17em}a\hspace{0.17em}substring\hspace{0.17em}of\hspace{0.17em}}S\right[1‥k+\mathcal{l}-2\left]\right\}.$$

In other words, $\mathsf{LPS}\left[k\right]$ is the length of the longest prefix of $S_k=S[k‥n]$ that matches a substring appearing before position k. If no such prefix exists, then $\mathsf{LPS}\left[k\right]$ is equal to 0. The array $\mathsf{PrevOcc}$ stores the position j where $\mathsf{LPS}$ occurs for each position k. More specifically, $\mathsf{PrevOcc}\left[k\right]$ stores the position j such that $j<k$ and $\mathsf{LPS}\left[k\right]$ occur at that position j. If no such position exists, then $\mathsf{PrevOcc}\left[k\right]$ is undefined.

The algorithm for computing these arrays is conceptualized using a graph-based representation of $\mathsf{SA}$ and $\mathsf{LCP}$, which was originally introduced by Crochemore and Ilie (2008). The graph has $\left|S\right|+2$ nodes, where there exists a node for every $(i,\mathsf{SA}[i])$, where i is a position in SA, and nodes with labels $(0,\mathsf{SA}[0])$ and $(n+1,\mathsf{SA}[n+1])$. Two nodes $(i,\mathsf{SA}[i])$ and $(i+1,\mathsf{SA}[i+1])$ are connected by an edge marked with $\mathsf{LCP}[i+1]$. A peak in this graph occurs when $\mathsf{SA}[i-1]<\mathsf{SA}\left[i\right]$ and $\mathsf{SA}[i+1]<\mathsf{SA}\left[i\right]$. After the construction of this graph, the peaks are then identified and eliminated to derive $\mathsf{LPS}$ and $\mathsf{PrevOcc}$ values. Each peak is eliminated by removing the node $(i,\mathsf{SA}[i])$ and replacing it with a new edge labeled $\min (\mathsf{LCP}[i],\mathsf{LCP}[i+1])$ between $(i-1,\mathsf{SA}[i-1])$ and $(i+1,\mathsf{SA}[i+1])$. For each eliminated peak node i, LPS and PrevOcc values are determined as follows: $\mathsf{LPS}\left[\mathsf{SA}\right[i\left]\right]$ = $\max \left\{\mathsf{LCP}\right[i],\mathsf{LCP}[i+1\left]\right\}$. The value of $\mathsf{PrevOcc}\left[\mathsf{SA}\right[i\left]\right]$ is set to $\mathsf{SA}[i-1]$ if $\mathsf{LCP}\left[i\right]>\mathsf{LCP}[i+1]$. Otherwise, $\mathsf{PrevOcc}\left[\mathsf{SA}\right[i\left]\right]$ = $\mathsf{SA}[i+1]$ if $\mathsf{LCP}\left[i\right]\le \mathsf{LCP}[i+1]$. After construction of the $\mathsf{LPS}$ and $\mathsf{PrevOcc}$ arrays, the $\mathsf{LZ}$77 factorization is computed using the $\mathsf{LPS}$ array to find the length of the longest previous match at each position, while the $\mathsf{PrevOcc}$ array provides the starting position of this match. If no match is found, the current character is encoded directly.

This algorithm is further optimized by interleaving the computation $\mathsf{LCP}$ with the peak elimination process. Instead of pre-computing the entire $\mathsf{LCP}$ array, their enhanced version performs incremental $\mathsf{LCP}$ computation while simultaneously processing potential peaks. This optimization reduces memory usage while achieving better runtime performance, especially for highly repetitive strings.

The second algorithm improves the space efficiency of the first algorithm by performing a backward search on $\mathsf{BWT}$ of $\mathsf{rev}(S)$. The algorithm first constructs a $\mathsf{SA}$ for $\mathsf{rev}(S)$. From this $\mathsf{SA}$, it derives the $\mathsf{BWT}$ using the formula, $\mathsf{BWT}\left[i\right]=\mathsf{rev}(S[\mathsf{SA}\left[i\right]-1])$ and computes the Previous Greater Value ( $\mathsf{PGV}$) and Next Greater Value ( $\mathsf{NGV}$)—instead of $\mathsf{PSV}$ and $\mathsf{NSV}$—needed since the algorithm processes $\mathsf{rev}(S)$. The boundary cases are handled by setting $\mathsf{SA}\left[0\right]=\mathsf{SA}[n+1]=\infty$, where

$$\mathsf{PGV}=\max \{\hspace{0.17em}j:0\le j<i\text{\hspace{0.17em}and\hspace{0.17em}}\mathsf{SA}[j]>\mathsf{SA}[i\left]\right\},$$

and

$$\mathsf{NGV}=\min \{\hspace{0.17em}j:i<j\le n+1\text{\hspace{0.17em}and\hspace{0.17em}}\mathsf{SA}[j]>\mathsf{SA}[i\left]\right\}.$$

After constructing the $\mathsf{BWT}$ of the $\mathsf{rev}(S)$ and the $\mathsf{PGV}$ and $\mathsf{NGV}$ arrays, the $\mathsf{LZ}$77 factorization is computed as follows. For each position k in S, the algorithm identifies an interval $[{\mathcal{l}}_b,r_b]$ that contains suffixes that begin with the substring $S[k‥n]$ using a backward search in $\mathsf{BWT}$ of $\mathsf{rev}(S)$. Then it checks if $\mathsf{PGV}$ or $\mathsf{NGV}$ falls within this interval. If neither $\mathsf{PGV}\left[k\right]$ nor $\mathsf{NGV}\left[k\right]$ falls within this interval, then it follows that S[k] has no previous occurrence, and the factor $(S[k],0)$ is outputted. However, if $\mathsf{PGV}\left[k\right]$ or $\mathsf{NGV}\left[k\right]$ falls within this interval, then it continues the process by extending the pattern to $S[k‥k+1]$ through backward search to get a new interval, and again checks if $\mathsf{PGV}[k+1]$ or $\mathsf{NGV}[k+1]$ falls within this new interval. This process continues until the maximum length substring, denoted as $\mathcal{l}$, is found such that the interval contains either $\mathsf{PGV}$ or $\mathsf{NGV}$. The starting position p of the factor is then determined using the position corresponding to $\mathsf{PGV}$ or $\mathsf{NGV}$. The algorithm outputs each $\mathsf{LZ}$77 factor as a pair $(p,\mathcal{l})$.

3.2. Lazy $\mathsf{LZ}$77 construction

Kempa and Puglisi (2013) and Goto and Bannai (2013) independently developed algorithms to calculate $\mathsf{LZ}$77 based on the knowledge that, given a position i in S, the new $\mathsf{LZ}$77 factor can be found by comparing suffixes at positions $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$. In particular, the length of the $\mathsf{LZ}$77 factor at position i can be determined by computing $\max \{\mathsf{lcp}(i,\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]),\mathrm{lcp}(i,\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right])\}$. Notably, both Kempa and Puglisi (2013) and Goto and Bannai (2013) only compute the $\mathsf{lcp}(i,\mathsf{NS}{\mathsf{V}}_{\mathit{text}}[i])$ and $\mathsf{lcp}(i,\mathsf{PS}{\mathsf{V}}_{\mathit{text}}[i])$ when i is the starting position of a phrase, and thus, are often referred to as “lazy evaluation” of $\mathsf{LCP}$ values. The methods differ in the way $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ are computed. We now discuss each of these algorithms in detail.

Goto and Bannai define three linear-time algorithms that differ on the basis of their memory use. First, they define the BGT algorithm, which operates directly in text order, and use the array $\Phi$ to maintain a mapping between each suffix and its preceding lexicographic suffix. Given the input string S of length n, this algorithm begins by computing $\mathsf{SA}$ and $\mathsf{ISA}$. With these structures in place, the algorithm then calculates $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ using the $\Phi$ array, defined as $\Phi \left[\mathsf{SA}\right[i\left]\right]=\mathsf{SA}[i-1]$ for i greater than 0 and $\Phi \left[\mathsf{SA}\right[1\left]\right]\hspace{0.17em}=\hspace{0.17em}n+1$. Using this array, $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ is assigned the value of $\Phi \left[i\right]$ if it exists; otherwise, it is set to 0. Meanwhile, $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ is computed by scanning the $\Phi$ array starting from position i. Once the values for $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ are calculated, the algorithm proceeds to determine the LPF for each position i in S using the expression $\mathsf{LPF}\left[i\right]=\max \{\mathrm{lcp}(i,\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]),\mathrm{lcp}(i,\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right])\}$. Finally, the $\mathsf{LZ}$77 factors are constructed using the $\mathsf{LPF}$ values. If $\mathsf{LPF}\left[i\right]$ is equal to 0, then the factor represents a new character at position i. Otherwise, the $\mathsf{LZ}$77 factor consists of a reference to an earlier occurrence, with the position of the matching substring determined based on the computation of $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$. The algorithm then advances by $\mathsf{LPF}\left[i\right]$ characters. The BGT algorithm requires $3n\hspace{0.17em}\log \hspace{0.17em}n$ bits of space, making it the most memory-efficient algorithm among the three algorithms.

Next, they introduced the BGS algorithm that computes $\mathsf{PS}{\mathsf{V}}_{\mathit{lex}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{lex}}$ using a stack. Similarly to BGT, this algorithm begins by computing the $\mathsf{SA}$ and $\mathsf{ISA}$. Next, it traverses the $\mathsf{SA}$, computing the $\mathsf{PS}{\mathsf{V}}_{\mathit{lex}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{lex}}$ using a stack. Subsequently, these values are converted to text order using $\mathsf{ISA}$, resulting in $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$. In particular, the transformations are defined by the equations: $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]=\mathsf{SA}[\mathsf{NS}{\mathsf{V}}_{\mathsf{lex}}(\mathsf{ISA}\left[i\right])]$, and $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right]=\mathsf{SA}[\mathsf{PS}{\mathsf{V}}_{\mathsf{lex}}(\mathsf{ISA}\left[i\right])]$, Lastly, the $\mathsf{LZ}$77 is computed in the same way as in the BGT algorithm. This algorithm is faster than the BGT algorithm but uses additional space. In particular, it requires $(4n+s)\log \hspace{0.17em}n$ bits of space, where s denotes the maximum size of the stack.

The third algorithm they introduced is BGL, which computes $\mathsf{PS}{\mathsf{V}}_{\mathit{lex}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{lex}}$ without a stack, instead employing the peak elimination algorithm previously defined by Ohlebusch and Gog (2011). After this, the algorithm maintained the same computations as the BGS algorithm. The workspace required by BGL is $4n\hspace{0.17em}\log \hspace{0.17em}n$.

The algorithm of Kempa and Puglisi begins by computing the $\mathsf{SA}$ and the $\mathsf{ISA}$ for the input string S. The algorithm begins by constructing the predecessor ( $\mathsf{PS}{\mathsf{V}}_{\mathit{lex}}$) and successor ( $\mathsf{NS}{\mathsf{V}}_{\mathit{lex}}$) arrays using a stack and scanning the $\mathsf{SA}$ while maintaining a stack of positions. For each position i, the algorithm compares $\mathsf{SA}\left[i\right]$ with $\mathsf{SA}\left[j\right]$, where j is the position that is at the top of the stack. If the stack is not empty and $\mathsf{SA}\left[j\right]$ is greater than $\mathsf{SA}\left[i\right]$, then j is popped from the stack. This process is repeated until $\mathsf{SA}\left[j\right]\le \mathsf{SA}\left[i\right]$ or the stack is empty. If the stack is not empty, then $\mathsf{PS}{\mathsf{V}}_{\mathit{lex}}\left[i\right]$ is equal to j. Otherwise, if the stack is empty, $\mathsf{PS}{\mathsf{V}}_{\mathit{lex}}\left[i\right]$ is equal to −1, indicating that there are no previous smaller suffixes. The index i is then pushed onto the stack, and the next position is considered. The array $\mathsf{NS}{\mathsf{V}}_{\mathit{lex}}$ is computed in an identical manner, but the traversal of $\mathsf{SA}$ is in reverse order. Next, the algorithm uses the arrays $\mathsf{ISA}$, $\mathsf{SA}$, $\mathsf{PS}{\mathsf{V}}_{\mathit{lex}}$, and $\mathsf{NS}{\mathsf{V}}_{\mathit{lex}}$ to simulate $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ and calculate the factorization $\mathsf{LZ}$77, with the property that for each position i, $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ equals $\mathsf{SA}[\mathsf{NS}{\mathsf{V}}_{\mathit{lex}}(\mathsf{ISA}\left[i\right])]$ and similarly, $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ equals $\mathsf{SA}[\mathsf{PS}{\mathsf{V}}_{\mathit{lex}}(\mathsf{ISA}\left[i\right])]$. The next $\mathsf{LZ}$77 factor is determined by identifying the longest previous match for the current position i, which is computed using $\mathrm{lcp}(i,\mathsf{NS}{\mathsf{V}}_{\mathit{text}}[i])$ and $\mathrm{lcp}(i,\mathsf{PS}{\mathsf{V}}_{\mathit{text}}[i])$.

3.3. The KKP algorithms

Kärkkäinen et al. (2013a) introduced a suite of algorithms collectively called the KKP algorithms, named after the authors. Among these, the KKP3 algorithm requires $3n\hspace{0.17em}\log \hspace{0.17em}n$ bits of working space and shares close ties with the algorithms of Goto and Bannai (2013). KKP3 is a simple algorithm that the authors describe in 13 lines of pseudocode. It begins by first computing $\mathsf{SA}$. After the $\mathsf{SA}$ is constructed, the $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ arrays are calculated using the approach of the BGS algorithm, i.e., using a stack and $\mathsf{SA}$. After computing $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$, the $\mathsf{LZ}$77 factors are calculated using lazy factorization, which is similar to the BGT algorithm described in Section 3.2. KKP3 does not construct $\mathsf{ISA}$ or $\Phi$; (2) KKP3 stores the arrays $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ in an interweaved manner so that $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ and $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ are next to each other, making access more cache efficient; and lastly, (3) is more memory efficient in the construction of $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$, that is, SA is overwritten by the elements of the stack, so only 2 arrays of size $\mathsf{O}(n)$ are ever stored in main memory.

KKP2 reduces memory consumption compared to KKP3. Instead of precomputing both $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ before computing the $\mathsf{LZ}$77 factorization, KKP2 only precomputes the $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ array. More specifically, given the string S, KKP2 starts by constructing $\mathsf{SA}$, then computes $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ using the same approach as KKP3, and stores the values of $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ in the array $\Phi$, that is, $\Phi \left[i\right]=\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$, for each position i. Next, for each position t, KKP2 computes $\mathsf{PS}{\mathsf{V}}_{\text{text}}\left[t\right]$ using $\Phi$ and then determines the next $\mathsf{LZ}$77 factor in the same way as KKP3 by comparing $\mathsf{PS}{\mathsf{V}}_{\text{text}}\left[t\right]$, $\mathsf{NS}{\mathsf{V}}_{\text{text}}\left[t\right]$, and t. This design ensures that both the values $\mathsf{NS}{\mathsf{V}}_{\text{text}}$ and $\mathsf{PS}{\mathsf{V}}_{\text{text}}$ can be retrieved directly from the $\Phi$ array, eliminating the need for an additional $\mathsf{PS}{\mathsf{V}}_{\text{text}}$ array. By avoiding the precomputation and storage of $\mathsf{PS}{\mathsf{V}}_{\text{text}}$, KKP2 reduces the working space to $\mathsf{O}(2n\hspace{0.17em}\log \hspace{0.17em}n)$ bits while maintaining linear time complexity.

KKP1 will be discussed in a later section, where external memory methods are discussed. The names of the methods, KKP3, KKP2, and KKP1, originate from the number of arrays (of length n) that are stored in memory. KKP3 stores SA, PSV, and NSV. KKP2 stores SA and $\Phi$. KKP1 assumes that SA is stored in external memory and only stores NSV in main memory.

3.4. The BGone algorithm

Motivated by the fact that the KKP algorithms store two arrays, Goto and Bannai (2014) developed algorithms to compute the $\mathsf{LZ}$77 factorization using a single array. The name of the algorithm refers to the authors’ names and the number of auxiliary arrays used. They refer to their previous work that we described in Subsection 3.2 as BGtwo.

In order to describe BGone, we require some additional definitions. Given an input string S, we define the suffix $S[i‥n]$ to be either an S-type or an L-type based on lexicographical comparisons. A suffix $S[i‥n]$ is S-type if $S\left[i\right]<S[i+1]$, or if $S\left[i\right]=S[i+1]$ and $S[i+1‥n]$ are S-type suffixes. Furthermore, $S[i‥n]$ is a L-type suffix if $S\left[i\right]>S[i+1]$, or if $S\left[i\right]=S[i+1]$ and $S[i+1‥n]$ is a L-type suffix. Additionally, a suffix is defined as a Left-Most S-type (LMS) if it is a S-type and the suffix immediately preceding it is an L-type suffix.

Given a string S, BGone constructs the array $\Phi$ using an algorithm that is similar to Nong (2013)’s induced sorting algorithm, which sorts a selected subset of suffixes (in this case, the LMS suffixes) and then uses their ordering to sort the remaining suffixes. Hence, the algorithm begins by scanning S from right to left and classifying each suffix as either L-type or S-type, initializing an auxiliary array A, and sorting the identified LMS suffixes lexicographically. This produces a lexicographically sorted list of LMS suffixes, which we denote as $LM{S}_{\mathsf{SA}}$. These suffixes are then placed in alternating positions of the array A: specifically, for each i from 1 to k, we set $A\left[2i\right]=LM{S}_{\mathsf{SA}}\left[i\right]$, while the positions $A[2i-1]$ are left empty. Then, for each LMS suffix position $j_1=A\left[2i\right]$ and its lexicographic successor $j_2=A[2(i+1)]$, the algorithm attempts to assign $A\left[j_1\right]=j_2$. If a value has already been assigned to $A\left[j_1\right]$—that is, if $j_1$ appears in the rearranged $LM{S}_{\mathsf{SA}}$—the algorithm instead assigns $A[j_1-1]=j_2$. Once all assignments are complete, BGone performs two passes of induced sorting to incorporate all remaining suffixes into A: in the first pass, it incorporates all L-type suffixes in A, and in the second pass, it incorporates all S-type suffixes in A. After both passes are complete, the working array A is transformed into the $\Phi$ array. Finally, BGone computes the $\mathsf{LZ}$77 factorization using the $\Phi$ array, following the same method as the KKP2 algorithm described in Subsection 3.3. By combining these in-place algorithms, Goto and Bannai present the first $\mathsf{LZ}$77 factorization method that uses only a single auxiliary array of length n. The BGone algorithm runs in linear time and requires $n\hspace{0.17em}\log \hspace{0.17em}n+\mathsf{O}(|\Sigma |\log \hspace{0.17em}n)$ bits of space.

3.5. The LZoneT and LZoneSA algorithms

Building on the development of BGone, Liu et al. (2016) introduced the LZone algorithm, which maintains the same theoretical runtime and memory usage as BGone, but offers practical improvements in speed. Like BGone, it uses a single auxiliary array, but achieves better performance by more precisely handling L-type and S-type suffixes. This is achieved through a reorganization of the suffix sorting process to avoid unnecessary scans and random access operations. LZoneT begins by counting the number of L-type and S-type suffixes, then adapts its strategy based on which suffix type is the most prevalent. If the number of L-type suffixes is less than the number of S-type suffixes, then $\mathsf{S}{\mathsf{A}}_L$ is computed; otherwise, $\mathsf{S}{\mathsf{A}}_S$ is computed. Next, LZoneT computes ${\Phi }_L$ or ${\Psi }_S$. If L-type suffixes are less than the number of S-type suffixes, then $\mathsf{S}{\mathsf{A}}_L$ is used to construct ${\Phi }_L$. In ${\Phi }_L$, all L-type suffixes are connected in descending lexicographic order. For each L-type suffix at position i, ${\Phi }_L\left[i\right]$ stores the position of the L-type suffix that immediately precedes suffix i in lexicographic order. To complete the structure as a circular linked list, we handle the lexicographically largest L-type suffix at position $j=\mathsf{S}{\mathsf{A}}_L\left[\right|\mathsf{S}{\mathsf{A}}_L\left|\right]$—by setting ${\Phi }_L\left[j\right]=0$, indicating that it has no lexicographic predecessor. Furthermore, ${\Phi }_L\left[0\right]$ is set to $\mathsf{S}{\mathsf{A}}_L\left[1\right]$, the position of the lexicographically smallest L-type suffix. This structure enables efficient sequential access to all L-type suffixes in descending lexicographical order using a single scan. Similarly, if S-type suffixes are less prevalent, ${\Psi }_S$ is constructed by linking the S-type suffixes from the smallest to the largest, with ${\Psi }_S\left[i\right]$ storing the position of the S-type suffix that immediately follows i lexicographically.

Next, $\Psi$ or $\Phi$ is computed. If the number of L-type suffixes is less than the number of S-type suffixes, then $\Psi$ is computed from ${\Psi }_S$ using induced sorting to insert L-type suffixes into the linked structure ${\Psi }_S$. Otherwise, $\Phi$ is computed from ${\Phi }_L$. LZone then derives $\mathsf{NSV}$ or $\mathsf{PSV}$ from $\Phi$ or $\Psi$ using the same method as KKP2 (Section 3.3). Finally, LZone computes the $\mathsf{LZ}$77 factors using the same procedure as BGone (Subsection 3.4).

The LZoneSA algorithm is a variant of LZoneT designed to efficiently compute the $\mathsf{LZ}$77 factorization using a pre-existing $\mathsf{SA}$ of the input string S. LZoneSA extracts the $\mathsf{S}{\mathsf{A}}_L$ or $\mathsf{S}{\mathsf{A}}_S$ directly from the available SA.

3.6. $\mathsf{LZ}$77 construction with the RLBWT and sparse sampling

Policriti and Prezza (2018) presented an algorithm to compute the $\mathsf{LZ}$77 factorization by using the $\mathsf{RLBWT}$ of the $\mathsf{rev}(S)$. The algorithm begins by constructing $\mathsf{RLBWT}$ for $\mathsf{rev}(S)$ through an online process that reads the input string S from left to right, where online means that each character is processed and inserted into $\mathsf{RLBWT}$ immediately upon reading, without needing the entire sequence in memory. This online construction specifically employs a dynamic run-length encoded string data structure that supports rank, access, and insert operations. During $\mathsf{RLBWT}$ construction, for each position i in S, the algorithm attempts to find the longest prefix of the substring $S[i-\mathit{len}‥i-1]$ that has already occurred earlier in S before position i, where len denotes the length of this current longest prefix. This search is accomplished by maintaining an interval $[\mathcal{l},r]$ within $\mathsf{RLBWT}$ that corresponds to the potential positions of the current reversed prefix in $\mathsf{RLBWT}$. As each new character is processed c, the algorithm performs a backward search to locate the matching positions. More specifically, the algorithm considers whether the character c can extend the current prefix without creating a new $\mathsf{LZ}$77 phrase boundary. To determine this, the algorithm counts the number of runs in $\mathsf{BWT}$ intersecting the interval $[\mathcal{l},r]$ in the $\mathsf{RLBWT}$. For example, in a $\mathsf{BWT}$ string aaaaabbbaadd, there are four runs: aaaaa, bbb, aa, and dd. If there is only one such run, this indicates that the current prefix can be extended without forming a new $\mathsf{LZ}$77 phrase, as all occurrences of the prefix are followed by c. If more than one run intersects the interval, the prefix is considered right-maximal, indicating that it might end at this position and create a new $\mathsf{LZ}$77 phrase. When a right maximal prefix is identified, the algorithm checks whether this prefix actually matches a substring occurring earlier in S. For this, it used sampled $\mathsf{SA}$ stored in a red-black tree. These samples are recorded for each run, and the positions where characters appear in each run are stored. If an earlier occurrence of the prefix is found within the interval, it indicates that the prefix can be extended, updated to its length, and the interval $[\mathcal{l},r]$ adjusted to represent the new extended interval in $\mathsf{RLBWT}$. Otherwise, if no match is found, this prefix forms a new $\mathsf{LZ}$77 phrase, and its position and length are recorded.

The authors further introduced an alternative approach to optimize $\mathsf{SA}$ sample storage by concentrating the samples at $\mathsf{LZ}$77 factor boundaries, leveraging the observation that the number of $\mathsf{LZ}$77 factors z is typically much smaller than the number of $\mathsf{RLBWT}$ runs. In this approach, the algorithm marks positions in the $\mathsf{BWT}$‘s F column (first column of the sorted $\mathsf{BWT}$ matrix) with $\mathsf{LZ}$77 factor numbers, and later uses these marks to identify the source position of $\mathsf{LZ}$77 factors. This improved algorithm follows three main steps.

First, the algorithm constructs $\mathsf{RLBWT}$ of $\mathsf{rev}(S)$ while identifying $\mathsf{LZ}$77 phrase boundaries by backward search. When processing the i-th character c in S, the algorithm performs a backward search in RLBWT, maintaining an interval $[\mathcal{l},r]$ (initially set to $[0,n-1]$, n is the length of S) to determine if the current substring appended with c has appeared before in the string up to position $i-1$. If the resulting interval $[{\mathcal{l}}^{\prime },r^{\prime }]$ is empty, then the current substring with c has not appeared before in the string at position i, marking this position as a $\mathsf{LZ}$77 phrase boundary. At each such boundary, the algorithm stored a sample mapping the corresponding $\mathsf{BWT}$ position to position i in S. The key difference from the first algorithm is that the samples are only stored at these boundary positions rather than at each $\mathsf{BWT}$ run.

Next, the algorithm identifies the source positions of each $\mathsf{LZ}$77 factor. It scans S, and for each marked boundary position, it determines the starting position of the $\mathsf{LZ}$77 phrase in the original string and stores these positions in an array. Finally, this algorithm rebuilds the $\mathsf{RLBWT}$ and outputs the $\mathsf{LZ}$77 factors using this $\mathsf{RLBWT}$. More specifically, for each position i in the string S, the algorithm reads the current character c and performs the backward search in $\mathsf{RLBWT}$ using the current interval $[\mathcal{l},r]$ and the character c. This will return a new interval $[{\mathcal{l}}^{\prime },r^{\prime }]$. If this interval $[{\mathcal{l}}^{\prime },r^{\prime }]$ is empty, then the substring formed by the current $\mathsf{LZ}$77 factor cannot be extended further in S. Therefore, the algorithm uses the corresponding value in the array of starting positions to output the current $\mathsf{LZ}$77 factor, resets the interval $[\mathcal{l},r]$ to [0, i], and inserts the character c into $\mathsf{RLBWT}$. If the interval $[{\mathcal{l}}^{\prime },r^{\prime }]$ is not empty, then the algorithm extends the current $\mathsf{LZ}$77 factor and adjusts $[\mathcal{l},r]$ accordingly. Repeatedly, until all character’s c in S are calculated.

3.7. PFP- $\mathsf{LZ}$77

Given an input string S, Fischer et al. (2009) demonstrated a data structure that supports access to $\mathsf{S}{\mathsf{A}}_S$, $\mathsf{IS}{\mathsf{A}}_S$, $\mathsf{BW}{\mathsf{T}}_S$, and $\mathsf{LC}{\mathsf{P}}_S$, and some additional queries on $\mathsf{LC}{\mathsf{P}}_S$ can emulate a compressed suffix tree (CST). Given that insight, Boucher et al. (2021) described how the output of prefix-free parsing (PFP) can be used as the main components of a data structure that supports CST queries, and thus simulate a CST. Building upon this foundation, Hong (2023) demonstrated how PFP can be applied to the LZ77 factorization. Here, we will briefly describe PFP, the components of the PFP data structure, and the queries that are supported. Then we show how it can be applied to compute the LZ77 factorization. The discussion of the data structure is not intended to be exhaustive.

PFP takes as input a string $S[1‥n]$ and positive integers w and p, and produces a parse of S (denoted as $\mathsf{P}$) and a dictionary (denoted as $\mathsf{D}$). The dictionary consists of unique substrings of S (referred to as phrases), and the parse is an encoding of S as a sequence of these dictionary phrases. Instead of storing the phrases themselves, the parse is represented as a sequence of integers corresponding to the ranks of the dictionary phrases. Next, we briefly go over the algorithm for producing this dictionary and parse. First, we let T be an arbitrary set of w-length strings over $\Sigma$, and $T^{\prime }=T\cup \left\{{\$}^w\right\}$ where $is a special symbol lexicographically less than any symbol in $\Sigma$. We call $T^{\prime }$ the set of trigger strings. We append w copies of $ to S, i.e., $S^{\prime }=S{\$}^w$, and consider it cyclic.

We let the dictionary $\mathsf{D}=d_1,‥,d_{\Vert \Vert \mathsf{D}\Vert \Vert }$ be equal to the set of all substrings of $S^{\prime }$ such that the following holds for each $d_i$: (1) exactly one proper prefix of $d_i$ is contained $T^{\prime }$, (2) exactly one proper suffix of $d_i$ is contained in $T^{\prime }$, (3) and no other substring of $d_i$ is in $T^{\prime }$. Therefore, since we assume ${\$}^w\in T^{\prime }$ we can construct $\mathsf{D}$ by scanning ${\$}^{w}S{\$}^w$ from right to left to find all occurrences of $T^{\prime }$ and adding to $\mathsf{D}$ each substring of ${\$}^{w}S{\$}^w$ that starts and ends at a trigger string inclusive of the starting and ending trigger string. We can also construct the list of occurrences of $\mathsf{D}$ in S, which defines the parse $\mathsf{P}$. Hence, to do this, we let f be the Karp-Rabin (1987) fingerprint of strings of length w and slide a window of length w over ${\$}^{w}S{\$}^w$. We define Q as the sequence of substrings $r=S^{\prime }[s,s+w-1]$ such that $f(r)\equiv 0\mathrm{mod}p$ or $r={\$}^w$ ordered by their position in S, i.e., $Q=(S^{\prime }[s_1,s_1+w-1],‥,S^{\prime }[s_k,s_k+w-1])$ where $s_1<s_2<‥<s_k$. The dictionary is the set of unique substrings s of $S^{\prime }$ such that $\left|s\right|>w$, and the first and last w characters of s correspond to consecutive elements in Q. Lastly, the dictionary is sorted in lexicographical order, and the parse is defined as the ranks of the substrings of $\mathsf{D}$ in sorted order.

Lastly, we illustrate PFP using an example. Suppose that we have the following string $S^{\prime }=\mathrm{AGACGACT}\$\mathrm{AGATACT}\$\mathrm{AGATTCGAG}\$\$$, $w=2$, and a set of trigger strings $T^{\prime }=\{\mathrm{AC},\mathrm{TC},\$\$,\mathrm{T}\$\}$ obtained from taking those that have a fingerprint $0\mathrm{mod}p$. It follows that Q = {$$AGAC, ACGACT$, $AGATAC, ACT$, T$AGATTC, TCGAG$$}. It follows then that the dictionary is equal to {$$AGAC, $AGATAC, ACGACT$, ACT$, T$AGATTC, TCGAG$$} and parse is equal to [1, 3, 2, 4, 5, 6]. As we can see, the parse is the string S parsed by phrases.

Boucher et al. construct the following data structures from the parse $\mathsf{P}$ from the PFP of the input string S: (1) the suffix array of $\mathsf{P}$ (denoted as $\mathsf{S}{\mathsf{A}}_{\mathsf{P}}$), (2) the inverse suffix array of $\mathsf{P}$ (denoted as $\mathsf{IS}{\mathsf{A}}_{\mathsf{P}}$), (3) a cyclic bitvector (denoted as ${\mathsf{B}}_{\mathsf{P}}$) that marks the start position of each trigger string in the string S, and (4) a bitvector (denoted as ${\mathsf{B}}_{\mathsf{BWT}}$) that, for each unique proper phrase suffix of length at least w, has a 1 for the first position in the $\mathsf{S}{\mathsf{A}}_S$ that stores that phrase. Lastly, Boucher et al. construct LCP of $\mathsf{P}$ but in terms of the phrases of $\mathsf{P}$ (not the ranks of $\mathsf{D}$), which is denoted as $\mathsf{LC}{\mathsf{P}}_{\mathsf{P}}$.

Next, Boucher et al. construct the following data structures from the dictionary $\mathsf{D}$: (1) the longest common prefix array of $\mathsf{D}$ (denoted as $\mathsf{LC}{\mathsf{P}}_{\mathsf{D}}$); and (2) a list of size $\mathsf{D}$ that, for each position in $\mathsf{D}$, which starts with a specific proper suffix $\alpha$, records the rank of $\alpha$ among the distinct proper suffixes of length a minimum w (denoted as $\mathsf{IS}{\mathsf{A}}_{\mathsf{D}}$). We define some of these a bit more clearly since they are non-standard. If $\mathsf{P}$ is considered as a string, then we can consider all rotations of $\mathsf{P}$, sort them lexicographically, and lastly, consider the starting location of each of the rotations in sorted order. This is the suffix array of $\mathsf{P}$. Next, if we can consider the concatenation of the phrases of $\mathsf{D}$, consider all rotations of this in lexicographical order, then we can compute the $\mathsf{LCP}$ on the sorted rotations. This is the LCP of $\mathsf{D}$.

An additional structure built from $\mathsf{D}$ and $\mathsf{P}$ by Boucher et al. is a two-dimensional discrete grid that we denote as $\mathsf{W}$. $\mathsf{W}$ is used to support queries $\mathsf{S}{\mathsf{A}}_S$ and $\mathsf{IS}{\mathsf{A}}_S$. The x coordinates of $\mathsf{W}$ correspond to the entries of $\mathsf{BW}{\mathsf{T}}_{\mathsf{P}}$, and the y coordinates of $\mathsf{W}$ correspond to the phrases of $\mathsf{D}$ in co-lexicographic order, which is lexicographic order from right to left. Given a phase p of the parse $\mathsf{P}$, let $\mathsf{colex}(p)$ be the co-lexicographic rank of p in D. The set points on the grid are defined as $\mathsf{W}=\{(\mathsf{BW}{\mathsf{T}}_{\mathsf{P}}\left[i\right],\mathsf{colex}(\mathsf{BW}{\mathsf{T}}_{\mathsf{P}}[i]))\mid 1\le i\le |\mathsf{P}\left|\right\}$, where $\mathsf{BW}{\mathsf{T}}_{\mathsf{P}}$ is the BWT of the parse. We pause here for a second to define the BWT of the parse more clearly; we remind the reader that the parse is a list of integers corresponding to the rank of the dictionary phrases. The BWT is computed on this list of interest. Continuing our previous example, we have $\mathsf{BW}{\mathsf{T}}_{\mathsf{P}}$ = [5,0,1,3,2,2,4] and, therefore, 5 corresponds to the phrase TCGAG # #. Given a position p and rank r, $\mathsf{W}$ can return how many phrases of the first p phrases in $\mathsf{BW}{\mathsf{T}}_{\mathsf{P}}$ have a co-lexicographic rank at least r.

We define ${\mathsf{D}}_w$ as the set of all proper phrase suffixes of $\mathsf{D}$ of length at least w, and we denote $\mathsf{rev}({\mathsf{D}}_w)$ to be the set of phrases of ${\mathsf{D}}_w$ reversed and stored in co-lexicographic order. Given these definitions, Boucher et al. define a table $\mathsf{M}$ where $\mathsf{M}\left[r\right]$ stores (1) the length $\mathcal{l}$ of the lexicographically r-th proper suffix $\alpha$ of length at least w; and (2) the range in ${\mathsf{D}}_w$ of phrases starting with $\alpha$ reversed, starting with 0. Using the dictionary $\mathsf{D}$, the parse $\mathsf{P}$, and the auxiliary data structures discussed above, Boucher et al. demonstrate that a number of different queries can be supported, including $\mathsf{SA}$, $\mathsf{ISA}$, $\mathsf{LCP}$, $\mathsf{PSV}$, and $\mathsf{NSV}$, among others. Using these queries, Hong et al. showed how to compute the $\mathsf{LZ}$77 factorization as follows.

PFP-LZ77 considers positions $1\le i\le n$ that are the start of an $\mathsf{LZ}$77 factor. For each such position, the algorithm uses ${\mathsf{B}}_{\mathsf{P}}$ to find the phrase q of $\mathsf{P}$ that contains position i in S. Next, ${\mathsf{B}}_{\mathsf{D}}$ is used to find its offset within the phrase and use this offset to identify the proper phrase suffix $\alpha$ to the phrase q in $\mathsf{D}$.

Given a position d of $\alpha$ in $\mathsf{D}$, the algorithm calculates $r=\mathsf{IS}{\mathsf{A}}_{\mathsf{D}}\left[d\right]$, which is equal to its lexicographic rank r among phrases. Next, using $\mathsf{M}$ the algorithm finds the range [y1, y2] of phrases ending with $\alpha$—this corresponds to the boundary in $\mathsf{T}$ along the y-coordinate. Then $\mathsf{IS}{\mathsf{A}}_{\mathsf{P}}$ is used to find the phrase q of $\mathsf{P}$ in the $\mathsf{BW}{\mathsf{T}}_{\mathsf{P}}$ order. This is denoted by x. This defines the ranges of interest for which the rightmost and leftmost point are considered, which are $[1,x-1]\times [y1,y2]\times [1,q]$ and $[x+1,|\mathsf{P}\left|\right]\times [y1,y2]\times [1,q]$, respectively. Using these ranges and the definitions described previously, the algorithm computes $\mathsf{LCE}(i,\mathsf{PS}{\mathsf{V}}_{\mathit{text}}[i])$ and $\mathsf{LCE}(i,\mathsf{PS}{\mathsf{V}}_{\mathit{text}}[i])$. If $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ exists, then the phrase and the phrase length are defined using $\mathsf{LC}{\mathsf{P}}_{\mathsf{P}}$, ${\mathsf{B}}_{\mathsf{P}}$ and $\mathsf{S}{\mathsf{A}}_{\mathsf{P}}$. Otherwise, $\mathsf{PS}{\mathsf{V}}_{\mathsf{D}}$, $S_{\mathsf{D}}$ and $\mathsf{LC}{\mathsf{P}}_d$ are used to find the phrase and the phrase length of the LPF within the region $\mathsf{PS}{\mathsf{V}}_{\mathsf{D}}\left[r\right]+1,r$. $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ is computed analogously. Lastly, the LPF for position i is computed as $\mathsf{LCE}(\mathsf{PS}{\mathsf{V}}_{\mathit{text}}[i],\mathsf{NS}{\mathsf{V}}_{\mathit{text}}[i])$.

3.8. Algorithms with runtime dependent on the number of $\mathsf{LZ}$77 factors

Ellert (2023) presents an algorithm in which the running time is dependent on the number of $\mathsf{LZ}$77 factors z, achieving a complexity of $\mathsf{O}\left(n/{\log }_{\sigma }\hspace{0.17em}n+z\hspace{0.17em}{\log }^{3+ϵ}\hspace{0.17em}z\right)$, which simplifies to $\mathsf{O}\left(n/{\log }_{\sigma }\hspace{0.17em}n\right)$ when z is sufficiently small, yet still exhibits a worst-case running time of $\mathsf{O}(n/{\log }_{\sigma }\hspace{0.17em}n)$.

Given a parameter $\delta$ in the range [1,| S |], the algorithm distinguishes between phrases that have a length shorter than $\delta$, which are referred to as short, and phrases that have a length of at least $\delta$, referred to as long. To compute the factorization $\mathsf{LZ}$77, the algorithm processes the input string S from left to right, and at each position i, it seeks the leftmost source, the earliest position $j<i$ where the substring starting at j matches the substring starting at i with the maximum possible length.

If $i<\delta$, the algorithm computes $\mathsf{LCE}(j,i)$ for each $j\in [1,i)$, with the maximum $\mathsf{LCE}$ value indicating the length of the current $\mathsf{LZ}$77 factor and the corresponding index j marking its starting position. If all $\mathsf{LCE}$ values are 0, then the factor at position i is a literal.

If $i\ge \delta$, the algorithm considers three different methods to find the longest factor. The first method, called close sources, applies when the leftmost is $j\in [i-\delta ,i)$. The close sources method computes $\mathsf{LCE}(j^{\prime },i)$ for each $j^{\prime }\in [i-\delta ,i)$ and selects the longest matching substring within this nearby window. The second method, called long phrases with far sources, applies when $j\in [1,i-\delta )$ and the current $\mathsf{LZ}$77 factor length is at least $\delta$. It uses a set of evenly spaced sample positions A[1. | S |], where each $A\left[h\right]=h\delta$ and $h=\lceil j/\delta \rceil$. This approach exploits the property that any valid source position j must have an interval $[j,j+\delta )$ that contains at least one sample point A[h]. As a result, instead of examining all possible positions j, the algorithm focuses only on the positions near these sample points. For each A[h] and offset $k\in [0,\delta )$, it computes $\mathsf{LCE}(A[h]-k,i)$ and uses this array to determine the longest matching substring. The third method, called short phrases with far sources, addresses cases where $j\in [1,i-\delta )$ and the current $\mathsf{LZ}$77 factor length is less than $\delta$. It uses a subset Y of sample positions drawn from an array B, which is sorted in increasing order and constructed to ensure that any interval of sufficient length contains at least one sample. Specifically, for each sample position B[h], the condition $B\left[h\right]\le j+\text{factor\hspace{0.17em}length}\le j+\delta <i$ ensures that the samples adequately cover the relevant portion of the string. For each $B\left[h\right]\in Y$, the algorithm performs queries to compute the corresponding $\mathsf{LCE}$ values, allowing it to determine the longest matching substring for the current phrase. At each position $i\ge \delta$, the algorithm applies all three methods and selects the one that corresponds to the longest match, ensuring correct identification of the $\mathsf{LZ}$77 factors.

Although there is no implementation of this algorithm, it has an efficient theoretical running time when the number of $\mathsf{LZ}$77 factors z is relatively small compared to the input size n.

3.9. Sublinear-Time $\mathsf{LZ}$77 factorization

Most recently, Kempa and Kociumaka (2024) presented the first algorithm to compute the $\mathsf{LZ}$77 factorization in $o(n)$ time. For a binary alphabet, the algorithm runs in $\mathsf{O}\left(n/\sqrt{\log \hspace{0.17em}n}\right)$ time and uses $\mathsf{O}(n/\log \hspace{0.17em}n)$ space. To achieve this sublinear-time $\mathsf{LZ}$77 algorithm, they establish a more general result: For any constant $ϵ\in (0,1)$ and any string $T\in {[0‥\sigma )}^n$, they show that it is possible to construct an index of optimal size $O(n/{\log }_{\sigma }\hspace{0.17em}n)$ in $O((n\hspace{0.17em}\log \hspace{0.17em}\sigma )/\sqrt{\log \hspace{0.17em}n})$ time, using $O(n/{\log }_{\sigma }\hspace{0.17em}n)$ working space. This index enables computing the leftmost occurrence of any substring $P=T[j‥j+\mathcal{l})$, given by the pair $(j,\mathcal{l})$, in $O({\log }^{ϵ}\hspace{0.17em}n)$ time. The authors’ approach to achieving sublinear-time $\mathsf{LZ}$77 factorization begins with the development of the first index that has a sublinear construction time that can quickly locate the leftmost occurrences of substrings of S. The primary challenge in building such an index with previous methods lies in the process of finding leftmost occurrences, which generally relies on RMQ applied to SA, and thus makes it impossible to achieve the construction of $o(n)$ times. Instead, Kempa and Kociumaka show that building an index that supports locating the leftmost occurrences of substrings of S can be accomplished by supporting a new type of query called a prefix Range Minimum Query (prefix RMQ). More formally, given an array of integers $A[1\dots m]$ and a sequence S of m strings over an alphabet $\Sigma$, a prefix RMQ with arguments $b,e\in [0\dots m]$ and a string $X\in {\Sigma }^{*}$ asks for the position i that minimizes A[i] among all indices $i\in (b\dots e]$ such that X is a prefix of S[i]. After defining this new query type, they then propose a space-efficient data structure to support it and describe its fast construction. This, in turn, leads to fast access to LPF for S, and thus the $\mathsf{LZ}$77 factorization. Hence, their efficient method for solving prefix RMQ queries, combined with a reduction showing that such improvements directly enhance substring indexing and the computation of leftmost occurrences, leads to faster execution of key operations in $\mathsf{LZ}$77 factorization.

3.10. The external KKP algorithms

Kärkkäinen et al. (2014) demonstrated the construction of $\mathsf{LZ}$77 factorization in external memory by presenting the following algorithms: External Memory Longest Previous Factor (EM-LPF), External Memory LZscan (EM-LZscan), and semi-external KKP (SE-KKP).

EM-LPF uses eSAIS (Bingmann et al., 2016) to compute the $\mathsf{SA}$ and $\mathsf{LCP}$ arrays in external memory. Once these arrays are computed, EM-LPF uses a linear-time algorithm to compute the $\mathsf{LZ}$77 factorization, based on Crochemore and Ilie (2008). This algorithm first computes $\mathsf{LPF}$ by scanning the $\mathsf{SA}$ and $\mathsf{LCP}$ arrays from left to right, while maintaining a stack of $\mathsf{SA}$ values and their corresponding $\mathsf{LCP}$ values. At each position i in $\mathsf{SA}$, the algorithm compares the current value with elements at the top of the stack to identify previous occurrences of substrings and their lengths. This computes the position, source, and length of each $\mathsf{LPF}$, where position corresponds to the SA index, the source is the previous stack entry or the current $\mathsf{SA}\left[i\right]$ value (whichever gives the longest match), and the length is the corresponding LCP value. Then, the elements $\mathsf{LPF}$ are sorted in text order with STXXL Dementiev et al. (2005) using external memory. Finally, the algorithm sequentially processes the sorted $\mathsf{LPF}$ elements to extract the $\mathsf{LZ}$77 factors by identifying nonoverlapping segments based on position and length, using the approach developed by Crochemore and Ilie (2008). EM-LPF relies heavily on disk I/O operations and requires substantial hard disk space, estimated at approximately 54n bytes.

EM-LZscan is an adaptation of LZscan (Kärkkäinen et al., 2013b), which computes the $\mathsf{LZ}$77 factorization based on matching statistics. Matching statistics are defined as follows: given two strings Y and Z, the matching statistics of Y with respect to Z, denoted as $M{S}_{Y|Z}$, is an array of |Y| pairs

$$(p_1,{\mathcal{l}}_1),(p_2,{\mathcal{l}}_2),\dots ,(p_{\left|Y\right|},{\mathcal{l}}_{\left|Y\right|})$$

where, for each position i in Y, the pair $(p_i,{\mathcal{l}}_i)$ indicates that the longest substring starting at position i in Y that also appears in Z is given by

$$Y[i\dots i+\mathcal{l})=Z[p_i\dots p_i+{\mathcal{l}}_i).$$

The algorithm processes the input string S in fixed-size blocks to efficiently handle strings that exceed the available memory. For each block $B=S[kb+1\dots (k+1)b]$, where k is the block index and b is the block size, the processed prefix is represented as $A=S[1\dots kb]$. EM-LZscan computes the arrays $\mathsf{S}{\mathsf{A}}_B$, $\mathsf{LC}{\mathsf{P}}_B$, and $\mathsf{LP}{\mathsf{F}}_B$ for the current block B, storing these data structures in RAM. Next, it computes matching statistics $M{S}_{B|A}$ by sequentially reading and scanning A from the disk in small buffers from right to left, identifying for each position i in A the longest substring $A[i\dots i+\mathcal{l})$ that also appears in B. The algorithm then transforms $M{S}_{A|B}$ into $M{S}_{B|A}$ using $\mathsf{S}{\mathsf{A}}_B$ and $\mathsf{LC}{\mathsf{P}}_B$ with the algorithm of Kärkkäinen et al. (2013b). By combining $M{S}_{B|A}$ with $\mathsf{LP}{\mathsf{F}}_B$, the algorithm computes $\mathsf{LP}{\mathsf{F}}_{AB}$, which represents, for each position i in B, the position of the first occurrence of the longest substring that starts at position i and occurs earlier in the concatenated string A-B. Using $\mathsf{LP}{\mathsf{F}}_{AB}$, the algorithm computes the $\mathsf{LZ}$77 factors of block B using the same method described in Section 3.2. This process is repeated for each block B until all $\mathsf{LZ}$77 factors in the whole input string have been computed. EM-LZscan runs in $\mathsf{O}(n^2{t}_{\mathit{rank}}/M)$ time, where M is the size of RAM and $t_{\mathit{rank}}$ is the time complexity of the rank operation, which is used during the computation of matching statistics.

Lastly, SE-KKP represents a semi-external memory approach. It first computes $\mathsf{SA}$ using the SAscan (Kärkkäinen and Kempa, 2017), and then scans $\mathsf{SA}$ to compute $\mathsf{PS}{\mathsf{V}}_{\mathit{lex}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{lex}}$ for each position in the string using a stack. This follows the same algorithm described in Subsection 3.3. After constructing $\mathsf{PS}{\mathsf{V}}_{\mathit{lex}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{lex}}$, the algorithm uses STXXL Dementiev et al. (2005) to covert $\mathsf{PS}{\mathsf{V}}_{\mathit{lex}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{lex}}$ to $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}$ by performing an external memory sort with fixed-size blocks of data. Next, for each position i, if $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ is not defined, the starting position of the $\mathsf{LZ}$77 factor will be $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$. If $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ is undefined, then the starting position of the current $\mathsf{LZ}$77 factor will be $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$. If both are defined, the algorithm computes the lcp between the substrings at the positions $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ and $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$, selecting the position that corresponds to the longest match. This match length is denoted as $\mathcal{l}$, and the $\mathsf{LZ}$77 factor is defined as $(p,\mathcal{l})$, where p is the position corresponding to $\mathsf{PS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$ or $\mathsf{NS}{\mathsf{V}}_{\mathit{text}}\left[i\right]$. The SE-KKP algorithms achieve an I/O complexity of $\mathsf{O}(n/B)$, where B is the block size, and requires approximately 21n bytes of space.

4. EXPERIMENTS

The experiments were run on a server with an AMD EPYC 75F3 CPU with Red Hat Enterprise Linux 7.7 (64-bit, kernel 3.10.0). The compiler was g++ version 12.2.0. Running time and memory usage were recorded by the Snakemake benchmark facility (Mölder et al., 2021). We set a memory limit of 128 GB and a time limit of 24 hours. We evaluated the performance of several algorithms described previously, specifically KKP2, RLE-LZ, and PFP-LZ77. We obtained the source code from the respective articles of the authors. The implementations were primarily through build systems such as CMake or Makefile. We followed the instructions provided to compile and run each algorithm.

KKP1 and KKP3 were excluded from the evaluation due to their implementations being unable to support large datasets. LZone was also not compared because the source code for LZone was unavailable, and currently, there is no implementation available for the sublinear algorithm. In addition, the methods of Ellert (2023) and Kempa and Kociumaka (2024) do not have an implementation and thus were not compared. We selected SE-KKP to represent the construction of LZ77 in external memory, as it demonstrated the best performance in external memory settings in a previous benchmark study by Hong et al. (2023).

4.1. Datasets

We used real-world data sets from Pizza&Chili repetitive corpus, Salmonella genomes obtained from the NCBI website, and human chromosome 19 genomes from The 1000 Genomes Project Consortium (2015). The Pizza&Chili repetitive corpus consists of repetitive texts that vary in length and alphabet size. For Salmonella data sets, we used four variants that contained 50, 100, 500, and 1,000 copies of the Salmonella genome sequence. Similarly, we used eight variants that contained 1, 2, 4, 8, 16, 32, 64, and 128 copies of the chromosome 19 genome sequence. In Table 2, we report the number of $\mathsf{LZ}$77 factors produced and the average factor lengths ( $n/z$). Where n represents the size of the input file in bytes and z represents the number of $\mathsf{LZ}$77 factors. This ratio indicates the compression potential of the data. The higher $n/z$ suggests greater repetitiveness in the data set.

Table 2.

Factor Number, File Size, and Average Phrase Length of Different Datasets

Dataset	Number of factors	File size (bytes)	$n/z$
Cere	1,700,630	461,286,644	271.25
Einstein.de.txt	34,572	92,758,441	2,683.05
Einstein.en.txt	89,467	467,626,544	5,226.80
Escherichia_Coli	2,078,512	112,689,515	54.22
Influenza	769,286	154,808,555	201.24
Kernel	793,915	257,961,616	324.92
Para	2,332,657	429,265,758	184.02
World_leaders	175,740	46,968,181	267.26
salmonella_50	3,166,246	261,523,734	82.60
salmonella_100	3,843,418	508,556,447	132.32
salmonella_500	12,576,165	2,553,875,813	203.07
salmonella_1000	22,139,715	5,097,148,124	230.23
chr1	3,750,850	59,128,983	15.76
chr2	3,855,058	118,255,109	30.68
chr4	3,957,472	236,505,070	59.76
chr8	4,049,288	473,004,898	116.81
chr16	4,144,317	945,998,727	228.26
chr32	4,257,171	1,891,989,453	444.42
chr64	4,389,413	3,783,968,805	862.07
chr128	4,574,714	7,567,919,010	1654.29

4.2. Results on Pizza&Chili

All methods, except the RLE-LZ algorithm, required less than 5GB of memory and were completed in less than 20 minutes. Figure 1 illustrates the memory (in bytes) and time (in seconds) of all methods. The SE-KKP algorithm used more time and memory than all other algorithms. Since SE-KKP is semi-external memory, it used a substantial amount of hard disk for each experiment. In particular, SE-KKP consumed between 2.47GB and 36.51GB of hard disk space across different datasets. The PFP-LZ77 required the least memory and time out of all methods on the Einstein.de.txt, Einstein.en.txt datasets, as these two datasets have a higher repetitiveness. The performance of the BGone algorithm is comparable to that of PFP-LZ77.

FIG. 1.

Time and memory required by all algorithms for the Pizza&Chili dataset. The left graph shows the time required to build the LZ77 factors. The graph on the right shows the memory required to compute the LZ77.

4.3. Results on Salmonella

In this experiment, we set a time limit of 24 hours. However, the RLE-LZ algorithm was unable to complete the experiment within this limit. To evaluate its performance more precisely, we extended the time limit and observed that it required 27 hours to process the data set with 1,000 copies of Salmonella. Although RLE-LZ consumed the least memory, it required the most time across all datasets. The SE-KKP algorithm consistently used the most memory and required 19.71 GB, 38.78 GB, 211.03 GB, and 428.60 GB of hard disk space in various scenarios. KKP2 was the fastest in this experiment, and its memory usage was similar to that of PFP-LZ77. However, our analysis indicates that KKP2’s memory consumption increases at a faster rate than the PFP-LZ77 method when processing larger datasets. The BGone algorithm could only complete processing 100 copies of the Salmonella dataset, as its memory requirements exceeded our available 128 GB RAM when attempting to process larger datasets. In the experiments it could complete, BGone demonstrated the second-lowest memory consumption and the second-fastest execution time, indicating its suitability for smaller datasets. Results shown in Figure 2.

FIG. 2.

Time and memory required by all the algorithms on the Salmonella dataset. The left graph shows the time required to build the LZ77 factors. The graph on the right shows the memory required to compute the LZ77.

4.4. Results on chromosome 19

In this experiment, we set a time limit of 24 hours. The BGone algorithm could only process data sets smaller than 64 copies of chromosome 19. For datasets with fewer than 16 copies, BGone uses the least memory and is only slower than the RLE-LZ and KKP2 algorithms. Although other algorithms showed increasing resource requirements as the size of the dataset increased, the PFP-LZ77 method demonstrated a more stable performance trend. As the size of the data set increased, SE-KKP consistently showed the poorest performance, becoming the slowest and most memory-intensive. Specifically, when the data set is larger than 16 copies, SE-KKP is 7.63 to 8.45 times slower than the KKP2 algorithm and requires 2.92 to 11.64 times more RAM than PFP-LZ77. As shown in Figure 3, the PFP-LZ77 method demonstrated the most stable performance in both time and space on varying dataset sizes. This stability indicates that, for large datasets, PFP-LZ77 provides a more reliable approach to construct the $\mathsf{LZ}$77 factors.

FIG. 3.

Time and Memory required by all the algorithms on the Chromosome 19 dataset. The left graph shows the time required to build the LZ77 factors. The graph on the right shows the memory required to compute the LZ77.

4.5. Recommendations

Based on our experimental evaluation, we recommend KKP2 and PFP-LZ77 as balanced and reliable choices to compress large-scale genomic data, offering strong performance in both runtime and memory usage across a variety of datasets. These algorithms consistently performed well on highly repetitive sequences, such as those in the Pizza&Chili corpus, as well as on real-world genomic data, including Salmonella genomes. KKP2, in particular, was notable for its speed, while PFP-LZ77 offers additional flexibility through its use of PFP and support for CST queries. RLE-LZ, although the most memory-efficient algorithm tested, demonstrated significantly longer runtimes and did not complete on the largest datasets, making it suitable only in highly memory-constrained environments where execution time is not a concern. SE-KKP proved effective in external memory settings, making it an appropriate option for datasets that exceed main memory limits, though it comes with increased resource usage. In practice, the availability of robust, well-documented, and actively maintained implementations, such as those for KKP2 and PFP-LZ77, should be a key consideration when selecting a compression strategy, as this can significantly affect integration and long-term usability. Although algorithms like KKP3 and BGone employ lazy evaluation techniques, our results do not indicate substantial runtime advantages of this design, particularly compared to alternatives such as KKP2. Ultimately, no single algorithm dominates in all scenarios; therefore, researchers and practitioners should carefully consider the trade-offs between time, space, and implementation maturity when selecting an algorithm based on $\mathsf{LZ}$77 for genomic data compression.

5. CONCLUSION AND FUTURE WORK

Throughout this review, we have thoroughly explored the evolution and enhancement of the $\mathsf{LZ}$77 algorithm and its derivatives, particularly focusing on their applications in genomic data compression. The continuous growth of genomic data requires efficient compression solutions that can effectively manage and utilize these vast datasets. Traditional $\mathsf{LZ}$77 algorithms, while fundamental in the realm of lossless data compression, require significant adaptations to meet the unique demands of genomic sequences characterized by their repetitive nature. The advancements discussed in this article highlight significant strides in algorithmic innovation aimed at optimizing $\mathsf{LZ}$77 for better handling of large-scale genomic data. We have examined a variety of specialized algorithms that not only improve compression ratios but also enhance processing efficiencies in both software and hardware implementations. These adaptations make it feasible to manage, store, and analyze genomic data more effectively, supporting the broader goals of genomic research and clinical applications.

Moreover, the integration of LZ77-based algorithms into hardware platforms represents a promising avenue for overcoming the throughput limitations of software-based compression. This approach can potentially unlock real-time processing capabilities, essential for applications requiring immediate data analysis, such as the compression of pangenomic datasets. In the future, the ongoing development of LZ77-based compression technologies will undoubtedly play a critical role in the bioinformatics field. Continued innovations are expected to further refine these algorithms, making them more adaptable to the evolving landscape of genomic data. Furthermore, exploring synergies between algorithmic improvements and hardware accelerations will be crucial to achieving the scalability and efficiency required for future applications in biological data compression, software distribution, streaming media, and medical imaging data. In the following, we highlight some specific areas that warrant future research.

5.1. Merging $\mathsf{LZ}$77 factorizations

There are many large sequence datasets that grow rapidly, such as GenomeTrakr (Timme et al., 2019) and MetaSub (Mason et al., 2016), which update frequently. Currently, when applying $\mathsf{LZ}$77 factorization to these datasets, we need to regenerate the $\mathsf{LZ}$77 factorization each time the datasets are updated. This process is time-consuming and computationally expensive. However, if we can update the $\mathsf{LZ}$77 factors generated from previous versions of the dataset, it would save a significant amount of time and computational resources. As described by Oliva et al. (2021) and Muggli et al. (2019), dynamic data structures and algorithms for merging BWTs have been successfully applied to address similar challenges in other contexts. These techniques could be adapted to the $\mathsf{LZ}$77 construction process.

5.2. External memory construction

Although there have been several methods that use external memory for $\mathsf{LZ}$77 factorization—as discussed in Subsection 3.10—this area still deserves continued attention since data sets continue to grow substantially larger and will surpass main memory. Current algorithms that rely on external memory require substantially more time than existing methods, which is unsurprising given the gap between the time required to access main memory and access to the hard disk. However, there are still opportunities for advancement in this area. For instance, consider the PFP-LZ77 algorithm. Previously, the PFP parsing for the input string S was computed in a single step within main memory. However, this process can be adapted to handle larger data sets by dividing the input string into manageable blocks that fit into main memory. Each block can then be processed independently using the PFP-LZ77 algorithm to produce partial $\mathsf{LZ}$77 factorizations. After processing all blocks, a merging phase can combine these partial factorizations into a global factorization. Multithreading can be applied to this process, significantly speeding up the processing time. Furthermore, using disk-based data structures, such as B-trees, can facilitate the efficient search operations required by the $\mathsf{LZ}$77 algorithm, enhancing the algorithm’s performance in an external memory context.

5.3. GPU implementation

Exploring the implementation of $\mathsf{LZ}$77 on GPU is a promising direction for future research. GPUs are well-suited for parallel processing tasks due to their ability to handle many computations simultaneously. Although some existing research has successfully applied GPUs to fundamental data structures such as suffix arrays (Büren et al., 2019), more complex algorithms such as $\mathsf{LZ}$77 have not been extensively explored. A key challenge in implementing $\mathsf{LZ}$77 on GPUs is efficiently partitioning input data and managing memory to ensure that each GPU thread operates independently. One approach is to divide the input string into smaller blocks that fit into GPU memory and process each block in parallel. After processing, the partial $\mathsf{LZ}$77 factorizations can be merged into a global factorization. Minimizing data transfer between the CPU and the GPU and using asynchronous transfers can further enhance performance. Future research could focus on optimizing these parallel algorithms, exploring hybrid CPU-GPU approaches, and testing the implementation on real-world datasets to validate its effectiveness.

DATA AVAILABILITY STATEMENT

We used the real-world datasets from the Pizza&Chili repetitive corpus, Salmonella genomes obtained from the NCBI website. The Pizza&Chili repetitive corpus consists of repetitive texts that vary in length and alphabet size, providing a diverse set of inputs to test the performance of the algorithms. This data set is widely used for benchmarking as it includes standard test cases that have been extensively studied in previous work on compression. This data set can be accessed at https://pizzachili.dcc.uchile.cl/biblio.html. For Salmonella data sets, we used four sets that contain 50, 100, 500, and 1,000 copies of the Salmonella genome sequence. All Salmonella genome sequences are available at https://www.ncbi.nlm.nih.gov/datasets/genome/?taxon=590. For human chromosome 19 analysis, we constructed eight datasets containing 1, 2, 4, 8, 16, 32, 64, and 128 copies of the chromosome 19 genome sequence. All chromosome 19 sequences were accessed through the NCBI gene database https://www.ncbi.nlm.nih.gov/gene?Db=gene&Cmd=DetailsSearch&Term=148223.

AUTHORS’ CONTRIBUTIONS

A.H.: Theoretical and formal analysis, data curation, writing—original draft, writing—review and editing, experiments. C.B.: Funding acquisition, supervision, writing—original draft, writing—review and editing, reviewing experiments and analysis.

AUTHOR DISCLOSURE STATEMENT

The authors have no conflict of interest.

FUNDING INFORMATION

This research was funded by NSF:SCH:INT (Grant No. 2013998) and NIH/NHGRI (Grant No. R01HG011392).