Asymptotic Analysis of the kth Subword Complexity

Abstract

Patterns within strings enable us to extract vital information regarding a string’s randomness. Understanding whether a string is random (Showing no to little repetition in patterns) or periodic (showing repetitions in patterns) are described by a value that is called the kth Subword Complexity of the character string. By definition, the kth Subword Complexity is the number of distinct substrings of length k that appear in a given string. In this paper, we evaluate the expected value and the second factorial moment (followed by a corollary on the second moment) of the kth Subword Complexity for the binary strings over memory-less sources. We first take a combinatorial approach to derive a probability generating function for the number of occurrences of patterns in strings of finite length. This enables us to have an exact expression for the two moments in terms of patterns’ auto-correlation and correlation polynomials. We then investigate the asymptotic behavior for values of $k=\mathsf{\Theta }(logn)$. In the proof, we compare the distribution of the kth Subword Complexity of binary strings to the distribution of distinct prefixes of independent strings stored in a trie. The methodology that we use involves complex analysis, analytical poissonization and depoissonization, the Mellin transform, and saddle point analysis.

1. Introduction

Analyzing and understanding occurrences of patterns in a character string is helpful for extracting useful information regarding the nature of a string. We classify strings to low-complexity and high-complexity, according to their level of randomness. For instance, we take the binary string $X=10101010...$, which is constructed by repetitions of the pattern $w=10$. This string is periodic, and therefore has low randomness. Such periodic strings are classified as low-complexity strings, whereas strings that do not show periodicity are considered to have high complexity. An effective way of measuring a string’s randomness is to count all distinct patterns that appear as contiguous subwords in the string. This value is called the Subword Complexity. The name is given by Ehrenfeucht, Lee, and Rozenberg [1], and initially was introduced by Morse and Hedlund in 1938 [2]. The higher the Subword Complexity, the more complex the string is considered to be.

Assessing information about the distribution of the Subword Complexity enables us to better characterize strings, and determine atypically random or periodic strings that have complexities far from the average complexity [3]. This type of string classification has applications in fields such as data compression [4], genome analysis (see [5,6,7,8,9]), and plagiarism detection [10]. For example, in data compression, a data set is considered compressible if it has low complexity, as consists of repeated subwords. In computational genomics, Subword Complexity (known as k-mers) is used in detection of repeated sequences and DNA barcoding [11,12]. k-mers are composed of A, T, G, and C nucleotides. For instance, 7-mers for a DNA sequence GTAGAGCTGT is four, meaning that there are 4-hour distinct substrings of length 7 in the given DNA sequence. Counting k-mers becomes challenging for longer DNA sequences. Our results can be easily extended to the alphabet $\{A,T,G,C\}$ and directly applied in theoretical analysis of the genomic k-mer distributions under the Bernoulli probabilistic model, particularly when the length n of the sequence approaches infinity.

There are two variations for the definition of the Subword Complexity: the one that counts all distinct subwords of a given string (also known as Complexity Index and Sequence Complexity [13]), and the one that only counts the subwords of the same length, say k, that appear in the string. In our work, we analyze the latter, and we call it the kth Subword Complexity to avoid any confusion.

Throughout this work, we consider the kth Subword Complexity of a random binary string of length n over a memory-less source, and we denote it by $X_{n,k}$. We analyze the first and second factorial moments of $X_{n,k}$ (1) for the range $k=\mathsf{\Theta }(logn)$, as $n\to \infty$. More precisely, will divide the analysis into three ranges as follows.

• $i.$${\displaystyle \frac{1}{log{q}^{-1}}}logn<k<{\displaystyle \frac{2}{log{q}^{-1}+log{p}^{-1}}}logn$,
• $ii.$${\displaystyle \frac{2}{log{q}^{-1}+log{p}^{-1}}}logn<k<{\displaystyle \frac{1}{qlog{q}^{-1}+plog{p}^{-1}}}logn$, and
• $iii.$${\displaystyle \frac{1}{qlog{q}^{-1}+plog{p}^{-1}}}logn<k<{\displaystyle \frac{1}{log{p}^{-1}}}logn$.

Our approach involves two major steps. First, we choose a suitable model for the asymptotic analysis, and afterwards we provide proofs for the derivation of the asymptotic expansion of the first two factorial moments.

1.1. Part I

This part of the analysis is inspired by the earlier work of Jacquet and Szpankowski [14] on the analysis of suffix trees by comparing them to independent tries. A trie, first introduced by René de la Briandais in 1959 (see [15]), is a search tree that stores n strings, according to their prefixes. A suffix tree, introduced by Weiner in 1973 (see [16]), is a trie where the strings are suffixes of a given string. An example of these data structures are given in Figure 1.

Figure 1.

The suffix tree in (a) is built over the first four suffixes of string $X=101110...$, and the trie in (b) is build over strings $X_1=111...$, $X_2=101...$, $X_3=100$, and $X_4=010...$.

A direct asymptotic analysis of the moments is a difficult task, as patterns in a string are not independent from each other. However, we note that each pattern in a string can be regarded as a prefix of a suffix of the string. Therefore, the number of distinct patterns of length k in a string is actually the number of nodes of the suffix tree at level k and lower. It is shown by I. Gheorghiciuc and M. D. Ward [17] that the expected value of the k-th Subword Complexity of a Bernoulli string of length n is asymptotically comparable to the expected value of the number of nodes at level k of a trie built over n independent strings generated by a memory-less source.

We extend this analysis to the desired range for k, and we prove that the result holds for when k grows logarithmically with n. Additionally, we show that asymptotically, the second factorial moment of the k-th Subword Complexity can also be estimated by admitting the same independent model generated by a memory-less source. The proof of this theorem heavily relies on the characterization of the overlaps of the patterns with themselves and with one another. Autocorrelation and correlation polynomials explicitly describe these overlaps. The analytic properties of these polynomials are key to understanding repetitions of patterns in large Bernoulli strings. This, in conjunction with Cauchy’s integral formula (used to compare the generating functions in the two models) and the residue theorem, provides solid verification that the second factorial moment in the Subword Complexity behaves the same as in the independent model.

To make this comparison, we derive the generating functions of the first two factorial moments in both settings. In a paper published by F. Bassino, J. Clément, and P. Nicodème in 2012 [18], the authors provide a multivariate probability generating function $f(z,x)$ for the number of occurrences of patterns in a finite Bernoulli string. That is, given a pattern w, the coefficient of the term $z^n{x}^m$ in $f(z,x)$ is the probability in the Bernoulli model that a random string of size n has exactly m occurrences of the pattern w. Following their technique, we derive the exact expression for the generating functions of the first two factorial moments of the kth Subword Complexity. In the independent model, the generating functions are obtained by basic probability concepts.

1.2. Part II

This part of the proof is analogous to the analysis of profile of tries [19]. To capture the asymptotic behavior, the expressions for the first two factorial moments in the independent trie are further improved by means of a Poisson process. The poissonized version yields generating functions in the form of harmonic sums for each of the moments. The Mellin transform and the inverse Mellin transforms of these harmonic sums establish a connection between the asymptotic expansion and singularities of the transformed function. This methodology is sufficient for when the length k of the patterns are fixed. However, allowing k to grow with n, makes the analysis more challenging. This is because for large k, the dominant term of the poissonized generating function may come from the term involving k, and singularities may not be significant compared to the growth of k. This issue is treated by combining the singularity analysis with a saddle point method [20]. The outcome of the analysis is a precise first-order asymptotics of the moments in the poissonized model. Depoissonization theorems are then applied to obtain the desired result in the Bernoulli model.

2. Results

For a binary string $X=X_1{X}_2...X_n$, where $X_i$’s ($i=1,...,n$) are independent and identically distributed random variables, we assume that $\mathbf{P}(X_i=1)=p$, $\mathbf{P}(X_i=0)=q=1-p$, and $p>q$. We define the kth Subword Complexity, $X_{n,k}$, to be the number of distinct substrings of length k that appear in a random string X with the above assumptions. In this work, we obtain the first order asymptotics for the average and the second factorial moment of $X_{n,k}$. The analysis is done in the range $k=\mathsf{\Theta }(logn)$. We rewrite this range as $k=alogn$, and by performing a saddle point analysis, we will show that

$$\begin{array}{c}\hfill 1/log{q}^{-1}<a<1/log{p}^{-1}\end{array}$$ (1)

In the first step, we compare the kth Subword Complexity to an independent model constructed in the following way: We store a set of n independently generated strings by a memory-less source in a trie. This means that each string is a sequence of independent and identically distributed Bernoulli random variables from the binary alphabet $\mathcal{A}=\{0,1\}$, with $\mathbf{P}\left(1\right)=p$, $\mathbf{P}\left(0\right)=q=1-p$. We denote the number of distinct prefixes of length k in the trie by ${\widehat{X}}_{n,k}$, and we call it the kth prefix complexity. Before proceeding any further, we remind that factorial moments of a random variable are defined as following.

Definition 1.

The jth factorial moment of a random variable X is defined as

$$\begin{array}{c}\hfill \mathbf{E}\left[{\left(X\right)}_j\right]=\mathbf{E}[\left(X\right)(X-1)(X-2)...(X-j+1)],\end{array}$$ (2)

where j = 1, 2, … will show that the first and second factorial moments of $X_{n,k}$ are asymptotically comparable to those of ${\widehat{X}}_{n,k}$, when $k=\mathsf{\Theta }(logn)$. We have the following theorems.

Theorem 1.

For large values of n, and for $k=\mathsf{\Theta }(logn)$, there exists $M>0$ such that

$$\mathbf{E}\left[X_{n,k}\right]-\mathbf{E}\left[{\widehat{X}}_{n,k}\right]=O\left(n^{-M}\right).$$

We also prove a similar result for the second factorial moments of the kth Subword Complexity and the kth Prefix Complexity:

Theorem 2.

For large values of n, and for $k=\mathsf{\Theta }(logn)$, there exists $ϵ>0$ such that

$$\mathbf{E}\left[{\left(X_{n,k}\right)}_2\right]-\mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]=O\left(n^{-ϵ}\right).$$

In the second part of our analysis, we derive the first order asymptotics of the kth Prefix Complexity. The methodology used here is analogous to the analysis of profile of tries [19]. The rate of the asymptotic growth depends on the location of the value a as seen in (1). For instance, for the average kth Subword Complexity, $\mathbf{E}\left[X_{n,k}\right]$, we have the following observations.

• i.For the range $I_1:{\displaystyle \frac{1}{log{q}^{-1}}}<a<{\displaystyle \frac{2}{log{q}^{-1}+log{p}^{-1}}}$, the growth rate is of order $O\left(2^k\right)$,
• ii.in the range $I_2:{\displaystyle \frac{2}{log{q}^{-1}+log{p}^{-1}}}<a<{\displaystyle \frac{1}{qlog{q}^{-1}+plog{p}^{-1}}}$, we observe some oscillations with n, and
• iii.in the range $I_3:{\displaystyle \frac{1}{qlog{q}^{-1}+plog{p}^{-1}}}<a<{\displaystyle \frac{1}{log{p}^{-1}}}$, the average has a linear growth $O\left(n\right)$.

The above observations will be discussed in depth in the proofs of the following theorems.

Theorem 3.

The average of the kth Prefix Complexity has the following asymptotic expansion

• i. 
  For $a\in I_1$,

  $$\begin{array}{cc}\hfill \mathbf{E}\left[{\widehat{X}}_{n,k}\right]& =2^k-{\mathsf{\Phi }}_1\left((1+logp){log}_{p/q}n\right){\displaystyle \frac{n^{\nu }}{\sqrt{logn}}}\left(1+O\left({\displaystyle \frac{1}{\sqrt{logn}}}\right)\right),\hfill \end{array}$$ (3)
  where $\nu =-r_0+alog(p^{-r_0}+q^{-r_0})$, and

  $$\begin{array}{cc}\hfill {\mathsf{\Phi }}_1\left(x\right)& ={\displaystyle \frac{{(p/q)}^{-r_0/2}+{(p/q)}^{r_0/2}}{\sqrt[]{2\pi }logp/q}}\sum _{j\in \mathbb{Z}}\mathsf{\Gamma }(r_0+i{t}_j)e^{-2\pi ijx}\hfill \end{array}$$

  is a bounded periodic function.

• ii. 
  For $a\in I_2$,

  $$\begin{array}{cc}\hfill \mathbf{E}\left[{\widehat{X}}_{n,k}\right]& ={\mathsf{\Phi }}_1\left((1+logp){log}_{p/q}n\right){\displaystyle \frac{n^{\nu }}{\sqrt{logn}}}\left(1+O\left({\displaystyle \frac{1}{\sqrt{logn}}}\right)\right).\hfill \end{array}$$
• iii. 
  For $a\in I_3$

  $$\begin{array}{cc}\hfill \mathbf{E}\left[{\widehat{X}}_{n,k}\right]& =n+O\left(n^{{\nu }_0}\right),\hfill \end{array}$$

  for some ${\nu }_0<1$.

Theorem 4.

The second factorial moment of the kth Prefix Complexity has the following asymptotic expansion.

• i. 
  For $a\in I_1$,

  $$\begin{array}{c}\hfill \mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]={\left(2^k-{\mathsf{\Phi }}_1\left({log}_{p/q}n(1+logp)\right){\displaystyle \frac{n^{\nu }}{\sqrt{logn}}}\left(1+O\left({\displaystyle \frac{1}{\sqrt{logn}}}\right)\right)\right)}^2.\end{array}$$
• ii. 
  For $a\in I_2$,

  $$\begin{array}{cc}\hfill \mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]& ={\mathsf{\Phi }}_1^2\left({log}_{p/q}n(1+logp)\right){\displaystyle \frac{n^{2\nu }}{logn}}\left(1+O\left({\displaystyle \frac{1}{logn}}\right)\right).\hfill \end{array}$$
• iii. 
  For $a\in I_3$,

  $$\begin{array}{c}\hfill \mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]=n^2+O\left(n^{2{\nu }_0}\right).\end{array}$$

The periodic function ${\mathsf{\Phi }}_1\left(x\right)$ in Theorems 3 and 4 is shown in Figure 2.

Figure 2.

Left: ${\mathsf{\Phi }}_1\left(x\right)$ at $p=0.90$, and various levels of $r_0$. The amplitude increases as $r_0$ increases. Right: ${\mathsf{\Phi }}_1\left(x\right)$ at $r_0=1$, and various levels of p. The amplitude tends to zero as $p\to 1/2^{+}$.

The results in Theorem 4 will follow for the second moment of the kth Subword Complexity as the analysis can be easily extended from the second factorial moment to the second moment. The variance however, as seen in Figure 3, does not show the same asymptotic behavior as the variance of kth Subword Complexity.

Figure 3.

Approximated second moments (left), and variances (right) of the kth Subword Complexity (red), and the kth Prefix Complexity (blue), for $n=4000$, at different probability levels, averaged over 10,000 iterations.

3. Proofs and Methods

3.1. Groundwork

We first introduce a few terminologies and lemmas regarding overlaps of patterns and their number of occurrences in texts. Some of the notations we use in this work are borrowed from [18] and [21].

Definition 2.

For a binary word $w=w_1...w_k$ of length k, The autocorrelation set ${\mathcal{S}}_w$ of the word w is defined in the following way.

$$\begin{array}{c}\hfill {\mathcal{S}}_w=\{w_{i+1}...w_k|w_1...w_i=w_{k-i+1}...w_k\}.\end{array}$$ (4)

The autocorrelation index set is

$$\begin{array}{c}\hfill \mathcal{P}\left(w\right)=\left\{i\right|w_1...w_i=w_{k-i+1}...w_k\},\end{array}$$ (5)

And the autocorrelation polynomial is

$$\begin{array}{c}\hfill S_w\left(z\right)=\sum _{i\in \mathcal{P}\left(w\right)}\mathbf{P}(w_{i+1}...w_k)z^{k-i}.\end{array}$$ (6)

Definition 3.

For the distinct binary words $w=w_1...w_k$ and $w^{\prime }=w_1^{\prime }...w_k^{\prime }$, the correlation set ${\mathcal{S}}_{w,w^{\prime }}$ of the words w and $w^{\prime }$ is

$$\begin{array}{c}\hfill {\mathcal{S}}_{w,w^{\prime }}=\{w_{i+1}^{\prime }...w_k^{\prime }|w_1^{\prime }...w_i^{\prime }=w_{k-i+1}...w_k\}.\end{array}$$ (7)

The correlation index set is

$$\begin{array}{c}\hfill \mathcal{P}(w,w^{\prime })=\left\{i\right|w_1^{\prime }...w_i^{\prime }=w_{k-i+1}...w_k\},\end{array}$$ (8)

The correlation polynomial is

$$\begin{array}{c}\hfill S_{w,w^{\prime }}\left(z\right)=\sum _{i\in \mathcal{P}(w,w^{\prime })}\mathbf{P}(w_{i+1}^{\prime }...w_k^{\prime })z^{k-i}.\end{array}$$ (9)

The following two lemmas present the probability generating functions for the number of occurrences of a single pattern and a pair of distinct pattern, respectively, in a random text of length n. For a detailed dissection on obtaining such generating functions, refer to [18].

Lemma 1.

The Occurrence probability generating function for a single pattern w in a binary text over a memoryless source is given by $F_w(z,x-1)$, where

$$\begin{array}{c}\hfill F_w(z,t)={\displaystyle \frac{1}{1-A\left(z\right)-{\displaystyle \frac{t\mathbf{P}\left(w\right)z^k}{1-t(S_w\left(z\right)-1)}}}},\end{array}$$ (10)

The coefficient $\left[z^n{x}^m\right]F_w(z,x-1)$ is the probability that a random binary string of length n has m occurrences of the pattern w.

Lemma 2.

The Occurrence PGF for two distinct Patterns of length k in a Bernoulli random text is given by $F_{w,w^{\prime }}(z,x_1-1,x_2-1)$ where,

$$\begin{array}{c}\hfill F_{w,w^{\prime }}(z,t_1,t_2)={\displaystyle \frac{1}{1-A\left(z\right)-M(z,t_1,t_2)}},\end{array}$$ (11)

and

$$M(z,t_1,t_2)=\left(\begin{array}{cc}\mathbf{P}\left(w\right)z^k{t}_1& \mathbf{P}\left(w^{\prime }\right)z^k{t}_2\end{array}\right){\left(\begin{array}{c}\mathbb{I}-\left(\begin{array}{cc}(S_w\left(z\right)-1)t_1& S_{w,w^{\prime }}\left(z\right)t_2\\ S_{w^{\prime },w}\left(z\right)t_1& (S_{w^{\prime }}\left(z\right)-1)t_2\end{array}\right)\end{array}\right)}^{-1}\left(\begin{array}{c}1\\ 1\end{array}\right).$$

The coefficient $\left[z^n{x}_1^{m_1}{x}_2^{m_2}\right]F_{w,w^{\prime }}(z,x_1-1,x_2-1)$ is the probability that there are $m_1$ occurrences of w and $m_2$ occurrences of $w^{\prime }$ in a random string of length n.

The above results will be used to find the generating functions for the first two factorial moments of the kth Subword Complexity in the following section.

3.2. Derivation of Generating Functions

Lemma 3.

For generating functions $H_k\left(z\right)={\sum }_{n\ge 0}\mathbf{E}\left[X_{n,k}\right]z^n$ and $G_k\left(z\right)={\sum }_{n\ge 0}\mathbf{E}\left[{\left(X_{n,k}\right)}_2\right]z^n$, we have

• i. 
  $$\begin{array}{c}\hfill H_k\left(z\right)=\sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{S_w\left(z\right)}{D_w\left(z\right)}}\right),\end{array}$$ (12)

  where $D_w\left(z\right)=\mathbf{P}\left(w\right)z^k+(1-z)S_w\left(z\right)$, and
• ii. 
  $$\begin{array}{c}\hfill G_k\left(z\right)=\sum _{\begin{array}{c}w,w^{{}^{\prime }}\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{S_w\left(z\right)}{D_w\left(z\right)}}-{\displaystyle \frac{S_{w^{{}^{\prime }}}\left(z\right)}{D_{w^{\prime }}\left(z\right)}}+{\displaystyle \frac{S_w\left(z\right)S_{w^{{}^{\prime }}}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{{}^{\prime }},w}\left(z\right)}{D_{w,w^{\prime }}\left(z\right)}}\right),\end{array}$$ (13)

  where

  $$\begin{array}{c}{D}_{w,w^{\prime }}\left(z\right)=(1-z)(S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right))\hfill \\ \hfill +z^k\left(\mathbf{P}\left(w\right)(S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right))+\mathbf{P}\left(w^{\prime }\right)(S_w\left(z\right)-S_{w^{\prime },w}\left(z\right))\right).\end{array}$$ (14)

Proof. 

$i.$ We define

$$X_{n,k}^{\left(w\right)}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}w\mathrm{appears}\mathrm{at}\mathrm{least}\mathrm{once}\mathrm{in}\mathrm{string}X\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

This yields

$$\begin{array}{cc}\hfill \mathbf{E}\left[X_{n,k}^{\left(w\right)}\right]& =\mathbf{P}(X_{n,k}^{\left(w\right)}=1)\hfill \\ \hfill & =1-P(X_{n,k}^{\left(w\right)}=0)\hfill \\ \hfill & =1-\left[z^n{x}^0\right]F_w(z,x).\hfill \end{array}$$ (15)

We observe that $\left[z^n{x}^0\right]F_w(z,x)=\left[z^n\right]F_w(z,0)$. By defining $f_w\left(z\right)=F_w(z,0)$ and from (10), we obtain

$$\begin{array}{cc}\hfill f_w\left(z\right)& ={\displaystyle \frac{S_w\left(z\right)}{\mathbf{P}\left(w\right)z^k+(1-z)S_w\left(z\right)}}.\hfill \end{array}$$ (16)

Having the above function, we derive the following result.

$$\begin{array}{cc}\hfill H\left(z\right)& =\sum _{n\ge 0}\mathbf{E}\left[X_{n,k}\right]z^n\hfill \\ \hfill & =\sum _{n\ge 0}\sum _{w\in {\mathcal{A}}^k}(1-\left[z^n\right]f_w\left(z\right))z^n\hfill \\ \hfill & =\sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{1}{1-z}}-f_w\left(z\right)\right)\hfill \\ \hfill & =\sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{S_w\left(z\right)}{D_w\left(z\right)}}\right).\hfill \end{array}$$ (17)

$ii.$ For this part, we first note that

$$\begin{array}{cc}\hfill \mathbf{E}\left[{\left(X_{n,k}\right)}_2\right]& =\mathbf{E}\left[X_{n,k}^2\right]-\mathbf{E}\left[X_{n,k}\right]\hfill \\ \hfill & =\mathbf{E}\left[{(X_{n,k}^{\left(w\right)}+...+X_{n,k}^{\left(w^{\left(r\right)}\right)})}^2\right]-\mathbf{E}\left[X_{n,k}^{\left(w\right)}+...+X_{n,k}^{(w^{\left(r\right)})}\right]\hfill \\ \hfill & =\sum _{w\in {\mathcal{A}}^k}\mathbf{E}\left[{\left(X_{n,k}^{\left(w\right)}\right)}^2\right]+\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\mathbf{E}\left[X_{n,k}^{\left(w\right)}{X}_{n,k}^{\left(w^{\prime }\right)}\right]-\sum _{w\in {\mathcal{A}}^k}\mathbf{E}\left[X_{n,k}^{\left(w\right)}\right]\hfill \\ \hfill & =\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\mathbf{E}\left[X_{n,k}^{\left(w\right)}{X}_{n,k}^{\left(w^{\prime }\right)}\right].\hfill \end{array}$$ (18)

Due to properties of indicator random variables, we observe that the expected value of the second factorial moment has only one term:

$$\begin{array}{c}\hfill \mathbf{E}\left[{\left(X_{n,k}\right)}_2\right]=\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\mathbf{E}\left[X_{n,k}^{\left(w\right)}{X}_{n,k}^{\left(w^{\prime }\right)}\right].\end{array}$$ (19)

We proceed by defining a second indicator variable as following.

$$X_{n,k}^{\left(w\right)}{X}_{n,k}^{\left(w^{\prime }\right)}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{X}_{n,k}^{\left(w\right)}=X_{n,k}^{\left(w^{\prime }\right)}=1\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

This gives

$$\begin{array}{cc}\hfill \mathbf{E}\left[X_{n,k}^{\left(w\right)}{X}_{n,k}^{\left(w^{\prime }\right)}\right]& =\mathbf{P}\left(X_{n,k}^{\left(w\right)}=1,X_{n,k}^{\left(w^{\prime }\right)}=1\right)\hfill \\ \hfill & =1-\mathbf{P}\left(X_{n,k}^{\left(w\right)}=0\cup X_{n,k}^{\left(w^{\prime }\right)}=0\right)\hfill \\ \hfill & =1-\mathbf{P}\left(X_{n,k}^{\left(w\right)}=0\right)-\mathbf{P}\left(X_{n,k}^{\left(w^{\prime }\right)}=0\right)+\mathbf{P}\left(X_{n,k}^{\left(w\right)}=0,X_{n,k}^{\left(w^{\prime }\right)}=0\right).\hfill \end{array}$$

Finally, we are able to express $\mathbf{E}\left[{\left(X_{n,k}\right)}_2\right]$ in the following

$$\begin{array}{cc}\hfill \mathbf{E}\left[{\left(X_{n,k}\right)}_2\right]& =\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left(1-\left[z^n\right]f_w\left(z\right)-\left[z^n\right]f_{w^{\prime }}\left(z\right)+\left[z^n\right]f_{w{w}^{\prime }}\left(z\right)\right),\hfill \end{array}$$ (20)

where $f_{w,w^{\prime }}\left(z\right)=F_{w,w^{\prime }}(z,0,0)$ and $\left[z^n\right]F_{w,w^{\prime }}(z,0,0)=\left[z^n{x}_1^0{x}_2^0\right]F_{w,w^{\prime }}(z,x_1,x_2)$. By (11) we have

$$\begin{array}{cc}\hfill f_{w,w^{\prime }}\left(z\right)& ={\displaystyle \frac{S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)}{D_{w,w^{\prime }}\left(z\right)}}\hfill \end{array}$$ (21)

Having the above expression, we finally obtain

$$\begin{array}{cc}\hfill G_k\left(z\right)& =\sum _{n\ge 0}\mathbf{E}\left[{\left(X_{n,k}\right)}_2\right]z^n\hfill \\ \hfill & =\sum _{\begin{array}{c}w,w^{{}^{\prime }}\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\sum _{n\ge 0}\left(1-\left[z^n\right]f_w\left(z\right)-\left[z^n\right]f_{w^{{}^{\prime }}}\left(z\right)+\left[z^n\right]f_{w,w^{{}^{\prime }}}\left(z\right)\right)z^n\hfill \\ \hfill & =\sum _{\begin{array}{c}w,w^{{}^{\prime }}\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left({\displaystyle \frac{1}{1-z}}-f_w\left(z\right)-f_{w^{{}^{\prime }}}\left(z\right)+f_{w,w^{{}^{\prime }}}\left(z\right)\right)\hfill \\ \hfill & =\sum _{\begin{array}{c}w,w^{{}^{\prime }}\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{S_w\left(z\right)}{D_w\left(z\right)}}-{\displaystyle \frac{S_{w^{{}^{\prime }}}\left(z\right)}{D_{w^{\prime }}\left(z\right)}}+{\displaystyle \frac{S_w\left(z\right)S_{w^{{}^{\prime }}}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{{}^{\prime }},w}\left(z\right)}{D_{w,w^{\prime }}\left(z\right)}}\right).\hfill \end{array}$$ (22)

□

In the following lemma, we present the generating functions for the first two factorial moments for the kth Prefix Complexity in the independent model.

Lemma 4.

For ${\widehat{H}}_k\left(z\right)={\sum }_{n\ge 0}\mathbf{E}\left[{\widehat{X}}_{n,k}\right]z^n$ and ${\widehat{G}}_k\left(z\right)={\sum }_{n\ge 0}\mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]z^n$, which are the generating functions for $\mathbf{E}\left[{\widehat{X}}_{n,k}\right]$ and $\mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]$ respectively, we have

• i. 
  $$\begin{array}{cc}\hfill {\widehat{H}}_k\left(z\right)& =\sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{1}{1-(1-\mathbf{P}(w\left)\right)z}}\right).\hfill \end{array}$$ (23)
• ii. 
  $$\begin{array}{cc}\hfill {\widehat{G}}_k\left(z\right)& =\sum _{\begin{array}{c}w,w^{{}^{\prime }}\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{1}{1-(1-\mathbf{P}(w\left)\right)z}}-{\displaystyle \frac{1}{1-(1-\mathbf{P}\left(w^{\prime }\right))z}}\right)\hfill \\ \hfill & +\sum _{\begin{array}{c}w,w^{{}^{\prime }}\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}{\displaystyle \frac{1}{1-(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))z}}.\hfill \end{array}$$ (24)

Proof. 

$i.$ We define the indicator variable ${\widehat{X}}_{n,k}^{\left(w\right)}$ as follows.

$${\widehat{X}}_{n,k}^{\left(w\right)}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}w\mathrm{is}\mathrm{a}\mathrm{prefix}\mathrm{of}\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{string}\mathrm{in}P\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

For each ${\widehat{X}}_{n,k}^{\left(w\right)}$, we have

$$\begin{array}{cc}\hfill \mathbf{E}\left[{\widehat{X}}_{n,k}^{\left(w\right)}\right]& =\mathbf{P}({\widehat{X}}_{n,k}^{\left(w\right)}=1)\hfill \\ \hfill & =1-P({\widehat{X}}_{n,k}^{\left(w\right)}=0)\hfill \\ \hfill & =1-{\left(1-\mathbf{P}\left(w\right)\right)}^n.\hfill \end{array}$$ (25)

Summing over all words w of length k, determines the generating function $\widehat{H}\left(z\right)$:

$$\begin{array}{cc}\hfill \widehat{H}\left(z\right)& =\sum _{n\ge 0}\mathbf{E}\left[{\widehat{X}}_{n,k}\right]z^n\hfill \\ \hfill & =\sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{1}{1-(1-\mathbf{P}(w\left)\right)z}}\right).\hfill \end{array}$$ (26)

$ii.$ Similar to in (18) and (20), we obtain

$$\begin{array}{cc}\hfill \mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]& =\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\mathbf{E}\left[{\widehat{X}}_{n,k}^{\left(w\right)}{\widehat{X}}_{n,k}^{\left(w^{\prime }\right)}\right]\hfill \\ \hfill & =\sum _{\begin{array}{c}w,w^{{}^{\prime }}\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left(1-{(1-\mathbf{P}\left(w\right))}^n-{(1-\mathbf{P}\left(w^{\prime }\right))}^n+{(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))}^n\right).\hfill \end{array}$$ (27)

Subsequently, we obtain the generating function below.

$$\begin{array}{cc}\hfill \widehat{G}\left(z\right)& =\sum _{n\ge 0}\mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]z^n\hfill \\ \hfill & =\sum _{\begin{array}{c}w,w^{{}^{\prime }}\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\sum _{n\ge 0}\left(1-{(1-\mathbf{P}\left(w\right))}^n-{(1-\mathbf{P}\left(w^{\prime }\right))}^n+{(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))}^n\right)z^n\hfill \\ \hfill & =\sum _{\begin{array}{c}w,w^{{}^{\prime }}\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{1}{1-(1-\mathbf{P}(w\left)\right)z}}-{\displaystyle \frac{1}{1-(1-\mathbf{P}\left(w^{\prime }\right))z}}\right)\hfill \\ \hfill & +\sum _{\begin{array}{c}w,w^{{}^{\prime }}\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}{\displaystyle \frac{1}{1-(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))z}}.\hfill \end{array}$$ (28)

□

Our first goal is to compare the coefficients of the generating functions in the two models. The coefficients are expected to be asymptotically equivalent in the desired range for k. To compare the coefficients, we need more information on the analytic properties of these generating functions. This will be discussed in Section 3.3.

3.3. Analytic Properties of the Generating Functions

Here, we turn our attention to the smallest singularities of the two generating functions given in Lemma 3. It has been shown by Jacquet and Szpankowski [21] that $D_w\left(z\right)$ has exactly one root in the disk $\left|z\right|\le \rho$. Following the notations in [21], we denote the root within the disk $\left|z\right|\le \rho$ of $D_w\left(z\right)$ by $A_w$, and by bootstrapping we obtain

$$\begin{array}{cc}\hfill & A_w=1+{\displaystyle \frac{1}{S_w\left(1\right)}}\mathbf{P}\left(w\right)+O\left(\mathbf{P}{\left(w\right)}^2\right).\hfill \end{array}$$ (29)

We also denote the derivative of $D_w\left(z\right)$ at the root $A_w$, by $B_w$, and we obtain

$$\begin{array}{c}\hfill B_w=-S_w\left(1\right)+\left(k-{\displaystyle \frac{2S_w^{\prime }\left(1\right)}{S_w\left(1\right)}}\mathbf{P}\left(w\right)\right)+O\left(\mathbf{P}{\left(w\right)}^2\right).\end{array}$$ (30)

In this paper, we will prove a similar result for the polynomial $D_{w,w^{\prime }}\left(z\right)$ through the following work.

Lemma 5.

If w and $w^{\prime }$ are two distinct binary words of length k and $\delta =\sqrt{p}$, there exists $\rho >1$, such that $\rho \delta <1$ and

$$\begin{array}{c}\hfill \sum _{w\in {\mathcal{A}}^k}\left[\right[|S_{w,w^{\prime }}{\left(\rho \right)|\le \left(\rho \delta \right)}^k\theta \left]\right]\mathbf{P}\left(w\right)\ge 1-\theta {\delta }^k.\end{array}$$ (31)

Proof. 

If the minimal degree of $S_{w,w^{\prime }}\left(z\right)$ is greater than $>\lfloor k/2\rfloor$, then

$$\begin{array}{c}\hfill |S_{w,w^{\prime }}{\left(\rho \right)|\le \left(\rho \delta \right)}^k\theta .\end{array}$$ (32)

for $\theta ={(1-p)}^{-1}$. For a fixed $w^{\prime }$, we have

$$\begin{array}{cc}\hfill & \sum _{w\in {\mathcal{A}}^k}\left[[S_{w,w^{\prime }}\left(z\right)\mathrm{has}\mathrm{minimal}\mathrm{degree}\le \lfloor k/2\rfloor ]\right]\mathbf{P}\left(w\right)\hfill \\ \hfill & =\sum _{i=1}^{\lfloor k/2\rfloor }\sum _{w\in {\mathcal{A}}^k}\left[[S_{w,w^{\prime }}\left(z\right)\mathrm{has}\mathrm{minimal}\mathrm{degree}=i]\right]\mathbf{P}\left(w\right)\hfill \\ \hfill & =\sum _{i=1}^{\lfloor k/2\rfloor }\sum _{w_1...w_i\in {\mathcal{A}}^i}\mathbf{P}(w_1...w_i)\hfill \\ \hfill & \sum _{w_{i+1}...w_k\in {\mathcal{A}}^{k-i}}\left[[S_{w,w^{\prime }}\left(z\right)\mathrm{has}\mathrm{minimal}\mathrm{degree}=i]\right]\mathbf{P}(w_{i+1}...w_k)\hfill \\ \hfill & \le \sum _{i=1}^{\lfloor k/2\rfloor }\sum _{w_1..w_i\in {\mathcal{A}}^i}\mathbf{P}(w_{i+1}...w_k)p^{k-i}\hfill \\ \hfill & =\sum _{i=1}^{\lfloor k/2\rfloor }{p}^{k-i}\sum _{w_1..w_i\in {\mathcal{A}}^i}\mathbf{P}(w_1...w_i)\hfill \\ \hfill & =\sum _{i=1}^{\lfloor k/2\rfloor }{p}^{k-i}\le {\displaystyle \frac{p^{k-\lfloor k/2\rfloor }}{1-p}}.\hfill \end{array}$$ (33)

This leads to the following

$$\begin{array}{cc}\hfill & \sum _{w\in {\mathcal{A}}^k}\left[[\mathrm{every}\mathrm{term}\mathrm{of}{S}_{w,w^{\prime }}\left(z\right)\mathrm{is}\mathrm{of}\mathrm{degree}>\lfloor k/2\rfloor ]\right]\mathbf{P}\left(w\right)\hfill \\ \hfill & =1-\sum _{w\in {\mathcal{A}}^k}\left[[S_{w,w^{\prime }}\left(z\right)\mathrm{has}\mathrm{a}\mathrm{term}\mathrm{of}\mathrm{degree}\le \lfloor k/2\rfloor ]\right]\mathbf{P}\left(w\right)\hfill \\ \hfill & \ge 1-{\displaystyle \frac{p^{\lceil k/2\rceil }}{1-p}}\ge 1-\theta {\delta }^k.\hfill \end{array}$$ (34)

□

Lemma 6.

There exist $K^{\prime }>0$, and $\rho >1$ such that $p\rho <1$, and such that, for every pair of distinct words w, and $w^{\prime }$ of length $k\ge K^{\prime }$, and for $\left|z\right|\le \rho$, we have

$$\begin{array}{c}\hfill |S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)|>0.\end{array}$$ (35)

In other words, $S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)$ does not have any roots in $\left|z\right|\le \rho$.

Proof. 

There are three cases to consider:

Case $i.$ When either $S_w\left(z\right)=1$ or $S_{w^{\prime }}\left(z\right)=1$, then every term of $S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)$ has degree k or larger, and therefore

$$\begin{array}{c}\hfill |S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)|\le k{\displaystyle \frac{{\left(p\rho \right)}^k}{1-p\rho }}.\end{array}$$ (36)

There exists $K_1>0$, such that for $k>K_1$, we have ${lim}_{k\to \infty }k{\displaystyle \frac{{\left(p\rho \right)}^k}{1-p\rho }}=0$. This yields

$$\begin{array}{cc}\hfill |S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)|& \ge |S_w\left(z\right)S_{w^{\prime }}\left(z\right)|-|S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)|\hfill \\ \hfill & \ge 1-k{\displaystyle \frac{{\left(p\rho \right)}^k}{1-p\rho }}>0.\hfill \end{array}$$ (37)

Case $ii.$ If the minimal degree for $S_w\left(z\right)-1$ or $S_{w^{\prime }}\left(z\right)-1$ is greater than $\lfloor k/2\rfloor$, then every term of $S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)$ has degree at least $k/2$. We also note that, by Lemma 9, $|S_w\left(z\right)S_{w^{\prime }}\left(z\right)|>0$. Therefore, there exists $K_2>0$, such that

$$\begin{array}{cc}\hfill |S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w^{\prime },w}\left(z\right)S_{w,w^{\prime }}\left(z\right)|& \ge |S_w\left(z\right)S_{w^{\prime }}\left(z\right)|-|S_{w^{\prime },w}\left(z\right)S_{w,w^{\prime }}\left(z\right)|\hfill \\ \hfill & >0\mathrm{for}k>K_2.\hfill \end{array}$$ (38)

Case $iii.$ The only remaining case is where the minimal degree for $S_w\left(z\right)-1$ and $S_{w^{\prime }}\left(z\right)-1$ are both less than or equal to $\lfloor k/2\rfloor$. If $w=w_1...w_k$, then $w^{\prime }=u{w}_1...w_{k-m}$, where u is a word of length $m\ge 1$. Then we have

$$\begin{array}{c}\hfill S_{w^{\prime },w}\left(z\right)=\mathbf{P}(w_{k-m+1}...w_k)z^m\left(S_w\left(z\right)-O\left({\left(pz\right)}^{k-m}\right)\right).\end{array}$$ (39)

There exists $K_3>0$, such that

$$\begin{array}{cc}\hfill |S_{w^{\prime },w}\left(z\right)|& \le {\left(p\rho \right)}^m\left(|S_w(z\right)|+O\left({\left(p\rho \right)}^{k-m}\right)\hfill \\ \hfill & ={\left(p\rho \right)}^m\left|S_w\left(z\right)\right|+O\left({\left(p\rho \right)}^k\right)\hfill \\ \hfill & <|S_w\left(z\right)|\mathrm{for}k>K_3.\hfill \end{array}$$ (40)

Similarly, we can show that there exists $K_3^{\prime }$, such that $|S_{w,w^{\prime }}\left(z\right)|<|S_{w^{\prime }}\left(z\right)|$. Therefore, for $k>K_3^{\prime }$ we have

$$\begin{array}{cc}\hfill |S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)|& \ge |S_w\left(z\right)\left|\right|S_{w^{\prime }}\left(z\right)|-|S_{w,w^{\prime }}\left(z\right)\left|\right|S_{w^{\prime },w}\left(z\right)|\hfill \\ \hfill & >|S_w\left(z\right)\left|\right|S_{w^{\prime }}\left(z\right)|-|S_w\left(z\right)\left|\right|S_{w^{\prime }}\left(z\right)|=0.\hfill \end{array}$$ (41)

We complete the proof by setting $K^{\prime }=max\{K_1,K_2,K_3,K_3^{\prime }\}$. □

Lemma 7.

There exist $K_{w,w^{\prime }}>0$ and $\rho >1$ such that $p\rho <1$, and for every word w and $w^{\prime }$ of length $k\ge K_{w,w^{\prime }}$, the polynomial

$$\begin{array}{c}{D}_{w,w^{\prime }}\left(z\right)=(1-z)(S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right))\hfill \\ \hfill +z^k\left(\mathbf{P}\left(w\right)(S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right))+\mathbf{P}\left(w^{\prime }\right)(S_w\left(z\right)-S_{w^{\prime },w}\left(z\right))\right),\end{array}$$ (42)

has exactly one root in the disk $\left|z\right|\le \rho$.

Proof. 

First note that

$$\begin{array}{cc}\hfill |S_w\left(z\right)-S_{w^{\prime },w}\left(z\right)|& \le |S_w\left(z\right)|+|S_{w^{\prime },w}\left(z\right)|\hfill \\ \hfill & \le {\displaystyle \frac{1}{1-p\rho }}+{\displaystyle \frac{p\rho }{1-p\rho }}={\displaystyle \frac{1+p\rho }{1-p\rho }}.\hfill \end{array}$$ (43)

This yields

$$\begin{array}{cc}\hfill & \left|z^k\left(\mathbf{P}\left(w\right)(S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right))+\mathbf{P}\left(w^{\prime }\right)(S_w\left(z\right)-S_{w^{\prime },w}\left(z\right))\right)\right|\hfill \\ \hfill & \le {\left(p\rho \right)}^k\left(|S_w\left(z\right)-S_{w^{\prime },w}\left(z\right)|+|S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)|\right)\hfill \\ \hfill & \le {\left(p\rho \right)}^k\left({\displaystyle \frac{2(1+p\rho )}{1-p\rho }}\right).\hfill \end{array}$$ (44)

There exist $K^{\prime }$, $K^{″}$ large enough, such that, for $k>K^{\prime }$, we have

$$\left|\right(S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)\left)\right|\ge \beta >0,$$

and for $k>K^{″}$,

$${\left(p\rho \right)}^k\left({\displaystyle \frac{2(1+p\rho )}{1-p\rho }}\right)<(\rho -1)\beta .$$

If we define $K_{w,w^{\prime }}=max\{K^{\prime },K^{″}\}$, then we have, for $k\ge K_{w,w^{\prime }}$,

$$\begin{array}{cc}\hfill {\left(p\rho \right)}^k\left({\displaystyle \frac{2(1+p\rho )}{1-p\rho }}\right)& <(\rho -1)\beta \hfill \\ \hfill & <\left|(1-z)\right(S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)\left)\right|.\hfill \end{array}$$ (45)

by Rouché’s theorem, as $(1-z)(S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right))$ has only one root in $\left|z\right|\le \rho$, then also $D_{w,w^{\prime }}\left(z\right)$ has exactly one root in $\left|z\right|\le \rho$. □

We denote the root within the disk $\left|z\right|\le \rho$ of $D_{w,w^{\prime }}\left(z\right)$ by ${\alpha }_{w,w^{\prime }}$, and by bootstrapping we obtain

$$\begin{array}{cc}\hfill & {\alpha }_{w,w^{\prime }}=1+{\displaystyle \frac{S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w\right)\hfill \\ \hfill & +{\displaystyle \frac{S_w\left(1\right)-S_{w^{\prime },w}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w^{\prime }\right)+O\left(p^{2k}\right).\hfill \end{array}$$ (46)

We also denote the derivative of $D_{w,w^{\prime }}\left(z\right)$ at the root ${\alpha }_{w,w^{\prime }}$, by ${\beta }_{w,w^{\prime }}$, and we obtain

$$\begin{array}{cc}\hfill & {\beta }_{w,w^{\prime }}=S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)-S_w\left(1\right)S_{w^{\prime }}\left(1\right)+O\left(k{p}^k\right).\hfill \end{array}$$ (47)

We will refer to these expressions in the residue analysis that we present in the next section.

3.4. Asymptotic Difference

We begin this section by the following lemmas on the autocorrelation polynomials.

Lemma 8

(Jacquet and Szpankowski, 1994). For most words w, the autocorrelation polynomial $S_w\left(z\right)$ is very close to 1, with high probably. More precisely, if w is a binary word of length k and $\delta =\sqrt{p}$, there exists $\rho >1$, such that $\rho \delta <1$ and

$$\begin{array}{c}\hfill \sum _{w\in {\mathcal{A}}^k}\left[\right[|S_w\left(\rho \right)-1|\le {\left(\rho \delta \right)}^k\theta \left]\right]\mathbf{P}\left(w\right)\ge 1-\theta {\delta }^k,\end{array}$$ (48)

where $\theta ={(1-p)}^{-1}$. We use Iverson notation

$$\left[\left[A\right]\right]=\left\{\begin{array}{cc}1\hfill & ifAholds\hfill \\ 0\hfill & otherwise\hfill \end{array}\right.$$

Lemma 9

(Jacquet and Szpankowski, 1994). There exist $K>0$ and $\rho >1$, such that $p\rho <1$, and for every binary word w with length $k\ge K$ and $\left|z\right|\le \rho$, we have

$$\begin{array}{c}\hfill |S_w\left(z\right)|>0.\end{array}$$ (49)

In other words, $S_w\left(z\right)$ does not have any roots in $\left|z\right|\le \rho$.

Lemma 10.

With high probability, for most distinct pairs $\{w,w^{\prime }\}$, the correlation polynomial $S_{w,w^{\prime }}\left(z\right)$ is very close to 0. More precisely, if w and $w^{\prime }$ are two distinct binary words of length k and $\delta =\sqrt{p}$, there exists $\rho >1$, such that $\rho \delta <1$ and

$$\begin{array}{c}\hfill \sum _{w\in {\mathcal{A}}^k}\left[\right[|S_{w,w^{\prime }}{\left(\rho \right)|\le \left(\rho \delta \right)}^k\theta \left]\right]\mathbf{P}\left(w\right)\ge 1-\theta {\delta }^k\end{array}$$ (50)

We will use the above results to prove that the expected values in the Bernoulli model and the model built over a trie are asymptotically equivalent. We now prove Theorem 1 below.

Proof of Theorem 1.

From Lemmas 3 and 4, we have

$$\begin{array}{c}\hfill H\left(z\right)=\sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{S_w\left(z\right)}{D_w\left(z\right)}}\right),\end{array}$$

and

$$\begin{array}{c}\hfill \widehat{H}\left(z\right)=\sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{1}{1-(1-\mathbf{P}(w\left)\right)z}}\right).\end{array}$$

subtracting the two generating functions, we obtain

$$\begin{array}{c}\hfill H\left(z\right)-\widehat{H}\left(z\right)=\sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{1}{1-(1-\mathbf{P}(w\left)\right)z}}-{\displaystyle \frac{S_w\left(z\right)}{D_w\left(z\right)}}\right).\end{array}$$ (51)

We define

$$\begin{array}{c}\hfill {\Delta }_w\left(z\right)={\displaystyle \frac{1}{1-(1-\mathbf{P}(w\left)\right)z}}-{\displaystyle \frac{S_w\left(z\right)}{D_w\left(z\right)}}.\end{array}$$ (52)

Therefore, by Cauchy integral formula (see [20]), we have

$$\begin{array}{cc}\hfill \left[z^n\right]{\Delta }_w\left(z\right)& ={\displaystyle \frac{1}{2\pi i}}\oint {\Delta }_w\left(z\right){\displaystyle \frac{dz}{z^{n+1}}}={\mathrm{Res}}_{z=0}{\Delta }_w\left(z\right){\displaystyle \frac{dz}{z^{n+1}}},\hfill \end{array}$$ (53)

where the path of integration is a circle about zero with counterclockwise orientation. We note that the above integrand has poles at $z=0$, $z={\displaystyle \frac{1}{1-\mathbf{P}\left(w\right)}}$, and $z=A_w$ (refer to expression (29)). Therefore, we define

$$\begin{array}{cc}\hfill I^w\left(\rho \right)& :={\displaystyle \frac{1}{2\pi i}}{\int }_{\left|z\right|=\rho }{\Delta }_w\left(z\right){\displaystyle \frac{dz}{z^{n+1}}},\hfill \end{array}$$ (54)

where the circle of radius $\rho$ contains all of the above poles. By the residue theorem, we have

$$\begin{array}{cc}\hfill I^w\left(\rho \right)& ={\mathrm{Res}}_{z=0}{\displaystyle \frac{{\Delta }_w\left(z\right)}{z^{n+1}}}+{\mathrm{Res}}_{z=A_w}{\displaystyle \frac{{\Delta }_w\left(z\right)}{z^{n+1}}}+{\mathrm{Res}}_{z=1/1-\mathbf{P}\left(w\right)}{\displaystyle \frac{{\Delta }_w\left(z\right)}{z^{n+1}}}\hfill \\ \hfill & =\left[z^n\right]{\Delta }_w\left(z\right)-{\mathrm{Res}}_{z=A_w}{\displaystyle \frac{H_w\left(z\right)}{z^{n+1}}}+{\mathrm{Res}}_{z=1/1-\mathbf{P}\left(w\right)}{\displaystyle \frac{{\widehat{H}}_w\left(z\right)}{z^{n+1}}}\hfill \end{array}$$ (55)

We observe that

$$\begin{array}{cc}\hfill & {\mathrm{Res}}_{z=A_w}{\displaystyle \frac{{\Delta }_w\left(z\right)}{z^{n+1}}}={\displaystyle \frac{S_w\left(A_w\right)}{B_w{A}_w^{n+1}}},\mathrm{where}{B}_w\mathrm{is}\mathrm{as}\mathrm{in}\left(30\right)\hfill \\ \hfill & {\mathrm{Res}}_{z=1/1-\mathbf{P}\left(w\right)}{\displaystyle \frac{{\widehat{H}}_w\left(z\right)}{z^{n+1}}}=-{(1-\mathbf{P}\left(w\right))}^{n+1}.\hfill \end{array}$$

Then we obtain

$$\begin{array}{cc}\hfill \left[z^n\right]{\Delta }_w& =I^w\left(\rho \right)-{\displaystyle \frac{S_w\left(A_w\right)}{B_w{A}_w^{n+1}}}-{(1-\mathbf{P}\left(w\right))}^{n+1},\hfill \end{array}$$ (56)

and finally, we have

$$\begin{array}{cc}\hfill \left[z^n\right](H\left(z\right)-\widehat{H}\left(z\right))& =\sum _{w\in {\mathcal{A}}^k}\left[z^n\right]{\Delta }_w\hfill \\ \hfill & =\sum _{w\in {\mathcal{A}}^k}{I}_n^w\left(\rho \right)-\sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{S_w\left(A_w\right)}{B_w{A}_w^{n+1}}}+{(1-\mathbf{P}\left(w\right))}^{n+1}\right).\hfill \end{array}$$ (57)

First, we show that, for sufficiently large n, the sum ${\sum }_{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{S_w\left(A_w\right)}{B_w{A}_w^{n+1}}}+{(1-\mathbf{P}\left(w\right))}^{n+1}\right)$ approaches zero. □

Lemma 11.

For large enough n, and for $k=\mathsf{\Theta }(logn)$, there exists $M>0$ such that

$$\begin{array}{c}\hfill \sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{S_w\left(A_w\right)}{B_w{A}_w^{n+1}}}+{(1-\mathbf{P}\left(w\right))}^{n+1}\right)=O\left(n^{-M}\right).\end{array}$$ (58)

Proof. 

We let

$$\begin{array}{cc}\hfill r_w\left(z\right)& ={(1-\mathbf{P}\left(w\right))}^z+{\displaystyle \frac{S_w\left(A_w\right)}{B_w{A}_w^z}}.\hfill \end{array}$$ (59)

The Mellin transform of the above function is

$$\begin{array}{c}\hfill r_w^{*}\left(s\right)=\mathsf{\Gamma }\left(s\right){log}^{-s}\left(\frac{1}{1-\mathbf{P}\left(w\right)}\right)-{\displaystyle \frac{S_w\left(A_w\right)}{B_w}}\mathsf{\Gamma }\left(s\right){log}^{-s}\left(A_w\right).\end{array}$$ (60)

We define

$$\begin{array}{cc}\hfill C_w={\displaystyle \frac{S_w\left(A_w\right)}{B_w}}& ={\displaystyle \frac{S_w\left(A_w\right)}{-S_w\left(1\right)+O\left(k\mathbf{P}\left(w\right)\right)}},\hfill \end{array}$$ (61)

which is negative and uniformly bounded for all w. Also, for a fixed s, we have

$$\begin{array}{cc}\hfill {ln}^{-s}\left(\frac{1}{1-\mathbf{P}\left(w\right)}\right)& ={ln}^{-s}\left(1+\mathbf{P}\left(w\right)+O\left(\mathbf{P}{\left(w\right)}^2\right)\right)\hfill \\ \hfill & ={\left(\mathbf{P}\left(w\right)+O\left(\mathbf{P}{\left(w\right)}^2\right)\right)}^{-s}\hfill \\ \hfill & =\mathbf{P}{\left(w\right)}^{-s}{\left(1+O\left(\mathbf{P}\left(w\right)\right)\right)}^{-s}\hfill \\ \hfill & =\mathbf{P}{\left(w\right)}^{-s}\left(1+O\left(\mathbf{P}\left(w\right)\right)\right),\hfill \end{array}$$ (62)

$$\begin{array}{cc}\hfill {ln}^{-s}\left(A_w\right)& ={ln}^{-s}\left(1-\left(-{\displaystyle \frac{\mathbf{P}\left(w\right)}{S_w\left(1\right)}}+O\left(\mathbf{P}{\left(w\right)}^2\right)\right)\right)\hfill \\ \hfill & ={\left({\displaystyle \frac{\mathbf{P}\left(w\right)}{S_w\left(1\right)}}+O\left(\mathbf{P}{\left(w\right)}^2\right)\right)}^{-s}\hfill \\ \hfill & ={\left({\displaystyle \frac{\mathbf{P}\left(w\right)}{S_w\left(1\right)}}\right)}^{-s}{\left(1+O\left(\mathbf{P}\left(w\right)\right)\right)}^{-s}\hfill \\ \hfill & ={\left({\displaystyle \frac{\mathbf{P}\left(w\right)}{S_w\left(1\right)}}\right)}^{-s}\left(1+O\left(\mathbf{P}\left(w\right)\right)\right),\hfill \end{array}$$ (63)

and therefore, we obtain

$$\begin{array}{cc}\hfill r_w^{*}\left(s\right)& =\mathsf{\Gamma }\left(s\right)\mathbf{P}{\left(w\right)}^{-s}\left(1-{\displaystyle \frac{1}{S_w{\left(1\right)}^{-s}}}\right)O\left(1\right).\hfill \end{array}$$ (64)

From this expression, and noticing that the function has a removable singularity at $s=0$, we can see that the Mellin transform $r_w^{*}\left(s\right)$ exists on the strip where $\Re \left(s\right)>-1$. We still need to investigate the Mellin strip for the sum ${\sum }_{w\in {\mathcal{A}}^k}{r}_w^{*}\left(s\right)$. In other words, we need to examine whether summing $r_w^{*}\left(s\right)$ over all words of length k (where k grows with n) has any effect on the analyticity of the function. We observe that

$$\begin{array}{cc}\hfill \sum _{w\in {\mathcal{A}}^k}\left|r_w^{*}\left(s\right)\right|& =\sum _{w\in {\mathcal{A}}^k}\left|\mathsf{\Gamma }\left(s\right)\mathbf{P}{\left(w\right)}^{-s}\left(1-{\displaystyle \frac{1}{S_w{\left(1\right)}^{-s}}}\right)O\left(1\right)\right|\hfill \\ \hfill & \le \left|\mathsf{\Gamma }\left(s\right)\right|\sum _{w\in {\mathcal{A}}^k}\mathbf{P}{\left(w\right)}^{-\Re \left(s\right)}\left(1-{\displaystyle \frac{1}{S_w{\left(1\right)}^{-\Re \left(s\right)}}}\right)O\left(1\right)\hfill \\ \hfill & ={\left(q^k\right)}^{-\Re \left(s\right)-1}\left|\mathsf{\Gamma }\left(s\right)\right|\sum _{w\in {\mathcal{A}}^k}\mathbf{P}\left(w\right)(1-S_w{\left(1\right)}^{\Re \left(s\right)})O\left(1\right).\hfill \end{array}$$

Lemma 8 allows us to split the above sum between the words for which $S_w\left(1\right)\le 1+O\left({\delta }^k\right)$ and words that have $S_w\left(1\right)>1+O\left({\delta }^k\right)$.

Such a split yields the following

$$\begin{array}{cc}\hfill \sum _{w\in {\mathcal{A}}^k}\left|r_w^{*}\left(s\right)\right|& ={\left(q^k\right)}^{-\Re \left(s\right)-1}\left|\mathsf{\Gamma }\left(s\right)\right|O\left({\delta }^k\right).\hfill \end{array}$$ (65)

This shows that ${\sum }_{w\in {\mathcal{A}}^k}{r}_w^{*}\left(s\right)$ is bounded above for $\Re \left(s\right)>-1$ and, therefore, it is analytic. This argument holds for $k=\mathsf{\Theta }(logn)$ as well, as ${\left(q^k\right)}^{-\Re \left(s\right)-1}$ would still be bounded above by a constant $M_{s,k}$ that depends on s and k.

We would like to approximate ${\sum }_{w\in {\mathcal{A}}^k}{r}_w^{*}\left(s\right)$ when $z\to \infty$. By the inverse Mellin transform, we have

$$\begin{array}{cc}\hfill \sum _{w\in {\mathcal{A}}^k}{r}_w\left(z\right)& ={\displaystyle \frac{1}{2\pi i}}{\int }_{c-i\infty }^{c+i\infty }\left(\sum _{w\in {\mathcal{A}}^k}{r}_w^{*}\left(s\right)\right)z^{-s}ds.\hfill \end{array}$$ (66)

We choose $c\in (-1,M)$ for a fixed $M>0$. Then by the direct mapping theorem [22], we obtain

$$\begin{array}{c}\hfill \sum _{w\in {\mathcal{A}}^k}{r}_w\left(z\right)=O\left(z^{-M}\right).\end{array}$$ (67)

and subsequently, we get

$$\begin{array}{c}\hfill \sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{S_w\left(A_w\right)}{B_w{A}_w^{n+1}}}+{(1-\mathbf{P}\left(w\right))}^{n+1}\right)=O\left(n^{-M}\right).\end{array}$$ (68)

□

We next prove the asymptotic smallness of $I_n^w\left(\rho \right)$ in (54).

Lemma 12.

Let

$$\begin{array}{cc}\hfill I_n^w\left(\rho \right)& ={\displaystyle \frac{1}{2\pi i}}{\int }_{\left|z\right|=\rho }\left({\displaystyle \frac{1}{1-(1-\mathbf{P}(w\left)\right)z}}-{\displaystyle \frac{S_w\left(z\right)}{D_w\left(z\right)}}\right){\displaystyle \frac{dz}{z^{n+1}}}.\hfill \end{array}$$ (69)

For large n and $k=\mathsf{\Theta }(logn)$, we have

$$\begin{array}{c}\hfill \sum _{w\in {\mathcal{A}}^k}{I}_n^w\left(\rho \right)=O\left({\rho }^{-n}{\left(\rho \delta \right)}^k\right).\end{array}$$ (70)

Proof. 

We observe that

$$\begin{array}{c}\hfill |I_n^w\left(\rho \right)|\le {\displaystyle \frac{1}{2\pi }}{\int }_{\left|z\right|=\rho }\left|{\displaystyle \frac{\mathbf{P}\left(w\right)z\left(z^{k-1}-S_w\left(z\right)\right)}{D_w\left(z\right)(1-(1-\mathbf{P}\left(w\right))z)}}{\displaystyle \frac{1}{z^{n+1}}}\right|dz.\end{array}$$ (71)

For $\left|z\right|=\rho$, we show that the denominator in (71) is bounded away from zero.

$$\begin{array}{cc}\hfill |D_w\left(z\right)|& =|(1-z)S_w\left(z\right)+\mathbf{P}\left(w\right)z^k|\hfill \\ \hfill & \ge |1-z||S_w\left(z\right)|-\mathbf{P}\left(w\right)|z^k|\hfill \\ \hfill & \ge (\rho -1)\alpha -{\left(p\rho \right)}^k,\mathrm{where}\alpha >0\mathrm{by}\mathrm{Lemma}9.\hfill \\ \hfill & >0,\mathrm{we}\mathrm{assume}k\mathrm{to}\mathrm{be}\mathrm{large}\mathrm{enough}\mathrm{such}\mathrm{that}{\left(p\rho \right)}^k<\alpha (\rho -1).\hfill \end{array}$$ (72)

To find a lower bound for $|1-(1-\mathbf{P}\left(w\right)\left)z\right|$, we can choose $K_w$ large enough such that

$$\begin{array}{cc}\hfill |1-(1-\mathbf{P}\left(w\right)\left)z\right|& \ge \left|1-(1-\mathbf{P}(w\left)\right)\left|z\right|\right|\hfill \\ \hfill & \ge |1-\rho (1-p^{K_w}\left)\right|\hfill \\ \hfill & >0.\hfill \end{array}$$ (73)

We now move on to finding an upper bound for the numerator in (71), for $\left|z\right|=\rho$.

$$\begin{array}{cc}\hfill |z^{k-1}-S_w\left(z\right)|& \le |S_w\left(z\right)-1|+|1-z^{k-1}|\hfill \\ \hfill & \le (S_w\left(\rho \right)-1)+(1+{\rho }^{k-1})\hfill \\ \hfill & =(S_w\left(\rho \right)-1)+O\left({\rho }^k\right).\hfill \end{array}$$ (74)

Therefore, there exists a constant $\mu >0$ such that

$$\begin{array}{cc}\hfill |I_n^w|& \le \mu \rho \mathbf{P}\left(w\right)\left((S_w\left(\rho \right)-1)+O\left({\rho }^k\right)\right){\displaystyle \frac{1}{{\rho }^{n+1}}}\hfill \\ \hfill & =O\left({\rho }^{-n}\right)\left(\mathbf{P}\left(w\right)(S_w\left(\rho \right)-1)+\mathbf{P}\left(w\right)O\left({\rho }^k\right)\right).\hfill \end{array}$$ (75)

Summing over all patterns w, and applying Lemma 8, we obtain

$$\begin{array}{cc}\hfill \sum _{w\in {\mathcal{A}}^k}\left|I_n^w\left(\rho \right)\right|& =O\left({\rho }^{-n}\right)\sum _{w\in {\mathcal{A}}^k}\mathbf{P}\left(w\right)(S_w\left(\rho \right)-1)+O\left({\rho }^{-n+k}\right)\sum _{w\in {\mathcal{A}}^k}\mathbf{P}\left(w\right)\hfill \\ \hfill & =O\left({\rho }^{-n}\right)\left(\theta {\left(\rho \delta \right)}^k+{\displaystyle \frac{p\rho }{1-p\rho }}\theta {\delta }^k\right)+O\left({\rho }^{-n+k}\right)\hfill \\ \hfill & =O\left({\rho }^{-n}{\left(\rho \delta \right)}^k\right),\hfill \end{array}$$ (76)

which approaches zero as $n\to \infty$ and $k=\mathsf{\Theta }(logn)$. This completes the proof of of Theorem 1. □

Similar to Theorem 1, we provide a proof to show that the second factorial moments of the kth Subword Complexity and the kth Prefix Complexity, have the same first order asymptotic behavior. We are now ready to state the proof of Theorem 2.

Proof of Theorem 2.

As discussed in Lemmas 3 and 4, the generating functions representing $\mathbf{E}\left[{\left(X_{n,k}\right)}_2\right]$ and $\mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]$ respectively, are

$$\begin{array}{c}\hfill G\left(z\right)=\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{S_w\left(z\right)}{D_w\left(z\right)}}-{\displaystyle \frac{S_{w^{\prime }}\left(z\right)}{D_{w^{\prime }}\left(z\right)}}+{\displaystyle \frac{S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)}{D_{w,w^{\prime }}\left(z\right)}}\right),\end{array}$$

and

$$\begin{array}{cc}\hfill \widehat{G}\left(z\right)& =\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left({\displaystyle \frac{1}{1-z}}-{\displaystyle \frac{1}{1-(1-\mathbf{P}(w\left)\right)z}}-{\displaystyle \frac{1}{1-(1-\mathbf{P}\left(w^{\prime }\right))z}}\right)\hfill \\ \hfill & +\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}{\displaystyle \frac{1}{1-(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))z}}.\hfill \end{array}$$

Note that

$$\begin{array}{cc}\hfill G\left(z\right)-\widehat{G}\left(z\right)& =\sum _{\begin{array}{c}{w}^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{1}{1-(1-\mathbf{P}(w\left)\right)z}}-{\displaystyle \frac{S_w\left(z\right)}{D_w\left(z\right)}}\right)\hfill \end{array}$$ (77)

$$\begin{array}{cc}\hfill & +\sum _{\begin{array}{c}w\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\sum _{w^{\prime }\in {\mathcal{A}}^k}\left({\displaystyle \frac{1}{1-(1-\mathbf{P}\left(w^{\prime }\right))z}}-{\displaystyle \frac{S_{w^{\prime }}\left(z\right)}{D_{w^{\prime }}\left(z\right)}}\right)\hfill \end{array}$$ (78)

$$\begin{array}{cc}\hfill & +\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left({\displaystyle \frac{1}{1-(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))z}}-{\displaystyle \frac{S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)}{D_{w,w^{\prime }}\left(z\right)}}\right)\hfill \end{array}$$ (79)

In Theorem 1, we proved that for every $M>0$ (which does not depend on n or k), we have

$$\begin{array}{cc}\hfill H\left(z\right)-\widehat{H}\left(z\right)& =\sum _{w\in {\mathcal{A}}^k}\left({\displaystyle \frac{1}{1-(1-\mathbf{P}(w\left)\right)z}}-{\displaystyle \frac{S_w\left(z\right)}{D_w\left(z\right)}}\right)=O\left(n^{-M}\right).\hfill \end{array}$$

Therefore, both (77) and (78) are of order $(2^k-1)O\left(n^{-M}\right)=O\left(n^{-M+alog2}\right)$ for $k=alogn$. Thus, to show the asymptotic smallness, it is enough to choose $M=alog2+ϵ$, where $ϵ$ is a small positive value. Now, it only remains to show (79) is asymptotically negligible as well. We define

$$\begin{array}{c}\hfill {\Delta }_{w,w^{\prime }}\left(z\right)={\displaystyle \frac{1}{1-(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))z}}-{\displaystyle \frac{S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)}{D_{w,w^{\prime }}\left(z\right)}}.\end{array}$$ (80)

Next, we extract the coefficient of $z^n$

$$\begin{array}{cc}\hfill \left[z^n\right]{\Delta }_{w,w^{\prime }}\left(z\right)& ={\displaystyle \frac{1}{2\pi i}}\oint {\Delta }_{w,w^{\prime }}\left(z\right){\displaystyle \frac{dz}{z^{n+1}}},\hfill \end{array}$$ (81)

where the path of integration is a circle about the origin with counterclockwise orientation. We define

$$\begin{array}{c}\hfill I_n^{w,w^{\prime }}\left(\rho \right)={\displaystyle \frac{1}{2\pi i}}{\int }_{\left|z\right|=\rho }{\Delta }_{w,w^{\prime }}\left(z\right){\displaystyle \frac{dz}{z^{n+1}}},\end{array}$$ (82)

The above integrand has poles at $z=0$, $z={\alpha }_{w,w^{\prime }}$ (as in (46)), and $z=\frac{1}{1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right)}$. We have chosen $\rho$ such that the poles are all inside the circle $\left|z\right|=\rho$. It follows that

$$\begin{array}{cc}\hfill I_n^{w,w^{\prime }}\left(\rho \right)& ={\mathrm{Res}}_{z=0}{\displaystyle \frac{{\Delta }_{w,w^{\prime }}\left(z\right)}{z^{n+1}}}+{\mathrm{Res}}_{z={\alpha }_{w,w^{\prime }}}{\displaystyle \frac{{\Delta }_{w,w^{\prime }}\left(z\right)}{z^{n+1}}}+{\mathrm{Res}}_{z=\frac{1}{1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right)}}{\displaystyle \frac{{\Delta }_w\left(z\right)}{z^{n+1}}},\hfill \end{array}$$ (83)

and the residues give us the following.

$$\begin{array}{c}\hfill {\mathrm{Res}}_{z=\frac{1}{1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right)}}{\displaystyle \frac{1}{1-(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))z)z^{n+1}}}=-{(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))}^{n+1},\end{array}$$

and

$$\begin{array}{cc}\hfill & {\mathrm{Res}}_{z={\alpha }_{w,w^{\prime }}}{\displaystyle \frac{S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)}{D_{w,w^{\prime }}\left(z\right)}}=\hfill \\ \hfill & {\displaystyle \frac{S_w\left({\alpha }_{w,w^{\prime }}\right)S_{w^{\prime }}\left({\alpha }_{w,w^{\prime }}\right)-S_{w,w^{\prime }}\left({\alpha }_{w,w^{\prime }}\right)S_{w^{\prime },w}\left({\alpha }_{w,w^{\prime }}\right)}{{\beta }_{w,w^{\prime }}{\alpha }_{w,w^{\prime }}^{n+1}}},\hfill \end{array}$$

where ${\beta }_{w,w^{\prime }}$ is as in (47). Therefore, we get

$$\begin{array}{cc}\hfill \sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left[z^n\right]{\Delta }_{w,w^{\prime }}\left(z\right)& =\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}{I}_n^{w,w^{\prime }}\left(\rho \right)\hfill \\ \hfill & -\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}({\displaystyle \frac{S_w\left({\alpha }_{w,w^{\prime }}\right)S_{w^{\prime }}\left({\alpha }_{w,w^{\prime }}\right)-S_{w,w^{\prime }}\left({\alpha }_{w,w^{\prime }}\right)S_{w^{\prime },w}\left({\alpha }_{w,w^{\prime }}\right)}{{\beta }_{w,w^{\prime }}{\alpha }_{w,w^{\prime }}^{n+1}}}\hfill \\ \hfill & +{(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))}^{n+1}).\hfill \end{array}$$ (84)

We now show that the above two terms are asymptotically small. □

Lemma 13.

There exists $ϵ>0$ where the sum

$$\begin{array}{c}\hfill \sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left({\displaystyle \frac{S_w\left({\alpha }_{w,w^{\prime }}\right)S_{w^{\prime }}\left({\alpha }_{w,w^{\prime }}\right)-S_{w,w^{\prime }}\left({\alpha }_{w,w^{\prime }}\right)S_{w^{\prime },w}\left({\alpha }_{w,w^{\prime }}\right)}{{\beta }_{w,w^{\prime }}{\alpha }_{w,w^{\prime }}^{n+1}}}+{(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))}^{n+1}\right)\end{array}$$

is of order O($n^{-ϵ}$).

Proof. 

We define

$$\begin{array}{cc}\hfill r_{w,w^{\prime }}\left(z\right)& ={\displaystyle \frac{S_w\left({\alpha }_{w,w^{\prime }}\right)S_{w^{\prime }}\left({\alpha }_{w,w^{\prime }}\right)-S_{w,w^{\prime }}\left({\alpha }_{w,w^{\prime }}\right)S_{w^{\prime },w}\left({\alpha }_{w,w^{\prime }}\right)}{{\beta }_{w,w^{\prime }}{\alpha }_{w,w^{\prime }}^z}}+{(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))}^z.\hfill \end{array}$$

The Mellin transform of the above function is

$$\begin{array}{c}\hfill r_{w,w^{\prime }}^{*}\left(s\right)=\mathsf{\Gamma }\left(s\right){log}^{-s}\left(\frac{1}{1-\mathbf{P}\left(w\right)-\mathbf{p}\left(w^{\prime }\right)}\right)+C_{w,w^{\prime }}\mathsf{\Gamma }\left(s\right){log}^{-s}\left({\alpha }_{w,w^{\prime }}\right),\end{array}$$ (85)

where $C_{w,w^{\prime }}={\displaystyle \frac{S_w\left({\alpha }_{w,w^{\prime }}\right)S_{w^{\prime }}\left({\alpha }_{w,w^{\prime }}\right)-S_{w,w^{\prime }}\left({\alpha }_{w,w^{\prime }}\right)S_{w^{\prime },w}\left({\alpha }_{w,w^{\prime }}\right)}{{\beta }_{w,w^{\prime }}}}$. We note that $C_{w,w^{\prime }}$ is negative and uniformly bounded from above for all $w,w^{\prime }\in {\mathcal{A}}^k$.For a fixes s, we also have,

$$\begin{array}{cc}\hfill {ln}^{-s}\left(\frac{1}{1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right)}\right)& ={ln}^{-s}\left(1+\mathbf{P}\left(w\right)+\mathbf{P}\left(w^{\prime }\right)+O\left(p^{2k}\right)\right)\hfill \\ \hfill & ={\left(\mathbf{P}\left(w\right)+\mathbf{P}\left(w^{\prime }\right)+O\left(p^{2k}\right)\right)}^{-s}\hfill \\ \hfill & ={(\mathbf{P}\left(w\right)+\mathbf{P}\left(w^{\prime }\right))}^{-s}{\left(1+O\left(p^k\right)\right)}^{-s}\hfill \\ \hfill & ={(\mathbf{P}\left(w\right)+\mathbf{P}\left(w^{\prime }\right))}^{-s}\left(1+O\left(p^k\right)\right),\hfill \end{array}$$ (86)

and

$$\begin{array}{cc}\hfill {ln}^{-s}\left({\alpha }_{w,w^{\prime }}\right)& =({\displaystyle \frac{S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w\right)\hfill \\ \hfill & +{\displaystyle \frac{S_w\left(1\right)-S_{w^{\prime },w}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w^{\prime }\right)+O\left(p^{2k}\right){)}^{-s}\hfill \\ \hfill & =({\displaystyle \frac{S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w\right)\hfill \\ \hfill & +{\displaystyle \frac{S_w\left(1\right)-S_{w^{\prime },w}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w^{\prime }\right){)}^{-s}\left(1+O\left(p^k\right)\right).\hfill \end{array}$$ (87)

Therefore, we have

$$\begin{array}{cc}\hfill r_{w,w^{\prime }}^{*}\left(s\right)& =\mathsf{\Gamma }\left(s\right){\left(\mathbf{P}\left(w\right)+\mathbf{P}\left(w^{\prime }\right)\right)}^{-s}(1+O\left(p^k\right))\hfill \\ \hfill & -\mathsf{\Gamma }\left(s\right)({\displaystyle \frac{S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w\right)\hfill \\ \hfill & +{\displaystyle \frac{S_w\left(1\right)-S_{w^{\prime },w}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w^{\prime }\right){)}^{-s}\left(1+O\left(p^k\right)\right)O\left(1\right).\hfill \end{array}$$ (88)

To find the Mellin strip for the sum ${\sum }_{w\in {\mathcal{A}}^k}{r}_{w,w^{\prime }}^{*}\left(s\right)$, we first note that

$$\begin{array}{cc}\hfill {(x+y)}^a& \le x^a+y^a,\mathrm{for}\mathrm{any}\mathrm{real}x,y>0\mathrm{and}a\le 1.\hfill \end{array}$$

Since $-\Re \left(s\right)<1$, we have

$$\begin{array}{c}\hfill {\left(\mathbf{P}\left(w\right)+\mathbf{P}\left(w^{\prime }\right)\right)}^{-\Re \left(s\right)}\le \mathbf{P}{\left(w\right)}^{-\Re \left(s\right)}+\mathbf{P}{\left(w^{\prime }\right)}^{-\Re \left(s\right)},\end{array}$$ (89)

and

$$\begin{array}{cc}\hfill & {\left({\displaystyle \frac{S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w\right)ın+{\displaystyle \frac{S_w\left(1\right)-S_{w^{\prime },w}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w^{\prime }\right)\right)}^{-\Re \left(s\right)}\hfill \\ \hfill & \le {\left({\displaystyle \frac{S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w\right)\right)}^{-\Re \left(s\right)}\hfill \\ \hfill & +{\left({\displaystyle \frac{S_w\left(1\right)-S_{w^{\prime },w}\left(1\right)}{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}}\mathbf{P}\left(w^{\prime }\right)\right)}^{-\Re \left(s\right)}.\hfill \end{array}$$ (90)

Therefore, we get

$$\begin{array}{cc}\hfill \sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left|r_{w,w^{\prime }}^{*}\left(s\right)\right|& \le \left|\mathsf{\Gamma }\right(s\left)\right|O\left(1\right)\hfill \\ \hfill & (\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\mathbf{P}{\left(w\right)}^{-\Re \left(s\right)}\left(1-{\left({\displaystyle \frac{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}{S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)}}\right)}^{\Re \left(s\right)}\right)\hfill \\ \hfill \\ \hfill & +\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\mathbf{P}{\left(w^{\prime }\right)}^{-\Re \left(s\right)}\left(1-{\left({\displaystyle \frac{S_w\left(1\right)S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)S_{w^{\prime },w}\left(1\right)}{S_w\left(1\right)-S_{w^{\prime },w}\left(1\right)}}\right)}^{\Re \left(s\right)}\right))\hfill \\ \hfill & \le {\left(q^k\right)}^{-\Re \left(s\right)-1}\left|\mathsf{\Gamma }\left(s\right)\right|O\left(1\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & (\sum _{\begin{array}{c}{w}^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\sum _{w\in {\mathcal{A}}^k}\mathbf{P}\left(w\right)\left(1-{\left(S_w\left(1\right)\right)}^{\Re \left(s\right)}{\left(1-{\displaystyle \frac{S_{w,w^{\prime }}\left(1\right)}{S_{w^{\prime }}\left(1\right)}}\right)}^{-\Re \left(s\right)}\right)\hfill \end{array}$$ (91)

$$\begin{array}{cc}\hfill & +\sum _{\begin{array}{c}{w}^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\sum _{w\in {\mathcal{A}}^k}\mathbf{P}\left(w\right)S_{w,w^{\prime }}{\left(1\right)}^{\Re \left(s\right)}{\left({\displaystyle \frac{S_{w^{\prime }}\left(1\right)-S_{w,w^{\prime }}\left(1\right)}{S_{w^{\prime },w}\left(1\right)}}\right)}^{-\Re \left(s\right)}\hfill \end{array}$$ (92)

$$\begin{array}{cc}\hfill & +\sum _{\begin{array}{c}w\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\sum _{w^{\prime }\in {\mathcal{A}}^k}\mathbf{P}\left(w^{\prime }\right)\left(1-{\left(S_{w^{\prime }}\left(1\right)\right)}^{\Re \left(s\right)}{\left(1-{\displaystyle \frac{S_{w^{\prime },w}\left(1\right)}{S_w\left(1\right)}}\right)}^{-\Re \left(s\right)}\right)\hfill \end{array}$$ (93)

$$\begin{array}{cc}\hfill & +\sum _{\begin{array}{c}w\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\sum _{w^{\prime }\in {\mathcal{A}}^k}\mathbf{P}\left(w^{\prime }\right)S_{w^{\prime },w}{\left(1\right)}^{\Re \left(s\right)}{\left({\displaystyle \frac{S_w\left(1\right)-S_{w^{\prime },w}\left(1\right)}{S_{w,w^{\prime }}\left(1\right)}}\right)}^{-\Re \left(s\right)}).\hfill \end{array}$$ (94)

By Lemma 10, with high probability, a randomly selected w has the property $S_{w,w^{\prime }}\left(1\right)=O\left({\delta }^k\right)$, and thus

$${\left(1-{\displaystyle \frac{S_{w,w^{\prime }}\left(1\right)}{S_{w^{\prime }}\left(1\right)}}\right)}^{-\Re \left(s\right)}=1+O\left({\delta }^k\right).$$

With that and by Lemma 8, for most words w,

$$1-S_w{\left(1\right)}^{\Re \left(s\right)}(1+O\left({\delta }^k\right))=O\left({\delta }^k\right).$$

Therefore, both sums (91) and (93) are of the form $(2^k-1)O\left({\delta }^k\right)$. The sums (92) and (94) are also of order $(2^k-1)O\left({\delta }^k\right)$ by Lemma 10. Combining all these terms we will obtain

$$\begin{array}{cc}\hfill \sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left|r_{w,w^{\prime }}^{*}\left(s\right)\right|& \le (2^k-1){\left(q^k\right)}^{-\Re \left(s\right)-1}\left|\mathsf{\Gamma }\left(s\right)\right|O\left({\delta }^k\right)O\left(1\right).\hfill \end{array}$$ (95)

By the inverse Mellin transform, for $k=alogn$, $M=alog2+ϵ$ and $c\in (-1,M)$, we have

$$\begin{array}{cc}\hfill \sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}{r}_{w,w^{\prime }}\left(z\right)& ={\displaystyle \frac{1}{2\pi i}}{\int }_{c-i\infty }^{c+i\infty }\left(\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}{r}_{w,w^{\prime }}^{*}\left(s\right)\right)z^{-s}ds=O\left(z^{-M}\right)O\left(2^k\right)\hfill \\ \hfill & =O\left(z^{-ϵ}\right).\hfill \end{array}$$ (96)

□

In the following lemma we show that the first term in (85) is asymptotically small.

Lemma 14.

Recall that

$$\begin{array}{cc}\hfill I_n^{w,w^{\prime }}\left(\rho \right)& ={\displaystyle \frac{1}{2\pi i}}{\int }_{\left|z\right|=\rho }{\Delta }_{w,w^{\prime }}\left(z\right){\displaystyle \frac{dz}{z^{n+1}}}.\hfill \end{array}$$

We have

$$\begin{array}{cc}\hfill \sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}{I}_n^{w,w^{\prime }}\left(\rho \right)& =O\left({\rho }^{-n+2k}{\delta }^k\right).\hfill \end{array}$$ (97)

Proof. 

First note that

$$\begin{array}{cc}\hfill {\Delta }_{w,w^{\prime }}\left(z\right)& ={\displaystyle \frac{1}{1-(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))z}}-{\displaystyle \frac{S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)}{D_{w,w^{\prime }}\left(z\right)}}\hfill \\ \hfill & ={\displaystyle \frac{z\mathbf{P}\left(w\right)\left(S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)-S_w\left(z\right)S_{w^{\prime }}\left(z\right)+z^{k-1}{S}_{w^{\prime }}\left(z\right)-z^{k-1}{S}_{w,w^{\prime }}\left(z\right)\right)}{\left(1-(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))z\right)D_{w,w^{\prime }}\left(z\right)}}\hfill \\ \hfill & +{\displaystyle \frac{z\mathbf{P}\left(w^{\prime }\right)\left(S_{w^{\prime },w}\left(z\right)S_{w,w^{\prime }}\left(z\right)-S_{w^{\prime }}\left(z\right)S_w\left(z\right)+z^{k-1}{S}_w\left(z\right)-z^{k-1}{S}_{w^{\prime },w}\left(z\right)\right)}{\left(1-(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))z\right)D_{w,w^{\prime }}\left(z\right)}}.\hfill \end{array}$$ (98)

We saw in (73) that $|1-(1-\mathbf{P}\left(w^{\prime }\right))z|\ge c_2$, and therefore, it follows that

$$\begin{array}{c}\hfill |1-(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))z|\ge c_1\end{array}$$ (99)

For $z=\rho$, $|D_{w,w^{\prime }}\left(z\right)|$ is also bounded below as the following

$$\begin{array}{cc}\hfill |D_{w,w^{\prime }}\left(z\right)|& =\left|(1-z)\right(S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right))\hfill \\ \hfill & +z^k\left(\mathbf{P}\left(w\right)(S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right))+\mathbf{P}\left(w^{\prime }\right)(S_w\left(z\right)-S_{w^{\prime },w}\left(z\right))\right)|\hfill \\ \hfill & \ge \left|(1-z)\right(S_w\left(z\right)S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)\left)\right|\hfill \\ \hfill & -\left|z^k\right|\left|\left(\mathbf{P}\left(w\right)(S_{w^{\prime }}\left(z\right)-S_{w,w^{\prime }}\left(z\right))+\mathbf{P}\left(w^{\prime }\right)(S_w\left(z\right)-S_{w^{\prime },w}\left(z\right))\right)\right|\hfill \\ \hfill & \ge (\rho -1)\beta -{\left(p\rho \right)}^k\left({\displaystyle \frac{2(1+p\rho )}{1-p\rho }}\right),\hfill \end{array}$$ (100)

which is bounded away from zero by the assumption of Lemma 7. Additionally, we show that the numerator in (98) is bounded above, as follows

$$\begin{array}{cc}\hfill & |S_{w,w^{\prime }}\left(z\right)S_{w^{\prime },w}\left(z\right)-S_w\left(z\right)S_{w^{\prime }}\left(z\right)+z^{k-1}{S}_{w^{\prime }}\left(z\right)-z^{k-1}{S}_{w,w^{\prime }}\left(z\right)|\le \hfill \\ \hfill & |S_{w^{\prime }}\left(z\right)(z^{k-1}-S_w\left(z\right))|+|S_{w,w^{\prime }}\left(z\right)(S_{w^{\prime },w}\left(z\right)-z^{k-1})|\hfill \\ \hfill & \le S_{w^{\prime }}\left(\rho \right)\left((S_w\left(\rho \right)-1)+O\left({\rho }^k\right)\right)+S_{w,w^{\prime }}\left(\rho \right)\left(S_{w^{\prime },w}\left(\rho \right)+O\left({\rho }^k\right)\right).\hfill \end{array}$$ (101)

This yields

$$\begin{array}{cc}\hfill \sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left|I_n^{w,w^{\prime }}\right|& \le O\left({\rho }^{-n}\right)\sum _{\begin{array}{c}{w}^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}{S}_{w^{\prime }}\left(\rho \right)\sum _{w\in {\mathcal{A}}^k}\mathbf{P}\left(w\right)\left((S_w\left(\rho \right)-1)+O\left({\rho }^k\right)\right)\hfill \\ \hfill & +O\left({\rho }^{-n}\right)\sum _{\begin{array}{c}{w}^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\sum _{w\in {\mathcal{A}}^k}\mathbf{P}\left(w\right)S_{w,w^{\prime }}\left(\rho \right)\left(S_{w^{\prime },w}\left(\rho \right)+O\left({\rho }^k\right)\right).\hfill \end{array}$$ (102)

By (75), the first term above is of order $(2^k-1)O\left({\rho }^{-n+k}\right)$ and by Lemma 10 and an analysis similar to (75), the second term yields $(2^k-1)O\left({\rho }^{-n+k}\right)$ as well. Finally, we have

$$\begin{array}{cc}\hfill \sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left|I_n^{w,w^{\prime }}\right|& \le O\left({\rho }^{-n+2k}{\delta }^k\right).\hfill \end{array}$$

Which goes to zero asymptotically, for $k=\mathsf{\Theta }(logn)$. □

This lemma completes our proof of Theorem 2.

3.5. Asymptotic Analysis of the kth Prefix Complexity

We finally proceed to analyzing the asymptotic moments of the kth Prefix Complexity. The results obtained hold true for the moments of the kth Subword Complexity. Our methodology involves poissonization, saddle point analysis (the complex version of Laplace’s method [23]), and depoissonization.

Lemma 15

(Jacquet and Szpankowski, 1998). Let $\tilde{G}\left(z\right)$ be the Poisson transform of a sequence $g_n$. If $\tilde{G}\left(z\right)$ is analytic in a linear cone $S_{\theta }$ with $\theta <\pi /2$, and if the following two conditions hold:

(I) For $z\in S_{\theta }$ and real values B, $r>0$, ν

$$\begin{array}{c}\hfill \left|z\right|>r\to |\tilde{G}\left(z\right)|\le B|z^{\nu }\left|\mathsf{\Psi }\right(\left|z\right|),\end{array}$$ (103)

where $\mathsf{\Psi }\left(x\right)$ is such that, for fixed t, ${lim}_{x\to \infty }{\displaystyle \frac{\mathsf{\Psi }\left(tx\right)}{\mathsf{\Psi }\left(x\right)}}=1$;

(II) For $z\notin S_{\theta }$ and $A,\alpha <1$

$$\begin{array}{c}\hfill \left|z\right|>r\to |\tilde{G}\left(z\right)e^z|\le A{e}^{\alpha \left|z\right|}.\end{array}$$ (104)

Then, for every non-negative integer n, we have

$$g_n=\tilde{G}\left(n\right)+O\left(n^{\nu -1}\mathsf{\Psi }\left(n\right)\right).$$

On the Expected Value: To transform the sequence of interest, ${\left(\mathbf{E}\left[{\widehat{X}}_{n,k}\right]\right)}_{n\ge 0}$, into a Poisson model, we recall that in (25) we found

$$\begin{array}{cc}\hfill \mathbf{E}\left[{\widehat{X}}_{n,k}\right]& =\sum _{w\in {\mathcal{A}}^k}\left(1-{\left(1-\mathbf{P}\left(w\right)\right)}^n\right).\hfill \end{array}$$

Thus, the Poisson transform is

$$\begin{array}{cc}\hfill {\tilde{E}}_k\left(z\right)& =\sum _{n=0}^{\infty }\mathbf{E}\left[{\widehat{X}}_{n,k}\right]{\displaystyle \frac{z^n}{n!}}{e}^{-z}\hfill \\ \hfill & =\sum _{n=0}^{\infty }\sum _{w\in {\mathcal{A}}^k}\left(1-{(1-\mathbf{P}\left(w\right))}^n\right){\displaystyle \frac{z^n}{n!}}{e}^{-z}\hfill \\ \hfill & =\sum _{w\in {\mathcal{A}}^k}\left(1-e^{-z\mathbf{P}\left(w\right)}\right).\hfill \end{array}$$ (105)

To asymptotically evaluate this harmonic sum, we turn our attention to the Mellin Transform once more. The Mellin transform of ${\tilde{E}}_k\left(z\right)$ is

$$\begin{array}{cc}\hfill {\tilde{E}}_k^{*}\left(s\right)& =-\mathsf{\Gamma }\left(s\right)\sum _{w\in {\mathcal{A}}^k}P{\left(w\right)}^{-s}\hfill \\ \hfill & =-\mathsf{\Gamma }\left(s\right){(p^{-s}+q^{-s})}^k,\hfill \end{array}$$ (106)

which has the fundamental strip $s\in \langle -1,0\rangle$. For $c\in (-1,0)$, the inverse Mellin integral is the following

$$\begin{array}{cc}\hfill {\tilde{E}}_k\left(z\right)& ={\displaystyle \frac{1}{2\pi i}}{\int }_{c-i\infty }^{c+i\infty }{\tilde{E}}_k^{*}\left(s\right)·z^{-s}ds\hfill \\ \hfill & ={\displaystyle \frac{-1}{2\pi i}}{\int }_{c-i\infty }^{c+i\infty }{z}^{-s}\mathsf{\Gamma }\left(s\right){(p^{-s}+q^{-s})}^{k}ds\hfill \\ \hfill & ={\displaystyle \frac{-1}{2\pi i}}{\int }_{c-i\infty }^{c+i\infty }\mathsf{\Gamma }\left(s\right)e^{-k(s\frac{logz}{k}-log(p^{-s}+q^{-s}))}ds\hfill \\ \hfill & ={\displaystyle \frac{-1}{2\pi i}}{\int }_{c-i\infty }^{c+i\infty }\mathsf{\Gamma }\left(s\right)e^{-kh\left(s\right)}ds,\hfill \end{array}$$ (107)

where we define $h\left(s\right)={\displaystyle \frac{s}{a}}-log(p^{-s}+q^{-s})$ for $k=alogz$. We emphasize that the above integral involves k, and k grows with n. We evaluate the integral through the saddle point analysis. Therefore, we choose the line of integration to cross the saddle point $r_0$. To find the saddle point $r_0$, we let $h^{\prime }\left(r_0\right)=0$, and we obtain

$$\begin{array}{c}\hfill {\left(p/q\right)}^{-r_0}={\displaystyle \frac{alog{p}^{-1}-1}{1-alog{q}^{-1}}},\end{array}$$ (108)

and therefore,

$$\begin{array}{cc}\hfill & r_0={\displaystyle \frac{-1}{logp/q}}log\left({\displaystyle \frac{alog{q}^{-1}-1}{1-alog{p}^{-1}}}\right),\hfill \end{array}$$ (109)

where ${\displaystyle \frac{1}{log{q}^{-1}}}<a<{\displaystyle \frac{1}{log{p}^{-1}}}$.

By (108) and the fact that ${\left(p/q\right)}^{i{t}_j}=1$ for $t_j={\displaystyle \frac{2\pi j}{logp/q}}$ and $j\in \mathbb{Z}$, we can see that there are actually infinitely many saddle points $z_j$ of the form $r_0+i{t}_j$ on the line of integration.

We remark that the location of $r_0$ depends on the value of a. We have $r_0\to \infty$ as $a\to {\displaystyle \frac{1}{log{q}^{-1}}}$, and $r_0\to -\infty$ as $a\to {\displaystyle \frac{1}{log{p}^{-1}}}$. We divide the analysis into three parts, for the three ranges $r_0\in (0,\infty )$, $r_0\in (-1,0)$, and $r_0\in (-\infty ,-1)$.

In the first range, which corresponds to

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{log{q}^{-1}}}<a<{\displaystyle \frac{2}{log{q}^{-1}+log{p}^{-1}}},\end{array}$$ (110)

we perform a residue analysis, taking into account the dominant pole at $s=-1$. In the second range, we have

$$\begin{array}{c}\hfill {\displaystyle \frac{2}{log{q}^{-1}+log{p}^{-1}}}<a<{\displaystyle \frac{1}{qlog{q}^{-1}+plog{p}^{-1}}},\end{array}$$ (111)

and we get the asymptotic result through the saddle point method. The last range corresponds to

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{qlog{q}^{-1}+plog{p}^{-1}}}<a<{\displaystyle \frac{1}{log{p}^{-1}}},\end{array}$$ (112)

and we approach it with a combination of residue analysis at $s=0$, and the saddle point method. We now proceed by stating the proof of Theorem 3.

Proof of Theorem 3.

We begin with proving part $ii$ which requires a saddle point analysis. We rewrite the inverse Mellin transform with integration line at $\Re \left(s\right)=r_0$ as

$$\begin{array}{cc}\hfill {\tilde{E}}_k\left(z\right)& ={\displaystyle \frac{-1}{2\pi }}{\int }_{-\infty }^{\infty }{z}^{-(r_0+it)}\mathsf{\Gamma }(r_0+it){(p^{-(r_0+it)}+q^{-(r_0+it)})}^{k}dt\hfill \\ \hfill & ={\displaystyle \frac{-1}{2\pi }}{\int }_{-\infty }^{\infty }\mathsf{\Gamma }(r_0+it)e^{-k((r_0+it)\frac{logz}{k}-log(p^{-(r_0+it)}+q^{-(r_0+it)}))}dt.\hfill \end{array}$$ (113)

Step one: Saddle points’ contribute to the integral estimation

First, we are able to show those saddle points with $|t_j|>\sqrt{logn}$ do not have a significant asymptotic contribution to the integral. To show this, we let

$$\begin{array}{cc}\hfill T_k\left(z\right)& ={\int }_{\left|t\right|>\sqrt{logn}}{z}^{-r_0-it}\mathsf{\Gamma }(r_0+it){(p^{-r_0-it}+q^{-r_0-it})}^{k}dt.\hfill \end{array}$$ (114)

Since $|\mathsf{\Gamma }(r_0+it){|=O(\left|t\right|}^{r_0-\frac{1}{2}}{e}^{\frac{-\pi \left|t\right|}{2}})$ as $\left|t\right|\to \pm \infty$, we observe that

$$\begin{array}{cc}\hfill T_k\left(z\right)& =O\left(z^{-r_0}{(p^{-r_0}+q^{-r_0})}^k{\int }_{\sqrt{logn}}^{\infty }{t}^{r_0/2-1/2}{e}^{-\pi t/2}dt\right)\hfill \\ \hfill & =O\left(z^{-r_0}{(p^{-r_0}+q^{-r_0})}^k{(logn)}^{r_0/4-1/4}{\int }_{\sqrt{logn}}^{\infty }{e}^{-\pi t/2}dt\right)\hfill \\ \hfill & =O\left(z^{-r_0}{(p^{-r_0}+q^{-r_0})}^k{(logn)}^{r_0/4-1/4}{e}^{-\pi \sqrt{logn}/2}\right)\hfill \\ \hfill & =O\left({(logn)}^{r_0/4-1/4}{e}^{-\pi \sqrt{logn}/2}\right),\hfill \end{array}$$ (115)

which is very small for large n. Note that for $t\in (\sqrt{logn},\infty )$, $t^{r_0/2-1/2}$ is decreasing, and bounded above by ${(logn)}^{r_0/4-1/4}$.

Step two: Partitioning the integral

There are now only finitely many saddle points to work with. We split the integral range into sub-intervals, each of which contains exactly one saddle point. This way, each integral has a contour traversing a single saddle point, and we will be able to estimate the dominant contribution in each integral from a small neighborhood around the saddle point. Assuming that $j^{*}$ is the largest j for which ${\displaystyle \frac{2\pi j}{logp/q}}\le \sqrt{logn}$, we split the integral ${\tilde{E}}_k\left(z\right)$ as following

$$\begin{array}{cc}\hfill {\tilde{E}}_k\left(z\right)& =-{\displaystyle \frac{1}{2\pi }}\left(\sum _{\left|j\right|<j^{*}}{\int }_{|t-t_j|\le \frac{\pi }{logp/q}}{z}^{-r_0+it}\mathsf{\Gamma }(r_0+it){(p^{-r_0-it}+q^{-r_0-it})}^{k}dt\right)\hfill \\ \hfill & -{\displaystyle \frac{1}{2\pi }}{\int }_{\frac{\pi }{logp/q}\le \left|t_j^{*}\right|<\sqrt{logn}}\mathsf{\Gamma }(r+it)z^{-r_0+it}{(p^{-r_0-it}+q^{-r_0-it})}^{k}dt.\hfill \end{array}$$ (116)

By the same argument as in (115), the second term in (116) is also asymptotically negligible. Therefore, we are only left with

$$\begin{array}{cc}\hfill {\tilde{E}}_k\left(z\right)& =\sum _{\left|j\right|<j^{*}}{S}_j\left(z\right),\hfill \end{array}$$ (117)

where $S_j\left(z\right)=-{\displaystyle \frac{1}{2\pi }}{\int }_{|t-t_j|\le \frac{\pi }{logp/q}}{z}^{-r_0+it}\mathsf{\Gamma }(r_0+it){(p^{-r_0-it}+q^{-r_0-it})}^{k}dt)$.

Step three: Splitting the saddle contour

For each integral $S_j$, we write the expansion of $h\left(t\right)$ about $t_j$, as follows

$$\begin{array}{cc}\hfill h\left(t\right)& =h\left(t_j\right)+{\displaystyle \frac{1}{2}}{h}^{{}^{″}}\left(t_j\right){(t-t_j)}^2+O\left({(t-t_j)}^3\right).\hfill \end{array}$$ (118)

The main contribution for the integral estimate should come from an small integration path that reduces $kh\left(t\right)$ to its quadratic expansion about $t_j$. In other words, we want the integration path to be such that

$$\begin{array}{c}\hfill k{(t-t_j)}^2\to \infty ,\mathrm{and}k{(t-t_j)}^3\to 0.\end{array}$$ (119)

The above conditions are true when $|t-t_j|\gg k^{-1/2}$ and $|t-tj|\ll k^{-1/3}$. Thus, we choose the integration path to be $|t-t_j|\le k^{-2/5}$. Therefore, we have

$$\begin{array}{cc}\hfill S_j\left(z\right)& =-{\displaystyle \frac{1}{2\pi }}{\int }_{|t-t_j|\le k^{-2/5}}{z}^{-r_0+it}\mathsf{\Gamma }(r_0+it){(p^{-r_0-it}+q^{-r_0-it})}^{k}dt\hfill \\ \hfill & -{\displaystyle \frac{1}{2\pi }}{\int }_{k^{-2/5}<|t-t_j|<\frac{\pi }{logp/q}}{z}^{-r_0+it}\mathsf{\Gamma }(r_0+it){(p^{-r_0-it}+q^{-r_0-it})}^{k}dt.\hfill \end{array}$$ (120)

Saddle Tails Pruning.

We show that the integral is small for $k^{-2/5}<|t-t_j|<{\displaystyle \frac{\pi }{logp/q}}$. We define

$$\begin{array}{c}\hfill S_j^{\left(1\right)}\left(z\right)=-\frac{1}{2\pi }{\int }_{k^{-2/5}<|t-t_j|<\frac{\pi }{logp/q}}{z}^{-r_0+it}\mathsf{\Gamma }(r_0+it){(p^{-r_0-it}+q^{-r_0-it})}^{k}dt.\end{array}$$ (121)

Note that for $|t-t_j|\le {\displaystyle \frac{\pi }{logp/q}}$, we have

$$\begin{array}{cc}\hfill |p^{-r_0-it}+q^{-r_0-it}|& =(p^{-r_0}+q^{-r_0})\sqrt{1-{\displaystyle \frac{2p^{-r_0}{q}^{-r_0}}{{(p^{-r_0}+q^{-r_0})}^2}}(1-cos\left(tlogp/q)\right)}\hfill \\ \hfill & \le (p^{-r_0}+q^{-r_0})\left(1-{\displaystyle \frac{p^{-r_0}{q}^{-r_0}}{{(p^{-r_0}+q^{-r_0})}^2}}(1-cos\left(t-t_j)logp/q\right)\right)\hfill \\ \hfill & \sin \mathrm{ce}\sqrt[]{1-x}\le 1-{\displaystyle \frac{x}{2}}\mathrm{for}x\in [0,1]\hfill \\ \hfill & \le (p^{-r_0}+q^{-r_0})\left(1-{\displaystyle \frac{2p^{-r_0}{q}^{-r_0}}{{\pi }^2{(p^{-r_0}+q^{-r_0})}^2}}{((t-t_j)logp/q)}^2\right)\hfill \\ \hfill & \sin \mathrm{ce}1-cosx\ge {\displaystyle \frac{2x^2}{{\pi }^2}}\mathrm{for}\left|x\right|\le \pi \hfill \\ \hfill & \le (p^{-r_0}+q^{-r_0})e^{-\gamma {(t-t_j)}^2},\hfill \end{array}$$ (122)

where $\gamma ={\displaystyle \frac{2p^{-r_0}{q}^{-r_0}{log}^{2}p/q}{{\pi }^2{(p^{-r_0}+q^{-r_0})}^2}}$. Thus,

$$\begin{array}{cc}\hfill S_j^{\left(1\right)}\left(z\right)& =O\left(z^{-r_0}|\mathsf{\Gamma }(r_0+it)|{\int }_{k^{-2/5}<|t-t_j|<\frac{\pi }{logp/q}}|p^{-r_0-it}+q^{-r_0-it}|dt\right)\hfill \\ \hfill & =O\left(z^{-r_0}{(p^{-r_0}+q^{-r_0})}^k{\int }_{k^{-2/5}}^{\infty }{e}^{-\gamma k{u}^2}du\right)\hfill \\ \hfill & =O\left(z^{-r_0}{(p^{-r_0}+q^{-r_0})}^k{k}^{-3/5}{e}^{-\gamma k^{1/5}}\right),\sin \mathrm{ce}\mathrm{erf}\left(x\right)=O\left(e^{-x^2}/x\right).\hfill \end{array}$$ (123)

Central Approximation.

Over the main path, the integrals are of the form

$$\begin{array}{cc}\hfill S_j^{\left(0\right)}\left(z\right)& =-{\displaystyle \frac{1}{2\pi }}{\int }_{|t-t_j|\le k^{-2/5}}\mathsf{\Gamma }(r_0+it)z^{-r_0+it}{(p^{-r_0-it}+q^{-r_0-it})}^{k}dt\hfill \\ \hfill & =-{\displaystyle \frac{1}{2\pi }}{\int }_{|t-t_j|\le k^{-2/5}}\mathsf{\Gamma }(r_0+it)e^{-kh\left(t\right)}dt.\hfill \end{array}$$

We have

$$\begin{array}{c}\hfill h^{{}^{\prime \prime }}\left(t_j\right)={\displaystyle \frac{{log}^{2}p/q}{{({(p/q)}^{-r_0/2}+{(p/q)}^{r_0/2})}^2}},\end{array}$$ (124)

and

$$\begin{array}{c}\hfill p^{-r_0-i{t}_j}+q^{-r_0-i{t}_j}=p^{-i{t}_j}(p^{-r_0}+q^{-r_0}).\end{array}$$ (125)

Therefore, by Laplace’s theorem (refer to [22]) we obtain

$$\begin{array}{cc}\hfill S_j^{\left(0\right)}\left(z\right)& ={\displaystyle \frac{1}{\sqrt{2\pi k{h}^{{}^{\prime \prime }}\left(t_j\right)}}}\mathsf{\Gamma }(r_0+i{t}_j)e^{-kh\left(t_j\right)}(1+O\left(k^{-1/2}\right))\hfill \\ \hfill & ={\displaystyle \frac{{(p/q)}^{-r_0/2}+{(p/q)}^{r_0/2}}{\sqrt[]{2\pi }logp/q}}\hfill \\ \hfill & \times z^{-r_0}{(p^{-r_0}+q^{-r_0})}^k\mathsf{\Gamma }(r_0+i{t}_j)z^{-i{t}_j}{p}^{-ik{t}_j}{k}^{-1/2}\left(1+O\left({\displaystyle \frac{1}{\sqrt{k}}}\right)\right).\hfill \end{array}$$ (126)

We finally sum over all j$\left(\right|j|<j^{*})$, and we get

$$\begin{array}{cc}\hfill {\tilde{E}}_k\left(z\right)& ={\displaystyle \frac{{(p/q)}^{-r_0/2}+{(p/q)}^{r_0/2}}{\sqrt[]{2\pi }logp/q}}\hfill \\ \hfill & \times \sum _{\left|j\right|<j^{*}}{z}^{-r_0}{(p^{-r_0}+q^{-r_0})}^k\mathsf{\Gamma }(r_0+i{t}_j)z^{-i{t}_j}{p}^{-ik{t}_j}{k}^{-1/2}\left(1+O\left({\displaystyle \frac{1}{\sqrt{k}}}\right)\right).\hfill \end{array}$$ (127)

We can rewrite ${\tilde{E}}_k\left(z\right)$ as

$$\begin{array}{cc}\hfill {\tilde{E}}_k\left(z\right)& ={\mathsf{\Phi }}_1\left((1+alogp){log}_{p/q}n\right){\displaystyle \frac{z^{\nu }}{\sqrt{logn}}}\left(1+O\left({\displaystyle \frac{1}{\sqrt{logn}}}\right)\right),\hfill \end{array}$$ (128)

where $\nu =-r_0+alog(p^{-r_0}+q^{-r_0})$, and

$$\begin{array}{cc}\hfill {\mathsf{\Phi }}_1\left(x\right)& ={\displaystyle \frac{{(p/q)}^{-r_0/2}+{(p/q)}^{r_0/2}}{\sqrt[]{2a\pi }logp/q}}\sum _{\left|j\right|<j^{*}}\mathsf{\Gamma }(r_0+i{t}_j)e^{-2\pi ijx}.\hfill \end{array}$$ (129)

For part $ii$, we move the line of integration to $r_0\in (0,\infty )$. Note that in this range, we must consider the contribution of the pole at $s=0$. We have

$$\begin{array}{cc}\hfill {\tilde{E}}_k\left(z\right)& ={\mathrm{Res}}_{s=0}{\tilde{E}}_k^{*}\left(s\right)z^{-s}+{\int }_{r_0-i\infty }^{r_0+i\infty }{\tilde{E}}_k^{*}\left(z\right)z^{-s}ds.\hfill \end{array}$$ (130)

Computing the residue at $s=0$, and following the same analysis as in part i for the above integral, we arrive at

$$\begin{array}{cc}\hfill {\tilde{E}}_k\left(z\right)& =2^k-{\mathsf{\Phi }}_1\left((1+alogp){log}_{p/q}n\right){\displaystyle \frac{z^{\nu }}{\sqrt{logn}}}\left(1+O\left({\displaystyle \frac{1}{\sqrt{logn}}}\right)\right).\hfill \end{array}$$ (131)

For part $iii.$ of Theorem 3, we shift the line of integration to $c_0\in (-2,-1)$, then we have

$$\begin{array}{cc}\hfill {\tilde{E}}_k\left(z\right)& ={\mathrm{Res}}_{s=-1}{\tilde{E}}_k^{*}\left(s\right)z^{-s}+{\int }_{c-i\infty }^{c+i\infty }{\tilde{E}}_k^{*}\left(z\right)z^{-s}ds\hfill \\ \hfill & =z+O\left(z^{-c_0}{(p^{-c_0}+q^{-c_0})}^k\right)\hfill \\ \hfill & =z^{alog2}+O\left(z^{{\nu }_0}\right),\hfill \end{array}$$ (132)

where ${\nu }_0=-c_0+alog(p^{-c_0}+q^{-c_0})<1$.

Step four: Asymptotic depoissonization

To show that both conditions in (15) hold for ${\tilde{E}}_k\left(z\right)$, we extend the real values z to complex values $z=n{e}^{i\theta }$, where $\left|\theta \right|<\pi /2$. To prove (103), we note that

$$\begin{array}{c}\hfill |e^{-i\theta (r_0+it)}\mathsf{\Gamma }(r_0+it){|=O(\left|t\right|}^{r_0-1/2}{e}^{t\theta -\pi \left|t\right|/2}),\end{array}$$ (133)

and therefore

$$\begin{array}{cc}\hfill {\tilde{E}}_k\left(n{e}^{i\theta }\right)& ={\displaystyle \frac{1}{2\pi }}{\int }_{-\infty }^{\infty }{e}^{-i\theta (r_0+it)}{n}^{-r_0-it}\mathsf{\Gamma }(r_0+it){(p^{-r_0-it}+q^{-r_0-it})}^{k}dt\hfill \end{array}$$ (134)

is absolutely convergent for $\left|\theta \right|<\pi /2$. The same saddle point analysis applies here and we obtain

$$\begin{array}{cc}\hfill |{\tilde{E}}_k\left(z\right)|& \le B{\displaystyle \frac{|z^{\nu }|}{\sqrt{logn}}},\hfill \end{array}$$ (135)

where $B=|{\mathsf{\Phi }}_1\left((1+alogp){log}_{p/q}n\right)|$, and $\nu$ is as in (128). Condition (103) is therefore satisfied. To prove condition (104) We see that for a fixed k,

$$\begin{array}{c}\hfill |{\tilde{E}}_k\left(z\right)e^z|\le \sum _{w\in {\mathcal{A}}^k}|e^z-e^{z(1-\mathbf{P}(w\left)\right)}|\\ \hfill \le 2^{k+1}{e}^{\left|z\right|cos\left(\theta \right)}.\end{array}$$ (136)

Therefore, we have

$$\begin{array}{c}\hfill \mathbf{E}\left[{\widehat{X}}_{n,k}\right]=\tilde{E}\left(n\right)+O\left({\displaystyle \frac{n^{\nu -1}}{\sqrt{logn}}}\right).\end{array}$$ (137)

This completes the proof of Theorem 3. □

On the Second Factorial Moment: We poissonize the sequence ${\left(\mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]\right)}_{n\ge 0}$ as well. By the analysis in (27),

$$\begin{array}{cc}\hfill \mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]& =\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left(1-{(1-\mathbf{P}\left(w\right))}^n-{(1-\mathbf{P}\left(w^{\prime }\right))}^n+{(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))}^n\right),\hfill \end{array}$$

which gives the following poissonized form

$$\begin{array}{cc}\hfill \tilde{G}\left(z\right)& =\sum _{n\ge 0}\mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}_2\right]{\displaystyle \frac{z^n}{n!}}{e}^{-z}\hfill \\ \hfill & =\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}1-e^{-\mathbf{P}\left(w\right)z}-e^{-\mathbf{P}\left(w^{\prime }\right)z}+e^{-(\mathbf{P}\left(w\right)+\mathbf{P}\left(w^{\prime }\right))z}\hfill \\ \hfill & =\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left(1-e^{-\mathbf{P}\left(w^{\prime }\right)z}\right)\left(1-e^{-\mathbf{P}\left(w\right)z}\right)\hfill \\ \hfill & ={\left(\sum _{w\in {\mathcal{A}}^k}\left(1-e^{-\mathbf{P}\left(w\right)z}\right)\right)}^2-\sum _{w\in {\mathcal{A}}^k}{\left(1-e^{-\mathbf{P}\left(w\right)z}\right)}^2\hfill \\ \hfill & ={\left({\tilde{E}}_k\left(z\right)\right)}^2-\sum _{w\in {\mathcal{A}}^k}{\left(1-e^{-\mathbf{P}\left(w\right)z}\right)}^2\hfill \\ \hfill & ={\left({\tilde{E}}_k\left(z\right)\right)}^2-\sum _{w\in {\mathcal{A}}^k}\left(1-2e^{-\mathbf{P}\left(w\right)z}+e^{-2\mathbf{P}\left(w\right)z}\right).\hfill \end{array}$$ (138)

We show that in all ranges of a the leftover sum in (138) has a lower order contribution to ${\tilde{G}}_k\left(z\right)$ compared to ${\left({\tilde{E}}_k\left(z\right)\right)}^2$. We define

$$\begin{array}{c}\hfill {\tilde{L}}_k\left(z\right)=\sum _{w\in {\mathcal{A}}^k}\left(1-2e^{-\mathbf{P}\left(w\right)z}+e^{-2\mathbf{P}\left(w\right)z}\right).\end{array}$$ (139)

In the first range for k, we take the Mellin transform of ${\tilde{L}}_k\left(z\right)$, which is

$$\begin{array}{cc}\hfill {\tilde{L}}_k^{*}\left(s\right)& =-2\mathsf{\Gamma }\left(s\right)\sum _{w\in {\mathcal{A}}^k}\mathbf{P}{\left(w\right)}^{-s}+\mathsf{\Gamma }\left(s\right)\sum _{w\in {\mathcal{A}}^k}{\left(2\mathbf{P}\left(w\right)\right)}^{-s}\hfill \\ \hfill & =-2\mathsf{\Gamma }\left(s\right){(p^{-s}+q^{-s})}^k+\mathsf{\Gamma }\left(s\right)2^{-s}{(p^{-s}+q^{-s})}^k\hfill \\ \hfill & =\mathsf{\Gamma }\left(s\right){(p^{-s}+q^{-s})}^k(2^{-s-1}-1),\hfill \end{array}$$ (140)

and we note that the fundamental strip for this Mellin transform of is $\langle -2,0\rangle$ as well. The inverse Mellin transform for $c\in (-2,0)$ is

$$\begin{array}{cc}\hfill {\tilde{L}}_k\left(z\right)& ={\displaystyle \frac{1}{2\pi i}}{\int }_{c-i\infty }^{c+i\infty }{\tilde{L}}_k^{*}\left(s\right)z^{-s}ds\hfill \\ \hfill & ={\displaystyle \frac{1}{\pi i}}{\int }_{c-i\infty }^{c+i\infty }\mathsf{\Gamma }\left(s\right){(p^{-s}+q^{-s})}^k(2^{-s-1}-1)z^{-s}ds\hfill \end{array}$$ (141)

We note that this range of $r_0$ corresponds to

$$\begin{array}{c}\hfill {\displaystyle \frac{2}{log{q}^{-1}+log{p}^{-1}}}<a<{\displaystyle \frac{p^2+q^2}{q^{2}log{q}^{-1}+p^{2}log{p}^{-1}}}.\end{array}$$ (142)

The integrand in (141) is quite similar to the one seen in (107). The only difference is the extra term $2^{-s-1}-1$. However, we notice that $2^{-s-1}-1$ is analytic and bounded. Thus, we obtain the same saddle points with the real part as in (109) and the same imaginary parts in the form of ${\displaystyle \frac{2\pi ij}{logp/q}}$, $j\in \mathbb{Z}$. Thus, the same saddle point analysis for the integral in (107) applies to ${\tilde{L}}_k\left(z\right)$ as well. We avoid repeating the similar steps, and we skip to the central approximation, where by Laplace’s theorem (ref. [22]), we get

$$\begin{array}{cc}\hfill {\tilde{L}}_k\left(z\right)& ={\displaystyle \frac{{(p/q)}^{-r_0/2}+{(p/q)}^{r_0/2}}{\sqrt[]{2\pi }logp/q}}\hfill \\ \hfill & \times \sum _{\left|j\right|<j^{*}}{z}^{-r_0}{(p^{-r_0}+q^{-r_0})}^k(2^{-r_0-1-i{t}_j}-1)\hfill \\ \hfill & \times \mathsf{\Gamma }(r_0+i{t}_j)z^{-i{t}_j}{p}^{-ik{t}_j}{k}^{-1/2}\left(1+O\left({\displaystyle \frac{1}{\sqrt{k}}}\right)\right),\hfill \end{array}$$ (143)

which can be represented as

$$\begin{array}{cc}\hfill {\tilde{L}}_k\left(z\right)& ={\mathsf{\Phi }}_2\left((1+alogp){log}_{p/q}n\right){\displaystyle \frac{z^{\nu }}{\sqrt{logn}}}\left(1+O\left({\displaystyle \frac{1}{\sqrt{logn}}}\right)\right),\hfill \end{array}$$ (144)

where

$$\begin{array}{cc}\hfill {\mathsf{\Phi }}_2\left(x\right)& ={\displaystyle \frac{{(p/q)}^{-r_0/2}+{(p/q)}^{r_0/2}}{\sqrt[]{2a\pi }logp/q}}\sum _{\left|j\right|<j^{*}}(2^{-r_0-1-i{t}_j}-1)\mathsf{\Gamma }(r_0+i{t}_j)e^{-2\pi ijx}.\hfill \end{array}$$ (145)

This shows that ${\tilde{L}}_k\left(z\right)=O\left({\displaystyle \frac{z^{\nu }}{\sqrt{logn}}}\right)$, when

$${\displaystyle \frac{2}{log{q}^{-1}+log{p}^{-1}}}<a<{\displaystyle \frac{p^2+q^2}{q^{2}log{q}^{-1}+p^{2}log{p}^{-1}}}.$$

Subsequently, for ${\displaystyle \frac{1}{log{q}^{-1}}}<a<{\displaystyle \frac{2}{log{q}^{-1}+log{p}^{-1}}}$, we get

$$\begin{array}{cc}\hfill {\tilde{L}}_k\left(z\right)& =2^k-{\mathsf{\Phi }}_2\left((1+alogp){log}_{p/q}n\right){\displaystyle \frac{z^{\nu }}{\sqrt{logn}}}\left(1+O\left({\displaystyle \frac{1}{\sqrt{logn}}}\right)\right),\hfill \end{array}$$ (146)

and for ${\displaystyle \frac{p^2+q^2}{q^{2}log{q}^{-1}+p^{2}log{p}^{-1}}}<a<{\displaystyle \frac{1}{log{p}^{-1}}}$, we get

$$\begin{array}{cc}\hfill {\tilde{L}}_k\left(z\right)& =O\left(n^2\right).\hfill \end{array}$$ (147)

It is not difficult to see that for each range of a as stated above, ${\tilde{L}}_k\left(z\right)$ has a lower order contribution to the asymptotic expansion of ${\tilde{G}}_k\left(z\right)$, compared to ${\left({\tilde{E}}_k\left(z\right)\right)}^2$. Therefore, this leads us to Theorem 4, which will be proved bellow.

Proof of Theorem 4. 

It is only left to show that the two depoissonization conditions hold: For condition (103) in Theorem 15, from (135) we have

$$\begin{array}{cc}\hfill |{\tilde{G}}_k\left(z\right)|& \le B^2{\displaystyle \frac{|z^{2\nu }|}{logn}},\hfill \end{array}$$ (148)

and for condition (104), we have, for fixed k,

$$\begin{array}{cc}\hfill |{\tilde{G}}_k\left(z\right)e^z|& \le \sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left|e^z-e^{(1-\mathbf{P}(w\left)\right)z}-e^{(1-\mathbf{P}\left(w^{\prime }\right))z}+e^{(1-(\mathbf{P}\left(w\right)+\mathbf{P}\left(w^{\prime }\right)))z}\right|\hfill \\ \hfill & \le 4^k{e}^{\left|z\right|cos\theta }.\hfill \end{array}$$ (149)

Therefore both depoissonization conditions are satisfied and the desired result follows. □

Corollary. A Remark on the Second Moment and the Variance

For the second moment we have

$$\begin{array}{cc}\hfill \mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}^2\right]& =\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\mathbf{E}\left[{\widehat{X}}_{n,k}^{\left(w\right)}{\widehat{X}}_{n,k}^{\left(w^{\prime }\right)}\right]+\sum _{w\in {\mathcal{A}}^k}\mathbf{E}\left[{\widehat{X}}_{n,k}^{\left(w\right)}\right]\hfill \\ \hfill & =\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left(1-{(1-\mathbf{P}\left(w\right))}^n-{(1-\mathbf{P}\left(w^{\prime }\right))}^n+{(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))}^n\right)\hfill \\ \hfill & +\sum _{w\in {\mathcal{A}}^k}\left(1-{\left(1-\mathbf{P}\left(w\right)\right)}^n\right).\hfill \end{array}$$ (150)

Therefore, by (105) and (138) the Poisson transform of the second moment, which we denote by ${\tilde{G}}_k^{\left(2\right)}\left(z\right)$ is

$$\begin{array}{cc}\hfill {\tilde{G}}_k^{\left(2\right)}\left(z\right)& ={\left({\tilde{E}}_k\left(z\right)\right)}^2+{\tilde{E}}_k\left(z\right)-\sum _{w\in {\mathcal{A}}^k}\left(1-2e^{-\mathbf{P}\left(w\right)z}+e^{-2\mathbf{P}\left(w\right)z}\right),\hfill \end{array}$$ (151)

which results in the same first order asymptotic as the second factorial moment. Also, it is not difficult to extend the proof in Chapter 6 to show that the second moments of the two models are asymptotically the same. For the variance we have

$$\begin{array}{cc}\hfill \mathrm{Var}\left[{\widehat{X}}_{n,k}\right]& =\mathbf{E}\left[{\left({\widehat{X}}_{n,k}\right)}^2\right]-{\left(\mathbf{E}\left[{\widehat{X}}_{n,k}\right]\right)}^2\hfill \\ \hfill & =\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left(1-{(1-\mathbf{P}\left(w\right))}^n-{(1-\mathbf{P}\left(w^{\prime }\right))}^n+{(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))}^n\right)\hfill \\ \hfill & +\sum _{w\in {\mathcal{A}}^k}\left(1-{\left(1-\mathbf{P}\left(w\right)\right)}^n\right)\hfill \\ \hfill & -\sum _{\begin{array}{c}w,w^{\prime }\in {\mathcal{A}}^k\\ w\ne w^{\prime }\end{array}}\left(1-{(1-\mathbf{P}\left(w\right))}^n-{(1-\mathbf{P}\left(w^{\prime }\right))}^n+{(1-\mathbf{P}\left(w\right)-\mathbf{P}\left(w^{\prime }\right))}^n\right)\hfill \\ \hfill & -\sum _{w\in {\mathcal{A}}^k}\left(1-{\left(1-\mathbf{P}\left(w\right)\right)}^n-{\left(1-\mathbf{P}\left(w\right)\right)}^n+{\left(1-\mathbf{P}\left(w\right)\right)}^{2n}\right)\hfill \\ \hfill & =\sum _{w\in {\mathcal{A}}^k}\left({\left(1-\mathbf{P}\left(w\right)\right)}^n-{\left(1-\mathbf{P}\left(w\right)\right)}^{2n}\right).\hfill \end{array}$$ (152)

Therefore the Poisson transform, which we denote by ${\tilde{G}}_k^{\mathrm{var}}\left(z\right)$ is

$$\begin{array}{c}\hfill {\tilde{G}}_k^{\mathrm{var}}\left(z\right)=\sum _{w\in {\mathcal{A}}^k}\left(e^{-\mathbf{P}\left(w\right)z}-e^{-(2\mathbf{P}\left(w\right)+{\left(\mathbf{P}\left(w\right)\right)}^2)z}\right).\end{array}$$ (153)

The Mellin transform of the above function has the following form

$$\begin{array}{cc}\hfill {\widetilde{G^{*}}}_k^{\mathrm{var}}\left(z\right)& =\mathsf{\Gamma }\left(s\right){(p^{-s}+q^{-s})}^k(-1+O\left(\mathbf{P}\left(w\right)\right)).\hfill \end{array}$$ (154)

This is quite similar to what we saw in (106), which indicates that the variance has the same asymptotic growth as the expected value. But the variance of the two models do not behave in the same way (cf. Figure 2).

4. Summary and Conclusions

We studied the first-order asymptotic growth of the first two (factorial) moments of the kth Subword Complexity. We recall that the kth Subword Complexity of a string of length n is denoted by $X_{n,k}$, and is defined as the number of distinct subwords of length k, that appear in the string. We are interested in the asymptotic analysis for when k grows as a function of the string’s length. More specifically, we conduct the analysis for $k=\mathsf{\Theta }(logn)$, and as $n\to \infty$.

The analysis is inspired by the earlier work of Jacquet and Szpankowski on the analysis of suffix trees, where they are compared to independent tries (cf. [14]). In our work, we compare the first two moments of the kth Subword Complexity to the kth Prefix Complexity over a random trie built over n independently generated binary strings. We recall that we define the kth Prefix Complexity as the number of distinct prefixes that appear in the trie at level k and lower.

We obtain the generating functions representing the expected value and the second factorial moments as their coefficients, in both settings. We prove that the first two moments have the same asymptotic growth in both models. For deriving the asymptotic behavior, we split the range for k into three intervals. We analyze each range using the saddle point method, in combination with residue analysis. We close our work with some remarks regarding the comparison of the second moment and the variance to the kth Prefix Complexity.

5. Future Challenges

The intervals’ endpoints for a in Theorems 3 and 4 are not investigated in this work. The asymptotic analysis of the end points can be studied using van der Waerden saddle point method [24].

The analogous results are not (yet) known in the case where the underlying probability source has Markovian dependence or in the case of dynamical sources.

Abbreviations

The following abbreviations are used in this manuscript:

PGF	Probabilty Generating Function
$\mathbf{P}$	Probability
$\mathbf{E}$	Expected value
Var	Variance
$\mathbf{E}\left[{\left(X_{n,k}\right)}_2\right]$	The second factorial moment of $X_{n,k}$

Author Contributions

This paper is based on a Ph.D. dissertation conducted by the L.A. under the supervision of the M.D.W. All authors have read and agreed to the published version of the manuscript.

Funding

M.D.W. Ward’s research is supported by FFAR Grant 534662, by the USDA NIFA Food and Agriculture Cyberinformatics and Tools (FACT) initiative, by NSF Grant DMS-1246818, by the NSF Science & Technology Center for Science of Information Grant CCF-0939370, and by the Society Of Actuaries.

Conflicts of Interest

The authors declare no conflict of interest.