Optimal Permutation Recovery in Permuted Monotone Matrix Model

Abstract

Motivated by recent research on quantifying bacterial growth dynamics based on genome assemblies, we consider a permuted monotone matrix model Y = ΘΠ+ Z, where the rows represent different samples, the columns represent contigs in genome assemblies and the elements represent log-read counts after preprocessing steps and Guanine-Cytosine (GC) adjustment. In this model, Θ is an unknown mean matrix with monotone entries for each row, Π is a permutation matrix that permutes the columns of Θ, and Z is a noise matrix. This paper studies the problem of estimation/recovery of Π given the observed noisy matrix Y. We propose an estimator based on the best linear projection, which is shown to be minimax rate-optimal for both exact recovery, as measured by the 0–1 loss, and partial recovery, as quantified by the normalized Kendall’s tau distance. Simulation studies demonstrate the superior empirical performance of the proposed estimator over alternative methods. We demonstrate the methods using a synthetic metagenomics dataset of 45 closely related bacterial species and a real metagenomic dataset to compare the bacterial growth dynamics between the responders and the non-responders of the IBD patients after 8 weeks of treatment.

1. INTRODUCTION

1.1. A Motivation Example from Microbiome Studies

The statistical problem considered in this paper is motivated by the problem of estimating the bacterial growth dynamics based on shotgun metagenomics data (Myhrvold et al. 2015; Abel et al. 2015; Korem et al. 2015; Brown et al. 2016). The growth dynamics of microbial populations reflects their physiological states and drives variation of microbial compositions, which provide important feature summary of the microbes in a given community. One way of studying such communities is through shotgun metagenomic sequencing, which involve direct DNA sequencing of all the microbiome genomes in a given microbial community. Korem et al. (2015) presented the first paper on quantifying the bacterial growth dynamics based on shotgun metagenomics data, where the uneven sequencing read coverage resulting from the bidirectional DNA replications provides information on the rates of microbial DNA replications. For bacterial species with known complete genome sequences, Korem et al. (2015) proposed to use the peak-to-trough ratio (PTR) of read coverages to quantify the bacterial growth dynamics after aligning the sequencing reads to the complete genome sequences.

However, in many applications, it is of importance to quantify the bacterial growth dynamics based on genome assemblies for the bacterial species with unknown genomes. These genome assemblies may represent new bacterial species that we have seen or sequenced before. The genome assembly of a bacterium species consists of a collection of contigs (called bin) constructed based on the overlapping of the sequencing reads (Li et al. 2015; Wu et al. 2014). Compared to the complete genome, the genome assembled bins are more fragmented and often contained errors or contaminations. The noisy read coverage data due to intraspecific variations, interspecific/intraspecific repeated sequences, limited sequencing depths and the inability of binning algorithms to correctly cluster all the contigs further complicate the estimation of growth dynamics based on read coverages of the contigs. Besides these noisy count data, one key difficulty in estimating the growth dynamic based on contig counts is that the accurate locations of the contigs on the original genome are unknown. It is therefore not feasible to measure the microbial growth rate directly using peak-to-trough coverage ratio for the assembled genomes (Brown et al. 2016; Gao and Li 2018).

Brown et al. (2016) presented the first method (called iRep) of estimating the bacterial growth dynamics based on genome assemblies, where the contigs are ordered based on the GC-adjusted counts for each sample separately. However, due to noise in the count data, such an ordering method often leads to wrong ordering of the contigs and therefore inaccurate estimates of the growth dynamics. Gao and Li (2018) developed a computational algorithm, DEMIC, to accurately compare growth dynamics of a given assembled species existing in multiple samples by taking advantage of highly fragmented contigs assembled in typical metagenomics studies. One key step of DEMIC is to apply a principal components analysis (PCA)-based method to recover the true ordering of the contigs along the underlying unknown bacterial complete genomes. Gao and Li (2018) reported excellent empirical performance of DEMIC over existing methods. The goal of this paper is to provide a rigorous statistical framework to study the problem of optimal permutation recovery in a permuted monotone matrix model.

1.2. A Permuted Monotone Matrix Model

For a given genome assembly with p contigs, DEMIC first obtains the read coverage for each of the sliding window of size 5000 bps, denoted by X_ijl for the ith sample, jth contig and kth window. In order to account for the GC-content of the kth window, Gao and Li (2018) considered the following mixed-effects model,

$${\log }_2{X}_{ijk}=\alpha +G{C}_{jk}\beta +W_{ij}+e_{ijk},$$

where GC_jk is the centred GC count of the kth window of the jth contig, W_ij is the sample- and contig- specific random intercept, α is the intercept, β is the regression coefficient, and e_ijk is the random error. This model is fitted for each contig to obtain the best linear unbiased predictor of W_ij, which is used as the GC-adjusted log-read count Y_ij for the ith sample and jth contig. Here Y_ij can be regarded as average read coverage over non-overlapping windows of a contig and is approximately normally distributed.

Let Y be the GC-adjusted log-contig count matrix of n samples and p contigs of a genome assembly with Y_ij as its entries. Given this, we consider the following permuted monotone matrix model:

$$Y=\mathrm{\Theta }\mathrm{\Pi }+Z,$$ (1)

where $\mathrm{\Theta }\in {ℝ}^{n\times p}$ is an unknown nonnegative signal matrix with nondecreasing rows, $Z\in {ℝ}^{n\times p}$ is a zero-mean noise matrix, and $\mathrm{\Pi }\in {ℝ}^{p\times p}$ is a permutation matrix corresponding to some permutation Π from the symmetric group ${\mathcal{S}}_p$. That is, after a suitable permutation of the columns of Y, all the rows of the mean matrix are nondecreasing sequences. In microbiome applications, Θ is the matrix of true log-coverage of n samples over p contigs along the circular genome of the bacterium, which is generally hypothesized to have non-decreasing rows. Π represents a permutation due to unknown locations of the contigs relative to the replication origin. Throughout this paper, we denote the parameter space

$$(\mathrm{\Theta },\pi )\in \mathcal{D}=\left\{\mathrm{\Theta }=\left({\theta }_{ij}\right)\in {ℝ}^{n\times p},\pi \in {\mathcal{S}}_p:0\le {\theta }_{i,j-1}\le {\theta }_{i,j}<\infty \text{for}\text{all}1\le i\le n,2\le j\le p\right\}.$$

The focus of this paper is to optimally estimate the permutation π from the noisy observation Y.

1.3. Related Problems and Other Applications

The permutation recovery problem under permuted monotone matrix model bears some similarity to other problems studied in machine learning literature, including the feature matching between two sets of observations (Collier and Dalalyan 2016) and linear regression model with permuted data, where the correspondences between the response and the predictors are unknown (Pananjady, Wainwright, and Courtade 2016; Slawski and Ben-David 2017; Pananjady, Wainwright, and Courtade 2017). More recently, Flammarion, Mao, and Rigollet (2019) considered the problem of statistical seriation, which has a close affinity to our model (1). However, the focus of Flammarion, Mao, and Rigollet (2019) is to optimally estimate the signal matrix Θ rather than the underlying permutation.

Model (1) can be thought as a natural extension of the shape constrained matrix denoising model studied in the isotonic regression literature. Specifically, under Π = I_p, risk bounds and the minimax rate-optimal estimator Model (1) with known for Θ under the Frobenius norm was obtained in Chatterjee, Guntuboyina, and Sen (2015) for n = 1 and later in Chatterjee, Guntuboyina, and Sen (2018) for general n > 1. Using the idea of optimal transport, a minimax optimal estimator of the underlying signals was obtained by Rigollet and Weed (2018). However, their goal is not to recover the underlying permutation.

Besides the microbiome applications, the permuted monotone matrix model is generic and has other applications. For instance, the problem of permutation recovery is usually equivalent to statistical ranking/sorting from noisy observations, which arises commonly in finance (Currie and Pandher 2011), sport analytics (Deshpande and Jensen 2016), and recommendation systems (Rendle et al. 2009). Specifically, in the latter case, the task of tag recommendation is to provide a user with a personalized ranked list of tags for a specific item. Under the permuted monotone matrix model, we can treat the entries of Y, say Y_ij, as an indicator of the jth tag being related to the ith item by a given customer, and Θ as a probability matrix characterizing the customer’s tagging preferences across multiple items. As a result, recovering the underlying permutation provides a solution of a tag recommender.

1.4. Main Contributions and Organization

In this paper, we investigate the problem of permutation recovery in the permuted monotone matrix model (1), which relies on certain invariance property of the singular subspace of the monotone matrices. The properties of the proposed method in terms of both the exact and partial recovery are studied in detail. In particular, we obtained regions of the signal-to-noise ratio (defined later as Γ /σ) that are subject to exact/partial recovery (Figure 1). For both exact and partial permutation recovery, we obtained the matching minimax lower bounds and established the minimax rate-optimality of the proposed method over a wide range of parameter space (Figure 1). For partial recovery, the proof of the lower bound relies on a version of Fano’s lemma and the sphere packing of the symmetric group equipped with the Kendall’s tau metric.

Fig. 1.

A graphical illustration of the main result obtained in this paper about the regions of the signal-to-noise ratio Γ /σ that correspond to exact/partial recovery, and the region with proved minimax optimality.

The rest of this paper is organized as follows. After a brief introduction of notation and definitions, we present in Section 2 the proposed permutation estimator. The theoretical properties of the proposed method are studied, first under a more illustrative linear growth model in Section 3 and then under a general growth model in Section 4. Section 5 provides results on minimax lower bounds and the optimality of the proposed estimator. We evaluate the methods using both simulated data, synthetic and real microbiome datasets and compare with other methods in Section 6. In Section 7, we discuss some implications and extensions of the methods. Finally, the proofs of our main results are given in Section 8.

1.5. Notation and Definitions

Throughout, we define the permutation π as a bijection from the set {1,2,..., p} onto itself. For simplicity, we denote π = (π(1), π(2),...,π(p)). All permutations of the set {1,2,..., p} form a symmetric group, equipped with the function composition operation °, denoted as ${\mathcal{S}}_p$. For any $\pi \in {\mathcal{S}}_p$, we denote ${\pi }^{-1}\in {\mathcal{S}}_p$ as its group inverse, so that π°π⁻¹ = π⁻¹°π = id, and denote rev(π) = (π(p), π(p−1),... π(1)). In particular, we may use π and its corresponding permutation matrix $\mathrm{\Pi }\in {ℝ}^{p\times p}$ interchangeably, depending on the context. For a vector $\mathbf{a}={\left(a_1,\dots ,a_n\right)}^{\top }\in {ℝ}^n$, we define the ℓ_p norm $\Vert \mathbf{a}{\Vert }_p={\left(\sum _{i=1}^n{a}_i^p\right)}^{1/p}$, and the ℓ_∞ norm $\Vert \mathbf{a}{\Vert }_{\infty }={\max }_{1\le j\le n}\left|a_i\right|$. For a matrix $\mathrm{\Theta }\in {ℝ}^{p_1\times p_2}$, we denote ${\mathrm{\Theta }}_{.i}\in {ℝ}^{p_1}$ as its i-th norm column, ${\mathrm{\Theta }}_{i.}\in {ℝ}^{p_2}$ as its i-th row, and denote its (ordered) singular values as ${\lambda }_1(\mathrm{\Theta })\ge {\lambda }_2(\mathrm{\Theta })\ge \dots \ge {\lambda }_{\min \{p_1,p_2\}}(\mathrm{\Theta })$. Furthermore, for sequences {a_n} and {b_n}, we write $a_n=o\left(b_n\right)$ if $\underset{n}{\lim }{a}_n/b_n=0$, and write a_n = O(b_n), a_n ≲ b_n or b_n ≳ a_n if there exists a constant C such that a_n ≤ Cb_n for all n. We write a_n≍b_n if a_n≲b_n and a_n≳b_n. For a finite set A, we denote |A| as its cardinality. We use the logical symbols ∧ and ∨ to represent “and” and “or,” respectively. Lastly, C, C_0, C₁,... are constants that may vary from place to place.

2. PERMUTATION RECOVERY VIA BEST LINEAR PROJECTION

In the following, we first make some key observations about the connection between the underlying permutation π and the column linear projections of the observed matrix Y, which motivate our construction of the proposed estimator.

2.1. Linear Projection

Given the observed noisy matrix Y, we consider the class of the linear projection statistics of the form $w^{\top }Y\in {ℝ}^p$ where $w\in {ℝ}^n$ and ||w||₂ = 1. Intuitively, by projecting each column of Y onto the subspace generated by w, the components of w^⊤Y (hereafter referred as “projection scores”) would quantify the relative position of the columns of Y, so that their order statistics can be used to recover the original orders of the columns of Θ. To fix ideas, we define the following ranking operator.

Definition 1 (Ranking Operator). The ranking operator $\mathfrak{r}:{ℝ}^p\to {\mathcal{S}}_p$ is defined such that for any vector $\mathbf{x}\in {ℝ}^p,\mathfrak{r}(\mathbf{x})$ is the vector of ranks for components of x in increasing order. Whenever there are ties, increasing orders are assigned from left to right.

For example, given a vector x = (2,5,1,6,2)^⊤, we have $\mathfrak{r}(\mathbf{x})=(2,4,1,5,3)$. The following proposition concerning the invariance property of the column spacing of Θ is the key to our construction of the minimax optimal estimator.

Proposition 1. Suppose $(\mathrm{\Theta },\pi )\in \mathcal{D}$. For any nonnegative unit vector $w\in {ℝ}^n$, we have

$$\mathfrak{r}\left(w^{\top }\mathrm{\Theta }\mathrm{\Pi }\right)={\pi }^{-1}.$$ (2)

Apparently, under the noiseless setting, any nonnegative unit vector would lead to the exact recovery of the underlying permutation as in this case the relative orders of the columns are exactly coded by the relative magnitudes of the projection scores w^⊤Y = w^⊤ΘΠ. However, with the noisy observations, w^⊤Y = w^⊤ΘΠ+ w^⊤Z so that the relative orders of the columns are only partially preserved by the noisy projection scores w^⊤Y, up to some random perturbations.

Consequently, the best linear projection vector w₀ would correspond to the case where $w_0^{\top }\mathrm{\Theta }\mathrm{\Pi }$ has the most separated components such that their relative orders are most immune to the random noises. Specifically, since for any given $w\in {ℝ}^n$, the i-th component of w^⊤ΘΠ has the expression w^⊤ΘΠe_i where ${\left\{e_i\right\}}_{i=1}^p$ is the canonical basis of the Euclidean space ${ℝ}^p$, we define

$$w_0={\mathrm{argmax}}_{\begin{array}{l}w\in {ℝ}^n\\ \left|\right|w|{|}_2=1\end{array}}\sum _{\begin{array}{c}1\le i,j\le p\\ i\ne j\end{array}}{\left(w^{\top }\mathrm{\Theta }\mathrm{\Pi }{e}_i-w^{\top }\mathrm{\Theta }\mathrm{\Pi }{e}_j\right)}^2={\mathrm{argmax}}_{\begin{array}{l}w\in {ℝ}^n\\ \left|\right|w|{|}_2=1\end{array}}\sum _{i=1}^p{\left(w^{\top }\mathrm{\Theta }\mathrm{\Pi }{e}_i-\frac{1}{p}\sum _{j=1}^p{w}^{\top }\mathrm{\Theta }\mathrm{\Pi }{e}_j\right)}^2,$$

which maximizes the pairwise distances of the components under the squared distance. Now since w₀ relies on the unknown ΘΠ and is not computable from the data, we substitute ΘΠ by its sample/noisy counterpart Y and define our data-driven best linear projection vector as

$$\widehat{w}=\arg {\max }_{\begin{array}{l}w\in {ℝ}^n\\ \left|\right|w|{|}_2=1\end{array}}\sum _{i=1}^p{\left(w^{\top }Y{e}_i-\frac{1}{p}\sum _{i=1}^p{w}^{\top }Y{e}_i\right)}^2,$$ (3)

which is actually the first eigenvector of the symmetric matrix

$$A=Y\sum _{i=1}^p\left(e_i-\frac{1}{p}\sum _{i=1}^p{e}_i\right){\left(e_i-\frac{1}{p}\sum _{i=1}^p{e}_i\right)}^{\top }{Y}^{\top },$$ (4)

and can be immediately solved by performing an eigen-decomposition on A. Once $\widehat{w}$ is obtained, we define our proposed permutation estimator as

$$\widehat{\pi }={(\mathfrak{r}({\widehat{w}}^{\top }Y))}^{-1}.$$ (5)

Intuitively, the projection vector $\widehat{w}$ assigns different weights to the rows of Y so that more weight is given to the rows whose elements are better separated and therefore more informative in distinguishing the columns of Y or Θ.

2.2. Evaluation Criteria

The main focus of this paper is to investigate the theoretical properties of our proposed estimator (5) under various loss measures and parameter spaces. For any given estimator $\check{\pi }$, we first consider the 0–1 loss

$$\ell (\check{\pi },\pi )=1\{\check{\pi }\ne \pi \},$$

with the corresponding risk $\mathbb{E}\ell (\check{\pi },\pi )=P(\check{\pi }\ne \pi )$. The 0–1 loss is used to evaluate the exact recovery, which can be a strong requirement for practical applications. As an alternative, we also consider the more flexible partial recovery, where the loss function is given by the normalized Kendall’s tau distance (Kendall 1938) defined as

$${\tau }_K\left({\pi }_1,{\pi }_2\right)=\frac{\left\{\#\text{of}\text{discordant}\text{pairs}\text{between}{\pi }_1\text{and}{\pi }_2\right\}}{\left(\begin{array}{l}n\hfill \\ 2\hfill \end{array}\right)}.$$ (6)

Technically, for two permutations π₁ and π₂, the set of discordant pairs is defined as

$$\mathcal{G}\left({\pi }_1,{\pi }_2\right)=\left\{(i,j):i<j,\left[{\pi }_1(i)<{\pi }_1(j)\wedge {\pi }_2(i)>{\pi }_2(j)\right]\vee \left[{\pi }_1(i)>{\pi }_1(j)\wedge {\pi }_2(i)<{\pi }_2(j)\right]\right\}$$

so that the numerator in (6) is equal to the cardinality $|\mathcal{G}\left({\pi }_1,{\pi }_2\right)|$, which, in fact, is also the minimum number of pairwise adjacent transpositions converting ${\pi }_1^{-1}$ into ${\pi }_2^{-1}$ (Diaconis 1988). The denominator $\left(\begin{array}{l}n\hfill \\ 2\hfill \end{array}\right)$ ensures that τ_K (π₁, π₂) ∈[0,1] where τ_K (π₁, π₂) = 0 corresponds to π₁ = π_2.

3. A LINEAR GROWTH MODEL

We start with a simpler case where the pair (Θ,π) is from the subspace

$${\mathcal{D}}_L=\{(\mathrm{\Theta },\pi )\in \mathcal{D}:\begin{array}{l}{\theta }_{ij}=a_i{\eta }_j+b_i,\text{where}{a}_i,b_i\ge 0\text{for}1\le i\le n,\hfill \\ 0\le {\eta }_j\le {\eta }_{j+1}\text{for}1\le j\le p-1\hfill \end{array}\}.$$ (7)

In other words, each row of Θ has a linear growth pattern with possibly different intercepts and different slopes. In the context of bacterial growth dynamics, this model is sometimes referred as the Cooper-Helmstetter model (Cooper and Helmstetter 1968; Bremer and Churchward 1977) that associates the copy number of genes with their relative distances to the replication origin. Specifically, a_i is the ratio of genome replication time and doubling time, which can be used to quantify the bacterial growth dynamics for the ith sample, η_j is related to distance from the replication origin for the jth contig, and b_i is related to the read counts at the replication origin and the sequencing depth. If the bacterium is non-dividing in sample i, a_i is zero.

For the linear growth model (7), there are two key quantities that are relevant to permutation recovery.

Definition 2. For any $\mathrm{\Theta }\in {\mathcal{D}}_L$, we define

$$\mathrm{\Gamma }={\left(\sum _{i=1}^n{a}_i^2\right)}^{1/2}\cdot \underset{1\le i<j\le p}{\min }\left|{\eta }_i-{\eta }_j\right|$$ (8)

as the local minimal signal gap of Θ, and define

$$\mathrm{\Lambda }=\left(\sum _{i=1}^n{a}_i^2\right)·\frac{1}{p}\sum _{1\le i<j\le p}{\left({\eta }_i-{\eta }_j\right)}^2=\left(\sum _{i=1}^n{a}_i^2\right)\cdot \sum _{j=1}^p{\left({\eta }_j-\overline{\eta }\right)}^2$$ (9)

as the global signal strength of Θ, where $\overline{\eta }=\sum _{j=1}^p{\eta }_j/p$.

Intuitively, both quantities involve the set ${\left\{\left|{\eta }_j-{\eta }_i\right|\right\}}_{1\le i<j\le p}$ and the ℓ₂ norm of the vector a = (a₁,...,a_n)^⊤, which characterize the column spacings and the growth rates (slopes) of Θ, respectively. Throughout this paper, we assume (A1) the additive noise matrix $Z\in {ℝ}^{n\times p}$ has i.i.d. entries z_ij ~ N(0,σ²). The Gaussian assumption simplifies our theoretical analysis. But this is not essential because all the theoretical results remain true if Z has independent sub-Gaussian entries with parameters bounded by σ². The following theorem provides conditions on Γ and Λ such that exact recovery of π can be obtained by $\widehat{\pi }$ in (5).

Theorem 1 (Exact Recovery, Linear). Suppose (A1) hold, $(\mathrm{\Theta },\pi )\in {\mathcal{D}}_L$ and Θ satisfies

$$\mathrm{\Gamma }>C_0\sigma \sqrt{\log p},\mathrm{\Lambda }>C_1{\sigma }^2\left(n\max \left\{{\sigma }^{2}n/{\mathrm{\Gamma }}^2,1\right\}+\sqrt{np\max \left\{{\sigma }^{2}n/{\mathrm{\Gamma }}^2,1\right\}}\right)$$ (10)

for some C₀, C₁ > 0. Then with probability at least 1−O(p^−c) for some constant c > 0, up to a permutation reversion, we have $\widehat{\pi }=\pi$.

Remark 1. Due to non-identifiability between $\widehat{w}$ and $-\widehat{w}$ defined in (3), in Theorem 1, as well as all the other theoretical results concerning $\widehat{\pi }$, the statement is up to a possible reversion of $\widehat{\pi }$. For example, for permutation π= (2,4,1,5,3), its reversion would be rev(π) = (4,2,5,1,3). In fact, such indeterminacy can be avoided by noting that a_i ≥ 0 for all i’s, but we will not pursue such a direction in this study as the practical interest only concerns relative orders of the permuted elements.

Since Γ depends on a only through its ℓ₂ ∥a∥_2, the local minimal signal gap (MSG) condition $\mathrm{\Gamma }\ge C\sigma \sqrt{\log p}$ allows for the presence of non-informative signals in the sense that some components of a can be 0. In contrast, the condition on Λ (GSS) depends on a trade-off between Γ and $\sigma \sqrt{n}$. One the one hand, when $\mathrm{\Gamma }>\sigma \sqrt{n}$, the condition on Λ becomes $\mathrm{\Lambda }\ge {\sigma }^2\left(C_{0}n+C_1\sqrt{np}\right)$, which is independent of Γ, and is minimax optimal for left singular subspace estimation (Cai and Zhang 2018). On the other hand, when $\mathrm{\Gamma }<\sigma \sqrt{n}$, stronger condition on Λ is posed, as a compensation for small Γ.

In some cases, the GSS condition in (10) can be implied by the MSG condition. We summarize our results in the following proposition.

Proposition 2. Suppose Γ /σ>1/ p and the MSG condition hold. Then the GSS condition can be implied by either one of the following conditions

1. $\mathrm{\Gamma }\gtrsim \sigma \sqrt{n}$;

2. $\mathrm{\Gamma }\lesssim \sigma \sqrt{n}$, and either ${\left({\sigma }^4{n}^2/{\mathrm{\Gamma }}^4\right)}^{1/3}\lesssim p\lesssim {\sigma }^2{n}^2/{\mathrm{\Gamma }}^{2}orp\gtrsim {\sigma }^2{n}^2/{\mathrm{\Gamma }}^2+{\left({\sigma }^{3}n/{\mathrm{\Gamma }}^3\right)}^{2/5}$.

We next turn to the partial recovery and study the rate of convergence of $\widehat{\pi }$ measured by the normalized Kendall’s tau distance under the linear growth model. In particular, we will assume an approximate uniform assignment of ${\left\{{\eta }_j\right\}}_{j=1}^p$ over some subinterval of [0,∞). In other words, the minimal element and maximal element of the set ${\left\{\left|{\eta }_j-{\eta }_{j+1}\right|\right\}}_{j=1}^{p-1}$ should have roughly the same magnitude, so that $\mathrm{\Gamma }=\Vert a{\Vert }_2\cdot {\min }_{1\le j\le p-1}\left|{\eta }_j-{\eta }_{j+1}\right|\asymp \Vert a{\Vert }_2\cdot {\max }_{1\le j\le p-1}\left|{\eta }_j-{\eta }_{j+1}\right|$. This is equivalent to assuming that the contigs in genome assemblies are approximately uniformly spaced along the circular genome.

Theorem 2 (Partial Recovery, Linear). Suppose (A1) hold, $(\mathrm{\Theta },\pi )\in {\mathcal{D}}_L$, and Θ satisfies

1. there exist some C₀ > 0 such that ${\max }_{1\le j\le p-1}\left|{\eta }_j-{\eta }_{j+1}\right|<C_0{\min }_{1\le j\le p-1}\left|{\eta }_j-{\eta }_{j+1}\right|$ for all p > 0, and

2. $\mathrm{\Lambda }>C_1{\sigma }^2\left(\max \{\frac{{\sigma }^2{(n+\log p)}^2}{{\mathrm{\Gamma }}^2},n\}+\sqrt{p}\max \left\{\frac{\sigma (n+\log p)}{\mathrm{\Gamma }},\sqrt{n}\right\}\right)$ for some C₁ > 0.

Then, up to a permutation reversion,

$$\mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]\le 1\wedge \left(\frac{c_0\sigma }{p\mathrm{\Gamma }}\min \left\{1,e^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\log \left(1+\frac{2{\sigma }^2}{{\mathrm{\Gamma }}^2}\right)\right\}+\frac{c_1{e}^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}}{p(\mathrm{\Gamma }/\sigma +\sqrt{8/\pi })}+\frac{c_2}{p^{c+2}}\right)$$

for some c, c₀, c_1, c₂ > 0.

Remark 2. The risk upper bound derived in the above theorem can be simplified as

$$\mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]\lesssim \{\begin{array}{ll}\frac{\sigma }{p\mathrm{\Gamma }}\wedge 1\hfill & \text{if}\mathrm{\Gamma }/\sigma \to 0\hfill \\ \frac{\sigma }{p\mathrm{\Gamma }}{e}^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}+1/p^{c+2}\hfill & \text{otherwise}\hfill \end{array}$$

for some c > 0. In the case of Γ /σ→∞, simple calculation yields $e^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\sigma /(p\mathrm{\Gamma })+1/p^{c+2}\asymp e^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\sigma /\mathrm{\Gamma }$ when $\mathrm{\Gamma }<\sigma \sqrt{2(c+1)\log p}$, whereas $e^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\sigma /(p\mathrm{\Gamma })+1/p^{c+2}\asymp 1/p^{c+2}$ when $\mathrm{\Gamma }\ge \sigma \sqrt{2(c+1)\log p}$. As a result, we also have

$$\mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]\lesssim \{\begin{array}{ll}1/p^{c+2}\hfill & \text{if}\mathrm{\Gamma }/\sigma \ge \sqrt{2(c+1)\log p}\hfill \\ \frac{\sigma }{p\mathrm{\Gamma }}{e}^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\hfill & \text{if}1\lesssim \mathrm{\Gamma }/\sigma <\sqrt{2(c+1)\log p}\hfill \\ \frac{\sigma }{p\mathrm{\Gamma }}\wedge 1\hfill & \text{if}\mathrm{\Gamma }/\sigma \lesssim 1\hfill \end{array}.$$ (11)

See Figure 2 for an illustration.

Fig. 2.

A graphical illustration of the risk upper bound for $\mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]$, as a function of signal-to-noise ratio Γ /σ.

In general, Theorem 2 shows that, even with a weaker condition on Γ that is below the requirement for the exact recovery, our proposed estimator $\widehat{\pi }$ is still able to obtain a partial recovery of π with an exponential rate of convergence if Γ /σ ≳1 and a polynomial rate of convergence if 1/ p < Γ /σ ≲1. As for Λ, the requirement is essentially the same as the exact recovery, except for an additional log p term, which is negligible in the exact recovery scenario.

Some implications about the practically preferable settings of n and p should be clarified. Firstly, although Theorem 1 implies that the difficulty for exact recovery increases as p grows (see also Table 1 from our simulations), our theory suggests a wide range of feasible choices for p. For example, if the underlying signals θ_ij and the noise level σ² are of constant order, then we have $\mathrm{\Gamma }\asymp \sqrt{n}$ and Λ ≍ np³, so the conditions of Theorem 1 imply that the exact recovery can be guaranteed as long as log p≲ n. In other words, p is allowed to grow exponentially with n, which is in line with the modern high-dimensional setting. Secondly, our Theorem 2 implies that, even if some conditions (such as MSG) for the exact recovery are not satisfied, one can still hope to partially recover the underlying permutation. In accordance to our theoretical result (11), our numerical results (Figure 4) show that, for the partial recovery, increasing p indeed reduces the overall risk of the proposed estimator. Finally, as to the sample size n, we argue that, without assuming additional structural assumptions such as row-sparsity, it is very unlikely that including more samples will result in a worse estimate (see Table 1 and Figure 4 for numerical evidences).

Table 1.

The empirical risks of the estimators under the 0–1 loss based on 200 simulations for various combinations of the parameters (p, n, α). $\widehat{\pi }$: proposed method; Π_mean: mean-based method; Π_max: max-based method.

p = 75	S1(σ2 = 0.025)		S2(σ2 = 0.1)		S3(σ2 = 0.0075)		S4(σ2 = 0.025)
n = 40	α = 0.1	0.2	0.1	0.2	0.1	0.2	0.1	0.2
$\widehat{\pi }$	0.775	0.575	0.415	0.000	0.025	0.020	0.025	0.000
Πmean	0.925	0.815	0.955	0.015	0.155	0.135	0.880	0.005
Πmax	1.000	1.000	1.000	0.995	0.995	0.970	0.840	0.430
n = 40	S1(σ2 = 0.025)		S2(σ2 = 0.1)		S3(σ2 = 0.0075)		S4(σ2 = 0.025)
α = 0.1	p = 60	90	60	90	60	90	60	90
$\widehat{\pi }$	0.410	0.930	0.340	0.470	0.010	0.115	0.000	0.010
Πmean	0.720	0.985	0.910	0.980	0.070	0.245	0.775	0.900
Πmax	1.000	1.000	1.000	1.000	0.975	1.000	0.815	0.875
p = 75	S1(σ2 = 0.025)		S2(σ2 = 0.1)		S3(σ2 = 0.0075)		S4(σ2 = 0.025)
α = 0.1	n = 40	60	40	60	40	60	40	60
$\widehat{\pi }$	0.765	0.440	0.475	0.095	0.050	0.020	0.010	0.005
Πmean	0.920	0.645	0.940	0.700	0.175	0.045	0.900	0.905
Πmax	1.000	1.000	1.000	1.000	0.995	0.995	0.855	0.820

Fig. 4.

Boxplots of the empirical normalized Kendall’s distance between the estimated and true permutations under different models. $\widehat{\pi }$: proposed estimator; Π_mean : mean-based estimator; Π_max: max-based estimatior.

4. A GENERAL GROWTH MODEL

In this section we study the permutation recovery over the general parameter space $\mathcal{D}$ where the growth pattern is not necessarily linear and therefore is more realistic inasmuch as the noisy nature of the shotgun metagenomic datasets (Boulund et al. 2018; Gao and Li 2018). The analysis relies on a deeper understanding of the relationship between the row-monotonic matrices and its leading singular vectors.

Specifically, for any $\mathrm{\Theta }\in \mathcal{D}$, we define the row-centered matrix

$${\mathrm{\Theta }}^{\prime }=\mathrm{\Theta }\left(I-p^{-1}e{e}^{\top }\right)\in {ℝ}^{n\times p}$$ (12)

whose singular value decomposition (SVD) is given by ${\mathrm{\Theta }}^{\prime }=\sum _{i=1}^r{\lambda }_i\left({\mathrm{\Theta }}^{\prime }\right)u_i^{\prime }{v}_i^{\prime \top }$, with r ≤ min{n, p}. The following proposition is essential to our analysis of the general growth model.

Proposition 3. Let Θ′ be defined as above, then its first right singular vector v′₁ is a monotone vector, i.e., either v′₁₁ ≤ v′₁₂ ≤…≤ v′_1p or v′₁₁ ≥ v′₁₂ ≥…≥ v′_1p.

Together with Proposition 1, the above proposition justifies our construction of the permutation estimator $\widehat{\pi }$ using a PCA based approach. To overcome the identifiability issue, we further assume λ₁(Θ′) has multiplicity one. We first introduce the several quantities that play the key roles in permutation recovery over $\mathcal{D}$.

Definition 3. For any $\mathrm{\Theta }\in \mathcal{D}$ and the corresponding Θ′ defined as above, we define

$$\mathrm{\Gamma }=\underset{1\le i<j\le p}{\min }\left|u_1^{\prime \top }\left({\mathrm{\Theta }}_{.i}^{\prime }-{\mathrm{\Theta }}_{.j}^{\prime }\right)\right|={\lambda }_1\left({\mathrm{\Theta }}^{\prime }\right)\underset{1\le i<j\le p}{\min }\left|v_{1i}^{\prime }-v_{1j}^{\prime }\right|,$$

as the local minimal signal gap, define

$$\mathrm{\Xi }=\underset{1\le i\le p-1}{\max }{\Vert {\mathrm{\Theta }}_{.i}^{\prime }-{\mathrm{\Theta }}_{.i+1}^{\prime }\Vert }_2=\underset{1\le i\le p-1}{\max }{\left(\sum _{j=1}^r{\lambda }_j^2\left({\mathrm{\Theta }}^{\prime }\right){\left|v_{ji}^{\prime }-v_{j,i+1}^{\prime }\right|}^2\right)}^{1/2},$$

as the local maximal signal gap, and define

$$\mathrm{\Lambda }={\lambda }_1^2\left({\mathrm{\Theta }}^{\prime }\right)-{\lambda }_2^2\left({\mathrm{\Theta }}^{\prime }\right)$$

as the global signal strength of Θ.

In particular, the above definitions of Γ and Λ generalize the ones given earlier in the linear growth model as these quantities coincide for $\mathrm{\Theta }\in {\mathcal{D}}_L$. The following theorem concerns the exact permutation recovery with $\widehat{\pi }$ over $\mathcal{D}$.

Theorem 3 (Exact Recovery, General). Suppose (A1) hold, $n\lesssim p,(\mathrm{\Theta },\pi )\in \mathcal{D}$, and Θ satisfies $\mathrm{\Gamma }>C_0\sigma \sqrt{\log p}$ and

$$\mathrm{\Lambda }>C_1{\sigma }^2\left[\left(n+\frac{{\mathrm{\Xi }}^2}{{\sigma }^2}\right)\max \left\{\frac{(n+\log p){\sigma }^2}{{\mathrm{\Gamma }}^2},1\right\}+\sqrt{p}\left(\sqrt{n}+\frac{\mathrm{\Xi }}{\sigma }\right)\max \left\{\frac{\sigma \sqrt{n+\log p}}{\mathrm{\Gamma }},1\right\}\right]$$

for some C₀, C₁ > 0. Then with probability at least 1−O (p^−c) for some constant c > 0, up to a permutation reversion, we have $\widehat{\pi }=\pi$.

As in the case of linear growth model (Theorem 1), in Theorem 3, to guarantee exact recovery, we need the MSG condition $\mathrm{\Gamma }>C_0\sigma \sqrt{\log p}$. Unlike the linear growth model, here Γ only implicitly depends on the elements of Θ through its spectral quantities, which makes its interpretation less clear. To address this issue, we make the following observation that links the minimal singular vector gap ${\min }_{1\le i<j\le p}\left|v_{1i}^{\prime }-v_{1j}^{\prime }\right|$ in the definition of Γ to the elements of Θ.

Proposition 4. Let Θ′ in (12) be such that there exists a δ> 0 being the lower bound of the normalized minimum gap between any two entries in the same row, i.e.

$$\underset{1\le k\le n}{\min }\frac{\left|{\theta }_{k,i}^{\prime }-{\theta }_{k,j}^{\prime }\right|}{{\Vert {\mathrm{\Theta }}_{k.}^{\prime }\Vert }_2}\ge \delta \text{for}\text{some}i\ne j.$$

Then the first singular vector $v_1^{\prime }\in {ℝ}^p$ of Θ′ satisfies $\left|v_{1,i}^{\prime }-v_{1,j}^{\prime }\right|\ge \delta$.

Consequently, the implicit requirement that ${\min }_{1\le i<j\le p}\left|v_{1i}^{\prime }-v_{1j}^{\prime }\right|$ is large can be guaranteed when the normalized minimum distance ${\min }_{1\le i<j\le p}{\min }_{1\le k\le n}\left|{\theta }_{k,i}^{\prime }-{\theta }_{k,j}^{\prime }\right|/{\Vert {\mathrm{\Theta }}_{k.}^{\prime }\Vert }_2$ is large. Our next theorem concerns the partial recovery over the general parameter space $\mathcal{D}$.

Theorem 4 (Partial Recovery, General). Suppose (A1) hold, $n\lesssim p,(\mathrm{\Theta },\pi )\in \mathcal{D}$, and Θ satisfies

1. there exits some C₀ > 0 such that ${\max }_{1\le j\le p-1}\left|v_{1j}^{\prime }-v_{1,j+1}^{\prime }\right|<C_0{\min }_{1\le j\le p-1}|v_{1j}^{\prime }-v_{1,j+1}^{\prime }|$ for all p > 0, and

2. $\mathrm{\Lambda }>C_1{\sigma }^2\left[\max \left\{\frac{{(n+\log p)}^2{\sigma }^2}{{\mathrm{\Gamma }}^2},n+\frac{{\mathrm{\Xi }}^2}{{\sigma }^2}\right\}+\sqrt{p}\max \left\{\frac{\sigma (n+\log p)}{\mathrm{\Gamma }},\sqrt{n}+\frac{\mathrm{\Xi }}{\sigma }\right\}\right]$ for some C₁ > 0.

Then, up to a permutation reversion,

$$\mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]\le 1\wedge \left(\frac{c_0\sigma }{p\mathrm{\Gamma }}\min \left\{1,e^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\log \left(1+\frac{2{\sigma }^2}{{\mathrm{\Gamma }}^2}\right)\right\}+\frac{c_1{e}^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}}{p(\mathrm{\Gamma }/\sigma +\sqrt{8/\pi })}+\frac{c_2}{p^{c+2}}\right)$$

for some c, c₀, c₁, c₂ > 0.

Condition (i) of Theorem 4 parallels the one given in Theorem 2. It essentially requires an even distancing of the elements (the projected columns of Θ) whose ordering is to be tracked by $\widehat{\pi }$. In contrast, in both Theorem 3 and 4, the conditions on Λ are slightly more complicated than those in Theorem 1 and 2, as it further depends on the relative magnitude between Ξ/σ and $\sqrt{n}$. In particular, if $\mathrm{\Xi }/\sigma \lesssim \sqrt{n}$, the conditions reduce to the ones required in the linear growth models. Interestingly, the risk upper bound obtained in Theorem 4 remains the same as in the linear growth model, which only depends on p and the signal-to-noise ratio Γ /σ.

5. MINIMAX LOWER BOUNDS AND OPTIMALITY

In this section, we establish the minimax lower bounds for both exact and partial recovery considered in previous sections, in relation to different levels of the signal-to-noise ratio Γ /σ. In the following theorem, we show the MSG condition for exact recovery is asymptotically sharp.

Theorem 5. Suppose (A1) hold. Let ${\mathcal{D}}_1={\mathcal{D}}_L\cap \left\{(\mathrm{\Theta },\pi ):\mathrm{\Gamma }\le \frac{\sigma }{4}\sqrt{\log p}\right\}$ and ${\mathcal{D}}_1^{\prime }=\mathcal{D}\cap \left\{(\mathrm{\Theta },\pi ):\mathrm{\Gamma }\le \frac{\sigma }{4}\sqrt{\log p}\right\}$. Then for any p ≥10, we have

$$\underset{\widehat{\pi }}{\inf }\underset{(\mathrm{\Theta },\pi )\in {\mathcal{D}}_1^{\prime }}{\sup }P(\widehat{\pi }\ne \pi )\ge \underset{\widehat{\pi }}{\inf }\underset{(\mathrm{\Theta },\pi )\in {\mathcal{D}}_1}{\sup }P(\widehat{\pi }\ne \pi )\ge 0.3,$$

where the infimum is over all the permutation estimators $\widehat{\pi }$.

This theorem along with Theorem 1 and Theorem 3 indicates that our proposed estimator is minimax rate-optimal over ${\mathcal{D}}_L$ and $\mathcal{D}$ in terms of the MSG condition on Γ. In light of Proposition 2, in some situations the MSG condition can be both necessary and sufficient for the exact recovery, which includes practically important cases such as n ≍ p, n < log p, etc. Using the information-theoretic language, we have therefore obtained both the achievability result, i.e., the existence of an algorithm or estimator that exactly recovers signal with high probability, and the converse result, namely, an upper bound on the probability of exact recovery that applies to any estimators (Cullina and Kiyavash 2016). See Figure 3 for an illustration.

Fig. 3.

A graphical illustration of the achievability/converse result for exact recovery.

Our next theorem establishes a minimax lower bound for the expected rate of convergence for the partial recovery.

Theorem 6. Suppose (A1) hold, ${\mathcal{D}}_2(t)={\mathcal{D}}_L\cap \{(\mathrm{\Theta },\pi ):ct\le \mathrm{\Gamma }\le Ct\},{\mathcal{D}}_2^{\prime }(t)=\mathcal{D}\cap \{(\mathrm{\Theta },\pi ):ct\le \mathrm{\Gamma }\le Ct\}$ for some C, c > 0, and t/σ≥ 2. Then there exist constants C₁, C₂ > 0 such that

$$\underset{\widehat{\pi }}{\inf }\underset{(\mathrm{\Theta },\pi )\in {\mathcal{D}}_2^{\prime }(t)}{\sup }\mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]\ge \underset{\widehat{\pi }}{\inf }\underset{(\mathrm{\Theta },\pi )\in {\mathcal{D}}_2(t)}{\sup }\mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]\ge \frac{C_1\sigma }{pt}{e}^{-t^2/2{\sigma }^2}+\frac{C_2}{p^2}.$$

Comparing the above minimax lower bound to the risk upper bounds obtained in Theorem 2 and 4, we conclude that our proposed estimator $\widehat{\pi }$ is minimax rate-optimal in terms of the partial recovery for both the linear growth model and the general growth model over the range whenever Γ /σ does not diminish (Figure 1). In particular, in Theorem 5 and 6, since the minimax lower bounds only concern the worst-case scenarios, the same lower bounds should hold for any parameter spaces whenever the same worst cases are included. Similarly, the assumption (A1) does not pose a restriction to the general applicability of such results.

6. NUMERICAL STUDIES

6.1. Simulation with Model-Generated Data

To demonstrate our theoretical results and compare with alternative methods, we generate data from model (1) with various configurations of the signal matrix Θ. We compare the empirical performance of our proposed estimator $\widehat{\pi }$ with the following alternatives:

• π_mean : Order the columns of Y by the magnitude of its column means;

• π_max : Order the columns of Y by the magnitude of its column maximums.

We use both the 0–1 loss and the normalized Kendall’s tau distance in comparing these methods. Due to the identifiability issue, the performance of each estimator is evaluated up to a complete reversion of the permutation. For example, we use $\min \left\{{\tau }_K(\widehat{\pi },\pi ),{\tau }_K(\widehat{\pi },\mathrm{rev}(\pi ))\right\}$ as the empirical Kendall’s tau distance. By symmetry, we set the underlying permutation π= id. The signal matrix $\mathrm{\Theta }=\left({\theta }_{ij}\right)\in {ℝ}^{n\times p}$ is generated under the following four regimes:

1. S₁(α, n, p): For any 1 ≤ j ≤ p, θ_ij = log(1 + jα_i +β_i) where α_i ~ Unif(α/2, α) for 1 ≤ i ≤ n/2, α_i ~ Unif(0,0.01) for n/2 < i ≤ n, and β_i ~ Unif(1,3) for all 1 ≤ i ≤ n;

2. S₂(α, n, p): For any 1 ≤ j ≤ p, θ_ij = jα_i + β_i where α_i ~ Unif(α/2, α) for 1 ≤ i ≤ n/2, α_i ~ Unif(0, α/10) for n/2 < i ≤ n, and β_i ~ Unif(1,3) for all 1 ≤ i ≤ n;

3. S₃(α, n, p): For any 1 ≤ j ≤ p, θ_ij = log(1 + jα_i + β_i) where α_i ~ Unif(α/2, α) for 1 ≤ i ≤ 3, α_i ~ Unif(0,0.01) for 4 < i ≤ n, and β_i ~ Unif(1,3) for all 1 ≤ i ≤ n;

4. S₄(α, n, p): For any 1 ≤ j ≤ p, θ_ij = jα_i + β_i where α_i ~ Unif(α/2, α) for 1 ≤ i ≤ 3, α_i ~ Unif(0, α/10) for 4 < i ≤ n, and β_i ~ Unif(1,3) for all 1 ≤ i ≤ n.

Specifically, under each regime, the sample-specific “growth rate” parameter α_i is randomly and uniformly generated either from the interval [α/2, α] or an interval with much smaller values, namely, [0, α/10] in ${\mathcal{S}}_2$ and ${\mathcal{S}}_4$ and [0,0.01] in ${\mathcal{S}}_1$ and ${\mathcal{S}}_3$. By construction, the four regimes consist of the nonlinear growth model where the signals spread out over many samples (${\mathcal{S}}_1$) or concentrate at a few rows (${\mathcal{S}}_3$) and the linear growth model where the signals spread out over many samples (S₂) or concentrate at a few rows (S₄). In particular, in accordance to our theory, for the supposedly “non-informative” samples, we allow the corresponding growth rates to be small but non-zero, which shows the flexibility of our proposed method. The entries of Z are drawn from i.i.d. centred normal distributions whose variance σ₂ will be given explicitly. In each setting, we evaluate the empirical performance of each method over a range of n, p or α. Each setting is repeated for 200 times.

For the exact recovery, in Table 1, we reported the empirical risks of the estimators under the 0–1 loss for various regimes and parameter combinations. The noise level σ² is chosen for each regime to better illustrate the differences in the empirical risks among the estimators. From our simulation results, in consistent to our theory, our proposed estimator has the smallest empirical risk over all the settings, and the estimation risk decreases as we increase α, n or decrease p.

For partial recovery, in Figure 4, we show boxplots of the empirical normalized Kendall’s tau between each estimator and the true permutation π. Again, our proposed method outperforms the alternatives in all the cases. As expected from our theory, under all the four regimes, increasing p while keeping other parameters fixed results to smaller estimation risk. As for the dependence on n, under ${\mathcal{S}}_1$ and ${\mathcal{S}}_2$, increasing n leads to smaller risk as it is equivalent to increasing Γ, whereas under ${\mathcal{S}}_3$ and ${\mathcal{S}}_4$, the risk roughly remains the same across different n’s as in these case Γ doesn’t change much.

To offer more intuitive interpretation of why $\widehat{\pi }$ performs better than the alternative methods, we assessed the weight vectors $\widehat{w}$ of our proposed estimator $\widehat{\pi }$ under each regime after 200 rounds of simulations (Figure 3 in Supplemented Material). In comparison, the weight vector for π_mean is simply $(1/\sqrt{n},\dots ,1/\sqrt{n})$, which assigns equal weight to all the samples. On the other hand, since π_max cannot be written in the form of ${\left(\mathfrak{r}\left(w^{\top }Y\right)\right)}^{-1}$ for some weight vector w and therefore does not belong to the class of linear projection estimators, we reported instead the pseudo-weight vector $\tilde{w}\in {ℝ}^n$ where the i-th component is the proportion that the i-th sample is used among the p coordinates. In general, we found that $\tilde{w}\in {ℝ}^n$ assigns larger weights to only a few samples among those with higher signal strength, and the weight vector for π_mean fails to distinguish the informative samples from the non-informative ones. In contrast, the weight vectors $\widehat{w}$ for our proposed estimator $\widehat{\pi }$ would automatically adapt to the varying signal strengths across the samples and assign larger weights to the samples with more significant signal changes. This also explains the interesting phenomenon in Figure 4 that, under the regime ${\mathcal{S}}_1$ and ${\mathcal{S}}_2$, $\widehat{\pi }$ and π_mean perform better than π_max, whereas under ${\mathcal{S}}_3$ and ${\mathcal{S}}_4$, $\widehat{\pi }$ and π_max perform better. In summary, methods that are able to detect and assign larger weight to the more informative samples would perform better than methods that are not. Observably, $\widehat{\pi }$ combines the advantages of π_mean and π_max in that it finds the best weights (projection scores) in a data-driven manner.

6.2. Evaluation Using Synthetic Metagenomic Data

We evaluate the empirical performance of our proposed method using a synthetic metagenomic sequencing dataset used in Gao and Li (2018) by generating sequencing reads based on 45 bacterial genomes. Instead of estimating the PTRs, which was the focus of Gao and Li (2018), our goal is to recover the unknown relative orders of the contigs assembled in typical metagenomics studies. In addition to assisting the estimation of PTRs, such ordering of the contigs could be of independent interest for other applications, including genome assemblies based on shotgun metagenomics data.

Gao and Li (2018) presented a synthetic shotgun metagenomic sequencing dataset of a community of 45 phylogenetically related species from 15 genera of five different phyla with known RefSeq ID, taxonomy and replication origin (Gao, Luo, and Zhang 2013) (see Figure 2 in our Supplementary Material). To generate metagenomics reads, reference genome sequences of randomly selected three species in each genus were downloaded from NCBI. Read coverages were generated along the genome based on an exponential distribution with a specified peak-to-trough ratio and a function of accumulative distribution of read coverages along the genome was calculated. Sequencing reads were next generated using the above accumulative distribution function and a random location of each read on the genome, until the total read number achieved a randomly assigned average coverage between 0.5 and 10 folds for the species in a sample. Sequencing errors including substitution, insertion and deletion were simulated in a position- and nucleotide-specific pattern according to a recent study on metagenomic sequencing error profiles of Illumina.

For the final dataset, the average nucleotide identities (ANI) between species within each genus ranged from 66.6% to 91.2% The probability of one species existing in each of the 50 simulated samples was set as 0.6, and a total of 1,336 average coverages and the corresponding PTRs were randomly and independently assigned. After the same processing and filtering steps and CG-adjustment step as in Gao and Li (2018), the final dataset included genome assemblies of 41 species. For each species, we obtained the permuted matrix of log-contig counts, with the number of samples ranging from 29 to 46, and the number of contigs ranging from 47 to 482.

Our proposed method $(\widehat{\pi })$ was used to estimate the unknown orders of the contigs for each species and each sample. As a comparison, we also considered the iRep estimator proposed in Brown et al. (2016), where the contigs of a given species were ordered for each sample separately based on the read counts observed. We evaluate these methods by comparing the estimated contig orders to their true orders as measured by the normalized Kendall’s tau distance. To generalize our evaluation to diverse metagenomic datasets, we also evaluate the effect of sample size as well as contig numbers by randomly selecting subsets of samples or contigs from each dataset. The selection was made with replacement.

The results are summarized in Figure 5 by comparing the normalized Kendall’s tau distances. As n or p varies, our proposed estimator performs consistently better than iRep in recovering the true contig orders, which explains partially why the DEMIC algorithm worked better in estimating the bacterial growth dynamics. The results of our methods are not sensitive to the sample size and the number of contigs from the genome assemblies. Our estimator also shows smaller variability.

Fig. 5.

Boxplots of the normalized Kendall’s distance between the estimated contig orders and the true orders for different sample sizes n and different numbers of contigs p. The lighter ones correspond to our proposed method and the darker ones correspond to the iRep estimation method.

6.3. Analysis of a Real Microbiome Metagenomic Data Set

Finally, we complete our numerical studies by analyzing a real metagenomic dataset from the Pediatric Longitudinal Study of Elemental Diet and Stool Microbiome Composition (PLEASE) study, a prospective cohort study to investigate the treatment effects on the gut microbiome and reduction of inflammation in pediatric Crohn’s disease patients (Lewis et al. 2015). In particular, sequencing data from the fecal samples of 86 Crohn’s disease children were obtained at baseline, 1 week and 8 weeks after antiTNF or enteral diet treatment. In our analysis, the sequencing data at the 8th week after treatment was used to compare the bacterial growth dynamics for non-responders (n = 34) and responders (n = 47). The reads were downloaded from NCBI short read archive (SRP057027) with the corresponding metadata. After the same coassembly, alignment and binning steps as in Gao and Li (2018), the DEMIC algorithm was applied to estimate the bacterial growth rate of a given species represented by a contig cluster (bin) for each sample. In particular, DEMIC applied our proposed method to the GC-adjusted contig coverage data to recover the original order of the contigs. After obtaining the ordered contigs, a simple linear regression was fitted to obtain estimates of the PTRs (ePTRs).

In order to compare the baterial growth rates between responders and non-responders, our analysis focused on ePTRs of 8 contig clusters over subsets of the non-responders (n₁) and the responders (n₂) after 8 weeks of treatment with min{n₁, n₂} > 5. Other contig clusters were rare and only appeared in a few samples. For each contig cluster, we compare the ePTRs of the responders and non-responders Wilcoxon rank sum test (Table S.1 in Supplementary Material). The taxonomic annotations of these eight contig clusters were obtain by applying the BAT algorithm (von Meijenfeldt et al. 2019) that compares the metagenomic assembled bins to a taxonomy database. In Table S.1, we show the final taxonomic annotations for each bin to the finest possible resolution, with the lineage scores indicating the quality of each taxonomic classification.

Among the 8 contig clusters, bin.026 showed a significant difference in ePTRs between responders and non-responders after either antiTNF or enteral diet treatment for 8 weeks (p=0.0418), where the growth rate was higher in Crohn’s disease patients who did not respond to the treatment. The taxonomic classification (Table S.1) shows that this contig cluster belongs to the phylum Firmicutes and the order Clostridiales. Since BAT algorithm was not able to classify the order Clostridiales to finer taxonomic level of known species, this contig cluster may represent a new species that is important to the treatment outcome of Crohn’s disease patients.

7. DISCUSSION

In this paper, partial recovery was studied under the normalized Kendall’s tau distance. Another commonly used metric is the normalized Spearman’s footrule distance defined by

$$\rho \left({\pi }_1,{\pi }_2\right)=\frac{2}{p(p-1)}\sum _{i=1}^p\left|{\pi }_1(i)-{\pi }_2(i)\right|,{\pi }_1,{\pi }_2\in {\mathcal{S}}_p.$$

A celebrated result by Diaconis and Graham (1977) shows that ${\tau }_K\left({\pi }_1,{\pi }_2\right)\le \rho \left({\pi }_1,{\pi }_2\right)\le 2{\tau }_K\left({\pi }_1,{\pi }_2\right)$, which means the two distances are equivalent. As a consequence, all the theoretical results presented in this paper concerning the Kendall’s tau distance also hold for the Spearman’s footrule distance without any change.

The minimax optimality of the proposed estimator $\widehat{\pi }$ was investigated in Section 5 by examining the asymptotic sharpness of the MSG condition for exact recovery, and by obtaining the matching minimax risk lower bound for partial recovery. There are a few issues that deserve further investigation. For both exact and partial recovery, it is unclear to what extent the GSS condition is necessary. In our risk analysis, the perturbation bound for the left singular subspace (Cai and Zhang 2018) was used. In fact, similar results can be obtained using the concentration bound for the linear functionals of singular vectors (Koltchinskii and Xia 2016). Nevertheless, it remains to show whether the GSS condition is also asymptotically sharp. In addition, in Theorem 6, the matching minimax lower bound was obtained only for nonvanishing Γ /σ. It remains to show whether the rate σ/(pΓ) is minimax optimal when Γ /σ → 0. The difficulty lies in finding a p^1+δ -sphere packing of the group ${\mathcal{S}}_p$ equipped with the Kendall’s tau distance for any 0 <δ<1, while the pairwise ℓ₂ distances of the packing elements are also well controlled. Some initial steps have been made in the so-called rank modulation theory (Barg and Mazumdar 2010; Mazumdar, Barg, and Zemor 2013).

There are several related problems that are also of significant theoretical and practical interest. Firstly, although we used the Kendall’s tau distance or the equivalent Spearman’s footrule distance as the metric for partial recovery, other distances such the Hamming distance, Spearman’s rank correlation distance, and Ulam’s distance have also been used as the performance metrics for partial recovery in other permutation estimation problems (Göloğlu et al. 2015; Mukherjee 2016). It is therefore of interest to see how $\widehat{\pi }$ performs under these losses. Secondly, our proposed estimator $\widehat{\pi }$ implicitly performs a (linear) dimension reduction technique and only uses the information contained in the first eigenvector of A in (4). A natural extension is to consider the eigen-subspace spanned by the first k eigenvectors and to estimate the permutation in a sequential manner.

The present paper focuses on the estimation of the permutation matrix Π. It is also of interest to estimate the underlying signal matrix Θ or some functionals of it. For example, in microbiome growth dynamics studies, it is of significant interest to estimate the peak-to-trough ratio $\exp \left({\theta }_{kp}-{\theta }_{k1}\right)$ for k =1,...,n, which measures the microbial growth rate for the kth sample, and to identify the samples with peak-to-trough ratio of 1. It is also interesting to identify the bacteria that show differential growth dynamics between disease and normal individuals. Finally, robust permutation recovery methods that can relax the Gaussian or sub-Gaussian assumption of the noise in the permuted monotone matrix model are needed. For example, in some applications, the columns of the noise matrix are not independent, or the variance levels across the noise matrix are not identical. In these cases, we argue that, as long as the marginal distributions of the noise matrix entries remain sub-Gaussian, the analytical framework of the current paper can still be applied, but with more efforts to control the underlying heteroskedasticity. Toward this end, results from the recent work of Zhang, Cai, and Wu (2018) can be very useful, in terms of the new technical tools that parallel the ones used in the current paper to analyse the homoskedastic PCA (cf. Lemma 2 and 3). Finally, to account for non-informative samples, sparse PCA (Cai, Ma, and Wu 2013; Yuan and Zhang 2013) can be considered. These are interesting problems left for future research.

8. PROOFS OF THE MAIN THEOREMS

In this section, we prove Theorems 1 and 2 in detail and briefly sketch the proofs of Theorems 3 and 4. We also prove the minimax lower bounds in Theorems 5 and 6. Proofs of other results including the technical lemmas can be found in the online Supplementary Materials.

Proof of Theorem 1

Let X =Θ+Z. It follows that Y = XΠ. By right invariance of the 0–1 loss with respect to permutation composition, we have

$$\ell ({(\mathfrak{r}({\widehat{w}}^{\top }Y))}^{-1},\pi )=\ell ({(\mathfrak{r}({\widehat{w}}^{\top }X\mathrm{\Pi }))}^{-1},\pi )=\ell ({(\mathfrak{r}({\widehat{w}}^{\top }X))}^{-1}\circ \pi ,\pi )=\ell ({(\mathfrak{r}({\widehat{w}}^{\top }X))}^{-1},id).$$

Thus it suffices to study the risk $\mathbb{E}\ell ({(\mathfrak{r}({\widehat{w}}^{\top }X))}^{-1},id)=P({(\mathfrak{r}({\widehat{w}}^{\top }X))}^{-1}\ne id)$. In fact,

$$\begin{gathered} P\left({\left(\mathfrak{r}\left({\widehat{w}}^{\top }X\right)\right)}^{-1}\ne id\right)\le P\left(\underset{i=1}{\overset{p-1}{\cup }}\left\{\sum _{k=1}^n{\widehat{w}}_k{X}_{ki}\ge \sum _{k=1}^n{\widehat{w}}_k{X}_{k,i+1}\right\}\right) \\ \le \sum _{i=1}^{p-1}P\left(\sum _{k=1}^n{\widehat{w}}_k{X}_{ki}\ge \sum _{k=1}^n{\widehat{w}}_k{X}_{k,i+1}\right)=\sum _{i=1}^{p-1}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{k,i+1}\right)\ge 0\right), \end{gathered}$$ (13)

which further reduces to obtaining an upper bound for $P_i=P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{k,i+1}\right)\ge 0\right)$. By definition, $\widehat{w}$ is the first eigenvector of $A=Y\left(I-\frac{1}{p}e{e}^{\top }\right)\left(I-\frac{1}{p}e{e}^{\top }\right)Y^{\top }$. Simple calculation yields $\mathrm{\Pi }\left(I-\frac{1}{p}e{e}^{\top }\right)\left(I-\frac{1}{p}e{e}^{\top }\right){\mathrm{\Pi }}^{\top }=\left(I-\frac{1}{p}e{e}^{\top }\right)\left(I-\frac{1}{p}e{e}^{\top }\right)$ for any $\mathrm{\Pi }\in {\mathcal{S}}_p$. So $\widehat{w}$ is also the first eigenvector of $A=X\left(I-\frac{1}{p}e{e}^{\top }\right)\left(I-\frac{1}{p}e{e}^{\top }\right)X^{\top }\equiv T{T}^{\top }$, where $T\in {ℝ}^{n\times p}$. Note that T admits the decomposition $T={\mathrm{\Theta }}^{\prime }+E\in {ℝ}^{n\times p}$ where $E_{ij}\sim N\left(0,(p-1){\sigma }^2/p\right)$ and ${\mathrm{\Theta }}^{\prime }=a{\eta }^{\prime \top },{\eta }_j^{\prime }={\eta }_j-\frac{1}{p}\sum _{i=1}^p{\eta }_i$. In particular, $T_{.i}=X_{.i}-{\overline{X}}_{row}$ where ${\overline{X}}_{row}=p^{-1}\sum _{i=1}^p{X}_{.i}\in {ℝ}^n$ is the vector of row means of X. We denote ϕ_ij = T_.i − T_.j = X_.i − X_.j and denote $w=a/\Vert a{\Vert }_2\in {ℝ}^n$ as the first eigenvector of the rank-one matrix Θ′Θ′^⊤. Now following (13), we have

$$\begin{gathered} P_i=P\left(\sum _{k=1}^n{\widehat{w}}_k(X_{ki}-X_{k,i+1}\right)\ge 0)=P\left(w^{\top }{\varphi }_{ij}+{(\widehat{w}-w)}^{\top }{\varphi }_{ij}\ge 0\right) \\ =P\left(w^{\top }{\varphi }_{i,i+1}+{(\widehat{w}-w)}^{\top }{\varphi }_{ij}\ge 0,\left|1-{\left({\widehat{w}}^{\top }w\right)}^2\right|\le \delta \right) \\ +P\left(w^{\top }{\varphi }_{i,i+1}+{(\widehat{w}-w)}^{\top }{\varphi }_{ij}\ge 0,\left|1-{\left({\widehat{w}}^{\top }w\right)}^2\right|>\delta \right) \end{gathered}$$

for some δ > 0. By definition, up to a change of sign for $\widehat{w}$, we have $0\le {\widehat{w}}^{\top }w\le 1$. Then $\left|1-{\left({\widehat{w}}^{\top }w\right)}^2\right|\le \delta$ implies $\left|{(\widehat{w}-w)}^{\top }{\varphi }_{i,i+1}\right|\le \Vert \widehat{w}-w{\Vert }_2{\Vert {\varphi }_{i,i+1}\Vert }_2\le \sqrt{2\delta }{\Vert {\varphi }_{i,i+1}\Vert }_2$, where the first inequality follows from Cauchy-Schwartz and the second inequality used $\Vert \widehat{w}-w{\Vert }_2=\sqrt{2\left(1-{\widehat{w}}^{\top }w\right)}\le \sqrt{2\left(1-{\left({\widehat{w}}^{\top }w\right)}^2\right)}$. Thus

$$P_i\le P\left(w^{\top }{\varphi }_{i,i+1}\ge -\sqrt{2\delta }{\Vert {\varphi }_{i,i+1}\Vert }_2\right)+P\left(\left|1-{\left({\widehat{w}}^{\top }w\right)}^2\right|>\delta \right).$$ (14)

The following lemmas provide upper bounds for the two probability events in the last expression.

Lemma 1. Under the conditions of Theorem 1, denote ${\mathrm{\Gamma }}_i=\Vert a{\Vert }_2\left({\eta }_i-{\eta }_{i+1}\right)$, then for any δ > 0, we have

$$P\left(w^{\top }{\varphi }_{i,i+1}\ge -\sqrt{2\delta }\Vert {\varphi }_{i,i+1}\Vert \right)\le \mathrm{\Phi }\left(C_1\sqrt{\delta }{\mathrm{\Psi }}_i^{1/2}+\frac{{\mathrm{\Gamma }}_i}{\sigma }\right)+\frac{C_2}{p^c}$$ (15)

for ${\mathrm{\Psi }}_i={(\sqrt{n}+\sqrt{\log p})}^2+\frac{{\mathrm{\Gamma }}_i^2}{{\sigma }^2}+\frac{\left|{\mathrm{\Gamma }}_i\right|}{\sigma }\sqrt{\log p}$ and some constants C₁, C₂, c > 0.

Lemma 2. Suppose ${\lambda }_1^2\left({\mathrm{\Theta }}^{\prime }\right)\ge C{\sigma }^2(n+\sqrt{np})$ for some C > 0, it follows that

$$P\left(\left|1-{\left({\widehat{w}}^{\top }w\right)}^2\right|\le C_1\frac{{\sigma }^2\left({\lambda }_1^2\left({\mathrm{\Theta }}^{\prime }\right)+{\sigma }^{2}p\right)(n+\log p)}{{\lambda }_1^4\left({\mathrm{\Theta }}^{\prime }\right)}\right)\ge 1-\frac{C_2}{p^c}$$

for some C₁, C₂, c > 0.

Now since $\frac{1}{p}\sum _{1\le i<j\le p}{\left({\eta }_i-{\eta }_j\right)}^2=\sum _{j=1}^p{\left({\eta }_j-\frac{1}{p}\sum _{i=1}^p{\eta }_i\right)}^2=\sum _{j=1}^p\eta {\prime }_j^2$, we have ${\lambda }_1^2\left({\mathrm{\Theta }}^{\prime }\right)=\mathrm{\Lambda }>C_0{\sigma }^2(n+\sqrt{np})$ for some C₀ > 0. Set $\delta =C_0{\sigma }^2\frac{\left({\lambda }_1^2\left({\mathrm{\Theta }}^{\prime }\right)+{\sigma }^{2}p\right)(n+\log p)}{{\lambda }_1^4\left({\mathrm{\Theta }}^{\prime }\right)}$. It follows that δ= o(1). Combining Lemma 1 and Lemma 2, we have

$$P_i\le \mathrm{\Phi }\left(C\sqrt{\delta }{\left[{(\sqrt{n}+\sqrt{\log p})}^2+\frac{{\mathrm{\Gamma }}_i^2}{{\sigma }^2}+\frac{\left|{\mathrm{\Gamma }}_i\right|}{\sigma }\sqrt{\log p}\right]}^{1/2}+\frac{{\mathrm{\Gamma }}_i}{\sigma }\right)+\frac{C}{p^c}$$ (16)

for some C, c > 0. The rest of the analysis is divided into several cases.

Case 1. log p≲n.

In this case, we have $P_i\le \mathrm{\Phi }\left(C\sqrt{\delta }{\left[n+\frac{{\mathrm{\Gamma }}_i^2}{{\sigma }^2}+\frac{\left|{\mathrm{\Gamma }}_i\right|}{\sigma }\sqrt{\log p}\right]}^{1/2}+\frac{{\mathrm{\Gamma }}_i}{\sigma }\right)+\frac{C}{p^c}$. In addition, if $\left|{\mathrm{\Gamma }}_i\right|/\sigma \lesssim \sqrt{n}$, we have $P_i\le \mathrm{\Phi }\left(C\sqrt{\delta n}+\frac{{\mathrm{\Gamma }}_i}{\sigma }\right)+\frac{C}{p^c}\le \frac{C^{\prime }}{p^c}$, where the last inequality follows from $\sqrt{\log p}\lesssim \mathrm{\Gamma }/\sigma \le \left|{\mathrm{\Gamma }}_i\right|/\sigma \lesssim \sqrt{n}$ and $\mathrm{\Lambda }\gtrsim {\sigma }^{2}n\left(\frac{{\sigma }^{2}n}{{\mathrm{\Gamma }}^2}+\frac{\sigma \sqrt{p}}{\mathrm{\Gamma }}\right)$. If instead $\left|{\mathrm{\Gamma }}_i\right|/\sigma \gtrsim \sqrt{n}$, we have $P_i\le \mathrm{\Phi }\left(C\sqrt{\delta }\frac{\left|{\mathrm{\Gamma }}_i\right|}{\sigma }+\frac{{\mathrm{\Gamma }}_i}{\sigma }\right)+\frac{C}{p^c}\le \frac{C^{\prime }}{p^c}$, where the last inequality follows from $\left|{\mathrm{\Gamma }}_i\right|/\sigma \gtrsim \sqrt{n}\gtrsim \sqrt{\log p}$ and δ= o(1). Hence, in Case 1, (16) can be bounded by O(p^−c).

Case 2. log p≳n.

In this case, we have $P_i\le \mathrm{\Phi }\left(C\sqrt{\delta }{\left[\log p+\frac{{\mathrm{\Gamma }}_i^2}{{\sigma }^2}+\frac{\left|{\mathrm{\Gamma }}_i\right|}{\sigma }\sqrt{\log p}\right]}^{1/2}+\frac{{\mathrm{\Gamma }}_i}{\sigma }\right)+\frac{C}{p^c}$. In addition, since $\left|{\mathrm{\Gamma }}_i\right|\ge \mathrm{\Gamma }\gtrsim \sigma \sqrt{\log p}$ and δ= o(1), we have $P\left(\sum _{k=1}^n{\widehat{u}}_k\left(X_{ki}-X_{k,i+1}\right)\ge 0\right)\le \mathrm{\Phi }\left(\frac{C\sqrt{\delta }}{\sigma }\left|{\mathrm{\Gamma }}_i\right|+\frac{{\mathrm{\Gamma }}_i}{\sigma }\right)+\frac{C}{p^c}\le \frac{C^{\prime }}{p^c}$. This shows that, in Case 2, (16) can also be bounded by O(p^−c).

As a result, it follows that, up to a change of sign for $\widehat{w},P({(\mathfrak{r}({\widehat{w}}^{\top }X))}^{-1}\ne id)=O\left(p^{-c}\right)$ for some constant c > 0. □

Proof of Theorem 2

Firstly, by invariance property of Kendall’s tau distance, $\mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]=\mathbb{E}\left[{\tau }_K\left({\left(\mathfrak{r}\left({\widehat{w}}^{\top }X\right)\right)}^{-1},id\right)\right]=\mathbb{E}\left[{\tau }_K\left(\left(\mathfrak{r}\left({\widehat{w}}^{\top }X\right)\right),id\right)\right]$. It then follows

$$\begin{gathered} \mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]=\frac{2}{p(p-1)}\sum _{i<j}P\left({\left[\mathfrak{r}\left({\widehat{w}}^{\top }X\right)\right]}_i\ge {\left[\mathfrak{r}\left({\widehat{w}}^{\top }X\right)\right]}_j\right) \\ =\frac{2}{p(p-1)}\sum _{i<j}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right). \end{gathered}$$

The summation in the last expression can be divided into two parts, namely, the consecutive differences and non-consecutive differences, i.e.,

$$\begin{gathered} \sum _{i<j}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right)=\sum _{(i,j):j=i+1}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right) \\ +\sum _{(i,j):j>i+1}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right). \end{gathered}$$

In the following, we first show

$$P_i=P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{k,i+1}\right)\ge 0\right)\le \frac{c{e}^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}}{\mathrm{\Gamma }/\sigma +\sqrt{{\mathrm{\Gamma }}^2/{\sigma }^2+8/\pi }}+\frac{C}{p^c}$$ (17)

so that

$$\sum _{(i,j):j=i+1}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right)\le \frac{cp{e}^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}}{\mathrm{\Gamma }/\sigma +\sqrt{{\mathrm{\Gamma }}^2/{\sigma }^2+8/\pi }}+\frac{C}{p^c}.$$ (18)

for some C, c> 0. Then we show that

$$\sum _{(i,j):j>i+1}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right)\le C\frac{p\sigma }{\mathrm{\Gamma }}\min \left\{1,e^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\log \left(1+\frac{2{\sigma }^2}{{\mathrm{\Gamma }}^2}\right)\right\}+\frac{C}{p^c}.$$ (19)

Combining (18) and (19), we conclude that

$$\mathbb{E}\left[{\tau }_K\left({\widehat{w}}^{\top }Y,\pi \right)\right]\le \frac{C\sigma }{p\mathrm{\Gamma }}\min \left\{1,e^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\log \left(1+\frac{2{\sigma }^2}{{\mathrm{\Gamma }}^2}\right)\right\}+\frac{C{e}^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}}{p(\mathrm{\Gamma }/\sigma +\sqrt{8/\pi })}+\frac{C}{p^{c+2}},$$

which completes the proof, as the bound $\mathbb{E}\left[{\tau }_K\left({\widehat{w}}^{\top }Y,\pi \right)\right]\le 1$ is trivial.

Proof of (17). Following the same argument as the proof of Theorem 1, we have for 1 ≤ i ≤ p −1 and $\delta =\frac{{\sigma }^2(n+\log p)\left({\lambda }_1^2+{\sigma }^{2}p\right)}{{\lambda }_1^4},P_i\le P\left(w^{\top }{\varphi }_{ij}\ge -\sqrt{2\delta }{\Vert {\varphi }_{i,i+1}\Vert }_2\right)+P\left(\left|1-{\left({\widehat{w}}^{\top }w\right)}^2\right|>\delta \right),$ where the second term can be bounded using Lemma 2. For the first term, by Lemma 1, for ${\lambda }_1^4\left({\mathrm{\Theta }}^{\prime }\right)\ge {\sigma }^2\left(p+{\sigma }^2{\lambda }_1^2\left({\mathrm{\Theta }}^{\prime }\right)\right)(\log p+n)$, we have $P_i\le \mathrm{\Phi }\left(C\sqrt{\delta }{\left[{(\sqrt{n}+\sqrt{\log p})}^2+\frac{{\mathrm{\Gamma }}_i^2}{{\sigma }^2}+\frac{\left|{\mathrm{\Gamma }}_i\right|}{\sigma }\sqrt{\log p}\right]}^{1/2}+\frac{{\mathrm{\Gamma }}_i}{\sigma }\right)+\frac{C}{p^c}$.

Using same argument as the proof of Theorem 1, it holds that $P_i\le \mathrm{\Phi }\left(\frac{{\mathrm{\Gamma }}_i}{\sigma }\right)+\frac{C}{p^c}$. Equation (17) then follows by using formula 7.1.13 of Abramowitz and Stegun (1965) that $\mathrm{\Phi }(-t)<\frac{2}{t+\sqrt{t^2+8/\pi }}\varphi (t)$ for t ≥ 0.

Proof of (19). For the set of indices S = {(i, j) :1 ≤ i < j ≤ p, j > i +1}, we further divide it into two subsets $S_1=\{(i,j):1\le i<j\le p,j>i+\lfloor \sigma \sqrt{C\log p}/\mathrm{\Gamma }\rfloor \}$ and $S_2=\{(i,j):1\le i<j\le p,i+1<j\le i+\lfloor \sigma \sqrt{C\log p}/\mathrm{\Gamma }\rfloor \}$ for some constant C > 0. Apparently we have the decomposition

$$\begin{gathered} \sum _{(i,j):j>i+1}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right)=\sum _{(i,j)\in S_1}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right) \\ +\sum _{(i,j)\in S_2}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right) \end{gathered}$$ (20)

For the first term, by construction, it can be shown using the same argument (see supplementary materials) in Theorem 1 that

$$\sum _{(i,j)\in S_1}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right)\le \frac{C\left|S_1\right|}{p^c}\le \frac{C}{p^{c_0}}.$$ (21)

Now for the second term in (20), similar argument yields, for (i, j) ∈ S₂,

$$P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right)\le \frac{c\exp \left(-|i-j{|}^2{\mathrm{\Gamma }}^2/\left(2{\sigma }^2\right)\right)}{|i-j|\mathrm{\Gamma }/\sigma +\sqrt{|i-j{|}^2{\mathrm{\Gamma }}^2/{\sigma }^2+8/\pi }}+\frac{C}{p^c}.$$

Note that, on the one hand,

$$\frac{e^{-|i-j{|}^2{\mathrm{\Gamma }}^2/\left(2{\sigma }^2\right)}}{|i-j|\mathrm{\Gamma }/\sigma +\sqrt{|i-j{|}^2{\mathrm{\Gamma }}^2/{\sigma }^2+8/\pi }}\le \frac{\sigma e^{-|i-j{|}^2{\mathrm{\Gamma }}^2/2{\sigma }^2}}{|i-j|\mathrm{\Gamma }}.$$

We have

$$\sum _{(i,j)\in S_2}\frac{\sigma e^{-|i-j{|}^2{\mathrm{\Gamma }}^2/2{\sigma }^2}}{|i-j|\mathrm{\Gamma }}=\frac{\sigma }{\mathrm{\Gamma }}\sum _{k=2}^{p\wedge \lfloor \sqrt{\log p}/\mathrm{\Gamma }\rfloor }\left(\frac{e^{-k^2{\mathrm{\Gamma }}^2/2{\sigma }^2}p}{k}-e^{-k^2{\mathrm{\Gamma }}^2/2{\sigma }^2}\right)=T_1-T_2.$$

For the rest of the proof, we assume $\mathrm{\Gamma }\le \sigma \sqrt{\log p}/2$, otherwise the set ${\mathcal{S}}_2$ will vanish. Then

$$T_1=\frac{\sigma p}{\mathrm{\Gamma }}\sum _{k=2}^{p\wedge \lfloor \sigma \sqrt{\log p}/\mathrm{\Gamma }\rfloor }\frac{e^{-k^2{\mathrm{\Gamma }}^2/2{\sigma }^2}}{k}\le \frac{\sigma p}{\mathrm{\Gamma }}{\int }_1^{p\wedge \sigma \sqrt{\log p}/\mathrm{\Gamma }}\frac{e^{-x^2{\mathrm{\Gamma }}^2/2{\sigma }^2}}{x}dx$$

where the last inequality used monotonicity of the integrand. The integral in the last inequality, after change of variable, can be bounded by an exponential integral Ei(Γ² / 2σ² ), which has an upper bound

$${\int }_1^{p\wedge \sigma \sqrt{\log p}/\mathrm{\Gamma }}\frac{e^{-x^2{\mathrm{\Gamma }}^2/2{\sigma }^2}}{x}dx=\frac{1}{2}{\int }_{{\mathrm{\Gamma }}^2/2{\sigma }^2}^{\left(p^2{\mathrm{\Gamma }}^2/2{\sigma }^2\right)\wedge (\log p/2)}\frac{e^{-t}}{t}dt\le \frac{1}{2}{\int }_{{\mathrm{\Gamma }}^2/2{\sigma }^2}^{\infty }\frac{e^{-t}}{t}dx\le e^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\log \left(1+\frac{2{\sigma }^2}{{\mathrm{\Gamma }}^2}\right)$$

so that $T_1\le \frac{\sigma p}{\mathrm{\Gamma }}{e}^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\log \left(1+\frac{2{\sigma }^2}{{\mathrm{\Gamma }}^2}\right)$. For T₂, we have $T_2=\frac{\sigma }{\mathrm{\Gamma }}\sum _{k=2}^{p\wedge \lfloor \sigma \sqrt{\log p}/\mathrm{\Gamma }\rfloor }{e}^{-k^2{\mathrm{\Gamma }}^2/2{\sigma }^2}\ge \frac{\sigma }{\mathrm{\Gamma }}{e}^{-2{\mathrm{\Gamma }}^2/{\sigma }^2}$. Therefore,

$$\sum _{(i,j)\in S_2}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right)\le \frac{C\sigma p}{\mathrm{\Gamma }}{e}^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\log \left(1+\frac{2{\sigma }^2}{{\mathrm{\Gamma }}^2}\right)+\frac{C}{p^c}.$$ (22)

On the other hand, note that

$$\frac{e^{-|i-j{|}^2{\mathrm{\Gamma }}^2/\left(2{\sigma }^2\right)}}{|i-j|\mathrm{\Gamma }/\sigma +\sqrt{|i-j{|}^2{\mathrm{\Gamma }}^2/{\sigma }^2+8/\pi }}\le c{e}^{-|i-j{|}^2{\mathrm{\Gamma }}^2/\left(2{\sigma }^2\right)}.$$

We have

$$\begin{gathered} \sum _{(i,j)\in S_2}{e}^{-|i-j{|}^2{\mathrm{\Gamma }}^2/\left(2{\sigma }^2\right)}=\sum _{k=2}^{p\wedge \lfloor \sigma \sqrt{\log p}/\mathrm{\Gamma }\rfloor }p{e}^{-k^2{\mathrm{\Gamma }}^2/\left(2{\sigma }^2\right)}-\sum _{k=2}^{p\wedge \lfloor \sigma \sqrt{\log p}/\mathrm{\Gamma }\rfloor }k{e}^{-k^2{\mathrm{\Gamma }}^2/\left(2{\sigma }^2\right)} \\ \le p{\int }_1^{\infty }{e}^{-k^2{\mathrm{\Gamma }}^2/2{\sigma }^2}dk-2e^{-2{\mathrm{\Gamma }}^2/{\sigma }^2}\le Cp\sigma /\mathrm{\Gamma }. \end{gathered}$$

Thus

$$\sum _{(i,j)\in S_2}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right)\le Cp\sigma /\mathrm{\Gamma }+\frac{C^{\prime }}{p^c}.$$ (23)

Combining (22) and (23), we have

$$\sum _{(i,j)\in S_2}P\left(\sum _{k=1}^n{\widehat{w}}_k\left(X_{ki}-X_{kj}\right)\ge 0\right)\le C\frac{p\sigma }{\mathrm{\Gamma }}\min \left\{1,e^{-{\mathrm{\Gamma }}^2/2{\sigma }^2}\log \left(1+\frac{2{\sigma }^2}{{\mathrm{\Gamma }}^2}\right)\right\}+\frac{C}{p^c}$$ (24)

Combining (21) and (24), we have (19). □

Proof of Theorem 3 and Theorem 4

Here we only provide a sketch of the proofs. We refer the readers to our Supplementary Material for detailed proofs. The proofs follow essentially from the same argument as the proofs of Theorem 1 and Theorem 2, respectively. However, in place of Lemma 2 used therein, we need the following lemma that provides a perturbation bound for the leading eigenvector of approximate rank-one matrices, which could be of independent interest.

Lemma 3. Suppose p≳n and ${\lambda }_1^2\left({\mathrm{\Theta }}^{\prime }\right)\ge {\lambda }_2^2\left({\mathrm{\Theta }}^{\prime }\right)+C{\sigma }^2(n+\sqrt{np})$ for some C > 0. Let $w=u_1^{\prime }$ be the first left singular vector of Θ′, it follows that,

$$P\left(\left|1-{\left({\widehat{w}}^{\top }w\right)}^2\right|\le \frac{C{\sigma }^2\left({\lambda }_1^2\left({\mathrm{\Theta }}^{\prime }\right)+{\sigma }^{2}p\right)(n+\log p)}{{\left({\lambda }_1^2\left({\mathrm{\Theta }}^{\prime }\right)-{\lambda }_2^2\left({\mathrm{\Theta }}^{\prime }\right)\right)}^2}\right)\ge 1-\frac{C}{p^c}.$$

The proof of Lemma 3 is nontrivial, which depends on a combination of the generic perturbation bound obtained by Cai and Zhang (2018) and new concentration inequalities of approximate rank-one matrices (see Supplementary Materials).

Proof of Theorem 5

The proof relies on the following lemma adapted from (Tsybakov 2009).

Lemma 4. Assume that for some integer M ≥ 2 there exist distinct parameters θ₀,..., θ_M from the parameter space Θ and mutually absolutely continuous probability measures P₀,...,P_M with $P_j=P_{{\theta }_j}$ for j = 0,1,...,M, defined on a common probability space $(\mathrm{\Omega },\mathcal{F})$ such that the averaged K-L divergence $\frac{1}{M}\sum _{j=1}^{M}D\left(P_j,P_0\right)\le \frac{1}{8}\log M$. Then, for every measurable mapping $\widehat{\theta }:\mathrm{\Omega }\to \mathrm{\Theta }$, $\underset{j=0,\dots M}{\max }{P}_j\left(\widehat{\theta }\ne {\theta }_j\right)\ge \frac{\sqrt{M}}{\sqrt{M}+1}\left(\frac{3}{4}-\frac{1}{2\sqrt{\log M}}\right)$.

We construct the (M +1) =p points parameter space as follows. We define p permutations from ${\mathcal{S}}_p$ as an identity plus (p −1) consecutive swaps, i.e., π₀ = id, π_k = (k, k +1) for k = 1,..., p −1. The signal matrix Θ₀ = aη^⊤ where $a={(1,\dots ,1)}^{\top }\in {ℝ}^n$ and $\eta ={(0,\delta ,\dots ,(p-1)\delta )}^{\top }\in {ℝ}^p$, $\delta =\frac{\sigma }{4}\sqrt{\log p/n}$. In this way, we have $\mathrm{\Gamma }=\Vert a{\Vert }_2\cdot {\min }_{1\le i\le p-1}\left|{\eta }_j-{\eta }_{j+1}\right|=\frac{\sigma }{4}\sqrt{\log p}$. Let P_k corresponds to the joint probability measure of Y under (Θ₀, π_k) for k = 0,1,..., p −1, and let p_k be the pdf of P_k, we have $p_0(x)=\prod _{i=1}^n\prod _{j=1}^p{\varphi }_{{\eta }_j}\left(x_{ij}\right)$, $p_k(x)=\prod _{i=1}^n\prod _{j=1}^p{\varphi }_{{\eta }_{{\pi }_k(j)}}\left(x_{ij}\right)$ for k = 1,..., p −1, where ϕ_μ is the pdf of Gaussian distribution N(μ,σ²). Now we calculate the KL-divergence

$$\begin{gathered} D\left(P_k,P_0\right)=\int \log \left(\frac{p_k(x)}{p_0(x)}\right)p_0(x)dx=\int \frac{n}{2{\sigma }^2}\sum _{i=1}^p\left[{\left(x_{1j}-{\eta }_{{\pi }_k(j)}\right)}^2-{\left(x_{1j}-{\eta }_j\right)}^2\right]p_0(x)dx \\ =\frac{n{\delta }^2}{{\sigma }^2}=\frac{\log p}{16}. \end{gathered}$$

Then, we have for $p\ge 10,\frac{1}{p-1}\sum _{k=1}^{p-1}D\left(P_k,P_0\right)=\frac{\log p}{16}\le \frac{1}{8}\log (p-1)$. It follows from Lemma 4 that, $\underset{\widehat{\pi }}{\inf }\underset{(\pi ,\mathrm{\Theta })\in {\mathcal{D}}_1}{\sup }P(\widehat{\pi }\ne \pi )\ge \underset{\widehat{\mathbf{\pi }}}{\inf }{\max }_{j=0,\dots ,p-1}{P}_j\left(\widehat{\pi }\ne {\pi }_j\right)\ge 0.3$ as long as p ≥ 10. In addition, $\underset{\widehat{\pi }}{\inf }\underset{(\pi ,\mathrm{\Theta })\in {\mathcal{D}}_1^{\prime }}{\sup }P(\widehat{\pi }\ne \pi )\ge \underset{\widehat{\pi }}{\inf }\underset{(\pi ,\mathrm{\Theta })\in {\mathcal{D}}_1}{\sup }P(\widehat{\pi }\ne \pi )$ as ${\mathcal{D}}_1\subset {\mathcal{D}}_1^{\prime }$. □

Proof of Theorem 6

The proof relies on the following lemma from Tsybakov (2009).

Lemma 5. Assume that M ≥ 2 and suppose that Θ contains elements θ₀, θ₁,..., θ_M such that: (i) d(θ_j, θ_k) ≥ 2s > 0 for any 0 ≤ j < k ≤ M; (ii) for any $j=1,\dots ,M,\frac{1}{M}\sum _{j=1}^{M}D\left(P_j,P_0\right)\le \alpha \log M$ with 0 < α < 1/8 and $P_j=P_{{\theta }_j}$ for j = 0,1,...,M. Then

$$\underset{\widehat{\theta }}{\inf }\underset{\theta \in \mathrm{\Theta }}{\sup }{P}_{\theta }(d(\widehat{\theta },\theta )\ge s)\ge \frac{\sqrt{M}}{1+\sqrt{M}}(1-2\alpha -\sqrt{\frac{2\alpha }{\log M}})>0.$$

.

We also need the following sphere packing lemma proved by Mao, Weed, and Rigollet (2017), which is a direct consequence of the well-celebrated Varshamov-Gilbert bound.

Lemma 6. For any r < p / 2, there exists a subset ${\mathcal{Q}}_r$ of ${\mathcal{S}}_p$ such that (i) $\log \left|{\mathcal{Q}}_r\right|\ge \frac{r}{5}\log (p/r)$, (ii) for any elements ${\pi }_1,{\pi }_2\in {\mathcal{Q}}_r$, we have $\left(\begin{array}{l}p\hfill \\ 2\hfill \end{array}\right)\cdot {\tau }_K\left({\pi }_1,{\pi }_2\right)\ge r$, and (iii) for any $\pi \in {\mathcal{Q}}_r$, we have $\Vert \pi -id{\Vert }_2^2\le 2r$.

For t / σ ≥ 2, we set $r=\frac{p\sigma }{t}{e}^{-t^2/2{\sigma }^2}<p/2$. Let π₀ = id and ${\pi }_1,\dots ,{\pi }_{\left|{\mathcal{Q}}_r\right|}$ be the elements of ${\mathcal{Q}}_r$. The signal matrix Θ₀ = aη^⊤ where $a={(1/\sqrt{320n},\dots ,1/\sqrt{320n})}^{\top }\in {ℝ}^n$ and $\eta ={(t,\dots ,pt)}^{\top }\in {ℝ}^p$. Let P_k be the joint probability measure of Y under (Θ_0, π_k) for $k=0,1,\dots ,\left|{\mathcal{Q}}_r\right|$, and let p_k be the pdf of P_k. By Lemma 6, the KL-divergence

$$D\left(P_k,P_0\right)=\int \log \left(\frac{p_k(x)}{p_0(x)}\right)p_0(x)dx=\frac{t^2}{320{\sigma }^2}{\Vert {\pi }_k-id\Vert }_2^2\le \frac{pt}{160\sigma }{e}^{-t^2/2{\sigma }^2}.$$

and therefore

$$\frac{1}{p-1}\sum _{k=1}^{p-1}D\left(P_k,P_0\right)\le \frac{pt}{160\sigma }{e}^{-t^2/2{\sigma }^2}\le \frac{p\sigma }{80t}{e}^{-t^2/2{\sigma }^2}\log \left(\frac{t}{\sigma }{e}^{t^2/2{\sigma }^2}\right)\le \frac{1}{16}\log |{\mathcal{Q}}_r|.$$

Without loss of generality, we assume $\left|{\mathcal{Q}}_r\right|\ge 2$. By Lemma 5, it then follows that,

$$\underset{\widehat{\pi }}{\inf }\underset{(\mathrm{\Theta },\pi )\in {\mathcal{D}}_2(t)}{\sup }P\left({\tau }_K(\widehat{\pi },\pi )\ge \frac{\sigma }{2pt}{e}^{-t^2/2{\sigma }^2}\right)\ge C_1,$$

for some absolute constant C₁ > 0. By Markov’s inequality, we have

$$\underset{\widehat{\pi }}{\inf }\underset{(\mathrm{\Theta },\pi )\in {\mathcal{D}}_2(t)}{\sup }\mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]\ge \frac{\sigma }{2pt}{e}^{-t^2/2{\sigma }^2}\underset{\widehat{\pi }}{\inf }\underset{(\mathrm{\Theta },\pi )\in {\mathcal{D}}_2(t)}{\sup }P\left({\tau }_K(\widehat{\pi },\pi )\ge \frac{\sigma }{2pt}{e}^{-t^2/2{\sigma }^2}\right)\ge \frac{C_1\sigma }{pt}{e}^{-t^2/2{\sigma }^2}.$$

.

The relationship $\underset{\widehat{\pi }}{\inf }\underset{(\mathrm{\Theta },\pi )\in {\mathcal{D}}_2^{\prime }(t)}{\sup }\mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]\ge \underset{\widehat{\pi }}{\inf }\underset{(\mathrm{\Theta },\pi )\in {\mathcal{D}}_2(t)}{\sup }\mathbb{E}\left[{\tau }_K(\widehat{\pi },\pi )\right]$ follows from ${\mathcal{D}}_L\subset \mathcal{D}$. The rate 1 / p² follows by setting $t=C_2\sigma \sqrt{\log p}$ for some C₂ > 0. □