Phase transitions in semidefinite relaxations

Significance

Modern data analysis requires solving hard optimization problems with a large number of parameters and a large number of constraints. A successful approach is to replace these hard problems by surrogate problems that are convex and hence tractable. Semidefinite programming relaxations offer a powerful method to construct such relaxations. In many instances it was observed that a semidefinite relaxation becomes very accurate when the noise level in the data decreases below a certain threshold. We develop a new method to compute these noise thresholds (or phase transitions) using ideas from statistical physics.

Abstract

Statistical inference problems arising within signal processing, data mining, and machine learning naturally give rise to hard combinatorial optimization problems. These problems become intractable when the dimensionality of the data is large, as is often the case for modern datasets. A popular idea is to construct convex relaxations of these combinatorial problems, which can be solved efficiently for large-scale datasets. Semidefinite programming (SDP) relaxations are among the most powerful methods in this family and are surprisingly well suited for a broad range of problems where data take the form of matrices or graphs. It has been observed several times that when the statistical noise is small enough, SDP relaxations correctly detect the underlying combinatorial structures. In this paper we develop asymptotic predictions for several detection thresholds, as well as for the estimation error above these thresholds. We study some classical SDP relaxations for statistical problems motivated by graph synchronization and community detection in networks. We map these optimization problems to statistical mechanics models with vector spins and use nonrigorous techniques from statistical mechanics to characterize the corresponding phase transitions. Our results clarify the effectiveness of SDP relaxations in solving high-dimensional statistical problems.

Many information processing tasks can be formulated as optimization problems. This idea has been central to data analysis and statistics at least since Gauss and Legendre’s invention of the least-squares method in the early 19th century (1).

Modern datasets pose new challenges to this centuries-old framework. On one hand, high-dimensional applications require the simultaneous estimation of millions of parameters. Examples span genomics (2), imaging (3), web services (4), and so on. On the other hand, the unknown object to be estimated has often a combinatorial structure: In clustering we aim at estimating a partition of the data points (5). Network analysis tasks usually require identification of a discrete subset of nodes in a graph (6, 7). Parsimonious data explanations are sought by imposing combinatorial sparsity constraints (8).

There is an obvious tension between the above requirements. Although efficient algorithms are needed to estimate a large number of parameters, the maximum likelihood (ML) method often requires the solution of NP-hard (nondeterministic polynomial-time hard) combinatorial problems. A flourishing line of work addresses this conundrum by designing effective convex relaxations of these combinatorial problems (9–11).

Unfortunately, the statistical properties of such convex relaxations are well understood only in a few cases [compressed sensing being the most important success story (12–14)]. In this paper we use tools from statistical mechanics to develop a precise picture of the behavior of a class of semidefinite programming relaxations. Relaxations of this type appear to be surprisingly effective in a variety of problems ranging from clustering to graph synchronization. For the sake of concreteness we will focus on three specific problems.

${\mathrm{\mathbb{Z}}}_2$ Synchronization

In the general synchronization problem, we aim at estimating $x_{\mathrm{0,1}},x_{\mathrm{0,2}},\dots ,x_{0,n}$, which are unknown elements of a known group $\mathfrak{G}$. This is done using data that consist of noisy observations of relative positions $Y_{ij}=x_{0,i}^{-1}{x}_{0,j}+\mathsf{noise}$. A large number of practical problems can be modeled in this framework. For instance, the case $\mathfrak{G}=SO\left(3\right)$ (the orthogonal group in three dimensions) is relevant for camera registration and molecule structure reconstruction in electron microscopy (15).

${\mathrm{\mathbb{Z}}}_2$ synchronization is arguably the simplest problem in this class and corresponds to $\mathfrak{G}={\mathrm{\mathbb{Z}}}_2$ (the group of integers modulo 2). Without loss of generality, we will identify this with the group $\left(\left\{+1,-1\right\},\cdot \right)$ (elements of the group are $+1$, $-1$, and the group operation is ordinary multiplication). We assume observations to be distorted by Gaussian noise; namely, for each $i<j$ we observe $Y_{ij}=\left(\lambda /n\right)\hspace{0.17em}{x}_{0,i}{x}_{0,j}+W_{ij}$, where $W_{ij}\sim \mathsf{N}\left(0,1/n\right)$ are independent standard normal random variables. This fits the general definition because $x_{0,i}^{-1}=x_{0,i}$ for $x_{0,i}\in \left\{+1,-1\right\}$.

In matrix notation, we observe a symmetric matrix $\mathit{Y}={\mathit{Y}}^{\text{*}}\in {\mathrm{ℝ}}^{n\times n}$ given by

$$\mathit{Y}=\frac{\lambda }{n}\hspace{0.17em}{\mathit{x}}_0\hspace{0.17em}{\mathit{x}}_0^{\text{*}}+\mathit{W}.$$ [1]

(Note that entries on the diagonal carry no information.) Here ${\mathit{x}}_0\in {\left\{+1,-1\right\}}^n$ and ${\mathit{x}}_0^{\text{∗}}$ denote the transpose of ${\mathit{x}}_0$, and $\mathit{W}={\left(W_{ij}\right)}_{i,j\le n}$ is a random matrix from the Gaussian orthogonal ensemble (GOE), i.e., a symmetric matrix with independent entries (up to symmetry) ${\left(W_{ij}\right)}_{1\le i<j\le n}{\sim }_{i.i.d.}\mathsf{N}\left(0,1/n\right)$ and ${\left(W_{ii}\right)}_{1\le i\le n}{\sim }_{i.i.d.}\mathsf{N}\left(0,2/n\right)$.

A solution of the ${\mathrm{\mathbb{Z}}}_2$ synchronization problem can be interpreted as a bipartition of the set $\left\{1,\dots ,n\right\}$. Hence, this has been used as a model for partitioning signed networks (16, 17).

$U\left(1\right)$ Synchronization

This is again an instance of the synchronization problem. However, we take $\mathfrak{G}=U\left(1\right)$. This is the group of complex number of modulus one, with the operation of complex multiplication $\mathfrak{G}=\left(\left\{x\in \mathrm{ℂ}:\left|x\right|=1\right\},\cdot \right)$.

As in the previous case, we assume observations to be distorted by Gaussian noise; that is, for each $i<j$ we observe $Y_{ij}=\left(\lambda /n\right)\hspace{0.17em}{x}_{0,i}{\overline{x}}_{0,j}+W_{ij}$, where $\overline{z}$ denotes complex conjugation^† and $W_{ij}\sim \mathsf{CN}\left(0,1/n\right)$.

In matrix notations, this model takes the same form as [1], provided we interpret ${\mathit{x}}_0^{*}$ as the conjugate transpose of vector ${\mathit{x}}_0\in {\mathrm{ℂ}}^n$, with components $x_{0,i}$, $\left|x_{0,i}\right|=1$. We will follow this convention throughout.

$U\left(1\right)$ synchronization has been used as a model for clock synchronization over networks (18, 19). It is also closely related to the phase-retrieval problem in signal processing (20–22). An important qualitative difference with respect to the previous example (${\mathrm{\mathbb{Z}}}_2$ synchronization) lies in the fact that $U\left(1\right)$ is a continuous group. We regard this as a prototype of synchronization problems over compact Lie groups [e.g., $SO\left(3\right)$].

Hidden Partition

The hidden (or planted) partition (also known as community detection) model is a statistical model for the problem of finding clusters in large network datasets (see refs. 7, 23, 24 and references therein for earlier work). The data consist of graph $G=\left(V,E\right)$ over vertex set $V=\left[n\right]\equiv \left\{\mathrm{1,2},\dots ,n\right\}$ generated as follows. We partition $V=V_{+}\mathrm{\hspace{0.17em}\cup }{V}_{-}$ by setting $i\in V_{+}$ or $i\in V_{-}$ independently across vertices with $\mathrm{\mathbb{P}}\left(i\in V_{+}\right)=\mathrm{\mathbb{P}}\left(i\in V_{-}\right)=1/2$. Conditional on the partition, edges are independent with

$$\mathrm{\mathbb{P}}\left\{\left(i,j\right)\in E|V_{+},V_{-}\right\}=\{\begin{array}{ll}a/n\hfill & \text{if}\hspace{0.17em}\left\{i,j\right\}\subseteq V_{+}\hspace{0.17em}\text{or}\hspace{0.17em}\left\{i,j\right\}\subseteq V_{-}.\hfill \\ b/n\hfill & \text{otherwise}.\hfill \end{array}$$ [2]

Here $a>b>0$ are model parameters that will be kept of order one as $n\to \infty$. This corresponds to a random graph with bounded average degree $d=\left(a+b\right)/2$ and a cluster (also known as block or community) structure corresponding to the partition $V_{+}\mathrm{\cup }{V}_{-}$. Given a realization of such a graph, we are interested in estimating the underlying partition.

We can encode the partition $V_{+}$, $V_{-}$ by a vector ${\mathit{x}}_0\in {\left\{+1,-1\right\}}^n$, letting $x_{0,i}=+1$ if $i\in V_{+}$ and $x_{0,i}=-1$ if $i\in V_{-}$. An important insight, which we will further develop below (25, 26), is that this problem is analogous to ${\mathrm{\mathbb{Z}}}_2$ synchronization, with signal strength $\lambda =\left(a-b\right)/\sqrt{2\left(a+b\right)}$. The parameters’ correspondence is obtained, at a heuristics level, by noting that if ${\mathit{A}}_G$ is the adjacency matrix of G, then $\mathbb{E}\langle {\mathit{x}}_0,{\mathit{A}}_G{\mathit{x}}_0\rangle /{\left(n\mathbb{E}{\Vert {\mathit{A}}_G\Vert }_F^2\right)}^{1/2}\approx \left(a-b\right)/\sqrt{2\left(a+b\right)}$. (Here and below, $\langle \mathit{a},\mathit{b}\rangle ={\displaystyle {\sum }_i{a}_i{b}_i}$ denotes the standard scalar product between vectors.)

A generalization of this problem to the case of more than two blocks has been studied since the 1980s as a model for social network structure (27), under the name of “stochastic block model.” For the sake of simplicity, we will focus here on the two-blocks case.

Illustrations

As a first preview of our results, Fig. 1 reports our analytical predictions for the estimation error in the ${\mathrm{\mathbb{Z}}}_2$ synchronization problem, comparing them with numerical simulations using semidefinite programming (SDP). An estimator is a map $\widehat{\mathit{x}}:{\mathrm{ℝ}}^{n\times n}\to {\mathrm{ℝ}}^n$, $\mathit{Y}\mapsto \widehat{\mathit{x}}\left(\mathit{Y}\right)$. We compare various estimators in terms of their per-coordinate mean square error (MSE):

$${\text{MSE}}_n\left(\widehat{\mathit{x}}\right)\equiv \frac{1}{n}\mathbb{E}\left\{\underset{s\in \left\{+1,-1\right\}}{\text{min}}{\Vert \widehat{\mathit{x}}\left(\mathit{Y}\right)-s\hspace{0.17em}{\mathit{x}}_0\Vert }_2^2\right\},$$ [3]

where expectation is with respect to the noise model [1] and ${\mathit{x}}_0\in {\left\{+1,-1\right\}}^n$ uniformly random. Note the minimization with respect to the sign $s\in \left\{+1,-1\right\}$ inside the expectation: because of the symmetry of [1], the vector ${\mathit{x}}_0$ can only be estimated up to a global sign. We will be interested in the high-dimensional limit $n\to \infty$ and will omit the subscript n—thus writing $\text{MSE}\left(\widehat{\mathit{x}}\right)$—to denote this limit. Note that a trivial estimator that always returns 0 has error ${\text{MSE}}_n\left(\mathbf{0}\right)=1$.

Fig. 1.

Estimating ${\mathit{x}}_0\in {\left\{+1,-1\right\}}^n$ under the noisy ${\mathrm{\mathbb{Z}}}_2$ synchronization model of Eq. 1. Curves correspond to (asymptotic) analytical predictions, and dots correspond to numerical simulations (averaged over 100 realizations).

Classical statistical theory suggests two natural reference estimators: the Bayes optimal and the maximum likelihood estimators. We will discuss these methods first, to set the stage for SDP relaxations.

Bayes Optimal Estimator.

The Bayes optimal estimator (also known as minimum MSE) provides a lower bound on the performance of any other approach. It takes the conditional expectation of the unknown signal given the observations:

$${\widehat{\mathit{x}}}^{\text{\hspace{0.17em}Bayes}}\left(\mathit{Y}\right)=\mathbb{E}\left\{\mathit{x}|\left(\lambda /n\right)\mathit{x}{\mathit{x}}^{\text{∗}}+\mathit{W}=\mathit{Y}\right\}.$$ [4]

Explicit formulas are given in SI Appendix. We note that ${\widehat{\mathit{x}}}^{\text{\hspace{0.17em}Bayes}}\left(\mathit{Y}\right)$ assumes knowledge of the prior distribution. The red dashed curve in Fig. 1 presents our analytical prediction for the asymptotic MSE for ${\widehat{\mathit{x}}}^{\text{\hspace{0.17em}Bayes}}\left(\cdot \right)$. Notice that $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}Bayes}}\right)=1$ for all $\lambda \le 1$ and $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}Bayes}}\right)<1$ strictly for all $\lambda >1$, with $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}Bayes}}\right)\to 0$ quickly as $\lambda \to \infty$. The point ${\lambda }_c^{\text{Bayes}}=1$ corresponds to a phase transition for optimal estimation, and no method can have nontrivial $\text{MSE}$ for $\lambda \le {\lambda }_c^{\text{Bayes}}$.

Maximum Likelihood.

The estimator ${\widehat{\mathit{x}}}^{\text{\hspace{0.17em}ML}}\left(Y\right)$ is given by the solution of

$${\widehat{\mathit{x}}}^{\text{\hspace{0.17em}ML}}\left(\mathit{Y}\right)=c\left(\lambda \right)\arg {\max }_{\mathit{x}\in {\left\{+1,-1\right\}}^n}\langle \mathit{x},\mathit{Yx}\rangle .$$ [5]

Here $c\left(\lambda \right)$ is a scaling factor^‡ that is chosen according to the asymptotic theory as to minimize the MSE. As for the Bayes optimal curve, we obtain $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}ML}}\right)=1$ for $\lambda \le {\lambda }_c^{\text{ML}}=1$ and $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}ML}}\right)<1$ (and rapidly decaying to 0) for $\lambda >{\lambda }_c^{\text{ML}}$ . (We refer to SI Appendix for this result.)

Semidefinite Programming.

Neither the Bayes nor the maximum likelihood approaches can be implemented efficiently. In particular, solving the combinatorial optimization problem in Eq. 5 is a prototypical NP-complete problem. Even worse, approximating the optimum value within a sublogarithmic factor is computationally hard (28) (from a worst case perspective). SDP relaxations allow us to obtain tractable approximations. Specifically, and following a standard lifting idea, we replace the problem [5] by the following semidefinite program over the symmetric matrix $\mathit{X}\in {\mathrm{ℝ}}^{n\times n}$ (18. 29, 30):

$$\begin{array}{l}\text{maximize}\hspace{0.17em}\langle \mathit{X},\mathit{Y}\rangle ,\\ \text{subject}\hspace{0.17em}\text{to}\hspace{0.17em}\hspace{0.17em}\mathit{X}\mathrm{\succcurlyeq }0,X_{ii}=1\forall i\in \left[n\right].\end{array}$$ [6]

We use $\langle \cdot ,\cdot \rangle$ to denote the scalar product between matrices, namely, $\langle \mathit{A},\mathit{B}\rangle \equiv \mathsf{Tr}\left({\mathit{A}}^{\ast }\mathit{B}\right)$, and $\mathit{A}\mathrm{\succcurlyeq }0$ to indicate that $\mathit{A}$ is positive semidefinite^§ (PSD). If we assume $\mathit{X}=\mathit{x}{\mathit{x}}^{\text{*}}$, the SDP [6] reduces to the maximum-likelihood problem [5]. By dropping this condition, we obtain a convex optimization problem that is solvable in polynomial time. Given an optimizer ${\mathit{X}}_{\text{opt}}={\mathit{X}}_{\text{opt}}\left(\mathit{Y}\right)$ of this convex problem, we need to produce a vector estimate. We follow a different strategy from standard rounding methods in computer science, which is motivated by our analysis below. We compute the eigenvalue decomposition ${\mathit{X}}_{\text{opt}}={\displaystyle {\sum }_{i=1}^n{\xi }_i\hspace{0.17em}{\mathit{v}}_i{\mathit{v}}_i^{\text{*}}}$, with eigenvalues ${\xi }_1\ge {\xi }_2\ge \mathrm{\cdots }\ge {\xi }_n\ge 0$ and eigenvectors ${\mathit{v}}_i={\mathit{v}}_i\left({\mathit{X}}_{\text{opt}}\left(\mathit{Y}\right)\right)$, with ${\Vert {\mathit{v}}_i\Vert }_2=1$. We then return the estimate

$${\widehat{\mathit{x}}}^{\text{SDP}}\left(\mathit{Y}\right)=\sqrt{n}\hspace{0.17em}{c}^{\text{SDP}}\left(\lambda \right)\hspace{0.17em}{\mathit{v}}_1\left({\mathit{X}}_{\text{opt}}\left(\mathit{Y}\right)\right),$$ [7]

with $c^{\text{SDP}}\left(\lambda \right)$ a certain scaling factor (SI Appendix).

Our analytical prediction for $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}SDP}}\right)$ is plotted as blue solid line in Fig. 1. Dots report the results of numerical simulations with this relaxation for increasing problem dimensions. The asymptotic theory appears to capture these data very well already for $n=200$. For further comparison, alongside the above estimators, we report the asymptotic prediction for $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}PCA}}\right)$, the mean square error of principal component analysis (PCA). This method simply returns the principal eigenvector of $\mathit{Y}$, suitably rescaled (SI Appendix).

Fig. 1 reveals several interesting features.

First, it is apparent that optimal estimation undergoes a phase transition. Bayes optimal estimation achieves nontrivial accuracy as soon as $\lambda >{\lambda }_c^{\text{Bayes}}=1$. The same is achieved by a method as simple as PCA (blue-dashed curve). On the other hand, for $\lambda <1$, no method can achieve $\text{MSE}\left(\widehat{\mathit{x}}\right)<1$ strictly [whereas $\text{MSE}\left(\widehat{\mathit{x}}\right)=1$ is trivial by $\widehat{\mathit{x}}=0$].

Second, PCA is suboptimal at large signal strength. PCA can be implemented efficiently but does not exploit the information $x_{0,i}\in \left\{+1,-1\right\}$. As a consequence, its estimation error is significantly suboptimal at large λ (SI Appendix).

Third, the SDP-based estimator is nearly optimal. The tractable estimator ${\widehat{\mathit{x}}}^{\text{\hspace{0.17em}SDP}}\left(\mathit{Y}\right)$ achieves the best of both worlds. Its phase transition coincides with the Bayes optimal one ${\lambda }_c^{\text{Bayes}}=1$, and $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}SDP}}\right)$ decays exponentially at large λ, staying close to $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}Bayes}}\right)$ and strictly smaller than $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}PCA}}\right)$, for $\lambda \ge 1$.

We believe that the above features are generic: as shown in SI Appendix, $U\left(1\right)$ synchronization confirms this expectation.

Fig. 2 illustrates our results for the community detection problem under the hidden partition model of Eq. 2. Recall that we encode the ground truth by a vector ${\mathit{x}}_0\in {\left\{+1,-1\right\}}^n$. In the present context, an estimator is required to return a partition of the vertices of the graph. Formally, it is a function on the space of graphs with n vertices ${\mathcal{G}}_n$, namely, $\widehat{\mathit{x}}:{\mathcal{G}}_n\to {\left\{+1,-1\right\}}^n$, $G\mapsto \widehat{\mathit{x}}\left(G\right)$. We will measure the performances of such an estimator through the overlap,

$${\text{Overlap}}_n\left(\widehat{\mathit{x}}\right)=\frac{1}{n}\mathbb{E}\left\{\left|\langle \widehat{\mathit{x}}\left(G\right),{\mathit{x}}_0\rangle \right|\right\},$$ [8]

and its asymptotic $n\to \infty$ limit (for which we omit the subscript). To motivate the SDP relaxation we note that the maximum likelihood estimator partitions V in two sets of equal size to minimize the number of edges across the partition (the minimum bisection problem). Formally,

$${\widehat{\mathit{x}}}^{\text{\hspace{0.17em}ML}}\left(G\right)\equiv \text{arg}\underset{\mathit{x}\in {\left\{+1,-1\right\}}^n}{\text{max}}\left\{{\displaystyle \sum _{\left(i,j\right)\in E}{x}_i{x}_j}:\langle \mathit{x},1\rangle =0\right\},$$ [9]

where $\mathbf{1}=\left(\mathrm{1,1},\dots ,1\right)$ is the all-ones vector. Once more, this problem is hard to approximate (31), which motivates the following SDP relaxation:

$$\begin{array}{l}\text{maximize}\hspace{0.17em}{\displaystyle \sum _{\left(i,j\right)\in E}{X}_{ij}},\\ \text{subject}\hspace{0.17em}\text{to}\hspace{0.17em}\mathit{X}\mathrm{\succcurlyeq }0,\mathit{X}1=0,\hspace{0.17em}\hspace{0.17em}{X}_{ii}=1\hspace{0.17em}\forall i\in \left[n\right].\end{array}$$ [10]

Given an optimizer ${\mathit{X}}_{\text{opt}}={\mathit{X}}_{\text{opt}}\left(G\right)$, we extract a partition of the vertices V as follows. As for the ${\mathrm{\mathbb{Z}}}_2$ synchronization problem, we compute the principal eigenvector ${\mathit{v}}_1\left({\mathit{X}}_{\text{opt}}\right)$. We then partition V according to the sign of ${\mathit{v}}_1\left({\mathit{X}}_{\text{opt}}\right)$. Formally,

$${\widehat{\mathit{x}}}^{\text{\hspace{0.17em}SDP}}\left(G\right)=\text{sign}\left({\mathit{v}}_1\left({\mathit{X}}_{\text{\hspace{0.17em}opt}}\left(G\right)\right)\right).$$ [11]

Let us emphasize a few features of Fig. 2:

Fig. 2.

Community detection under the hidden partition model of Eq. 2, for average degree $\left(a+b\right)/2=5$. Dots indicate performance of the SDP reconstruction method (averaged over 500 realizations). Dashed curve indicates asymptotic analytical prediction for the Gaussian model (which captures the large-degree behavior). Solid curve indicates analytical prediction for the sparse graph case (within the vectorial ansatz; SI Appendix).

First, both the GOE theory and the cavity method are accurate. The dashed curve of Fig. 2 reports the analytical prediction within the ${\mathrm{\mathbb{Z}}}_2$ synchronization model, with Gaussian noise (the GOE theory). This can be shown to capture the large degree limit: $d=\left(a+b\right)/2\to \mathrm{\infty }$, with $\lambda =\left(a-b\right)/\sqrt{2\left(a+b\right)}$ fixed, and is an excellent approximation already for $d=5$. The continuous curve is our prediction for $d=5$, obtained by applying the cavity method from statistical mechanics to the community detection problem (see next section and SI Appendix). This approach describes very accurately the empirical data and the small discrepancy from the GOE theory.

Second, SDP is superior to PCA. A sequence of recent papers (ref. 7 and references therein) demonstrate that classical spectral methods—such as PCA—fail to detect the hidden partition in graphs with bounded average degree. In contrast, Fig. 2 shows that a standard SDP relaxation does not break down in the sparse regime. See refs. 25, 32 for rigorous evidence toward the same conclusion.

Third, SDP is nearly optimal. As proven in ref. 33, no estimator can achieve ${\text{Overlap}}_n\left(\widehat{\mathit{x}}\right)\ge \delta >0$ as $n\to \infty$, if $\lambda =\left(a-b\right)/\sqrt{2\left(a+b\right)}<1$. Fig. 2 (and the theory developed in the next section) suggests that SDP has a phase transition threshold. Namely, there exists ${\lambda }_c^{\text{SDP}}={\lambda }_c^{\text{SDP}}\left(d\right)$ such that if

$$\lambda =\frac{a-b}{\sqrt{2\left(a+b\right)}}\ge {\lambda }_c^{\text{SDP}}\left(d=\left(a+b\right)/2\right),$$ [12]

then SDP achieves overlap bounded away from zero: $\text{Overlap}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}SDP}}\right)>0$. Fig. 2 also suggests ${\lambda }_c^{\text{SDP}}\left(5\right)\approx {\lambda }_c^{\text{Bayes}}=1$; that is, SDP is nearly optimal.

Below we will derive an accurate approximation for the critical point ${\lambda }_c^{\text{SDP}}\left(d\right)$. The factor ${\lambda }_c^{\text{SDP}}\left(d\right)$ measures the suboptimality of SDP for graphs of average degree d.

Fig. 3 plots our prediction for the function ${\lambda }_c^{\text{SDP}}\left(d\right)$, together with empirically determined values for this threshold, obtained through Monte Carlo experiments for $d\in \left\{\mathrm{2,5,10}\right\}$ (red circles). These were obtained by running the SDP estimator on randomly generated graphs with size up to $n=\mathrm{64,000}$ (total CPU time was about 10 y). In particular, we obtain ${\lambda }_c^{\text{SDP}}\left(d\right)>1$ strictly, but the gap ${\lambda }_c^{\text{SDP}}\left(d\right)-1$ is very small (at most of the order of $2\text{\%}$) for all d. This confirms in a precise quantitative way the conclusion that SDP is nearly optimal for the hidden partition problem.

Fig. 3.

Phase transition for the SDP estimator: for $\lambda >{\lambda }_c^{\text{SDP}}\left(d\right)$, the SDP estimator has positive correlation with the ground truth; for $\lambda \le {\lambda }_c^{\text{SDP}}\left(d\right)$ the correlation is vanishing [here $\lambda =\left(a-b\right)/\sqrt{2\left(a+b\right)}$ and $d=\left(a+b\right)/2$]. Solid line indicates prediction ${\tilde{\lambda }}_c^{\text{SDP}}\left(d\right)$ from the cavity method (vectorial ansatz; SI Appendix) (compare Eq. 25). Dashed line indicates ideal phase transition $\lambda =1$. Red circles indicate numerical estimates of the phase transition location for $d=2$, 5, and 10.

Simulations results are in broad agreement with our predictions but present small discrepancies (below $0.5\text{\%}$). These discrepancies might be due to the extrapolation from finite-n simulations to $n\to \infty$ or to the inaccuracy of our analytical approximation.

Analytical Results.

Our analysis is based on a connection with statistical mechanics. The models arising from this connection are spin models in the so-called “large-N” limit, a topic of intense study across statistical mechanics and quantum field theory (34). Here we exploit this connection to apply nonrigorous but sophisticated tools from the theory of mean field spin glasses (35, 36). The paper (25) provides partial rigorous evidence toward the predictions developed here.

We will first focus on the simpler problem of synchronization under Gaussian noise, treating together the ${\mathrm{\mathbb{Z}}}_2$ and $U\left(1\right)$ cases. We will then discuss the new features arising within the sparse hidden partition problem. Most technical derivations are presented in SI Appendix. To treat the real (${\mathrm{\mathbb{Z}}}_2$) and complex [$U\left(1\right)$] cases jointly, we will use $\mathbb{F}$ to denote any of the fields of reals or complex numbers, i.e., either $\mathrm{ℝ}$ or $\mathrm{ℂ}$.

Gibbs Measures and Vector Spin Models.

We start by recalling that a matrix $\mathit{X}\in {\mathbb{F}}^{n\times n}$ is PSD if and only if it can be written as $\mathit{X}=\mathit{\sigma }{\mathit{\sigma }}^{\text{*}}$ for some $\mathit{\sigma }\in {\mathbb{F}}^{n\times m}$. Indeed, without loss of generality, one can take $m=n$, and any $m\ge n$ is equivalent.

Letting ${\mathit{\sigma }}_1$, … ${\mathit{\sigma }}_n\in {\mathbb{F}}^m$ be the rows of $\mathit{\sigma }$, the SDP [6] can be rewritten as

$$\begin{array}{l}\text{maximize}\hspace{0.17em}{\displaystyle \sum _{\left(i,j\right)}{Y}_{ij}\langle {\mathit{\sigma }}_i,{\mathit{\sigma }}_j\rangle },\\ \text{subject}\hspace{0.17em}\text{to}\hspace{0.17em}{\mathit{\sigma }}_i\in S^{m-1}\hspace{0.17em}\forall i\in \left[n\right],\end{array}$$ [13]

with $S^{m-1}=\left\{\mathit{z}\in {\mathbb{F}}^m:{\Vert \mathit{z}\Vert }_2=1\right\}$ the unit sphere in m dimensions. The SDP relaxation corresponds to any case $m\ge n$ or, following the physics parlance, $m=\infty$. Note, however, that cases with bounded (small) m are of independent interest. In particular, for $m=1$ we have ${\mathit{\sigma }}_i\in \left\{-1,+1\right\}$ (for the real case) or ${\mathit{\sigma }}_i\in U\left(1\right)\subset \mathrm{ℂ}$ (for the complex case). Hence, we recover the maximum-likelihood estimator setting $m=1$. It is also known that (under suitable conditions on $\mathit{Y}$) for $m>\sqrt{2n}$, the problem [13] has no local optima except the global ones (37).

A crucial question is how the solution of [13] depends on the spin dimensionality m, for $m\ll n$. Denote by $\text{OPT}\left(\mathit{Y};m\right)$ the optimum value when the dimension is m (in particular, $\text{OPT}\left(\mathit{Y};m\right)$ is also the value of [6] for $m\ge n$). It was proven in ref. 25 that there exists a constant C independent of m and n such that

$$\left(1-\frac{C}{m}\right)\text{OPT}\left(\mathit{Y};\infty \right)\le \text{OPT}\left(\mathit{Y};m\right)\le \text{OPT}\left(\mathit{Y};\infty \right),$$ [14]

with probability converging to one as $n\to \infty$ (whereby $\mathit{Y}$ is chosen with any of the distributions studied in the present paper). The upper bound in Eq. 14 follows immediately from the definition. The lower bound is a generalization of the celebrated Grothendieck inequality from functional analysis (38).

The above inequalities imply that we can obtain information about the SDP [6] in the $n\to \infty$ limit, by taking $m\to \infty$ after $n\to \infty$. This is the asymptotic regime usually studied in physics under the term “large-N limit.”

Finally, we can associate to the problem [13] a finite-temperature Gibbs measure as follows:

$$p_{\beta ,m}\left(\text{d}\mathit{\sigma }\right)=\frac{1}{Z}\text{exp}\left\{2m\beta {\displaystyle \sum _{i<j}\mathrm{\Re }\left(Y_{ij}\langle {\mathit{\sigma }}_i,{\mathit{\sigma }}_j\rangle \right)}\right\}{\displaystyle \prod _{i=1}^n{p}_0\left(\text{d}{\mathit{\sigma }}_i\right)},$$ [15]

where $p_0\left(\text{d}{\mathit{\sigma }}_i\right)$ is the uniform measure over the m-dimensional sphere $S^{m-1}$ and $\mathrm{\Re }\left(z\right)$ denotes the real part of z. This allows us to treat in a unified framework all of the estimators introduced above. The optimization problem [13] is recovered by taking the limit $\beta \to \infty$ (with maximum likelihood for $m=1$ and SDP for $m\to \infty$). The Bayes optimal estimator is recovered by setting $m=1$ and $\beta =\lambda /2$ (in the real case) or $\beta =\lambda$ (in the complex case).

Cavity Method: ${\mathrm{\mathbb{Z}}}_2$ and $U\left(1\right)$ Synchronization.

The cavity method from spin-glass theory can be used to analyze the asymptotic structure of the Gibbs measure [15] as $n\to \infty$. Below we will state the predictions of our approach for the SDP estimator ${\widehat{\mathit{x}}}^{\text{\hspace{0.17em}SDP}}$.

Here we list the main steps of our analysis for the expert reader, deferring a complete derivation to the SI Appendix: (i) We use the cavity method to derive the replica symmetric predictions for the model (15) in the limit $n\to \infty$. (ii) By setting $m=1$, $\beta =\lambda /2$ (in the real case), or $\beta =\lambda$ (in the complex case) we obtain the Bayes optimal error $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}Bayes}}\right)$: on the basis of ref. 39, we expect the replica symmetric assumption to hold and these predictions to be exact. (See also ref. 40 for related work.) (iii) By setting $m=1$ and $\beta \to \infty$ we obtain a prediction for the error of maximum likelihood estimation $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}ML}}\right)$. Although this prediction is not expected to be exact (because of replica symmetry breaking), it should be nevertheless rather accurate, especially for large λ. (iv) By setting $m\to \infty$ and $\beta \to \infty$, we obtain the SDP estimation error $\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}SDP}}\right)$, which is our main object of interest. Notice that the inversion of limits $m\to \infty$ and $n\to \infty$ is justified (at the level of objective value) by Grothendieck inequality. Further, because the $m=\infty$ case is equivalent to a convex program, we expect the replica symmetric prediction to be exact in this case.

The properties of the SDP estimator are given in terms of the solution of a set of three nonlinear equations for the three scalar parameters μ, q, and $b\in \mathrm{ℝ}$ that we state next. Let $Z\sim \mathsf{N}\left(\mathrm{0,1}\right)$ (in the real case) or $Z\sim \mathsf{CN}\left(\mathrm{0,1}\right)$ (in the complex case). Define $\rho =\rho \left(Z;\mu ,q,r\right)$ as the only nonnegative solution of the following equation in $\left(0,\infty \right)$:

$$1=\frac{{\left|\mu +\sqrt{q}Z\right|}^2}{{\left(\rho +r\right)}^2}+\frac{1-q}{{\rho }^2}.$$ [16]

Then μ, q, and r satisfy

$$\mathit{\mu }=\lambda \mathbb{E}\left\{\frac{\mu +\sqrt{q}\hspace{0.17em}\mathrm{\Re }\left(Z\right)}{\rho +r}\right\},q=\mathbb{E}\left\{\frac{{\left|\mu +\sqrt{q}\hspace{0.17em}Z\right|}^2}{{\left(\rho +r\right)}^2}\right\},$$ [17]

$$r=\mathbb{E}\left\{\frac{1}{\rho }-\frac{\mu }{\sqrt{q}}\frac{\mathrm{\Re }\left(Z\right)}{\rho +r}-\frac{{\left|Z\right|}^2}{\rho +r}\right\}.$$ [18]

These equations can be solved by iteration, after approximating the expectations on the right-hand side numerically. The properties of the SDP estimator can be derived from this solution. Concretely, we have

$$\text{MSE}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}SDP}}\right)=1-\frac{\mu {\left(\lambda \right)}^2}{{\lambda }^{2}q\left(\lambda \right)}.$$ [19]

The corresponding curve is reported in Fig. 1 for the real case $\mathfrak{G}={\mathrm{\mathbb{Z}}}_2$. We can also obtain the asymptotic overlap from the solution of these equations. The cavity prediction is

$$\text{Overlap}\left({\widehat{\mathit{x}}}^{\text{\hspace{0.17em}SDP}}\right)=1-2\mathrm{\Phi }\left(-\frac{\mu \left(\lambda \right)}{\sqrt{q\left(\lambda \right)}}\right).$$ [20]

The corresponding curve is plotted in Fig. 2.

More generally, for any dimension m and inverse temperature β, we obtain equations that are analogous to Eqs. 17 and 18. The parameters $\mu ,q\text{,\hspace{0.17em}and\hspace{0.17em}}b$ characterize the asymptotic structure of the probability measure $p_{\beta ,m}\left(\text{d}\mathit{\sigma }\right)$ defined in Eq. 15, as follows. We assume, for simplicity ${\mathit{x}}_0=\left(+1,\dots ,+1\right)$. Define the following probability measure on unit sphere $S^{m-1}$, parametrized by $\mathit{\xi }\in {\mathrm{ℝ}}^m$, $r\in \mathrm{ℝ}$:

$${\nu }_{\mathit{\xi },r}\left(\text{d}\mathit{\sigma }\right)=\frac{1}{z\left(\mathit{\xi },r\right)}\text{exp}\left\{2\beta \hspace{0.17em}m\mathrm{\Re }\langle \mathit{\xi },\mathit{\sigma }\rangle -\beta \hspace{0.17em}mr{\left|{\sigma }_1\right|}^2\right\}\hspace{0.17em}{p}_0\left(\text{d}\mathit{\sigma }\right).$$ [21]

For ν a probability measure on $S^{m-1}$ and R an orthogonal (or unitary) matrix, let ${\nu }^R$ be the measure obtained by^¶ rotating ν. Finally, let $p_{i\left(1\right),\dots ,i\left(k\right)}^{\left(m,\beta \right)}$ denote the joint distribution of ${\mathit{\sigma }}_{i\left(1\right)},\cdots ,{\mathit{\sigma }}_{i\left(k\right)}$ under $p_{m,\beta }$. Then, for any fixed k, and any sequence of k-uples ${\left(i\left(1\right),\dots ,i\left(k\right)\right)}_n\in \left[n\right]$, we have

$$p_{i\left(1\right),\dots ,i\left(k\right)}^{\left(m,\beta \right)}\Rightarrow {\displaystyle \int {\nu }_{{\mathit{\xi }}_1,r}^R\left(\cdot \right)\times \cdots \times {\nu }_{{\mathit{\xi }}_k,r}^R\left(\cdot \right)\text{d}R}.$$ [22]

Here $\text{d}R$ denotes the uniform (Haar) measure on the orthogonal group, $\Rightarrow$ denotes convergence in distribution (note that $p_{i\left(1\right),\dots ,i\left(k\right)}^{\left(m,\beta \right)}$ is a random variable), and ${\mathit{\xi }}_1,\dots ,{\mathit{\xi }}_k{\sim }_{iid}\mathsf{N}\left(\mu {\mathit{e}}_1,\mathit{Q}\right)$ with $\mathit{Q}=\text{diag}\left(q,q_0\dots ,q_0\right)$, $q_0=\left(1-q\right)/\left(m-1\right)$.

Cavity Method: Community Detection in Sparse Graphs.

We next consider the hidden partition model, defined by Eq. 2. As above, we denote by $d=\left(a+b\right)/2$ the asymptotic average degree of the graph G and by $\lambda =\left(a-b\right)/\sqrt{2\left(a+b\right)}$ the signal-to-noise ratio. As illustrated by Fig. 2 (and further simulations presented in SI Appendix), ${\mathrm{\mathbb{Z}}}_2$ synchronization appears to be a very accurate approximation for the hidden partition model already at moderate d.

The main change with respect to the dense case is that the phase transition at $\lambda =1$, is slightly shifted, as per Eq. 12. Namely, SDP can detect the hidden partition with high probability if and only if $\lambda \ge {\lambda }_c^{\text{SDP}}\left(d\right)$, for some ${\lambda }_c^{\text{SDP}}\left(d\right)>1$.

Our prediction for the curve ${\lambda }_c^{\text{SDP}}\left(d\right)$ will be denoted by ${\tilde{\lambda }}_c^{\text{SDP}}\left(d\right)$ and is plotted in Fig. 3. It is obtained by finding an approximate solution of the RS cavity equations, within a scheme that we name “vectorial ansatz” (see SI Appendix for details). We see that ${\tilde{\lambda }}_c^{\text{SDP}}\left(d\right)$ approaches very quickly the ideal value $\lambda =1$ for $d\to \infty$. Indeed, our prediction implies ${\tilde{\lambda }}_c^{\text{SDP}}\left(d\right)=1+1/\left(8d\right)+O\left(d^{-2}\right)$. Also, ${\tilde{\lambda }}_c^{\text{SDP}}\left(d\right)\to 1$ as $d\to 1$. This is to be expected because the constraints $a\ge b\ge 0$ imply $\left(a-b\right)/2\le d$, with $b=0$ at $\left(a-b\right)/2=d$. Hence, the problem becomes trivial at $\left(a-b\right)/2=d$: it is sufficient to identify the connected components in G, whence ${\lambda }_c^{\text{SDP}}\left(d\right)\le \sqrt{d}$.

More interestingly, ${\tilde{\lambda }}_c^{\text{SDP}}\left(d\right)$ admits a characterization in terms of a distributional recursion, which can be evaluated numerically and is plotted as a continuous line in Fig. 3. Surprisingly, the SDP detection threshold appears to be suboptimal at most by 2%. To state this characterization, consider first the recursive distributional equation (RDE)

$$\mathsf{c}\stackrel{\text{d}}{=}{\displaystyle \sum _{i=1}^L\frac{{\mathsf{c}}_i}{1+{\mathsf{c}}_i}}.$$ [23]

Here $\stackrel{\text{d}}{=}$ denotes equality in distribution, $L\sim \text{Poisson}\left(d\right)$, and ${\mathsf{c}}_1,\dots ,{\mathsf{c}}_L$ are independent and identically distributed (i.i.d.) copies of $\mathsf{c}$. This has to be read as an equation for the law of the random variable $\mathsf{c}$ (see, e.g., ref. 41 for further background on RDEs). We are interested in a specific solution of this equation, constructed as follows. Set $c^0=\infty$ almost surely, and for $\ell \ge 0$, let ${\mathsf{c}}^{\ell +1}\stackrel{\text{d}}{=}{\displaystyle {\sum }_{i=1}^L{\mathsf{c}}_i^{\ell }/\left(1+{\mathsf{c}}_i^{\ell }\right)}$. It is proved in ref. 42 that the resulting sequence converges in distribution to a solution of Eq. 23: ${\mathsf{c}}^{\ell }\stackrel{\text{d}}{\Rightarrow }{\mathsf{c}}_{\text{*}}$.

The quantity ${\mathsf{c}}_{\text{∗}}$ has a useful interpretation. Consider a (rooted) Poisson Galton–Watson tree with branching number d, and imagine each edge to be a conductor with unit conductance. Then ${\mathsf{c}}_{\text{∗}}$ is the total conductance between the root and the boundary of the tree at infinity. In particular, ${\mathsf{c}}_{\text{∗}}=0$ almost surely for $d\le 1$, and ${\mathsf{c}}_{\text{∗}}>0$ with positive probability if $d>1$ (see ref. 42 and SI Appendix).

Next consider the distributional recursion

$$\left({\mathsf{c}}^{\ell +1};h^{\ell +1}\right)\stackrel{\text{d}}{=}\left({\displaystyle \sum _{i=1}^{L_{+}+L_{-}}\frac{{\mathsf{c}}_i^{\ell }}{1+{\mathsf{c}}_i^{\ell }}};{\displaystyle \sum _{i=1}^{L_{+}+L_{-}}\frac{s_i{h}_i^{\ell }}{\sqrt{1+{\mathsf{c}}_i^{\ell }}}}\right),$$ [24]

where $s_1,\dots ,s_{L_{+}}=+1$, $s_{L_{+}+1},\dots ,s_{L_{+}+L_{-}}=-1$, and we use initialization $\left({\mathsf{c}}^0,h^0\right)=\left(+\infty ,1\right)$. This recursion determines sequentially the distribution of $\left({\mathsf{c}}^{\ell +1},h^{\ell +1}\right)$ from the distribution of $\left({\mathsf{c}}^{\ell },h^{\ell }\right)$. Here $L_{+}\sim \text{Poisson}\left(\left(d+\lambda \right)/2\right)$, $L_{-}\sim \text{Poisson}\left(\left(d-\lambda \right)/2\right)$, and $\left({\mathsf{c}}_1^{\ell },h_1^{\ell }\right),\dots ,\left({\mathsf{c}}_L^{\ell },h_L^{\ell }\right)$ are i.i.d. copies of $\left({\mathsf{c}}^{\ell },h^{\ell }\right)$, independent of $L_{+},L_{-}$. Notice that because $L_{+}+L_{-}\sim \text{Poisson}\left(d\right)$, we have ${\mathsf{c}}^{\ell }\stackrel{\text{d}}{\Rightarrow }{\mathsf{c}}_{\text{*}}$. The threshold ${\tilde{\lambda }}_c^{\text{SDP}}\left(d\right)$ is defined as the smallest λ such that the $h^t$ diverges exponentially:

$${\tilde{\lambda }}_c^{\text{SDP}}\left(d\right)\equiv \text{inf}\left\{\lambda \in \left[0,\sqrt{d}\right]:\text{lim}\underset{t\to \infty }{\text{inf}}\frac{1}{t}\log \mathbb{E}\left({\left|h^t\right|}^2\right)>0\right\}.$$ [25]

This value can be computed numerically, for instance, by sampling the recursion [24]. The results of such an evaluation are plotted as a continuous line in Fig. 3.

Final Algorithmic Considerations

We have shown that ideas from statistical mechanics can be used to precisely locate phase transitions in SDP relaxations for high-dimensional statistical problems. In the problems investigated here, we find that SDP relaxations have optimal thresholds [in ${\mathrm{\mathbb{Z}}}_2$ and $U\left(1\right)$ synchronization] or nearly optimal thresholds (in community detection under the hidden partition model). Here near-optimality is to be interpreted in a precise quantitative sense: SDP’s threshold is suboptimal—at most—by a 2% factor. As such, SDPs provide a very useful tool for designing computationally efficient algorithms that are also statistically efficient.

Let us emphasize that other polynomial–time algorithms can be used for the specific problems studied here. In the synchronization problem, naive PCA achieves the optimal threshold $\lambda =1$. In the community detection problem, several authors recently developed ingenious spectral algorithms that achieve the information theoretically optimal threshold $\left(a-b\right)/\sqrt{2\left(a+b\right)}=1$ (see, e.g., refs. 7, 23, 24, 43, 44).

However, SDP relaxations have the important feature of being robust to model misspecifications (see also refs. 30, 45 for independent investigations of robustness issues). To illustrate this point, we perturbed the hidden partition model as follows. For a perturbation level $\alpha \in \left[\mathrm{0,1}\right]$, we draw $n\alpha$ vertices $i_1,\dots ,i_{n\alpha }$ uniformly at random in G. For each such vertex $i_{\ell }$ we connect by edges all of the neighbors of $i_{\ell }$. In our case, this results in adding $O\left(n{d}^2\alpha \right)$ edges.

In SI Appendix, we compare the behavior of SDP and the Bethe Hessian algorithm of ref. 44 for this perturbed model: although SDP appears to be rather insensitive to the perturbation, the performance of Bethe Hessian are severely degraded by it. We expect a similar fragility in other spectral algorithms.