Statistical properties of convex clustering

Abstract

In this manuscript, we study the statistical properties of convex clustering. We establish that convex clustering is closely related to single linkage hierarchical clustering and k-means clustering. In addition, we derive the range of the tuning parameter for convex clustering that yields a non-trivial solution. We also provide an unbiased estimator of the degrees of freedom, and provide a finite sample bound for the prediction error for convex clustering. We compare convex clustering to some traditional clustering methods in simulation studies.

1. Introduction

Let $\mathbf{X}\in {\mathbb{R}}^{n\times p}$ be a data matrix with n observations and p features. We assume for convenience that the rows of X are unique. The goal of clustering is to partition the n observations into K clusters, D₁, . . . , D_K, based on some similarity measure. Traditional clustering methods such as hierarchical clustering, k-means clustering, and spectral clustering take a greedy approach (see, e.g., Hastie, Tibshirani and Friedman, 2009).

In recent years, several authors have proposed formulations for convex clustering (Pelckmans et al., 2005; Hocking et al., 2011; Lindsten, Ohlsson and Ljung, 2011; Chi and Lange, 2014a). Chi and Lange (2014a) proposed efficient algorithms for convex clustering. In addition, Radchenko and Mukherjee (2014) studied the theoretical properties of a closely related problem to convex clustering, and Zhu et al. (2014) studied the condition needed for convex clustering to recover the correct clusters.

Convex clustering of the rows, X_1., . . . , X_n., of a data matrix X involves solving the convex optimization problem

$$\underset{\mathbf{U}\in {\mathbb{R}}^{n\times p}}{\mathrm{minimize}}\frac{1}{2}\sum _{i=1}^n{\Vert {\mathbf{X}}_{i.}-{\mathbf{U}}_{i.}\Vert }_2^2+\lambda {\mathrm{Q}}_q\left(\mathbf{U}\right),$$ (1)

where ${\mathrm{Q}}_q\left(\mathbf{U}\right)={\sum }_{i<i^{\prime }}{\Vert {\mathbf{U}}_{i.}-{\mathbf{U}}_{i^{\prime }.}\Vert }_q$ for q ∈ {1, 2, ∞}. The penalty Q_q(U) generalizes the fused lasso penalty proposed in Tibshirani et al. (2005), and encourages the rows of Û, the solution to (1), to take on a small number of unique values. On the basis of Û, we define the estimated clusters as follows.

Definition 1

The ith and i′th observations are estimated by convex clustering to belong to the same cluster if and only if Û_i. = Û_i′..

The tuning parameter λ controls the number of unique rows of Û, i.e., the number of estimated clusters. When λ = 0, Û = X, and so each observation belongs to its own cluster. As λ increases, the number of unique rows of Û will decrease. For sufficiently large λ, all rows of Û will be identical, and so all observations will be estimated to belong to a single cluster. Note that (1) is strictly convex, and therefore the solution Û is unique.

To simplify our analysis of convex clustering, we rewrite (1). Let $\mathbf{x}=\mathrm{vec}\left(\mathbf{X}\right)\in {\mathbb{R}}^{np}$ and let $\mathbf{u}=\mathrm{vec}\left(\mathbf{U}\right)\in {\mathbb{R}}^{np}$, where the vec(·) operator is such that x_(i–1)p+j = X_ij and u_(i–1)p+j = U_ij. Construct $\mathbf{D}\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]\times np}$, and define the index set $\mathcal{C}(i,i^{\prime })$ such that the p × np submatrix ${\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}$ satisfies ${\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\mathbf{u}={\mathbf{U}}_i-{\mathbf{U}}_{i^{\prime }}$. Furthermore, for a vector $\mathbf{b}\in {\mathbb{R}}^{p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)}$, we define

$${\mathrm{P}}_q\left(\mathbf{b}\right)=\sum _{i<i^{\prime }}{\Vert {\mathbf{b}}_{\mathcal{C}(i,i^{\prime })}\Vert }_q.$$ (2)

Thus, we have ${\mathrm{P}}_q\left(\mathbf{Du}\right)={\sum }_{i<i^{\prime }}{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\mathbf{u}\Vert }_q={\sum }_{i<i^{\prime }}{\Vert {\mathbf{U}}_{i.}-{\mathbf{U}}_{i^{\prime }.}\Vert }_q={\mathrm{Q}}_q\left(\mathbf{U}\right)$. Problem (1) can be rewritten as

$$\underset{\mathbf{u}\in {\mathbb{R}}^{np}}{\mathrm{minimiz}}\frac{1}{2}{\Vert \mathbf{x}-\mathbf{u}\Vert }_2^2+\lambda {\mathrm{P}}_q\left(\mathbf{Du}\right).$$ (3)

When q = 1, (3) is an instance of the generalized lasso problem studied in Tibshirani and Taylor (2011). Let û be the solution to (3). By Definition 1, the ith and i′th observations belong to the same cluster if and only if ${\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}=0$. In what follows, we work with (3) instead of (1) for convenience.

Let ${\mathbf{D}}^{†}\in {\mathbb{R}}^{np\times [p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}$ be the Moore-Penrose pseudo-inverse of D. We state some properties of D and D^† that will prove useful in later sections.

Lemma 1

The matrices D and D^† have the following properties.

• (i)rank(D) = p(n – 1).
• (ii)${\mathbf{D}}^{†}=\frac{1}{n}{\mathbf{D}}^T$.
• (iii)(D^TD)^†D^T = D^† and (DD^T)^†D = (D^T)^†.
• (iv)$\mathbf{D}{\left({\mathbf{D}}^T\mathbf{D}\right)}^{†}{\mathbf{D}}^T=\frac{1}{n}{\mathbf{DD}}^T$ is a projection matrix onto the column space of D.
• (v)Define Λ_min(D) and Λ_max(D) as the minimum non-zero singular value and maximum singular value of the matrix D, respectively. Then, Λ_min(D) = Λ_max(D) = √n.

In this manuscript, we study the statistical properties of convex clustering. In Section 2, we study the dual problem of (3) and use it to establish that convex clustering is closely related to single linkage hierarchical clustering. In addition, we establish a connection between k-means clustering and convex clustering. In Section 3, we present some properties of convex clustering. More specifically, we characterize the range of the tuning parameter λ in (3) such that convex clustering yields a non-trivial solution. We also provide a finite sample bound for the prediction error, and an unbiased estimator of the degrees of freedom for convex clustering. In Section 4, we conduct numerical studies to evaluate the empirical performance of convex clustering relative to some existing proposals. We close with a discussion in Section 5.

2. Convex clustering, single linkage hierarchical clustering, and k-means clustering

In Section 2.1, we study the dual problem of convex clustering (3). Through its dual problem, we establish a connection between convex clustering and single linkage hierarchical clustering in Section 2.2. We then show that convex clustering is closely related to k-means clustering in Section 2.3.

2.1. Dual problem of convex clustering

We analyze convex clustering (3) by studying its dual problem. Let s, q ∈ {1, 2, ∞} satisfy $\frac{1}{s}+\frac{1}{q}=1$. For a vector $\mathbf{b}\in {\mathbb{R}}^{p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)}$, let ${\mathrm{P}}_q^{\ast }\left(\mathbf{b}\right)$ denote the dual norm of P_q(b), which takes the form

$${\mathrm{P}}_q^{\ast }\left(\mathbf{b}\right)=\underset{i<i^{\prime }}{\max }{\Vert {\mathbf{b}}_{\mathcal{C}(i,i^{\prime })}\Vert }_s.$$ (4)

We refer the reader to Chapter 6 in Boyd and Vandenberghe (2004) for an overview of the concept of duality.

Lemma 2

The dual problem of convex clustering (3) is

$$\underset{\nu \in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}}{\mathrm{minimize}}\frac{1}{2}{\Vert \mathbf{x}-{\mathbf{D}}^T\nu \Vert }_2^2\text{subject to}{\mathrm{P}}_q^{\ast }\left(\nu \right)\le \lambda ,$$ (5)

where $\nu \in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}$ is the dual variable. Furthermore, let û and $\widehat{\nu }$ be the solutions to (3) and (5), respectively. Then,

$$\mathbf{D}\widehat{\mathbf{u}}=\mathbf{Dx}-{\mathbf{DD}}^T\widehat{\nu }.$$ (6)

While (3) is strictly convex, its dual problem (5) is not strictly convex, since D is not of full rank by Lemma 1(i). Therefore, the solution $\widehat{\nu }$ to (5) is not unique. Lemma 1(iv) indicates that $\frac{1}{n}{\mathbf{DD}}^T$ is a projection matrix onto the column space of D. Thus, the solution Dû in (6) can be interpreted as the difference between Dx, the pairwise difference between rows of X, and the projection of a dual variable onto the column space of D.

We now consider a modification to the convex clustering problem (3). Recall from Definition 1 that the ith and i'th observations are in the same estimated cluster if ${\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}=0$. This motivates us to estimate γ = Du directly by solving

$$\underset{\gamma \in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}}{\mathrm{minimize}}\frac{1}{2}{\Vert \mathbf{Dx}-\gamma \Vert }_2^2+\lambda {\mathrm{P}}_q\left(\gamma \right).$$ (7)

We establish a connection between (3) and (7) by studying the dual problem of (7).

Lemma 3

The dual problem of (7) is

$$\underset{{\nu }^{\prime }\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}}{\mathrm{minimize}}\frac{1}{2}{\Vert \mathbf{Dx}-{\nu }^{\prime }\Vert }_2^2\text{subject to}{\mathrm{P}}_q^{\ast }\left({\nu }^{\prime }\right)\le \lambda ,$$ (8)

where ${\nu }^{\prime }\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}$ is the dual variable. Furthermore, let $\widehat{\gamma }$ and ${\widehat{\nu }}^{\prime }$ be the solutions to (7) and (8), respectively. Then,

$$\widehat{\gamma }=\mathbf{Dx}-{\widehat{\nu }}^{\prime }.$$ (9)

Comparing (6) and (9), we see that the solutions to convex clustering (3) and the modified problem (7) are closely related. In particular, both Dû in (6) and $\widehat{\gamma }$ in (9) involve taking the difference between Dx and some function of a dual variable that has ${\mathrm{P}}_q^{\ast }(\cdot )$ norm less than or equal to λ. The main difference is that in (6), the dual variable is projected into the column space of D.

Problem (7) is quite simple, and in fact it amounts to a thresholding operation on Dx when q = 1 or q = 2, i.e., the solution $\widehat{\gamma }$ is obtained by performing soft thresholding on Dx, or group soft thresholding on ${\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\mathbf{x}$ for all i < i′, respectively (Bach et al., 2011). When q = ∞, an efficient algorithm was proposed by Duchi and Singer (2009).

2.2. Convex clustering and single linkage hierarchical clustering

In this section, we establish a connection between convex clustering and single linkage hierarchical clustering. Let ${\widehat{\gamma }}^q$ be the solution to (7) with P_q(·) norm and let s, q ∈ {1, 2, ∞} satisfy $\frac{1}{s}+\frac{1}{q}=1$. Since (7) is separable in ${\gamma }_{\mathcal{C}(i,i^{\prime })}$ for all i < i′, by Lemma 2.1 in Haris, Witten and Simon (2015), it can be verified that

$${\widehat{\gamma }}_{\mathcal{C}(i,i^{\prime })}^q=0\text{if and only if}{\Vert {\mathbf{X}}_{i.}-{\mathbf{X}}_{i^{\prime }.}\Vert }_s\le \lambda .$$ (10)

It might be tempting to conclude that a pair of observations (i, i′) belong to the same cluster if ${\widehat{\gamma }}_{\mathcal{C}(i,i^{\prime })}^q=0$. However, by inspection of (10), it could happen that ${\widehat{\gamma }}_{\mathcal{C}(i,i^{\prime })}^q=0$ and ${\widehat{\gamma }}_{\mathcal{C}(i^{\prime },i^{″})}^q=0$, but ${\widehat{\gamma }}_{\mathcal{C}(i,i^{″})}^q\ne 0$.

To overcome this problem, we define the n × n adjacency matrix A^q(λ) as

$${\left[{\mathbf{A}}^q\left(\lambda \right)\right]}_{i{i}^{\prime }}=\{\begin{array}{cc}1\hfill & \text{if}i=i^{\prime },\hfill \\ 1\hfill & \text{if}{\widehat{\gamma }}_{\mathcal{C}(i,i^{\prime })}^q=0,\hfill \\ 0\hfill & \text{if}{\widehat{\gamma }}_{\mathcal{C}(i,i^{\prime })}^q\ne 0.\hfill \end{array}\phantom{\}}$$ (11)

Subject to a rearrangement of the rows and columns, A^q(λ) is a block-diagonal matrix with some number of blocks, denoted as R. On the basis of A^q(λ), we define R estimated clusters: the indices of the observations in the rth cluster are the same as the indices of the observations in the rth block of A^q(λ).

We now present a lemma on the equivalence between single linkage hierarchical clustering and the clusters identified by (7) using (11). The lemma follows directly from the definition of single linkage clustering (see, for instance, Chapter 3.2 of Jain and Dubes, 1988).

Lemma 4

Let Ê₁, . . . , Ê_R index the blocks within the adjacency matrix A_q(λ). Let s satisfy $\frac{1}{s}+\frac{1}{q}=1$. Let D̂₁, . . . , D̂_K denote the clusters that result from performing single linkage hierarchical clustering on the dissimilarity matrix defined by the pairwise distance between the observations ∥X_i. – X_i′∥_s, and cutting the dendrogram at the height of λ > 0. Then K = R, and there exists a permutation π : {1, . . . , K} → {1, . . . , K} such that D_k = E_π(k) for k = 1, . . . , K.

In other words, Lemma 4 implies that single linkage hierarchical clustering and (7) yield the same estimated clusters. Recalling the connection between (3) and (7) established in Section 2.1, this implies a close connection between convex clustering and single linkage hierarchical clustering.

2.3. Convex clustering and k-means clustering

We now establish a connection between convex clustering and k-means clustering. k-means clustering seeks to partition the n observations into K clusters by minimizing the within cluster sum of squares. That is, the clusters are given by the partition D̂₁, . . . , D̂_K of {1, . . . , n} that solves the optimization problem

$$\underset{{\mu }_1,\dots ,{\mu }_K\in {\mathbb{R}}^p,D_1,\dots ,D_K}{\mathrm{minimize}}\sum _{k=1}^K\sum _{i\in D_k}{\Vert {\mathbf{X}}_{i.}-{\mu }_k\Vert }_2^2.$$ (12)

We consider convex clustering (1) with q = 0,

$$\underset{\mathbf{U}\in {\mathbb{R}}^{n\times p}}{\mathrm{minimize}}\frac{1}{2}\sum _{i=1}^n{\Vert {\mathbf{X}}_{i.}-{\mathbf{U}}_{i.}\Vert }_2^2+\lambda \sum _{i<i^{\prime }}\mathbb{I}({\mathbf{U}}_{i.}\ne {\mathbf{U}}_{i^{\prime }.}),$$ (13)

where $\mathbb{I}({\mathbf{U}}_{i.}\ne {\mathbf{U}}_{i^{\prime }.})$ is an indicator function that equals one if U_i. ≠ U_i′.. Note that (13) is no longer a convex optimization problem.

We now establish a connection between (12) and (13). For a given value of λ, (13) is equivalent to

$$\underset{\mathbf{U}\in {\mathbb{R}}^{n\times p},K,{\mu }_1,\dots ,{\mu }_K\in {\mathbb{R}}^p,E_1,\dots ,E_K}{\mathrm{minimize}}\frac{1}{2}\sum _{k=1}^K\sum _{i\in E_k}{\Vert {\mathbf{X}}_{i.}-{\mu }_k\Vert }_2^2+\lambda \sum _{i<i^{\prime }}\sum _{k=1}^K\mathbb{I}(i\in E_k,i^{\prime }\notin E_k),$$ (14)

subject to the constraint that {μ₁, . . . , μ_K} are the unique rows of U and E_k = {i : U_i. = μ_k}. Note that $\mathbb{I}(i\in E_k,i^{\prime }\notin E_k)$ is an indicator function that equals to one if i ∈ E_k and $i^{\prime }\notin E_k$. Thus, we see from (12) and (14) that k-means clustering is equivalent to convex clustering with q = 0, up to a penalty term $\lambda {\sum }_{i<i^{\prime }}{\sum }_{k=1}^K\mathbb{I}(i\in E_k,i^{\prime }\notin E_k)$.

To interpret the penalty term, we consider the case when there are two clusters E₁ and E₂. The penalty term reduces to λ|E₁| · (n – |E₁|), where |E₁| is the cardinality of the set E₁. The term λ|E₁| · (n – |E₁|) is minimized when |E₁| is either 1 or n – 1, encouraging one cluster taking only one observation. Thus, compared to k-means clustering, convex clustering with q = 0 has the undesirable behavior of producing clusters whose sizes are highly unbalanced.

3. Properties of convex clustering

We now study the properties of convex clustering (3) with q ∈ {1, 2}. In Section 3.1, we establish the range of the tuning parameter λ in (3) such that convex clustering yields a non-trivial solution with more than one cluster. We provide finite sample bounds for the prediction error of convex clustering in Section 3.2. Finally, we provide unbiased estimates of the degrees of freedom for convex clustering in Section 3.3.

3.1. Range of λ that yields non-trivial solution

In this section, we establish the range of the tuning parameter λ such that convex clustering (3) yields a solution with more than one cluster.

Lemma 5

Let

$${\lambda }_{\text{upper}}≔\{\begin{array}{cc}\underset{\omega }{\min }{\Vert \frac{1}{n}\mathbf{Dx}+(\mathbf{I}-\frac{1}{n}{\mathbf{DD}}^T)\omega \Vert }_{\infty }\hfill & \text{for}q=1,\hfill \\ \underset{\omega }{\min }\left\{\underset{i<i^{\prime }}{\max }\left\{{\Vert {(\frac{1}{n}\mathbf{Dx}+(\mathbf{I}-\frac{1}{n}{\mathbf{DD}}^T)\omega )}_{\mathcal{C}(i,i^{\prime })}\Vert }_2\right\}\right\}\hfill & \text{for}q=2.\hfill \end{array}\phantom{\}}$$ (15)

Convex clustering (3) with q = 1 or q = 2 yields a non-trivial solution of more than one cluster if and only if λ < λ_upper.

By Lemma 5, we see that calculating λ_upper boils down to solving a convex optimization problem. This can be solved using a standard solver such as CVX in MATLAB. In the absence of such a solver, a loose upper bound on λ_upper is given by ${\Vert \frac{1}{n}\mathbf{Dx}\Vert }_{\infty }$ for q = 1, or $\underset{i<i^{\prime }}{\max }{\Vert \frac{1}{n}{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\mathbf{x}\Vert }_2$ for q = 2.

Therefore, to obtain the entire solution path of convex clustering, we need only consider values of λ that satisfy λ ≤ λ_upper.

3.2. Bounds on prediction error

In this section, we assume the model x = u + ϵ, where $ϵ\in {\mathbb{R}}^{np}$ is a vector of independent sub-Gaussian noise terms with mean zero and variance σ², and u is an arbitrary np-dimensional mean vector. We refer the reader to pages 24-25 in Boucheron, Lugosi and Massart (2013) for the properties of sub-Gaussian random variables. We now provide finite sample bounds for the prediction error of convex clustering (3). Let λ be the tuning parameter in (3) and let ${\lambda }^{\prime }=\frac{\lambda }{np}$.

Lemma 6

Suppose that x = u + ϵ, where $ϵ\in {\mathbb{R}}^{np}$ and the elements of ϵ are independent sub-Gaussian random variables with mean zero and variance σ². Let û be the estimate obtained from (3) with q = 1. If ${\lambda }^{\prime }\ge 4\sigma \sqrt{\frac{\log (p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right))}{n^3{p}^2}}$, then

$$\frac{1}{2np}{\Vert \widehat{\mathbf{u}}-\mathbf{u}\Vert }_2^2\le \frac{3{\lambda }^{\prime }}{2}{\Vert \mathbf{Du}\Vert }_1+{\sigma }^2\left[\frac{1}{n}+\sqrt{\frac{\log \left(np\right)}{n^{2}p}}\right]$$

holds with probability at least $1-\frac{2}{p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)}-\exp \{-\min (c_1\log \left(np\right),c_2\sqrt{p\log \left(np\right)})\}$, where c₁ and c₂ are positive constants appearing in Lemma 10.

We see from Lemma 6 that the average prediction error is bounded by the oracle quantity ∥Du∥₁ and a second term that decays to zero as n, p → ∞. Convex clustering with q = 1 is prediction consistent only if λ′∥Du∥₁ = o (1). We now provide a scenario for which λ′∥Du∥₁ = o (1) holds.

Suppose that we are in the high-dimensional setting in which p > n and the true underlying clusters differ only with respect to a fixed number of features (Witten and Tibshirani, 2010). Also, suppose that each element of Du — that is, U_ij–U_i′j — is of order O(1). Therefore, ∥Du∥₁ = O(n²), since by assumption only a fixed number of features have different means across clusters. Assume that $\sqrt{\frac{n\log (p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right))}{p^2}}=o\left(1\right)$. Under these assumptions, convex clustering with q = 1 is prediction consistent.

Next, we present a finite sample bound on the prediction error for convex clustering with q = 2.

Lemma 7

Suppose that x = u + ϵ, where $ϵ\in {\mathbb{R}}^{np}$ and the elements of ϵ are independent sub-Gaussian random variables with mean zero and variance σ². Let û be the estimate obtained from (3) with q = 2. If ${\lambda }^{\prime }\ge 4\sigma \sqrt{\frac{\log (p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right))}{n^{3}p}}$, then

$$\frac{1}{2np}{\Vert \widehat{\mathbf{u}}-\mathbf{u}\Vert }_2^2\le \frac{3{\lambda }^{\prime }}{2}\sum _{i<i^{\prime }}{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\mathbf{u}\Vert }_2+{\sigma }^2\left[\frac{1}{n}+\sqrt{\frac{\log \left(np\right)}{n^{2}p}}\right]$$

holds with probability at least $1-\frac{2}{p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)}-\exp \{-\min (c_1\log \left(np\right),c_2\sqrt{p\log \left(np\right)})\}$, where c₁ and c₂ are positive constants appearing in Lemma 10.

Under the scenario described above, ${\parallel {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\mathbf{u}\parallel }_2=O\left(1\right)$, and therefore ${\sum }_{i<i^{\prime }}{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\mathbf{u}\Vert }_2=O\left(n^2\right)$. Convex clustering with q = 2 is prediction consistent if $\sqrt{\frac{n\log (p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right))}{p}}=o\left(1\right)$.

3.3. Degrees of freedom

Convex clustering recasts the clustering problem as a penalized regression problem, for which the notion of degrees of freedom is established (Efron, 1986). Under this framework, we provide an unbiased estimator of the degrees of freedom for clustering. Recall that û is the solution to convex clustering (3). Suppose that Var(x) = σ²I. Then, the degrees of freedom for convex clustering is defined as $\frac{1}{{\sigma }^2}{\sum }_{\mathrm{j}=1}^{np}\mathrm{Cov}({\widehat{u}}_j,x_j)$ (see, e.g., Efron, 1986). An unbiased estimator of the degrees of freedom for convex clustering with q = 1 follows directly from Theorem 3 in Tibshirani and Taylor (2012).

Lemma 8

Assume that x ~ MVN(u, σ²I), and let û be the solution to (3) with q = 1. Furthermore, let ${\widehat{\mathcal{B}}}_1=\{j:{\left(\mathbf{D}\widehat{\mathbf{u}}\right)}_j\ne 0\}$. We define the matrix ${\mathbf{D}}_{-{\widehat{\mathcal{B}}}_1}$ by removing the rows of D that correspond to ${\widehat{\mathcal{B}}}_1$. Then

$$\widehat{\mathrm{d}}{\mathrm{f}}_1=\mathrm{tr}\left(\mathbf{I}-{\mathbf{D}}_{-{\widehat{\mathcal{B}}}_1}^T{\left({\mathbf{D}}_{-{\widehat{\mathcal{B}}}_1}{\mathbf{D}}_{-{\widehat{\mathcal{B}}}_1}^T\right)}^{†}{\mathbf{D}}_{-{\widehat{\mathcal{B}}}_1}\right)$$ (16)

is an unbiased estimator of the degrees of freedom of convex clustering with q = 1.

The following corollary follows directly from Corollary 1 in Tibshirani and Taylor (2011).

Corollary 1

Assume that x ~ MVN(u, σ²I), and let û be the solution to (3) with q = 1. The fit û has degrees of freedom

$${\mathrm{df}}_1\left(\widehat{\mathbf{u}}\right)=\mathrm{E}\left[\text{number of unique elements in}\widehat{\mathbf{u}}\right].$$

There is an interesting interpretation of the degrees of freedom estimator for convex clustering with q = 1. Suppose that there are K estimated clusters, and all elements of the estimated means corresponding to the K estimated clusters are unique. Then the degrees of freedom is Kp, the product of the number of estimated clusters and the number of features.

Next, we provide an unbiased estimator of the degrees of freedom for convex clustering with q = 2.

Lemma 9

Assume that x ~ MVN(u, σ²I), and let û be the solution to (3) with q = 2. Furthermore, let ${\widehat{\mathcal{B}}}_2=\{(i,i^{\prime }):{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2\ne 0\}$. We define the matrix ${\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}$ by removing rows of D that correspond to ${\widehat{\mathcal{B}}}_2$. Let $\mathbf{P}=(\mathbf{I}-{\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}^T{\left({\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}{\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}^T\right)}^{†}{\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2})$ be the projection matrix onto the complement of the space spanned by the rows of ${\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}$. Then

$$\widehat{\mathrm{d}}{\mathrm{f}}_2=\mathrm{tr}\left({\left[\mathbf{I}+\lambda \mathbf{P}\sum _{(i,i^{\prime })\in {\widehat{\mathcal{B}}}_2}\left(\frac{{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}}{{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2}-\frac{{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}{\widehat{\mathbf{u}}}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}}{{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2^3}\right)\right]}^{-1}\mathbf{P}\right)$$ (17)

is an unbiased estimator of the degrees of freedom of convex clustering with q = 2.

When λ = 0, ${\parallel {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\parallel }_2\ne 0$ for all i < i′. Therefore, $\mathbf{P}=\mathbf{I}\in {\mathbb{R}}^{np\times np}$ and the degrees of freedom estimate is equal to tr(I) = np. When λ is sufficiently large that ${\widehat{\mathcal{B}}}_2$ is an empty set, one can verify that P = I – D^T (DD^T)^†D is a projection matrix of rank p, using the fact that rank(D) = p(n – 1) from Lemma 1(i). Therefore d̂f₂ = tr(P) = p.

We now assess the accuracy of the proposed unbiased estimators of the degrees of freedom. We simulate Gaussian clusters with K = 2 as described in Section 4.1 with n = p = 20 and σ = 0.5. We perform convex clustering with q = 1 and q = 2 across a fine grid of tuning parameters λ. For each λ, we compare the quantities (16) and (17) to

$$\frac{1}{{\sigma }^2}\sum _{j=1}^{np}({\widehat{u}}_j-u_j)(x_j-u_j),$$ (18)

which is an unbiased estimator of the true degrees of freedom, $\frac{1}{{\sigma }^2}{\sum }_{j=1}^{np}\mathrm{Cov}({\widehat{u}}_j,x_j)$, averaged over 500 data sets. In addition, we plot the point-wise intervals of the estimated degrees of freedom (mean ± 2 × standard deviation). Note that (18) cannot be computed in practice, since it requires knowledge of the unknown quantity u. Results are displayed in Figure 1. We see that the estimated degrees of freedom are quite close to the true degrees of freedom.

Fig 1.

We compare the true degrees of freedom of convex clustering (x-axis), given in (18), to the proposed unbiased estimators of the degrees of freedom (y-axis), given in Lemmas 8 and 9. Panels (a) and (b) contain the results for convex clustering with q = 1 and q = 2, respectively. The red line is the mean of the estimated degrees of freedom for convex clustering over 500 data sets, obtained by varying the tuning parameter λ. The shaded bands indicate the point-wise intervals of the estimated degrees of freedom (mean ± 2 × standard deviation), over 500 data sets. The black line indicates y = x.

4. Simulation studies

We compare convex clustering with q = 1 and q = 2 to the following proposals:

1. Single linkage hierarchical clustering with the dissimilarity matrix defined by the Euclidean distance between two observations.

2. The k-means clustering algorithm (Lloyd, 1982).

3. Average linkage hierarchical clustering with the dissimilarity matrix defined by the Euclidean distance between two observations.

We apply convex clustering (3) with q = {1, 2} using the R package cvxclustr (Chi and Lange, 2014b). In order to obtain the entire solution path for convex clustering, we use a fine grid of λ values for (3), in a range guided by Lemma 5. We apply the other methods by allowing the number of clusters to vary over a range from 1 to n clusters. To evaluate and quantify the performance of the different clustering methods, we use the Rand index (Rand, 1971). A high value of the Rand index indicates good agreement between the true and estimated clusters.

We consider two different types of clusters in our simulation studies: Gaussian clusters and non-convex clusters.

4.1. Gaussian clusters

We generate Gaussian clusters with K = 2 and K = 3 by randomly assigning each observation to a cluster with equal probability. For K = 2, we create the mean vectors μ₁ = 1_p and μ₂ = −1_p. For K = 3, we create the mean vectors μ₁ = −3 · 1_p, μ₂ = 0_p, and μ₃ = 3 · 1_p. We then generate the n × p data matrix X according to X_i. ~ MVN(μ_k, σ²I) for i ∈ D_k. We consider n = p = 30 and σ = {1, 2}. The Rand indices for K = 2 and K = 3, averaged over 200 data sets, are summarized in Figures 2 and 3, respectively.

Fig 2.

Simulation results for Gaussian clusters with K = 2, n = p = 30, averaged over 200 data sets, for two noise levels σ = {1, 2}. Colored lines correspond to single linkage hierarchical -clustering (), average linkage hierarchical clustering (), k-means clustering (1074i03), convex clustering with q = 1 (), and convex clustering with q = 2 ().

Fig 3.

Simulation results for Gaussian clusters with K = 3, n = p = 30, averaged over 200 data sets, for two noise levels σ = {1, 2}. Line types are as described in Figure 2.

Recall from Section 2.2 that there is a connection between convex clustering and single linkage clustering. However, we note that the two clustering methods are not equivalent. From Figure 2(a), we see that single linkage hierarchical clustering performs very similarly to convex clustering with q = 2 when the signal-to-noise ratio is high. However, from Figure 2(b), we see that single linkage hierarchical clustering outperforms convex clustering with q = 2 when the signal-to-noise ratio is low.

We also established a connection between convex clustering and k-means clustering in Section 2.3. From Figure 2(a), we see that k-means clustering and convex clustering with q = 2 perform similarly when two clusters are estimated and the signal-to-noise ratio is high. In this case, the first term in (14) can be made extremely small if the clusters are correctly estimated, and so both k-means and convex clustering yield the same (correct) cluster estimates. In contrast, when the signal-to-noise ratio is low, the first term in (14) is relatively large regardless of whether or not the clusters are correctly estimated, and so convex clustering focuses on minimizing the penalty term in (14). Therefore, when convex clustering with q = 2 estimates two clusters, one cluster is of size one and the other is of size n – 1, as discussed in Section 2.3. Figure 2(b) illustrates this phenomenon when both methods estimate two clusters: convex clustering with q = 2 has a Rand index of approximately 0.5 while k-means clustering has a Rand index of one.

All methods outperform convex clustering with q = 1. Moreover, k-means clustering and average linkage hierarchical clustering outperform single linkage hierarchical clustering and convex clustering when the signal-to-noise ratio is low. This suggests that the minimum signal needed for convex clustering to identify the correct clusters may be larger than that of average linkage hierarchical clustering and k-means clustering. We see similar results for the case when K = 3 in Figure 3.

4.2. Non-convex clusters

We consider two types of non-convex clusters: two circles clusters (Ng, Jordan and Weiss, 2002) and two half-moon clusters (Hocking et al., 2011; Chi and Lange, 2014a). For two circles clusters, we generate 50 data points from each of the two circles that are centered at (0, 0) with radiuses two and 10, respectively. We then add Gaussian random noise with mean zero and standard deviation 0.1 to each data point. For two half-moon clusters, we generate 50 data points from each of the two half-circles that are centered at (0, 0) and (30, 3) with radius 30, respectively. We then add Gaussian random noise with mean zero and standard deviation one to each data point. Illustrations of both types of clusters are given in Figure 4. The Rand indices for both types of clusters, averaged over 200 data sets, are summarized in Figure 5.

Fig 4.

Illustrations of two circles clusters and two half-moons clusters with n = 100.

Fig 5.

Simulation results for the two circles and two half-moons clusters with n = 100, averaged over 200 data sets. Line types are as described in Figure 2.

We see from Figure 5 that convex clustering with q = 2 and single linkage hierarchical clustering have similar performance, and that they outperform all of the other methods. Single linkage hierarchical clustering is able to identify non-convex clusters since it is an agglomerative algorithm that merges the closest pair of observations not yet belonging to the same cluster into one cluster. In contrast, average linkage hierarchical clustering and k-means clustering are known to perform poorly on identifying non-convex clusters (Ng, Jordan and Weiss, 2002; Hocking et al., 2011). Again, convex clustering with q = 1 has the worst performance.

4.3. Selection of the tuning parameter λ

Convex clustering (3) involves a tuning parameter λ, which determines the estimated number of clusters. Some authors have suggested a hold-out validation approach to select tuning parameters for clustering problems (see, for instance, Tan and Witten, 2014; Chi, Allen and Baraniuk, 2014). In this section, we present an alternative approach for selecting λ using the unbiased estimators of the degrees of freedom derived in Section 3.3.

The Bayesian Information Criterion (BIC) developed in Schwarz (1978) has been used extensively for model selection. However, it is known that the BIC does not perform well unless the number of observations is far larger than the number of parameters (Chen and Chen, 2008, 2012). For convex clustering (3), the number of observations is equal to the number of parameters. Thus, we consider the extended BIC (Chen and Chen, 2008, 2012), defined as

$${\mathrm{eBIC}}_{q,\gamma }=np\cdot \log \left(\frac{{\mathrm{RSS}}_q}{np}\right)+\widehat{\mathrm{d}}{\mathrm{f}}_q\cdot \log \left(np\right)+2\gamma \cdot \widehat{\mathrm{d}}{\mathrm{f}}_q\cdot \log \left(np\right),$$ (19)

where ${\mathrm{RSS}}_q={\Vert \mathbf{x}-{\widehat{\mathbf{u}}}_q\Vert }_2^2,{\widehat{\mathbf{u}}}_q$ is the convex clustering estimate for a given value of q and λ, γ ∈ [0, 1], and d̂f_q is given in Section 3.3. Note that we suppress the dependence of û_q and d̂f_q on λ for notational convenience. We see that when γ = 0, the extended BIC reduces to the classical BIC.

To evaluate the performance of the extended BIC in selecting the number of clusters, we generate Gaussian clusters with K = 2 and K = 3 as described in Section 4.1, with n = p = 20, and σ = 0.5. We perform convex clustering with q = 2 over a fine grid of λ, and select the value of λ for which the quantity eBIC_q,γ is minimized. We consider γ ∈ {0, 0.5, 0.75, 1}. Table 1 reports the proportion of datasets for which the correct number of clusters was identified, as well as the average Rand index.

Table 1.

Simulation study to evaluate the performance of the extended BIC for tuning parameter selection for convex clustering with q = 2. Results are reported over 100 simulated data sets. We report the proportion of data sets for which the correct number of clusters was identified, and the average Rand index.

	eBIC2,γ	Correct number of clusters	Rand index
Gaussian clusters, K = 2	γ = 0	0.94	0.9896
	γ = 0.5	0.98	0.9991
	γ = 0.75	0.99	0.9995
	γ =1	0.99	0.9995
Gaussian clusters, K = 3	γ = 0	0.06	0.7616
	γ = 0.5	0.59	0.9681
	γ = 0.75	0.70	0.9768
	γ =1	0.84	0.9873

From Table 1, we see that the extended BIC is able to select the true number of clusters accurately for K = 2. When K = 3, the classical BIC (γ = 0) fails to select the true number of clusters. In contrast, the extended BIC with γ = 1 has the best performance.

5. Discussion

Convex clustering recasts the clustering problem into a penalized regression problem. By studying its dual problem, we show that there is a connection between convex clustering and single linkage hierarchical clustering. In addition, we establish a connection between convex clustering and k-means clustering. We also establish several statistical properties of convex clustering. Through some numerical studies, we illustrate that the performance of convex clustering may not be appealing relative to traditional clustering methods, especially when the signal-to-noise ratio is low.

Many authors have proposed a modification to the convex clustering problem (1),

$$\underset{\mathbf{U}\in {\mathbb{R}}^{n\times p}}{\mathrm{minimize}}\frac{1}{2}\sum _{i=1}^n{\Vert {\mathbf{X}}_{i.}-{\mathbf{U}}_{i.}\Vert }_2^2+\lambda {\mathrm{Q}}_q(\mathbf{W},\mathbf{U}),$$ (20)

where W is an n × n symmetric matrix of positive weights, and ${\mathrm{Q}}_q(\mathbf{W},\mathbf{U})={\sum }_{i<i^{\prime }}{W}_{i{i}^{\prime }}{\Vert {\mathbf{U}}_{i.}-{\mathbf{U}}_{i^{\prime }.}\Vert }_q$ (Pelckmans et al., 2005; Hocking et al., 2011; Lindsten, Ohlsson and Ljung, 2011; Chi and Lange, 2014a). For instance, the weights can be defined as $W_{i{i}^{\prime }}=\exp (-\varphi {\Vert {\mathbf{X}}_{i.}-{\mathbf{X}}_{i^{\prime }.}\Vert }_2^2)$ for some constant ϕ > 0. This yields better empirical performance than (1) (Hocking et al., 2011; Chi and Lange, 2014a). We leave an investigation of the properties of (20) to future work.

Appendix A: Proof of Lemmas 2–3

Proof of Lemma 2

We rewrite problem (3) as

$$\underset{\mathbf{u}\in {\mathbb{R}}^{np},{\eta }_1\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}}{\mathrm{minimize}}\frac{1}{2}{\Vert \mathbf{x}-\mathbf{u}\Vert }_2^2+\lambda {\mathrm{P}}_q\left({\eta }_1\right)\text{subject to}{\eta }_1=\mathbf{Du},$$

with the Lagrangian function

$$\mathcal{L}(\mathbf{u},{\eta }_1,\nu )\frac{1}{2}{\Vert \mathbf{x}-\mathbf{u}\Vert }_2^2+\lambda {\mathrm{P}}_q\left({\eta }_1\right)+{\nu }^T(\mathbf{Du}-{\eta }_1),$$ (A-1)

where $\nu \in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}$ is the Lagrangian dual variable. In order to derive the dual problem, we need to minimize the Lagrangian function over the primal variables u and η₁. Recall from (4) that ${\mathrm{P}}_q^{\ast }(\cdot )$ is the dual norm of ${\mathrm{P}}_q(\cdot )$. It can be shown

$$\underset{{\eta }_1\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}}{\inf }\mathcal{L}(\mathbf{u},{\eta }_1,\nu )=\{\begin{array}{cc}\frac{1}{2}{\Vert \mathbf{x}-\mathbf{u}\Vert }_2^2+{\nu }^T\mathrm{Du}\hfill & \text{if}{\mathrm{P}}_q^{\ast }\left(\nu \right)\le \lambda ,\hfill \\ -\infty \hfill & \text{otherwise},\hfill \end{array}\phantom{\}}$$

and

$$\underset{{\eta }_1\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]},\mathbf{u}\in {\mathbb{R}}^{np}}{\inf }\mathcal{L}(\mathbf{u},{\eta }_1,\nu )=\{\begin{array}{cc}-\frac{1}{2}{\Vert \mathbf{x}-{\mathbf{D}}^T\nu \Vert }_2^2+\frac{1}{2}{\Vert \mathbf{x}\Vert }_2^2\hfill & \text{if}{\mathrm{P}}_q^{\ast }(\nu \le \lambda .\hfill \\ -\infty \hfill & \text{otherwise}.\hfill \end{array}\phantom{\}}$$

Therefore, the dual problem for (3) is

$$\underset{\nu \in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}}{\mathrm{minimize}}\frac{1}{2}{\Vert \mathbf{x}-{\mathbf{D}}^T\nu \Vert }_2^2\text{subject to}{\mathrm{P}}_q^{\ast }\left(\nu \right)\le \lambda .$$ (A-2)

We now establish an explicit relationship between the solution to convex clustering and its dual problem. differentiating the Lagrangian function (A-1) with respect to u and setting it equal to zero, we obtain

$$\widehat{\mathbf{u}}=\mathbf{x}-{\mathbf{D}}^T\widehat{\nu },$$

where $\widehat{\nu }$ is the solution to the dual problem, which satisfies ${\mathrm{P}}_q^{\ast }\left(\widehat{\nu }\right)\le \lambda$ by (A-2). Multiplying both sides by D, we obtain the relationship (6).

Proof of Lemma 3

We rewrite (7) as

$$\underset{\gamma \in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]},{\eta }_2\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}}{\mathrm{minimize}}\frac{1}{2}{\Vert \mathbf{Dx}-\gamma \Vert }_2^2+\lambda {\mathrm{P}}_q\left({\eta }_2\right)\text{subject to}{\eta }_2=\gamma ,$$

with the Lagrangian function

$$\mathcal{L}(\gamma ,{\eta }_2,{\nu }^{\prime })=\frac{1}{2}{\Vert \mathbf{Dx}-\gamma \Vert }_2^2+\lambda {\mathrm{P}}_q\left({\eta }_2\right)+{\left({\nu }^{\prime }\right)}^T(\gamma -{\eta }_2),$$ (A-3)

where ${\nu }^{\prime }\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}$ is the Lagrangian dual variable. In order to derive the dual problem, we minimize the Lagrangian function over the primal variables γ and η₂. It can be shown that

$$\underset{{\eta }_2\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}}{\inf }\mathcal{L}(\gamma ,{\eta }_2,{\nu }^{\prime })=\{\begin{array}{cc}\frac{1}{2}{\Vert \mathbf{Dx}-\gamma \Vert }_2^2+{\left({\nu }^{\prime }\right)}^T\gamma \hfill & \text{if}{\mathrm{P}}_q^{\ast }\left({\nu }^{\prime }\right)\le \lambda ,\hfill \\ -\infty \hfill & \text{otherwise},\hfill \end{array}\phantom{\}}$$

and

$$\underset{{\eta }_2\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]},\gamma \in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}}{\inf }\mathcal{L}(\gamma ,{\eta }_2,{\nu }^{\prime })=\{\begin{array}{cc}-\frac{1}{2}{\Vert \mathbf{Dx}-{\nu }^{\prime }\Vert }_2^2+\frac{1}{2}{\Vert \mathbf{Dx}\Vert }_2^2\hfill & \text{if}{\mathrm{P}}_q^{\ast }\left({\nu }^{\prime }\right)\le \lambda .\hfill \\ -\infty \hfill & \text{otherwise}.\hfill \end{array}\phantom{\}}$$

Therefore, the dual problem for (7) is

$$\underset{{\nu }^{\prime }\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}}{\mathrm{minimize}}\frac{1}{2}{\Vert \mathbf{Dx}-{\nu }^{\prime }\Vert }_2^2\text{subject to}{\mathrm{P}}_q^{\ast }\left({\nu }^{\prime }\right)\le \lambda .$$ (A-4)

We now establish an explicit relationship between the solution to (7) and its dual problem. Differentiating the Lagrangian function (A-3) with respect to γ and setting it equal to zero, we obtain

$$\widehat{\gamma }=\mathbf{Dx}-{\widehat{\nu }}^{\prime },$$

where ${\widehat{\nu }}^{\prime }$ is the solution to the dual problem, which we know from (A-4) satisfies ${\mathrm{P}}_q^{\ast }\left({\widehat{\nu }}^{\prime }\right)\le \lambda$.

Appendix B: Proof of Lemma 5

Proof of Lemma 5

Since D is not of full rank by Lemma 1(i), the solution to (5) in the absence of constraint is not unique, and takes the form

$$\begin{array}{cc}\hfill \widehat{\nu }& ={\left({\mathbf{DD}}^T\right)}^{†}\mathbf{Dx}+(\mathbf{I}-\mathbf{D}{\left({\mathbf{D}}^T\mathbf{D}\right)}^{†}{\mathbf{D}}^T)\omega \hfill \\ \hfill & ={\left({\mathbf{D}}^T\right)}^{†}\mathbf{x}+(\mathbf{I}-{\mathbf{DD}}^{†})\omega \hfill \\ \hfill & =\frac{1}{n}\mathbf{Dx}+(\mathbf{I}-\frac{1}{n}{\mathbf{DD}}^T)\omega ,\hfill \end{array}$$ (B-1)

for $\omega \in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}$. The second equality follows from Lemma 1(iii) and the last equality follows from Lemma 1(ii).

Let û be the solution to (3). Substituting $\widehat{\nu }$ given in (B-1) into (6), we obtain

$$\begin{array}{cc}\hfill \mathbf{D}\widehat{\mathbf{u}}& =\mathbf{Dx}-{\mathbf{DD}}^T\widehat{\nu }\hfill \\ \hfill & =\mathbf{Dx}-\frac{1}{n}{\mathbf{DD}}^T\mathbf{Dx}-{\mathbf{DD}}^T\omega +\frac{1}{n}{\mathbf{DD}}^T{\mathbf{DD}}^T\omega \hfill \\ \hfill & =\mathbf{Dx}-\mathbf{Dx}-{\mathbf{DD}}^T\omega +{\mathbf{DD}}^T\omega \hfill \\ \hfill & =0.\hfill \end{array}$$

Recall from Definition 1 that all observations are estimated to belong to the same cluster if Dû = 0. For any $\widehat{\nu }$ in (B-1), picking $\lambda ={\mathrm{P}}_q^{\ast }\left(\widehat{\nu }\right)$) guarantees that the constraint on the dual problem (5) is inactive, and therefore that convex clustering has a trivial solution of Dû = 0.

Since $\widehat{\nu }$ is not unique, ${\mathrm{P}}_q^{\ast }\left(\widehat{\nu }\right)$ is not unique. In order to obtain the smallest tuning parameter λ such that Dû = 0, we take

$${\lambda }_{\text{upper}}≔\underset{\omega \in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}}{\min }{\mathrm{P}}_q^{\ast }\left(\frac{1}{n}\mathbf{Dx}+\left(\mathbf{I}-\frac{1}{n}{\mathbf{DD}}^T\right)\omega \right).$$

Any tuning parameter λ ≥ λ_upper results in an estimate for which all observations belong to a single cluster. The proof is completed by recalling the definition of the dual norm ${\mathrm{P}}_q^{\ast }(\cdot )$ in (4).

Appendix C: Proof of Lemmas 6–7

To prove Lemmas 6 and 7, we need a lemma on the tail bound for quadratic forms of independent sub-Gaussian random variables.

Lemma 10

(Hanson and Wright, 1971) Let z be a vector of independent sub-Gaussian random variables with mean zero and variance σ². Let M be a symmetric matrix. Then, there exists some constants c₁, c₂ > 0 such that for any t > 0,

$$\mathrm{Pr}({\mathbf{z}}^T\mathbf{Mz}\ge t+{\sigma }^2\mathrm{tr}\left(\mathbf{M}\right))\le \exp \left\{-\min \left(\frac{c_1{t}^2}{{\sigma }^4{\Vert \mathbf{M}\Vert }_{\mathrm{F}}},\frac{c_{2}t}{{\sigma }^2{\Vert \mathbf{M}\Vert }_{\mathrm{sp}}}\right)\right\},$$

where ∥ · ∥_F and ∥ · ∥_sp are the Frobenius norm and spectral norm, respectively.

In order to simplify our analysis, we start by reformulating (3) as in Liu, Yuan and Ye (2013). Let $\mathbf{D}=\mathbf{A}\Lambda {\mathbf{V}}_{\beta }^T$ be the singular value decomposition of D, where $\mathbf{A}\in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]\times p(n-1)},\Lambda \in {\mathbb{R}}^{p(n-1)\times p(n-1)}$, and ${\mathbf{V}}_{\beta }\in {\mathbb{R}}^{np\times p(n-1)}$. Construct ${\mathbf{V}}_{\alpha }\in {\mathbb{R}}^{np\times p}$ such that $\mathbf{V}=[{\mathbf{V}}_{\alpha },{\mathbf{V}}_{\beta }]\in {\mathbb{R}}^{np\times np}$ is an orthogonal matrix, that is, V^TV = VV^T = I. Note that ${\mathbf{V}}_{\alpha }^T{\mathbf{V}}_{\beta }=0$.

Let $\beta ={\mathbf{V}}_{\beta }^T\mathbf{u}\in {\mathbb{R}}^{p(n-1)}$ and $\alpha ={\mathbf{V}}_{\alpha }^T\mathbf{u}\in {\mathbb{R}}^p$. Also, let ${\lambda }^{\prime }=\frac{\lambda }{np}$. Optimization problem (3) then becomes

$$\underset{\alpha \in {\mathbb{R}}^p,\beta \in {\mathbb{R}}^{p(n-1)}}{\mathrm{minimize}}\frac{1}{2np}{\Vert \mathbf{x}-{\mathbf{V}}_{\alpha }\alpha -{\mathbf{V}}_{\beta }\beta \Vert }^2+{\lambda }^{\prime }{\mathrm{P}}_q\left(\mathbf{Z}\beta \right),$$ (C-1)

where $\mathbf{Z}=\mathbf{A}\Lambda \in {\mathbb{R}}^{[p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]\times p(n-1)}$. Note that rank(Z) = p(n – 1) and therefore, there exists a pseudo-inverse ${\mathbf{Z}}^{†}\in {\mathbb{R}}^{p(n-1)\times [p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)]}$ such that Z^†Z = I. Recal from Section 1 that the set $\mathcal{C}(i,i^{\prime })$ contains the row indices of D such that ${\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\mathbf{u}={\mathbf{U}}_{i.}-{\mathbf{U}}_{i^{\prime }.}$. Let the submatrices ${\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}$ and ${\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}$ denote the rows of Z and the columns of Z^†, respectively, corresponding to the indices in the set $\mathcal{C}(i,i^{\prime })$. By Lemma 1(v),

$$\begin{array}{cc}\hfill & {\Lambda }_{\min }\left(\mathbf{Z}\right)={\Lambda }_{\min }\left(\mathbf{D}\right)=\frac{1}{{\Lambda }_{\max }\left({\mathbf{Z}}^{†}\right)}=\sqrt{n}\hfill \\ \hfill & {\Lambda }_{\max }\left(\mathbf{Z}\right)={\Lambda }_{\max }\left(\mathbf{D}\right)=\frac{1}{{\Lambda }_{\min }\left({\mathbf{Z}}^{†}\right)}=\sqrt{n}.\hfill \end{array}$$ (C-2)

Let $\widehat{\alpha }$ and $\widehat{\beta }$ denote the solution to (C-1).

Proof of Lemma 6

We establish a finite sample bound for the prediction error of convex clustering with q = 1 by analyzing (C-1). First, note that $\widehat{\mathbf{u}}={\mathbf{V}}_{\alpha }\widehat{\alpha }+{\mathbf{V}}_{\beta }\widehat{\beta }$ and $\mathbf{u}={\mathbf{V}}_{\alpha }\alpha +{\mathbf{V}}_{\beta }\beta$. Thus, $\frac{1}{2np}{\Vert \widehat{\mathbf{u}}-\mathbf{u}\Vert }^2=\frac{1}{2np}{\Vert {\mathbf{V}}_{\alpha }(\widehat{\alpha }-\alpha )+{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )\Vert }^2$. Recall from (2) that P₁(Zβ) = ∥Zβ∥₁. By the definition of $\widehat{\alpha }$ and $\widehat{\beta }$, we have

$$\frac{1}{2np}{\Vert \mathbf{x}-({\mathbf{V}}_{\alpha }\widehat{\alpha }+{\mathbf{V}}_{\beta }\widehat{\beta })\Vert }^2+{\lambda }^{\prime }{\Vert \mathbf{Z}\widehat{\beta }\Vert }_1\le \frac{1}{2np}{\Vert \mathbf{x}-({\mathbf{V}}_{\alpha }\alpha +{\mathbf{V}}_{\beta }\beta )\Vert }^2+{\lambda }^{\prime }{\Vert \mathbf{Z}\beta \Vert }_1,$$

implying

$$\frac{1}{2np}{\Vert {\mathbf{V}}_{\alpha }(\widehat{\alpha }-\alpha )+{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )\Vert }^2+{\lambda }^{\prime }{\Vert \mathbf{Z}\widehat{\beta }\Vert }_1\le \frac{1}{np}\mathrm{G}(\widehat{\alpha },\widehat{\beta })+{\lambda }^{\prime }{\Vert \mathbf{Z}\beta \Vert }_1,$$ (C-3)

where $\mathrm{G}(\widehat{\alpha },\widehat{\beta })={ϵ}^T[{\mathbf{V}}_{\alpha }(\widehat{\alpha }-\alpha )+{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )]$. Recall that ${\mathbf{V}}_{\alpha }^T{\mathbf{V}}_{\alpha }=\mathbf{I}$ and ${\mathbf{V}}_{\alpha }^T{\mathbf{V}}_{\beta }=0$. By the optimality condition of (C-1),

$$\begin{array}{cc}\hfill \widehat{\alpha }& ={\mathbf{V}}_{\alpha }^T(\mathbf{x}-{\mathbf{V}}_{\beta }\widehat{\beta })\hfill \\ \hfill & ={\mathbf{V}}_{\alpha }^T({\mathbf{V}}_{\alpha }\alpha +{\mathbf{V}}_{\beta }\beta +ϵ-{\mathbf{V}}_{\beta }\widehat{\beta })\hfill \\ \hfill & =\alpha +{\mathbf{V}}_{\alpha }^Tϵ.\hfill \end{array}$$

Therefore, substituting $\widehat{\alpha }-\alpha ={\mathbf{V}}_{\alpha }^Tϵ$ into $\mathrm{G}(\widehat{\alpha },\widehat{\beta })$, we obtain

$$\begin{array}{cc}\hfill \frac{1}{np}\mid \mathrm{G}(\widehat{\alpha },\widehat{\beta })\mid & =\frac{1}{np}\mid {ϵ}^T\left[{\mathbf{V}}_{\alpha }(\widehat{\alpha }-\alpha )+{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )\right]\mid \hfill \\ \hfill & =\frac{1}{np}\mid {ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ+{ϵ}^T{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )\mid \hfill \\ \hfill & \le \frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ+\frac{1}{np}\mid {ϵ}^T{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )\mid \hfill \\ \hfill & =\frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ+\frac{1}{np}\mid {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}^{†}\mathbf{Z}(\widehat{\beta }-\beta )\mid \hfill \\ \hfill & \le \frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ+\frac{1}{np}{\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}^{†}\Vert }_{\infty }{\Vert \mathbf{Z}(\widehat{\beta }-\beta )\Vert }_1.\hfill \end{array}$$

We now establish bounds for $\frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ$ and $\frac{1}{np}{\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}^{†}\Vert }_{\infty }$ that hold with high probability.

Bound for $\frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ$

First, note that ${\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^T$ is a projection matrix of rank p, and therefore ${\Vert {\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^T\Vert }_{\mathrm{sp}}=1$ and ${\Vert {\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^T\Vert }_F=p$. By Lemma 10 and taking z = ϵ and $\mathbf{M}={\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^T$, we have that

$$\mathrm{Pr}({ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ\ge t+{\sigma }^{2}p)\le \exp \left\{-\min \left(\frac{c_1{t}^2}{{\sigma }^{4}p},\frac{c_{2}t}{{\sigma }^2}\right)\right\},$$

where c₁ and c₂ are constants in Lemma 10. Picking $t={\sigma }^2\sqrt{p\log \left(np\right)}$, we have

$$\mathrm{Pr}\left(\frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ\ge {\sigma }^2\left[\frac{1}{n}+\sqrt{\frac{\log \left(np\right)}{n^{2}p}}\right]\right)\le \exp \left\{-\min \left(c_1\log \left(np\right),c_2\sqrt{p\log \left(np\right)}\right)\right\}.$$ (C-4)

Bound for $\frac{1}{np}{\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}^{†}\Vert }_{\infty }$

Let e_j be a vector of length $p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)$ with a one in the jth entry and zeroes in the remaining entries. Let $v_j=e_j^T{\left({\mathbf{Z}}^{†}\right)}^T{\mathbf{V}}_{\beta }^Tϵ$. Using the fact that Λ_max(V_β) = 1 and ${\Lambda }_{\max }\left({\mathbf{Z}}^{†}\right)=\frac{1}{\sqrt{n}}$ (C-2), we know that each v_j is a sub-Gaussian random variable with zero mean and variance at most $\frac{{\sigma }^2}{n}$. Therefore, by the union bound,

$$\mathrm{Pr}\left(\underset{j}{\max }\mid v_j\mid \ge z\right)\le p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)\cdot \mathrm{Pr}(\mid v_j\mid \ge z)\le 2p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)\exp \left(-\frac{n{z}^2}{2{\sigma }^2}\right).$$

Picking $z=2\sigma \sqrt{\frac{\log (p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right))}{n}}$, we obtain

$$\mathrm{Pr}\left({\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}^{†}\Vert }_{\infty }\ge 2\sigma \sqrt{\frac{\log (p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right))}{n}}\right)\le \frac{2}{p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)}.$$ (C-5)

Combining the two upper bounds

Setting ${\lambda }^{\prime }>4\sigma \sqrt{\frac{\log (p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right))}{n^3{p}^2}}$ and combining the results from (C-4) and (C-5), we obtain

$$\frac{1}{np}\mathrm{G}(\widehat{\alpha },\widehat{\beta })\le {\sigma }^2\left[\frac{1}{n}+\sqrt{\frac{\log \left(np\right)}{n^{2}p}}\right]+\frac{{\lambda }^{\prime }}{2}{\Vert \mathbf{Z}(\widehat{\beta }-\beta )\Vert }_1$$ (C-6)

with probability at least $1-\frac{2}{p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)}-\exp \{-\min (c_1\log \left(np\right),c_2\sqrt{p\log \left(np\right)})\}$. Substituting (C-6) into (C-3), we obtain

$$\begin{array}{cc}\hfill & \frac{1}{2np}{\Vert {\mathbf{V}}_{\alpha }(\widehat{\alpha }-\alpha )+{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )\Vert }^2+{\lambda }^{\prime }{\Vert \mathbf{Z}\widehat{\beta }\Vert }_1\hfill \\ \hfill \le & {\sigma }^2\left[\frac{1}{n}+\sqrt{\frac{\log \left(np\right)}{n^{2}p}}\right]+\frac{{\lambda }^{\prime }}{2}{\Vert \mathbf{Z}(\widehat{\beta }-\beta )\Vert }_1+{\lambda }^{\prime }{\Vert \mathbf{Z}\beta \Vert }_1.\hfill \end{array}$$

We get Lemma 6 by an application of the triangle inequality and by rearranging the terms.

Proof of Lemma 7

We establish a finite sample bound for the prediction error of convex clustering with q = 2 by analyzing (C-1). Recall from (2) that ${\mathrm{P}}_2\left(\mathbf{Z}\beta \right)={\sum }_{i<i^{\prime }}{\Vert {\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}\beta \Vert }_2$. By the definition of $\widehat{\alpha }$ and $\widehat{\beta }$, we have

$$\begin{array}{cc}\hfill & \frac{1}{2np}{\Vert \mathbf{x}-({\mathbf{V}}_{\alpha }\widehat{\alpha }+{\mathbf{V}}_{\beta }\widehat{\beta })\Vert }^2+{\lambda }^{\prime }\sum _{i<i^{\prime }}{\Vert {\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}\widehat{\beta }\Vert }_2\hfill \\ \hfill & \le \frac{1}{2np}{\Vert \mathbf{x}-({\mathbf{V}}_{\alpha }\alpha +{\mathbf{V}}_{\beta }\beta )\Vert }^2+{\lambda }^{\prime }\sum _{i<i^{\prime }}{\Vert {\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}\beta \Vert }_2,\hfill \end{array}$$

implying

$$\begin{array}{cc}\hfill & \frac{1}{2np}{\Vert {\mathbf{V}}_{\alpha }(\widehat{\alpha }-\alpha )+{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )\Vert }^2+{\lambda }^{\prime }\sum _{i<i^{\prime }}{\Vert {\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}\widehat{\beta }\Vert }_2\hfill \\ \hfill & \le \frac{1}{np}\mathrm{G}(\widehat{\alpha },\widehat{\beta })+{\lambda }^{\prime }\sum _{i<i^{\prime }}{\Vert {\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}\beta \Vert }_2,\hfill \end{array}$$ (C-7)

where $\mathrm{G}(\widehat{\alpha },\widehat{\beta })={ϵ}^T[{\mathbf{V}}_{\alpha }(\widehat{\alpha }-\alpha )+{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )]$. Again, by the optimality condition of (C-1), we have that $\widehat{\alpha }-\alpha ={\mathbf{V}}_{\alpha }^Tϵ$. Substituting this into $\frac{1}{np}\mathrm{G}(\widehat{\alpha },\widehat{\beta })$, we obtain

$$\begin{array}{cc}\hfill \frac{1}{np}\mid \mathrm{G}(\widehat{\alpha },\widehat{\beta })\mid & =\frac{1}{np}\mid {ϵ}^T\left[{\mathbf{V}}_{\alpha }(\widehat{\alpha }-\alpha )+{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )\right]\mid \hfill \\ \hfill & =\frac{1}{np}\mid {ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ+{ϵ}^T{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )\mid \hfill \\ \hfill & \le \frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ+\frac{1}{np}\mid {ϵ}^T{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )\mid \hfill \\ \hfill & =\frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ+\frac{1}{np}\mid {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}^{†}\mathbf{Z}(\widehat{\beta }-\beta )\mid \hfill \\ \hfill & =\frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ+\frac{1}{np}\mid \sum _{i<i^{\prime }}\left({ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\right)\left({\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}(\widehat{\beta }-\beta )\right)\mid \hfill \\ \hfill & \le \frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ+\frac{1}{np}\sum _{i<i^{\prime }}\mid \left({ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\right)\left({\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}(\widehat{\beta }-\beta )\right)\mid \hfill \\ \hfill & \le \frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ+\frac{1}{np}\sum _{i<i^{\prime }}{\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\Vert }_2{\Vert {\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}(\widehat{\beta }-\beta )\Vert }_2\hfill \\ \hfill & \le \frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ+\frac{1}{np}\cdot \underset{i<i^{\prime }}{\max }{\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\Vert }_2\sum _{i<i^{\prime }}{\Vert {\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}(\widehat{\beta }-\beta )\Vert }_2,\hfill \end{array}$$

where the second inequality follows from an application of the triangle inequality and the third inequality from an application of the Cauchy-Schwarz inequality. We now establish bounds for $\frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ$ and $\frac{1}{np}\cdot \underset{i<i^{\prime }}{\max }{\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\Vert }_2$ that hold with large probability.

Bound for $\frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ$

This is established in the proof of Lemma 6 in (C-4), i.e.,

$$\mathrm{Pr}\left(\frac{1}{np}{ϵ}^T{\mathbf{V}}_{\alpha }{\mathbf{V}}_{\alpha }^Tϵ\ge {\sigma }^2\left[\frac{1}{n}+\sqrt{\frac{\log \left(np\right)}{n^{2}p}}\right]\right)\le \frac{1}{np}.$$

Bound for $\frac{1}{np}\cdot \underset{i<i^{\prime }}{\max }{\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\Vert }_2$

First, note that there are p indices in each set $\mathcal{C}(i,i^{\prime })$. Therefore,

$${\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\Vert }_2\le \sqrt{p}\cdot {\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\Vert }_{\infty }.$$

Note that

$$\frac{1}{np}\cdot \underset{i<i^{\prime }}{\max }{\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\Vert }_2\le \sqrt{\frac{1}{n^{2}p}}\cdot \underset{i<i^{\prime }}{\max }{\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\Vert }_{\infty }=\sqrt{\frac{1}{n^{2}p}}\cdot {\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}^{†}\Vert }_{\infty }.$$ (C-8)

Therefore, using (C-8),

$$\begin{array}{cc}\hfill & \mathrm{Pr}\left(\frac{1}{np}\cdot \underset{i<i^{\prime }}{\max }{\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\Vert }_2\ge 2\sigma \sqrt{\frac{\log (p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right))}{n^{3}p}}\right)\hfill \\ \hfill \le & \mathrm{Pr}\left({\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}^{†}\Vert }_{\infty }\ge 2\sigma \sqrt{\frac{\log (p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right))}{n}}\right)\hfill \\ \hfill \le & \frac{2}{p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)},\hfill \end{array}$$ (C-9)

where the last inequality follows from (C-5) in the proof of Lemma 6.

Therefore, for ${\lambda }^{\prime }>4\sigma \sqrt{\frac{\log (p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right))}{n^{3}p}}$, we have $\frac{{\lambda }^{\prime }}{2}<\frac{1}{np}\cdot \underset{i<i^{\prime }}{\max }{\Vert {ϵ}^T{\mathbf{V}}_{\beta }{\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}^{†}\Vert }_2$ with probability at most $\frac{2}{p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)}$. Combining the results from (C-4) and (C-9), we have that

$$\frac{1}{np}\mathrm{G}(\widehat{\alpha },\widehat{\beta })\le {\sigma }^2\left[\frac{1}{n}+\sqrt{\frac{\log \left(np\right)}{n^{2}p}}\right]+\frac{{\lambda }^{\prime }}{2}\sum _{i<i^{\prime }}{\Vert {\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}(\widehat{\beta }-\beta )\Vert }_2$$ (C-10)

with probability at least $1-\frac{2}{p\cdot \left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)}-\exp \{-\min (c_1\log \left(np\right),c_2\sqrt{p\log \left(np\right)})\}$. Substituting (C-10) into (C-7), we obtain

$$\begin{array}{cc}\hfill & \frac{1}{2np}{\Vert {\mathbf{V}}_{\alpha }(\widehat{\alpha }-\alpha )+{\mathbf{V}}_{\beta }(\widehat{\beta }-\beta )\Vert }^2+{\lambda }^{\prime }\sum _{i<i^{\prime }}{\Vert {\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}\widehat{\beta }\Vert }_2\hfill \\ \hfill \le & {\sigma }^2\left[\frac{1}{n}+\sqrt{\frac{\log \left(np\right)}{n^{2}p}}\right]+\frac{{\lambda }^{\prime }}{2}\sum _{i<i^{\prime }}{\Vert {\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}(\widehat{\beta }-\beta )\Vert }_2+{\lambda }^{\prime }\sum _{i<i^{\prime }}{\Vert {\mathbf{Z}}_{\mathcal{C}(i,i^{\prime })}\beta \Vert }_2.\hfill \end{array}$$

We get Lemma 7 by an application of the triangle inequality and by rearranging the terms.

Appendix D: Proof of Lemma 9

Proof of Lemma 9

Directly from the dual problem (5), ${\mathbf{D}}^T\widehat{\nu }$ is the projection of x onto the convex set $K=\{{\mathbf{D}}^T\nu :{\mathrm{P}}_2^{\ast }\left(\nu \right)\le \lambda \}$. Using the primal-dual relationship $\widehat{\mathbf{u}}=\mathbf{x}-{\mathbf{D}}^T\widehat{\nu }$, we see that û is the residual from projecting x onto the convex set K. By Lemma 1 of Tibshirani and Taylor (2012), û is continuous and almost differentiable with respect to x. Therefore, by Stein's formula, the degrees of freedom can be characterized as $\mathrm{E}\left[\mathrm{tr}\left(\frac{\partial \widehat{\mathbf{u}}}{\partial \mathbf{x}}\right)\right]$.

Recall that ${\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}$ denotes the rows of D corresponding to the indices in the set $\mathcal{C}(i,i^{\prime })$. Let ${\widehat{\mathcal{B}}}_2=\{(i,i^{\prime }):{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2\ne 0\}$. By the optimality condition of (3) with q = 2, we obtain

$$(\mathbf{x}-\widehat{\mathbf{u}})=\lambda \sum _{i<i^{\prime }}{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{g}_{\mathcal{C}(i,i^{\prime })},$$ (D-1)

where

$$g_{\mathcal{C}(i,i^{\prime })}=\{\begin{array}{cc}\frac{{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}}{{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2}\hfill & \text{if}(i,i^{\prime })\in {\widehat{\mathcal{B}}}_2.\hfill \\ \in \{\Gamma :{\Vert \Gamma \Vert }_2\le 1\}\hfill & \text{if}(i,i^{\prime })\notin {\widehat{\mathcal{B}}}_2.\hfill \end{array}\phantom{\}}$$

We define the matrix ${\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}$ by removing the rows of D that correspond to elements in ${\widehat{\mathcal{B}}}_2$. Let $\mathbf{P}=(\mathbf{I}-{\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}^T{\left({\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}{\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}^T\right)}^{†}{\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2})$ be the projection matrix onto the complement of space spanned by the rows of ${\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}$.

By the definition of ${\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}$, we obtain ${\mathbf{D}}_{-{\widehat{\mathcal{B}}}_2}\widehat{\mathbf{u}}=0$. Therefore, Pû = û. Multiplying P onto both sides of (D-1), we obtain

$$\mathbf{Px}-\widehat{\mathbf{u}}=\lambda \mathbf{P}\sum _{i<i^{\prime }}{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{g}_{\mathcal{C}(i,i^{\prime })}=\lambda \mathbf{P}\sum _{(i,i^{\prime })\in {\widehat{\mathcal{B}}}_2}\frac{{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}}{{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2},$$ (D-2)

where the second equality follows from the fact that ${\mathbf{PD}}_{\mathcal{C}(i,i^{\prime })}^T=0$ for any $(i,i^{\prime })\notin {\widehat{\mathcal{B}}}_2$.

Vaiter et al. (2014) showed that there exists a neighborhood around almost every x such that the set ${\widehat{\mathcal{B}}}_2$ is locally constant with respect to x. Therefore, the derivative of (D-2) with respect to x is

$$\mathbf{P}=\frac{\partial \widehat{\mathbf{u}}}{\partial \mathbf{x}}=\lambda \mathbf{P}\sum _{(i,i^{\prime })\in {\widehat{\mathcal{B}}}_2}\left(\frac{{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}}{{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2}-\frac{{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}{\widehat{\mathbf{u}}}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}}{{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2^3}\right)\frac{\partial \widehat{\mathbf{u}}}{\partial \mathbf{x}},$$ (D-3)

using the fact that for any matrix A with ${\Vert \mathbf{Av}\Vert }_2\ne 0,\frac{\partial }{\partial \mathbf{v}}\frac{{\mathbf{A}}^T\mathbf{Av}}{{\Vert \mathbf{Av}\Vert }_2}=\frac{{\mathbf{A}}^T\mathbf{A}}{{\Vert \mathbf{Av}\Vert }_2}-\frac{{\mathbf{A}}^T{\mathbf{Avv}}^T{\mathbf{A}}^T\mathbf{A}}{{\Vert \mathbf{Av}\Vert }_2^3}$.

Solving (D-3) for $\frac{\partial \widehat{\mathbf{u}}}{\partial \mathbf{x}}$, we have

$$\frac{\partial \widehat{\mathbf{u}}}{\partial \mathbf{x}}={\left[\mathbf{I}+\lambda \mathbf{P}\sum _{(i,i^{\prime })\in {\widehat{\mathcal{B}}}_2}\left(\frac{{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}}{{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2}-\frac{{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}{\widehat{\mathbf{u}}}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}}{{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2^3}\right)\right]}^{-1}\mathbf{P}.$$ (D-4)

Therefore, an unbiased estimator of the degrees of freedom is of the form

$$\mathrm{tr}\left(\frac{\partial \widehat{\mathbf{u}}}{\partial \mathbf{x}}\right)=\mathrm{tr}\left({\left[\mathbf{I}+\lambda \mathbf{P}\sum _{(i,i^{\prime })\in {\widehat{\mathcal{B}}}_2}\left(\frac{{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}}{{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2}-\frac{{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}{\widehat{\mathbf{u}}}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}^T{\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}}{{\Vert {\mathbf{D}}_{\mathcal{C}(i,i^{\prime })}\widehat{\mathbf{u}}\Vert }_2^3}\right)\right]}^{-1}\mathbf{P}\right).$$