Sparse kernel sufficient dimension reduction

Abstract

The sufficient dimension reduction (SDR) with sparsity has received much attention for analysing high-dimensional data. We study a nonparametric sparse kernel sufficient dimension reduction (KSDR) based on the reproducing kernel Hilbert space, which extends the methodology of the sparse SDR based on inverse moment methods. We establish the statistical consistency and efficient estimation of the sparse KSDR under the high-dimensional setting where the dimension diverges as the sample size increases. Computationally, we introduce a new nonconvex alternating directional method of multipliers (ADMM) to solve the challenging sparse SDR and propose the nonconvex linearised ADMM to solve the more challenging sparse KSDR. We study the computational guarantees of the proposed ADMMs and show an explicit iteration complexity bound to reach the stationary solution. We demonstrate the finite-sample properties in simulation studies and a real application.

1. Introduction

Consider the sufficient dimension reduction (SDR) of high-dimensional random vector $\mathit{x}\in {ℝ}^p$ in the presence of response $y\in ℝ$. We aim to find the low-dimensional central subspace ${𝒮}_{y|\text{x}}\subset {ℝ}^p$ such that dim $\left({𝒮}_{y|\text{x}}\right)=d\ll p$ and $y\mathbf{\perp }\mathit{x}|P_{{𝒮}_{y|\mathit{x}}}\left(\mathit{x}\right)$, where the dimension $p$ may diverge as the sample size increases, and $P_{{𝒮}_{y|\mathit{x}}}\left(\cdot \right)$ is the projection of a vector onto the subspace ${𝒮}_{y|\mathit{x}}$. The inverse moment methods are extensively studied under the fixed-dimension setting, including the sliced inverse regression (SIR) (Li 1991), sliced average variance estimation (SAVE) (Cook and Weisberg 1991), and directional regression (DR) (Li and Wang 2007). Fan, Xue, and Yao (2017), Luo, Xue, Yao, and Yu (2022), and Yu, Yao, and Xue (2022) extended these inverse moment methods and introduced sufficient forecasting using factor models for high-dimensional predictors when both $n$, $p\to \infty$. Ying and Yu (2022), Zhang, Xue, and Li (2024), and Zhang, Li, and Xue (2024) studied the sufficient dimension reduction for Fréchet regression and distribution-on-distribution regression. (Lin, Zhao, and Liu 2018; Lin, Zhao, and Liu 2019) introduced the Lasso-SIR under the high-dimensional setting when $p/n\to 0$ and the diagonal thresholding SIR (DT-SIR) introduced a new diagonal thresholding screening under the ultra high-dimensional setting when $p/n\to \infty$. However, the restricted linearity condition on the conditional expectation of $\mathit{x}$ given $P_{{𝒮}_{y|\mathit{x}}}\left(\mathit{x}\right)$ is required for inverse moment methods. To remove the linearity condition, the kernel sufficient dimension reduction (KSDR) was proposed by Fukumizu, Bach, and Jordan (2009). The nonconvex KSDR does not require the linearity condition nor the elliptical distribution of $\mathit{x}$, but it poses a significant computational challenge. Wu, Miller, Chang, Sznaier, and Dy (2019) solved this problem using an iterative spectral method, but such a method is hard to extend when a sparsity assumption is assumed in high-dimensional settings. Shi, Huang, Feng, and Suykens (2019) studies nonconvex penalty in a kernel regression problem. Although both the theoretical property and optimisation algorithm cannot be applied in SDR settings, it shows the promises of nonconvex penalised KSDR in high-dimensional settings.

The sparse estimation of central subspace has received considerable attention in the past decade. The main focus of the literature is on sparse SDR based on inverse moment methods, which is equivalent to solving the generalised eigenvectors from the estimated kernel matrix (Li 2007). For example, the kernel matrix for SIR is the conditional covariance matrix of $\mathit{x}$ given $y$. In view of such equivalence, Li (2007) followed the similar spirit of the sparse principal component analysis (SPCA) (Zou, Hastie, and Tibshirani 2006; Zou and Xue 2018) to propose the sparse SDR using ${\ell }_1$ penalisation. Bondell and Li (2009) proposed a shrinkage inverse regression estimation and studied the asymptotic consistency and asymptotic normality. Chen, Zou, and Cook (2010) proposed a coordinate-independent sparse estimation method and established the oracle property when the dimension $p$ is fixed. Lin et al. (2019) established the consistency of the Lasso-SIR by assuming the number of slices also diverges. Neykov, Lin, and Liu (2016) followed the convex semidefinite programming (d’Aspremont, Ghaoui, Jordan, and Lanckriet 2005; Amini and Wainwright 2008) to give a high-dimensional analysis for the sign consistency of the DT-SIR in a single-index model. Tan, Wang, Zhang, Liu, and Cook (2018) proposed a sparse SIR estimator based on convex formulation and established the Frobenius norm convergence of projection subspace. Lin et al. (2018) established the consistency of DT-SIR by assuming a stability condition for $E\left(\mathit{x}|y\right)$. Tan, Shi, and Yu (2020) proved the optimality of ${\ell }_0$-constrained sparse solution and provided a three-steps adaptive estimation to approximate $\ell -0$ solution with near-optimal property. The sparse estimation in the KSDR area has not been well studied.

In this paper, we propose the nonconvex estimation scheme for sparse KSDR, motivated by the nonconvex estimation of sparse SDR (Li 2007). The proposed methods target the ideal ${\ell }_0$-constrained estimator and can be efficiently solved with statistical and computational guarantees after approximation. We follow d’Aspremont et al. (2005) to reformulate sparse SDR and sparse KSDR as the ${\ell }_0$-constrained trace maximisation problem. It is worth pointing out that KSDR is also equivalent to solving the eigenvectors from the nonconvex residual covariance estimator in RKHS (Fukumizu et al. 2009). The residual covariance estimator in RKHS plays a similar role as the kernel matrix in inverse moment methods. Thus, we propose that sparse SDR solves the ${\ell }_0$-constrained convex M-estimation problem, whereas sparse KSDR solves the ${\ell }_0$-constrained nonconvex M-estimation problem. We provide a high-dimensional analysis for the global solution of the ${\ell }_0$-constrained SDR and prove its asymptotic properties, including both variable selection consistency and convergence rate. We demonstrate the power of this general theory by obtaining the explicit convergence rates for the ${\ell }_0$-constrained SDR under the high-dimensional setting when the dimension $p$ diverges. Furthermore, we also provide a high-dimensional analysis of the global solution of the ${\ell }_0$-constrained KSDR and prove its asymptotic properties, such as the consistency in variable selection and estimation, without assuming the linearity condition.

Computationally, the ${\ell }_0$-constrained trace maximisation is NP-hard. There has been some work showing success in adjusting the weight of coefficients adaptively, using convex penalties such as adaptive-lasso (Zou 2006) and adaptive-ridge regression (Frommlet and Nuel 2016). The performance of adaptive-based approaches heavily relies on the accuracy of initial estimation. Seeking for a more robust ${\ell }_0$ penalisation approximation without adaptive procedure, we use the folded concave penalisation (Fan and Li 2001; Fan, Xue, and Zou 2014) as an alternative to the ${\ell }_1$ penalisation in Li (2007), Lin et al. (2019), and Lin et al. (2018). We propose the novel nonconvex alternating direction method of multipliers (ADMM) to solve the folded concave penalised SDR and the nonconvex linearised ADMM to solve the folded concave penalised KSDR. In particular, sparse KSDR has not only a non-convex loss function but also a folded concave penalty and a positive definite constraint. We provide the important convergence guarantees of the proposed nonconvex ADMMs for solving two challenging nonconvex and nonsmooth problems. We also establish the explicit iteration complexity bound for the proposed nonconvex ADMM and its linearised variant to reach the stationary solution of sparse SDR and sparse KSDR, respectively. To the best of our knowledge, our work is the first in the literature to study the challenging nonconvex and nonsmooth optimisation problem for sparse KSDR.

The rest of this paper is organised as follows. We will revisit the methods for sparse SDR in Section 2 and then present the proposed methods for sparse KSDR in Section 3. The proposed nonconvex ADMM and its linearised variant are presented in Sectio 4, and statistical and computational guarantees are presented in Section 5. Section 6 evaluates the numerical performance in several different simulation settings. Section 7 applies sparse KDR in a real data analysis. Section 7 includes a few concluding remarks.

Before proceeding, we define some useful notations. Let ${‖\mathit{\theta }‖}_0=card\left\{{\theta }_j\ne 0\text{,}j=1\text{,}\dots d\right\}$, and $\left|𝒮\right|$ is the cardinality of a set $𝒮$. For a matrix $\mathit{A}$, let ${‖\mathit{A}‖}_{\infty }$ be the maximum norm, ${‖\mathit{A}‖}_1$ the matrix ${\ell }_1$ norm, ${‖\mathit{A}‖}_2$ the spectral norm, ${‖\mathit{A}‖}_F$ the Frobenius norm, ${‖\mathit{A}‖}_{tr}$ the trace norm, defined as ${‖\mathit{A}‖}_{tr}=Tr\left[{\left({\mathit{A}}^T\mathit{A}\right)}^{1/2}\right]$. Let ${‖\cdot ‖}_{{\mathscr{H}}_d}$ denote the norm in $d$-dimensional Hilbert space. To estimate the central subspace, it is equivalent to finding a low-rank matrix $\mathbf{\Theta }\in {ℝ}^{p\times d}$ of full column rank such that $y\mathit{\perp }\mathit{x}|{\mathbf{\Theta }}^T\mathit{x}$. We may also denote the central subspace by ${\mathit{S}}_{y|\mathit{x}}=\mathrm{span}\left(\mathbf{\Theta }\right)$, as the column space of matrix $\mathbf{\Theta }$.

2. Preliminaries

This section first revisits the sparse SDR methods in Li (2007) and then follows d’Aspremont et al. (2005) to reformulate sparse SDR via the trace maximisation problem. Note that SDR aims to estimate the linear combination $\mathbf{\Theta }$ such that $y\mathit{\perp }\mathit{x}|{\mathbf{\Theta }}^T\mathit{x}$ given independently and identically distributed data $\left({\mathit{x}}_1\text{,}{y}_1\right)\text{,}\dots \text{,}\left({\mathit{x}}_n\text{,}{y}_n\right)$. Li (2007) showed that SDR methods based on inverse moments, such as SIR, DR, SAVE, and their variants, essentially solve the generalised eigenvector of a certain kernel matrix. When we assume the product of the inverse covariance matrix and kernel matrix is a normal matrix, the problem is also equivalent to solving an eigenvector of a certain kernel matrix. More specifically, at population level, the kernel matrix of SIR is ${\mathit{\text{M}}}_{sir}^{★}={\mathbf{\Sigma }}_x^{-1}\text{cov}\left[E\left\{\mathit{z}|y\right\}\right]$; the kernel matrix of DR is:

$$\begin{array}{l}\begin{array}{l}{\mathit{\text{M}}}_{dr}^{★}=2{\sum }_x^{-1}\text{(}E\left[E^2\left(\mathit{z}{\mathit{z}}^T|y\right)\right]+E^2\left[E\left(\mathit{z}|y\right)E\left({\mathit{z}}^T|y\right)\right]\\ +E\left[E\left({\mathit{z}}^T|y\right)E\left(\mathit{z}|y\right)\right]E\left[E\left(\mathit{z}|y\right)E\left({\mathit{z}}^T|y\right)\right]-{\mathit{I}}_p\text{),}\end{array}\end{array}$$

where $\mathit{z}=\mathit{x}-E\left(\mathit{x}\right)$ and ${\mathbf{\Sigma }}_{\mathit{x}}$ the covariance matrix of all predictors $\mathit{x}$; the kernel matrix of DT-SIR (Lin et al. 2018) is ${\mathit{\text{M}}}_{DT-sir}^{★}={\sum }_{x_{DT}}^{-1}\text{cov}\left[E\left\{\left({\mathit{x}}_{DT}-E{\mathit{x}}_{DT}\right)|y\right\}\right]$, where ${\mathit{x}}_{DT}$ is the subset of random variables in $\mathit{x}$ after the diagonal thresholding.

Given the finite sample size $n$, we have an empirical version of such kernel matrices $\widehat{\mathit{\text{M}}}$, by replacing population covariance matrix ${\mathbf{\Sigma }}_{\mathit{x}}$ with sample covariance matrix ${\widehat{\mathbf{\Sigma }}}_{\mathit{x}}$, and all means $E\left(\cdot \right)$ with empirical mean $E_n\left(\cdot \right)$. For example, ${\widehat{\mathit{\text{M}}}}_{sir}={\widehat{\mathbf{\Sigma }}}_x^{-1}\text{cov}\left[E_n\left\{\mathit{z}|y\right\}\right]$. Li (2007) shows these SDR methods are equivalent to solving the leading eigenvector of the empirical version of such kernel matrices.

After estimating the kernel matrix $\widehat{\mathit{\text{M}}}$, we study the sparse estimation of the leading SDR direction, i.e. the largest eigenvector of $\widehat{\mathit{\text{M}}}$. We use the well-known fact: solving the largest eigenvector ${\mathit{\theta }}_{\text{1}}$ is equivalent to trace optimisation (Wold, Esbensen, and Geladi 1987) as follows:

$$\begin{array}{llll}\underset{\mathit{\theta }}{\text{max}}& \text{tr}\left({\mathit{\theta }}^T\widehat{M}\mathit{\theta }\right)& \text{s.t.}& {‖\mathit{\theta }‖}_2^2\le 1\end{array}\text{.}$$

To impose the sparsity, we introduce ${\ell }_0$ constraint in the following trace maximisation to penalise the cardinality of non-zero elements in vector $\mathit{\theta }$:

$$\begin{array}{lllll}{\widehat{\mathit{\theta }}}_{\text{1}}=\underset{\mathit{\theta }}{\text{arg}\text{max}}& \text{tr}\left({\mathit{\theta }}^T\widehat{M}\mathit{\theta }\right)& \text{s.t.}& {‖\mathit{\theta }‖}_2^2\le 1,& {‖\mathit{\theta }‖}_0\le t\end{array}$$

Note that $\text{span}\left({\mathit{\theta }}^{T}x\right)=\text{span}\left(\mathit{\theta }{\mathit{\theta }}^{T}x\right)$. With $\mathbf{\Theta }\text{=}\mathit{\theta }{\mathit{\theta }}^T$, then it is equivalent to solving the positive semidefinite estimate ${\widehat{\mathbf{\Theta }}}_{\text{1}}$ from

$$\begin{array}{lllllll}\underset{\mathbf{\Theta }\in {ℝ}^{p\times p}}{\text{max}}& \text{tr}\left(\widehat{M}\mathbf{\Theta }\right)& \text{s.t.}& \text{tr}\left(\mathbf{\Theta }\right)=1\text{;}& 1^T{‖\mathbf{\Theta }‖}_{0}1\le t^2\text{;}& \mathbf{\Theta }\underset{\_}{\succ }0\text{;}& \text{rank}\left(\mathbf{\Theta }\right)=1\end{array}$$

where ${‖\mathbf{\Theta }‖}_0$ is defined as the element-wise ${\ell }_0$ norm. When $\mathbf{\Theta }\underset{\_}{\succ }0$ and $\text{rank}\left(\mathbf{\Theta }\right)=1$, $\text{tr}\left(\mathbf{\Theta }\right)=1$ implies ${‖\mathit{\theta }‖}_2^2=1$, and $1^T{‖\mathbf{\Theta }‖}_{0}1\le t^2$ is equivalent to ${‖\mathit{\theta }‖}_0\le t$. However, the rank-one constraint that $\text{rank}\left(\mathbf{\Theta }\right)=1$ is hard to solve efficiently in practice. Therefore, we consider a relaxation of removing this constraint. Instead, we solve the dominant eigenvector of ${\widehat{\mathbf{\Theta }}}_{\text{1}}$ as ${\widehat{\mathit{\theta }}}_{\text{1}}$, which is the SDR direction of our interest. Generally, this relaxation tends to have a solution very close to rank one matrix (d’Aspremont et al. 2005).

After removing the rank-one constraint, it is equivalent to solve ${\widehat{\mathbf{\Theta }}}_{\text{1}}$ from the following ${\ell }_0$ penalised estimation problem for some sufficiently large $\gamma$:

$$\begin{array}{lllll}\underset{\mathbf{\Theta }\in {ℝ}^{p\times p}}{\text{max}}& \text{tr}\left(\widehat{M}\mathbf{\Theta }\right)-\gamma {\mathbf{1}}^T{‖\mathbf{\Theta }‖}_0\mathbf{1}& \text{s.t.}& \text{tr}\left(\mathbf{\Theta }\right)=1\text{;}& \mathbf{\Theta }\underset{\_}{\succ }0\text{,}\end{array}$$ (1)

where given ${\widehat{\mathbf{\Theta }}}_{\text{1}}$, we follow d’Aspremont et al. (2005) to truncate it to obtain the dominant (sparse) eigenvector of ${\widehat{\mathit{\theta }}}_{\text{1}}$, and it will be the leading sparse SDR direction of our interest.

In the sequel, given ${\widehat{\mathit{\theta }}}_{\text{1}}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k$, we solve the following sparse SDR direction ${\widehat{\mathit{\theta }}}_{k+1}$ in a sequential fashion. We denote ${\widehat{\mathit{\text{M}}}}_{\text{1}}=\widehat{\mathit{\text{M}}}$, and for $k\ge 1$, let ${\widehat{\mathit{\text{M}}}}_{k+1}$ be the projection of ${\widehat{\mathit{\text{M}}}}_k$ on the space orthogonal to previous SDR directions. To remove the effect of previous eigenvectors, ${\widehat{\mathit{\theta }}}_{k+1}$ is estimated as the largest (sparse) eigenvector of matrix ${\widehat{\mathit{\text{M}}}}_{k+1}$, which is a Hotelling’s deflation from previous vectors:

$${\widehat{\mathit{M}}}_{k+1}={\widehat{\mathit{M}}}_k-\left({\widehat{\mathbf{\theta }}}_k^T{\widehat{\mathit{M}}}_k{\widehat{\mathbf{\theta }}}_k\right){\widehat{\mathbf{\theta }}}_k{\widehat{\mathbf{\theta }}}_k^T.$$

Mackey (2009) studied the property of Hotelling’s deflation in this type of sparse eigenvector estimation problem. Especially when we estimate the sparse eigenvector on the true support, this deflation helps eliminate the influence of a given eigenvector by setting the associated eigenvalue to zero and achieve ${\widehat{\mathit{\theta }}}_1\mathit{\perp }{\widehat{\mathit{\theta }}}_2\mathit{\perp }\dots \mathit{\perp }{\widehat{\mathit{\theta }}}_k$. For the sparse estimation of ${\widehat{\mathit{\theta }}}_{k+1}$, we plug in ${\widehat{\mathit{\text{M}}}}_{k+1}$ to (1), then solve ${\mathit{\theta }}_{k+1}$ from:

$$\begin{array}{lllll}\underset{\mathit{\theta }\in {ℝ}^p}{\text{max}}& \text{tr}\left({\mathit{\theta }}^T{\widehat{M}}_{k+1}\mathit{\theta }\right)& \text{s.t.}& {‖\mathit{\theta }‖}_2^2\le 1\text{,}& {‖\mathit{\theta }‖}_{\text{0}}\le t\text{,}\end{array}$$

which is further equivalent to the following constrained problem following the same rank relaxation:

$$\begin{array}{lllll}\underset{\mathbf{\Theta }\in {ℝ}^{p\times p}}{\text{max}}& \text{tr}\left({\widehat{M}}_{k+1}\mathbf{\Theta }\right)-\gamma {\mathbf{1}}^T{‖\mathbf{\Theta }‖}_0\mathbf{1}& \text{s.t.}& \text{tr}\left(\mathbf{\Theta }\right)=1\text{;}& \mathbf{\Theta }\underset{\_}{\succ }0.\end{array}$$ (2)

(2) is still a nonconvex problem since ${\ell }_0$ constraint is included. To solve the computational challenge, one existing relaxation is using folded concave penalty to approximate ${\ell }_0$ constraint. Let $P_{\lambda }\left(\left|\mathbf{\Theta }\right|\right)$ be a folded concave penalty function such as the SCAD (Fan and Li 2001) and MCP (Zhang 2010), where $\left|\mathbf{\Theta }\right|$ is the matrix such that taking the absolute value of matrix $\mathbf{\Theta }$ element-wisely.

Specifically, in this paper, we consider the element-wise SCAD penalty $P_{\delta }\left(t\right)$ as:

$$P_{\delta }(t)=\left\{\begin{array}{ll}\frac{2at}{(a+1)\delta }\hfill & \left|t\right|\le \frac{\delta }{a},\hfill \\ 1-\frac{{(\delta -\left|t\right|)}^2}{\left(1-\frac{1}{a^2}\right){\delta }^2}\hfill & \frac{\delta }{a}\le \left|t\right|\le \delta ,\hfill \\ 1\hfill & \left|t\right|\ge \delta .\hfill \end{array}\right.$$

where $a$ is a constant, usually chosen as 3.7 in the SCAD penalty. In Section 5, we discuss how such penalisation can well approximate ${\ell }_0$ penalisation. In this paper, We use SCAD penalty in this paper for simplicity. It is worth mentioning that we can also apply other nonconvex penalties and approximations here, such as MCP (Zhang 2010) and coefficient thresholding (Liu, Zhang, Xue, Song, and Kang 2024) to achieve the same theoretical property and similar numerical performance.

With this approximation, we solve ${\widehat{\mathbf{\Theta }}}_{\text{1}}$ from the following folded concave penalized estimation of sparse SDR:

$$\begin{array}{lllll}\underset{\mathbf{\Theta }\in {ℝ}^{p\times p}}{\text{max}}& \text{tr}\left(\widehat{M}\mathbf{\Theta }\right)-\lambda P_{\delta }\left(\left|\mathbf{\Theta }\right|\right)& \text{s.t.}& \text{tr}\left(\mathbf{\Theta }\right)=1\text{;}& \mathbf{\Theta }\underset{\_}{\succ }0\text{,}\end{array}$$ (3)

and then for $k=1\text{,}2\text{,}\dots \text{,}d-1$, sequentially solve ${\widehat{\mathbf{\Theta }}}_{k+1}$, given ${\widehat{\mathit{\text{M}}}}_{k+1}$, from

$$\begin{array}{lllll}\underset{\mathbf{\Theta }\in {ℝ}^{p\times p}}{\text{max}}& \text{tr}\left({\widehat{M}}_{k+1}\mathbf{\Theta }\right)-\lambda P_{\delta }\left(\left|\mathbf{\Theta }\right|\right)& \text{s.t.}& \text{tr}\left(\mathbf{\Theta }\right)=1\text{;}& \mathbf{\Theta }\underset{\_}{\succ }0.\end{array}$$

Without loss of generality, this idea of estimating leading sparse eigenvalues can also be applied to the sparse Principal Component Analysis (PCA) problem.

Theorem 4 will provide the computational guarantee for the above approximation to ${\ell }_0$ penalisation. Section 6 will also present the numerical performance.

3. Methodology

The methodology and theory of the sparse SDR depend on the linearity condition and constant variance condition, which usually hold when predictors follow elliptical distributions, but elliptical conditions can be violated in many cases. In this section, following the framework built by Fukumizu et al. (2009) and Li, Chun, and Zhao (2012), we extend the framework to achieve sparse KSDR, which does not require linearity and constant variance conditions. The key concept is that we construct a conditional covariance operator in reproducing kernel Hilbert space (RKHS), which evaluates the amount of information $y$ that cannot be explained by the function of linear combinations of $\mathit{x}$. Then, we derive the empirical estimator of such conditional covariance operator with finite samples and finally minimise the trace of such estimator with sparsity constraints.

First, we provide the necessary definitions in RKHS to construct the conditional covariance operator. Let $k_{\mathit{x}}:{\mathbf{\text{Ω}}}_{\mathit{x}}\times {\mathbf{\text{Ω}}}_{\mathit{x}}\to ℝ$ and $k_y:{\mathbf{\text{Ω}}}_y\times {\mathbf{\text{Ω}}}_y\to ℝ$ be positive definite kernel functions and ${\mathscr{H}}_{\mathit{x}}$ and ${\mathscr{H}}_y$ be the RKHS generated by $k_{\mathit{x}}$ and $k_y$, such that ${\mathscr{H}}_{\mathit{x}}$ satisfies reproducing property:

$$\forall \mathit{x}\in {\mathbf{\text{Ω}}}_{\mathit{x}}\text{,}k\left(\cdot \text{,}\mathit{x}\right)\in {\mathscr{H}}_{\mathit{x}}\text{;}$$

$$\forall \mathit{x}\in {\mathbf{\text{Ω}}}_{\mathit{x}}\text{,}\forall f\in {\mathscr{H}}_{\mathit{x}}\text{,}〈f\text{,}k\left(\cdot \text{,}\mathit{x}\right)〉{\mathscr{H}}_{\mathit{x}}=f\left(\mathit{x}\right)\text{.}$$

And ${\mathscr{H}}_y$ satisfies such reproducing property accordingly in space ${\mathscr{H}}_y$. Then define ${\mathbf{\Sigma }}_{xx}:{\mathscr{H}}_{\mathit{x}}\to {\mathscr{H}}_{\mathit{x}}$ be the variance operator of $\mathit{x}$, which satisfies the relation $〈f,{\mathbf{\Sigma }}_{xx}g〉{\mathscr{H}}_{\mathit{x}}=\text{cov}\left(f\left(x\right)\text{,}g\left(x\right)\right)$ for all $f$, $g\in {\mathscr{H}}_{\mathit{x}}$, and so as ${\mathbf{\Sigma }}_{yy}$. Define ${\mathbf{\Sigma }}_{yx}:{\mathscr{H}}_{\mathit{x}}\to {\mathscr{H}}_y$ as the covariance operator satisfying $〈g,{\mathbf{\Sigma }}_{yx}f〉{\mathscr{H}}_y=\text{cov}\left(g\left(y\right)\text{,}f\left(x\right)\right)$ for all $g\in {\mathscr{H}}_y$ and $f\in {\mathscr{H}}_{\mathit{x}}$, and ${\mathbf{\Sigma }}_{xy}={\mathbf{\Sigma }}_{yx}^{*}$ be the adjoint operator of ${\mathbf{\Sigma }}_{yx}$. Based on Baker (1973), we know there exists unique bounded operators ${\mathit{V}}_{y\mathit{x}}$ and ${\mathit{V}}_{\mathit{x}y}$ satisfy:

$${\mathbf{\Sigma }}_{y\mathit{x}}={\mathbf{\Sigma }}_{yy}^{1/2}{\mathit{V}}_{y\mathit{x}}{\mathbf{\Sigma }}_{\text{xx}}^{1/2};{\mathbf{\Sigma }}_{\mathit{x}y}={\mathbf{\Sigma }}_{\text{xx}}^{1/2}{\mathit{V}}_{\mathit{x}y}{\mathbf{\Sigma }}_{yy}^{1/2}.$$

Then, the conditional covariance operator can be defined as:

$${\mathbf{\Sigma }}_{yy\mid \mathit{x}}={\mathbf{\Sigma }}_{yy}-{\mathbf{\Sigma }}_{yy}^{1/2}{\mathit{V}}_{y\mathit{x}}{\mathit{V}}_{\mathit{x}y}{\mathbf{\Sigma }}_{yy}^{1/2}.$$

For convenience, we sometime write it as: ${\mathbf{\Sigma }}_{yy|\mathit{x}}={\mathbf{\Sigma }}_{yy}-{\mathbf{\Sigma }}_{y\mathit{x}}{\mathbf{\Sigma }}_{\text{xx}}^{†}{\mathbf{\Sigma }}_{\mathit{x}y}$. where ${\mathbf{\Sigma }}_{\text{xx}}^{†}$ is the pseudo inverse of ${\mathbf{\Sigma }}_{\text{xx}}$. Specifically, ${\mathbf{\Sigma }}_{\text{xx}}^{†}={\left({\mathbf{\Sigma }}_{\text{xx}}+{ϵ}_{\mathit{s}}\mathit{\text{I}}\right)}^{-1}$, as the Tychonoff regularised inverse matrix of ${\mathbf{\Sigma }}_{\text{xx}}^{†}$, given some sufficient small constant ${ϵ}_{\mathit{s}}$.

Also, define the sparse orthogonal space ${\mathbb{S}}_d^p\left(ℝ\right)=\left\{\mathbf{\Theta }\in M(p\times d;ℝ))|{\mathbf{\Theta }}^T\mathbf{\Theta }={\mathbf{\text{I}}}_d;{‖\mathbf{\Theta }{.}_i‖}_0\le k\mathrm{for}\mathrm{i}=1,\dots ,\mathrm{d}\right\}$, representing the leading $d$ sparse SDR directions. When $d=1$, it represents the leading sparse SDR vector.

In what follows, we study the sparse estimation of $\mathbf{\Theta }\in {\mathbb{S}}_d^p\left(ℝ\right)$ such that $y\mathbf{\perp }\mathit{x}|{\mathbf{\Theta }}^T\mathit{x}$. To simplify notation, we introduce $\mathit{S}\text{=}{\mathbf{\Theta }}^T\mathit{x}$, and $k_{\mathit{\text{s}}}$, ${\mathbf{\Sigma }}_{\mathit{ss}}$, ${\mathbf{\Sigma }}_{y\mathit{\text{s}}}$ and ${\mathbf{\Sigma }}_{\text{s}y}$ are defined in the same manner as $k_{\mathit{x}}$, ${\mathbf{\Sigma }}_{\text{xx}}$, ${\mathbf{\Sigma }}_{y\mathit{x}}$ and ${\mathbf{\Sigma }}_{\mathit{x}y}$. Then we have:

$${\mathbf{\Sigma }}_{yy\mid \mathit{\text{s}}}={\mathbf{\Sigma }}_{yy}-{\mathbf{\Sigma }}_{y\mathit{\text{s}}}{\mathbf{\Sigma }}_{\mathit{\text{ss}}}^{†}{\mathbf{\Sigma }}_{\mathit{\text{s}}y}$$

As shown in Theorem4 of Fukumizu et al. (2009), if ${\mathscr{H}}_{\mathit{x}}$ and ${\mathscr{H}}_y$ are rich enough RKHS, $y\mathbf{\perp }\mathit{x}|{\mathbf{\Theta }}^T\mathit{x}\iff {\mathbf{\Sigma }}_{yy\mid \mathit{\text{s}}}={\mathbf{\Sigma }}_{yy|\mathit{x}}$, and also for any $\mathbf{\Theta }\in {\mathbb{S}}_d^p\left(ℝ\right)$, we have ${\mathbf{\Sigma }}_{yy|{\mathbf{\Theta }}^T\mathit{x}}\ge {\mathbf{\Sigma }}_{yy|\mathit{x}}$. This conclusion holds for our sparse orthogonal space as well, under the sparsity assumption of SDR vectors.

This result indicates that such a conditional covariance operator is a good estimator for sufficient dimension reduction at the population level. However, the population-level operator is not available in practice. At the sample level, we need to derive the empirical version of such an operator in coordinate representation and minimise it. To start with, we have the coordinate representation of the kernel matrix as:

$${\text{Κ}}_{\mathit{x}}=\left(\begin{array}{lll}{k}_{\mathit{x}}\left({\mathit{x}}_1\text{,}{\mathit{x}}_1\right)& \cdots & k_{\mathit{x}}\left({\mathit{x}}_1\text{,}{\mathit{x}}_n\right)\\ ⋮& \ddots & ⋮\\ k_{\mathit{x}}\left({\mathit{x}}_n\text{,}{\mathit{x}}_1\right)& \cdots & k_{\mathit{x}}\left({\mathit{x}}_n\text{,}{\mathit{x}}_n\right)\end{array}\right)\text{.}$$

Let ${\mathit{G}}_{\mathit{\text{x}}}=\mathit{Q}{\mathit{K}}_{\mathit{\text{x}}}\mathit{Q}$ be the centralised gram matrix, where $\mathit{Q}={\text{I}}_n-{\text{1}}_n{\text{1}}_n^T/n$. Similarly, we define ${\mathit{K}}_y$, ${\mathit{K}}_{\text{s}}$, ${\mathit{G}}_y$, ${\mathit{G}}_{\text{s}}$. Following Lemma1 of Li et al. (2012), under the spanning systems:

$${\widehat{\mathscr{H}}}_X=span\left\{k_x\left(\cdot \text{,}{\mathit{\text{x}}}_i\right)-{\mathbb{E}}_n{k}_x\left(\cdot \text{,}\mathit{\text{x}}\right):i\text{=}1\text{,}\dots \text{,}n\right\}\text{,}$$

$${\widehat{\mathscr{H}}}_Y=span\left\{k_y\left(\cdot \text{,}{y}_i\right)-{\mathbb{E}}_n{k}_y\left(\cdot \text{,}y\right):i\text{=}1\text{,}\dots \text{,}n\right\}\text{.}$$

The variance and covariance operators can be written as:

$$\begin{array}{llll}{\widehat{\mathbf{\Sigma }}}_{\text{xx}}={\mathit{G}}_{\mathit{\text{x}}}\text{,}& {\widehat{\mathbf{\Sigma }}}_{y\mathit{\text{x}}}={\mathit{G}}_{\mathit{\text{x}}}\text{,}& {\widehat{\mathbf{\Sigma }}}_{\mathit{\text{x}}y}={\mathit{G}}_y\text{,}& {\widehat{\mathbf{\Sigma }}}_{yy}={\mathit{G}}_y\end{array}$$

$$\begin{array}{lll}{\widehat{\mathbf{\Sigma }}}_{\mathit{ss}}={\mathit{G}}_{\mathit{s}}\text{,}& {\widehat{\mathbf{\Sigma }}}_{\mathit{s}y}={\mathit{G}}_y\text{,}& {\widehat{\mathbf{\Sigma }}}_{y\mathit{s}}={\mathit{G}}_{\mathit{s}}\text{.}\end{array}$$

Placing these coordinate representations into ${\mathbf{\Sigma }}_{yy|\mathit{s}}$, we derive the coordinate representation of ${\widehat{\mathbf{\Sigma }}}_{yy|\mathit{s}}$ as:

$${\widehat{\mathbf{\Sigma }}}_{yy|\mathit{s}}={\mathit{G}}_y-{\mathit{G}}_{\mathit{s}}{\mathit{G}}_{\mathit{s}}^{‡}{\mathit{G}}_y$$

where ${\mathit{G}}_{\mathit{s}}^{‡}={\left({\mathit{G}}_{\mathit{s}}+{ϵ}_{\mathit{s}}\text{I}\right)}^{-1}$ is the Tychonoff regularised inverse matrix of ${\mathit{G}}_{\mathit{s}}$ given some sufficient small constant ${ϵ}_{\mathit{s}}$. Then we minimise the trace of $\text{tr}\left({\widehat{\mathbf{\Sigma }}}_{yy|{\mathbf{\Theta }}^T\mathit{\text{x}}}\right)$ to estimate sparse SDR directions from $\mathbf{\Theta }\in {\mathbb{S}}_d^p\left(ℝ\right)$.

To deal with the orthogonality constraint in ${\mathbb{S}}_d^p\left(ℝ\right)$, we use a similar sequential procedure as in Section 2. To this end, we solve ${\widehat{\mathit{\theta }}}_{\text{1}}$ from the following trace minimisation to minimise the sum of conditional variance:

$$\begin{array}{ll}\underset{\mathit{\theta }\in \mathbb{S}{\mathbb{P}}_1^p\left(ℝ\right)}{\text{min}}& \text{tr}\left({\widehat{\mathbf{\Sigma }}}_{yy|{\mathit{\theta }}^T\mathit{\text{x}}}\right)\end{array}$$

where $\mathbb{S}{\mathbb{P}}_1^p\left(ℝ\right)=\left\{\text{θ|θ}\in {ℝ}^p\text{,}{‖\mathit{\theta }‖}_2=1\text{;}{‖\mathit{\theta }‖}_0\le t\right\}$. It is worth pointing out the equivalence between the above trace minimisation problem and the following regularised nonparametric regression problem:

$$\underset{\mathbf{\Theta }\in {\mathbb{S}}_1^p\left(ℝ\right)}{\text{min}}\underset{f\in {\mathscr{H}}_{𝒳}^{𝒮}}{\text{min}}\frac{1}{n}\sum _{i=1}^n\sum _{a=1}^{\infty }E|{\xi }_a\left(Y_i\right)-\frac{1}{n}\sum _{j=1}^n\left({\xi }_a\left(Y_j\right)\right)-\left(f\left(X_i\right)-\frac{1}{n}\sum _{j=1}^n\left[f\left(X_j\right)\right]\right){|}^2\text{}}+\frac{{ϵ}_n}{n}{‖f‖}^2{\mathscr{H}}_{𝒳}^{𝒮}$$

This equivalence provides insight into sparse KSDR, and it is also useful to establish the theoretical properties in Section 5.

In sparse KSDR, we assume $y$ is the function of ${\mathit{\theta }}_1^{★T}\mathit{x}\text{,}\dots \text{,}{\mathit{\theta }}_d^{★T}\mathit{\text{x}}$. In this case, ${\mathit{\theta }}_1^{★}\text{,}\dots \text{,}{\mathit{\theta }}_d^{★}$ are the true sparse SDR vectors of our interest. Previously, we have introduced how to estimate ${\mathit{\theta }}_1^{★}$. Here, we introduce how to solve these non-leading SDR directions sequentially.

Given that we have already estimated orthogonal sufficient directions ${\widehat{\mathit{\theta }}}_{\text{1}}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k$ for $k\ge 1$, we sequentially solve the following KSDR direction ${\mathit{\theta }}^{k+1}$ from the minimisation of $\text{tr}\left({\mathbf{\Sigma }}_{yy|{\widehat{\mathit{\theta }}}_1^T\mathit{x}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k^T\mathit{x}\text{,}{\mathit{\theta }}^T\mathit{\text{x}}}\right)$. By orthogonality of all sufficient directions, $\text{tr}\left({\mathbf{\Sigma }}_{yy|{\widehat{\mathit{\theta }}}_1^T\mathit{x}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k^T\mathit{x}\text{,}{\mathit{\theta }}^T\mathit{x}}\right)=\text{tr}\left({\mathbf{\Sigma }}_{yy|\left({\widehat{\mathit{\theta }}}_{\text{1}}{\widehat{\mathit{\theta }}}_1^T\text{+}\dots +{\widehat{\mathit{\theta }}}_k{\widehat{\mathit{\theta }}}_k^T\text{,}\mathit{\theta }{\mathit{\theta }}^T\right)\mathit{x}}\right)$. Hence, we only need to solve

$${\widehat{\mathit{\theta }}}_{k+1}^{init}=\underset{\mathit{\theta }\in \mathbb{S}{\mathbb{P}}_1^p\left(ℝ\right)}{\text{min}}\text{tr}\left({\mathbf{\Sigma }}_{yy|\left({\widehat{\mathit{\theta }}}_{\text{1}}{\widehat{\mathit{\theta }}}_1^T\text{+}\dots +{\widehat{\mathit{\theta }}}_k{\widehat{\mathit{\theta }}}_k^T\text{,}\mathit{\theta }{\mathit{\theta }}^T\right)\mathit{x}}\right)\text{,}$$

Then we project ${\widehat{\mathit{\theta }}}_{k+1}^{init}$ to the space orthogonal to all previous KSDR vectors, denoted as the final estimator ${\widehat{\mathit{\theta }}}_{k+1}$.

Here, we justify why this projection procedure is correct. Define ${\mathit{\theta }}_p={𝒫}_{{\widehat{\mathit{\theta }}}_{\text{1}}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k}^{\mathbf{\perp }}\mathit{\theta }$ as the projection of $\mathit{\theta }$ on the direction orthogonal to all previous $k$ KSDR vectors, such that $\text{span}\left({\widehat{\mathit{\theta }}}_{\text{1}}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k\text{,}\mathit{\theta }\right)=\text{span}\left({\widehat{\mathit{\theta }}}_{\text{1}}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k\text{,}{\mathit{\theta }}_p\right)$. By definition of the conditional covariance operator, we have:

$$\text{tr}\left({\mathbf{\Sigma }}_{yy|{\widehat{\mathit{\theta }}}_1^T\mathit{x}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k^T\mathit{x}\text{,}{\mathit{\theta }}^T\mathit{x}}\right)=\text{tr}\left({\mathbf{\Sigma }}_{yy|{\widehat{\mathit{\theta }}}_1^T\mathit{x}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k^T\mathit{x}\text{,}{\mathit{\theta }}_p^T\mathit{x}}\right)\text{.}$$

Since the non-orthogonal direction to previous KSDR directions plays no effect in reducing the trace, we only need to consider the projection ${\mathit{\theta }}_p={𝒫}_{{\widehat{\mathit{\theta }}}_{\text{1}}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k}^{\mathbf{\perp }}\mathit{\theta }$ for the sparse estimation of the following KSDR direction ${\mathit{\theta }}^{k+1}$. By the projection procedure, we guarantee that ${\widehat{\mathit{\theta }}}_{\text{1}}\mathbf{\perp }{\widehat{\mathit{\theta }}}_2\mathbf{\perp }\dots \mathbf{\perp }{\widehat{\mathit{\theta }}}_k$.

Finally, we discuss solving ${\ell }_0$ constraint and rank one constraint. Recall that $P_{\lambda }\left(\left|\mathbf{\Theta }\right|\right)$ is a folded concave penalty. In practice, we solve the following folded concave relaxation of sparse KSDR to approximate the ${\ell }_0$ constraint inside $\mathbb{S}{\mathbb{P}}_1^p\left(ℝ\right)$, and solve ${\widehat{\mathbf{\Theta }}}_{\text{1}}$ from

$$\begin{array}{llllll}\underset{\mathbf{\Theta }\in {ℝ}^{p\times p}}{\text{min}}& \text{tr}\left({\text{∑}}_{yy|{\mathbf{\Theta }}^T\mathit{x}}\right)& \text{s.t.}& \text{tr}\left(\mathbf{\Theta }\right)=1\text{;}& P_{\delta }\left(\left|\mathbf{\Theta }\right|\right)\le t^{\prime }\text{;}& \mathbf{\Theta }\underset{\_}{\succ }0\text{,}\end{array}$$

which can be rewritten in terms of a folded concave penalised estimation problem:

$$\begin{array}{lllll}\underset{\mathbf{\Theta }\in {ℝ}^{p\times p}}{\text{min}}& \text{tr}\left({\text{∑}}_{yy|{\mathbf{\Theta }}^T\mathit{x}}\right)+\lambda P_{\delta }\left(\left|\mathbf{\Theta }\right|\right)& \mathit{s}\text{.t.}& \text{tr}\left(\mathbf{\Theta }\right)=1\text{;}& \mathbf{\Theta }\underset{\_}{\succ }0.\end{array}$$

Given ${\widehat{\mathbf{\Theta }}}_{\text{1}}$, we follow Section 2 to truncate it to obtain the first KSDR direction ${\widehat{\mathit{\theta }}}_{\text{1}}$. Next, in the sequential fashion, given ${\widehat{\mathit{\theta }}}_{\text{1}}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k$ for $k\ge 1$, we solve ${\widehat{\mathbf{\Theta }}}_{k+1}$ from

$$\begin{array}{lllll}\underset{\mathbf{\Theta }\in {ℝ}^{p\times p}}{\text{min}}& \text{tr}\left({\mathbf{\Sigma }}_{yy|\left({\widehat{\mathit{\theta }}}_{\text{1}}{\widehat{\mathit{\theta }}}_1^T\text{+}\dots +{\widehat{\mathit{\theta }}}_k{\widehat{\mathit{\theta }}}_k^T\text{,}{\mathbf{\Theta }}^T\right)\mathit{x}}\right)+P_{\lambda }\left(\left|\mathbf{\Theta }\right|\right)& \mathit{s}\text{.t.}& \text{tr}\left(\mathbf{\Theta }\right)=1\text{;}& \mathbf{\Theta }\underset{\_}{\succ }0\text{,}\end{array}$$

where we used the fact that $\text{tr}\left({\mathbf{\Sigma }}_{yy|{\widehat{\mathit{\theta }}}_1^T\mathit{x}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k^T\mathit{x}\text{,}{\mathit{\theta }}^T\mathit{x}}\right)=\text{tr}\left({\mathbf{\Sigma }}_{yy|{\widehat{\mathit{\theta }}}_1^T\mathit{x}\text{,}\dots \text{,}{\widehat{\mathit{\theta }}}_k^T\mathit{x}\text{,}{\mathit{\theta }}_p^T\mathit{x}}\right)$. Now, we compute the projection ${\mathbf{\Theta }}_{k+1}^{proj}\text{=}{\mathbb{P}}_{{\mathit{\theta }}_{\text{1}}\oplus \cdots \oplus {\mathit{\theta }}_k}^{\mathbf{\perp }}{\widehat{\mathbf{\Theta }}}_{k+1}$ and solve the dominant (sparse) eigenvector as the desired sparse KSDR direction ${\widehat{\mathit{\theta }}}_{k+1}$.

4. Nonconvex optimisation

Section 4 first proposes an efficient nonconvex ADMM to solve the nonconvex optimisation of sparse SDR and then presents an efficient nonconvex linearised ADMM to solve the even more challenging nonconvex optimisation of sparse KSDR. Both nonconvex ADMMs enjoy the important computational guarantees, which will be presented in Section 5.

4.1. Nonconvex ADMM for sparse SDR

In this section, we propose a novel nonconvex ADMM to solve the challenging nonconvex and nonsmooth optimisation in the sparse SDR. The proposed nonconvex ADMM enjoys the convergence guarantees to the ‘$ϵ$-stationary solution’ (see the definition in Theorem 5) with an explicit iteration complexity bound. The convergence guarantee of nonconvex algorithms can be defined as varying from algorithms. The $ϵ$-stationary solution introduced in Jiang, Lin, Ma, and Zhang (2019) is an applicable one to depict the convergence in the aspect of the Lagrangian function. More details about the $ϵ$-stationary solution and computational guarantees can be found in Theorem 5 of Section 5.1.

To begin with, we introduce a new variable $\mathbf{\text{Ω}}$ and also a slacking variable $\mathbf{\text{Φ}}$. By adding the equality constraint $\mathbf{\Theta }\text{−}\mathbf{\text{Ω}}-\mathbf{\text{Φ}}=0$, (3) is equivalent to:

$$\begin{array}{llllll}\underset{\mathbf{\Theta }\in {ℝ}^{p\times p}}{\text{min}}& -\text{tr}\left(\widehat{M}\mathbf{\Theta }\right)+P_{\lambda }\left(\left|\mathbf{\Theta }\right|\right)+\frac{\mu }{2}{‖\mathbf{\text{Φ}}‖}_F^2& \mathit{s}\text{.t.}& \text{tr}\left(\text{Ω}\right)=1\text{;}& \mathbf{\Theta }\underset{\_}{\succ }0\text{,}& \mathbf{\Theta }\text{−}\mathbf{\text{Ω}}-\mathbf{\Phi }=0.\end{array}$$

The slacking variable $\mathbf{\Phi }$ plays an essential role in the convergence guarantee of our proposed nonconvex ADMM. Accordingly, the augmented Lagrange function is

$$\begin{array}{l}\begin{array}{l}{ℒ}_p\left(\mathbf{\Theta }\text{,}\mathbf{\text{Ω}},\mathbf{\Phi }\text{,}\mathbf{\text{Λ}}\right)=-\text{tr}\left(\widehat{M}\mathbf{\Theta }\right)+P_{\lambda }\left(\left|\mathbf{\Theta }\right|\right)+\frac{\mu }{2}{‖\mathbf{\Phi }‖}_F^2+{ℐ}_{\left\{\text{tr}\left(\text{Ω}\right)=1\text{;}\mathbf{\Theta }\underset{\_}{\succ }0\right\}}\\ +〈\mathbf{\Lambda }\text{,}\mathbf{\Theta }\text{−}\mathbf{\Omega }-\mathbf{\Phi }〉+\frac{\rho }{2}{‖\mathbf{\Theta }\text{−}\mathbf{\Omega }-\mathbf{\Phi }‖}_F^2\text{.}\end{array}\end{array}$$

where $\mathbf{\text{Λ}}$ is the Lagrange variable and $\rho \text{>}0$. Given the initial value (${\mathbf{\text{Ω}}}^0$, ${\mathbf{\Phi }}^0$, ${\mathbf{\text{Λ}}}^0$), we solve (${\mathbf{\Theta }}^{q+1}$, ${\mathbf{\text{Ω}}}^{q+1}$, ${\mathbf{\Phi }}^{q+1}$, ${\mathbf{\text{Λ}}}^{q+1}$) iteratively for $q=0\text{,}1\text{,}2\text{,}\text{…}$ as follows:

$${\mathbf{\Theta }}^{q+1}=\underset{\mathbf{\Theta }}{\text{arg}\text{min}}{ℒ}_p\left(\mathbf{\Theta }\text{,}{\mathbf{\text{Ω}}}^q\text{,}{\mathbf{\Phi }}^q\text{,}{\mathbf{\text{Λ}}}^q\right)+\frac{h}{2}{‖{\mathbf{\Theta }}^q-\mathbf{\Theta }‖}_F^2$$

$${\mathbf{\text{Ω}}}^{q+1}=\underset{\mathbf{\text{Ω}}}{\text{arg}\text{min}}{ℒ}_p\left({\mathbf{\Theta }}^{q+1}\text{,}\mathbf{\Omega }\text{,}{\mathbf{\Phi }}^q\text{,}{\mathbf{\text{Λ}}}^q\right)$$

$${\mathbf{\Phi }}^{q+1}={\mathbf{\Phi }}^q-\gamma \frac{\partial }{\partial {\mathbf{\Phi }}^q}{ℒ}_p\left({\mathbf{\Theta }}^{q+1}\text{,}{\mathbf{\text{Ω}}}^{q+1}\text{,}\mathbf{\Phi }\text{,}{\mathbf{\text{Λ}}}^q\right)$$

$${\mathbf{\text{Λ}}}^{q+1}={\mathbf{\text{Λ}}}^q-\rho \left({\mathbf{\Theta }}^{q+1}-{\mathbf{\text{Ω}}}^{q+1}\right)\text{,}$$

where $h>0$ is chosen to guarantee the convexity of $\mathbf{\Theta }$ step and $\gamma$ & $\rho$ are chosen to guarantee the convergence. Their choices will be discussed in Theorem 5. Note that the $\mathbf{\Phi }$ step and the $\mathbf{\text{Λ}}$ step are straightforward. Fortunately, we also have the closed-form solutions in both $\mathbf{\Theta }$ step and $\mathbf{\text{Ω}}$ step. In the $\mathbf{\Theta }$ step, we know that

$$\begin{array}{l}\underset{\mathbf{\Theta }}{\text{min}}{ℒ}_p\left(\mathbf{\Theta }\text{,}{\mathbf{\text{Ω}}}^q\text{,}{\mathbf{\Phi }}^q\text{,}{\mathbf{\text{Λ}}}^q\right)+\frac{h}{2}{‖{\mathbf{\Theta }}^q-\mathbf{\Theta }‖}_F^2\\ =\underset{\mathbf{\Theta }}{\text{min}}\frac{\rho +h}{2}{‖\mathbf{\Theta }-\frac{1}{\rho +h}\left(\rho {\mathbf{\text{Ω}}}^q+\rho {\mathbf{\Phi }}^q+h{\mathbf{\Theta }}^q-\text{,}{\mathbf{\text{Λ}}}^q-\widehat{\mathit{\text{M}}}\right)‖}_F^2+P_{\lambda }\left(\left|\mathbf{\Theta }\right|\right)\text{,}\end{array}$$

which can be analytically solved by using the univariate folded concave thresholding rule $S_{\lambda }\left(b\text{,}c\right)=\text{arg}{\text{min}}_u\frac{1}{2}{‖u-b‖}^2+c{P}_{\lambda }\left(\left|u\right|\right)$. More specifically, we have

$${\mathbf{\Theta }}^{q+1}=S_{\lambda }\left(\frac{1}{\rho +h}\left(\rho {\mathbf{\text{Ω}}}^q+\rho {\mathbf{\Phi }}^q+h{\mathbf{\Theta }}^q-\text{,}{\mathbf{\text{Λ}}}^q-\widehat{\mathit{\text{M}}}\right)\text{,}\frac{\rho +h}{2}\right)\text{.}$$

In the $\mathbf{\text{Ω}}$ step, it is easy to see that

$$\underset{\mathbf{\text{Ω}}}{\text{min}}{ℒ}_p\left({\mathbf{\Theta }}^{q+1}\text{,}\mathbf{\Omega }\text{,}{\mathbf{\Phi }}^q\text{,}{\mathbf{\text{Λ}}}^q\right)=\underset{\mathbf{\Omega }\underset{\_}{\succ }0\text{;tr}\left(\mathbf{\Omega }\right)=1}{\text{min}}{‖\mathbf{\Omega }+{\mathbf{\Phi }}^q-{\mathbf{\Theta }}^{q+1}-\frac{1}{\rho }{\mathbf{\text{Λ}}}^q‖}_F^2\text{,}$$

which can be solved by using the projection onto the simplex in the Euclidean space (See Algorithm 1 of Ma 2013). To solve ${\mathbf{\Omega }}^{q+1}$, we take the singular value decomposition of ${\mathit{W}}^q={\mathbf{\Theta }}^{q+1}-{\mathbf{\Phi }}^q+\frac{1}{\rho }{\mathbf{\text{Λ}}}^q$ as ${\mathit{W}}^q=\mathit{U}\text{diag}\left({\sigma }_1\text{,}\dots \text{,}{\sigma }_p\right){\mathit{U}}^T$. After analytically solving the projection of $\mathbf{\text{σ}}$ onto the simplex in the Euclidean space. Denoting $\widehat{\mathit{\xi }}=\left({\widehat{\xi }}_1\text{,}\text{…}{\widehat{\xi }}_p\right)$, it has closed-form solution:

$$\begin{array}{llll}\widehat{\mathit{\xi }}=\underset{\mathit{\xi }}{\text{arg}\text{min}}\frac{1}{2}{‖\mathit{\xi }-\mathbf{\text{σ}}‖}_2^2& \mathit{s}\text{.t.}& \sum _{j=1}^p{\xi }_j=t\text{,}& {\xi }_j=0.\end{array}$$

Then we obtain the closed-form solution ${\mathbf{\Omega }}^{q+1}=\mathit{U}\text{diag}\left({\widehat{\xi }}_1\text{,}\text{…}\text{,}{\widehat{\xi }}_p\right){\mathit{U}}^T$. In the $\mathbf{\Phi }$ step, we solve the derivative as:

$$\frac{\partial }{\partial {\mathbf{\Phi }}^q}{ℒ}_p\left({\mathbf{\Theta }}^{q+1}\text{,}{\mathbf{\Omega }}^{q+1}\text{,}\mathbf{\Phi }\text{,}{\mathbf{\text{Λ}}}^q\right)=\mu \mathbf{\Phi }\text{−}\mathbf{\Lambda }+\rho \left(\mathbf{\Phi }+\mathbf{\Omega }\text{−}\mathbf{\Theta }\right)\text{.}$$

Now, we summarise the proposed nonconvex ADMM in Algorithm 1.

4.2. Nonconvex linearised ADMM for sparse KSDR

In this section, we provide a detailed derivation of the nonconvex linearised ADMM algorithm for sparse KSDR. We present the algorithm for solving the leading KSDR vector,

$$\overline{\underset{¯}{\begin{array}{l}\underset{¯}{\mathbf{Algorithm}\mathbf{1}\text{Proposed}\text{nonconvex}\text{ADMM}\text{for}\text{solving}\text{the}\text{sparse}\text{SDR}}\\ \mathbf{Require}\mathbf{:}\text{Given}\text{(}{\mathbf{\Omega }}^0\text{,}{\mathbf{\Phi }}^{\text{0}}\text{,}{\mathbf{\Lambda }}^0\text{)}\in {ℝ}^{p\times p}\times {ℝ}^{p\times p}\times {ℝ}^{p\times p}\text{,}\\ \begin{array}{l}\mathbf{for}\text{q=0,}\text{1,}\dots \mathbf{do}\hfill \\ {\mathbf{\Theta }}^{q+1}={𝒮}_{\lambda }\left(\frac{1}{\rho +h}\text{(}\rho \left({\mathbf{\Omega }}^q+{\mathbf{\Phi }}^q\right)+h{\mathbf{\Theta }}^q-{\mathbf{\Lambda }}^q-\widehat{M}\text{),}\frac{\rho +h}{2}\right)\hfill \\ {\mathbf{\Omega }}^{q+1}=\mathit{U}\text{diag}\left({\widehat{\xi }}_1\text{,}\text{…}\text{,}{\widehat{\xi }}_p\right){\mathit{U}}^T\text{,}\text{where}{\mathbf{\Theta }}^{q+1}-{\mathbf{\Phi }}^q+\frac{1}{\rho }{\mathbf{\Lambda }}^q=\mathit{U}\text{diag}\left({\sigma }_1\text{,}\text{…}\text{,}{\sigma }_p\right){\mathit{U}}^T\hfill \\ {\mathbf{\Phi }}^{q+1}={\mathbf{\Phi }}^q-\gamma \left(\mu +\rho \right)\mathbf{\Phi }-\mathbf{\Lambda }+\rho \left(\mathbf{\Omega }-\mathbf{\Phi }\right)\hfill \\ {\mathbf{\Lambda }}^{q+1}={\mathbf{\Lambda }}^q-\rho \left({\mathbf{\Theta }}^{q+1}-{\mathbf{\Omega }}^{q+1}\right)\hfill \end{array}\end{array}}}$$

and the algorithm for the following vectors follows similarly. To begin with, we introduce new variables ${\mathbf{\Omega }}_{\text{1}}$ and ${\mathbf{\Omega }}_2$ and slacking variables ${\mathbf{\Phi }}_1$ and ${\mathbf{\Phi }}_2$. By adding two equality constraints that $\mathbf{\Theta }\text{−}{\mathbf{\Omega }}_{\text{1}}-{\mathbf{\Phi }}_{\text{1}}=0$, $\mathbf{\Theta }\text{−}{\mathbf{\Omega }}_2-{\mathbf{\Phi }}_{\text{2}}=0$, (2.7) is equivalent to:

$$\begin{array}{lllllll}\underset{\left(\mathbf{\Theta }\text{,}{\mathbf{\Omega }}_{\text{1}}\text{,}{\mathbf{\Omega }}_{\text{2}}\text{,}{\mathbf{\Phi }}_{\text{1}}\text{,}{\mathbf{\Phi }}_2\right)}{\text{min}}\hfill & \mathrm{tr}\left({\widehat{\mathbf{\Sigma }}}_{yy|{\mathbf{\Theta }}^T\mathit{x}}\right)+P_{\lambda }\left(\left|{\mathbf{\Omega }}_{\text{1}}\right|\right)+\frac{\mu }{2}\left({‖{\mathbf{\Phi }}_{\text{1}}‖}_F^2+{‖{\mathbf{\Phi }}_2‖}_F^2\right)\hfill & s.t\hfill & \mathbf{\Theta }-{\mathbf{\Omega }}_{\text{1}}-{\mathbf{\Phi }}_{\text{1}}=0,\hfill & \mathbf{\Theta }-{\mathbf{\Omega }}_2-{\mathbf{\Phi }}_2=0.\hfill & \mathrm{tr}\left({\mathbf{\Omega }}_2\right)=1,\hfill & {\mathbf{\Omega }}_2\underset{\_}{\succ }0.\hfill \end{array}$$

The slacking variables ${\mathbf{\Phi }}_1$ and ${\mathbf{\Phi }}_2$ are essential for the convergence guarantee of our proposed algorithm. Accordingly, the augmented Lagrange function is

$$\begin{array}{l}{ℒ}_{\rho }\left(\mathbf{\Theta }\text{,}{\mathbf{\Omega }}_1,{\mathbf{\Omega }}_2,{\mathbf{\Lambda }}_{\text{1}}\text{,}{\mathbf{\Lambda }}_2\right)=\text{tr}\left({\widehat{\mathbf{\Sigma }}}_{yy|\mathbf{\Theta }x}\right)+{ℐ}_{\left\{\text{tr}\left({\mathbf{\text{Ω}}}_2\right)=1\text{;}{\mathbf{\text{Ω}}}_2\underset{\_}{\succ }0\right\}}+\frac{\mu }{2}\left({‖{\mathbf{\Phi }}_{\text{1}}‖}_F^2+{‖{\mathbf{\Phi }}_2‖}_F^2\right)\\ +P_{\lambda }\left(\left|{\text{Ω}}_1\right|\right)+〈{\mathbf{\Lambda }}_{\text{1}}\text{,}\mathbf{\Theta }-{\mathbf{\Omega }}_{\text{1}}-{\mathbf{\Phi }}_{\text{1}}〉+〈{\mathbf{\Lambda }}_{\text{2}}\text{,}\mathbf{\Theta }-{\mathbf{\Omega }}_{\text{2}}-{\mathbf{\Phi }}_2〉\\ +\frac{\rho }{2}{‖\mathbf{\Theta }-{\mathbf{\Omega }}_{\text{1}}-{\mathbf{\Phi }}_{\text{1}}‖}_F^2+\frac{\rho }{2}{‖\mathbf{\Theta }-{\mathbf{\Omega }}_{\text{2}}-{\mathbf{\Phi }}_2‖}_F^2,\end{array}$$

where ${\mathbf{\Lambda }}_{\text{1}}$ and ${\mathbf{\Lambda }}_2$ are the Lagrange variable and $\rho >0$.

It is important to point out that $\mathrm{tr}({\widehat{\mathbf{\Sigma }}}_{yy|{\mathbf{\Theta }}^{T}x})$ introduces the additional computational challenges since the $\mathbf{\Theta }$ step does not lead to a closed-form solution. To this end, we introduce a linearised counterpart of this trace function as

$$\mathrm{tr}({\widehat{\mathbf{\Sigma }}}_{yy|{\mathbf{\Theta }}^{(q)T}x})+{\left(\mathbf{\Theta }-{\mathbf{\Theta }}^q\right)}^T\frac{\partial }{\partial \mathbf{\Theta }}\mathrm{tr}({\widehat{\mathbf{\Sigma }}}_{yy|{\mathbf{\Theta }}^{(q)T}x}).$$

After plugging this linearised term, we denote the new Lagrange function as ${ℒ}_{\rho }^{*}\left(\mathbf{\Theta }\text{,}{\mathbf{\Theta }}^q\text{,}{\mathbf{\Omega }}_1,{\mathbf{\Omega }}_2,{\mathbf{\Phi }}_{\text{1}},{\mathbf{\Phi }}_2\text{,}{\mathbf{\Lambda }}_{\text{1}}\text{,}{\mathbf{\Lambda }}_2\right)$. Given the initial solution (${\mathbf{\Omega }}_1^0$, ${\mathbf{\Omega }}_2^0$, ${\mathbf{\Phi }}_1^0$, ${\mathbf{\Phi }}_2^0$, ${\mathbf{\Lambda }}_1^0$, ${\mathbf{\Lambda }}_2^0$), we solve (${\mathbf{\Theta }}^{q+1}$,${\mathbf{\Omega }}_1^{q+1}$, ${\mathbf{\Omega }}_2^{q+1}$, ${\mathbf{\Phi }}_1^{q+1}$, ${\mathbf{\Phi }}_2^{q+1}$, ${\mathbf{\Lambda }}_1^{q+1}$, ${\mathbf{\Lambda }}_2^{q+1}$) iteratively for $q=0,1,2,\dots$ as follows:

$${\mathbf{\Theta }}^{q+1}=\underset{\mathbf{\Theta }}{\arg \min }{ℒ}_{\rho }^{*}\left(\mathbf{\Theta }\text{,}{\mathbf{\Omega }}_1^q,{\mathbf{\Omega }}_2^q,{\mathbf{\Phi }}_1^q,{\mathbf{\Phi }}_2^q\text{,}{\mathbf{\Lambda }}_1^q\text{,}{\mathbf{\Lambda }}_2^q\right)+\frac{h}{2}{‖\mathbf{\Theta }-{\mathbf{\Theta }}^q‖}_F^2$$

$${\mathbf{\Omega }}_1^{q+1}=\underset{{\mathbf{\Omega }}_1}{\arg \min }{ℒ}_{\rho }\left({\mathbf{\Theta }}^{q+1}\text{,}{\mathbf{\Omega }}_{\text{1}},{\mathbf{\Omega }}_2^q,{\mathbf{\Phi }}_1^q,{\mathbf{\Phi }}_2^q\text{,}{\mathbf{\Lambda }}_1^q\text{,}{\mathbf{\Lambda }}_2^q\right)$$

$${\mathbf{\Omega }}_2^{q+1}=\underset{{\mathbf{\Omega }}_2}{\arg \min }{ℒ}_{\rho }\left({\mathbf{\Theta }}^{q+1}\text{,}{\mathbf{\Omega }}_1^{q+1},{\mathbf{\Omega }}_{\text{2}},{\mathbf{\Phi }}_1^q,{\mathbf{\Phi }}_2^q\text{,}{\mathbf{\Lambda }}_1^q\text{,}{\mathbf{\Lambda }}_2^q\right)$$

$${\mathbf{\Phi }}_1^{q+1}={\mathbf{\Phi }}_1^q-\gamma \frac{\partial }{\partial {\mathbf{\Phi }}_1^q}{ℒ}_{\rho }\left({\mathbf{\Theta }}^{q+1}\text{,}{\mathbf{\Omega }}_1^{q+1},{\mathbf{\Omega }}_2^{q+1},{\mathbf{\Phi }}_1^q,{\mathbf{\Phi }}_2^q\text{,}{\mathbf{\Lambda }}_1^q\text{,}{\mathbf{\Lambda }}_2^q\right)$$

$${\mathbf{\Phi }}_2^{q+1}={\mathbf{\Phi }}_2^q-\gamma \frac{\partial }{\partial {\mathbf{\Phi }}_2^q}{ℒ}_{\rho }\left({\mathbf{\Theta }}^{q+1}\text{,}{\mathbf{\Omega }}_1^{q+1},{\mathbf{\Omega }}_2^{q+1},{\mathbf{\Phi }}_1^q,{\mathbf{\Phi }}_2^q\text{,}{\mathbf{\Lambda }}_1^q\text{,}{\mathbf{\Lambda }}_2^q\right)$$

$${\mathbf{\Lambda }}_1^{q+1}={\mathbf{\Lambda }}_1^q-\rho \left({\mathbf{\Theta }}^{q+1}+{\mathbf{\Omega }}_1^{q+1}\right)$$

$${\mathbf{\Lambda }}_2^{q+1}={\mathbf{\Lambda }}_2^q-\rho \left({\mathbf{\Theta }}^{q+1}+{\mathbf{\Omega }}_2^{q+1}\right),$$

where $h>0$ is chosen to guarantee the convexity of $\mathbf{\Theta }$ step and $\gamma$, $\rho$ are chosen to guarantee the algorithmic convergence (see Theorem 10). In what follows, we present the closed-form solution for each subproblem.

For $\mathbf{\Theta }$ step, with the linearisation, ${\mathbf{\Theta }}^{q+1}$ can be solved by taking derivatives as:

$$\frac{\partial }{\partial \mathbf{\Theta }}{ℒ}_{\rho }^{*}\left(\mathbf{\Theta }\text{,}{\mathbf{\Theta }}^q,{\mathbf{\Omega }}_1^q,{\mathbf{\Omega }}_2^q,{\mathbf{\Phi }}_1^q,{\mathbf{\Phi }}_2^q\text{,}{\mathbf{\Lambda }}_1^q\text{,}{\mathbf{\Lambda }}_2^q\right)+h\left(\mathbf{\Theta }-{\mathbf{\Theta }}^q\right)=0.$$

Defining $g\left(\mathbf{\Theta }\right)={\mathit{G}}_s{\left({\mathit{G}}_s+{ϵ}_{n}I\right)}^{-1}$, then $\mathrm{tr}({\widehat{\mathbf{\Sigma }}}_{yy|\mathbf{\Theta }x})=tr\left(g\left(\mathbf{\Theta }\right){\mathit{G}}_y\right)$, then the derivatives can be solved by using the chain rule of matrix derivatives:

$$\frac{\partial }{\partial \mathbf{\Theta }}\mathrm{tr}({\widehat{\mathbf{\Sigma }}}_{yy|\mathbf{\Theta }x})=vec{\left(\frac{\partial }{\partial \mathbf{\Theta }}g\left(\mathbf{\Theta }\right)\right)}^{T}vec\left(\frac{\partial }{\partial g\mathbf{\Theta }}\mathrm{tr}({\widehat{\mathbf{\Sigma }}}_{yy|\mathbf{\Theta }x})\right),$$

which has a complex closed form. For space consideration, the detailed calculation of closed-form derivatives and its numerical approximation is deferred to the supplement. Based on $\frac{\partial }{\partial \mathbf{\Theta }}\mathrm{tr}({\mathbf{\Sigma }}_{yy|\mathbf{\Theta }x})$, we can obtain the close-form derivative. By solving the linearisation, we obtain the minimiser:

$$\begin{array}{l}{q}_1\left(\mathbf{\Theta }\text{,}{\mathbf{\Omega }}_1,{\mathbf{\Omega }}_2,{\mathbf{\Phi }}_{\text{1}},{\mathbf{\Phi }}_2\text{,}{\mathbf{\Lambda }}_{\text{1}}\text{,}{\mathbf{\Lambda }}_2\right)\hfill \\ =\frac{\rho \left({\mathbf{\Omega }}_1+{\mathbf{\Phi }}_{\text{1}}+{\mathbf{\Omega }}_2+{\mathbf{\Phi }}_2\right)-{\mathbf{\Lambda }}_{\text{1}}-{\mathbf{\Lambda }}_2-\frac{\partial }{\partial g\left(\mathbf{\Theta }\right)}\mathrm{tr}({\widehat{\mathbf{\Sigma }}}_{yy|\mathbf{\Theta }x})+h{\mathbf{\Theta }}^q}{2\rho +h}.\hfill \end{array}$$

For ${\mathbf{\Omega }}_{\text{1}}$ step, it is equivalent to solve the subproblem

$$\begin{array}{l}\underset{{\mathbf{\Omega }}_1}{\text{min}}{P}_{\lambda }\left(\mathbf{\Theta }\right)+〈{\mathbf{\Lambda }}_1\text{,}\mathbf{\Theta }\text{−}{\mathbf{\Omega }}_{\text{1}}-{\mathbf{\Phi }}_1〉+\frac{\rho }{2}{‖\mathbf{\Theta }\text{−}{\mathbf{\Omega }}_{\text{1}}-{\mathbf{\Phi }}_1‖}_F^2\\ =\underset{{\mathbf{\Omega }}_1}{\text{min}}{‖{\mathbf{\Omega }}_{\text{1}}-\left(\mathbf{\Theta }-{\mathbf{\Phi }}_1+\frac{{\mathbf{\Lambda }}_1}{\rho }\right)‖}_F^2+P_{\lambda }\left(\left|{\mathbf{\Omega }}_1\right|\right)\text{.}\end{array}$$

This can be efficiently solved by using the univariate penalised least squares solution:

$${\mathbf{\Omega }}_1^{q+1}=S_{\lambda }\left(\mathbf{\Theta }-{\mathbf{\Phi }}_1+\frac{{\mathbf{\Lambda }}_1}{\rho },2\right),$$

where $S_{\lambda }$ has been defined in Section 4.1. For ${\mathbf{\Omega }}_2$ step, we are solving the optimisation problem:

$${\mathbf{\Omega }}_2=\underset{\text{tr}\left({\mathbf{\Omega }}_2\right)=1}{\arg \min -}{\mathbf{\Lambda }}_2{\mathbf{\Omega }}_2+\frac{\rho }{2}{‖\mathbf{\Theta }-{\mathbf{\Omega }}_2-{\mathbf{\Phi }}_2‖}_F^2=\underset{\text{tr}\left({\mathbf{\Omega }}_2\right)=1}{\arg \min -}+\frac{\rho }{2}{‖{\mathbf{\Omega }}_2+{\mathbf{\Phi }}_2-\mathbf{\Theta }-{\mathbf{\Lambda }}_2/\rho ‖}_F^2.$$

It can be solved by projection onto the simplex in the Euclidean space. We have presented exactly the same algorithm in Section 4.1. Thus, we omit it here. For ${\mathbf{\Phi }}_1$ and ${\mathbf{\Phi }}_{\text{2}}$ steps, we can solve them straightforwardly:

$${\mathbf{\Phi }}_1^{q+1}={\mathbf{\Phi }}_1^q+\gamma \left(\mu {\mathbf{\Phi }}_1^q+{\mathbf{\Lambda }}_1+\rho \left({\mathbf{\Phi }}_1^q+{\mathbf{\Omega }}_1^{q+1}-{\mathbf{\Theta }}^{q+1}\right)\right)$$

$${\mathbf{\Phi }}_2^{q+1}={\mathbf{\Phi }}_2^q+\gamma \left(\mu {\mathbf{\Phi }}_2^q+{\mathbf{\Lambda }}_2+\rho \left({\mathbf{\Phi }}_2^q+{\mathbf{\Omega }}_2^{q+1}-{\mathbf{\Theta }}^{q+1}\right)\right).$$

Now we can summarise the proposed nonconvex linearised ADMM in Algorithm 2:

$$\overline{\underset{¯}{\begin{array}{l}\underset{¯}{\mathbf{Algorithm}\mathbf{2}\text{Nonconvex}\text{linearised}\text{ADMM}\text{for}\text{solving}\text{sparse}\text{KSDR}}\\ \mathbf{Require}\mathbf{:}\text{Given}\text{(}{\mathbf{\Omega }}_1^0\text{,}{\mathbf{\Omega }}_2^0\text{,}{\mathbf{\Phi }}_1^0\text{,}{\mathbf{\Phi }}_2^0\text{,}{\mathbf{\Lambda }}_1^0\text{,}{\mathbf{\Lambda }}_2^0\text{)}\text{all}\text{in}{ℝ}^{p\times p}\\ \begin{array}{l}\mathbf{for}\text{k=0,}\text{1,}\dots \mathbf{do}\hfill \\ \begin{array}{l}{\mathbf{\Theta }}^{q+1}=q_1\left(\mathbf{\Theta },{\mathbf{\Omega }}_1^q\text{,}{\mathbf{\Omega }}_2^q\text{,}{\mathbf{\Phi }}_1^q\text{,}{\mathbf{\Phi }}_2^q\text{,}{\mathbf{\Lambda }}_1^q\text{,}{\mathbf{\Lambda }}_2^q\right)\text{.}\\ {\mathbf{\Omega }}_1^{q+1}=S_{\lambda }\left(\mathbf{\Theta }-{\mathbf{\Phi }}_1+\frac{{\mathbf{\Lambda }}_1}{\rho }\text{,}2\right)\end{array}\hfill \\ {\mathbf{\Omega }}_2^{q+1}=\mathit{U}\text{diag}\left({\widehat{\xi }}_1\text{,}\text{…}\text{,}{\widehat{\xi }}_p\right){\mathit{U}}^T\text{,}\text{where}{\mathbf{\Theta }}^{q+1}-{\mathbf{\Phi }}_2^q+\frac{1}{\rho }{\mathbf{\Lambda }}_2^q=\mathit{U}\text{diag}\left({\sigma }_1\text{,}\text{…}\text{,}{\sigma }_p\right){\mathit{U}}^T\hfill \\ {\mathbf{\Phi }}_1^{q+1}={\mathbf{\Phi }}_1^q+\gamma \left({\mathbf{\Lambda }}_1{\mathbf{\Phi }}_1^q+\rho \left({\mathbf{\Phi }}_1^q+{\mathbf{\Omega }}_1^{q+1}-{\mathbf{\Theta }}^{q+1}\right)\right)\hfill \\ \begin{array}{l}{\mathbf{\Phi }}_2^{q+1}={\mathbf{\Phi }}_2^q+\gamma \left({\mathbf{\Lambda }}_2{\mathbf{\Phi }}_2^q+\rho \left({\mathbf{\Phi }}_2^q+{\mathbf{\Omega }}_2^{q+1}-{\mathbf{\Theta }}^{q+1}\right)\right)\\ {\mathbf{\Lambda }}_1^{q+1}={\mathbf{\Lambda }}_1^q-\rho \left({\mathbf{\Theta }}^{q+1}-{\mathbf{\Omega }}_1^{q+1}\right)\\ {\mathbf{\Lambda }}_2^{q+1}={\mathbf{\Lambda }}_2^q-\rho \left({\mathbf{\Theta }}^{q+1}-{\mathbf{\Omega }}_2^{q+1}\right)\end{array}\hfill \end{array}\end{array}}}$$

5. Theoretical properties

In this section, we establish a theoretical guarantee for both sparse SDR and sparse KSDR estimators. For both SDR and KSDR, first we study the statistical guarantee of ${\ell }_0$-constrained estimator, showing their consistency, allowing $p$ diverging as $n\to \infty$. Then, we study how our folded concave penalised estimator can approximate ${\ell }_0$-constrained estimator and prove the computation guarantee of our proposed nonconvex ADMM and its linearised version.

5.1. Asymptotic properties of sparse SDR

We study the statistical and computational properties of ${\ell }_0$-constrained SDR, such as the consistency and convergence rate in this subsection and the computational guarantees of folded concave penalised SDR in this subsection. The ${\ell }_0$-constrained SDR

$${\widehat{\theta }}_{\text{1}}\text{=}\begin{array}{lllll}\underset{\theta }{\arg \max }& \text{tr}\left({\theta }^T\widehat{\mathit{M}}\theta \right)& \mathit{s}\text{.t.}& {‖\mathit{\theta }‖}_2^2\le 1\text{,}& {‖\mathit{\theta }‖}_{\text{0}}\le t\end{array}$$

achieves the desired theoretical properties under the high-dimensional setting, while folded concave penalised SDR is computationally feasible and it asymptotically converges to ${\ell }_0$-constrained SDR. First, we define the true sufficient dimension vectors. For the selected method, $d$ is the smallest integer such that, the top $d$ eigenvectors of ${\mathit{M}}^{★}$ are ${\theta }_i^{★}$ for $i=1,\dots ,d$, such that $y\mathbf{\perp }\mathit{x}\text{|}{\theta }_i^{★}$ for $i=1,\dots ,d$. Then by definition, we know ${\theta }_i^{★},\dots ,{\theta }_d^{★}$ are true sufficient vectors. Now we introduce assumptions thm(A1)–thm(A3).

$E({\xi }^T\mathit{x}|{\mathit{\theta }}_1^T\mathit{x},\dots ,{\mathit{\theta }}_d^T\mathit{x})$ is a linear function of ${\mathit{\theta }}_1^T\mathit{x},\dots ,{\mathit{\theta }}_d^T\mathit{x}$ for any $\xi \in {ℝ}^p$.

$\mathrm{cov}(\mathit{x}|{\mathit{\theta }}_1^T\mathit{x},\dots ,{\mathit{\theta }}_d^T\mathit{x})$is degenerate.

True sufficient dimension vectors ${\theta }_i^{★}$ satisfy ${‖{\theta }_i^{★}‖}_0\le t$ for $i=1,\dots ,d$.

Note that thm(A1), thm(A2), and thm(A3) are also called the linearity condition, the constant variance condition, and the sparsity condition, respectively. thm(A1) and thm(A2) hold when $\mathit{x}$ follows elliptical distributions. thm(A3) is a standard assumption in sparse SDR literature. See Li (1991), Li and Wang (2007), Li (2007) and Luo et al. (2022) for justifications of these conditions.

Let sin $\angle ({\mathit{\theta }}_a,{\mathit{\theta }}_b)=\sqrt{1-{\left({\mathit{\theta }}_a^T{\mathit{\theta }}_b\right)}^2}$. We first study the convergence rate of the first estimated sparse SDR direction ${\widehat{\mathit{\theta }}}_{\text{1}}$. Theorem 1 derives this estimation bound based on the general estimation bound that ${‖\widehat{\mathit{M}}-{\mathit{M}}^{★}‖}_2=O_p\left(\tau \left(n,p\right)\right)$.

Theorem 5.1: Suppose (A1)–(A3), ${‖\widehat{\mathit{M}}-{\mathit{M}}^{★}‖}_2=O_p\left(\tau \left(n,p\right)\right)$, ${\lambda }_1>{\lambda }_2$, ${‖{\mathit{\theta }}_1^{★}‖}_0\le t$, and the spectral decomposition for ${\mathit{M}}^{★}$ exists. Then we have:

$$\sin \angle ({\widehat{\mathit{\theta }}}_{\text{1}},{\mathit{\theta }}_1^{★})=O_p\left(\tau \left(n,p\right)\right).$$

In practice, $O_p\left(\tau \left(n,p\right)\right)$ has been derived for each specific SDR method. Corollary 1 discusses the explicit convergence rate, given the explicit convergence rate $O_p\left(\tau \left(n,p\right)\right)$ for different methods.

Corollary 5.2: (a) Suppose (A1)–(A3), ${\lambda }_1>{\lambda }_2$, ${‖{\mathit{\theta }}_1^{★}‖}_0\le t$, covariance matrix ${\mathbf{\Sigma }}_x={\mathit{I}}_{p*p}$, and $p=o\left(n^{1/2}\right)$ as $n\to \infty$, then for sparse SIR and sparse DR, we have:

$$\sin \angle ({\widehat{\mathit{\theta }}}_{\text{1}},{\mathit{\theta }}_1^{★})=O_p\left(p{n}^{-1/2}\right)$$

(b) Let $v\in \left[0,1\right]$ corresponds the stability of the central curve $E\left[\mathit{x}|y\right]$. When $p=o\left(n\right)$, with proper choice of number of slices $H={\left(\frac{n}{p}\right)}^{\frac{1}{2v+2}}$ and (A3)–(A4) in Lin et al. (2018) are satisfied, for sparse SIR, we have:

$$\begin{array}{lll}\sin \angle ({\widehat{\mathit{\theta }}}_{\text{1}},{\mathit{\theta }}_1^{★})\to & as& n\to \infty .\end{array}$$

Further, if we have ${\mathbf{\Sigma }}_{\mathit{x}}={\mathit{I}}_{p*p}$, we have convergence rate:

$$\sin \angle ({\widehat{\mathit{\theta }}}_{\text{1}},{\mathit{\theta }}_1^{★})=O_p\left({\left(\frac{p}{n}\right)}^{\frac{1}{2v+2}}\right)$$

(c) Let $s=‖𝒮‖\ll p$ where $𝒮=\left\{i|{\mathit{\theta }}_j\left(i\right)\ne 0forsomej,i\le j\le d\right\}$. Under the high-dimensional setting that $p\gg n$, by assuming (A3)–(A6) and (S1) in Lin et al. (2018), and ${\mathbf{\Sigma }}_{\mathit{x}}={\mathit{I}}_{p*p}$, then for the sparse DT-SIR, with proper choice of $H={\left(\frac{n}{p}\right)}^{\frac{1}{2v+2}}$, we have convergence rate:

$$\sin \angle ({\widehat{\mathit{\theta }}}_{\text{1}},{\mathit{\theta }}_1^{★})=O_p\left({\left(\frac{s}{n}\right)}^{\frac{1}{2v+2}}\right).$$

where the stability $v$ of the central curve $E\left[\mathit{x}|y\right]$ is defined in definition 1 of Lin et al. (2019). We also provide the complete definition of it in the supplemental material.

It is worth mentioning that our framework is very flexible. By applying different SDR methods to estimate the the $\mathit{M}$ matrix, we can derive the convergence rate under different settings. Let $j^{★}=\left\{j:{\theta }_{1j}^{★}\ne 0\right\}$ is the support set of ${\theta }_1^{★}$. We further prove the variable selection consistency in Theorem 2.

Theorem 5.3: Under the assumption of Theorem 1, further if ${\min }_{j\in J^{★}}\left|{\theta }_{1j}^{★}\right|\ge C_0\tau \left(n,p\right)$ for some constant $C_0>0$, and ${‖{\mathit{\theta }}_1^{★}‖}_0=t$, then $\mathrm{supp}\left({\widehat{\mathit{\theta }}}_1\right)=\mathrm{supp}\left({\mathit{\theta }}_1^{★}\right)$ with probability tending to 1 as $n\to \infty$.

Next, recall that in Section 2, for $k\ge 2$, by the same optimisation procedure, we can further estimate ${\widehat{\theta }}_k$ from ${\widehat{\mathit{M}}}_k$. Next, we study the estimation bound and consistency for ${\widehat{\theta }}_k\left(k\ge 2\right)$ in Theorem 3.

Theorem 5.4: Suppose that the conditions in Theorems 1 and 2 are satisfied, ${‖{\mathit{\theta }}_k^{★}‖}_0\le t$, ${\lambda }_k>{\lambda }_{k+1}$ and ${\lambda }_k=𝒪\left(1\right)$ for $1\le k\le d-1$. Then for $1\le k\le d-1$, we have:

$${‖{\widehat{\mathit{M}}}_{k+1}-{\mathit{M}}_{k+1}^{★}‖}_2=O_p\left(\tau \left(n,p\right)\right)$$

$$\sin \angle ({\widehat{\mathit{\theta }}}_{k+1},{\mathit{\theta }}_{k+1}^{★})=O_p\left(\tau \left(n,p\right)\right).$$

where ${\widehat{\mathit{M}}}_{k+1}$ and ${\mathit{M}}_{k+1}^{★}$ are defined in Section 2.

Now, we provide the computational guarantees of our proposed nonconvex ADMM. First, we show that the folded concave penalised estimation well approximates the ${\ell }_0$ penalisation. To this end, we use the rescaled folded concave penalty function, as we defined in Section 2. It is easy to see ${\lim }_{\delta \to 0}{P}_{\delta }\left(t\right)={‖t‖}_0$ for any $t\in ℝ$. Thus, $P_{\delta }\left(t\right)$ converges to ${\left|t\right|}_0$ in the ${\ell }_0$ penalisation. For matrix $\mathbf{\Theta }$, we define the penalisation $P_{\delta }\left(\left|\mathbf{\Theta }\right|\right)={\sum }_{i=1}^p{\sum }_{j=1}^{d}P\left({\mathbf{\Theta }}_{ij}\right)$, which is the summation of univariate rescaled penalisation for all entries. Here, we reveal the connection between ${\ell }_0$ penalised programming (1) and scaled folded-concave penalised programming (3) in the following theorem.

Theorem 5.5: Suppose we choose $\left\{{\delta }_u\right\}$ to be sequence such that ${\text{lim}}_{\mathit{U}\to \infty }{\delta }_u=0^{+}$. For the programming (3), we choose penalisation function $P_{\delta }\left(t\right)$ as we defined. Let ${\mathbf{\Theta }}^u$ be the global minimiser of (3) when we choose $\delta ={\delta }_u$, and assume ${\mathbf{\Theta }}^{\infty }$ is a limit point of $\left\{{\mathbf{\Theta }}^u\right\}$. Then ${\mathbf{\Theta }}^{\infty }$ is also a global minimiser to (1).

Since the nonconvex problem (3) has widely varying curvature, it is nontrivial to prove the algorithmic convergence. By taking advantage of the structured nonconvexity, the next theorem shows that our proposed nonconvex ADMM converges to an $ϵ$-stationary solution with an explicit iteration complexity bound. In the following theorem, we use $vec\left(\mathbf{\Theta }\right)$ to represent the vectorisation of a matrix from ${ℝ}^{p\times d}$ to a vector ${ℝ}^{pd}$.

Theorem 5.6: Let L be the Lipschitz bound of loss function, $\rho >\max \left(\frac{18\sqrt{3}+6}{13}L,6hL\right)$ and

$$\gamma \in \left\{(\begin{array}{ll}\left(\frac{\sqrt{13{\rho }^2-12\rho L-72L^2}-2\rho }{9{\rho }^2-12\rho L-72L^2},+\infty \right)\hfill & \rho \in \left(\frac{2+2\sqrt{19}}{3}L,+\infty \right)\hfill \\ \frac{2\rho -\sqrt{13{\rho }^2-12\rho L-72L^2}}{72L^2-12\rho L-9{\rho }^2},\hfill \\ \frac{2\rho +\sqrt{13{\rho }^2-12\rho L-72L^2}}{72L^2+12\rho L-9{\rho }^2})\hfill & \rho \in \left(\frac{18\sqrt{3}+6}{13}L,\frac{2+2\sqrt{19}}{3}L\right]\hfill \end{array}\right.$$

For any $0<ϵ<\min \left\{\frac{1}{L},\frac{1}{3h}\right\}$, with no more than $𝒪\left(\frac{1}{{ϵ}^4}\right)$ number of iterations, our proposed algorithm obtains an $ϵ$-stationary solution ($\widehat{\mathbf{\Theta }}$, $\widehat{\mathbf{\Omega }}$, $\widehat{\mathbf{\Lambda }}$), as defined in Jiang et al. (2019), for any matrix $\mathbf{\Theta }\in {ℝ}^{p\times p}$ with finite Frobenius norm satisfying that:

$${\left(vec\left(\mathbf{\Theta }-\widehat{\mathbf{\Theta }}\right)\right)}^T\left[vec\left(\widehat{\mathit{M}}\text{+}\partial P_{\lambda }\left(\left|\mathbf{\Theta }\right|\right)-\widehat{\mathbf{\Lambda }}\mathbf{I}\right)\right]\ge -ϵ$$

$$‖\widehat{\mathbf{\Theta }}-\widehat{\mathbf{\Omega }}‖\le ϵ$$

$$\begin{array}{ll}\mathrm{tr}\left(\widehat{\mathbf{\Omega }}\right)=1,& \widehat{\mathbf{\Omega }}\underset{\_}{\succ }0.\end{array}$$

Remark: When $ϵ=0$, the constraint exactly implicates the KKT-condition. Thus, the solution will be a stationary solution. We will show numerical performance of such a near-stationary solution in the next section.

5.2. Asymptotic properties of sparse KSDR

In this section, we establish the consistency of kernel sparse dimension reduction under milder assumptions. Based on Fukumizu et al. (2009), we assume following assumptions thm(A4)–thm(A7).

For any bounded continuous function $g$ on $𝒴$, the mapping $\mathbf{\Theta }\to E_x\left[E_{y|{\mathbf{\Theta }}^T\mathit{x}}{[g\left(y\right)|{\mathbf{\Theta }}^T\mathit{X}]}^2\right]$ is continuous on $\mathbf{\Theta }\in {\mathbb{S}}_k^p\left(ℝ\right)$ for $\text{k}\text{=}\text{1,2,}\dots \text{d}$.

For any $k\text{=}\text{1,2,}\dots d$, let $\mathbf{\Theta }\in {\mathbb{S}}_k^p\left(ℝ\right)$, and $P_{\mathbf{\Theta }}$ be the probability distribution of the random variable ${\mathbf{\Theta }}^T\mathit{x}$ on $𝒳$. The Hilbert space ${\mathscr{H}}_{𝒳}^{𝒮}+ℝ$ is dense in $L^2\left(P_{\mathbf{\Theta }}\right)$ for any $\mathbf{\Theta }\in {\mathbb{S}}_k^p\left(ℝ\right)$.

There is a measurable function $\varphi :𝒳\to ℝ$ such that $E{\left|\varphi \left(\mathit{x}\right)\right|}^2<\infty$ and the Lipschitz condition: ${‖k_s\left({\mathbf{\Theta }}^T\mathit{x}\text{,}\cdot \right)-k_s\left({\tilde{\mathbf{\Theta }}}^T\mathit{x}\text{,}\cdot \right)‖}_{{\mathscr{H}}_d}\le \varphi \left(\mathit{x}\right)D\left(\mathbf{\Theta },\tilde{\mathbf{\Theta }}\right)$ holds for all $\mathbf{\Theta }$, $\tilde{\mathbf{\Theta }}\in \mathbb{S}{\mathbb{P}}_k^p\left(ℝ\right)$ and $x\in 𝒳$ for any $\text{k}\text{=}\text{1,2,}\dots \text{d}$, and the kernel $k_s\left(x\text{,}\cdot \right)$ is defined in Section 3. $D$is a distance which is compatible with the topology of $\mathbb{S}{\mathbb{P}}_k^p\left(ℝ\right)$. And we assume ${‖{\mathbf{\Sigma }}_{yy}‖}_{tr}$, ${‖{\mathbf{\Sigma }}_{y\mathit{x}}‖}_{HS}\le \infty$. The true sparse SDR directions ${\mathbf{\Theta }}^{★}=\left({\mathit{\theta }}_1^{★},\dots ,{\mathit{\theta }}_d^{★}\right)$ satisfies ${\mathbf{\Theta }}^{★}\in \mathbb{S}{\mathbb{P}}_d^p\left(ℝ\right)$.

Note that thm(A1) and thm(A2) are not required for the sparse KSDR, and we rewrite the sparsity assumption thm(A3) in our defined sparse orthogonal space as thm(A7). In thm(A4) to thm(A6), we introduce some mild regularity conditions in RKHS, which are well justified in Fukumizu et al. (2009).

Theorem 5.7: Under assumption (A4) to (A7), and we also suppose true variable set $\mathrm{supp}({\theta }_1^{★})$ are correlated with a finite number of variables; Suppose our choice of kernel function is continuous and bounded, and suppose the regularisation parameter ${ϵ}_n$ satisfies:

$$\begin{array}{llllll}{ϵ}_n\to 0;\hfill & {\left(\frac{n}{p^t}\right)}^{\frac{1}{t+2}}\hfill & {ϵ}_n\to \infty ;p\lesssim n^a\hfill & \mathrm{such}that\alpha <1/t\hfill & as\hfill & n\to \infty .\hfill \end{array}$$

Then the functions $\mathrm{tr}\left[{\widehat{\mathbf{\Sigma }}}_{yy|{\mathit{\theta }}^T\mathit{x}}\right]$ and $\mathrm{tr}\left[{\mathbf{\Sigma }}_{yy|{\mathit{\theta }}^T\mathit{x}}\right]$ are continuous on $\mathbb{S}{\mathbb{P}}_1^p\left(ℝ\right)$, and it converges in probability that:

$$\underset{\mathit{\theta }\in \mathbb{S}{\mathbb{P}}_1^p\left(ℝ\right)}{\text{sup}}\left|\text{tr}\left[{\widehat{\mathbf{\Sigma }}}_{yy|{\mathit{\theta }}^{T}x}^{\left(n\right)}\right]-\text{tr}\left[{\mathbf{\Sigma }}_{yy|{\mathit{\theta }}^{T}x}\right]\right|\to 0.$$

Theorem 6 establishes the consistency of the first KDR vector. The growth rate assumption in the theorem says that the growth rate of $p$ is negatively affected by the sparsity level $t$. When $t$ is smaller, $p$ can diverge faster as $n\to \infty$. Because of this uniform convergence, we establish the variable selection consistency by following Theorem 7, allowing a diverging dimension of $p$.

Theorem 5.8: Under all conditions in Theorem 6, for $p\lesssim n^{\alpha }$, such that $\alpha <1/t$ as $n\to \infty$, and $k=1,\dots ,d$, we have:

$$Pr\left(\mathrm{supp}\left({\widehat{\mathit{\theta }}}_k\right)=\mathrm{supp}\left({\theta }_k^{★}\right)\right)\to 1.$$

When variable selection consistency is achieved, we can further construct estimation consistency of $\widehat{\mathit{\theta }}$:

Theorem 5.9: If all conditions in Theorem 7 hold, and we assume $p\lesssim n^{\alpha }$, such that $\alpha <1/t$ as $n\to \infty$. Then we denote:

$$\begin{array}{lllll}{\widehat{\mathit{\theta }}}_{\text{1}}=\underset{\mathit{\theta }}{\text{arg}\text{min}}& \mathrm{tr}\left({\mathbf{\Sigma }}_{yy|{\mathit{\theta }}^T\mathit{x}}\right)& \mathit{s}\text{.t.}& {‖\mathit{\theta }‖}_2=1,& {‖\mathit{\theta }‖}_0\le t\end{array}$$

And define the corresponding matrix ${\tilde{\mathbf{\Theta }}}_1={\widehat{\mathit{\theta }}}_{\text{1}}{\widehat{\mathit{\theta }}}_1^T$. When ${\tilde{\mathbf{\Theta }}}_1$ is nonempty, for any arbitrary open set $U$ in ${\mathbb{S}}_1^p\left(ℝ\right)$ with true direction ${\mathit{\theta }}^{★}{\mathit{\theta }}^{★T}\in U$, we have

$$\underset{n\to \infty }{\lim }\mathrm{Pr}\left({\tilde{\mathbf{\Theta }}}_1\in U\right)=1.$$ (4)

For following directions, let ${\mathit{\theta }}_{\text{1}},\dots ,{\mathit{\theta }}_k$ correspond to the true first k KDR vectors, and

$$\begin{array}{lllll}{\widehat{\mathit{\theta }}}_{k+1}=\underset{\mathit{\theta }}{\text{arg}\text{min}}\hfill & \text{tr}\left({\mathbf{\Sigma }}_{yy|{\left({\sum }_{i=1}^k{\widehat{\mathit{\theta }}}_i{\widehat{\mathit{\theta }}}_i^T+\mathit{\theta }{\mathit{\theta }}^T\right)}^T\mathit{x}}\right)\hfill & \mathit{s}\text{.t.}\hfill & {‖\mathit{\theta }‖}_2=1,\hfill & {‖\mathit{\theta }‖}_0\le t.\hfill \end{array}$$

Denote${\tilde{\mathbf{\Theta }}}_{k+1}={\sum }_{i=1}^k{\widehat{\mathit{\theta }}}_i{\widehat{\mathit{\theta }}}_i^T+{\widehat{\mathit{\theta }}}_{k+1}{\widehat{\mathit{\theta }}}_{k+1}^T$, and the true space ${\mathbf{\Theta }}_{k+1}={\sum }_{i=1}^{k+1}{\left({\mathit{\theta }}_i^{★}{\mathit{\theta }}_i^{★}\right)}^T$. When ${\widehat{\mathit{\theta }}}_i$ are all not nonempty for $i=1,\dots ,k$, for any arbitrary open set $U$ in ${\mathbb{S}}_{k+1}^p\left(ℝ\right)$ with ${\mathbf{\Theta }}_{k+1}\in U$, we have

$$\underset{n\to \infty }{\lim }\mathrm{Pr}\left({\tilde{\mathbf{\Theta }}}_{k+1}\in U\right)=1.$$ (5)

Theorem 8 extends the KSDR consistency results from Fukumizu et al. (2009) to a diverging $p$ case. Making use of the ${\ell }_0$-constrained parameter space, we have both variable selection and estimation consistency under certain conditions.

Following the statistical property, Theorem 9 shows the asymptotic equivalency of folded concave constraint and ${\ell }_0$ constraint. Theorem 10 shows that our nonconvex linearised ADMM guarantees the convergence to an $ϵ$-stationary solution.

Similar with Theorem 4, for Theorem 9, we consider

$$\begin{array}{llll}\underset{\mathbf{\Theta }\in {ℝ}^{p\times p}}{\text{min}}& \text{tr}\left({\mathbf{\Sigma }}_{yy|{\mathbf{\Theta }}^{T}x}\right)+\gamma {\text{1}}^T{\left|\mathbf{\Theta }\right|}_0\text{1}& \text{tr}\left(\mathbf{\Theta }\right)=1\text{;}& \mathbf{\Theta }\underset{\_}{\succ }0\end{array}$$

$$\begin{array}{llll}\underset{\mathbf{\Theta }\in {ℝ}^{p\times p}}{\text{min}}& \text{tr}\left({\mathbf{\Sigma }}_{yy|{\mathbf{\Theta }}^{T}x}\right)+\gamma P_{\delta }\left(\left|\mathbf{\Theta }\right|\right)& \text{tr}\left(\mathbf{\Theta }\right)=1\text{;}& \mathbf{\Theta }\underset{\_}{\succ }0\end{array},$$

where $P_{\delta }\left(\left|\mathbf{\Theta }\right|\right)$ is defined in Section 2. Considering $d=1$, we can also solve sufficient directions iteratively, as introduced in Section 2.

Theorem 5.10: Suppose we choose $\left\{{\delta }_u\right\}$ to be sequence such that ${\text{lim}}_{\mathit{U}\to \infty }{\delta }_u=0^{+}$. For the programming (4.12), we choose penalisation function $P_{\delta }\left(t\right)$ as we defined. Let ${\mathbf{\Theta }}^u$ be the global minimiser of (4.12) when we choose $\delta ={\delta }_u$, and ${\mathbf{\Theta }}^{\infty }$ is a limit point of $\left\{{\mathbf{\Theta }}^u\right\}$. Then ${\mathbf{\Theta }}^{\infty }$ is also a global minimiser to (4.11).

Parallel to Theorem 5, we show the property for the $ϵ$-stationary solution of our proposed nonconvex ADMM in Theorem 10.

Theorem 5.11: Let L be the Lipschitz constant of the loss function. Let

$$\rho >\max \left(\frac{18\sqrt{3}+6}{13}L,6hL\right)$$

and

$$\gamma \in \left\{(\begin{array}{ll}\left(\frac{\sqrt{13{\rho }^2-12\rho L-72L^2}-2\rho }{9{\rho }^2-12\rho L-72L^2},+\infty \right)\hfill & \rho \in \left(\frac{2+2\sqrt{19}}{3}L,+\infty \right)\hfill \\ \frac{2\rho -\sqrt{13{\rho }^2-12\rho L-72L^2}}{72L^2-12\rho L-9{\rho }^2},\hfill \\ \frac{2\rho +\sqrt{13{\rho }^2-12\rho L-72L^2}}{72L^2+12\rho L-9{\rho }^2})\hfill & \rho \in \left(\frac{18\sqrt{3}+6}{13}L,\frac{2+2\sqrt{19}}{3}L\right]\hfill \end{array}\right..$$

For any $0<ϵ<\min \left\{\frac{1}{L},\frac{1}{3h}\right\}$, with no more than $𝒪\left(\frac{1}{{ϵ}^4}\right)$ number of iterations, our proposed algorithm obtains an $ϵ$-stationary solution ($\widehat{\mathbf{\Theta }}$, ${\widehat{\mathbf{\Omega }}}_{\text{1}}$, ${\widehat{\mathbf{\Omega }}}_2$, ${\widehat{\mathbf{\Lambda }}}_1$, ${\widehat{\mathbf{\Lambda }}}_2$), as defined in Jiang et al. (2019), for any $\mathbf{\Theta }\in {ℝ}^{p\times p}$ with finite Frobenius norm,satisfying that

$${\left(vec\left(\mathbf{\Theta }-\widehat{\mathbf{\Theta }}\right)\right)}^T\left[vec\left(\frac{\partial }{\partial \mathbf{\Theta }}\mathrm{tr}\left({\widehat{\mathbf{\Sigma }}}_{yy|{\widehat{\mathbf{\Theta }}}^{T}x}\right)-\left({\widehat{\mathbf{\Lambda }}}_1+{\widehat{\mathbf{\Lambda }}}_2\right)\mathbf{I}\right)\right]\ge -ϵ$$

$${\left({\mathbf{\Omega }}_{\text{1}}-{\widehat{\mathbf{\Omega }}}_{\text{1}}\right)}^T\left[{P^{\prime }}_{\lambda }\left({\mathbf{\Omega }}_{\text{1}}\right)-\left(\left({\widehat{\mathbf{\Lambda }}}_1+{\widehat{\mathbf{\Lambda }}}_2\right)\mathbf{I}\right)\right]\ge -ϵ$$

$$‖\widehat{\mathbf{\Theta }}-{\widehat{\mathbf{\Omega }}}_{\text{1}}‖\le ϵ$$

$$‖\widehat{\mathbf{\Theta }}-{\widehat{\mathbf{\Omega }}}_2‖\le ϵ$$

$$\begin{array}{ll}\mathrm{tr}\left({\widehat{\mathbf{\Omega }}}_{\text{2}}\right)=1,& {\widehat{\mathbf{\Omega }}}_{\text{2}}\underset{\_}{\succ }0.\end{array}$$

6. Numerical properties

In this section, we evaluate the performance of the sparse SIR/DR/KSDR estimator in our semidefinite relaxation framework, the DT-SIR estimator proposed by Lin et al. (2019), the variable selection of penalised linear regression with SCAD penalisation and marginal correlation screening.

Here we compare the numerical performance of these methods in four data generation models as follows with sample size $n=100$.

• Model 1: $y=\frac{0.5\left(x_1+x_2\right)}{0.5+{\left(0.5\left(x_1-x_2\right)+1.5\right)}^2}+{\left(1+.05\left(x_1-x_2\right)\right)}^2+{\sigma }^{2}e$, where $e\sim N\left(0,1\right)$, $\mathit{x}\sim N\left(0,{\mathit{I}}_{\text{10}}\right)$ and $\sigma =0.2,0.4,0.8$ for Model 1a–1c.

• Model 2: $y=\frac{1}{2}\left(x_1^2+x_2^2\right)+{\sigma }^{2}e$, where $e\sim N\left(0,1\right)$, $\mathit{x}\sim N\left(0,{\mathit{I}}_{\text{20}}\right)$, and $\sigma =0.1,0.2,0.3$ for Model 2a–2c.

• Model 3: $y=\frac{1}{2}{\left(x_1+1\right)}^2+{\sigma }^{2}e$, where $e\sim N\left(0,1\right)$, $\mathit{x}\sim N\left(0,{\text{I}}_{\text{40}}\right)$, and $\sigma =0.2,0.4,0.8$ for Model 3a–3c.

• Model 4: $y={\sin }^2\left(\pi x_1+1\right)+{\sigma }^{2}e$, where $\sigma =0.2,0.4,0.8$ for Model 4a-4c, $e\sim N\left(0,1\right)$, and $\mathit{x}$ is uniformly distributed on $\left\{\mathit{x}\in {ℝ}^{\text{10}}|\exists x_i>0.7,\mathrm{for}i=1,2,\dots 10\right\}$

To compare the numerical performance of these methods, we replicate our data generation procedure 100 times and apply all the methods to these 100 replications. We compute the estimated sparse SDR vector ${\widehat{\mathit{\theta }}}_k$ for $k=1,2,\dots ,d$ for each method each time. Then, we evaluate the performance of different methods using multiple $R^2$, True Positive Rate (TPR) and True Negative Rate (TNR). The multiple $R^2$ is defined as $R^2=\frac{1}{d}{{\sum }_{k=1}^d‖P_{\text{span}\left({\mathbf{\Theta }}^{★}\right)}{\widehat{\mathit{\theta }}}_k‖}_2^2$. True Positive Rate(TPR) is defined as the fraction of variables correctly specified as non-zero in at least one out of SDR vectors, and True Negative Rate(TNR) is the fraction of variables correctly specified as zero in all SDR vectors.

The simulation results are summarised in the following Table 1–3. In the following Tables 1 and 2, we report the mean and standard deviation for these three indicators, with sparse SIR/DR/KSDR, DT-SIR, and SCAD penalised regression. The marginal correlation method doesn’t have $R^2$, and we report its true positive/negative rate in Table 3. The tuning procedure is as follows. For the proposed methods, we choose the penalisation parameter $\lambda$ via the oracle tuning procedure. Specifically, we construct two independent datasets with the same sample size $n=100$ for training and tuning respectively. For each $\lambda$, we estimate the parameter $\mathbf{\Theta }$ on the training dataset and evaluate the corresponding loss function on the tuning dataset. We choose the best penalisation parameter to minimise the loss function on the tuning dataset. In practice, we may use the cross-validation to find the proper penalisation parameter.

Table 1.

Summary of $R^2$, TP and TN for model 1 and 2.

	SpKSDR	DT-SIR	SpSIR	SpDR	SCAD	SpKSDR	DT-SIR	SpSIR	SpDR	SCAD
	Model 1a					Model 2a
$R^2$	0.833	0.843	0.980	0.995	0.656	0.975	0.224	0.215	0.965	0.172
	(0.303)	(0.153)	(0.098)	(0.050)	(0.063)	(0.112)	(0.250)	(0.278)	(0.146)	(0.355)
TPR	0.917	0.834	0.965	0.916	1	0.975	0.570	0.220	0.965	0.060
	(0.096)	(0.223)	(0.067)	(0.013)	(0)	(0.224)	(0.341)	(0.278)	(0)	(0.192)
TNR	0.975	0.423	0.976	0.969	0.660	0.988	0.429	0.805	0.990	0.905
	(0.069)	(0.255)	(0.066)	(0.091)	(0.191)	(0.038)	(0.270)	(0.070)	(0.042)	(0.103)
	Model 1b					Model 2b
$R^2$	0.917	0.834	0.965	0.916	0.654	0.950	0.235	0.180	0.929	0.183
	(0.203)	(0.158)	(0.151)	(0.278)	(0.064)	(0.154)	(0.274)	(0.261)	(0.199)	(0.363)
TPR	0.917	1	0.970	0.916	1	0.950	0.490	0.180	0.940	0.060
	(0.231)	(0)	(0.119)	(0.187)	(0)	(0.230)	(0.362)	(0.261)	(0.191)	(0.192)
TNR	0.975	0.423	0.976	0.969	0.626	0.981	0.464	0.795	0.949	0.895
	(0.069)	(0.255)	(0.066)	(0.091)	(0.229)	(0.205)	(0.270)	(0.065)	(0.181)	(0.136)
	Model 1c					Model 2c
$R^2$	0.900	0.757	0.950	0.720	0.649	0.900	0.233	0.185	0.920	0.196
	(0.203)	(0.158)	(0.151)	(0.278)	(0.073)	(0.230)	(0.266)	(0.263)	(0.197)	(0.371)
TPR	0.917	1	0.955	0.720	1	0.950	0.525	0.195	0.930	0.060
	(0.190)	(0)	(0.144)	(0.278)	(0)	(0.230)	(0.392)	(0.274)	(0.173)	(0.192)
TNR	0.967	0.375	0.978	0.930	0.610	0.975	0.454	0.799	0.981	0.893
	(0.080)	(0.228)	(0.060)	(0.070)	(0.236)	(0.052)	(0.281)	(0.069)	(0.048)	(0.138)

Table 3.

Summary of TP and TN for marginal correlation method.

	Model 1c	Model 2c	Model 3c	Model 4c
TPR	0.980	0.425	1	0.550
	(0.098)	(0.329)	(0)	(0.500)
TNR	0.873	0.913	1	0.950
	(0.012)	(0.018)	(0)	(0.056)

Table 2.

Summary of $R^2$, TP and TN for model 3 and 4.

	SpKSDR	DT-SIR	SpSIR	SpDR	SCAD	SpKSDR	DT-SIR	SpSIR	SpDR	SCAD
	Model 3a					Model 4a
$R^2$	0.889	0.953	1	0.990	0.995	0.667	0.449	0.600	0.128	0.608
	(0.333)	(0.060)	(0)	(0.099)	(0.017)	(0.479)	(0.337)	(0.492)	(0.333)	(0.445)
TPR	0.889	1	1	1	1	0.667	0.900	0.600	0.130	0.700
	(0.333)	(0)	(0)	(0)	(0)	(0.479)	(0.302)	(0.462)	(0.338)	(0.461)
TNR	0.997	0.850	1	0.990	0.983	0.963	0.502	0.950	0.873	0.881
	(0.009)	(0.138)	(0)	(0.239)	(0.039)	(0.052)	(0.297)	(0.062)	(0.132)	(0.172)
	Model 3b					Model 4b
$R^2$	0.850	0.913	1	0.940	0.996	0.567	0.381	0.360	0.100	0.438
	(0.362)	(0.099)	(0)	(0.238)	(0.012)	(0.504)	(0.332)	(0.482)	(0.293)	(0.451)
TPR	0.850	1	1	1	1	0.567	0.840	0.360	0.110	0.550
	(0.365)	(0)	(0)	(0)	(0)	(0.504)	(0.368)	(0.482)	(0.314)	(0.500)
TNR	0.996	0.840	1	0.940	0.985	0.952	0.542	0.920	0.889	0.873
	(0.009)	(0.138)	(0)	(0.239)	(0.030)	(0.055)	(0.268)	(0.060)	(0.052)	(0.127)
	Model 3c					Model 4c
$R^2$	0.778	0.573	1	0.500	0.992	0.533	0.257	0.320	0.164	0.335
	(0.440)	(0.348)	(0)	(0.503)	(0.021)	(0.507)	(0.285)	(0.469)	(0.367)	(0.429)
TPR	0.778	0.900	1	0.500	1	0.533	0.730	0.320	0.210	0.450
	(0.440)	(0.302)	(0)	(0.503)	0	(0.507)	(0.447)	(0.469)	(0.409)	(0.500)
TNR	0.994	0.781	1	0.987	0.984	0.948	0.520	0.915	0.862	0.863
	(0.011)	(0.178)	(0)	(0.013)	(0.035)	(0.055)	(0.287)	(0.059)	(0.156)	(0.136)

For implementation, Sparse SIR/DR/KSDR are implemented using our proposed non-convex ADMM, DT-SIR is implemented using R package ‘LassoSIR’, and SCAD penalised regression is implemented using R package ‘ncvreg’. For the marginal correlation method, we compute the correlation between $y$ and each variable $x_i$, then rank the absolute value of the correlation and select the first $d$ variables, where $d$ is the true dimension.

We also compare the performance of $R^2$ for kernel dimension reduction in Fukumizu et al. (2009) in Model 4(a, b, c) as a benchmark. The average $R^2$ are 0.642, 0.546 and 0.513 for Model 4(a), 4(b) and 4(c) separately. The results are comparable with SKDR results. SKDR performs slightly better given it eliminates some noise from irrelevant variables, thanks to the penalisation. Given kernel dimension reduction doesn’t provide a sparse solution, and doesn’t provide a meaningful TPR and TNR, we didn’t include them in the main table.

We draw the following conclusions from the results: (1) sparse SIR and DT-SIR are compatible in most cases, while sparse SIR outperforms DT-SIR in $R^2$ in general. In model 1 and model 3, when the assumption for SIR holds, our sparse SIR is almost perfect. DT-SIR also performs well in these two settings but sometimes includes many irrelevant variables and leads to an unsatisfying TNR and $R^2$. (2) The sparse DR is also compatible with SIR in general. Although it may not outperform SIR in model 1 and model 3, when the function is symmetric over $\mathit{x}$ in model 2, SIR fails to recover the true space, while sparse DR is very stable in this case. (3) sparse KSDR performs best in model 4 when linearity conditions thm(A1) and thm(A2) don’t hold, and both SIR and DR lose consistency. While sparse KSDR also performs fairly well compared to SIR and DR in models 1–3, when the linearity condition holds and no matter whether the function is symmetric over $\mathit{x}$. (4) The penalised regression fails in model 2 and model 4 when the relationship between $\mathit{x}$ and $y$ cannot be approximated by linear regression. Also, regression cannot identify the linear combination $\mathit{\theta }$, which is also of interest. (5) Similarly, the marginal correlation method fails in model 2 when the correlation between $\mathit{x}$ and $y$ is completely non-linear, and they have population correlation zero. What’s more, using correlation-based methods also cannot identify the linear combination $\mathit{\theta }$ of our interest.

7. Application to microbiome data analysis

In this section, we apply our method to analyse the host transcriptomic data and 16S rRNA gene sequencing data from paired biopsies from IPAA patients with UC and familial adenomatous polyposis (Morgan et al. 2015). Morgan et al. (2015) extracted 5 principal components of microbiome 16s rRNA gene sequencing data, and analysed the relationship between these PCs and the PCs from host transcriptomic data. However, the result is hard to interpret since PCs are linear combinations of all host transcriptomics. Thus we apply our proposed sparse KSDR to explore the more interpretable sparse linear combinations which are relevant to these microbiome PCs.

First we select 20 representative genes of host transcriptomic data from two groups. The first group of genes are those most different from two locations (PPI and Porch) of a patient by Kolmogorov–Smirnov test. The second group of genes are important genes pointed out in Morgan et al. (2015), which are important to the pathogenesis of inflammatory bowel disease (IBD). And we treat 9 principal components of microbiome 16s rRNA gene sequencing as $y$ in our model separately, and apply our kernel sparse diemsnion reduction estimation on the host transcriptomic data of PPI for each patient, with sample size $n=196$ and $p=20$. The selected genes and their coefficients are summarised in the following Table 4.

Table 4.

Selected genes and corresponding coefficients using KSDR.

PCs	Gene 1	Gene 2	Gene 3	Gene 4
PC1	MMEL1(0.52)	IL1RN(−0.50)	TLR1(−0.30)	IL10(−0.27)
PC2	TNF (−0.42)	MUC6(0.37)	TLR1(−0.36)	NFKB1(−0.29)
PC3	MUC6(0.58)	IL1RN(0.50)	SOX14(0.45)	FUT2(0.24)
PC4	MUC6(0.74)	IL1RN7096 (0.36)	TLR1(0.26)	NFKB1(0.23)
PC5	TLR1(−0.49)	MUC6(0.41)	IFNG(−0.40)	NFKB1(0.39)

In the result, leading PCs (PC1-PC5) are mainly correlated with protein-related genes such as TLR1 and MMEL1 It reflects that these PCs may be mainly composed of antibiotic-signature microbiome. Several genes that have been widely studied that are important to the pathogenesis of IBD is also selected. For example, IL10 is selected in PC1, MUC6 is selected in PC2, 3, 4, and 5, and IL1RN is selected in PC1, 3, 4. Among them, IL10 is well studied that its effectiveness in signalling a subgroup of patients with IBD (Begue et al. 2011). MUC6 may have a role in epithelial wound healing after mucosal injury in IBD in addition to mucosal protection (Buisine et al. 2001). IL1RN has a well-established pathological role in IBD (Stokkers et al. 1998). Our result emphasises the effects of these genes may be actively related to microbiome RNA environment of IBD patients.

8. Conclusion

Motivated by the desired property of sparse SDR, we propose the nonconvex estimation scheme for sparse KSDR. In aspect of statistical property, we prove the asymptotic consistency for both estimators. In the aspect of optimisation, we show that they can both be solved by nonconvex ADMM algorithm efficiently with a provable convergence guarantee. We also show the practical usefulness of our proposed methods in various simulation settings and real data analysis.