Optimal Estimation of Genetic Relatedness in High-dimensional Linear Models

Abstract

Estimating the genetic relatedness between two traits based on the genome-wide association data is an important problem in genetics research. In the framework of high-dimensional linear models, we introduce two measures of genetic relatedness and develop optimal estimators for them. One is genetic covariance, which is defined to be the inner product of the two regression vectors, and another is genetic correlation, which is a normalized inner product by their lengths. We propose functional de-biased estimators (FDEs), which consist of an initial estimation step with the plug-in scaled Lasso estimator, and a further bias correction step. We also develop estimators of the quadratic functionals of the regression vectors, which can be used to estimate the heritability of each trait. The estimators are shown to be minimax rate-optimal and can be efficiently implemented. Simulation results show that FDEs provide better estimates of the genetic relatedness than simple plug-in estimates. FDE is also applied to an analysis of a yeast segregant data set with multiple traits to estimate the genetic relatedness among these traits.

1. Introduction

1.1. Motivation and Background

Genome-wide association studies (GWAS) have led to identification of thousands of genetic variants or single nucleotide polymorphisms (SNPs) that are associated with various complex phenotypes (Manolio, 2010). Results from these GWAS have shown that many complex phenotypes share common genetic variants, including various autoimmune diseases (Zhernakova et al., 2009) and psychiatric disorders (Lee et al., 2013). These empirical evidence of shared genetic etiology for various phenotypes provides important insights of common pathophysiologies for related disorders that can be explored for drug repositioning and for studying disease etiology. Such knowledge of genetic sharing can potentially be explored to increase the accuracy of genetic risk prediction (Maier et al., 2015; Wray et al., 2007; Purcell et al., 2009). The concept of genetic relatedness or genetic correlations has been proposed to describe the shared genetic associations within pairs of quantitative traits based on GWAS data. This is in contrast to the traditional approaches of estimating co-heritability based on twin or family studies, where measurements of both traits are required on the same set of individuals. Due to the availability of GWAS data sets of many important traits, there has been significant recent interest in methods for quantifying and estimating the genetic relatedness between two traits based on large scale genetic association data.

Several measures of genetic relatedness have been proposed using GWAS data. Lee et al. (2012) and Yang et al. (2013) extended the mixed-effect model framework to estimate genetic covariance and genetic correlation between two traits. In their models, each individual’s trait value is associated with a random genetic effect, which is correlated across individuals by virtue of sharing some of the genetic variants affecting the traits, and an environmental random effect. Co-heritability is then defined as the square-root of the ratio of the covariance of the genetic random effects to the product of the total variances. The mixed-effect model approach requires knowledge of the identity of the causal variants, and hence the covariance matrix. This is however not available. Lee et al. (2012) and Yang et al. (2013) approximated the genetic relationship between every pair of individuals across the set of causal variants by the genetic relationship across the set of all genotyped variants. However, the very large number of variants used for estimating the genetic correlations, most of them likely not causative, might mask out the correlations on the set of causal variants, leading to inaccurate and suboptimal estimation of heritability (Golan and Rosset, 2011). Bulik-Sullivan et al. (2015) studied the genetic relatedness based on another random effects model for the two traits and developed a cross-trait linkage disequilibrium (LD) score regression to estimate the genetic covariance and genetic correlation. This approach shares similarity with the mixed-effect model approach of Yang et al. (2013) but has the advantages of only using the GWAS summary statistics. Lee and van der Wer (2016) developed an algorithm for multivariate linear mixed model analysis and demonstrated its use in estimating co-heritability.

To alleviate the difficulty of estimating the covariance matrix in the commonly used mixed effect model framework of estimating the heritability or co-heritability, we take a regression approach with fixed genetic effects in high-dimensional settings. High-dimensional linear regression provides a natural framework for GWAS in order to identify the trait-associated genetic variants, and its advantages over the simple univariate analysis have been demonstrated (Wu et al., 2009). The study of heritability in high-dimensional regression analysis has been studied in Bonnet et al. (2015); Verzelen and Gassiat (2016); Janson et al. (2016). However, high-dimensional regression analysis has not been explored to study the genetic relatedness between two traits based on genetic association data. The goal of this paper is to define two quantities that can be used to measure the genetic relatedness between a pair of traits based on GWAS data in the framework of high-dimensional linear models. Our definitions of the genetic relatedness reflect covariance or correlation of the trait-associated genetic variants. This is different from the mixed-effects model-based approaches where the genetic relatedness is defined through the variance/covariance matrix of the individual-specific random effects and the data from all the genetic variants are used to approximate the true covariance matrix.

1.2. Definition and Problem Formulation

A pair of trait values ($\mathbf{y}$, $\mathbf{w}$) are modeled as a linear combination of $p$ genetic variants and an error term that includes environmental and unmeasured genetic effects,

$${\mathit{y}}_{n_1\times 1}={\mathit{X}}_{n_1\times p}{\mathit{\beta }}_{p\times 1}+{\mathit{ϵ}}_{n_1\times 1}\text{and}{\mathit{w}}_{n_2\times 1}={\mathit{\text{Z}}}_{n_2\times p}{\mathit{\text{γ}}}_{p\times 1}+{\mathit{\delta }}_{n_2\times 1},$$ (1)

where the rows ${\mathit{\text{X}}}_{i·}$ are i.i.d. p-dimensional Sub-gaussian random vectors with covariance matrix $\mathbf{\text{Σ}}$, the rows ${\mathit{\text{Z}}}_{i·}$ are i.i.d. p-dimensional Sub-gaussian random vectors with covariance matrix $\mathbf{\text{Γ}}$ and the error ${(\mathit{ϵ},\mathit{\text{δ}})}^{\top }$ follows the multivariate normal distribution with mean zero and covariance

$$\left(\begin{array}{cc}{\sigma }_1^2{\mathbf{I}}_{n_1\times n_1}& {\mathbf{0}}_{n_1\times n_2}\\ {\mathbf{0}}_{n_2\times n_1}& {\sigma }_2^2{\mathbf{I}}_{n_2\times n_2}\end{array}\right)$$

and is assumed to be independent of $\mathit{\text{X}}$ and $\mathit{\text{Z}}$.

In the study of genetic relatedness, the pair of traits $\mathit{\text{y}}$ and $\mathit{\text{w}}$ are assumed to have mean zero, and the jth column of $\mathit{\text{X}}$, ${\mathit{\text{X}}}_{·j}$, and the jth column of $\mathit{\text{Z}}$, ${\mathit{\text{Z}}}_{·j}$, are the numerically coded genetic markers at the jth genetic variant and are assumed to have mean zero and variance 1. Under this model, if the columns of $\mathit{\text{X}}$ and $\mathit{\text{Z}}$ are independent, for the i-th observation,

$$\text{Var}\left({\mathit{y}}_i\right)=\sum _j{\mathit{\beta }}_j^2+{\sigma }_1^2=\parallel \mathit{\beta }{\parallel }_2^2+{\sigma }_1^2,\text{and}\text{Var}\left({\mathit{w}}_i\right)=\sum _j{\mathit{\text{γ}}}_j^2+{\sigma }_2^2=\parallel \mathit{\text{γ}}{\parallel }_2^2+{\sigma }_2^2,$$

therefore $\parallel \mathit{\beta }{\parallel }_2^2/\left(\parallel \mathit{\beta }{\parallel }_2^2+{\sigma }_1^2\right)$ and $\parallel \mathit{\text{γ}}{\parallel }_2^2/\left(\parallel \mathit{\text{γ}}{\parallel }_2^2+{\sigma }_2^2\right)$ can then be interpreted as the narrow sense heritability (Bulik-Sullivan et al., 2015).

Based on this model, one measure of genetic relatedness is the inner product of the regression coefficients

$$\text{I}(\mathit{\beta },\mathit{\gamma })=\langle \mathit{\beta },\mathit{\gamma }\rangle$$ (2)

which measures the shared genetic effects between these two traits. Bulik-Sullivan et al. (2015) defined this quantity as the genetic covariance due to the $p$ genetic variants. Alternatively, a normalized inner product called genetic correlation, that is the ratio

$$\text{R}(\mathit{\beta },\mathit{\gamma })=\frac{\langle \mathit{\beta },\mathit{\gamma }\rangle }{\parallel \mathit{\beta }{\parallel }_2\parallel \mathit{\text{γ}}{\parallel }_2}\mathbf{1}\left(\parallel \mathit{\beta }{\parallel }_2\parallel \mathit{\text{γ}}{\parallel }_2>0\right),$$ (3)

can also be used. In the case where one of $\parallel \mathit{\beta }{\parallel }_2$ and $\parallel \mathit{\text{γ}}{\parallel }_2$ is vanishing, the ratio is defined as zero, which indicates no correlation between two traits when one of the regression vector is zero. With this normalization, $\text{R}(\mathit{\beta },\mathit{\gamma })$ is always between −1 and 1 and can be used to compare the genetic relatedness among multiple pairs. Note that to exhibit genetic correlation, the directions of effect must also be consistently aligned.

Although Bulik-Sullivan et al. (2015) defined (2) and (3) as genetic covariance and genetic correlation, they treated $\mathit{\text{β}}$ and $\mathit{\text{γ}}$ as random vectors with a particular covariance form and then proposed to apply LD regression to estimate the expectation of $\langle \mathit{\beta },\mathit{\text{γ}}\rangle$. The focus of this paper is to develop estimators for $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ and $\text{R}(\mathit{\beta },\mathit{\gamma })$ based on two GWAS data with genotype data measured on the same set of genetic markers, denoted by $({\mathit{y}}_i,{\mathit{X}}_{i\cdot },i=1,\cdots ,n_1)$ and $\left({\mathit{w}}_i,{\mathit{Z}}_{i\cdot },i=1,\cdots ,n_2\right)$.

1.3. Methods and Main Results

A naive estimator is to estimate $\mathit{\text{β}}$ and $\mathit{\text{γ}}$ first and then plug-in the estimators of $\mathit{\text{β}}$ and $\mathit{\text{γ}}$ into the expressions (2) and (3). For the problem of interest, usually there are more genetic markers than the sample size, that is $p\ge \text{max}\{n_1,n_2\}$. However, for any trait, one expect that only a few of these markers have nonzero effects. One can apply any high-dimensional sparse regression methods such as Lasso (Tibshirani, 1996), scaled Lasso (Sun and Zhang, 2012) and marginal regression with screening (McCarthy et al., 2008; Fan et al., 2012) to estimate these sparse regression coefficients. The aforementioned plug-in estimators, however, have several drawbacks in estimating the genetic relatedness. The Lasso approach shrinks the estimation towards 0, in particular, some weak effects might be shrunken to 0, yet the accumulation of these weak effects may contribute significant to the trait variability. It is possible that some genetic variants may have strong effects on one trait and weak effects on the other trait. Due to shrinkage, the plug-in of Lasso type estimators fails to capture this part of contribution to genetic relatedness from such genetic variants. Marginal regression calculates the regression score between the trait and each single marker (i.e., $\mathit{\text{y}}$ and ${\mathit{X}}_{\cdot j},1\le j\le p$), and screen for the large scores. This approach also suffers from the existence of weak effects, as the marginal scores must be large enough to survive in the screening step.

We propose a two-step procedure to estimate the genetic relatedness measure $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ defined in (2), where step 1 is involved with estimating the inner product $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ by the plug-in scaled Lasso estimator, and step 2 is involved with correcting the plug-in scaled Lasso estimator. Similar two-step procedures are proposed to estimate the quadratic functionals $\parallel \mathit{\beta }{\parallel }_2^2$ and $\parallel \mathit{\text{γ}}{\parallel }_2^2$. To estimate the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$ defined in (3), we plug-in the estimators of the inner product and quadratic functionals into the definition (3). Due to the correction step, we name our estimators as Functional De-biased Estimators (FDEs).

FDEs are shown to achieve the minimax optimal convergence rates of estimating $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ and $\text{R}(\mathit{\beta },\mathit{\gamma })$. The optimality of FDEs results from the unique way of balancing the bias and variance for estimating $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ and $\text{R}(\mathit{\beta },\mathit{\gamma })$. To illustrate this, we focus on estimation of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, take the plug-in estimator of the scaled Lasso estimators (Sun and Zhang, 2012) and the plug-in of the de-biased Lasso estimators (Javanmard and Montanari, 2014; van de Geer et al., 2014; Zhang and Zhang, 2014) as examples and compare them with FDE estimator of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$. Note that the scaled Lasso estimator achieves the optimal convergence rate of estimating the whole vector $\beta$ and the de-biased estimator achieves the optimal convergence rate of estimating the single coordinate ${\beta }_i$. However, simply plugging in the scaled Lasso estimators or the de-biased Lasso estimators does not lead to a good estimator of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ since the plug-in estimator of scaled Lasso estimators suffers from a large bias and the plug-in estimator of de-biased Lasso estimators suffers from the inflation of variance.

In contrast, FDE estimator of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ balances the bias and variance in the optimal way. Specifically, in the correction step of FDE estimator, the bias caused by plugging in the scaled Lasso estimator is corrected through adding the minimum amount of variance. As demonstrated in the simulation studies, FDE consistently outperforms the plug-in estimator of the scaled Lasso estimators and the plug-in estimator of the de-biased Lasso estimators. In addition, FDEs do not suffer from dependency among genetic markers. FDEs work for a broad class of dependency structure of genetic markers.

The theoretical analysis given in Section 3 establishes the optimal convergence rates of estimating $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, $\text{R}(\mathit{\beta },\mathit{\gamma })$, $\parallel \beta {\parallel }_2^2$ and $\parallel \gamma {\parallel }_2^2$. To facilitate the discussion, we control the ${\ell }_2$ norm of regression coefficients $\text{β}$ and $\text{γ}$ as $c{\eta }_0\le \parallel \mathit{\beta }{\parallel }_2\le C{M}_0$ and $c{\eta }_0\le \parallel \mathit{\gamma }{\parallel }_2\le C{M}_0$ where $c$, $C$ are positive constant independent of $n$, $p$. Here, we present the most interesting regime where the signals are strong in the sense of ${\eta }_0\ge C\sqrt{k\log p/n}$, where $p$ is the dimension, $n$ is the sample size, k is the maximum sparsity of $\mathit{\text{β}}$ and $\mathit{\text{γ}}$ and $C$ is a positive constant independent of $k$, $n$, $p$. We have shown that the optimal rate of convergence for estimating $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, $\parallel \mathit{\beta }{\parallel }_2^2$ and $\parallel \mathit{\text{γ}}{\parallel }_2^2$ is

$$M_0\left(\frac{1}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{k\log p}{n}.$$

The optimal rate depends not only on $p$, $n$ and $k$, but also the upper bound for the signal strength $M_0$. In addition, we have shown that the optimal convergence rate of estimating $\text{R}(\mathit{\beta },\mathit{\gamma })$ is

$$\frac{1}{{\eta }_0}\left(\frac{1}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{1}{{\eta }_0^2}\frac{k\log p}{n}.$$

In contrast to estimating $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, $\parallel \mathit{\beta }{\parallel }_2^2$ and $\parallel \mathit{\text{γ}}{\parallel }_2^2$, the optimal rate scales to the inverse of the lower bound for the signal strength, represented by $1/{\eta }_0$. The estimators $\widehat{\text{I}}(\mathit{\text{β}},\mathit{\text{γ}})$, $\widehat{\text{Q}}(\mathit{\text{β}})$, $\widehat{\text{Q}}(\mathit{\text{γ}})$ and $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$ proposed in Section 3 are shown to adaptively achieve the optimal rates for estimating $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, $\text{Q}(\mathit{\text{β}})$, $\text{Q}(\mathit{\text{γ}})$ and $\text{R}(\mathit{\beta },\mathit{\gamma })$, respectively.

1.4. Notation and Definitions

Basic notation and definitions used in the rest of the paper are defined here. For a matrix $\mathit{X}\in {\mathbf{R}}^{n\times p}$, ${\mathit{X}}_{i\cdot }$, ${\mathit{X}}_{\cdot j}$, and ${\mathit{X}}_{i,j}$ denote respectively the i-th row, j-th column, and (i, j)-th entry of the matrix $\mathit{X}$, ${\mathit{X}}_{i,-j}$ denotes the i-th row of $\mathit{X}$ excluding the j-th coordinate, and ${\mathit{X}}_{-j}$ denotes the sub-matrix of $\mathit{X}$ excluding the j-th column. Let $[p]=\{1,2,···,p\}$. For a subset $J\subset [p]$, ${\mathit{X}}_J$ denotes the sub-matrix of $\mathit{X}$ consisting of columns ${\mathit{X}}_{\cdot j}$ with $j\in J$ and for a vector $\mathit{\text{x}}\in {\mathbf{R}}^p$, ${\mathit{\text{x}}}_J$ is the sub-vector of $\mathit{\text{x}}$ with indices in $J$ and ${\mathit{\text{x}}}_{-J}$ is the sub-vector with indices in $J^c$. For a vector $\mathit{\text{x}}\in {\mathbf{R}}^p$, the ${\ell }_q$ norm of $\mathit{\text{x}}$ is defined as $\parallel \mathit{x}{\parallel }_q={\left({\sum }_{i=1}^q{\left|{\mathit{x}}_i\right|}^q\right)}^{\frac{1}{q}}$ for $q\ge 0$ with ${\Vert \mathit{\text{x}}\Vert }_0$ denoting the cardinality of non-zero elements of $\mathit{\text{x}}$ and $\parallel \mathit{x}{\parallel }_{\infty }={\max }_{1\le j\le p}\left|{\mathit{x}}_j\right|$. For a matrix $A$ and $1\le q\le \infty$, $\parallel \mathit{A}{\parallel }_q={\sup }_{\parallel \mathit{x}{\parallel }_q=1}\parallel \mathit{A}\mathit{x}{\parallel }_q$ is the matrix ${\ell }_q$ operator norm. In particular, $\parallel \mathit{A}{\parallel }_2$ is the spectral norm. For a symmetric matrix $\mathit{\text{A}}$, ${\lambda }_{\text{min}}(\mathit{\text{A}})$ and ${\lambda }_{\text{max}}(\mathit{\text{A}})$ denote respectively the smallest and largest eigenvalue of $\mathit{\text{A}}$. For a set $S$, $|S|$ denotes the cardinality of $S$. For $a\in \mathbf{R}$, $a_{+}=\text{max}\{a,0\}$ and $\text{sign}(a)$ is the sign of $a$, i.e., $\text{sign}(a)=1$ if $a>0$, $\text{sign}(a)=-1$ if $a<0$ and $\text{sign}(0)=0$. Define the Sub-gaussian norm $\parallel \mathit{x}{\parallel }_{{\psi }_2}$ of $\mathit{x}\in {\mathbf{R}}^p$ as $\parallel \mathit{x}{\parallel }_{{\psi }_2}={\sup }_{\mathit{v}\in S^{p-1}}{\sup }_{q\ge 1}{\left(\mathbf{E}{\left|{\mathit{v}}^{\top }\mathit{x}\right|}^q\right)}^{\frac{1}{q}}/\sqrt{q}$ where $S^{p-1}$ is the unit sphere in ${\mathbf{R}}^p$. The random vector $\mathit{x}\in {\mathbf{R}}^p$ is defined to be Sub-gaussian if its corresponding Sub-gaussian norm is bounded; see Vershynin (2012) for more on Subgaussian random variables. For the design matriices $\mathit{X}\in {\mathbf{R}}^{n_1\times p}$ and $\mathit{\text{Z}}\in {\mathbf{R}}^{n_2\times p}$, we define the corresponding sample covariance matrices as $\widehat{\mathbf{\text{Σ}}}={\mathit{X}}^{\top }\mathit{X}/n_1$ and $\widehat{\mathbf{\text{Γ}}}={\mathit{\text{Z}}}^{\top }\mathit{Z}/n_2$. Let $z_{\alpha /2}$ denote the upper $\alpha /2$ quantile of the standard normal distribution. For two positive sequences $a_n$ and $b_n$, $a_n\lesssim b_n$ means $a_n\le C{b}_n$ for all $n$, $a_n\gtrsim b_n$ if $b_n\lesssim a_n$ and $a_n\asymp b_n$ if $a_n\lesssim b_n$ and $b_n\lesssim a_n$. $c$ and $C$ are used to denote generic positive constants that may vary from place to place. For any two sequences of numbers $a_n$ and $b_n$, we will write $b_n\ll a_n$ if $\text{lim sup}{b}_n/a_n=0$.

1.5. Organization of the Paper

The rest of the paper is organized as follows. Section 2 presents the procedures for estimating $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, $\text{Q}(\mathit{\text{β}})$, $\text{Q}(\mathit{\text{γ}})$, and $\text{R}(\mathit{\beta },\mathit{\gamma })$ in details. In Section 3, minimax convergence rates for the estimation problems are established and the proposed estimators are shown to attain the optimal rates. In Section 4, simulation studies are conducted to evaluate the empirical performance of FDEs. A yeast cross data is used to illustrate the estimators in Section 5. Discussion is provided in Section 6. The proofs of main theorems are present in Section 7. The remaining proofs and the extended simulation studies are given in the supplementary materials.

2. Estimation Methods

2.1. Estimation of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$

Since the inner product $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ is of significant interest in its own right, we first consider the estimation of $\text{I}(\mathit{\beta },\mathit{\gamma })=\langle \mathit{\beta },\mathit{\gamma }\rangle$. The scaled Lasso estimators for high-dimensional linear model (1) are defined through the following optimization algorithm (Sun and Zhang, 2012),

$$\left\{\widehat{\mathit{\beta }},{\widehat{\sigma }}_1\right\}=\arg \underset{\mathit{\beta }\in {\mathbf{R}}^p,{\sigma }_1\in {\mathbf{R}}^{+}}{\min }\frac{\parallel \mathit{y}-\mathit{X}\mathit{\beta }{\parallel }_2^2}{2n_1{\sigma }_1}+\frac{{\sigma }_1}{2}+\frac{{\lambda }_0}{\sqrt{n_1}}\sum _{j=1}^p\frac{{\Vert {\mathit{X}}_{\cdot j}\Vert }_2}{\sqrt{n_1}}\left|{\mathit{\beta }}_j\right|,$$ (4)

and

$$\left\{\widehat{\mathit{\gamma }},{\widehat{\sigma }}_2\right\}=\arg \underset{\mathit{\gamma }\in {\mathbf{R}}^p,{\sigma }_2\in {\mathbf{R}}^{+}}{\min }\frac{\parallel \mathit{w}-\mathit{Z}\mathit{\gamma }{\parallel }_2^2}{2n_2{\sigma }_2}+\frac{{\sigma }_2}{2}+\frac{{\lambda }_0}{\sqrt{n_2}}\sum _{j=1}^p\frac{{\Vert {\mathit{Z}}_{\cdot j}\Vert }_2}{\sqrt{n_2}}\left|{\mathit{\text{γ}}}_j\right|,$$ (5)

where ${\text{λ}}_0=\sqrt{2.01\log p}$. To construct an optimal estimator of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, it is helpful to analyze the error of the plugin estimator $\langle \widehat{\mathit{\beta }},\widehat{\mathit{\text{γ}}}\rangle$,

$$\langle \widehat{\mathit{\beta }},\widehat{\mathit{\text{γ}}}\rangle -\langle \mathit{\beta },\mathit{\text{γ}}\rangle =\langle \widehat{\mathit{\text{γ}}},\widehat{\mathit{\beta }}-\mathit{\beta }\rangle +\langle \widehat{\mathit{\beta }},\widehat{\mathit{\text{γ}}}-\mathit{\text{γ}}\rangle -\langle \widehat{\mathit{\beta }}-\mathit{\beta },\widehat{\mathit{\text{γ}}}-\mathit{\text{γ}}\rangle .$$ (6)

The last term on the right hand side, $\langle \widehat{\mathit{\beta }}-\mathit{\beta },\widehat{\mathit{\text{γ}}}-\mathit{\text{γ}}\rangle$ is “small”, but the first two terms $\langle \widehat{\mathit{\text{γ}}},\widehat{\mathit{\beta }}-\mathit{\beta }\rangle$ and $\langle \widehat{\mathit{\beta }},\widehat{\mathit{\text{γ}}}-\mathit{\text{γ}}\rangle$ can be large. This provides the motivation for the proposed estimator, where we first estimate these two terms and then subtract them from $\langle \widehat{\mathit{\beta }},\widehat{\mathit{\text{γ}}}\rangle$ to obtain the final estimator of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$.

The intuition for estimating $\langle \widehat{\mathit{\text{γ}}},\mathit{\beta }-\widehat{\mathit{\beta }}\rangle$ is given first. Since

$$\frac{1}{n_1}{\mathit{X}}^{\top }(\mathit{y}-\mathit{X}\widehat{\mathit{\beta }})=\widehat{\mathbf{\text{Σ}}}(\mathit{\beta }-\widehat{\mathit{\beta }})+\frac{1}{n_1}{\mathit{X}}^{\top }ϵ,$$ (7)

multiplying both sides of (7) by a vector $\mathit{\text{u}}\in {\mathbf{R}}^p$ yields

$$\frac{1}{n_1}{\mathit{u}}^{\top }{\mathit{X}}^{\top }(\mathit{y}-\mathit{X}\widehat{\mathit{\beta }})={\mathit{u}}^{\top }\widehat{\mathbf{\text{Σ}}}(\mathit{\beta }-\widehat{\mathit{\beta }})+\frac{1}{n_1}{\mathit{u}}^{\top }{\mathit{X}}^{\top }ϵ,$$ (8)

which can be written as

$$\frac{1}{n_1}{\mathit{u}}^{\top }{\mathit{X}}^{\top }(\mathit{y}-\mathit{X}\widehat{\mathit{\beta }})-\langle \widehat{\mathit{\text{γ}}},\mathit{\beta }-\widehat{\mathit{\beta }}\rangle ={(\widehat{\mathbf{\text{Σ}}}\mathit{u}-\widehat{\mathit{\text{γ}}})}^{\top }(\mathit{\beta }-\widehat{\mathit{\beta }})+\frac{1}{n_1}{\mathit{u}}^{\top }{\mathit{X}}^{\top }ϵ.$$ (9)

If the vector $\mathit{\text{u}}$ can be chosen such that the right hand side of (9) is “small”, then ${\mathit{u}}^{\top }{\mathit{X}}^{\top }(\mathit{y}-\mathit{X}\widehat{\mathit{\beta }})/n_1$ is a good estimator of $\langle \widehat{\mathit{\text{γ}}},\mathit{\beta }-\widehat{\mathit{\beta }}\rangle$. Since the first term on the right hand side of (9) is upper bounded as $\left|{(\widehat{\mathbf{\text{Σ}}}\mathit{\text{u}}-\widehat{\mathit{\text{γ}}})}^{\top }(\mathit{\text{β}}-\widehat{\mathit{\text{β}}})\right|\le \parallel \widehat{\mathbf{\text{Σ}}}\mathit{\text{u}}-\widehat{\mathit{\text{γ}}}{\parallel }_{\infty }\parallel \widehat{\mathit{\text{β}}}-\mathit{\text{β}}{\parallel }_1$, we control the right hand side of (9) through constructing a projection vector $\mathit{\text{u}}$ such that $\parallel \widehat{\mathbf{\text{Σ}}}\mathit{\text{u}}-\widehat{\mathit{\text{γ}}}{\parallel }_{\infty }$ is constrained and the second term of (9) ${\mathit{u}}^{\top }{\mathit{X}}^{\top }ϵ/n_1$ is controlled through minimizing its variance ${\sigma }_1^2{\mathit{u}}^{\top }\widehat{\mathbf{\text{Σ}}}\mathit{u}/n_1$. This leads to the following convex optimization algorithm for identifying the projection vector $\mathit{\text{u}}$ for estimating $\langle \widehat{\mathit{\text{γ}}},\mathit{\beta }-\widehat{\mathit{\beta }}\rangle$,

$${\widehat{\mathit{u}}}_1=\underset{\mathit{u}\in {\mathbf{R}}^p}{\arg \min }\left\{{\mathit{u}}^{\top }\widehat{\mathbf{\text{Σ}}}\mathit{u}:\parallel \widehat{\mathbf{\text{Σ}}}\mathit{u}-\widehat{\mathit{\gamma }}{\parallel }_{\infty }\le \parallel \widehat{\mathit{\gamma }}{\parallel }_2\frac{{\lambda }_1}{\sqrt{n_1}}\right\},$$ (10)

where ${\lambda }_1=12{\lambda }_{\max }^2(\mathbf{\text{Σ}})\sqrt{\log p}$.

Remark 1.

The solution of the above optimization problem might not be unique and ${\widehat{\mathit{u}}}_1$ is defined as any minimizer of the optimization problem. The theory established in Section 3 still holds for any minimizer of (10). The optimization problem (10) is solved through its equivalent Lagrange dual problem, which is computationally efficient and scales well to the high-dimensional problem. See Step 2 in Table 1 for more details.

Table 1:

FDE algorithm without sample splitting for estimating the inner product, quadratic functionals and the normalized inner product.

	Input: design matrices: $\mathit{\text{X}}$, $\mathit{\text{Z}}$; response vectors: $\mathit{\text{y}}$, $\mathit{\text{w}}$; tuning parameters ${\lambda }_0$, $\lambda$.
	Output: $\widehat{\text{I}}(\mathit{\beta },\mathit{\gamma })$, $\widehat{\text{Q}}(\mathit{\beta })$, $\widehat{\text{Q}}(\mathit{\gamma })$ and $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$.
Initial Lasso estimators:
1.	Scaled Lasso: Calculate $\widehat{\mathit{\beta }}$ and $\widehat{\mathit{\gamma }}$ from (4) and (5) with the tuning parameter ${\text{λ}}_0$.
Inner product calculation:
2.	Projection vector ${\widehat{\mathit{u}}}_1$: Calculate ${\widehat{\mathit{u}}}_1=\arg {\text{min}}_{\mathit{u}}{\mathit{u}}^{\top }{\mathit{X}}^{\top }\mathit{X}\mathit{u}/4n_1+{\mathit{u}}^{\top }\widehat{\mathit{\gamma }}+{\lambda }^t\parallel \mathit{u}{\parallel }_1$, where ${\text{λ}}^t={\text{λ}}^{t-1}/1.5$, and ${\text{λ}}^0=\text{λ}/\sqrt{n_1}$. Repeat until ${\widehat{\mathit{u}}}_1$ can not be solved with ${\text{λ}}^t$ replaced by ${\text{λ}}^{t+1}$, or $t\ge 10$.
3.	Projection vector ${\widehat{\mathit{u}}}_2$: Calculate ${\widehat{\mathit{u}}}_2=\arg {\min }_{\mathit{u}}{\mathit{u}}^{\top }{\mathit{Z}}^{\top }\mathit{Z}\mathit{u}/4n_2+{\mathit{u}}^{\top }\widehat{\mathit{\beta }}+{\lambda }^t\parallel \mathit{u}{\parallel }_1$, where ${\text{λ}}^t={\text{λ}}^{t-1}/1.5$, and ${\text{λ}}^0=\text{λ}/\sqrt{n_1}$. Repeat until ${\widehat{\mathit{u}}}_2$ can not be solved with ${\text{λ}}^t$ replaced by ${\text{λ}}^{t+1}$, or $t\ge 10$.
4.	Correction: $\widehat{\text{I}}(\mathit{\beta },\mathit{\gamma })=\langle \widehat{\mathit{\beta }},\widehat{\mathit{\gamma }}\rangle +{\widehat{\mathit{u}}}_1^{\top }{\mathit{X}}^{\top }(\mathit{y}-\mathit{X}\widehat{\mathit{\beta }})/n_1+{\widehat{\mathit{u}}}_2^{\top }{\mathit{Z}}^{\top }(\mathit{w}-\mathit{Z}\widehat{\mathit{\gamma }})/n_2$.
Quadratic functional calculation:
5.	Projection vector ${\widehat{\mathit{u}}}_3$: Calculate ${\widehat{\mathit{u}}}_3=\arg {\min }_{\mathit{u}}{\mathit{u}}^{\top }{\mathit{X}}^{\top }\mathit{X}\mathit{u}/4n_1+{\mathit{u}}^{\top }\widehat{\mathit{\beta }}+{\lambda }^t\parallel \mathit{u}{\parallel }_1$, where ${\text{λ}}^t={\text{λ}}^{t-1}/1.5$, and ${\text{λ}}^0=\text{λ}/\sqrt{n_1}$. Repeat until ${\widehat{\mathit{u}}}_3$ can not be solved with ${\text{λ}}^t$ replaced by ${\text{λ}}^{t+1}$, or $t\ge 10$.
6.	Projection vector ${\widehat{\mathit{u}}}_4$: Calculate ${\widehat{\mathit{u}}}_4=\arg {\min }_{\mathit{u}}{\mathit{u}}^{\top }{\mathit{Z}}^{\top }\mathit{Z}\mathit{u}/4n_2+{\mathit{u}}^{\top }\widehat{\mathit{\gamma }}+{\lambda }^t\parallel \mathit{u}{\parallel }_1$, where ${\text{λ}}^t={\text{λ}}^{t-1}/1.5$, and ${\text{λ}}^0=\text{λ}/\sqrt{n_2}$. Repeat until ${\widehat{\mathit{u}}}_1$ can not be solved with ${\text{λ}}^t$ replaced by ${\text{λ}}^{t+1}$, or $t\ge 10$.
7.	Correction: $\widehat{\text{Q}}(\mathit{\beta })={\left(\parallel \widehat{\mathit{\beta }}{\parallel }^2+2{\widehat{\mathit{u}}}_3^{\top }{\mathit{X}}^{\top }(\mathit{y}-\mathit{X}\widehat{\mathit{\beta }})/n_1\right)}_{+},$
	$\widehat{\text{Q}}(\mathit{\gamma })={\left(\parallel \widehat{\mathit{\gamma }}{\parallel }^2+2{\widehat{\mathit{u}}}_4^{\top }{\mathit{Z}}^{\top }(\mathit{w}-\mathit{Z}\widehat{\mathit{\gamma }})/n_2\right)}_{+}.$
Ratio calculation:
8.	$\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })=\text{sign}(\widehat{\text{I}}(\mathit{\beta },\mathit{\gamma }))\cdot \min \{(|\widehat{\text{I}}(\mathit{\beta },\mathit{\gamma })\mid /\sqrt{\widehat{\text{Q}}(\mathit{\beta })\widehat{\text{Q}}(\mathit{\gamma })})\mathbf{1}\{\widehat{\text{Q}}(\mathit{\beta })\widehat{\text{Q}}(\mathit{\gamma })>0\},1\}.$

Once the projection vector ${\widehat{\mathit{u}}}_1$ is obtained, $\langle \widehat{\mathit{\text{γ}}},\mathit{\beta }-\widehat{\mathit{\beta }}\rangle$ is then estimated by ${\widehat{\mathit{u}}}_1^{\top }{\mathit{X}}^{\top }(\mathit{y}-\mathit{X}\widehat{\mathit{\beta }})/n_1$. Similarly, the projection vector for estimating $\langle \widehat{\mathit{\beta }},\mathit{\text{γ}}-\widehat{\mathit{\text{γ}}}\rangle$ can be obtained via the convex algorithm

$${\widehat{\mathit{u}}}_2=\underset{\mathit{u}\in {\mathbf{R}}^p}{\arg \min }\left\{{\mathit{u}}^{\top }\widehat{\mathbf{\text{Γ}}}\mathit{u}:\parallel \widehat{\mathbf{\text{Γ}}}\mathit{u}-\widehat{\mathit{\beta }}{\parallel }_{\infty }\le \parallel \widehat{\mathit{\beta }}{\parallel }_2\frac{{\text{λ}}_2}{\sqrt{n_2}}\right\},$$ (11)

where ${\text{λ}}_2=12{\text{λ}}_{\max }^2(\mathbf{\Gamma })\sqrt{\log p}$. Then $\langle \widehat{\mathit{\beta }},\mathit{\text{γ}}-\widehat{\mathit{\text{γ}}}\rangle$ is estimated by ${\widehat{\mathit{u}}}_2^{\top }{\mathit{Z}}^{\top }(\mathit{w}-\mathit{Z}\widehat{\mathit{\gamma }})/n_2$.

The final estimator $\widehat{\text{I}}(\mathit{\text{β}},\mathit{\text{γ}})$ of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ is given by

$$\widehat{\text{I}}(\mathit{\beta },\mathit{\gamma })=\langle \widehat{\mathit{\beta }},\widehat{\mathit{\gamma }}\rangle +{\widehat{\mathit{u}}}_1^{\top }\frac{1}{n_1}{\mathit{X}}^{\top }(\mathit{y}-\mathit{X}\widehat{\mathit{\beta }})+{\widehat{\mathit{u}}}_2^{\top }\frac{1}{n_2}{\mathit{Z}}^{\top }(\mathit{w}-\mathit{Z}\widehat{\mathit{\gamma }}).$$ (12)

It is clear from the above discussion that the key idea for the construction of the final estimator $\widehat{\text{I}}(\mathit{\text{β}},\mathit{\text{γ}})$ is to identify the projection vectors ${\widehat{\mathit{u}}}_1$ and ${\widehat{\mathit{u}}}_2$ such that $\langle \widehat{\mathit{\text{γ}}},\mathit{\beta }-\widehat{\mathit{\beta }}\rangle$ and $\langle \widehat{\mathit{\beta }},\mathit{\text{γ}}-\widehat{\mathit{\text{γ}}}\rangle$ are well approximated. It will be shown in Section 3 that the estimator $\widehat{\text{I}}(\mathit{\text{β}},\mathit{\text{γ}})$ is adaptively minimax rate-optimal.

Remark 2.

As mentioned, simply plugging in the Lasso, scaled Lasso, or de-biased estimator does not lead to a good estimator of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$. Another natural approach is to first threshold the de-biased estimator to obtain a sparse estimator of the coefficient vectors (see details in Zhang and Zhang (2014, Section 3.3), Guo et al. (2016, equation (10))) and then plugin this thresholded estimator. This estimator is referred to as the thresholded estimator. Simulations in Section 4 demonstrate that the proposed estimator defined in (12) outperforms the three plug-in estimators using the scaled Lasso, de-biased, and thresholded estimators.

2.2. Estimation of $\text{Q}(\mathit{\text{β}})$ and $\text{Q}(\mathit{\text{γ}})$

In order to estimate the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$, it is necessary to estimate the quadratic functionals $\text{Q}(\mathit{\beta })=\parallel \mathit{\beta }{\parallel }_2^2$ and $\text{Q}(\mathit{\text{γ}})=\parallel \mathit{\text{γ}}{\parallel }_2^2$. To this end, we randomly split the data $(\mathit{\text{y}},\mathit{\text{X}})$ into two subsamples $\left({\mathit{y}}^{(1)},{\mathit{X}}^{(1)}\right)$ with sample size $n_1/2$ and $\left({\mathit{y}}^{(2)},{\mathit{X}}^{(2)}\right)$ with sample size $n_1/2$ and the data $(\mathit{\text{w}},\mathit{\text{Z}})$ into two subsamples $\left({\mathit{w}}^{(1)},{\mathit{Z}}^{(1)}\right)$ with sample size $n_2/2$ and $\left({\mathit{w}}^{(2)},{\mathit{Z}}^{(2)}\right)$ with sample size $n_2/2$.

With a slight abuse of notation, let $\widehat{\mathit{\beta }}$ and $\widehat{\mathit{\text{γ}}}$ denote the optimizers of the scaled Lasso algorithm (4) applied to $\left({\mathit{y}}^{(1)},{\mathit{X}}^{(1)}\right)$ and (5) applied to $\left({\mathit{w}}^{(1)},{\mathit{Z}}^{(1)}\right)$, respectively. For the scaled Lasso algorithms, the sample sizes $n_1$ and $n_2$ are replaced by $n_1/2$ and $n_2/2$, respectively. Again, the simple plug-in estimator $\parallel \widehat{\mathit{\beta }}{\parallel }_2^2$ of $\text{Q}(\mathit{\text{β}})$ is not a good estimator of $\parallel \mathit{\beta }{\parallel }_2^2$ because of the following error decomposition,

$$\parallel \widehat{\mathit{\beta }}{\parallel }_2^2-\parallel \mathit{\beta }{\parallel }_2^2=2\langle \widehat{\mathit{\beta }},\widehat{\mathit{\beta }}-\mathit{\beta }\rangle -\parallel \widehat{\mathit{\beta }}-\mathit{\beta }{\parallel }_2^2,$$ (13)

where the second term on the right hand side of (13) is “small”, but the first can be large. Specifically, the term $2\langle \widehat{\mathit{\beta }},\mathit{\beta }-\widehat{\mathit{\beta }}\rangle$ is estimated first and then is added to $\parallel \widehat{\mathit{\beta }}{\parallel }_2^2$ to obtain the final estimator of $\parallel \mathit{\beta }{\parallel }_2^2$. To estimate $\langle \widehat{\mathit{\beta }},\mathit{\beta }-\widehat{\mathit{\beta }}\rangle$, a projection vector $\mathit{\text{u}}$ is identified such that the following difference is controlled,

$$\frac{1}{n_1/2}{\mathit{u}}^{\top }{\left({\mathit{X}}^{(2)}\right)}^{\top }\left({\mathit{y}}^{(2)}-{\mathit{X}}^{(2)}\widehat{\mathit{\beta }}\right)-\langle \widehat{\mathit{\beta }},\mathit{\beta }-\widehat{\mathit{\beta }}\rangle =\left({\mathit{u}}^{\top }{\widehat{\mathbf{\text{Σ}}}}^{(2)}-\widehat{\mathit{\beta }}\right)(\mathit{\beta }-\widehat{\mathit{\beta }})+\frac{1}{n_1/2}{\mathit{u}}^{\top }{\left({\mathit{X}}^{(2)}\right)}^{\top }ϵ,$$ (14)

with ${\widehat{\mathbf{\text{Σ}}}}^{(2)}={\left({\mathit{X}}^{(2)}\right)}^{\top }{\mathit{X}}^{(2)}/\left(n_1/2\right)$. Define the projection vector ${\widehat{\text{u}}}_3$ as the solution to the following optimization algorithm

$${\widehat{\mathit{u}}}_3=\underset{\mathit{u}\in {\mathbf{R}}^p}{\arg \min }\left\{{\mathit{u}}^{\top }{\widehat{\mathbf{\text{Σ}}}}^{(2)}\mathit{u}:{\Vert {\widehat{\mathbf{\text{Σ}}}}^{(2)}\mathit{u}-\widehat{\mathit{\beta }}\Vert }_{\infty }\le \parallel \widehat{\mathit{\beta }}{\parallel }_2\frac{{\lambda }_1}{\sqrt{n_1/2}}\right\},$$ (15)

where ${\text{λ}}_1=12{\text{λ}}_{\max }^2(\mathbf{\text{Σ}})\sqrt{\log p}$. We then estimate $\langle \widehat{\mathit{\beta }},\mathit{\beta }-\widehat{\mathit{\beta }}\rangle$ by ${\widehat{\mathit{u}}}_3^{\top }{\left({\mathit{X}}^{(2)}\right)}^{\top }\left({\mathit{y}}^{(2)}-{\mathit{X}}^{(2)}\widehat{\mathit{\beta }}\right)/\left(n_1/2\right)$ and propose the final estimator of $\parallel \mathit{\beta }{\parallel }_2^2$ as

$$\widehat{\text{Q}}(\mathit{\beta })={\left(\parallel \widehat{\mathit{\beta }}{\parallel }_2^2+2{\widehat{\mathit{u}}}_3^{\top }\frac{1}{n_1/2}{\left({\mathit{X}}^{(2)}\right)}^{\top }\left({\mathit{y}}^{(2)}-{\mathit{X}}^{(2)}\widehat{\mathit{\beta }}\right)\right)}_{+}.$$ (16)

Similarly, the estimator of $\parallel \mathit{\text{γ}}{\parallel }_2^2$ is given by

$$\widehat{\text{Q}}(\mathit{\text{γ}})={\left(\parallel \widehat{\mathit{\text{γ}}}{\parallel }_2^2+2{\widehat{\mathit{u}}}_4^{\top }\frac{1}{n_2/2}{\left({\mathit{Z}}^{(2)}\right)}^{\top }\left({\mathit{w}}^{(2)}-{\mathit{Z}}^{(2)}\widehat{\mathit{\text{γ}}}\right)\right)}_{+},$$ (17)

where

$${\widehat{\mathit{u}}}_4=\underset{\mathit{u}}{\arg \min }\left\{{\mathit{u}}^{\top }{\widehat{\mathbf{\text{Γ}}}}^{(2)}\mathit{u}:{\Vert {\widehat{\mathbf{\text{Γ}}}}^{(2)}\mathit{u}-\widehat{\mathit{\gamma }}\Vert }_{\infty }\le \parallel \widehat{\mathit{\gamma }}{\parallel }_2\frac{{\lambda }_2}{\sqrt{n_2/2}}\right\},$$ (18)

with ${\widehat{\mathbf{\text{Γ}}}}^{(2)}={\left({\mathit{Z}}^{(2)}\right)}^{\top }\left({\mathit{Z}}^{(2)}\right)/\left(n_2/2\right)$ and ${\lambda }_2=12{\lambda }_{\max }^2(\mathbf{\Gamma })\sqrt{\log p}$.

Remark 3.

Sample splitting is used here for the purpose of the theoretical analysis. In the simulation study (Section 4), the performance of the proposed estimator without sample splitting is investigated; see Steps 5 – 7 in Table 1. The proposed estimator without sample splitting performs even better numerically than with sample splitting since more observations are used in constructing the initial estimators $\parallel \hat{\mathit{\beta }}{\parallel }_2^2$ and $\parallel \hat{\mathit{\gamma }}{\parallel }_2^2$ and the projection vectors ${\widehat{\mathit{u}}}_3$ and ${\widehat{\mathit{u}}}_4$.

2.3. Estimation of $\text{R}(\mathit{\beta },\mathit{\gamma })$

Given the estimators $\widehat{\text{I}}(\mathit{\text{β}},\mathit{\text{γ}})$, $\widehat{\text{Q}}(\mathit{\text{β}})$ and $\widehat{\text{Q}}(\mathit{\text{γ}})$ constructed in Sections 2.1 and 2.2, a natural estimator for the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$ is given by

$$\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })=sign(\widehat{\text{I}}(\mathit{\beta },\mathit{\gamma }))\cdot \min \left\{\frac{\mid \widehat{\text{I}}(\mathit{\beta },\mathit{\gamma })\mid }{\sqrt{\widehat{\text{Q}}(\mathit{\beta })\widehat{\text{Q}}(\mathit{\gamma })}}1\{\widehat{\text{Q}}(\mathit{\beta })\widehat{\text{Q}}(\mathit{\gamma })>0\},1\right\},$$ (19)

where $\widehat{\text{I}}(\mathit{\text{β}},\mathit{\text{γ}})$, $\widehat{\text{Q}}(\mathit{\text{β}})$ and $\widehat{\text{Q}}(\mathit{\text{γ}})$ are estimators of $\langle \mathit{\beta },\mathit{\text{γ}}\rangle$, $\parallel \mathit{\beta }{\parallel }_2^2$ and $\parallel \mathit{\text{γ}}{\parallel }_2^2$ defined in (12), (16) and (17), respectively. It is possible that one of $\widehat{\text{Q}}(\mathit{\text{β}})$ and $\widehat{\text{Q}}(\mathit{\text{γ}})$ is 0 if $\parallel \mathit{\text{β}}{\parallel }_2^2$ and $\parallel \mathit{\text{γ}}{\parallel }_2^2$ are close to zero. In this case, the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$ is estimated as 0. Since $\text{R}(\mathit{\beta },\mathit{\gamma })$ is always between −1 and 1, the estimator $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$ is truncated to ensure that it is within the range. The FDE algorithm without sample splitting for calculating the estimators $\widehat{\text{I}}(\mathit{\text{β}},\mathit{\text{γ}})$, $\widehat{\text{Q}}(\mathit{\text{β}})$, $\widehat{\text{Q}}(\mathit{\text{γ}})$, and $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$ is detailed in Table 1.

3. Theoretical Analysis

3.1. Upper Bound Analysis

The samples sizes $n_1$ and $n_2$ are assumed to be of the same order, that is $n_1\asymp n_2$. Let $n=\text{min}\{n_1,n_2\}$ be the smallest of two sample sizes. The following assumptions are introduced to facilitate the theoretical analysis.

(A1) The population covariance matrices $\mathbf{\text{Σ}}$ and $\mathbf{\text{Γ}}$ satisfy $1/M_1\le {\lambda }_{\min }(\mathbf{\text{Σ}})\le {\lambda }_{\max }(\mathbf{\text{Σ}})\le M_1$, and $1/M_1\le {\text{λ}}_{\min }(\mathbf{\text{Γ}})\le {\text{λ}}_{\max }(\mathbf{\text{Γ}})\le M_1$, where $M_1\ge 1$ is a positive constant. The random design matrix $\mathit{\text{X}}$ is assumed to be independent of the other random design matrix $\mathit{\text{Z}}$. The noise levels ${\sigma }_1$ and ${\sigma }_2$ satisfy $\text{max}\{{\sigma }_1,{\sigma }_2\}\le M_2$, where $M_2>0$ is a positive constant.

(A2) The ${\ell }_2$ norms of the coefficient vectors $\mathit{\text{β}}$ and $\mathit{\text{γ}}$ are bounded away from zero in the sense that

$$\min \left\{\parallel \mathit{\beta }{\parallel }_2,\parallel \mathit{\gamma }{\parallel }_2\right\}\ge {\eta }_0\ge C\sqrt{k\log p/n},\text{where}k=\max \left\{\parallel \mathit{\beta }{\parallel }_0,\parallel \mathit{\gamma }{\parallel }_0\right\}.$$ (20)

Assumption (A1) places a condition on the spectrum of the covariance matrices $\mathbf{\text{Σ}}$ and $\mathbf{\text{Γ}}$ and an upper bound on the noise levels ${\sigma }_1$ and ${\sigma }_2$. Assumption (A2) requires that the total strengths of the signals have to be bounded away from zero by ${\eta }_0$, which is only used in the upper bound analysis of the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$.

The following theorem establishes the convergence rates of the estimators $\widehat{\text{I}}(\mathit{\text{β}},\mathit{\text{γ}})$, $\widehat{\text{Q}}(\mathit{\text{β}})$, and $\widehat{\text{Q}}(\mathit{\text{γ}})$, proposed in (12), (16) and (17), respectively.

Theorem 1.

Suppose the assumption (A1) holds and $k=\max \left\{\parallel \mathit{\beta }{\parallel }_0,\parallel \mathit{\gamma }{\parallel }_0\right\}\le cn/\log p$ for some $c>0$. Then for any fixed constant $0<\alpha <1/4$, with probability at least $1-4\alpha -p^{-c_0}$, we have

$$|\widehat{\text{I}}(\mathit{\beta },\mathit{\gamma })-\text{I}(\mathit{\beta },\mathit{\gamma })|\lesssim \left(\parallel \mathit{\beta }{\parallel }_2+\parallel \mathit{\gamma }{\parallel }_2\right)\left(\frac{z_{\alpha /2}}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{k\log p}{n},$$ (21)

$$|\widehat{\text{Q}}(\mathit{\beta })-\text{Q}(\mathit{\beta })|\lesssim \parallel \mathit{\beta }{\parallel }_2\left(\frac{z_{\alpha /2}}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{k\log p}{n},$$ (22)

$$|\widehat{\text{Q}}(\mathit{\text{γ}})-\text{Q}(\mathit{\text{γ}})|\lesssim \parallel \mathit{\text{γ}}{\parallel }_2\left(\frac{z_{\alpha /2}}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{k\log p}{n},$$ (23)

where $c_0$ is a positive constant.

The upper bound of estimating $\langle \mathit{\beta },\mathit{\text{γ}}\rangle$ not only depends on $k$, $n$ and $p$, but also scales to the signal strengths $\parallel \mathit{\beta }{\parallel }_2$ and $\parallel \mathit{\text{γ}}{\parallel }_2$. For the estimation of the quadratic functional $\text{Q}(\mathit{\text{β}})$ (or $\text{Q}(\mathit{\text{γ}})$), the convergence rate depends on $\parallel \mathit{\beta }{\parallel }_2$ (or $\parallel \mathit{\text{γ}}{\parallel }_2$). The following theorem establishes the convergence rate of the estimator $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$ proposed in (19).

Theorem 2.

Suppose the assumptions (A1) and (A2) hold and $k=\max \left\{\parallel \mathit{\beta }{\parallel }_0,\parallel \mathit{\gamma }{\parallel }_0\right\}\le cn/\log p$ for some $c>0$. Then for any fixed constant $0<\alpha <1/4$, with probability at least $1-4\alpha -p^{-c_0}$, we have

$$|\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })-\text{R}(\mathit{\beta },\mathit{\gamma })|\lesssim \frac{1}{{\eta }_0}\left(\frac{z_{\alpha /2}}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{1}{{\eta }_0^2}\frac{k\log p}{n},$$ (24)

where $c_0$ is a positive constant.

In contrast to Theorem 1, Theorem 2 requires the extra assumption (A2) on the signal strengths $\parallel \mathit{\beta }{\parallel }_2$ and $\parallel \mathit{\text{γ}}{\parallel }_2$. The convergence rate of estimating $\text{R}(\mathit{\beta },\mathit{\gamma })$ is scaled to the inverse of the signal strength, $1/{\eta }_0$. This is different from the error bound in Theorem 1, where the estimation accuracy is scaled to the signal strength. The lower bound results established in Theorem 3 will demonstrate the necessity of Assumption (A2) for estimation of $\text{R}(\mathit{\beta },\mathit{\gamma })$.

3.2. Minimax Lower Bounds

This section establishes the minimax lower bounds of estimating $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, $\text{Q}(\mathit{\text{β}})$, $\text{Q}(\mathit{\text{γ}})$ and $\text{R}(\mathit{\beta },\mathit{\gamma })$. We first introduce parameter spaces for $\mathit{\theta }=\left(\mathit{\beta },\mathbf{\text{Σ}},{\sigma }_1,\mathit{\gamma },\mathbf{\text{Γ}},{\sigma }_2\right)$, which is defined as the product of parameter spaces for ${\mathit{\text{θ}}}_{\mathit{\text{β}}}=(\mathit{\text{β}},\mathbf{\text{Σ}},{\sigma }_1)$ and ${\mathit{\text{θ}}}_{\mathit{\gamma }}=(\mathit{\text{γ}},\mathbf{\text{Γ}},{\sigma }_2)$. We define the following parameter space for both ${\mathit{\text{θ}}}_{\mathit{\text{β}}}=(\mathit{\text{β}},\mathbf{\text{Σ}},{\sigma }_1)$ and ${\mathit{\text{θ}}}_{\mathit{\text{γ}}}=(\mathit{\text{γ}},\mathbf{\text{Γ}},{\sigma }_2)$,

$$𝓖\left(k,M_0\right)=\left\{\left(\mathit{\beta },\mathbf{\text{Σ}},{\sigma }_1\right):\parallel \mathit{\beta }{\parallel }_0\le k,\parallel \mathit{\text{β}}{\parallel }_2\le M_0,\frac{1}{M_1}\le {\text{λ}}_{\min }(\mathbf{\text{Σ}})\le {\text{λ}}_{\max }(\mathbf{\text{Σ}})\le M_1,{\sigma }_1\le M_2\right\},$$ (25)

where $M_1\ge 1$ and $M_2>0$ are positive constants. The parameter space defined in (25) requires that the signal $\beta$ contains less than $k$ non-zero coefficients and the ${\ell }_2$ norm $\parallel \beta {\parallel }_2$ is upper bounded by $M_0$, where $M_0$ is allowed to grow with $n$ and $p$. The lower bound results in Theorem 3 show that the estimation difficulties of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, $\text{Q}(\mathit{\text{β}})$ and $\text{Q}(\mathit{\text{γ}})$ depend on $M_0$. The other conditions $1/M_1\le {\text{λ}}_{\text{min}}(\mathbf{\text{Σ}})\le {\text{λ}}_{\text{max}}(\mathbf{\text{Σ}})\le M_1$ and ${\sigma }_1\le M_2$ are regularity conditions. Based on the definition (25), the parameter space for $(\mathit{\text{β}},\mathbf{\text{Σ}},{\sigma }_1,\mathit{\text{γ}},\mathbf{\text{Γ}},{\sigma }_2)$ is defined as a product of two parameter spaces,

$$\text{Θ}\left(k,M_0\right)=\left\{\mathit{\theta }=\left(\mathit{\beta },\mathbf{\text{Σ}},{\sigma }_1,\mathit{\gamma },\mathbf{\text{Γ}},{\sigma }_2\right):\left(\mathit{\beta },\mathbf{\text{Σ}},{\sigma }_1\right)\in 𝓖\left(k,M_0\right),\left(\mathit{\gamma },\mathbf{\text{Γ}},{\sigma }_2\right)\in 𝓖\left(k,M_0\right)\right\}.$$ (26)

For establishing optimal bounds of$\text{R}(\mathit{\beta },\mathit{\gamma })$, we define the following parameter space

$$\text{Θ}\left(k,{\eta }_0\right)=\left\{\mathit{\theta }=\left(\mathit{\beta },\mathbf{\text{Σ}},{\sigma }_1,\mathit{\gamma },\mathbf{\text{Γ}},{\sigma }_2\right):\left(\mathit{\beta },\mathbf{\text{Σ}},{\sigma }_1\right)\in 𝓖\left(k,{\eta }_0\right),\left(\mathit{\gamma },\mathbf{\text{Γ}},{\sigma }_2\right)\in 𝓖\left(k,{\eta }_0\right)\right\},$$ (27)

where

$$𝓖\left(k,{\eta }_0\right)=\left\{\left(\mathit{\beta },\mathbf{\text{Σ}},{\sigma }_1\right):\parallel \mathit{\beta }{\parallel }_0\le k,\parallel \mathit{\beta }{\parallel }_2\ge {\eta }_0,\frac{1}{M_1}\le {\text{λ}}_{\min }(\mathbf{\text{Σ}})\le {\text{λ}}_{\max }(\mathbf{\text{Σ}})\le M_1,{\sigma }_1\le M_2\right\},$$

with ${\eta }_0\ge 0$. In contrast to the parameter space $𝓖\left(k,M_0\right)$ where $\parallel \beta {\parallel }_2$ is upper bounded by $M_0$, the parameter space $𝓖\left(k,{\eta }_0\right)$ requires the signal strength $\parallel \beta {\parallel }_2$ to be lower bounded by ${\eta }_0$, where ${\eta }_0$ is allowed to grow with $n$ and $p$. The lower bound in Theorem 3 shows that the estimation difficulty of $\text{R}(\mathit{\beta },\mathit{\gamma })$ depends on ${\eta }_0$.

The following theorem establishes the minimax lower bounds for the convergence rates of estimating the inner product $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, the quadratic functionals $\text{Q}(\mathit{\text{β}})$ and $\text{Q}(\mathit{\text{γ}})$ and the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$.

Theorem 3.

Suppose $k\le c\min \left\{n/\log p,p^{\nu }\right\}$ for some constants $c>0$ and $0\le \nu <\frac{1}{2}$. Then

$$\underset{\tilde{\text{I}}}{\inf }\underset{\mathit{\theta }\in \text{Θ}\left(k,M_0\right)}{\sup }{\text{ℙ}}_{\mathit{\theta }}\left(|\tilde{\text{I}}-\text{I}(\mathit{\beta },\mathit{\gamma })|\gtrsim \min \left\{M_0\left(\frac{1}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{k\log p}{n},M_0^2\right\}\right)\ge \frac{1}{4},$$ (28)

$$\underset{\tilde{\text{Q}}}{\inf }\underset{{\mathit{\theta }}_{\mathit{\beta }}\in 𝓖\left(k,M_0\right)}{\sup }{\text{ℙ}}_{{\mathit{\theta }}_{\mathit{\beta }}}\left(|\tilde{\text{Q}}-\text{Q}(\mathit{\beta })|\gtrsim \min \left\{M_0\left(\frac{1}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{k\log p}{n},M_0^2\right\}\right)\ge \frac{1}{4},$$ (29)

$$\underset{\tilde{\text{Q}}}{\inf }\underset{{\mathit{\theta }}_{\mathit{\gamma }}\in 𝓖\left(k,M_0\right)}{\sup }{\text{ℙ}}_{{\mathit{\theta }}_{\mathit{\gamma }}}\left(|\tilde{\text{Q}}-\text{Q}(\mathit{\gamma })|\gtrsim \min \left\{M_0\left(\frac{1}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{k\log p}{n},M_0^2\right\}\right)\ge \frac{1}{4},$$ (30)

$$\underset{\tilde{\text{R}}}{\inf }\underset{\mathit{\theta }\in \text{Θ}\left(k,{\eta }_0\right)}{\sup }{\text{ℙ}}_{\mathit{\theta }}\left(|\tilde{\text{R}}-\text{R}(\mathit{\beta },\mathit{\gamma })|\gtrsim \min \left\{\frac{1}{{\eta }_0}\left(\frac{1}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{1}{{\eta }_0^2}\frac{k\log p}{n},1\right\}\right)\ge \frac{1}{4}.$$ (31)

Remark 4.

Estimation of quadratic functionals has been extensively studied in the classical Gaussian sequence model. See, for example, Donoho and Nussbaum (1990); Efromovich and Low (1996); Laurent and Massart (2000); Cai and Low (2005, 2006); Collier et al. (2015) for details. In the regime $k\le c\text{min}\{n/\text{log}p,p^{\nu }\}$ for some constants $c>0$ and $0<v<\frac{1}{2}$, Theorem 2 in (Collier et al., 2015) gives a lower bound, $\min \left\{M_0/\sqrt{n}+k\log p/n,M_0^2\right\}$, for estimating ${\parallel \beta \parallel }_2^2$ in the sequence model. In contrast, an extra term $M_{0}k\text{log}p/n$ appears in the lower bound given in (29) for estimating ${\parallel \beta \parallel }_2^2$ in high-dimensional linear regression. One intuitive reason for this extra term is that high-dimensional linear regression is involved with an extra inverse process than Gaussian sequence model. Estimation of the quadratic functional ${\parallel \beta \parallel }_2^2$ in high-dimensional linear regression is fundamentally harder than that in the Gaussian sequence model. For the high-dimensional linear regression, the estimation lower bound $k\text{log}p/n$ in (29) can also be established by the general lower bounds developed in Cai and Guo (2017a). See Section 8 in Cai and Guo (2017a) for details.

3.3. Optimality of FDEs

In this section, we establish the optimality of FDEs by combining Theorems 1 and 2 over the parameter spaces $\text{Θ}(k,M_0)$ and $\text{Θ}(k,{\eta }_0)$ defined in (26) and (27), respectively.

Corollary 1.

Suppose $k\le c\text{min}\{n/\text{log}p,p^{\nu }\}$ and $M_0\ge {\eta }_0\ge C\sqrt{k\log p/n}$ for some constants $C$, $c>0$ and $0\le \nu <\frac{1}{2}$. Then

$$\underset{\mathit{\theta }\in \text{Θ}\left(k,M_0\right)}{\sup }{\text{ℙ}}_{\mathit{\theta }}\left(|\widehat{\text{I}}-\text{I}(\mathit{\beta },\mathit{\gamma })|\gtrsim M_0\left(\frac{1}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{k\log p}{n}\right)\ge 1-4\alpha -p^{-c_0},$$ (32)

$$\underset{{\mathit{\theta }}_{\mathit{\beta }}\in 𝓖\left(k,M_0\right)}{\sup }{\text{ℙ}}_{{\mathit{\theta }}_{\mathit{\beta }}}\left(|\widehat{\text{Q}}-\text{Q}(\mathit{\beta })|\gtrsim M_0\left(\frac{1}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{k\log p}{n}\right)\ge 1-4\alpha -p^{-c_0},$$ (33)

$$\underset{{\mathit{\theta }}_{\mathit{\gamma }}\in 𝓖\left(k,M_0\right)}{\sup }{\text{ℙ}}_{{\mathit{\theta }}_{\mathit{\gamma }}}\left(|\widehat{\text{Q}}-\text{Q}(\mathit{\gamma })|\gtrsim M_0\left(\frac{1}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{k\log p}{n}\right)\ge 1-4\alpha -p^{-c_0},$$ (34)

$$\underset{\mathit{\theta }\in \text{Θ}\left(k,{\eta }_0\right)}{\sup }{\text{ℙ}}_{\mathit{\theta }}\left(|\widehat{\text{R}}-\text{R}(\mathit{\beta },\mathit{\gamma })|\gtrsim \frac{1}{{\eta }_0}\left(\frac{1}{\sqrt{n}}+\frac{k\log p}{n}\right)+\frac{1}{{\eta }_0^2}\frac{k\log p}{n}\right)\ge 1-4\alpha -p^{-c_0}.$$ (35)

where $0<\alpha <1/4$ and $c_0$ is a positive constant.

Combined with Theorem 3, Corollary 1 implies that, for $M_0\ge C\sqrt{k\log p/n}$, the estimators $\widehat{\text{I}}(\mathit{\beta },\mathit{\gamma })$, $\widehat{\text{Q}}(\mathit{\beta })$, $\widehat{\text{Q}}(\mathit{\gamma })$ proposed in (12),(16) and (17) achieve the minimax lower bounds (28), (29) and (30) within a constant factor, that is, FDEs are minimax rate-optimal. On the other hand, if $M_0\ll \sqrt{k\log p/n}$, estimation of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, $\text{Q}(\mathit{\text{β}})$ and $\text{Q}(\mathit{\text{γ}})$ is uninteresting as the trivial estimator 0 achieves the minimax lower bound in this case. For estimation of $\text{R}(\mathit{\beta },\mathit{\gamma })$, under the assumption ${\eta }_0\ge C\sqrt{k\log p/n}$, Corollary 1 shows that the estimator $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$ given in (19) achieves the minimax lower bound $1/{\eta }_0\times (1/\sqrt{n}+k\log p/n)+1/{\eta }_0^2\times k\log p/n$ in (31). Hence $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$ is the rate-optimal estimator of $\text{R}(\mathit{\beta },\mathit{\gamma })$ under the assumption (A2). When ${\eta }_0\ll \sqrt{k\log p/n}$, estimation of $\text{R}(\mathit{\beta },\mathit{\gamma })$ becomes trivial as the simple estimator 0 attains the minimax lower bound. This demonstrates the necessity of the assumption (A2) in Theorem 2.

4. Simulation Evaluations and Comparisons

We compare the finite-sample performance of several estimators of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ and $\text{R}(\mathit{\beta },\mathit{\gamma })$ using simulations. These estimators included plug-in scaled Lasso estimator (Sun and Zhang, 2012), plug-in de-biased estimator (Javanmard and Montanari, 2014; van de Geer et al., 2014; Zhang and Zhang, 2014), plug-in thresholded estimator (Zhang and Zhang, 2014, Section 3.3) and the proposed estimator FDE. Specifically, they are defined as

• FDE: The inner product $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ is estimated by $\widehat{\text{I}}(\mathit{\text{β}},\mathit{\text{γ}})$ in (12) and the ratio $\text{R}(\mathit{\beta },\mathit{\gamma })$ is estimated by $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$ in (19). We consider FDE with sample splitting (FDE-S) and without sample splitting (FDE-NS) for $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$

• Plug-in scaled Lasso estimator (Lasso): The inner product $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ is estimated by $\langle \widehat{\mathit{\text{β}}},\widehat{\mathit{\text{γ}}}\rangle$ and the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$ is estimated by

  $$\left[\langle \widehat{\mathit{\beta }},\widehat{\mathit{\text{γ}}}\rangle /\left(\parallel \widehat{\mathit{\beta }}{\parallel }_2\parallel \widehat{\mathit{\text{γ}}}{\parallel }_2\right)\right]\mathbf{1}\left\{\parallel \widehat{\mathit{\beta }}{\parallel }_2\parallel \widehat{\mathit{\text{γ}}}{\parallel }_2>0\right\}.$$

• Plug-in de-biased estimator (De-biased): Denote the de-biased Lasso estimators as $\widetilde{\mathit{\text{β}}}$ and $\widetilde{\mathit{\text{γ}}}$. The inner product $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ is estimated by $\langle \tilde{\mathit{\beta }},\widetilde{\mathit{\text{γ}}}\rangle$ and the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$ is estimated by $\left[\langle \tilde{\mathit{\beta }},\widetilde{\mathit{\text{γ}}}\rangle /\left(\parallel \tilde{\mathit{\beta }}{\parallel }_2\parallel \widetilde{\mathit{\text{γ}}}{\parallel }_2\right)\right]\mathbf{1}\{\parallel \tilde{\mathit{\beta }}{\parallel }_2\parallel \widetilde{\mathit{\text{γ}}}{\parallel }_2>0\}$.

• Plug-in thresholded estimator (Thresholded): Denote the thresholded estimators as $\overline{\mathit{\text{β}}}$ and $\overline{\mathit{\text{γ}}}$. The inner product $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ is estimated by $\langle \tilde{\mathit{\beta }},\widetilde{\mathit{\text{γ}}}\rangle$ and the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$ is estimated by $\left[\langle \overline{\mathit{\beta }},\overline{\mathit{\text{γ}}}\rangle /\left(\parallel \overline{\mathit{\beta }}{\parallel }_2\parallel \overline{\mathit{\text{γ}}}{\parallel }_2\right)\right]\mathbf{1}\left\{\parallel \overline{\mathit{\beta }}{\parallel }_2\parallel \overline{\mathit{\text{γ}}}{\parallel }_2>0\right\}$.

Implementation of the de-biased, thresholded and FDE estimators requires the scaled Lasso estimators $\widehat{\mathit{\text{β}}}$ and $\widehat{\mathit{\text{γ}}}$ in the initial step. The scaled Lasso estimator is implemented by the equivalent square-root Lasso algorithm (Belloni et al., 2011). The theoretical tuning parameter is ${\text{λ}}_0=\sqrt{2.01\log p/n}$, which may be conservative in the numerical studies. Instead, the tuning parameter is chosen as ${\text{λ}}_0=b\sqrt{2.01\log p}$. However, the performances of all estimators are evaluated across a grid of tuning parameter values $b\in \{.25,.5,.75,1\}$ (see Supplementary Material, Section A.1). The results showed that b = .5 was a good choice for all the estimators. Hence, ${\lambda }_0=.5\sqrt{2.01\log p/n}$ was used for the numerical studies in this section and Section 5. To implement the FDE algorithm, the other tuning parameter $\text{λ}$ is chosen as $\sqrt{2.01\log p/n}$ for the correction Steps 2,3,5 and 6 in Table 1.

Comparisons of estimates of $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ and $\text{R}(\mathit{\beta },\mathit{\gamma })$ are presented below. Results on estimating the quadratic functionals are presented in the Supplementary Material, Section A.2. For each setting, with the parameters $(p,n_1,n_2,s,s_1,s_2)$, $\mathbf{\text{Σ}}$, $\mathbf{\text{Γ}}$, $F_{\mathit{\beta }}$, $F_{\mathit{\gamma }}$ specified, we generate the data and compare different methods as follows,

1. Generate sets $S_1\subset [p]$ and $S_2\subset [p]$, with $|S_1|=s_1$, $|S_2|=s_2$ and $|S_1\cap S_2|=s$. For $\mathit{\text{β}}\in {\mathbf{R}}^p$ and $\mathit{\text{γ}}\in {\mathbf{R}}^p$, generate ${\mathit{\beta }}_j\sim F_{\mathit{\beta }}$ and ${\mathit{\gamma }}_l\sim F_{\mathit{\gamma }}$, for $j\in S_1$, $l\in S_2$, and set ${\mathit{\beta }}_j=0$ and ${\mathit{\gamma }}_l=0$, for $j\notin S_1$, $l\notin S_2$.

2. Generate ${\mathit{X}}_i\stackrel{\text{i.i.d}}{\sim }N(\mathbf{0},\mathbf{\text{Σ}})$, $1\le i\le n_1$, and ${\mathit{\text{Z}}}_i\stackrel{\text{i.i.d}}{\sim }N(\mathbf{0},\mathbf{\text{Γ}})$, $1\le i\le n_2$.

3. Generate the noise ${\mathit{ϵ}}_i\stackrel{\text{i.i.d}}{\sim }N(0,1)$, $1\le i\le n_1$, and ${\mathit{\delta }}_i\stackrel{\text{i.i.d}}{\sim }N(0,1)$, $1\le i\le n_2$. Generate the outcome as $\mathit{\text{y}}=\mathit{\text{Xβ}}+\mathit{ϵ}$ and $\mathit{\text{w}}=\mathit{\text{Zγ}}+\mathit{\text{δ}}$.

4. With $\mathit{\text{X}}$, $\mathit{\text{y}}$, $\mathit{\text{Z}}$, and $\mathit{\text{w}}$, estimate $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ and $\text{R}(\mathit{\beta },\mathit{\gamma })$ through different estimators.

5. Repeat 2–4 for $L$ times.

We evaluate the performance of an estimator by the mean squared error (MSE), which is defined as

$$\text{MSE}(\widehat{\text{T}})=\frac{1}{L}\sum _{l=1}^L{(\widehat{\text{T}}(\mathit{X},\mathit{y},\mathit{Z},\mathit{w};l)-\text{T})}^2,$$ (36)

for a given quantity T and its estimate $\widehat{\text{T}}(\mathit{X},\mathit{y},\mathit{Z},\mathit{w};l)$ from l-th replication. We consider two different settings with two sets of parameters $(p,n_1,n_2,s,s_1,s_2)$, $\mathbf{\text{Σ}}$, $\mathbf{\text{Γ}}$, $F_{\mathit{\beta }}$ and $F_{\mathit{\gamma }}$ and the simulation for each setting is repeated L = 300 times.

Experiment 1.

The parameters are set as follows, $(p,n_1,n_2)=(600,400,400)$, the sparsity parameters $(s,s_1,s_2)=(15,30,25)$, and the covariance matrices $\mathbf{\text{Σ}}$ and $\mathbf{\text{Γ}}$ satisfy ${\mathbf{\text{Σ}}}_{ij}={\mathbf{\text{Γ}}}_{ij}={(0.8)}^{|i-j|}$. For given positive values ${\tau }_1$ and ${\tau }_2$, the signals of $\mathit{\text{β}}$ satisfy that ${\mathit{\beta }}_{ji}=(1+i/s_1){\tau }_1/2$, for $j_i\in S_1$, $i=1,2,···,s_1$, and the signals in $\mathit{\text{γ}}$ satisfy that ${\mathit{\text{γ}}}_j={\tau }_2$ for $j\in S_2$. This simulation aims to investigate the case where the coefficients for one regression are much larger than the other by varying the signal strength parameters as $({\tau }_1,{\tau }_2)\in \text{{(3.0,.1),(2.6,.2),(2.2,.3),(1.8,.4),(.1,1.6),(.2,1.4),(.3,1.2),(.4,1.0)}}$.

The results are summarized in Table 2. For all combinations of the signal strength parameters, in terms of estimating the inner product $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, FDE consistently outperformed the plug-in estimates with Lasso and Thresholded Lasso. Moreover, with increasing difference between ${\tau }_1$ and ${\tau }_2$, the advantage of FDE over the plugin estimate using Lasso or thresholded Lasso became larger. The same results were observed for estimation of the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$, where FDE-NS had consistent better performance than other methods. Although De-biased performed well in terms of estimating $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, it performed much worse than FDE-NS for estimating $\text{R}(\mathit{\beta },\mathit{\gamma })$.

Table 2:

Mean square errors (MSE) of the estimates of the inner product $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ and the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$ for various signal strength parameters. Lasso: plugin estimator with the scaled Lasso estimator; De-biased: plug-in estimator with the de-biased estimator; Thresholded: plug-in estimator with the thresholded estimator; FDE: the proposed estimator $\widehat{\text{I}}(\mathit{\text{β}},\mathit{\text{γ}})$; FDE-S: the proposed estimator $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$ with sample splitting; FDE-NS: the proposed estimator $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$ without sample splitting.

		Strength parameters, $({\tau }_1,{\tau }_2)$
		(1.8, .4)	(2.2, .3)	(2.6, .2)	(3, .1)	(.1, 1.6)	(.2, 1.4)	(.3, 1.2)	(.4, 1)
$\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$	Truth	8.088	7.414	5.841	3.370	1.797	3.145	4.044	4.493
MSE
	Lasso	9.295	11.564	12.560	7.279	2.377	4.889	5.409	4.800
	De-biased	1.733	2.191	2.324	1.386	.449	.838	.985	.886
	Thresholded	2.029	3.377	6.463	5.789	1.877	3.024	2.432	1.546
	FDE	1.847	2.471	2.662	2.118	.734	.995	1.028	.986
$\text{R}(\mathit{\text{β}},\mathit{\text{γ}})$	Truth	.5314	.5314	.5314	.5314	.5314	.5314	.5314	.5314
MSE
	Lasso	.0023	.0075	.0332	.1260	.1574	.0624	.0227	.0087
	De-biased	.0208	.0415	.0864	.1590	.1736	.1068	.0627	.0373
	Thresholded	.0045	.0139	.0585	.1753	.0964	.0981	.0389	.0153
	FDE-S	.0337	.0303	.0621	.0678	.2130	.1199	.0694	.0616
	FDE-NS	.0036	.0064	.0163	.0580	.0892	.0237	.0116	.0061

As discussed in Section 2, the sample splitting of estimating the normalized inner product is simply proposed to facilitate the theoretical analysis and might not be necessary for the algorithm. Our simulation results indicated that the proposed estimator without sample splitting (FDE-NS) performed quite well in all settings, even better than FDE-S, due to the fact that more samples were used for estimation and correction steps. Such observations led us to use the proposed estimator without sample splitting (FDE-NS) in the real data analysis in Section 5.

Experiment 2.

The parameters are set as follows, $(p,n_1,n_2)=(800,400,400)$, signal strength parameters $({\tau }_1,{\tau }_2)=(.2,.1)$ and the covariance matrices $\mathbf{\text{Σ}}$ and $\mathbf{\text{Γ}}$ satisfy ${\mathbf{\text{Σ}}}_{ij}={\mathbf{\text{Γ}}}_{ij}={(0.8)}^{|i-j|}$. The signals of $\mathit{\text{β}}$ follow that ${\mathit{\text{β}}}_{ji}=(1+i/s_1){\tau }_1/2$, for $j_i\in S_1$, $i=1,2,···,s_1$, and the signals in $\mathit{\text{γ}}$ satisfy that ${\mathit{\text{γ}}}_j={\tau }_2$ for $j\in S_2$. This simulation setting is set to investigate the relationship between the performance of estimators and the signal sparsity level and vary the sparsity levels of $\mathit{\text{β}}$ and $\mathit{\text{γ}}$ as $(s_2,s_2)\in \left\{\right(40,40),(50,50),(60,60\mathrm{)},(70,70),(80,80),(90,90),(100,100),(110,110)\}$, and fix the number of common signals at $s=20$. Since the number of the associated variants is very large for both coefficient vectors, large values of ${\tau }_1$ and ${\tau }_2$ would induce strong signals such that all the methods perform well. Instead, we consider a more challenging setting where the signal magnitude is small, that is ${\tau }_1=.2$ and ${\tau }_2=.1$.

The results are summarized in Table 3. Clearly, FDE outperformed the other methods. When the signals became denser, the improvement of FDE over other methods was more pronounced. For estimation of $\text{R}(\mathit{\beta },\mathit{\gamma })$, the results showed that FDE-NS consistently outperformed other estimators. As the number of signals increased, the MSE corresponding to FDE-NS decreased quickly.

Table 3:

Mean square errors (MSE) of the estimates of the inner product $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ and the normalized inner product $\text{R}(\mathit{\beta },\mathit{\gamma })$ for various sparsity parameters. Lasso: plug-in estimator with the scaled Lasso estimator; De-biased: plug-in estimator with the de-biased estimator; Thresholded: plug-in estimator with the thresholded estimator; FDE: the proposed estimator $\widehat{\text{I}}(\mathit{\text{β}},\mathit{\text{γ}})$; FDE-S: the proposed estimator $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$ with sample splitting; FDE-NS: the proposed estimator $\widehat{\text{R}}(\mathit{\beta },\mathit{\gamma })$ without sample splitting.

		Sparsity parameter, $s_1(s_2=s_1)$
		40	50	60	70	80	90	100	110
$\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$	Truth	.190	.170	.219	.212	.179	.221	.183	.221
MSE
	Lasso	.032	.025	.039	.036	.024	.035	.023	.028
	De-biased	.015	.015	.018	.017	.024	.027	.040	.066
	Thresholded	.027	.021	.031	.029	.020	.025	.018	.018
	FDE	.020	.014	.021	.022	.011	.013	.008	.008
$\text{R}(\mathit{\beta },\mathit{\gamma })$	Truth	.4027	.2908	.3122	.2592	.1914	.2110	.1573	.1725
MSE
	Lasso	.1157	.0517	.0539	.0370	.0166	.0180	.0097	.0079
	De-biased	.1267	.0601	.0659	.0411	.0160	.0173	.0063	.0059
	Thresholded	.1392	.0687	.0732	.0504	.0262	.0277	.0155	.0142
	FDE-S	.1154	.1225	.0779	.0574	.0456	.0499	.0450	.0493
	FDE-NS	.0847	.0340	.0368	.0294	.0115	.0091	.0055	.0047

5. Genetic Relatedness Yeast Colony Growths of Based on Genome Wide Association Data

Bloom et al. (2013) reported a large scale genome-wide association study of 46 quantitative traits based on 1,008 Saccharomyces cerevisiae segregants crossbred from a laboratory strain and a wine strain. The data set included 11,623 unique genotype markers. Since many of these markers are highly correlated and differ only in a few samples, Bloom et al. (2013) further selected a set of 4,410 markers that are weakly dependent based on the linkage disequilibrium information. Specifically, these markers were selected by picking one marker closest to each centimorgan position on the genetic map. The maker genotypes are coded as 1 or −1, according to which strain it came from and satisfy the sub-Gaussian conditions. The traits of interest were the end-point colony size normalized by the control growth under 46 different growth media, including Hydrogen Peroxide, Diamide, Calcium, Yeast Nitrogen Base (YNB) and Yeast extract Peptone Dextrose (YPD), etc. Bloom et al. (2013) showed that the genetic variants are associated with many of such trait values. It is therefore important to genetic relatedness among these related traits.

To demonstrate the genetic relatedness among these traits, eight traits were considered, including the normalized colony sizes under Calcium Chloride (Calcium), Diamide, Hydrogen Peroxide (Hydrogen), Paraquat, Raffinose, 6 Azauracil (Azauracil), YNB, and YPD. Each trait was normalized to have variance 1, so the quadratic norm represents the total genetic effects for each trait and an estimate the heritability. FDE was applied to every pair of these 8 traits without sample splitting, for a total of 28 pairs. The results are summarized in Table 4, including estimates of the heritability, genetic covariance and and genetic correlation for each of the 28 pairs. The genetic heritability of these traits ranged from 0.22 for Raffinose to 0.67 for YPD. About two thirds of these pairs had an estimated genetic correlation smaller than 0.1, indicating relatively weak genetic correlations among these traits.

Table 4:

FDE estimation for the heritability (bold diagonals), genetic covariance(upper diagonals) and genetic correlation (lower diagonals) among for each pair of 8 colony growth traits of the yeast segregants.

Traits	Calcium	Diamide	Hydrogen	Paraquat	Raffinose	Azauracil	YNB	YPD
Calcium	.3314	−.0189	−.1003	.0084	.0927	.0095	.0656	.−0134
Diamide	−.0286	.4390	.0598	−.0039	.0500	.0446	−.0159	.0803
Hydrogen	−.1579	.0942	.4033	.0576	−.1040	.0601	.0672	.0637
Paraquat	.0117	−.0053	.0799	.5199	.0023	.0365	.1148	.1029
Raffinose	.1972	.1065	−.2213	.0049	.2208	.0137	.0830	.0331
Azauracil	.0172	.0809	.1089	.0661	.0248	.3045	−.0259	.0703
YNB	.0968	−.0235	.0991	.1693	.1224	−.0383	.4594	.4246
YPD	−.0164	.0983	.0779	.1259	.0405	.0860	.5195	.6680

To further demonstrate the genetic relatedness among these pairs, for each trait, a Z-score was calculated based on regressing the trait value $\mathit{\text{y}}$ on genetic genetic marker ${\mathit{\text{X}}}_{·j}$, for $i\le j\le p$. A larger absolute value of the Z-score statistic implies a stronger effect of the marker on the trait. For any pair of traits, the scatter plot of the Z-statistics provides a way of revealing the shared genetic relationship between them. The scatterplots of the Z-scores for all 28 pairs of traits are included in Section D of the Supplemetal Materials. Figure 1 (a) shows the plots of several pairs of the traits, including the pairs with a large positive $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, YPD v.s. YNB and Paraquat v.s. YNB, pairs with a large negative $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$, Raffinose v.s. Hydrogen and Calcium v.s. Hydrogen, and pairs with $\text{I}(\mathit{\text{β}},\mathit{\text{γ}})$ near 0, including Paraquat v.s. Diamide and Paraquat v.s. Raffinose. The plot clearly indicates a strong positive genetic covariance between YPD and YNB. The genetic covariance between Paraquat and YNB/YPD is smaller. Raffinose/Hydrogen and Calcium/Hydrogen pair clearly show negative genetic correlation. There are several genetic variants with very large effects on Hydrogen, but they are not associated with the other traits such as Raffinose and Calcium. The shared genetic variants are relatively weak, leading to smaller genetic covariances. The plots on the bottom show the pairs of traits with weak weak genetic covariances. These plots indicate that the proposed genetic correlation measures can indeed capture the genetic sharing among different related traits.

Figure 1:

Scatter plots of marginal regression Z-score statistics for six pairs of traits ranked by the estimated genetic covariance (gcov) based on FDE (a) or LD regression (b), including the pairs with large positive genetic covariance (left panel), negative genetic covariance (middle panel), and small genetic covariance (right panel).

Figure 2 shows the six pairs of the phenotypes ranked by the estimated genetic correlations FDE, including two with the largest positive genetic correlations, two with the largest negative genetic correlations and two with the small genetic correlations. The pairs identified agree with the marginal Z-scores very well.

Figure 2:

Scatter plots of marginal regression Z-score statistics for six pairs of traits ranked by the estimated genetic correlation (gcor) based on FDE, including the pairs with large positive genetic correlation (left panel), negative genetic correlation (middle panel), and small genetic correlation (right panel).

As a comparison, we also obtained the estimated genetic covariance for each pair of the traits using the LD regression methods proposed by Bulik-Sullivan et al. (2015). The pairs of traits with large positive, negative or weak estimated covariance are presented in Figure 1 (b). The pairs with the largest positive and negative estimated covariance are different from those two pairs identified by FDE. Comparison of the scatter plots of the Z-score statistics in Figure 1 indicates the pairs identified by FDE seem to agree with the marginal Z-statistics better.

6. Discussion

Motivated by the problems of estimating the genetic relatedness between two traits using the GWAS data, we have considered the problem of estimating the different functionals of the regression coefficients of two linear models, including the inner product $\langle \mathit{\beta },\mathit{\text{γ}}\rangle$, the quadratic functionals $\text{Q}(\mathit{\text{β}})$ and $\text{Q}(\mathit{\text{γ}})$ and the ratio $\text{R}(\mathit{\beta },\mathit{\gamma })$. The proposed method is different from plugging in the de-biased estimators proposed in Javanmard and Montanari (2014); van de Geer et al. (2014); Zhang and Zhang (2014). The correction procedures are implemented on the inner product and quadratic functionals directly, which balance the bias and variance uniquely for these functionals and hence result in minimax rate optimal estimators. The proposed estimators were shown in simulations to result in smaller estimation errors than directly plugging in these de-biased estimators across different settings. Results from analysis of the yeast segregants data suggested that the yeast colony growth sizes were under similar genetic controls under certain growth medias such as YPD and YNB, but this was not true for all pairs of growth media considered.

The algorithm for obtaining the these estimates only involves applying the Lasso several times, which can be implemented efficiently using the coordinate descent algorithms. The Matlab codes to implement the proposed estimation methods are available at http://statgene.med.upenn.edu/software.html. An important future research is to quantify uncertainty of these proposed estimators and the upper bound analysis of (21)-(23) and (24) indicates the possibility of constructing confidence intervals, centering at the proposed estimators and of parametric length $1/\sqrt{n}$, under additional sparsity and other regularity conditions.

7. Proofs

In this Section, we prove Theorem 1 and (29) and (30) of Theorem 3. The proofs of Theorem 2, (28) and (31) of Theorem 3 and extra lemmas are presented in the supplementary materials.

7.1. Proof of Theorem 1

For simplicity of notation, we assume $n_1=n_2$ and use $n=n_1=n_2$ to represent the sample size throughout the proof. The proofs can be easily generalized to the case $n_1\asymp n_2$. Without loss of generality, we assume that the Sub-gaussian norm of random vectors ${\mathit{\text{X}}}_{i·}$ and ${\mathit{\text{Z}}}_{i·}$ are also upper bounded by $\sqrt{M_1}$, that is, $\max \left\{\parallel {\mathit{X}}_{i\cdot }{\parallel }_{{\psi }_2}^2,\parallel {\mathit{Z}}_{i\cdot }{\parallel }_{{\psi }_2}^2\right\}\le M_1$.

Proof of (21)

The upper bound is based on the following decomposition,

$$\begin{gathered} \widehat{\text{I}}(\mathit{\beta },\mathit{\gamma })-\text{I}(\mathit{\beta },\mathit{\gamma })=\langle \widehat{\mathit{\beta }},\widehat{\mathit{\text{γ}}}\rangle +{\widehat{\mathit{u}}}_1^{\top }\frac{1}{n}{\mathit{X}}^{\top }(\mathit{y}-\mathit{X}\widehat{\mathit{\beta }})+{\widehat{\mathit{u}}}_2^{\top }\frac{1}{n}{\mathit{Z}}^{\top }(w-\mathit{Z}\widehat{\mathit{\gamma }})-\langle \mathit{\beta },\mathit{\text{γ}}\rangle \\ =\left({\widehat{\mathit{u}}}_1^{\top }\frac{1}{n}{\mathit{X}}^{\top }(\mathit{y}-\mathit{X}\widehat{\mathit{\beta }})-\langle \widehat{\mathit{\text{γ}}},\mathit{\beta }-\widehat{\mathit{\beta }}\rangle \right)+\left({\widehat{\mathit{u}}}_2^{\top }\frac{1}{n}{\mathit{Z}}^{\top }(\mathit{w}-\mathit{Z}\widehat{\mathit{\text{γ}}})-\langle \widehat{\mathit{\beta }},\mathit{\gamma }-\widehat{\mathit{\text{γ}}}\rangle \right)-\langle \widehat{\mathit{\beta }}-\mathit{\beta },\widehat{\mathit{\text{γ}}}-\mathit{\text{γ}}\rangle \\ ={\widehat{\mathit{u}}}_1^{\top }\frac{1}{n}{\mathit{X}}^{\top }\mathit{ϵ}+\left({\widehat{\mathit{u}}}_1^{\top }\widehat{\mathbf{\text{Σ}}}-{\widehat{\mathit{\text{γ}}}}^{\top }\right)(\mathit{\beta }-\widehat{\mathit{\beta }})+{\widehat{\mathit{u}}}_2^{\top }\frac{1}{n}{\mathit{Z}}^{\top }\mathit{\delta }+\left({\widehat{\mathit{u}}}_2^{\top }\widehat{\mathbf{\text{Γ}}}-{\widehat{\mathit{\beta }}}^{\top }\right)(\mathit{\text{γ}}-\widehat{\mathit{\text{γ}}})-\langle \widehat{\mathit{\beta }}-\mathit{\beta },\widehat{\mathit{\text{γ}}}-\mathit{\text{γ}}\rangle . \end{gathered}$$ (37)

The following lemmas are introduced to control the terms in (37) and similar results were established in the analysis of Lasso, scaled Lasso and de-biased Lasso (Cai and Guo, 2017b; Ren et al., 2015; Sun and Zhang, 2012; Ye and Zhang, 2010). The proofs of the following lemmas can be found in the supplementary material, Section D.

Lemma 1.

Suppose the assumption (A1) holds and $k=\max \left\{\parallel \mathit{\beta }{\parallel }_0,\parallel \mathit{\gamma }{\parallel }_0\right\}\le cn/\log p$ for some $c>0$. Then with probability at least $1-p^{-c_0}$, we have

$$\parallel \widehat{\mathit{\beta }}-\mathit{\beta }{\parallel }_1\le Ck\sqrt{\frac{\log p}{n}},\parallel \widehat{\mathit{\gamma }}-\mathit{\gamma }{\parallel }_1\le Ck\sqrt{\frac{\log p}{n}},$$ (38)

$$\parallel \widehat{\mathit{\beta }}-\mathit{\beta }{\parallel }_2\le C\sqrt{\frac{k\log p}{n}},\parallel \widehat{\mathit{\gamma }}-\mathit{\gamma }{\parallel }_2\le C\sqrt{\frac{k\log p}{n}},$$ (39)

where $c_0$ and $C$ are positive constants.

Lemma 2.

Suppose the assumption (A1) holds and $k=\max \left\{\parallel \mathit{\beta }{\parallel }_0,\parallel \mathit{\gamma }{\parallel }_0\right\}\le cn/\log p$ for some $c>0$. Then with probability at least $1-2\alpha -p^{-c_0}$, we have

$$\left|\left({\widehat{\mathit{u}}}_1^{\top }\widehat{\mathbf{\text{Σ}}}-{\widehat{\mathit{\gamma }}}^{\top }\right)(\mathit{\beta }-\widehat{\mathit{\beta }})\right|\le C\parallel \widehat{\mathit{\text{γ}}}{\parallel }_2\frac{\parallel \mathit{\beta }{\parallel }_0\log p}{n}and\left|\left({\widehat{\mathit{u}}}_2^{\top }\widehat{\mathbf{\text{Γ}}}-{\widehat{\mathit{\beta }}}^{\top }\right)(\mathit{\text{γ}}-\widehat{\mathit{\text{γ}}})\right|\le C\parallel \widehat{\mathit{\beta }}{\parallel }_2\frac{\parallel \mathit{\text{γ}}{\parallel }_0\log p}{n};$$ (40)

$$\left|{\widehat{\mathit{u}}}_1^{\top }\frac{1}{n}{\mathit{X}}^{\top }\mathit{ϵ}\right|\le C\parallel \widehat{\mathit{\gamma }}{\parallel }_2\frac{z_{\alpha /2}}{\sqrt{n}},and\left|{\widehat{\mathit{u}}}_2^{\top }\frac{1}{n}{\mathit{Z}}^{\top }\mathit{\delta }\right|\le C\parallel \widehat{\mathit{\beta }}{\parallel }_2\frac{z_{\alpha /2}}{\sqrt{n}}\text{,}$$ (41)

where $c_0$ and $C$ are positive constants.

By the decomposition (37) and the inequalities (39), (40) and (41), we obtain that

$$|\widehat{\text{I}}(\mathit{\beta },\mathit{\gamma })-\text{I}(\mathit{\beta },\mathit{\gamma })|\le C\left(\parallel \widehat{\mathit{\beta }}{\parallel }_2+\parallel \widehat{\mathit{\gamma }}{\parallel }_2\right)\left(\frac{z_{\alpha /2}}{\sqrt{n}}+\frac{k\log p}{n}\right)+C\frac{k\log p}{n}.$$

By (39), we establish (21).

Proof of (22) and (23)

The proof of (23) is similar to that of (22) and only the proof of (22) is present in the following. We introduce the estimator $\overline{\text{Q}}(\mathit{\beta })=\parallel \widehat{\mathit{\beta }}{\parallel }_2^2+2{\widehat{\mathit{u}}}_3^{\top }{\left({\mathit{X}}^{(2)}\right)}^{\top }\left({\mathit{y}}^{(2)}-{\mathit{X}}^{(2)}\widehat{\mathit{\beta }}\right)/(n/2)$ and due to the fact that $\text{Q}(\mathit{\text{β}})$ is non-negative, we have $|\widehat{\text{Q}}(\mathit{\beta })-\text{Q}(\mathit{\beta })|\le |\overline{\text{Q}}(\mathit{\beta })-\text{Q}(\mathit{\beta })|$. We decompose the difference between $\overline{\text{Q}}(\mathit{\text{β}})$ and $\text{Q}(\mathit{\text{β}})$,

$$\begin{gathered} \overline{\text{Q}}(\mathit{\beta })-\text{Q}(\mathit{\beta })=\parallel \widehat{\mathit{\beta }}{\parallel }_2^2-\parallel \mathit{\beta }{\parallel }_2^2+2{\widehat{\mathit{u}}}_3^{\top }\frac{1}{n/2}{\left({\mathit{X}}^{(2)}\right)}^{\top }\left({\mathit{y}}^{(2)}-{\mathit{X}}^{(2)}\widehat{\mathit{\beta }}\right) \\ =2\left({\widehat{\mathit{u}}}_3^{\top }{\widehat{\mathbf{\text{Σ}}}}^{(2)}-{\widehat{\mathit{\beta }}}^{\top }\right)(\mathit{\beta }-\widehat{\mathit{\beta }})+2{\widehat{\mathit{u}}}_3^{\top }\frac{1}{n/2}{\left({\mathit{X}}^{(2)}\right)}^{\top }{\mathit{ϵ}}^{(2)}-\parallel \widehat{\mathit{\beta }}-\mathit{\beta }{\parallel }_2^2. \end{gathered}$$

Combined with the above argument, the upper bound (22) follows from (39) and the following lemma, whose proof can be found in the supplementary material Section D.

Lemma 3.

Suppose the assumption (A1) holds and $k=\max \left\{\parallel \mathit{\beta }{\parallel }_0,\parallel \mathit{\gamma }{\parallel }_0\right\}\le c\frac{n}{\log p}$ for some $c>0$. Then with probability at least $1-p^{-c_0}-\alpha$,

$$\left|{\widehat{\mathit{u}}}_3^{\top }\frac{1}{n/2}{\left({\mathit{X}}^{(2)}\right)}^{\top }{\mathit{ϵ}}^{(2)}\right|\le C\parallel \widehat{\mathit{\beta }}{\parallel }_2\frac{z_{\alpha /2}}{\sqrt{n}},$$ (42)

$$\left|\left({\widehat{\mathit{u}}}_3^{\top }{\widehat{\mathbf{\text{Σ}}}}^{(2)}-{\widehat{\mathit{\beta }}}^{\top }\right)(\mathit{\beta }-\widehat{\mathit{\beta }})\right|\le C\parallel \widehat{\mathit{\beta }}{\parallel }_2\frac{k\log p}{n},$$ (43)

where $c_0$ and $C$ are positive constants.

7.2. Proof of (29) and (30) in Theorem 3

We first introduce the notations used in the proof of lower bound results. Let $\pi$ denote the prior distribution supported on the parameter space $𝓗$. Let $f_{\pi }(z)$ denote the density function of the marginal distribution of the random variable $\mathit{Z}$ with the prior $\pi$ on $𝓗$. More specifically, $f_{\pi }(z)=\int f_{\theta }(z)\pi (\theta )d\theta$. We define the ${\chi }^2$ distance between two density functions $f_1$ and $f_0$ by

$${\chi }^2\left(f_1,f_0\right)=\int \frac{{\left(f_1(z)-f_0(z)\right)}^2}{f_0(z)}dz=\int \frac{f_1^2(z)}{f_0(z)}dz-1$$ (44)

and the $L_1$ distance by $L_1\left(f_1,f_0\right)=\int \left|f_1(z)-f_0(z)\right|dz$. It is well known that

$$L_1\left(f_1,f_0\right)\le \sqrt{{\chi }^2\left(f_1,f_0\right)}.$$ (45)

The proof of the lower bound is based on the following version of Le Cam’s Lemma (LeCam (1973); Yu (1997); Ren et al. (2015)).

Lemma 4.

Let $\text{T}(\mathit{\theta })$ denote a functional on $\mathit{\text{θ}}$. Suppose that ${𝓗}_0=\left\{{\mathit{\theta }}_0\right\}$, ${𝓗}_0$, ${𝓗}_1\subset \text{Θ}$ and $d={\min }_{\mathit{\theta }\in {𝓗}_1}\left|\text{T}(\mathit{\theta })-\text{T}\left({\mathit{\theta }}_0\right)\right|$. Let $\pi$ denote a prior on the parameter space ${𝓗}_1$. Then we have

$$\underset{\widehat{\text{T}}}{\inf }\underset{\mathit{\theta }\in {\mathscr{H}}_0\cup {\mathscr{H}}_1}{\sup }{\text{ℙ}}_{\mathit{\theta }}\left(|\widehat{\text{T}}-\text{T}(\mathit{\theta })|\ge \frac{d}{2}\right)\ge \frac{1-L_1\left(f_{\pi },f_{{\mathit{\theta }}_0}\right)}{2}.$$ (46)

The proofs of (29) and (30) are applications of Lemma 4. The key is to construct the parameter spaces ${𝓗}_0=\left\{{\mathit{\theta }}_0\right\}$, ${𝓗}_1$ and the prior on ${𝓗}_1$ such that (i) ${𝓗}_0$, ${𝓗}_1\subset \text{Θ}$, (ii) the $L_1$ distance $L_1\left(f_{\pi },f_{{\mathit{\theta }}_0}\right)$ is controlled and (iii) the distance $d={\min }_{\mathit{\theta }\in {𝓗}_1}\left|\text{T}(\mathit{\theta })-\text{T}\left({\mathit{\theta }}_0\right)\right|$ is maximized. In the following, we provide the detailed proof of (29). The proof of (30) is similar to that of (29) and is omitted here. In the discussion of lower bound results, we will assume that the design ${\mathit{\text{X}}}_{i·}$ and ${\mathit{\text{Z}}}_{i·}$ follow joint normal distribution with zero means. The lower bound (29) can be decomposed into the following three lower bounds,

$$\underset{\tilde{\text{Q}}}{\inf }\underset{{\mathit{\theta }}_{\mathit{\beta }}\in 𝓖\left(k,M_0\right)}{\sup }{\text{ℙ}}_{{\mathit{\theta }}_{\mathit{\beta }}}\left(|\tilde{\text{Q}}-\text{Q}(\mathit{\beta })|\ge c^{\prime }\min \left\{\frac{k\log p}{n},M_0^2\right\}\right)\ge \frac{1}{4}.$$ (47)

For $M_0\ge C\sqrt{k\log p/n}$, then

$$\underset{\tilde{\text{Q}}}{\inf }\underset{{\mathit{\theta }}_{\mathit{\beta }}\in 𝓖\left(k,M_0\right)}{\sup }{\text{ℙ}}_{{\mathit{\theta }}_{\mathit{\beta }}}\left(|\tilde{\text{Q}}-\text{Q}(\mathit{\beta })|\ge c^{\prime }{M}_0\frac{k\log p}{n}\right)\ge \frac{1}{4},$$ (48)

$$\underset{\tilde{\text{Q}}}{\inf }\underset{{\mathit{\theta }}_{\mathit{\beta }}\in 𝓖\left(k,M_0\right)}{\sup }{\text{ℙ}}_{{\mathit{\theta }}_{\mathit{\beta }}}\left(|\tilde{\text{Q}}-\text{Q}(\mathit{\beta })|\ge c^{\prime }{M}_0\frac{1}{\sqrt{n}}\right)\ge \frac{1}{4},$$ (49)

where $c^{\prime }>0$ is a positive constant. For $M_0\ge C\sqrt{k\log p/n}$, combining (48), (49) and (47), we have

$$\underset{\tilde{\text{Q}}}{\inf }\underset{{\mathit{\theta }}_{\mathit{\beta }}\in 𝓖\left(k,M_0\right)}{\sup }{\text{ℙ}}_{{\mathit{\theta }}_{\mathit{\beta }}}\left(|\tilde{\text{Q}}-\text{Q}(\mathit{\beta })|\gtrsim \max \left\{M_0\left(\frac{1}{\sqrt{n}}+\frac{k\log p}{n}\right),\frac{k\log p}{n}\right\}\right)\ge \frac{1}{4}.$$ (50)

For $M_0\ge C\sqrt{k\log p/n}$, by (47), we have

$$\underset{\tilde{\text{Q}}}{\inf }\underset{{\mathit{\theta }}_{\mathit{\beta }}\in 𝓖\left(k,M_0\right)}{\sup }{\text{ℙ}}_{{\mathit{\theta }}_{\mathit{\beta }}}\left(|\tilde{\text{Q}}-\text{Q}(\mathit{\beta })|\ge c^{\prime }{M}_0^2\right)\ge \frac{1}{4}.$$ (51)

We can establish (29) by combining (50) and (51). In the following, we will establish the lower bounds (48), (49) and (47) separately.

Proof of (48)

Under the Gaussian random design model, ${\mathbf{V}}_i=\left({\mathit{y}}_i,{\mathit{X}}_i\right)\in {\mathbf{R}}^{p+1}$ follows a joint Gaussian distribution with mean $\mathbf{0}$. Let ${\mathbf{\text{Σ}}}^v$ denote the covariance matrix of ${\mathbf{V}}_i$. For the indices of ${\mathbf{\text{Σ}}}^v$, we use 0 as the index of ${\mathit{\text{y}}}_i$ and $\{1,···,p\}$ as the indices for $\left({\mathit{X}}_{i1},\cdots ,{\mathit{X}}_{ip}\right)\in {\mathbf{R}}^p$. Decompose ${\mathbf{\text{Σ}}}^v$ into blocks $\left(\begin{array}{cc}{\mathbf{\text{Σ}}}_{yy}^v& {\left({\mathbf{\text{Σ}}}_{xy}^v\right)}^{\top }\\ {\mathbf{\text{Σ}}}_{xy}^v& {\mathbf{\text{Σ}}}_{xx}^v\end{array}\right)$, where ${\mathbf{\text{Σ}}}_{yy}^v$, ${\mathbf{\text{Σ}}}_{xx}^v$ and ${\mathbf{\text{Σ}}}_{xy}^v$ denote the variance of ${\mathit{\text{y}}}_i$, the variance of ${\mathit{\text{X}}}_{i·}$ and the covariance of ${\mathit{\text{y}}}_i$ and ${\mathit{\text{X}}}_{i·}$, respectively. Let $\mathbf{\text{Ω}}={\mathbf{\text{Σ}}}^{-1}$ denote the precision matrix. There exists a bijective function $h:{\mathbf{\text{Σ}}}^v\to \left(\mathit{\beta },\mathbf{\Omega },{\sigma }_1\right)$ and the inverse mapping $h^{-1}:\left(\mathit{\beta },\mathbf{\Omega },{\sigma }_1\right)\to {\mathbf{\text{Σ}}}^v$, where $h^{-1}\left(\left(\mathit{\beta },\mathbf{\Omega },{\sigma }_1\right)\right)=\left(\begin{array}{cc}{\mathit{\beta }}^{\top }{\mathbf{\Omega }}^{-1}\mathit{\beta }+{\sigma }_1^2& {\mathit{\beta }}^{\top }{\mathbf{\Omega }}^{-1}\\ {\mathbf{\text{Ω}}}^{-1}\mathit{\beta }& {\mathbf{\Omega }}^{-1}\end{array}\right)$ and

$$h\left({\mathbf{\text{Σ}}}^v\right)=\left({\left({\mathbf{\text{Σ}}}_{xx}^v\right)}^{-1}{\mathbf{\text{Σ}}}_{xy}^v,{\left({\mathbf{\text{Σ}}}_{xx}^v\right)}^{-1},{\mathbf{\text{Σ}}}_{yy}^v-{\left({\mathbf{\text{Σ}}}_{xy}^v\right)}^{\top }{\left({\mathbf{\text{Σ}}}_{xx}^v\right)}^{-1}{\mathbf{\text{Σ}}}_{xy}^v\right).$$ (52)

Based on the bijection, it is sufficient to control the ${\chi }^2$ distance between two multivariate Gaussian distributions. We introduce the null parameter space

$${𝓖}_0=\left\{{\mathit{\theta }}_0=\left({\mathit{\beta }}^{\ast },\text{I},{\sigma }_0\right):{\mathit{\beta }}^{\ast }=\left({\mathit{\beta }}_1^{\ast },{\eta }_0,0,\cdots ,0\right)\text{with}{\mathit{\beta }}_1^{\ast }=-\frac{M_0}{2}\text{and}{\sigma }_0=\sqrt{\frac{M_2}{8}}\right\}\text{,}$$

where $0\le {\eta }_0\le M_0/2$. Note that ${𝓖}_0\subset 𝓖\left(k,M_0\right)$. Define $p_1=p-2$. Based on the mapping $h$, we have the corresponding null parameter space for ${\mathbf{\text{Σ}}}^v$, ${𝓕}_0=\left\{{\mathbf{\text{Σ}}}_0^v\right\}$ where

$${\mathbf{\text{Σ}}}_0^v=\left(\begin{array}{cccc}{\left({\mathit{\beta }}_1^{\ast }\right)}^2+{\eta }_0^2+{\sigma }_0^2& {\mathit{\beta }}_1^{\ast }& {\eta }_0& 0\\ {\mathit{\beta }}_1^{\ast }& 1& 0& 0\\ {\eta }_0& 0& 1& 0\\ 0& 0& 0& {\mathbf{I}}_{p_1\times p_1}\end{array}\right).$$

We then introduce the alternative parameter space for ${\mathbf{\text{Σ}}}^v$, which will induce a parameter space for $(\mathit{\beta },\mathbf{\text{Ω}},{\sigma }_1)$ through the mapping $h$. Define ${𝓕}_1=\left\{{\mathbf{\text{Σ}}}_{\mathit{\alpha }}^v:\mathit{\alpha }\in \ell \left(p_1,k,\rho \right)\right\}$, where

$${\mathbf{\text{Σ}}}_{\mathit{\alpha }}^v=\left(\begin{array}{cccc}{\left({\mathit{\beta }}_1^{\ast }\right)}^2+{\eta }_0^2+{\sigma }_0^2& {\mathit{\beta }}_1^{\ast }& {\eta }_0& {\rho }_0{\mathit{\alpha }}^{\top }\\ {\mathit{\beta }}_1^{\ast }& 1& 0& {\mathit{\alpha }}^{\top }\\ {\eta }_0& 0& 1& 0\\ {\rho }_0\mathit{\alpha }& \mathit{\alpha }& 0& {\mathbf{I}}_{p_1\times p_1}\end{array}\right),$$ (53)

with ${\rho }_0={\mathit{\beta }}_1^{\ast }+{\sigma }_0$ and

$$\ell \left(p_1,k,\rho \right)=\left\{\mathit{\alpha }:\mathit{\alpha }\in {\mathbf{R}}^{p_1},\parallel \mathit{\alpha }{\parallel }_0=k-2,{\mathit{\alpha }}_i\in \{0,\rho \}\text{for}1\le i\le p_1\right\}.$$ (54)

Then we construct the corresponding parameter space ${𝓖}_1$ for $(\mathit{\text{β}},\mathbf{\text{Ω}},{\sigma }_1)$, which is induced by the mapping $h$ and the parameter space ${𝓕}_1$,

$${𝓖}_1=\left\{\left(\mathit{\beta },\mathbf{\Omega },{\sigma }_1\right):\left(\mathit{\beta },\mathbf{\text{Ω}},{\sigma }_1\right)=h\left({\mathbf{\text{Σ}}}^v\right)\text{for}{\mathbf{\text{Σ}}}^v\in {𝓕}_1\right\}.$$ (55)

Similar to equation (7.15) in Cai and Guo (2017b), we can show that for $\left(\mathit{\beta },\mathbf{\Omega },{\sigma }_1\right)\in {𝓖}_1$, the corresponding ${\mathit{\text{β}}}_1$ is $\frac{-\parallel \mathit{\alpha }{\parallel }_2^2{\rho }_0+{\mathit{\text{β}}}_1^{\ast }}{1-\parallel \mathit{\alpha }{\parallel }_2^2}$ and the difference between ${\mathit{\text{β}}}_1$ and ${\mathit{\beta }}_1^{\ast }$ is

$${\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }=\frac{\parallel \mathit{\alpha }{\parallel }_2^2\left({\mathit{\beta }}_1^{\ast }-{\rho }_0\right)}{1-\parallel \mathit{\alpha }{\parallel }_2^2}=\frac{-{\sigma }_0\parallel \mathit{\alpha }{\parallel }_2^2}{1-\parallel \mathit{\alpha }{\parallel }_2^2}.$$

By taking $\rho =\sqrt{\log \left(4p_1/k^2\right)/2n}$, we have $\left|{\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\right|\le C_{0}k\log p/n$ and hence ${\beta }_1^2+{\eta }_0^2+k{\rho }^2\le M_0^2$ for $M_0\ge C\sqrt{k\log p/n}$. Similar to the arguments between (7.15)-(7.18) in Cai and Guo (2017b), we can show that ${𝓖}_1\subset 𝓖\left(k,M_0\right)$. Let $\pi$ denote the uniform prior over the parameter space ${𝓖}_1$ induced by the uniform prior of $\mathit{\text{α}}$ over $\ell \left(p_1,k,\rho \right)$ where $\rho =\sqrt{\log \left(4p_1/k^2\right)/2n}$. The control of $L_1\left(f_{\pi },f_{{\mathit{\theta }}_0}\right)$ is established in the following lemma, which follows from Lemma 2 of Cai and Guo (2017b) and is established in (7.21) of Cai and Guo (2017b).

Lemma 5.

Suppose that $k\le c\left\{n/\log p,p^{\mathit{\gamma }}\right\}$, where $0\le \mathit{\gamma }<\frac{1}{2}$ and $c$ is a sufficient small positive constant. For $\rho =\sqrt{\log \left(4p_1/k^2\right)/2n}$, we establish that $\parallel \mathit{\alpha }{\parallel }_2\le \min \left\{1-1/M_1,M_1-1\right\}$ and

$$L_1\left(f_{\pi },f_{{\mathit{\theta }}_0}\right)\le \frac{1}{4}.$$ (56)

To apply Lemma 4, we consider the functional $\text{T}(\mathit{\theta })=\parallel \mathit{\beta }{\parallel }_2^2$ and calculate the distance

$$d=\left|{\mathit{\beta }}_1^2+{\left(\frac{{\sigma }_0}{1-\parallel \mathit{\alpha }{\parallel }_2^2}\right)}^2\parallel \mathit{\alpha }{\parallel }_2^2-{\left({\mathit{\beta }}_1^{\ast }\right)}^2\right|=\frac{{\sigma }_0}{1-\parallel \mathit{\alpha }{\parallel }_2^2}\parallel \mathit{\alpha }{\parallel }_2^2\left|{\mathit{\beta }}_1+{\mathit{\beta }}_1^{\ast }-\frac{{\sigma }_0}{1-\parallel \mathit{\alpha }{\parallel }_2^2}\right|.$$ (57)

Since ${\mathit{\beta }}_1^{\ast }=-M_0/2$ and $\left|{\mathit{\beta }}_1-{\mathit{\beta }}_1^{\ast }\right|\lesssim k\log p/n$, we have ${\mathit{\text{β}}}_1<0$ and hence

$$\frac{\sigma }{1-\parallel \mathit{\alpha }{\parallel }_2^2}\parallel \mathit{\alpha }{\parallel }_2^2\left|{\mathit{\beta }}_1+{\mathit{\beta }}_1^{\ast }-\frac{{\sigma }_0}{1-\parallel \mathit{\alpha }{\parallel }_2^2}\right|\ge ck\frac{\log p}{n}{\sigma }_0{M}_0,$$

where the last inequality follows from the fact that ${\mathit{\beta }}_1^{\ast }<0$, ${\mathit{\beta }}_1<0$ and $-\frac{{\sigma }_0}{1-\parallel \mathit{\alpha }{\parallel }_2^2}<0$. Combined with (56), an application of Lemma 4 leads to (48).

Proof of (49)

We construct the following parameter spaces,

$$\begin{gathered} {𝓖}_0=\left\{{\mathit{\theta }}_0=\left({\mathit{\beta }}^{\ast },\text{I},{\sigma }_0\right):{\mathit{\beta }}^{\ast }=\left({\mathit{\beta }}_1^{\ast },{\eta }_0,0\cdots ,0\right),\text{and}{\mathit{\beta }}_1^{\ast }=\frac{M_0}{2}\right\} \\ {𝓖}_1=\left\{{\mathit{\theta }}_1=\left(\mathit{\beta },\text{I},{\sigma }_0\right):\mathit{\beta }=\left({\mathit{\beta }}_1^{\ast }+{\sigma }_0\frac{\overline{ϵ}}{\sqrt{n}},{\eta }_0,0\cdots ,0\right)\right\}, \end{gathered}$$ (58)

where $0\le {\eta }_0\le M_0/2$ and $\overline{ϵ}=\sqrt{\log (17/16)/2}$. Since $M_0\ge C\sqrt{k\log p/n}$, we have ${\left({\mathit{\text{β}}}_1^{\ast }+{\sigma }_0\frac{\overline{ϵ}}{\sqrt{n}}\right)}^2+{\eta }_0^2\le M_0^2$ and ${𝓖}_0$, ${𝓖}_1\subset 𝓖\left(k,M_0\right)$.

The proof of the following lemma can be found in the supplementary material Section D.

Lemma 6.

If $\overline{ϵ}=\sqrt{\log (17/16)/2}$, then we have

$$L_1\left(f_{\pi },f_{{\mathit{\theta }}_0}\right)\le \frac{1}{4}.$$ (59)

To apply Lemma 4, we take ${\eta }_0=0$ and calculate the distance

$$d=\left|{\mathit{\beta }}_1^2-{\left({\mathit{\beta }}_1^{\ast }\right)}^2\right|=\left|2{\mathit{\beta }}_1^{\ast }\sigma \frac{\overline{ϵ}}{\sqrt{n}}+{\sigma }^2\frac{{\overline{ϵ}}^2}{n}\right|\ge c{M}_0\frac{1}{\sqrt{n}}.$$

Applying Lemma 4, we establish (49).

Proof of (47)

We introduce the following null and alternative parameter spaces,

$$\begin{gathered} {𝓖}_0=\left\{\left(\mathit{\beta },\mathbf{I},{\sigma }_1\right):\mathit{\beta }=\left({\eta }_0,0,0,\cdots ,0\right)\right\} \\ {𝓖}_1=\left\{\left(\mathit{\beta },\mathbf{I},{\sigma }_1\right):\mathit{\beta }=\left({\eta }_0,0,\mathit{\alpha }\right)\text{with}\mathit{\alpha }\in {\ell }_1(p,k,\rho )\right\}, \end{gathered}$$ (60)

where $0\le {\eta }_0\le M_0/2$ and

$${\ell }_1(p,k,\rho )=\left\{\mathit{\alpha }:\mathit{\alpha }\in {\mathbf{R}}^{p-2},\parallel \mathit{\alpha }{\parallel }_0=k-1,{\mathit{\alpha }}_i\in \{0,\rho \}\right\}.$$ (61)

Let $\pi$ denote the prior over the parameter space ${𝓖}_1$ induced by the uniform prior of $\mathit{\text{α}}$ over ${\ell }_1(p,k,\rho )$ where $\rho =\min \left\{\sqrt{\log \left(4p_1/{(k-1)}^2\right)/2n},\sqrt{\frac{M_0^2-{\eta }_0^2}{k-1}}\right\}$. The control of $L_1(f_{\pi },f_{{\mathit{\theta }}_0})$ is established in the following lemma, which follows from Lemma 7 of Cai and Guo (2017c) and is established in (1.6) of Cai and Guo (2017c).

Lemma 7.

Suppose that $k\le c\left\{n/\log p,p^{\mathit{\gamma }}\right\}$, where $0\le \mathit{\gamma }<\frac{1}{2}$ and $c$ is a sufficient small positive constant. For $\rho =\min \left\{\sqrt{\log \left(4p_1/{(k-1)}^2\right)/2n},\sqrt{\frac{M_0^2-{\eta }_0^2}{k-1}}\right\}$, we establish $L_1\left(f_{\pi },f_{{\mathit{\theta }}_0}\right)\le \frac{1}{8}$.

By specifying ${\eta }_0=0$, we have ${𝓖}_0$ and ${𝓖}_1$ defined in (60) are proper subspaces of the parameter space $𝓖\left(k,M_0\right)$. To apply Lemma 4, we calculate the distance $d=\parallel \mathit{\alpha }{\parallel }_2^2\ge c\min \left\{k\log p/n,M_0^2\right\}$. Applying Lemma 4, we establish (47).