A Review on Dimension Reduction

Summary

Summarizing the effect of many covariates through a few linear combinations is an effective way of reducing covariate dimension and is the backbone of (sufficient) dimension reduction. Because the replacement of high-dimensional covariates by low-dimensional linear combinations is performed with a minimum assumption on the specific regression form, it enjoys attractive advantages as well as encounters unique challenges in comparison with the variable selection approach. We review the current literature of dimension reduction with an emphasis on the two most popular models, where the dimension reduction affects the conditional distribution and the conditional mean, respectively. We discuss various estimation and inference procedures in different levels of detail, with the intention of focusing on their underneath idea instead of technicalities. We also discuss some unsolved problems in this area for potential future research.

1 Description of the Dimension Reduction Problem

Accompanying the advancement of sciences and technologies, scientific data has the tendency of growing in both size and complexity. One characteristic of such complexity is the sheer amount of available covariates, which makes it difficult to detect the dependence between a response variable and the collection of the covariates.

To reduce the problem of many covariates to one with a few covariates, mainly two different approaches exist in the statistical literature. One is variable selection, where the researchers believe that among all the available covariates, only a few are truly related to the response, all others are redundant and have no real explanatory effect. Literature in variable selection has experienced an explosion recently and we do not further elaborate in this article. The second approach to reduce the number of covariates is the so called (sufficient) dimension reduction, which is the focus of this review here. In contrast to the variable selection approach, dimension reduction approach assumes that the response variable relates to only a few linear combinations of the many covariates. Thus, it could happen that all the covariates have explanatory effect, but the effect is only represented in a few linear combinations. It is the goal of dimension reduction to identify these few linear combinations.

To set notations, we use Y ∈ ℝ to represent the response variable, and x = (X₁, …, X_p)^T ∈ ℝ^p to represent the covariates. A main class of dimension reduction problems concern the distribution of Y conditional on x, written as pr(Y ≤ y | x). If there exists a p × d matrix β such that

$$\text{pr}(Y\le y|\mathbf{x})=\text{pr}(Y\le y|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x}),\text{for all}y\in ℝ,$$ (1)

that is, conditional on x, the distribution of Y is the same as that conditional on β^Tx, then, as far as the relation between Y and x is concerned, the p-dimensional covariates x can be replaced by the d-dimensional linear combinations β^Tx. The dimension reduction model (1) is closely related to the effective dimension reduction model in Li (1991). It implies that Y is statistically independent of x when β^Tx is given (Cook, 1998), and the replacement of x by β^Tx retains all the dependence information of Y on x. In fact, Zeng & Zhu (2010) formally proved the equivalence of (1), the effective dimension reduction model of Li (1991) and the independence model of Cook (1998). If we can find a so called loading matrix β satisfying (1), we then convert a problem with a p-dimensional covariate to one with a d-dimensional covariate. Typically, d is much smaller than p, hence we achieve the goal of dimension reduction. It is easy to recognize that β is not identifiable, hence the goal of dimension reduction is to identify the central subspace (Cook 1994, 1998), defined as the column space of β which satisfies (1) with the smallest number of columns d. Central subspace is typically denoted by 𝒮_Y|x in the dimension reduction literature.

While the central subspace problem concerns the conditional distribution of Y on x hence provides a complete picture of their relation, sometimes one might be only interested in certain aspects of the dependence of Y on x. For example, one might be only concerned about the mean of Y conditional on x, expressed by E (Y | x). In such case, usually a weaker assumption

$$E(Y|\mathbf{x})=E(Y|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})$$ (2)

is made, where β is still a p × d matrix with d typically much smaller than p. This is referred to as a central mean subspace problem. In comparison to the central subspace assumption, the central mean subspace assumption only concerns the conditional mean function of the response, and it says that the dependence of the conditional mean of Y on x is completely described by the dependence of the conditional mean of Y on the d linear combinations β^Tx. Thus, to study the relationship between the mean of Y and x, it suffices to study the relationship between the mean of Y and β^Tx, which is a multivariate mean regression problem with d, instead of p, covariates. Thus, as long as we find a loading matrix β, we then reduce the p-covariate mean regression problem to a d-covariate mean regression problem. Similar to β in model (1), β in model (2) is also not identifiable. Thus, our interest is in estimating the central mean subspace (Cook & Li, 2002), which is defined as the column space of β satisfying (2) with the smallest number of columns d. Central mean subspace is typically denoted by 𝒮_E(Y|x).

Yin & Cook (2002) generalized the idea of the central mean subspace and defined the central k-th moment subspace, which is the column space of a p × d matrix β with the smallest number of columns d satisfying

$$E(Y^j|\mathbf{x})=E(Y^j|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x}),\text{for}j=1,\dots ,k.$$ (3)

Central k-th moment subspace is typically written as ${𝒮}_{Y|\mathbf{x}}^{(k)}$. To estimate the conditional variance when x is high dimensional, Zhu & Zhu (2009a) introduced the notion of central variance subspace, defined as the column space of a p × d matrix β with the smallest number of columns d satisfying

$$\text{var}(Y|\mathbf{x})=E\{\text{var}(Y|\mathbf{x})|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x}\}.$$ (4)

Central variance subspace is usually denoted as 𝒮_var(Y|x). If we are concerned with some other characteristic of the dependence of Y on x, we can develop the corresponding dimension reduction subspace as well. To deliver the most essential messages, in the following, we focus our discussion on the central subspace 𝒮_Y|x and central mean subspace 𝒮_E(Y|x) mainly.

2 The Current Approaches in Literature

The description of the dimension reduction problems in Section 1 indicates that finding the column space of β with the smallest dimension d is the target of the dimension reduction literature. In other words, dimension reduction is considered as a problem of estimating a space, instead of the more classic statistical problem of estimating parameters. Except for some degenerated cases, the smallest subspace usually exists and is uniquely defined (Cook, 2004). Another useful observation is that if we set z = V⁻¹(x − u) for any symmetric invertible matrix V and any p-dimensional vector u, then pr (Y | β^Tx) = pr {Y | (Vβ)^Tz}, E(Y^j | β^Tx) = E{Y^j | (Vβ)^Tz} and var(Y | β^Tx) = var{Y | (Vβ)^Tz}. Thus it follows immediately that 𝒮_Y|x = V⁻¹ 𝒮_Y|z, 𝒮_E(Y|x) = V⁻¹ 𝒮_E(Y|z), ${𝒮}_{Y|\mathbf{x}}^{(k)}={\mathbf{V}}^{-1}{𝒮}_{Y|\mathbf{z}}^{(k)}$ and 𝒮_var(Y|x) = V⁻¹ 𝒮_var(Y|z). This is the celebrated invariance property coined by Cook (1998). This property ensures that centring and normalizing the covariates x does not change the nature of the dimension reduction problem. Therefore for simplicity, we assume without loss of generality that the covariates x have mean zero and the identity variance–covariance matrix I_p in our subsequent exposition.

Generally speaking, there are mainly three classes of methods available in the literature for estimating the column space of β: the inverse regression based methods, the non-parametric methods and the semiparametric methods, respectively. We review these three classes of methods for estimating 𝒮_Y|x and 𝒮_E(Y|x) in detail. The methods for estimating ${𝒮}_{Y|\mathbf{x}}^{(k)}$ and 𝒮_var(Y|x) are similar hence are omitted.

2.1 Inverse Regression Based Methods

Inverse regression methods form the oldest class of dimension reduction methods and are still under active development currently, see Adragni & Cook (2009) for a comprehensive and detailed review on this class of methods. The origin of inverse regression is an ingenious idea of reversing the relation between the response variable and the covariates (Li, 1991; Li & Duan, 1991). In this class, instead of considering distributions or expectations of functions of Y conditional on x, which suffers the well-known curse of dimensionality when x is high dimensional, the methods consider expectations of functions of x conditional on Y, which is suddenly a low dimensional problem because Y is univariate. The inverse regression based methods often rely on some additional assumptions on the covariates to link the low dimensional problem and the original high dimensional problem. These additional assumptions include the linearity condition:

$$E(\mathbf{x}|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})=\mathbf{P}\mathbf{x},$$ (5)

which is indispensable for almost all inverse regression based methods. For some methods in this class, the constant variance condition is also needed, which requires

$$\text{cov}(\mathbf{x}|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})=\mathbf{Q}.$$ (6)

In the above displays (5) and (6), P = β(β^Tβ)⁻¹ β^T, Q = I_p − P, and β is a basis matrix of the corresponding dimension reduction subspaces.

For the inverse regression based methods to work, assumptions (5) and (6) only need to hold at the true value of β. But β is unknown. Hence to facilitate the practical verification of these conditions, they are often strengthened to hold for all possible β. If the linearity condition holds for all possible β, then (5) implies that x has an elliptically contoured distribution (Eaton, 1986). Hall & Li (1993) showed that the linearity condition (5) always offers a good approximation to the reality when p diverges to infinity while d remains fixed. If both conditions (5) and (6) are assumed to hold for all possible β, it implies that x has a multivariate normal distribution. If the covariates x do not satisfy these two conditions, useful treatment includes transformation (Box & Cox, 1964) and reweighting (Cook & Nachtsheim, 1994). Note that the presence of any categorical variable will violate the elliptical or normal distributional assumption and void the utility of transformation and reweighting techniques, hence the strengthened version of the linearity condition is not suitable in this case. Because these conditions are not always satisfied, and are not easy to check in practice, Li & Dong (2009) and Dong & Li (2010) relaxed the linearity condition to requiring E(x | β^Tx) to be a polynomial function of β^Tx, while retaining the constant variance condition (6).

For estimating 𝒮_Y|x, the most representative method in the inverse regression class is the sliced inverse regression (SIR, Li, 1991). It uses the eigenvectors associated with the d non-zero eigenvalues of the matrix Λ_SIR ≝ cov {E(x | Y)} to recover 𝒮_Y|x. In order for SIR to work, the linearity condition (5) is critical, as will be illustrated more clearly in Section 2.3. Inspired by SIR, Zhu & Fang (1996) proposed kernel inverse regression and Fung et al. (2002) proposed canonical correlation (CANCOR) analysis. Observing that SIR fails to identify some symmetric patterns, Cook & Weisberg (1991) developed sliced average variance estimation (SAVE) using the eigenvectors associated with the d non-zero eigenvalues of the matrix Λ_SAVE ≝ E {I_p − cov(x | Y)}². Cook & Lee (1999) proved that SAVE is more comprehensive than SIR, yet SIR is more efficient than SAVE. To combine the advantages of both SIR and SAVE, Zhu et al. (2007) suggested using a hybrid of SIR and SAVE through a convex combination αΛ_SIR + (1 − α)Λ_SAVE for some 0 ≤ α ≤ 1, and Li & Wang (2007) proposed direction regression (DR) with the kernel matrix defined by Λ_DR ≝ E[{2I_p − A(Y, Ỹ)}²], A(Y, Ỹ) = E{(x − x̃)(x − x̃)^T | Y, Ỹ}, and (x̃, Ỹ) is an independent copy of (x, Y). One can use either the slicing estimation or the kernel regression method to estimate the kernel matrices of SIR, SAVE, and DR. Yet how to select an optimal number of slices or how to decide an optimal bandwidth remains unknown in the literature. To avoid selecting the tuning parameters such as the number of slices in slicing estimation and the bandwidth in kernel regression, Zhu et al. (2010a) introduced the idea of the discretization–expectation method which improves the estimation accuracy of the slicing estimation, and Zhu et al. (2010d) proposed a family of cumulative slicing estimation in parallel to the development of SIR, SAVE, and DR. Li et al. (2005) proposed contour regression to identify the central subspace 𝒮_Y|x. These methods extend the inverse regression idea and are promising in recovering the central subspace.

The inverse regression methods are also used in a similar way to identify the central mean subspace 𝒮_E(Y|x). Li & Duan (1991) proved that the column space of the ordinary least squares (OLS) is a subspace of 𝒮_E(Y|x) if the covariates x satisfy the linearity condition (5), which can be viewed as the first attempt in this aspect. Li (1992) and Cook & Li (2002) later proposed the principal Hessian directions (PHD) and the iterative Hessian transformations methods, which are able to recover 𝒮_E(Y|x) if both (5) and (6) are satisfied.

Overall, it is fair to say that inverse regression methods are elegant, mysterious at first glance and relatively simple to implement, but are restricted by the moment conditions such as (5), (6) or their variations. It is also worth mentioning that through assuming the conditional normality of x on Y, Cook & Forzani (2008) proposed a likelihood based method. Because a fully parametric model is assumed on x given Y, the joint distribution of (x, Y) can now be fully specified as long as the marginal distribution of Y is obtained. Thus, the inherent difficulty of the dimension reduction problem is assumed away.

The inverse regression based methods can be readily adapted to accommodate regressions with multivariate response. In general the identification of the joint central subspace has been developed along three main streams. The first is to generalize the slicing methodology of Li (1991) through dividing the multivariate response data into several hypercubes. See, for instance, Aragon (1997), Hsing (1999), and Setodji & Cook (2004). The second is to recover the joint central subspace from marginal central subspaces. Some important work in this class includes Cook & Setodji (2003), Saracco (2005), and Yin & Bura (2006). The third approach utilizes one-dimensional projections of response variables. For example, Li et al. (2003) chose to project the multivariate response onto a few “optimal” projections, while Li et al. (2008) and Zhu et al. (2010e) considered to replace the multivariate responses with their random projections to preserve the integrity of the entire joint central subspace.

2.2 Non-Parametric Methods

The idea of non-parametric estimation for dimension reduction is relatively new. Conceptually, this class of estimators estimates the column space of β through minimizing a criterion that describes the fit of the dimension reduction models (1) and (2) to the observed data. Because the criterion inevitably involves unknown distributions or regression functions, non-parametric estimation is always involved. To facilitate our subsequent illustration in this subsection, we further require β to be an orthogonal matrix, that is, β^Tβ = I_d. For notational simplicity, even with the additional orthonormal requirement, we still retain the same notation β.

The most original non-parametric method is the minimum average variance estimation method (MAVE, Xia et al. 2002) for recovering the central mean subspace 𝒮_E(Y|x). The MAVE method is based on a very simple localizing idea used frequently in semiparametric/non-parametric estimation. Specifically, one can recast the dimension reduction model (2) into an equivalent but more familiar multiple-index model

$$Y=m({\mathit{\beta }}^{\mathrm{T}}\mathbf{x})+\epsilon ,$$

where m is an unspecified regression function, and the unobserved error term ε satisfies E(ε | x) = 0. From a Taylor expansion, m(β^Tx_i) ≈ m(β^Tx₀) + m′(β^Tx₀)(β^Tx_i − β^Tx₀) for x_i close to x₀, where m′ ≝ ∂m(β^Tx)/∂(x^Tβ) denotes the first derivative of m. With the m function being an unknown smooth function, we localize the usual ordinary least square criterion to obtain ${\sum }_{i=1}^n{\{Y_i-a_0-{\mathbf{b}}_0^{\mathrm{T}}({\mathit{\beta }}^{\mathrm{T}}{\mathbf{x}}_i-{\mathit{\beta }}^{\mathrm{T}}{\mathbf{x}}_0)\}}^2{w}_i$, where w_i = K_h(β^Tx_i − β^Tx₀) and K_h(·) = K(·/h)/h^d is a d-dimensional kernel function scaled by the bandwidth h. Minimizing this criterion subject to the constraint β^Tβ = I_d then yields an estimate of a₀, b₀, β. Considering that a common β is shared by different x₀ values, a natural compromise is to set x₀ to be the observed predictor values x₁, …, x_n, and obtain the corresponding a_j, b_j and β jointly. This leads to minimizing

$$\sum _{j=1}^n\sum _{i=1}^n{\{Y_i-a_j-{\mathbf{b}}_j^{\mathrm{T}}({\mathit{\beta }}^{\mathrm{T}}{\mathbf{x}}_i-{\mathit{\beta }}^{\mathrm{T}}{\mathbf{x}}_j)\}}^2{w}_{ij}$$

with respect to a_j’s, b_j’s, and β, where w_ij = K_h(β^Txi − β^Tx_j). The minimizer β̂ obtained from the above minimization can be used as an estimated basis matrix of 𝒮_E(Y|x). Differing from the aforementioned inverse regression based methods, the MAVE method guarantees to recover 𝒮_E(Y|x) if d is correctly specified. This property is named exhaustiveness, first introduced by Li et al. (2005). We point out a technique detail that in performing the non-parametric estimation to recover the unknown link function, MAVE has used local linear smoothing, while in the literature, there has been related work in single-index and multiple-index model framework, where kernel method is used (Ichimura, 1993). The local linear choice reduces boundary bias which leads to computational advantages.

To estimate the central subspace 𝒮_Y|x, a useful observation is that E {K_b(Y − y) | x} converges to the density of Y conditional on x as the bandwidth b shrinks to zero. Since the central subspace model (1) implies that the conditional density of Y given x is identical to the conditional density of Y given β^Tx, Xia (2007) adapted the idea of MAVE by regarding K_b(Y − y) as a response variable and proposed the density based MAVE (dMAVE) procedure targeting at recovering 𝒮_Y|x. Specifically, for any fixed y, the MAVE procedure can be adapted to minimizing

$$\sum _{j=1}^n\sum _{i=1}^n{\{K_b(Y_j-y)-a_j-{\mathbf{b}}_j^{\mathrm{T}}({\mathit{\beta }}^{\mathrm{T}}{\mathbf{x}}_i-{\mathit{\beta }}^{\mathrm{T}}{\mathbf{x}}_j)\}}^2{w}_{ij}$$

with respect to a_j, b_j, and β, where β is an orthonormal matrix and w_ij = K_h(β^Tx_i − β^Tx_j). The above minimization is carried out for an arbitrary y, and selecting y to be the observed response values yields minimizing

$$\sum _{k=1}^n\sum _{j=1}^n\sum _{i=1}^n{\{K_b(Y_i-Y_k)-a_{jk}-{\mathbf{b}}_{jk}^{\mathrm{T}}({\mathit{\beta }}^{\mathrm{T}}{\mathbf{x}}_i-{\mathit{\beta }}^{\mathrm{T}}{\mathbf{x}}_j)\}}^2{w}_{ij},$$

the most basic form of dMAVE. More elaborative forms of dMAVE are described in Xia (2007) and some variations such as sliced regression (SR) are proposed in Wang & Xia (2008). Both dMAVE and SR methods inherit the merit of MAVE in the sense that they can recover the entire 𝒮_Y|x exhaustively if d is correctly specified. Built on similar ideas of MAVE, Hernández & Velilla (2005) proposed to minimize a criterion function which involves kernel density estimation, yet their approach can only be applied to classification problems. Yin & Cook (2005) and Yin et al. (2008) proposed a method to recover the dimension reduction space via minimizing another criterion based on Kullback–Leibler distance.

None of the non-parametric methods requires the linearity condition (5) or the constant variance condition (6). Because of the implicit usage of smoothing, however, a common assumption of these non-parametric methods is that all the covariates are continuous. Consequently, as soon as one of the covariates is categorical, the description described above can break down. This issue is likely fixable through more careful handling of the smoothing procedure, although it will further increase the computational complexity of the methods. Overall, comparing with the inverse regression methods, non-parametric methods are conceptually more intuitive while computationally more complex. A review of both the inverse regression based methods and the non-parametric methods can be found in Yin (2010).

2.3 Semiparametric Methods

The semiparametric methods (Ma & Zhu, 2012a) use a geometric tool in Bickel et al. (1993) and Tsiatis (2006) to derive the complete family of influence functions, and subsequently obtain the complete class of estimators. This treatment enables to eliminate the moment conditions (5) and (6) required for the inverse regression methods and the continuity condition required for the non-parametric methods, hence is more flexible. The treatment also further reveals the underlying rationale behind various inverse regression methods and their interrelation, which in turn helps understanding these methods. The concept of influence function was originally introduced by Hampel (1968) in his PhD thesis as a tool to study robustness of an estimator, and has been used by Prendergast (2005, 2007) in the dimension reduction framework to analyse the statistical properties of various inverse regression methods. However, the semiparametric approach of Ma & Zhu (2012a) uses influence function in a completely different way. Instead of deriving the influence function of an existing method, Ma & Zhu (2012a) constructs a class of influence functions directly, then forms the estimators and studies their properties based on the influence functions.

The basic idea of semiparametric methods is very simple. It involves writing down the likelihood of one observation, recognizing that the essential problem of dimension reduction is equivalent to that of parameter estimation in the presence of nuisance components, and taking advantage of the influence function structure to avoid estimating all the nuisance components.

For example, to estimate the central subspace 𝒮_Y|x in model (1), the likelihood of one observation is η₁(x)η₂(Y, β^Tx), where η₁, η₂ are the marginal and conditional density functions and are viewed as nuisance, and the interest is solely on β. Thus, the geometric approach readily yields the unnormalized influence function class as

$$\{\mathbf{f}(Y,\mathbf{x})-E(\mathbf{f}|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x},Y):E(\mathbf{f}|\mathbf{x})=E(\mathbf{f}|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})\forall \mathbf{f}\}.$$

Some more explicit members in this class are for example

$$\sum _{i=1}^k\{{\mathbf{g}}_i(Y,{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})-E({\mathbf{g}}_i|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})\}\{{\mathit{\alpha }}_i(\mathbf{x})-E({\mathit{\alpha }}_i|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})\}$$

for any g_i, α_i, or simply

$$\{\mathbf{g}(Y,{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})-E(\mathbf{g}|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})\}\{\mathit{\alpha }(\mathbf{x})-E(\mathit{\alpha }|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})\}$$ (7)

for any g, α. The double centring through subtracting E(g | β^Tx) from g and subtracting E(α | β^Tx) from α is beneficial because it provides a double robustness property. This means that the mis-specification of either E(g | β^Tx) or E(α | β^Tx) can be tolerated and the resulting estimation is still consistent. In particular, mis-specifying E(g | β^Tx) while providing additional assumptions (such as the linearity condition (5) and the constant variance condition (6) when α is linear or quadratic) to enable the calculation of E(α | β^Tx) is the foundation under the inverse regression class.

The simplest illustration can be obtained from the SIR example. In (7), we select g(Y, β^Tx) = E(x | Y) and α(x) = x^T. The linearity condition yields E(α | β^Tx) = x^TP. Subsequently, the double robustness allows us to mis-specify E(g | β^Tx) = 0. Simple algebra shows that (7) then becomes Λ_SIRQ = 0, which is equivalent to SIR following some linear algebra manipulation. Different choices of g and α lead to different inverse regression methods, see Ma & Zhu (2012a) for details of more such exercises. The importance of this derivation is that the rationale behind the inverse regression methods is now clear. The true reason that this class of methods work is the double robustness property, which allows the linearity and/or constant variance condition alone to guarantee the consistency of the whole estimating equation. Understanding the underlying driving force is crucial for further relaxing these assumptions. For example, assuming a polynomial instead of a linear form of E(x | β^Tx) leads to the relaxed methods of Li & Dong (2009) and Dong & Li (2010). Furthermore, without assuming any structure on E(α | β^Tx), but instead, estimating this quantity through the data, will then completely eliminate the linearity and constant variance conditions and provide a generalized inverse regression estimator class.

The estimation of the central mean subspace 𝒮_E(Y|x) parallels that of the central subspace 𝒮_Y|x. The likelihood of one observation is η₁(x)η₂{Y − m(β^Tx), x}, where η₁, η₂ have the same meaning as before, but with the additional constraint that ∫ εη₂(ε, x)dμ(ε) = 0. The corresponding unnormalized influence function family is

$$[\{Y-E(Y|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})\}\{\mathit{\alpha }(\mathbf{x})-E(\mathit{\alpha }|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})\}:\forall \mathit{\alpha }].$$

This is similar to the central subspace analysis, except that the freedom of choosing g is lost to the sole choice of g(Y) = Y. Comparing the model specification, (1) assumes more structure than (2), hence it is no surprise that the estimator class for 𝒮_Y|x is richer than that for 𝒮_E(Y|x). Similarly, various choices of α leads to various inverse regression estimators, and estimating E(α | β^Tx) instead of assuming it leads to the relaxation of the linearity and constant variance conditions.

From the general theory of semiparametrics, the non-parametric methods, provided that they offer root-n consistent estimation, also have their corresponding influence functions, and should correspond to members in the semiparametric class. Establishing a clear link requires theoretical analysis of the non-parametric estimators and the derivation of the corresponding influence functions, which are not yet available in the dimension reduction literature.

Overall, semiparametric approach offers a birds’ eye view to dimension reduction. It allows a better understanding of the various methods and their connection, and enables the relaxation of some routine assumptions. The approach further provides tools for deriving new estimation and inference procedures, as is elaborated in Section 3.

3 Inference Issue

Because dimension reduction is generally viewed as a problem of estimating a space, inference is strikingly left out of the main stream research in this field. Except for some sporadic analysis growing out of particular needs of specific problems (Chen & Li, 1998; Cook, 2004; Li et al., 2010), no systematic studies have been devoted to inference. This has resulted in estimation of 𝒮_Y|x or 𝒮_E(Y|x) without knowing the variability, and certainly has made it impossible to perform a fair comparison between different estimation procedures.

The underlying difficulty of inference or lack of it is due to the concept of space estimation, instead of the more classical parameter estimation in the dimension reduction problems. None of the existing statistical methods apply if we are overwhelmed by the idea of space estimation. On the other hand, the spaces 𝒮_Y|x and 𝒮_E(Y|x) can be characterized by parameters contained in β. Although different parametrization of β may lead to the same space, for the space estimation purpose, these differences are uninteresting. In other words, as long as we make up our mind to stick with only one particular parametrization strategy, then the issue of space estimation is totally equivalent to that of parameter estimation. This will subsequently make inference more or less standard. We now summarize various different parametrizations and investigate the inference issue subsequently.

3.1 Various Parametrizations

The most widely used parametrization for both 𝒮_Y|x and 𝒮_E(Y|x) is β ∈ ℝ^p×d, where each column of β has length 1 and the d columns of β are mutually orthogonal to each other. This certainly does not uniquely match one space with one β matrix, since both β and βA satisfy the constraints and yield the same space. Here A is an arbitrary d × d orthogonal matrix. For this reason, when this parametrization is used, not only additional constraints β^Tβ = I_d needs to be respected, but one also has to take into account that different βs may actually correspond to the same space. Therefore, this parametrization can be viewed as a non-thorough conversion from space to parameter. However, in practice, because most numerical procedures look for roots or minimizers in a local neighbourhood, while βAs, with different orthogonal matrices A, form a discrete set of matrices, hence the one-to-many mapping issue hardly ever poses a real problem. The constraint of β^Tβ = I_d sometimes offer simplification in computations, although it leads to difficulty in analysing the estimation variability of β̂, because the domain of β is no longer an open set.

A second parametrization is motivated by the aim of representing the spaces 𝒮_Y|x and 𝒮_E(Y|x) without additional constraints on the parameters (Li & Dong, 2009). For d = 1, it parametrizes through requiring the k-th element of β to be sin(θ₁) … sin(θ_k−1) cos(θ_k) for k < p and the p-th element sin(θ₁) … sin(θ_p−2) sin(θ_p−1). Here θ₁, …, θ_p−1 are values in [0, 2π) and are otherwise without constraints. This parametrization is found to bring great computational challenge hence is not further explored for d > 1 cases in the literature.

A third parametrization avoids additional constraints in a different way from the second one. Recognizing that βA and β yield the same space as long as A has full rank, Ma & Zhu (2012b) directly requires the upper d × d submatrix of β to be the identity matrix I_d. This leaves the lower (p − d) × d submatrix of β completely free and provides a one-to-one mapping to the spaces. Similar parametrization was already proposed in Ma & Genton (2010) in a different context, and caution needs to be taken since the parametrization does require the first d components of the p covariates to be non-dismissable. Otherwise, the d rows in I_d will not be always cumulated in the upper part of β. Because with a suitable ordering of the covariates, this parametrization offers a one-to-one mapping between the parameters and spaces without any additional constraints, it is convenient for further investigating inference issues. Since (p − d)d free parameters are needed in this parametrization, it also becomes explicit that 𝒮_Y|x and 𝒮_E(Y|x) are effectively objects with (p − d)d degrees of freedom. Both the second and third parametrization can be viewed as a direct parametrization of any point of the Grassmann manifold (Cook et al., 2010), which is defined as the collection of all d-dimensional subspaces of the p-dimensional real space.

3.2 Inference and Efficient Estimation

Having obtained the unnormalized influence function for estimating 𝒮_Y|x and 𝒮_E(Y|x) in Section 2.3, adopting the third parametrization in Section 3.1, inference is more or less straightforward. Generally speaking, the raw form of the estimating equation system often contains more than (p − d)d equations. We first use a generalized method of moments (GMM) procedure to reduce the number of equations to (p − d)d. Denoting the resulting estimating equation ${\sum }_{i=1}^n\mathbf{f}(Y_i,{\mathbf{x}}_i;\mathit{\beta ̂})=\mathbf{0}$, a standard Taylor expansion then yields

$$\mathbf{0}=n^{-1/2}\sum _{i=1}^n\mathbf{f}(Y_i,{\mathbf{x}}_i;\mathit{\beta })+E\left\{\frac{\partial \mathbf{f}(Y_i,{\mathbf{x}}_i;\mathit{\beta })}{\partial \mathrm{v}\mathrm{e}\mathrm{c}\mathrm{l}{(\mathit{\beta })}^{\mathrm{T}}}\right\}\sqrt{n}\{\mathrm{v}\mathrm{e}\mathrm{c}\mathrm{l}(\mathit{\beta ̂})-\mathrm{v}\mathrm{e}\mathrm{c}\mathrm{l}(\mathit{\beta })\}+o_p(1),$$

and inference is therefore straightforward. Here, vecl(β) or vecl(β̂) is the concatenation of the lower (p − d) × d block of β or β̂ (remember the upper d × d block of β or β̂ is the identity matrix I_d).

With the availability of influence function family and inference tools, a curious issue is whether or not one can obtain efficient estimators. This is where the central subspace estimation problem splits from the central mean subspace estimation problem, because the latter turns out to be almost impossible due to the high dimensionality of x.

Let us start with the easier problem of estimating 𝒮_Y|x efficiently. Ma & Zhu (2012b) derived the efficient score for estimating β as

$${\mathbf{S}}_{\mathrm{e}\mathrm{f}\mathrm{f}}(Y,\mathbf{x},{\mathit{\beta }}^{\mathrm{T}}\mathbf{x},{\eta }_2)=\mathrm{v}\mathrm{e}\mathrm{c}\mathrm{l}[\{\mathbf{x}-E(\mathbf{x}|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})\}\partial \text{log}\{{\eta }_2(Y,{\mathit{\beta }}^{\mathrm{T}}\mathbf{x})\}/\partial ({\mathbf{x}}^{\mathrm{T}}\mathit{\beta })].$$

Hypothetically, the efficient estimator can be obtained through solving

$$\sum _{i=1}^n{\mathbf{S}}_{\mathrm{e}\mathrm{f}\mathrm{f}}(Y_i,{\mathbf{x}}_i,{\mathit{\beta }}^{\mathrm{T}}{\mathbf{x}}_i,{\eta }_2)=\mathbf{0}.$$

However, S_eff is not readily implementable because it contains the unknown quantities E(x | β^Tx) and ∂{logη₂(Y, β^Tx)}/∂(x^Tβ).

To circumvent this problem, we can estimate η₂ and its first derivative based on (Y_i, β^Tx_i), for i = 1, …, n by modifying the idea of the “double-kernel” local linear smoothing method studied in Fan et al. (1996). Let the resulting estimators be η̂₂(·) and ${\mathrm{\eta ̂}}_2^{\prime }(·)$. The estimation of E(x | β^Tx) is more straightforward and we denote the estimator by Ê(x | β^Tx). Inserting these estimated quantities into S_eff, the resulting estimator will indeed yield efficient estimation for 𝒮_Y|x (Ma & Zhu, 2012b).

The analysis appears to be similar for estimating 𝒮_E(Y|x). The efficient score function in this case is

$${\mathbf{S}}_{\mathrm{e}\mathrm{f}\mathrm{f}}(\epsilon ,\mathbf{x},{\mathit{\beta }}^{\mathrm{T}}\mathbf{x},m)=\mathrm{v}\mathrm{e}\mathrm{c}\mathrm{l}\left\{\frac{\epsilon }{E({\epsilon }^2|\mathbf{x})}\right(\mathbf{x}-\frac{E[\mathbf{x}{\{E({\epsilon }^2|\mathbf{x})\}}^{-1}|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x}]}{E[{\{E({\epsilon }^2|\mathbf{x})\}}^{-1}|{\mathit{\beta }}^{\mathrm{T}}\mathbf{x}]}\left)\frac{\partial m({\mathit{\beta }}^{\mathrm{T}}\mathbf{x})}{\partial ({\mathbf{x}}^{\mathrm{T}}\mathit{\beta })}\right\}.$$

To obtain the efficient estimator, it seems one just needs to insert various estimated quantities for ∂m(β^Tx)/∂(x^Tβ), E(ε² | x), E[{E(ε² | x)}⁻¹ | β^Tx], and E [x{E(ε² | x)}⁻¹ | β^Tx]. However, estimating E(ε² | x) involves a high dimensional problem and is the very same problem that motivated the original central mean subspace model. Hence, without additional assumptions, the efficient estimation of 𝒮_E(Y|x) is practically very hard to achieve because of the difficulty in estimating E(ε² | x) when x is high-dimensional.

A useful compromise here is to seek local efficiency, in which case one can, for example, set E(ε² | x) to a constant or other plausible variance function. Because other than the quantity E(ε² | x), all the non-parametric estimation involved in the efficient estimation is of lower dimension and can be handled rather easily, one can calculate the locally efficient estimation via setting a model for E(ε² | x) and estimating all other quantities non-parametrically. This will in general still lead to a consistent estimation of 𝒮_E(Y|x). In addition, when the model for E(ε² | x) happens to be true, then an efficient estimation will be obtained.

3.3 The Role of Linearity/Constant Variance Condition

Among the possible drawbacks of assuming the linearity condition (5) and the constant variance condition (6), the loss of estimation efficiency is certainly the last that one would expect. On the contrary, the typical conjecture would be just the opposite. One would usually suspect that giving up these conditions will incur a price of variability inflation. However, numerical experiments repeatedly support the counter-intuitive phenomenon of improved efficiency through giving up using the conditions (5) and (6) even when they hold, and the ability of performing inference finally allows a formal investigation of this issue.

In this review article we leave out the derivation and proof of this surprising phenomenon. One can refer to Ma & Zhu (2012c) for technical details. Even with the proof, to understand this phenomenon intuitively is not simple at all. One way to help understanding is to refer to Henmi & Eguchi (2004), where using a geometric tool, they explained a similar discovery in treating missing data via inverse probability weighting and provided some general insight. The problem here is unfortunately much more complex than the one they considered and their result or explanation does not apply and cannot be adapted. However, Henmi & Eguchi (2004) at least made one thing clear. In general estimation procedures, using a true quantity to replace an estimated quantity does not necessarily increase or decrease the variability of the estimation of the parameter of interest. The variance of an estimator could go either way, and in our case, using a known form increased the variability. Loosely speaking, when using inverse regression methods, assuming condition (5) and/or (6), E(α | β^Tx) is simply set to its true value when α is a linear or quadratic function of x. Compared with estimating E(α | β^Tx) non-parametrically, this practice eliminates a variance component. This variance component turns out to be in a direction that is beneficial to the variability of estimating β, which is why assuming the conditions would bring an eventual efficiency loss.

This discovery appears somewhat detrimental to the commonly used linearity and constant variance conditions. Because if these conditions are falsely assumed, the inverse regression methods are not consistent, and if they are correctly assumed, they cause inflated variation. In addition, linearity and constant variance conditions are completely irrelevant for the efficient estimation purpose, as is clear from Section 3.2. In fact, these conditions are useful only when α in (7) is taken to be linear or quadratic. In this aspect, if we had chosen to use some other α, we would simply assume a form for E(α | β^Tx) and carry out the subsequent derivation. However, it is also important to recognize the practical usefulness of these conditions. Computationally, these conditions have enabled great implementation simplification of the most familiar inverse regression estimators.

4 Determining the Dimension

Up till now, we have assumed the structural dimension d to be known. In practice, deciding d is not a simple task. In this section we review the literature on how to decide d.

When the linearity condition (5) and/or the constant variance condition (6) are satisfied, the inverse regression methods listed in Section 2.1 formulate the problem of estimating 𝒮_Y|x and 𝒮_E(Y|x) into an eigen-decomposition problem. Therefore, the determination of d becomes a problem of determining the number of non-zero eigenvalues of the corresponding matrix, termed kernel matrix in the literature. Let Λ be the kernel matrix of a specific inverse regression based dimension reduction method. For example, Λ_SIR = cov{E(x | Y)} in SIR (Li, 1991), Λ_SAVE = E{I_p − cov(x | Y)}² in SAVE (Cook & Weisberg, 1991) and Λ_PHD = E{Y − E(Y)}{x − E(x)}{x − E(x)}^T in PHD (Li, 1992). Since the structural dimension d is the number of non-zero eigenvalues of Λ in these methods, and the data provide an estimator Λ̂, thus any sensible way of deciding the number of non-zero eigenvalues through using Λ̂ will yield a reasonable way of deciding d. To this end, there are mainly four approaches in the literature: the sequential test method, the bootstrap method, the BIC type criterion and the sparse eigen-decomposition method.

To implement the sequential test, one tests a series of null hypotheses that Λ has l, from p − 1 down to 1, non-zero eigenvalues. The test procedure stops when the first time a null hypothesis is not rejected. In the context of deciding d for the central subspace 𝒮_Y|x, Li (1991) proposed a sequential chi-square test for the SIR method by assuming that x is normally distributed. Schott (1994), Velilla (1998), Bura & Cook (2001), Cook & Yin (2001), and Cook & Ni (2005) proposed more general versions to relax the normality assumption. To determine the dimension d of the central mean subspace 𝒮_E(Y|x), Li (1992), Cook & Li (2004) designed several sequential tests for principal Hessian directions (Li, 1992) and iterative Hessian transformation (Cook & Li, 2002). A nice summary of the sequential test in various inverse regression methods is given by Bura & Yang (2011). In general, the dimension d̂ obtained from these sequential tests is not consistent due to the presence of the type-I error. When the dimension p is large, such test procedures are computationally inefficient and the cumulative type-I error may not be ignorable. Moreover, it relies heavily on the asymptotic normality of the estimator of the kernel matrix, and requires plugging in an estimate of the asymptotic variance. Due to these theoretical and implementation issues, the sequential test method is usually studied case-by-case for each dimension reduction method in the literature.

Ye & Weiss (2003) proposed to decide d through a bootstrap procedure. Their intuition is that, at the true reduced dimension d, the distance between the estimated central (mean) subspace obtained from the bootstrap sample and that from the original sample must have the smallest variability under suitable distance measure. Therefore, choosing d that corresponds to the smallest estimated variability, assuming no or minimal bias, is a reasonable procedure. Zhu & Zeng (2006) modified the bootstrap procedure through using a different distance measure. The bootstrap procedure estimates the dimension in a data-driven manner, but it is computationally intensive (Zeng, 2008). In addition, the consistency of the bootstrap procedure is still unestablished in the literature.

For the SIR method, Zhu et al. (2006) imitated the classical BIC procedure and proposed a BIC type criterion. The estimated dimension obtained from the BIC type criterion is consistent if the estimator of the relevant kernel matrix converges. However, deciding the amount of penalty through a data-driven procedure is usually difficult in practice. Zhu & Zhu (2007) suggested a classical rate logn penalty, while Luo et al. (2009) used a cutoff value for the ratio of eigenvalues to decide d, both are somewhat ad hoc. Empirical studies suggest that the performance of the BIC type procedure can vary a great deal when different penalties are used, unless the sample size is very large.

Zhu et al. (2010c) proposed a sparse eigen-decomposition (SED) strategy. The main idea of the SED method is to cast the spectral decomposition problem resulting from an inverse regression method into a least squares problem. They then imposed an adaptive LASSO penalty (Tibshirani, 1996; Zou, 2006) to shrink some small estimated eigenvalues to zero directly. Different from the aforementioned methods, the SED method can estimate 𝒮_Y|x or 𝒮_E(Y|x) and the structural dimension d simultaneously. The minimization can be solved effectively by the LARS algorithm (Efron et al., 2004). Therefore, it is computationally attractive. In addition, the resulting estimators of the eigenvalues are consistent, hence d̂ is consistent.

The link to matrix eigen-decomposition is completely lost as soon as the estimation methods deviate from the inverse regression type. In the non-parametric methods we reviewed in Section 2.2, d is often determined through a leave-one-out cross-validation procedure. Specifically, at any fixed working dimension k, the average leave-one-out prediction error is calculated to yield CV(k), and subsequently, the k value that yields the smallest CV(k) is selected as d̂. For estimating 𝒮_E(Y|x), one can easily formulate

$$\mathrm{C}\mathrm{V}(k)=n^{-1}\sum _{i=1}^n{\{Y_i-{\mathrm{m̂}}_{-i}({\mathit{\beta ̂}}_{-i}^{\mathrm{T}}{\mathbf{x}}_i)\}}^2$$

calculated at a fixed working dimension, where the subscript “−i” stands for the usual estimate of the corresponding component with the i-th observation left out (Xia et al., 2002). For estimating 𝒮_Y|x, a similar formulation

$$\mathrm{C}\mathrm{V}(k)=n^{-1}\sum _{i=1}^n\sum _{j=1}^n{\{K_b(Y_i-Y_j)-{\mathrm{\eta ̂}}_{2,-i}(Y_j,{\mathit{\beta ̂}}_{-i}^{\mathrm{T}}{\mathbf{x}}_i)\}}^2$$

can be used.

Cross-validation in the non-parametric methods requires the estimation of m in the central mean subspace case and η₂ in the central subspace case, which are the quantities often avoided to estimate in the semiparametric methods we reviewed in Section 2.3. Thus, Ma & Zhu (2012a) proposed a bootstrap procedure to determine d in the semiparametric class. This procedure adapts the original idea of Ye & Weiss (2003) and Dong & Li (2010) in the inverse regression class.

For two arbitrary random vectors u and v, define

$$r^2(\mathbf{u},\mathbf{v})=\frac{1}{k}\sum _{i=1}^k{\lambda }_i,$$

where λ₁, …, λ_k are the non-zero eigenvalues of

$${\{\text{var}(\mathbf{u})\}}^{-1/2}\text{cov}(\mathbf{u},{\mathbf{v}}^{\mathrm{T}}){\{\text{var}(\mathbf{v})\}}^{-1}\text{cov}(\mathbf{v},{\mathbf{u}}^{\mathrm{T}}){\{\text{var}(\mathbf{u})\}}^{-1/2}.$$

For any working dimension k, let β̂_k be the semiparametric estimate based on the original sample, and β̂_k,b be the same estimate based on the b-th bootstrap sample, b = 1, …, B. Then the bootstrap method determines d through choosing the k value that maximizes the average r² value

$$\frac{1}{B}\sum _{b=1}^B{r}^2({\mathit{\beta ̂}}_k^{\mathrm{T}}\mathbf{x},{\mathit{\beta ̂}}_{k,b}^{\mathrm{T}}\mathbf{x}).$$

The rational behind the bootstrap method is the following. First, we note that r² is a kind of correlation measure. For any working dimension k equal to the true dimension d, β̂_k, and β̂_k,b both estimate the true dimension reduction space hence are largely identical, thus the average r² is expected to be close to 1. For k larger than d, β̂_k, and β̂_k,b contain too much freedom hence would have to contain some additional random directions that are contributed purely by the randomness of the sample. Thus, their corresponding average r² value would be diluted by the random directions hence would decrease. On the other hand, when k is smaller than d, not enough freedom is included in β, hence β̂_k and β̂_k,b will both only be the best approximation to the true β. This best approximation is purely decided by the data randomness and will be different when the random sample is different, hence the corresponding average r² value will also decrease.

5 The Issue of High Dimension

In the classic statistical literature, the dimension p of the covariates x is typically regarded as fixed. To study statistical properties in situations where p is fairly large in comparison with the sample size n, it is more convenient to consider the case where p diverges with n at some rate. In the dimension reduction context, Zhu et al. (2006) established the consistency for SIR when p = o(n^1/4) under the Frobenius norm. That is, they showed that ‖Λ̂_SIR − Λ_SIR‖₂ = o_p(1) as long as p = o(n^1/4) and the linearity condition (5) is true. Zhu et al. (2010d) obtained the consistency of the cumulative slicing estimation when p = o(n^1/2) under the Frobenius norm. We suspect that under the Frobenius norm, all the consistent methods mentioned in Section 2 for a fixed p actually remain consistent as long as p = o(n^1/2). We also believe that p = o(n^1/2) is the largest possible dimension to retain the consistency, if no additional assumption is made on the correlation structure of x.

Recent trend in statistics is to study the situation when the covariate dimensionality p is even larger than the sample size n. This poses new challenges to dimension reduction. For example, even the original normalization of the covariates x is no longer possible because the sample version of Σ = var(x) would be singular. Similarly, in many inverse regression based methods, the kernel matrices involve the computation of quantities such as Σ⁻¹. However, a consistent estimator of Σ⁻¹ is typically not available when p is larger than n. To implement the inverse regression methods in Section 2.1, a useful strategy is to combine the idea of partial least squares with the inverse regression based methods. This avoids the operation of matrix inversion which is not possible if p is larger than n. See, for example, Naik & Tsai (2000), Li et al. (2007), Cook et al. (2007), Zhu & Zhu (2009b), and Zhu et al. (2010b). This strategy is also useful when the covariates x are highly correlated such that Σ is a singular matrix.

Another strategy to handle ultrahigh dimensionality is to utilize the sparsity principle in the dimension reduction context, that is, we assume that only a moderate portion of the covariates are truly relevant to the response variable and need to be reduced further. This corresponds to assuming that most rows in β are simply zeros. In the context of linear regression, Tibshirani (1996), Fan & Li (2001), Candes & Tao (2007) proposed, respectively LASSO, SCAD, and Dantzig selector to penalize some small sample coefficients to zero to yield sparse solutions. One advantage of these methods is that these regularization methods can estimate the magnitude of these non-zero coefficients and screen out those zero coefficients simultaneously. If we only intend to select important covariates in the linear regression context, we can borrow the idea of sure independence screening (SIS) procedure (Fan & Lv, 2008). The SIS procedure ranks the importance of each covariate using the magnitude of the Pearson correlation coefficients. In the screening stage, they only selected the covariates associated with large squared correlation coefficients. They demonstrated that the SIS procedure has the sure screening property, that is, all important predictors will survive after the screening procedure if the sample size n is sufficiently large. With the normality assumption, their results can be applied to the central mean subspace model (2). Following the idea of Fan & Lv (2008), Zhu et al. (2012) proposed sure independence ranking and screening (SIRS) procedure to select important predictors from the central subspace model (1). They proved that the SIRS method has the ranking consistency property, that is, all important predictors will be ranked above the unimportant ones in an asymptotic sense. The ranking consistency property immediately implies the sure screening property provided that an ideal cut-off is available.

The aforementioned procedures all require that the covariates satisfy the linearity condition (5). However, how to handle ultrahigh dimensionality is still unknown when the linearity condition (5) is violated. Research along this line is in much demand.