Dimension Reduction for Fréchet Regression

Abstract

With the rapid development of data collection techniques, complex data objects that are not in the Euclidean space are frequently encountered in new statistical applications. Fréchet regression model (Peterson & Müller 2019) provides a promising framework for regression analysis with metric space-valued responses. In this paper, we introduce a flexible sufficient dimension reduction (SDR) method for Fréchet regression to achieve two purposes: to mitigate the curse of dimensionality caused by high-dimensional predictors, and to provide a visual inspection tool for Fréchet regression. Our approach is flexible enough to turn any existing SDR method for Euclidean ($X,Y$) into one for Euclidean $X$ and metric space-valued $Y$. The basic idea is to first map the metric space-valued random object $Y$ to a real-valued random variable $f\left(Y\right)$ using a class of functions, and then perform classical SDR to the transformed data. If the class of functions is sufficiently rich, then we are guaranteed to uncover the Fréchet SDR space. We showed that such a class, which we call an ensemble, can be generated by a universal kernel (cc-universal kernel). We established the consistency and asymptotic convergence rate of the proposed methods. The finite-sample performance of the proposed methods is illustrated through simulation studies for several commonly encountered metric spaces that include Wasserstein space, the space of symmetric positive definite matrices, and the sphere. We illustrated the data visualization aspect of our method by the human mortality distribution data from the United Nations Databases.

1. Introduction

With the rapid development of data collection techniques, complex data objects that are not in the Euclidean space are frequently encountered in new statistical applications, such as the graph Laplacians of networks, the covariance or correlation matrices for the brain functional connectivity in neuroscience (Ferreira and Busatto, 2013), and probability distributions in CT hematoma density data (Petersen and Müller, 2019). These data objects, also known as random objects, do not obey the operation rules of a vector space with an inner product or a norm, but instead reside in a general metric space. In a prescient paper, Fréchet (1948) proposed the Fréchet mean as a natural generalization of the expectation of a random vector. By extending the Fréchet mean to the conditional Fréchet mean, Petersen and Müller (2019) introduced the Fréchet regression model with random objects as the response and Euclidean vectors as predictors, which provides a promising framework for regression analysis with metric space-valued responses. Dubey and Müller (2019) showed the consistency of the sample Fréchet mean using the results of Petersen and Müller (2019), derived a central limit theorem for the sample Fréchet variance that quantifies the variation around the Fréchet mean, and further developed the Fréchet analysis of variance for random objects. Dubey and Müller (2020a) designed a method for change-point detection and inference in a sequence of metric-space-valued data objects.

The Fréchet regression employs the global least squares and the local linear or polynomial regression to fit the conditional Fréchet mean. It is well known that the global least squares is based on a restrictive assumption of the regression relation. Although the local regression is more flexible, it is effective only when the dimension of the predictor is relatively low. As this dimension gets higher, its accuracy drops significantly–a phenomenon known as the curse of dimensionality. To address this issue, it is essential to reduce the dimension of the predictor without losing the information about the response. For classical regression, this task is performed by sufficient dimension reduction (SDR; see Li 1991; Cook 1996 and Li 2018 among others). SDR projects the high-dimensional predictor onto a low-dimensional subspace that preserves the information about the response through the use of sufficiency.

Besides assisting regression in overcoming the curse of dimensionality, another important function of SDR for classical regression is to provide a data visualization tool to gain insights into how the regression surface looks in high-dimensional space before even fitting a model. By inspecting the sufficient plots of the response objects against the sufficient predictors, we can gain insights into the general trends of the response as the most informative part of the predictor varies, whether there are outlying observations, and whether there are subjects with high leverage that have undue influence on the regression estimates-the usual information a statistician looks for in the exploratory and model checking stages of the regression analysis. This function is also needed in Fréchet regression. In fact, it can be argued that data visualization is even more important for the regression of random objects, as the regression relation may be even more difficult to discern among the complex details of the objects.

To fulfill these demands, we systematically develop the theories and methodologies of sufficient dimension reduction for Fréchet regression in this paper. To set the stage, we first give an outline of SDR for classical regression. Let $X$ be a $p$-dimensional random vector in ${\mathbb{R}}^p$ and $Y$ a random variable in $\mathbb{R}$. The classical SDR aims to find a dimension reduction subspace $𝒮$ of ${\mathbb{R}}^p$ such that $Y$ and $X$ are independent conditioning on $P_{𝒮}X$, that is, $Y⫫X\mid P_{𝒮}X$, where $P_{𝒮}$ is the projection on to $𝒮$ with respect to the usual inner product in ${\mathbb{R}}^p$. This way, $P_{𝒮}X$ can be used as the synthetic predictor without losing regression information about the response $Y$. Under mild conditions, the intersection of all such dimension reduction subspaces is also a dimension reduction subspace, and the intersection is called the central subspace denoted by ${𝒮}_{Y\mid X}$ (Cook, 1996; Yin et al., 2008). For the situation where the primary interest is in estimating the regression function, Cook and Li (2002) introduced a weaker form of SDR, the mean dimension reduction subspace. A subspace $𝒮$ of ${\mathbb{R}}^p$ is a mean SDR subspace if it satisfies $E(Y\mid X)=E\left(Y\mid P_{𝒮}X\right)$, and the intersection of all such spaces if it is still a mean SDR subspace, is the central mean subspace, denoted by ${𝒮}_{E(Y\mid X)}$. The central mean subspace ${𝒮}_{E(Y\mid X)}$ is always contained in central subspace ${𝒮}_{Y\mid X}$ when they exist. Many estimating methods for the central subspace and the central mean subspace have been developed over the past decades. For example, for the central subspace, we have the sliced inverse regression (SIR; Li 1991), the sliced average variance estimate (SAVE; Cook and Weisberg 1991), the contour regression (CR; Li et al. 2005), and the directional regression (DR; Li and Wang 2007). For the central mean subspace, we have the ordinary least squares (OLS; Li and Duan 1989), the principal Hessian directions (PHD; Li 1992), the iterative Hessian transformation (IHT Cook and Li 2002), the outer product of gradients (OPG) and the minimum average variance estimator (MAVE) of Xia et al. (2002).

SDR has been extended to accommodate some complex data structures in the past, for example, to functional data (Ferré and Yao 2003; Hsing and Ren 2009; Li and Song 2017), to tensorial data (Li et al. 2010; Ding and Cook 2015), and to panel data (Fan et al., 2017; Yu et al., 2020; Luo et al., 2021). Most recently, Ying and Yu (2022) extended SIR to the case where the response takes values in a metric space, and Zhang et al. (2022) extended the generalized SIR (Lee et al., 2013) to the case where the response and predictors are distributional data. Taking a substantial step forward, in this paper, we introduce a comprehensive and flexible method that can adapt any existing SDR estimators to metric space-valued responses.

The basic idea of our method stems from the ensemble SDR for Euclidean $X$ and $Y$ of Yin and Li (2011), which recovers the central subspace ${𝒮}_{Y\mid X}$ by repeatedly estimating the central mean subspace ${𝒮}_{E\left[f\right(Y)\mid X]}$ for a family $\mathfrak{F}$ of functions $f$ that is rich enough to determine the conditional distribution of $Y|X$. Such a family $\mathfrak{F}$ is called an ensemble and satisfies ${𝒮}_{Y\mid X}=\cup \left\{{𝒮}_{E\left[f\right(Y)\mid X]}:f\in \mathfrak{F}\right.$. Using this relation, we can turn any method for estimating the central mean space into one that estimates the central subspace.

While borrowing the idea of the ensemble, our goal is different from Yin and Li (2011): we are not interested in turning an estimator for the central mean subspace into one for the central subspace. Instead, we are interested in turning any existing SDR method for Euclidean $(X,Y)$ into one for Euclidean $X$ and metric space-valued $Y$. Let $X$ be a random vector in ${\mathbb{R}}^p$ and $Y$ a random object that takes values in a metric space $\left({\mathrm{\Omega }}_Y,d\right)$. Still use the symbol $S_{Y\mid X}$ to represent the intersection of all subspaces of ${\mathbb{R}}^p$ satisfying $Y⫫X\mid P_{𝒮}X$. We call ${𝒮}_{Y\mid X}$ the central subspace for Fréchet SDR, or simply the Fréchet central subspace. Let $\mathfrak{F}$ be a family of functions $f:{\mathrm{\Omega }}_Y\to \mathbb{R}$ that are measurable with respect to the Borel $\sigma$-field on the metric space. We use two types of ensembles to connect classical SDR with Fréchet SDR:

• Central Mean Space ensemble (CMS-ensemble) is a family $\mathfrak{F}$ that is rich enough so that ${𝒮}_{Y\mid X}=\cup \left\{{𝒮}_{E\left[f\right(Y)\mid X]}:f\in \mathfrak{F}\right\}$. Note that we know how to estimate the spaces ${𝒮}_{E\left(f\right(Y)\mid X)}$ using the existing SDR methods since $f\left(Y\right)$ is a number. We use this ensemble to turn an SDR method that targets the central mean subspace into one that targets the Fréchet central subspace. We will focus on two forward regression methods: OPG and MAVE, and three moment estimators of the CMS.

• Central Space ensemble (CS-ensemble) is a family $\mathfrak{F}$ that is rich enough so that ${𝒮}_{Y\mid X}=\cup \left\{{𝒮}_{f\left(Y\right)\mid X}:f\in \mathfrak{F}\right\}$. We use this ensemble to turn an SDR method that targets the central subspace for real-valued response into one that targets the Fréchet central subspace. We will focus on three inverse regression methods: SIR, SAVE, and DR.

A key step in implementing both of the above schemes is to construct an ensemble $\mathfrak{F}$ in each case. For this purpose, we assume that the metric space $\left({\mathrm{\Omega }}_Y,d\right)$ is continuously embeddable into a Hilbert space. Under this assumption, one can construct a universal reproducing kernel, which leads to an $\mathfrak{F}$ that satisfies the required characterizing property.

As with classical SDR, the Fréchet SDR can also be used to assist data visualization. To illustrate this aspect, we consider an application involving factors that influence the mortality distributions of 162 countries (see Section 7 for details). For each country, the response is a histogram with the numbers of deaths for each five-year period from age 0 to age 100, which is smoothed to produce a density estimate, as shown in panel (a) of Figure 1. We considered nine predictors characterizing each country’s demography, economy, labor market, health care, and environment. Using our ensemble method, we obtained a set of sufficient predictors. In panel (b) of Figure 1, we show the mortality densities plotted against the first sufficient predictor. A clear pattern is shown in the plot: for countries with low values of the first sufficient predictor, the modes of the mortality distributions are at lower ages, and there are upticks at age 0, indicating high infant mortality rates; for countries with high values of the first sufficient predictor, the modes of the mortality distributions are significantly higher, and there are no upticks at age 0, indicating very low infant mortality rates. The information provided by the plot is useful, and many further insights can be gained about what affects the mortality distribution by taking a careful look at the loadings of the first sufficient predictor, as will be detailed in Section 7.

Figure 1:

Data visualization in Fréchet regression for mortality distributions of 162 countries. Panel (a) plots mortality densities that are placed in random order, and Panel (b) plots mortality densities versus the first sufficient predictor estimated by our ensemble method.

The rest of this paper is organized as follows. Section 2 defines the Fréchet SDR problem and provides sufficient conditions for a family $\mathfrak{F}$ to characterize the central subspace. Section 3 then constructs ensemble $\mathfrak{F}$ for the Wasserstein space of univariate distributions, the space of covariance matrix, and a special Riemannian manifold, the sphere. Section 4 proposes the CMS-ensembles by extending five SDR methods that target the central mean space for real-valued response: OLS, PHD, IHT, OPG and MAVE, and CS-ensembles by extending three SDR methods that target the central space for real-valued response: SIR, SAVE, and DR. Section 5 establishes the convergence rate of the proposed methods. Section 6 uses simulation studies to examine the numerical performances of different ensemble estimators in different settings, including distributional responses and covariance matrix responses. In Section 7, we analyze the mortality distribution data to demonstrate the usefulness of our methods. Section 8 includes a few concluding remarks and discussion. All the proofs and additional simulation studies and real applications are presented in the Supplementary Material.

2. Characterization of the Fréchet Central Subspace

Let $(\mathrm{\Omega },ℱ,P)$ be a probability space. Let $\left({\mathrm{\Omega }}_Y,d\right)$ be a metric space with metric $d$ and ${ℬ}_Y$ the Borel $\sigma$-field generated by the open sets in ${\mathrm{\Omega }}_Y$. Let ${\mathrm{\Omega }}_X$ be a subset of ${\mathbb{R}}^p$ and ${ℬ}_X$ the Borel $\sigma$-field generated by the open sets in ${\mathrm{\Omega }}_X$. Let $(X,Y)$ be a random element mapping from $\mathrm{\Omega }$ to ${\mathrm{\Omega }}_X\times {\mathrm{\Omega }}_Y$ measurable with respect to the product $\sigma$-field ${ℬ}_X\times {ℬ}_Y$. We denote the marginal distributions of $X$ and $Y$ by $P_X$ and $P_Y$, respectively, and the conditional distributions of $Y\mid X$ and $X\mid Y$ by $P_{Y\mid X}$ and $P_{X\mid Y}$, respectively. We formulate the Fréchet SDR problem as finding a subspace $𝒮$ of ${\mathbb{R}}^p$ such that $Y$ and $X$ are independent conditioning on $P_{𝒮}X$:

$$Y⫫X\mid P_{𝒮}X,$$ (1)

where $P_{𝒮}$ is the projection on to $𝒮$ with respect to the inner product in ${\mathbb{R}}^p$. As in the classical SDR, the intersection of all such subspaces $𝒮$ still satisfies (1) under mild conditions (Cook and Li, 2002). Indeed, it does not require any structure of the space ${\mathrm{\Omega }}_Y$. A sufficient condition shown in Yin et al. (2008) is that $X$ is supported by a matching set. For example, if the support of $X$ is convex, then this sufficient condition is satisfied. We call this subspace the Fréchet central subspace and denote it by ${𝒮}_{Y\mid X}$. Similar to Cook (1996), it can be shown that if the support of $\mathrm{X}$ is open and convex, the Fréchet central subspace ${𝒮}_{Y\mid X}$ satisfies (1).

2.1. Two types of ensembles and their sufficient conditions

Let $\mathfrak{F}$ be a family of measurable functions $f:{\mathrm{\Omega }}_Y\to \mathbb{R}$, and for an $f\in \mathfrak{F}$, let ${𝒮}_{E\left[f\right(Y)\mid X]}$ be the central mean subspace of $f\left(Y\right)$ versus $X$. As mentioned in Section 1, we use two types of ensembles to recover the Fréchet central subspace. The first type is any $\mathfrak{F}$ that satisfies

$$\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{n}\left\{{𝒮}_{E\left(f\right(Y)\mid X)}:f\in \mathfrak{F}\right\}={𝒮}_{Y|X}.$$ (2)

This is the same ensemble as that in Yin and Li (2011), except that, here, the right-hand side is the Fréchet central subspace. The relation (2) allows us to recover the Fréchet central subspace ${𝒮}_{Y\mid X}$ by a collection of classical central mean subspaces. We call a class $\mathfrak{F}$ that satisfies (2) a CMS-ensemble. The second type of ensemble is any family $\mathfrak{F}$ that satisfies

$$\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{n}\left\{{𝒮}_{f\left(Y\right)\mid X}:f\in \mathfrak{F}\right\}={𝒮}_{Y\mid X},$$ (3)

which we call a CS-ensemble. Proposition 1 shows that a CMS ensemble is a CS-ensemble.

PROPOSITION 1.

If $\mathfrak{F}$ is a CMS-ensemble, then it is a CS-ensemble.

We next develop a sufficient condition for an $\mathfrak{F}$ to be a CMS-ensemble and hence also a CS-ensemble. Let $\mathfrak{B}=\left\{I_B:B\right.\mathrm{i}\mathrm{s}\mathrm{B}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{t}\mathrm{i}\mathrm{n}\left.{\mathrm{\Omega }}_Y\right\}$ be the family of measurable indicator functions on ${\mathrm{\Omega }}_Y$, and let $\mathrm{span}(\mathfrak{F})=\left\{\sum _{i=1}^k{\alpha }_i{f}_i:k\in \mathbb{N},{\alpha }_1,\dots ,{\alpha }_k\in ℝ,f_1,\dots ,f_k\in \mathfrak{F}\right\}$ be the linear span of $\mathfrak{F}$, where $\mathbb{N}=\{1,2,\dots \}$. Yin and Li (2011) showed that if $\mathfrak{F}$ is a subset of $L_2\left(P_Y\right)$ that is dense in $\mathfrak{B}$, then (2) holds for the classical ${𝒮}_{Y\mid X}$. Here, we generalize that result to our setting by requiring only $\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{n}\left(\mathfrak{F}\right)$ to be dense in $\mathfrak{B}$.

LEMMA 1.

If $\mathfrak{F}$ is a subset of $L_2\left(P_Y\right)$ and $\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{n}\left\{\mathfrak{F}\right\}$ is dense in $\mathfrak{B}$ with respect to the $L_2\left(P_Y\right)$-metric, then $\mathfrak{F}$ is a CMS-ensemble and hence also a CS-ensemble.

2.2. Construction of the CMS-ensemble

To construct a CMS-ensemble, we resort to the notion of the universal kernel. Let $C\left({\mathrm{\Omega }}_y\right)$ be the family of continuous real-valued functions on ${\mathrm{\Omega }}_Y$. When ${\mathrm{\Omega }}_Y$ is compact, Steinwart (2001) defined a continuous kernel $\kappa$ as universal (we refer to it as c-universal) if its associated RKHS ${\mathscr{H}}_Y$ is dense in $C\left({\mathrm{\Omega }}_Y\right)$ under the uniform norm. To relax the compactness assumption, Micchelli et al. (2006) proposed the following notion of universality, which is referred to cc-univsersal in Sriperumbudur et al. (2011). For any compact set $K\subseteq {\mathrm{\Omega }}_Y$, let ${\mathscr{H}}_Y\left(K\right)$ be the RKHS generated by $\left\{\kappa \right(\cdot ,y):y\in K\}$. We should note that a member $f$ of ${\mathscr{H}}_Y\left(K\right)$ is supported on ${\mathrm{\Omega }}_Y$, rather than $K$. Let $f\mid K$ denote the restriction of $f$ on $K$, and $C\left(K\right)$ the class of all continuous functions with respect to the topology in $\left({\mathrm{\Omega }}_Y,d\right)$ restricted on $K$.

DEFINITION 1.

(Micchelli et al., 2006) We say that $\kappa$ is cc-universal if, for any compact set $K\subseteq {\mathrm{\Omega }}_Y$, any member $f$ of $C\left(K\right)$, and any $ϵ>0$, there is an $h\in {\mathscr{H}}_Y\left(K\right)$ such that $\parallel f-\left(h\mid K\right){\parallel }_{\infty }=\underset{y\in K}{\text{sup}}\left|f\left(y\right)-h\left(y\right)\right|<ϵ$.

When ${\mathrm{\Omega }}_Y$ is compact, Sriperumbudur et al. (2011) showed that two notions of universality are equivalent. In the following, we look into the conditions under which a metric space has a cc-universal kernel and how to construct such a kernel when it does.

Micchelli et al. (2006) showed that when ${\mathrm{\Omega }}_Y={\mathbb{R}}^d$, many standard kernels, including Laplacian kernels and Gaussian RBF kernels, are cc-universal. Unfortunately, when ${\mathrm{\Omega }}_Y$ is a general metric space, direct extensions of these types of kernels, for example, $k\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma d{\left(y,y^{\prime }\right)}^2)$, are no longer guaranteed to be cc-universal. Christmann and Steinwart (2010) showed that for compact ${\mathrm{\Omega }}_Y$, if there exists a separable Hilbert space $\mathscr{H}$ and a continuous injection $\rho :{\mathrm{\Omega }}_Y\to \mathscr{H}$, then for any analytic function $F:\mathbb{R}\to \mathbb{R}$ whose Taylor series at zero has strictly positive coefficients, the function $\kappa \left(y,y^{\prime }\right)=F\left({⟨\rho \left(y\right),\rho \left(y^{\prime }\right)⟩}_{\mathscr{H}}\right)$ defines a c-universal kernel on ${\mathrm{\Omega }}_Y$. They also provide an analogous definition of the Gaussian-type kernel in the above case. We extend this result to construct cc-universal kernels on non-compact metric space. The proof is given in the Supplementary Material.

PROPOSITION 2.

Suppose $\left({\mathrm{\Omega }}_Y,d\right)$ is a complete and separable metric space, and there exists a separable Hilbert space $\mathscr{H}$ and a continuous injection $\rho :{\mathrm{\Omega }}_Y\to \mathscr{H}$. If $F:\mathbb{R}\to \mathbb{R}$ is an analytic function of the form $F\left(t\right)=\sum _{n=0}^{\infty }{a}_n{t}^n$, $a_n\ge 0$ for all $n\ge 1$, then the function $\kappa :{\mathrm{\Omega }}_Y\times {\mathrm{\Omega }}_Y\to \mathbb{R}$ defined by $\kappa \left(y,y^{\prime }\right)=F\left({⟨\rho \left(y\right),\rho \left(y^{\prime }\right)⟩}_{\mathscr{H}}\right)$ is a positive definite kernel. Furthermore, if $a_n>0$ for all $n\ge 1$, then $\kappa$ is a cc-universal kernel on ${\mathrm{\Omega }}_Y$.

As an example, Corollary 1 shows that the Gaussian-type kernel is cc-universal on ${\mathrm{\Omega }}_Y$

COROLLARY 1.

Suppose the conditions in Proposition 2 are satisfied, then the Gaussian-type kernel ${\kappa }_{\gamma }\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma {∥\rho \left(y\right)-\rho \left(y^{\prime }\right)∥}_{\mathscr{H}}^2)$, where $\gamma >0$, is cc-universal. Furthermore, if the continuous function $\rho :{\mathrm{\Omega }}_Y\to \mathscr{H}$ is isometric, that is, $d\left(y,y^{\prime }\right)={∥\rho \left(y\right)-\rho \left(y^{\prime }\right)∥}_{\mathscr{H}}$, then Gaussian-type kernel ${\kappa }_{\gamma }\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left(-\gamma d^2\left(y,y^{\prime }\right)\right)$ is cc-universal.

The second part of Corollary 1 is straightforward since an isometry is an injection. Similar results can be established for Laplacian-type kernel ${\kappa }_{\gamma }(y,y^{\prime })=\mathrm{e}\mathrm{x}\mathrm{p}\left(-\gamma {∥\rho \left(y\right)-\rho \left(y^{\prime }\right)∥}_{\mathscr{H}}\right)$.

As far as we know, the idea of embedding a (semi) metric space to a Hilbert space was first proposed in Berg et al. (1984) and has been revisited by Sejdinovic et al. (2013) and Dubey and Müller (2020b). By Berg et al. (1984, Theorem 2.2), $\mathrm{e}\mathrm{x}\mathrm{p}\left(-\gamma d^2(\cdot ,\cdot )\right)$ is positive definite for all $\gamma >0$ if and only if $d^2(\cdot ,\cdot )$ is negative definite, which is guaranteed when the metric space can be isometrically embedded into a Hilbert space.

The continuous embedding condition in Proposition 2 covers several metric spaces often encountered in statistical applications. Section 3 employs it to construct cc-universal kernels on the space of univariate distributions endowed with Wasserstein-2 distance, correlation matrices endowed with Frobenius distance, and spheres endowed with geodesic distance.

By using the notion of regular probability measure, we connect the cc-universal kernel on $\left({\mathrm{\Omega }}_Y,d\right)$ with the CMS-ensemble, which is the theoretical foundation of our method. Recall that a measure $P_Y$ on $\left({\mathrm{\Omega }}_Y,d\right)$ is regular if, for any Borel subset $B\subseteq {\mathrm{\Omega }}_Y$ and any $\epsilon >0$, there is a compact set $K\subseteq B$ and an open set $G\supseteq B$, such that $P(G\backslash K)<\epsilon$.

THEOREM 1.

Suppose, on metric space $\left({\mathrm{\Omega }}_Y,d\right)$, (1) $\kappa$ is a bounded cc-universal kernel and (2) $P_Y$ is a regular probability measure. Then the family $\mathfrak{F}=\left\{\kappa (\cdot ,y):y\in {\mathrm{\Omega }}_Y\right\}$ is a CMS-ensemble.

The proof of Theorem 1 is given in the Supplementary Material. Condition (2), which requires $P_Y$ to be regular, is quite mild: it is known that any Borel measure on a complete and separable metric space is regular (see Granirer (1970, Chapter 2: Theorem 1.2, Theorem 3.2)). Thus, a sufficient condition of Condition (2) is $\left({\mathrm{\Omega }}_Y,d\right)$ being complete and separable, which is satisfied by all the metric spaces we consider. Specifically, note that if $M$ is separable and complete, then so is the Wasserstein-2 space $W_2\left(M\right)$ (Panaretos and Zemel, 2020, Proposition 2.2.8, Theorem 2.2.7). Therefore, $W_2\left(\mathbb{R}\right)$ is complete and separable. Similarly, the SPD matrix space endowed with Frobenius distance and the sphere endowed with geodesic distance are complete and separable metric spaces. Furthermore, the Gaussian kernel and Laplacian kernel we considered satisfy Condition (1) in Theorem 1.

Thus, Proposition 2 and Theorem 1 provide a general mechanism to construct the CMS-ensemble over any separable and complete metric space without a linear structure, provided it can be continuously embedded in a separable Hilbert space. For the case where multiple cc-universal kernels exist, we design a cross-validation framework in Section 6 to choose the kernel type and the bandwidth $\gamma$.

3. Important Metric Spaces and their CMS Ensembles

This section gives the construction of CMS-ensembles for three commonly used metric spaces.

3.1. Wasserstein space

Let $I$ be $\mathbb{R}$ or a closed interval of $\mathbb{R},ℬ\left(I\right)$ the $\sigma$-field of Borel subsets of $I$, and $𝒫\left(I\right)$ the collection of all probability measures on $\left(I,ℬ\left(I\right)\right)$. The Wasserstein space ${𝒲}_2\left(I\right)$ is defined as the subset of $𝒫\left(I\right)$ with finite second moment, that is, ${𝒲}_2\left(I\right)=\left\{\mu \in 𝒫\left(I\right):{\int }_I t^{2}d\mu \left(t\right)<\mathrm{\infty }\right\}$, endowed with the quadratic Wasserstein distance $d_{\mathrm{W}}\left({\mu }_1,{\mu }_2\right)={\left({\int }_0^1 {\left[F_{{\mu }_1}^{-1}(s)-F_{{\mu }_2}^{-1}(s)\right]}^{2}ds\right)}^{1/2}$, where ${\mu }_1$ and ${\mu }_2$ are members of ${𝒲}_2\left(I\right)$ and $F_{{\mu }_1}^{-1}$ and $F_{{\mu }_2}^{-1}$ are the quantile functions of ${\mu }_1$ and ${\mu }_2$, which we assume to be well defined. This distance can be equivalently written as $d_{\mathrm{W}}\left({\mu }_1,{\mu }_2\right)={\left({\int }_I {\left[F_{{\mu }_1}^{-1}{\circ F}_{{\mu }_2}(t)-t\right]}^{2}d{\mu }_2(t)\right)}^{1/2}$. The set ${𝒲}_2\left(I\right)$ endowed with $d_{\mathrm{W}}$ is a metric space with a formal Riemannian structure (Ambrosio et al., 2004).

Here, we present some basic results that characterize ${𝒲}_2\left(I\right)$, whose proofs can be found, for example, in Ambrosio et al. (2004) and Bigot et al. (2017). For ${\mu }_1,{\mu }_2\in {𝒲}_2\left(I\right)$, we say that a $ℬ\left(I\right)$-measurable map $r:I\to I$ transports ${\mu }_1$ to ${\mu }_2$ if ${\mu }_2={\mu }_1\circ r^{-1}$. This relation is often written as ${\mu }_2=r_{\#}{\mu }_1$. Let ${\mu }_0\in {𝒲}_2\left(I\right)$ be a reference measure with a continuous $F_{{\mu }_0}$. The tangent space at ${\mu }_0$ is $T_{{\mu }_0}={\mathrm{c}\mathrm{l}}_{L_2\left({\mu }_0\right)}\left\{\lambda \left(F_{\mu }^{-1}\circ F_{{\mu }_0}-\mathrm{i}\mathrm{d}\right):\mu \in {𝒲}_2\left(I\right),\lambda >0\right\}$, where, for a set $A\subseteq L_2\left({\mu }_0\right),{\mathrm{c}\mathrm{l}}_{L_2\left({\mu }_0\right)}\left(A\right)$ denotes the $L_2\left({\mu }_0\right)$-closure of $A$. The exponential map ${\mathrm{e}\mathrm{x}\mathrm{p}}_{{\mu }_0}$ from $T_{{\mu }_0}$ to ${𝒲}_2\left(I\right)$, defined by ${\mathrm{e}\mathrm{x}\mathrm{p}}_{{\mu }_0}(r)=(r+\mathrm{i}\mathrm{d}{)}_{\#}{\mu }_0$, is surjective. Therefore its inverse, ${\mathrm{l}\mathrm{o}\mathrm{g}}_{{\mu }_0}:{𝒲}_2(I)\to T_{{\mu }_0}$, defined by ${\mathrm{l}\mathrm{o}\mathrm{g}}_{{\mu }_0}(\mu )={F_{\mu }}^{-1}\circ F_{{\mu }_0}-\mathrm{i}\mathrm{d}$, is well defined on ${𝒲}_2(I)$. It is well known that the exponential map, restricted to the image of log map, denoted as ${\left.{\mathrm{e}\mathrm{x}\mathrm{p}}_{{\mu }_0}\right|}_{{\mathrm{l}\mathrm{o}\mathrm{g}}_{{\mu }_0}(\mu )\left({𝒲}_2(I)\right)}$, is an isometric homeomorphism (Bigot et al., 2017). Therefore, ${\mathrm{l}\mathrm{o}\mathrm{g}}_{{\mu }_0}$ is a continuous injection from ${𝒲}_2(I)$ to $L_2\left({\mu }_0\right)$. We can then construct CMS-ensembles using the general constructive method provided by Theorem 1 and Proposition 2. The next proposition gives two such constructions, where the subscripts “G” and “L” for the two kernels refer to “Gaussian” and “Laplacian”, respectively.

PROPOSITION 3.

For $I\subseteq \mathbb{R},{\kappa }_G\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma {\parallel {\mathrm{l}\mathrm{o}\mathrm{g}}_{{\mu }_0}(y)-{\mathrm{l}\mathrm{o}\mathrm{g}}_{{\mu }_0}\left(y^{\prime }\right)\parallel }_{{ℒ}_{{\mu }_0}^2}^2)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma d_{\mathrm{W}}{\left(y,y^{\prime }\right)}^2)$ and ${\kappa }_L\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma {\parallel {\mathrm{l}\mathrm{o}\mathrm{g}}_{{\mu }_0}(y)-{\mathrm{l}\mathrm{o}\mathrm{g}}_{{\mu }_0}\left(y^{\prime }\right)\parallel }_{{ℒ}_{{\mu }_0}^2})=\mathrm{e}\mathrm{x}\mathrm{p}\left(-\gamma d_{\mathrm{W}}\left(y,y^{\prime }\right)\right)$ are both cc-universal kernels on ${𝒲}_2(I)$. Consequently, the families ${\mathfrak{F}}_G=\{\exp (-\gamma d_{\mathrm{W}}(\cdot ,t{)}^2):t\in {𝒲}_2(I)\}$ and ${\mathfrak{F}}_L=\left\{\mathrm{e}\mathrm{x}\mathrm{p}\left(-\gamma d_{\mathrm{W}}(\cdot ,t)\right):t\in {𝒲}_2(I)\right\}$ are CMS-ensembles.

3.2. Space of symmetric positive definite matrices

We first introduce some notations. Let Sym ($r$) be the set of $r\times r$ invertible symmetric matrices with real entries and ${\mathrm{S}\mathrm{y}\mathrm{m}}^{+}(r)$ the set of $r\times r$ symmetric positive definite (SPD) matrices. For any $Y\in {\mathbb{R}}^{r\times r}$, the matrix exponential of $Y$ is defined as the infinite power series $\text{exp}(y)=\sum _{k=0}^{\infty }{Y}^k/k!$. For any $X\in {\mathrm{S}\mathrm{y}\mathrm{m}}^{+}(r)$, the matrix logarithm of $X$ is defined as any $r\times r$ matrix $Y$ such that $\mathrm{e}\mathrm{x}\mathrm{p}(y)=X$ and denoted by $\mathrm{l}\mathrm{o}\mathrm{g}\left(X\right)$.

Let $d_{\mathrm{F}}$ be the Frobenius metric. Then $\left.({\mathrm{s}\mathrm{y}\mathrm{m}}^{+}(r),d_{\mathrm{F}}\right)$ is a metric space continuously embedded by identity mapping in ${\mathrm{s}\mathrm{y}\mathrm{m}}^{+}(r)$, which is a Hilbert space with the Frobenius inner product $⟨A,B⟩=\mathrm{t}\mathrm{r}\left(A^{\mathrm{\top }}B\right)$. Also, the identity mapping id: ${\mathrm{S}\mathrm{y}\mathrm{m}}^{+}(r)\to \mathrm{S}\mathrm{y}\mathrm{m}(r)$ is obviously isometric. Therefore, by Corollary 1, the two types of radial basis function kernels for Wasserstein space can be similarly extended to ${\mathrm{S}\mathrm{y}\mathrm{m}}^{+}(r)$. That is, let ${\kappa }_G\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma d_{\mathrm{F}}{\left(y,y^{\prime }\right)}^2)$ and ${\kappa }_L\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left(-\gamma d_{\mathrm{F}}\left(y,y^{\prime }\right)\right)$, then ${\mathfrak{F}}_G=\left\{{\kappa }_G\left(y,y^{\prime }\right),y^{\prime }\in {\mathrm{S}\mathrm{y}\mathrm{m}}^{+}(r)\right\}$ and ${\mathfrak{F}}_L=\left\{{\kappa }_L\left(y,y^{\prime }\right),y^{\prime }\in {\mathrm{S}\mathrm{y}\mathrm{m}}^{+}(r)\right\}$ are CMS-ensembles.

Another widely used metric over ${\mathrm{S}\mathrm{y}\mathrm{m}}^{+}(r)$ is the log-Euclidean distance defined as $d_{\mathrm{l}\mathrm{o}\mathrm{g}}\left(Y_1,Y_2\right)={∥\mathrm{l}\mathrm{o}\mathrm{g}\left(Y_1\right)-\mathrm{l}\mathrm{o}\mathrm{g}\left(Y_2\right)∥}_{\mathrm{F}}$. It pulls the Frobenius metric on $\mathrm{S}\mathrm{y}\mathrm{m}(r)$ back to ${\mathrm{S}\mathrm{y}\mathrm{m}}^{+}(r)$ by the matrix logarithm map. The matrix logarithm $\mathrm{l}\mathrm{o}\mathrm{g}(\cdot )$ is a continuous injection to Hilbert $\mathrm{S}\mathrm{y}\mathrm{m}(r)$. By Corollary 1, the two types of radial basis function kernels ${\kappa }_{G,\mathrm{l}\mathrm{o}\mathrm{g}}\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma d_{\mathrm{l}\mathrm{o}\mathrm{g}}{\left(y,y^{\prime }\right)}^2)$ and ${\kappa }_{L,\mathrm{l}\mathrm{o}\mathrm{g}}\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left(-\gamma d_{\mathrm{l}\mathrm{o}\mathrm{g}}\left(y,y^{\prime }\right)\right)$ are cc-universal. Then, ${\mathfrak{F}}_{G,\mathrm{l}\mathrm{o}\mathrm{g}}=\left\{{\kappa }_{G,\mathrm{l}\mathrm{o}\mathrm{g}}\left(y,y^{\prime }\right),y^{\prime }\in {𝒲}_2(I)\right\}$ and ${\mathfrak{F}}_{L,\mathrm{l}\mathrm{o}\mathrm{g}}=\left\{{\kappa }_{L,\mathrm{l}\mathrm{o}\mathrm{g}}\left(y,y^{\prime }\right),y^{\prime }\in {𝒲}_2(I)\right\}$ are CMS-ensembles.

3.3. The sphere

Consider the random vector taking values in the sphere ${\mathbb{S}}^n=\left\{x\in {\mathbb{R}}^n:\parallel x\parallel =1\right\}$. To respect the nonzero curvature of ${\mathbb{S}}^n$, the geodesic distance $d_g\left(Y_1,Y_2\right)=\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{c}\mathrm{o}\mathrm{s}\left(Y_1^{\mathrm{\top }}{Y}_2\right)$, which is derived from its Riemannian geometry, is often used rather than the Euclidean distance. However, the popular Gaussian-type RBF kernel ${\kappa }_G\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma d_g{\left(y,y^{\prime }\right)}^2)$ is not positive definite on ${\mathbb{S}}^n$ (Jayasumana et al., 2013). In fact, Feragen et al. (2015) proved that for complete Riemannian manifold $M$ with its associated geodesic distance $d_g,{\kappa }_G\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma d_g{\left(y,y^{\prime }\right)}^2)$ is positive semidefinite only if $M$ is isometric to a Euclidean space. Honeine and Richard (2010) and Jayasumana et al. (2013) proved that the Laplacian-type kernel ${\kappa }_L\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left(-\gamma d_g\left(y,y^{\prime }\right)\right)$ is positive definite on the sphere ${\mathbb{S}}^n$. We show in the following proposition that ${\kappa }_L\left(y,y^{\prime }\right)$ is cc-universal.

PROPOSITION 4.

The Laplacian-type kernel ${\kappa }_L\left(y,y^{\prime }\right):{\mathbb{S}}^n\times {\mathbb{S}}^n\to \mathbb{R}$, defined by ${\kappa }_L\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left(-\gamma d_g\left(y,y^{\prime }\right)\right)$ where $d_g$ is the geodesic distance on ${\mathbb{S}}^n$, is a cc-universal kernel for any Consequently, ${\mathfrak{F}}_L=\left\{\mathrm{e}\mathrm{x}\mathrm{p}\left(-\gamma d_g(\cdot ,t)\right),t\in {\mathbb{S}}^n\right\}$ is a CMS-ensemble.

We note that the scope of Proposition 2 goes beyond Riemannian manifolds. For example, the Gaussian type kernel on the space of Borel probability measures on a compact metric space with Prohorov metric is universal (Christmann and Steinwart, 2010). We construct an explicit embedding and a universal kernel for any metric space of negative type in Theorem 4 in the Supplementary Material.

4. Fréchet Sufficient Dimension Reduction

In this section, we develop the Fréchet SDR estimators based on the CMS-ensembles and CS-ensembles and establish their Fisher consistency.

4.1. Ensembled moment estimators via CMS ensembles

We first develop a general class of Fréchet SDR estimators based on the ensembled moment estimators of the CMS, such as the OLS, PHD, and IHT. Let ${𝒫}_{XY}$ be the collection of all distributions of $(X,Y)$, and let $M:{𝒫}_{XY}\to {\mathbb{R}}^{p\times p}$ be a measurable function to be used as an estimator of the Fréchet central subspace ${𝒮}_{Y\mid X}$. A function defined on ${𝒫}_{XY}$ is called statistical functional; see, for example, Chapter 9 of Li (2018). In the SDR literature, such a function is also called a candidate matrix (Ye and Weiss, 2003). Let $F_{XY}$ be a generic member of ${𝒫}_{XY},F_{XY}^{\left(0\right)}$ the true distribution of $(X,Y)$, and ${\hat{F}}_{XY}^{\left(n\right)}$ the empirical distribution of $(X,Y)$ based on an i.i.d. sample $\left(X_1,Y_1\right),\dots ,\left(X_n,Y_n\right)$. Extending the terminology of classical SDR (see, for example, Li 2018, Chapter 2), we say that the estimate $M({\hat{F}}_{XY}^{\left(n\right)})$ is unbiased if $M(F_{XY}^{\left(0\right)})\subseteq {𝒮}_{Y\mid X}$, exhaustive if $M(F_{XY}^{\left(0\right)})\supseteq {𝒮}_{Y\mid X}$, and Fisher consistent if $M(F_{XY}^{\left(0\right)})={𝒮}_{Y\mid X}$. We refer to $M$ as the Fréchet candidate matrix.

Suppose we are given a CMS-ensemble $\mathfrak{F}$. Let $M_0:{𝒫}_{XY}\times \mathfrak{F}\to {\mathbb{R}}^{p\times p}$ be a function to be used as an estimator of ${𝒮}_{E[f(Y)\mid X]}$ for each $f$. In the classical sense, this is not a statistical functional, as it involves an additional set $\mathfrak{F}$. So, we redefine unbiasedness, exhaustiveness, and Fisher consistency for this type of augmented statistical functional.

DEFINITION 2.

We say that $M_0$ is unbiased for estimating $\left\{{𝒮}_{E[f(Y)\mid X]}:f\in \mathfrak{F}\right\}$ if, for each $f\in \mathfrak{F}$, $\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{n}\{M_0(F_{XY}^{\left(0\right)},f)\}\subseteq {𝒮}_{E[f(Y)\mid X]}$. Exhaustiveness and Fisher consistency of $M_0$ are defined by replacing ⊆ in the above by ⊇ and =, respectively.

Note that $M_0(\cdot ,f)$ is an estimator of the classical central mean subspace ${𝒮}_{E[f(Y)\mid X]}$, as $f(Y)$ is a random number rather than a random object. We refer to $M_0$ as the ensemble candidate matrix, or, when confusion is possible, CMS-ensemble candidate matrix. Our goal is to construct a Fréchet candidate matrix $M:{𝒫}_{XY}\to {\mathbb{R}}^{p\times p}$ from the ensemble candidate matrix $M_0:{𝒫}_{XY}\times \mathfrak{F}\to {\mathbb{R}}^{p\times p}$. To do so, we assume $\mathfrak{F}$ is of the form $\left\{\kappa (\cdot ,y):y\in {\mathrm{\Omega }}_Y\right\}$, where $\kappa :{\mathrm{\Omega }}_Y\times {\mathrm{\Omega }}_Y\to \mathbb{R}$ is a cc-universal kernel. Given such an $\mathfrak{F}$ and $M_0$, we define $M$ as follows

$$M\left(F_{XY}\right)={\int }_{{\mathrm{\Omega }}_Y} M_0\left(F_{XY},\kappa \left(\cdot ,y\right)\right)d{F}_Y\left(y\right),$$

where $F_Y$ is the distribution of $Y$ derived from $F_{XY}$.

We now adapt several estimates for the classical central mean subspace to the estimation of Fréchet SDR: the ordinary least squares (OLS; Li and Duan 1989), the principal Hessian directions (PHD; Li 1992), and the Iterative Hessian Transformation (IHT; Cook and Li 2002). These estimates are based on sample moments and require additional conditions on the predictor $X$ for their unbiasedness. Specifically, we make the following assumptions:

ASSUMPTION 1.

1. Linear Conditional Mean (LCM): $E(X\mid {\beta }^{\mathrm{\top }}X)$ is a linear function of ${\beta }^{\mathrm{\top }}X$, where $\beta$ is a basis matrix of the Fréchet central subspace ${𝒮}_{Y\mid X}$;

2. Constant Conditional Variance (CCV): $\mathrm{v}\mathrm{a}\mathrm{r}(X\mid {\beta }^{\mathrm{\top }}X)$ is a nonrandom matrix.

Under the first assumption, the ensemble OLS and IHT are unbiased for estimating the Fréchet central subspace; under both assumptions, the ensemble PHD is unbiased for estimating ${𝒮}_{Y\mid X}$. More detailed discussions on the unbiasedness and fisher consistency of ensemble estimators are presented in Section 4.4. In practice, the two assumptions above cannot be checked directly since we do not know $\beta$. However, as was shown by Eaton (1986), if Assumption 1 holds for all $\beta$, then the distribution of $X$ is elliptical, and vice versa. If further $X$ is multivariate normal, then Assumption 2 is satisfied. This means once the marginal distribution of predictor $X$ is regular, Assumption 1 holds without being affected by the nonlinear nature of the response. Currently, the scatter plot matrix is the most commonly used empirical method to check the elliptical distribution assumption. If non-elliptical features are observed, one can use marginal transformations of the predictors, such as the Box-Cox transformation, to mitigate the non-ellipticity problem. Furthermore, in practice, the SDR methods that require ellipticity usually still work reasonably well even when the elliptical distribution assumption is violated. This occurs particularly when the dimension $p$ of $X$ is high. See Hall and Li (1993) and Li and Yin (2007) for the theoretical supports. Our simulation results in Section 6 support this phenomenon.

It is most convenient to construct these ensemble estimators using standardized predictors. As stated in the next proposition, the theoretical basis for doing so is an equivariant property of the Fréchet central subspace.

PROPOSITION 5.

If ${𝒮}_{Y\mid X}$ is the Fréchet central subspace, $A\in {\mathbb{R}}^{p\times p}$ is a non-singular matrix, and $b$ is a vector in ${\mathbb{R}}^p$, then ${𝒮}_{Y\mid AX+b}=A^{\mathrm{\top }}{S}_{Y\mid X}$.

The proof is essentially the same as that for the classical central subspace (see, for example, Li, 2018, page 24) and is omitted. Using this property, we first transform $X$ to $Z=\mathrm{v}\mathrm{a}\mathrm{r}(X{)}^{-1/2}(X-EX)$, estimate the Fréchet central subspace ${𝒮}_{Y\mid Z}$, and then transform it by $\mathrm{v}\mathrm{a}\mathrm{r}(X{)}^{1/2}{𝒮}_{Y\mid Z}$, which is the same as ${𝒮}_{Y\mid X}$. The candidate matrices $M_0$ and $M$ for ensemble OLS, PHD, and IHT are formulated in Remark 1. Detailed motivation for each can be found in Li (2018, Chapter 8). The sample estimates can then be constructed by replacing the expectations in $M_0$ and $M$ with sample moments whenever possible. Algorithm 1 summarizes the steps to implement an ensembled moment estimator, where ${\kappa }_c\left(y,y^{\prime }\right)$ stands for the centered kernel $\kappa \left(y,y^{\prime }\right)-E_n\kappa \left(Y,y^{\prime }\right)$.

Algorithm 1.

Fréchet OLS, PHD, IHT, SIR, SAVE, DR

Step 1. Standardize predictors. Compute sample mean $\hat{\mu }=E_n\left(X\right)$ and sample variance $\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}={\mathrm{v}\mathrm{a}\mathrm{r}}_n(X)$. Then let $Z_i={\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}}^{-1/2}\left(X_i-\hat{\mu }\right)$.

Step 2. Compute ${\hat{M}}_0(y)$ for $y=Y_1,\dots ,Y_n$ according to Remarks 1 and 2.

Step 3. Compute $\hat{M}=\frac{1}{n}\sum _{i=1}^n{\hat{M}}_0\left(Y_i\right)$.

Step 4. Let ${\hat{v}}_1,\dots ,{\hat{v}}_{d_0}$ be the leading $d_0$ eigenvectors of $\hat{M}$, and let $u_k={\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}}^{1/2}{v}_k$, for $k=1,\dots ,d_0$. Then use $\left\{u_1,\dots ,u_{d_0}\right\}$ to estimate a basis of ${𝒮}_{Y\mid X}$.

REMARK 1.

The candidate matrices $M_0(y)$ for Fréchet OLS, PHD, and IHT are $(y)C(y{)}^{\mathrm{\top }}$ with $(y)=\mathrm{c}\mathrm{o}\mathrm{v}[Z,\kappa (Y,y)],$ $E\left[Z{Z}^{\mathrm{\top }}{\kappa }_c(Y,y)\right]$, and $W(y)W(y)$ with $W(y)=\left(C(y),H(y)C(y),\dots ,H(y{)}^{r}C(y)\right)$ and $H(y)=E\left[Z{Z}^{\mathrm{\top }}{\kappa }_c(Y,y)\right]$, respectively.

4.2. Ensembled forward regression estimators via CMS ensembles

We adopt the OPG (Xia et al. 2002), a popular method for estimating the classical CMS based on nonparametric forward regression, to the estimation of the Fréchet central subspace, which does not require LCM and CCV conditions. The adaption of another forward regression method MAVE is similar and presented in Section S.3.2 of the Supplementary Material. The framework of the statistical functional $M_0\left(F_{XY},f\right)$ is no longer sufficient to cover this case because we now have a tuning parameter here. So, we adopt the notion of tuned statistical functional in Section 11.2 of Li (2018) to accommodate a tuning parameter.

Let ${𝒫}_{XY},F_{XY},F_{XY}^{(0)}$ and ${\hat{F}}_{XY}^{(n)}$ be as defined in Section 4.1. For simplicity, we assume the tuning parameter $h$ to be a scalar, but it could also be a vector. Given a CMS-ensemble $\mathfrak{F}$, let $T_0:{𝒫}_{XY}\times \mathfrak{F}\times \mathbb{R}\to {\mathbb{R}}^{p\times p}$ be a tuned functional to be used as an estimator of ${𝒮}_{E[f(Y)\mid X]}$ for each $f$. We refer to $T_0$ as the ensemble-tuned candidate matrix. The unbiasedness, exhaustiveness, and Fisher consistency of $T_0$ are defined as follows.

DEFINITION 3.

We say that $T_0$ is unbiased for estimating $\left\{{𝒮}_{E[f(Y)\mid X]}:f\in \mathfrak{F}\right\}$ if, for each $f\in \mathfrak{F}$, $\text{span}\{\underset{h\to 0}{\text{lim}}{T}_0(F_{XY}^{(0)},f,h)\}\subseteq {𝒮}_{E[f(Y)\mid X]}$. Exhaustiveness and Fisher consistency of $T_0$ are defined by replacing ⊆ in the above by ⊇ and =, respectively.

Given $\mathfrak{F}=\left\{\kappa (\cdot ,y):y\in {\mathrm{\Omega }}_Y\right\}$ and $T_0$, we define the tuned Fréchet candidate matrix $T:{𝒫}_{XY}\times \mathbb{R}\to {\mathbb{R}}^{p\times p}$ as $T\left(F_{XY},h\right)={\int }_{{\mathrm{\Omega }}_Y} T_0\left(F_{XY},\kappa (S,y),h\right)d{F}_Y(y)$. We say that the estimate $T(T_{XY}^{(n)},h)$ is unbiased if $\text{span}(\underset{h\to 0}{\text{lim}}T(F_{XY}^{(0)},h))\subseteq {𝒮}_{Y\mid X}$, exhaustive if $\text{span}(\underset{h\to 0}{\text{lim}}T(F_{XY}^{(0)},h))\supseteq {𝒮}_{Y\mid X}$, and Fisher consistent if $\text{span}(\underset{h\to 0}{\text{lim}}T(F_{XY}^{(0)},h))={𝒮}_{Y\mid X}$.

In the following, for a function $h(x)$, we use $\partial h(X)/\partial X$ to denote $\partial h(x)/\partial x$ evaluated at $x=X$. The OPG aims to estimate central mean subspace ${𝒮}_{E[\kappa (Y,y)\mid X]}$ by $E\left[\frac{\partial E(\kappa (Y,y)\mid X)}{\partial X}\frac{\partial E(\kappa (Y,y)\mid X)}{\partial X^T}\right]$ where the gradient $\partial E(\kappa (Y,y)\mid X)/\partial X$ is estimated by local linear approximation as follows. Let $K_0:\mathbb{R}\to [0,\mathrm{\infty })$ be a kernel function as used in kernel estimation. For any $v\in {\mathbb{R}}^p$ and bandwidth $h>0$, let $K_h(v)=h^{-p}{K}_0(\parallel v\parallel /h)$. At the population level, for fixed $x\in {\mathrm{\Omega }}_x$ and $y\in {\mathrm{\Omega }}_Y$, we minimize the objective function

$$E\{{[\kappa (Y,y)-a-b^{\mathrm{\top }}(X-x)]}^2{K}_h(X-x)\}/E{K}_h(X-x)$$ (4)

over all $a\in \mathbb{R}$ and $b\in {\mathbb{R}}^{d_0}$. The minimizer depends on $x,y$ and we write it as $(a_h(x,y),\left.b_h(x,y)\right)$. The ensemble tuned candidate matrix for estimating the central mean subspace ${𝒮}_{E[\kappa (Y,y)\mid X]}$ is $T_0\left(F_{XY},\kappa (\cdot ,y),h\right)=E[b_h(X,y)b_h(X,y{)}^{\mathrm{\top }}]$ and the tuned Fréchet candidate matrix is $T\left(F_{XY},h\right)=E[b_h(X,Y)b_h(X,Y{)}^{\mathrm{\top }}]$.

At the sample level, we minimize, for each $j,k=1,\dots ,n$, the empirical objective function

$$\sum _{i=1}^n{w}_h(X_i,X_j){\left[{\kappa }_{\gamma }(Y_i,Y_k)-a_{jk}-b_{jk}^{\top }(X_i-X_j)\right]}^2$$ (5)

over $a_{jk}\in \mathbb{R}$ and $b_{jk}\in {\mathbb{R}}^p$, where $w_h(X_i,X_j)=K_h(X_i-X_j)/\sum _{l=1}^n{K}_h(X_l-X_j)$. Following Xia et al. (2002), we take the bandwidth to be $h=c_0{n}^{-1/\left(p_0+6\right)}$ where $p_0=\mathrm{m}\mathrm{a}\mathrm{x}\{p,3\}$ and $c_0=2.34$, which is slightly larger than the optimal $n^{-1/(p+4)}$ in terms of the mean integrated squared errors. As proposed in Li (2018, Lemma 11.6), instead of solving $b_{jk}$ from (5) $n^2$ times, we solve multivariate weighted least squares to obtain $b_{j1},\dots ,b_{jn}$ simultaneously. The tuned Fréchet candidate matrix is then estimated by $\hat{T}({\hat{F}}_{XY}^{(n)},h)=n^{-2}\sum _{j,k=1}^n{\hat{b}}_{jk}{\hat{b}}_{jk}^{\top }$. The first $d$ eigenvectors of $\hat{T}({\hat{F}}_{XY}^{(n)},h)$ form an estimate of the Fréchet central subspace.

We can further enhance the performance by projecting the original predictors onto the directions produced by $\hat{T}({\hat{F}}_{XY}^{(n)},h)$ to re-estimate ${𝒮}_{Y\mid X}$. Specifically, after compute $\hat{T}({\hat{F}}_{XY}^{(n)},h)$, we form the matrix $\hat{B}=\left({\hat{v}}_1,\dots ,{\hat{v}}_d\right)$ to contain the first $d$ eigenvectors of $\hat{T}({\hat{F}}_{XY}^{(n)},h)$. We then replace the kernel $K_h\left(X_j-X_i\right)$ by $K_h({\hat{B}}^{\mathrm{\top }}\left(X_j-X_i\right))$ with an updated bandwidth $h$, and complete ${\hat{b}}_{jk}$ from (5) again, which leads to an updated $\hat{B}$. We then iterate this process until convergence. In this way, we reduce the dimension of the kernel from $p$ to $d_0$ and mitigate the “curse of dimensionality.” For classical SDR problems, this procedure is called refined OPG, see Li (2018, Chapter 11.4). We call this refined estimator Fréchet OPG or FOPG. The algorithm for FOPG is summarized as Algorithm 2 in the Supplementary Material.

4.3. Ensembled inverse regression estimators via CS ensembles

In this subsection, we adapt several well-known estimators for the classical central subspace to Fréchet SDR, which include SIR (Li, 1991), SAVE (Cook and Weisberg, 1991), and DR (Li and Wang, 2007). We use the CS-ensemble to combine these classical estimates through (3). Let $\mathfrak{F}=\left\{\kappa (\cdot ,y):y\in {\mathrm{\Omega }}_y\right\}$ be a CS ensemble, where $\kappa$ is a cc-universal kernel. Let $M_0:{𝒫}_{XY}\times \mathfrak{F}\to {\mathbb{R}}^{p\times p}$ be a CS-ensemble candidate matrix. Let $M\left(F_{XY}\right)=\int M_0\left(F_{XY},\kappa (\cdot ,y)\right)d{F}_Y(y)$ be the Fréchet candidate matrix.

Again, we work with the standard predictor $Z$. The candidate matrices $M_0(y)$ for ensemble SIR, SAVE, and DR are formulated in Remark 2. Detailed motivation for each can be found in Li (2018, Chapter 3,5,6). At the sample level, we replace any unconditional moment $E$ by the sample average $E_n$, and replace any conditional moment, such as $E(Z\mid \kappa (Y,y))$, by the slice mean. The algorithms are also included in Algorithm 1.

REMARK 2.

The candidate matrices $M_0(y)$ for Fréchet SIR, SAVE, and DR are $\mathrm{v}\mathrm{a}\mathrm{r}[E(Z\mid \kappa (Y,y)]$, ${\left[I_p-\mathrm{v}\mathrm{a}\mathrm{r}(Z\mid \kappa (Y,y))\right]}^2$, and $2E{\left\{E\left[Z{Z}^{\mathrm{\top }}\mid \kappa (Y,y)\right]\right\}}^2+2E^2\left\{E[Z\mid \kappa (Y,y)]E\left[Z^{\mathrm{\top }}\mid \kappa (Y,y)\right]\right\}+2E\left\{E\left[Z^{\mathrm{\top }}\mid \kappa (Y,y)\right]E[Z\mid \kappa (Y,y)]\right\}E\left\{E[Z\mid \kappa (Y,y)]E\left[Z^{\mathrm{\top }}\mid \kappa (Y,y)\right]\right\}-2I_p$, respectively.

REMARK 3.

Regarding the time complexity of the Fréchet SDR methods, by construction, the ensemble estimator requires $n$ times the computing time of the original estimator because it needs to reapply the original estimator for each $\kappa \left(\cdot ,y_i\right),i=1,\dots ,n$. For example, if SAVE is used as the original estimator, then the largest matrix multiplication is $A_{p\times n}{B}_{n\times p}$ which requires $p^{2}n$ basic computation units; the largest matrix to invert or eigendecomposition to perform is a $p\times p$ matrix, which requires $p^3$ basic computation units. So the net computation complexity is $n\times \mathrm{m}\mathrm{a}\mathrm{x}\left(O\left(n{p}^2\right),O\left(p^3\right)\right)$.

4.4. Fisher consistency

In this subsection, we establish the unbiasedness and Fisher consistency of the tuned Fréchet candidate matrix. As a special case, the Fréchet candidate matrix constructed by any moment-based methods in Section 4.1 can be considered as tuned Fréchet candidate matrix with the tuning parameter $h$ taken to be 0. The next theorem shows that if $T_0$ is unbiased (or Fisher consistent), then $T$ is unbiased (or Fisher consistent). In the following, we say that a measure $\mu$ on ${\mathrm{\Omega }}_Y$ is strictly positive if and only if for any nonempty open set $U\subseteq {\mathrm{\Omega }}_Y,\mu (U)>0$. For a matrix $A,\parallel A\parallel$ represents the operator norm.

THEOREM 2.

Suppose $\mathfrak{F}=\left\{\kappa (\cdot ,y):y\in {\mathrm{\Omega }}_Y\right\}$ is a CMS-ensemble, where $\kappa$ is a cc-universal kernel. We have the following results regarding unbiasedness and Fisher consistency for $T$.

1. If $T_{0}isunbiasedfor\left\{{𝒮}_{E[\kappa (Y,y)\mid X]}:f\in \mathfrak{F}\right\}$ and $\parallel T_0(F_{XY}^{(0)},\kappa \left(\cdot ,Y^{\prime }\right),h)\parallel \le G\left(Y^{\prime }\right)$, where $G\left(Y^{\prime }\right)$ is a real-valued function with $E\left[G\left(Y^{\prime }\right)\right]<\mathrm{\infty }$, then $T$ is unbiased for ${𝒮}_{Y\mid X}$;

2. If (a) $T_0$ is Fisher consistent for $\left\{{𝒮}_{E[\kappa (Y,y)\mid X]}:f\in \mathfrak{F}\right\}$, (b) $T_0\left(F_{XY},\kappa (\cdot ,y),h\right)$ is positive semidefinite for each $y\in {\mathrm{\Omega }}_Y,h\in \mathbb{R}$ and $F_{XY}\in {𝒫}_{XY}$, (c) $\text{lim}\underset{h\to 0}{\text{sup}}{T}_0(F_{XY}^{(0)},\kappa (\cdot ,Y^{\prime }),h)\le G(Y^{\prime })$ with $E\left[G\left(Y^{\prime }\right)\right]<\mathrm{\infty }$, (d) $F_Y$ is strictly positive on ${\mathrm{\Omega }}_Y$, and (e) the mapping $y^{\prime }\mapsto \underset{h\to 0}{\text{lim}}{T}_0(F_{XY},\kappa (\cdot ,y^{\prime }),h)$ is continuous, then $T$ is Fisher consistent for ${𝒮}_{Y\mid X}$.

We similarly develop Fisher consistency for Fréchet SDR based on the CS-ensemble, including methods in Section 4.3. The next corollary says that if $M_0$ is Fréchet consistent for $\left\{{𝒮}_{\kappa (Y,y)\mid X}:y\in {\mathrm{\Omega }}_Y\right\}$, then $M$ is Fréchet consistent for ${𝒮}_{Y\mid X}$. The proof is similar to that of Theorem 2 and is omitted.

COROLLARY 2.

Suppose $\mathfrak{F}=\left\{\kappa (\cdot ,y):y\in {\mathrm{\Omega }}_Y\right\}$ is a CS-ensemble, where $\kappa$ is a cc-universal kernel. We have the following results regarding unbiasedness and Fisher consistency for $M$.

1. If $M_0$ is unbiased for $\left\{{𝒮}_{\kappa (Y,y)\mid X}:f\in \mathfrak{F}\right\}$, then $M$ is unbiased for ${𝒮}_{Y\mid X}$;

2. If $M_0$ is Fisher consistent for $\left\{{𝒮}_{\kappa (Y,y)\mid X}:f\in \mathfrak{F}\right\},M_0\left(F_{XY},\kappa (\cdot ,y)\right)$ is positive semidefinite for each $y\in {\mathrm{\Omega }}_X$ and $F_{XY}\in {𝒫}_{XY}$, $F_Y$ is strictly positive, and the mapping $y^{\prime }\mapsto M_0\left(F_{XY},\kappa \left(\cdot ,y^{\prime }\right)\right)$ is continuous, then $M$ is Fisher consistent for ${𝒮}_{Y\mid X}$.

Unbiasedness and Fisher consistency of $T_0$ or $M_0$ are satisfied by different sets of sufficient conditions for the moment-based or forward-regression-based estimators. We outline these conditions below.

1. For ensembled moment estimators in Section 4.1 and ensembled inverse regression estimators in Section 4.3, most of them are unbiased under either the LCM assumption or both the LCM and CCV assumption. For example, the unbiasedness of SIR, OLS, and IHT requires the LCM assumption, whereas the unbiasedness of SAVE, DR, and PHD requires both the LCM and the CCV assumptions. The estimators SIR, OLS, IHT, and PHD are generally not exhaustive (recall that unbiased along with exhaustiveness is equivalent to Fisher consistency). But sufficient conditions for SAVE and DR to be exhaustive are reasonably mild (see Li and Wang (2007) and Li (2018, Chapter 6)).

2. Sufficient conditions for Fisher consistency for OPG are given in Li (2018, Section 11.2). Specifically, it requires: (a) the smooth kernel function $K_0$ is a spherically-contoured p.d.f. with finite fourth moments; (b) the p.d.f. of $X$ is supported on ${\mathbb{R}}^p$ and has continuous bounded second derivatives. Note that neither LCM nor CCV assumption is needed for the OPG estimator.

In practice, when we observe a severe violation of the elliptical assumption among the predictors, for example, by exploratory data analysis tools such as the scatter plot matrix, it is favorable to use forward regression methods such as FOPG. Otherwise, moment-based methods are recommended since they are faster to compute and have a parametric ($n^{-1/2}$) convergence rate. Our experiences also indicate that the performance of the moment-based methods is relatively robust against the violation of ellipticity as long as it is not very severe.

5. Convergence Rates of the Ensemble Estimates

In this section, we develop the convergence rates of the ensemble estimates for Fréchet SDR. To save space, we will only consider the CMS-ensemble; the results for the CS-ensemble are largely parallel. To simplify the asymptotic development, we slightly modify the ensemble estimator, which does not result in any significant numerical difference from the original ensembles developed in the previous sections. For each $i=1,\dots ,n$, let ${\hat{F}}_{XY}^{(-i)}$ be the empirical distribution based on the sample with $i$th subject removed: $\left\{\left(X_1,Y_1\right),\dots ,\left(X_n,Y_n\right)\right\}\backslash \left\{\left(X_i,Y_i\right)\right\}$. Our modified ensemble estimate is of the form

$$T({\hat{F}}_{XY}^{(n)},h_n)=n^{-1}{\sum }_{i=1}^n T_0({\hat{F}}_{XY}^{(-i)},\kappa \left(\cdot ,Y_i\right),h_n).$$

The purpose of this modification is to break the dependence between the ensemble member $\kappa \left(\cdot ,Y_i\right)$ and the CMS estimate, which substantially simplifies the asymptotic argument. Here, we let the tuning parameter $h_n$ depend on $n$. Again, the Fréceht candidate matrix constructed by moment-based methods can be considered as a special case with $h_n=0$.

Rather than deriving the convergence rate of each individual ensemble estimate case by case, we will show that, under some mild conditions, the ensemble convergence rate is the same as the corresponding CMS-estimate’s rate. Since the convergence rates of many CMS-estimates are well established, including all the forward regression and sample moment-based estimates mentioned earlier, our general result covers all the CMS-ensemble estimates.

In this following, for a matrix $A,\parallel A\parallel$ represents the operator norm and $\parallel A{\parallel }_{\mathrm{F}}$ the Frobenius norm. If $\left\{a_n\right\}$ and $\left\{b_n\right\}$ are sequences of positive numbers, we write $a_n\prec b_n$ if $\underset{n\to \infty }{\text{lim}}{a}_n/b_n=0$; we write $a_n{b}_n$ if $a_n/b_n$ is a bounded sequence. We write $b_n\succ a_n$ (or $b_n⪰a_n$) if $a_n\prec b_n$ (or $a_n{b}_n$). We write $a_n\asymp b_n$ if $a_n{b}_n$ and $b_n{a}_n$. Let $T_0^{\text{*}}(F_{XY}^{(0)},\kappa (\cdot ,y))=\underset{h\to 0}{\text{lim}}{T}_0(F_{XY}^{(0)},\kappa (\cdot ,y),h)$ and $T^{\text{*}}(F_{XY}^{(0)})=\underset{h\to 0}{\text{lim}}T(F_{XY}^{(0)},h)$.

THEOREM 3.

Let $C_n(y)=E\parallel T_0({\hat{F}}_{XY}^{(n)},\kappa (\cdot ,y),h_n)-T_0^{\mathrm{*}}(F_{XY}^{(0)},\kappa (\cdot ,y))\parallel$ and $\left\{a_n\right\}$ be a positive sequence of numbers satisfying $a_{n+1}/a_n\asymp 1$ and $a_n⪰n^{-1/2}$. Suppose the entries of $T_0^{\mathrm{*}}(F_{XY}^{(0)},\kappa (\cdot ,Y))$ have finite variances. If $E\left[C_n(y)\right]=O\left(a_n\right)$, then $\parallel T({\hat{F}}_{XY}^{(n)},h_n)-T^{\mathrm{*}}(F_{XY}^{(0)})\parallel =O_P\left(a_n\right)$.

The above theorem says that, under some conditions, the convergence rate of an ensemble Fréchet SDR estimator is the same as the corresponding CMS estimator. This covers all the estimators developed in Section 4. Specifically:

1. For all moment-based ensemble methods, such as OLS, PHD, IHT, SIR, SAVE, DR, the ensemble candidate matrices can be written in the form $T_0({\hat{F}}_{XY}^{(n)},\kappa (\cdot ,y))=\stackrel{\mathrm{ˆ}}{\mathrm{\Lambda }}(y)\stackrel{\mathrm{ˆ}}{\mathrm{\Lambda }}(y{)}^{\mathrm{\top }}$, where $\stackrel{\mathrm{ˆ}}{\mathrm{\Lambda }}(y)$ is a matrix possessing the second order von Mises expansion, implying $E\left[C_n(y)\right]=O\left(n^{-1/2}\right)$. See, for example, Li (2018)

2. For nonparametric forward regression ensemble methods, OPG, the convergence rate of $C_n(y)$ was reported in Xia et al. (2002) as $O\left(h_n^2+h_n^{-1}{\delta }_n^2\right)$ where ${\delta }_n=\sqrt{(\mathrm{l}\mathrm{o}\mathrm{g}n)/n{h}_n^p}$. Although the convergence was established in terms of convergence in probability, under mild conditions such as uniformly integrability, we can obtain the same rate for $E\left[C_n(y)\right]$.

6. Simulations

We evaluate the performance of the proposed Fréchet SDR methods with distributions and symmetric positive definite matrices as responses. For space consideration, the additional simulation for spherical data is presented in the Supplementary Material.

6.1. Computational details

Choice of tuning parameters and kernel types.

We first implement a unified cross-validation procedure to select the kernel type and bandwidth $\gamma$ in the kernel. For both distributional response and symmetric positive definite matrix response, we consider Gaussian radial basis kernel ${\kappa }_{\mathrm{G}}\left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma d{\left(y,y^{\prime }\right)}^2)$ and Laplacian radial basis kernel ${\kappa }_{\mathrm{L}}\left(y,y^{\prime }\right)$ as candidates to construct the ensembles. For the bandwidth $\gamma$, we set the default value as

$${\gamma }_{\text{G}}=\frac{{\rho }_Y}{2{\sigma }_{\text{G}}^2},\text{where}{\sigma }_{\text{G}}^2={\left(\begin{array}{l}n\hfill \\ 2\hfill \end{array}\right)}^{-1}\sum _{i<j}d{(Y_i,Y_j)}^2,{\rho }_Y=1,$$ (6)

in the Gaussian radial basis kernel, and

$${\gamma }_{\text{L}}=\frac{{\rho }_Y}{2{\sigma }_L},\text{where}{\sigma }_{\text{L}}={\left(\begin{array}{l}n\hfill \\ 2\hfill \end{array}\right)}^{-1}\sum _{i<j}d(Y_i,Y_j),{\rho }_Y=1,$$

in the Laplacian radial basis kernel. The same choices were used in Lee et al. (2013) and Li and Song (2017). We then fine-tune ${\rho }_Y$ and kernel types together via the $k$-fold cross-validation as follows. Randomly split the whole sample into $k$ subsets of roughly equal sizes, say $D_1,\dots ,D_k$. For each $i=1,\dots ,k$, use $D_i$ as the test set and its complement as the training set. We first use the training set to implement the Fréchet SDR with an initial dimension $d$, say 5. This choice of a relatively large dimension helps to guarantee the unbiasedness of the estimated Fréchet central subspace. We then substitute the estimated $\hat{\beta }$ into the testing set to produce the sufficient predictor ${\hat{\beta }}^{T}X$ and then fit a global Fréchet regression model (Petersen and Müller, 2019) to predict the response in the testing set. Compute the prediction error for each $i$ and aggregate the error for all rotations $i=1,\dots ,k$, which yields an overall cross-validation error. This overall error depends on the tuning parameter ${\rho }_Y$ and kernel type, and is then minimized over a grid $\left\{{10}^{-2},{10}^{-1},1,10\right\}\times \left\{{\kappa }_{\mathrm{G}},{\kappa }_{\mathrm{L}}\right\}$ to obtain the optimal combinations.

Estimation of the dimensions.

For the ensemble estimators that possess a candidate matrix (such as the ensemble moment estimators in Section 4.1), the recently developed order-determination methods, such as the ladle estimate (Luo and Li, 2016), and predictor augmentation estimator (Luo and Li, 2021) can be directly applied to estimate $d_0$. In addition, the BIC-criterion introduced by Zhu et al. (2006) can also be used for this purpose.

In this paper, we adapted the predictor augmentation estimator to the current setting. A detailed introduction of the predictor augmentation method and more simulation results are included in the Supplementary Material. For the predictor augmentation estimator, we take the times of augmentations $s=10$ and the dimension of augmented predictors $r=\lceil p/2\rceil$, where $p$ is the original dimension of predictors.

Estimation error assessment.

We used the error measurement for subspace estimation as in Li et al. (2005): if ${𝒮}_1$ and ${𝒮}_2$ are two subspaces of ${\mathbb{R}}^p$ of the same dimension, then their distance is defined as $d\left({𝒮}_1,{𝒮}_2\right)={∥P_{{𝒮}_1}-P_{{𝒮}_2}∥}_{\mathrm{F}}$, where $P_{𝒮}$ is the projection on to $𝒮$, and $\parallel \cdot {\parallel }_{\mathrm{F}}$ is the Frobenius matrix norm. If $B_1$ and $B_2$ are two matrices whose columns form bases of ${𝒮}_1$ and ${𝒮}_2$ respectively, this distance can be equivalently written as ${\parallel B_1{\left(B_1^{\mathrm{\top }}{B}_1\right)}^{-1}{B}_1^{\mathrm{\top }}-B_2{\left(B_2^{\mathrm{\top }}{B}_2\right)}^{-1}{B}_2^{\mathrm{\top }}\parallel }_{\mathrm{F}}$. This distance is coordinate-free, as it is invariant to the basis matrices involved.

To facilitate the comparison, we also include the benchmark error, which is set as the expectation of the above distance when $B_1$ is taken as any basis matrix of the true central subspace, and entries of $B_2$ are generated randomly from i.i.d. $N(0,1)$. This expectation is computed by Monte Carlo with 1000 repeats.

6.2. Scenario I: Fréchet SDR for distributions

Let $\left({\mathrm{\Omega }}_Y,d_{\mathrm{w}}\right)$ be the metric space of univariate distributions endowed with Wasserstein metric $d_{\mathrm{w}}$, as described in Section 3. The construction of the ensembles requires computing the Wasserstein distances $d_{\mathrm{w}}\left(Y_i,Y_j\right)$ for $i,j=1,\dots ,n$. However, the distributions $Y_1,\dots ,Y_n$ are usually not fully observed in practice, which means we need to estimate them in the implementation of the proposed methods. There are multiple ways to do so, such as by estimating the c.d.f.’s, the quantile functions (Parzen, 1979), or the p.d.f’s (Petersen and Müller, 2016; Chen et al., 2021). For computation simplicity, we use the Wasserstein distances between the empirical measures. Specifically, suppose we observe $(X_1,{\left\{W_{1j}\right\}}_{j=1}^{m_1}),\dots ,(X_n,{\left\{W_{nj}\right\}}_{j=1}^{m_n})$, where ${\left\{W_{ij}\right\}}_{j=1}^{m_i}$ are independent samples from the distribution $Y_i$. Let ${\hat{Y}}_i$ be the empirical measure ${m_i}^{-1}\sum _{j=1}^{m_i}{\delta }_{w_{ij}}$, where ${\delta }_a$ is the Dirac measure at $a$, then we estimate $d_{\mathrm{w}}\left(Y_i,Y_k\right)$ by $d_{\mathrm{w}}(Y_i,{\hat{Y}}_k)$. For the theoretical justification, see Fournier and Guillin (2015) and Lei (2020). For simplicity, we assume the sample sizes $m_1,\dots ,m_n$ to be the same and denote the common sample size by $m$. Then the distance between empirical measures ${\hat{Y}}_i$ and ${\hat{Y}}_k$ is a simple function of the order statistics: $d_{\text{w}}({\hat{Y}}_i,{\hat{Y}}_k)={\{{\sum _{j=1}^m(W_{i(j)}-W_{k(j)})}^2\}}^{1/2}$, where $W_{i(j)}$ is the $j$-th order statistics of the sample $W_{i1}\dots ,W_{im}$.

Let ${\beta }_1^{\mathrm{\top }}=(1,1,0,\dots ,0)$, ${\beta }_2^{\mathrm{\top }}=(0,\dots ,0,1,1)$, ${\beta }_3^{\mathrm{\top }}=(1,2,0,\dots ,0,2)$ and ${\beta }_4^{\mathrm{\top }}=(0,0,1,2,2,\dots ,0)$. To generate univariate distributional response $Y$, we let $Y=N\left({\mu }_Y,{\sigma }_Y^2\right)$, where ${\mu }_Y$ and ${\sigma }_Y^2$ are random variables dependent on $X$, and ${\sigma }_Y>0$ almost surely, defined by the following models:

I-1 : ${\mu }_Y\mid X\sim N\left(\mathrm{e}\mathrm{x}\mathrm{p}\left({\beta }_1^{\mathrm{\top }}X\right),1\right)$ and ${\sigma }_Y=1$.

I-2 : ${\mu }_Y\mid X\sim N\left(\mathrm{e}\mathrm{x}\mathrm{p}\left({\beta }_1^{\mathrm{\top }}X\right),1\right)$ and ${\sigma }_Y={10}^{-1}\cdot 1\left\{\varsigma \left(X\right)<{10}^{-1}\right\}+\varsigma \left(X\right)\cdot 1\left\{{10}^{-1}\le \varsigma \left(X\right)\le 10\right\}+10\cdot 1\{\varsigma \left(X\right)>10\}$ where $\varsigma \left(X\right)=\text{exp}\left({\beta }_2^{\top }X\right)$.

To generate the predictor $X$, we consider both scenarios where Assumption 1 is satisfied and violated. Specifically, for Model I-1 and I-2, $X$ is generated by the following two scenarios:

1. $X\sim N(0,1)$; in this case both LCM and CCV in Assumption 1 are satisfied;

2. we first generate $U_1,\dots ,U_p$ from the $\mathbb{A}\mathbb{R}(1)$ model with mean 0 and covariance matrix $\mathrm{\Sigma }={({0.5}^{|i-j|})}_{i,j}$, and then generate $X$ by $\left(\mathrm{s}\mathrm{i}\mathrm{n}\left(U_1\right),\left|U_2\right|,U_3,\dots ,U_p\right)$. For this model, both LCM and CCS are violated.

Ying and Yu (2022) considered similar models to Model I-1 and Model I-2. For Model I-1, $B_0={\beta }_1$ and $d_0=1$; for Model I-2, $B_0=\left({\beta }_1,{\beta }_2\right)$ and $d_0=2$. In the simulation, we first generate $X_1,\dots ,X_n$, then generate $\left({\mu }_{Y_1},{\sigma }_{Y_1}\right),\dots ,\left({\mu }_{Y_n},{\sigma }_{Y_n}\right)$. For each $i=1,\dots ,n$, we then generate $W_{i1},\dots ,W_{im}$ independently from $N({\mu }_{Y_i},{\sigma }_{Y_i}^2)$. We take $(n,p)=(200,10),(400,20)$ and $m=100$.

We compare performances of the CMS ensemble methods and CS ensemble methods described in Section 4, including FOLS, FPHD, FIHT, FSIR, FSAVE, FDR, and FOPG (with refinement). We first implement the predictor augmentation (PA) method to estimate the dimension of the Fréchet central subspace. Then with estimated $\hat{d}$, we evaluate the estimation error. For FOPG, the number of iterations is set as 5, which is large enough to guarantee numerical convergence. For FSIR, FSAVE, the number of slices is chosen as $\lfloor n/2p\rfloor$; for FDR, the number of slices is chosen as $\lfloor n/6p\rfloor$. We also implement the weighted inverse regression ensemble (WIRE) method proposed by Ying and Yu (2022) for comparison. We repeat the experiments 500 times and summarize the proportion of correct identification of order and the mean and standard deviation of estimation error in Table 1. A smaller distance indicates a more accurate estimate, and the estimate with the smallest distance is shown in boldface. The benchmark distances are shown at the bottom of the table.

Table 2:

Mean(± standard deviation) of estimation error measured by ${∥P_{B_0}-P_{\hat{B}}∥}_{\mathrm{F}}$ for different methods for Scenario II. The benchmark for Model II-1 with $p=10,20$ are 1.334, 1.373 respectively, for Model II-2 with $p=10,20$ are 1.785, 1.893, respectively. The bold-faced number indicates the best performer.

Model	$(p,n)$	FOLS	FPHD	FIHT	FSIR	FSAVE	FDR	FOPG	WIRE
II-1-(a)		100%	71%	100%	99%	87%	98%	100%	100%
	(10,200)	0.157	0.865	0.157	0.170	0.299	0.171	0.154	0.152
		(0.055)	(0.288)	(0.055)	(0.093)	(0.285)	(0.138)	(0.041)	(0.038)
		100%	68%	100%	100%	92%	99%	100%	100%
	(20,400)	0.162	0.921	0.162	0.167	0.258	0.165	0.153	0.160
		(0.029)	(0.262)	(0.029)	(0.031)	(0.221)	(0.089)	(0.027)	(0.028)
II-1-(b)		99%	58%	99%	92%	51%	67%	97%	98%
	(10,200)	0.236	1.044	0.236	0.288	0.735	0.506	0.224	0.220
		(0.115)	(0.271)	(0.115)	(0.219)	(0.348)	(0.376)	(0.162)	(0.133)
		100%	52%	100%	96%	51%	70%	96%	99%
	(20,400)	0.235	1.126	0.235	0.260	0.726	0.472	0.233	0.222
		(0.047)	(0.245)	(0.047)	(0.155)	(0.336)	(0.364)	(0.157)	(0.09)
II-2-(a)		100%	20%	100%	100%	78%	99%	100%	100%
	(10,200)	0.292	1.20	0.292	0.306	0.615	0.358	0.151	0.290
		(0.078)	(0.155)	(0.078)	(0.088)	(0.285)	(0.122)	(0.05)	(0.078)
		100%	20%	100%	100%	85%	100%	100%	100%
	(20,400)	0.308	1.218	0.308	0.318	0.618	0.375	0.179	0.307
		(0.057)	(0.15)	(0.057)	(0.058)	(0.218)	(0.075)	(0.029)	(0.057)
II-2-(b)		99%	35%	98%	100%	58%	80%	98%	100%
	(10,200)	0.680	1.462	0.682	0.707	1.182	0.941	0.275	0.675
		(0.184)	(0.19)	(0.186)	(0.185)	(0.233)	(0.245)	(0.137)	(0.181)
		100%	42%	100%	100%	50%	92%	100%	100%
	(20,400)	0.694	1.505	0.694	0.710	1.228	0.933	0.336	0.691
		(0.128)	(0.187)	(0.128)	(0.126)	(0.189)	(0.194)	(0.079)	(0.128)

For Models I-1 and I-2, the best performer FOPG achieves 100% correct order determination percentage and enjoys the smallest estimation error. The moment-based ensemble methods are slightly less accurate than FOPG. Compared with the benchmark, all methods can successfully estimate the true central subspace except FPHD. Compared to the results from predictor settings (a) and (b), we see that most moment-based methods and inverse-regression-based methods have larger estimation error and less percentage of correct order determination under setting (b), but FOPG, which is free from the elliptical assumption of predictors, still give the most precise estimation. Overall, the correlation between predictors and non-ellipticity does not affect the results much compared with the benchmark error.

6.3. Scenario II: Fréchet SDR for positive definite matrices

Let ${\mathrm{\Omega }}_Y$ be the space ${\mathrm{s}\mathrm{y}\mathrm{m}}^{+}(r)$ endowed with Frobenius distance $d_{\mathrm{F}}\left(Y_1,Y_2\right)={∥Y_1-Y_2∥}_{\mathrm{F}}$. To accommodate the anatomical intersubject variability, Schwartzman (2006) introduced the symmetric matrix variate Normal distributions. We adopt this distribution to construct the regression model with correlation matrix response. We say that $Z\in \mathrm{S}\mathrm{y}\mathrm{m}(r)$ has the standard symmetric matrix variate Normal distribution $N_{rr}\left(0;I_r\right)$ if it has density $\phi (Z)=(2\pi {)}^{-q/2}\mathrm{e}\mathrm{x}\mathrm{p}(-\mathrm{t}\mathrm{r}(Z{)}^2/2)$ with respect to Lebesgue measure on ${\mathbb{R}}^{p(p+1)}$. As pointed out in Schwartzman (2006), this definition is equivalent to a symmetric matrix with independent $N(0,1)$ diagonal elements and $N(0,1/2)$ off-diagonal elements. We say $Y\in \mathrm{S}\mathrm{y}\mathrm{m}(r)$ has symmetric matrix variate Normal distribution $N_{rr}(M;\mathrm{\Sigma })$ if $Y=GZ{G}^T+M$ where $M\in \mathrm{S}\mathrm{y}\mathrm{m}(r)$, $G\in {\mathbb{R}}_{(r\times r)}$ is a non-singular matrix, and $\mathrm{\Sigma }=G^{T}G$. As a special case, we say $Y\in \mathrm{S}\mathrm{y}\mathrm{m}(r)\sim N_{rr}\left(M;{\sigma }^2\right)$ if $Y=\sigma Z+M$.

We generate predictors $X$ as in settings (a) and (b) of Scenario II. We generate $\mathrm{l}\mathrm{o}\mathrm{g}(y)$ following $N_{dd}(\mathrm{l}\mathrm{o}\mathrm{g}\{D(X)\},0.25)$, where $\mathrm{l}\mathrm{o}\mathrm{g}(\cdot )$ is the matrix logarithm defined in Section 3, and $D(X)$ is specified by the following models:

II-1: $(X)=\left(\begin{array}{cc}1& \rho (X)\\ \rho (X)& 1\end{array}\right)$, where $\rho \left(X\right)=\left[\text{exp}\left({\beta }_1^{\top }X\right)-1\right]/\left[\text{exp}\left({\beta }_1^{\top }X\right)+1\right]$.

II-2: $D(X)=\left(\begin{array}{ccc}1& {\rho }_1(X)& {\rho }_2(X)\\ {\rho }_1(X)& 1& {\rho }_1(X)\\ {\rho }_2(X)& {\rho }_1(X)& 1\end{array}\right)$, where ${\rho }_1\left(X\right)=0.4\left[\text{exp}\left({\beta }_1^{\top }X\right)-1\right]/\left[\text{exp}\left({\beta }_1^{\top }X\right)+1\right]$ and ${\rho }_2\left(X\right)=0.4\text{ sin}\left({\beta }_3^{\top }X\right)$.

In Model II-1, $B_0={\beta }_1$ and $d_0=1$; in Model II-2, $B_0=\left({\beta }_1,{\beta }_2\right)$ and $d_0=2$. We note that $D(x)$ is not necessarily the Fréchet conditional mean of $Y$ given $X$, but still measures the central tendency of the conditional distribution $Y\mid X$. We also compare performances of the CMS ensemble methods and CS ensemble methods, with $(n,p)=(200,10),(400,20)$. The experiments are repeated 500 times. The proportion of correct identification of order and the means and standard deviations of estimation errors are summarized in Table 2.

Table 3:

10-fold cross-validation prediction errors of GFR/LFR for mortality data

	GFR	LFR
9-dim full predictor	30.475	28.745
2-dim sufficient predictor	27.200	23.852

We conclude that all ensemble methods give reasonable estimation except FPHD. FOPG performs best in all settings except Model II-1-(b). To illustrate the relation between the response and estimated sufficient predictor ${\hat{\beta }}_1^{\mathrm{\top }}X$, we adopt the ellipsoidal representation of SPD matrices. Each $A\in {\mathrm{S}\mathrm{y}\mathrm{m}}^{+}(d)$ can be associated with an ellipsoid centered at the origin ${ℰ}_A=\left\{x:x^{\mathrm{\top }}{A}^{-1}x\le 1\right\}$. Figure 2 plots the responses ellipsoid versus the estimated sufficient predictor in panel (a), compared with the responses versus predictor $X_{10}$ for Model II-1-(a). We can tell a clear pattern of change in shape and rotation of response ellipsoids versus ${\hat{\beta }}_1^{\mathrm{\top }}X$.

Figure 2:

Ellipsoidal plots of the SPD matrix response versus the FOPG predictor ${\hat{\beta }}_1^{\mathrm{\top }}X$ and $X_{10}$ using Model II-1-(a) with $(n=200,p=10)$. Each horizontal ellipse is an Ellipsoidal representation of an SPD matrix, and the vertical axis is the value of (a) ${\hat{\beta }}_1^{\mathrm{\top }}X$; (b) $X_{10}$, colored according to the vertical axis values.

7. Application to the Human Mortality Data

This section presents an application concerning human life spans. Another application concerning intracerebral hemorrhage is presented in Section S. 6 of the Supplementary Material.

Compared with summary statistics such as the crude death rate, viewing the entire age-at-death distributions as data objects gives us a more comprehensive understanding of human longevity and health conditions. Mortality distributions are affected by many factors, such as economics, the health care system, and social and environmental factors. To investigate the potential factors that are related to the mortality distributions across different countries, we collect nine predictors listed below, covering demography, economics, labor market, nutrition, health, and environmental factors in 2015: (1) Population Density: population per square Kilometer; (2) Sex Ratio: number of males per 100 females in the population; (3) Mean Childbearing Age: the average age of mothers at the birth of their children; (4) Gross Domestic Product (GDP) per Capita; (5) Gross Value Added (GVA) by Agriculture: the percentage of agriculture, hunting, forestry, and fishing activities of gross value added, (6) Consumer price index: treat 2010 as the base year; (7) Unemployment Rate; (8) Expenditure on Health (percentage of GDP); (9) Arable Land (percentage of total land area). The data are collected from United Nation Databases (http://data.un.org/) and UN World Population Prospects 2019 Databases (https://population.un.org/wpp/Download). For each country and age, the life table contains the number of deaths $d(x,n)$ aggregated every five years. We treat these data as histograms of the number of deaths at age, with bin widths equal to 5 years. We smooth the histograms using the ‘frechet’ package available at (https://cran.r-project.org/web/packages/frechet/index.html) to obtain smoothed probability density functions and then calculate the Wasserstein distances between them. We collected the data for 152 countries in 2015 after removing 10 countries with extreme values in feature Population Density, Sex Ratio, CPI, and Expenditure on Health.

By checking the scatter plots matrix, we observe nonellipticity in the predictors. So we choose FOPG to analyze the data, which does not rely on Assumption 1. We use the Gaussian kernel $\kappa \left(y,y^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left(-\gamma d_w^2\left(y,y^{\prime }\right)\right)$ for the ensemble, where $\gamma$ is chosen according to (6) in Section 6.1. We standardize all covariates separately, then use the predictor augmentation method combined with FOPG to estimate the dimension of the Fréchet central subspace, which is estimated as 2. The first two directions obtained by FOPG are

$${\hat{\beta }}_1=(0.841,-0.155,-0.100,0.885,-0.361,-0.075,-0.108,0.214,-0.055{)}^{\mathrm{\top }},$$

$${\hat{\beta }}_2=(0.838,-0.706,-0.395,-0.424,-0.758,-0.005,-0.218,-0.102,-0.034{)}^{\mathrm{\top }}.$$

A plot of mortality densities versus the first sufficient predictor ${\hat{\beta }}_1^{T}X$ is shown in Figure 3(a). Clear and useful patterns emerge from Figure 3(a): the mode of the mortality distribution shifts from right to left (with left indicating a longer life span) as the value of the first sufficient predictor increases. Moreover, there is a significant uptick at the right-most end as the first sufficient predictor decreases, indicating high infant mortality. Meanwhile, the loadings of the first sufficieant predictor are strongly positive for the GDP per capita, which indicates the levels of economic development and health care of a country, with larger values associated with more developed countries and smaller values associated with less developed countries. From Figure 3(c), we see that the mean of the mortality distribution increases and the standard deviation decreases with the value of the first sufficient predictor. This also makes sense: the mean life span increases with the level of development, consistent with Figure 3(a). The standard deviation decreases with the first predictor because, as the development level increases, the life span is increasingly concentrated on the high values. Moreover, the high mortality in the lower region of the first sufficient predictor also contributes to the larger standard deviation in this region.

Figure 3:

(a) Mortality distributions versus the first sufficient predictors; (b) Cross-validation predicted mortality using sufficient predictors; (c) Mean and standard deviation of responses versus the first sufficient predictor.

We fit the global and local Fréchet regressions (GFR/LFR) with all nine predictors and two sufficient predictors, respectively. The 10 -fold cross-validation prediction errors are collected in the following Table 3. We see that using sufficient predictors achieves more accurate prediction results, especially in the local Fréchet regression model. The LFR gives more accurate predictions than the GFR, which indicates that the LFR is more flexible when a nonlinear regression pattern exists. This result is consistent with the recent findings in Bhattacharjee and Müller (2021). The predicted mortality densities versus the first sufficient predictor are shown in Figure 3(b). We also compared the cross-validation prediction errors by LFR using sufficient predictors of FOPG with those of FSIR and WIRE in Ying and Yu (2022), which require the linear conditional mean assumption. The 10 -fold cross-validation prediction error using WIRE and FSIR are 24.765 and 24.342, respectively. Compared with 23.852 by FOPG, we see that FOPG performs better than FSIR and WIRE, while the latter two perform similarly.

8. Discussion

In the classical regression setting, sufficient dimension reduction has been used as a tool for exploratory data analysis, regression diagnostics, and a mechanism to overcome the curse of dimensionality in regression. As a regression tool, it can help us to treat collinearity in the predictor effectively, detect heteroscedasticity in the response, find the most important linear combinations of predictors, and understand the general shape of the regression surface without fitting an elaborate regression model. Although regression with a metric-space valued random object is a new problem, as a regression problem, it shares the same set of issues, such as the need for exploratory analysis before regression, for model diagnostics after the regression, and for mitigating the curse of dimensionality. As shown in Figure 1 in the paper, the first sufficient predictor clearly reveals useful information about a general trend of mortality distributions among countries.

The proposed methodology is very flexible and versatile: it can be used to turn any existing SDR method into one that can deal with the metric-space-valued response variable. Furthermore, it applies to any separable and complete metric space of negative type with an explicit CMS ensemble. It significantly broadens the current field of sufficient dimension reduction and provides a useful set of tools for Fréchet regression.

The proposed method also has its limitations, one of which is that it only applies to metric spaces that permit the construction of a universal kernel. Another possible criticism is that the ensemble constructed by metric in the embedded Hilbert space is extrinsic to the original metric space. However, we do not regard this as a serious drawback for two reasons. First, the role played by the ensemble family is rather like that played by characteristic function, which need not be of the same nature as the original random variable. Second, in some important special cases (e.g., the Wasserstein space of univariate distributions and the space of SPD matrices), the embedding is isometric, so we are building the kernel using the original metric even though we work with the embedded Hilbert space. Nevertheless, when it is possible to use the original metric (such as in the isometric embedding case), it seems intuitively appealing to take that as our first choice, as we have done in all three examples.

Table 1:

The percentages of correct order determination, and the mean (standard deviation) of estimation error as measured by ${∥P_{B_0}-P_{\hat{B}}∥}_{\mathrm{F}}$ for Models I-1 and I-2 with settings (a) settings (a) and (b); the benchmark for Model I-1 with $p=10,20$ are 1.334, 1.373 respectively, and for Model I-2 with $p=10,20$ are respectively. The bold-faced number indicates the best performer.

Model	$(p,n)$	FOLS	FPHD	FIHT	FSIR	FSAVE	FDR	FOPG	WIRE
I-1-(a)		100%	97%	100%	100%	95%	97%	100%	100%
	(10,200)	0.334	0.593	0.341	0.260	0.437	0.336	0.167	0.236
		(0.088)	(0.158)	(0.09)	(0.081)	(0.199)	(0.144)	(0.054)	(0.057)
		100%	97%	100%	100%	97%	98%	100%	100%
	(20,400)	0.365	0.634	0.371	0.263	0.433	0.342	0.227	0.251
		(0.075)	(0.136)	(0.075)	(0.046)	(0.149)	(0.115)	(0.05)	(0.041)
I-1-(b)		99%	99%	99%	97%	95%	98%	100%	97%
	(10,200)	0.380	0.638	0.399	0.239	0.361	0.280	0.136	0.204
		(0.122)	(0.122)	(0.126)	(0.145)	(0.187)	(0.12)	(0.039)	(0.147)
		99%	99%	99%	98%	95%	99%	100%	97%
	(20,400)	0.387	0.648	0.404	0.237	0.365	0.275	0.194	0.211
		(0.098)	(0.096)	(0.094)	(0.123)	(0.176)	(0.092)	(0.053)	(0.151)
I-2-(a)		100%	91%	100%	100%	99%	100%	100%	100%
	(10,200)	0.409	1.032	0.412	0.370	0.528	0.413	0.267	0.304
		(0.11)	(0.254)	(0.109)	(0.082)	(0.134)	(0.09)	(0.112)	(0.061)
		100%	90%	100%	100%	100%	100%	100%	100%
	(20,400)	0.431	1.157	0.435	0.371	0.548	0.434	0.298	0.320
		(0.069)	(0.23)	(0.069)	(0.049)	(0.086)	(0.059)	(0.072)	(0.038)
I-2-(b)		100%	91%	100%	99%	100%	100%	100%	100%
	(10,200)	0.551	1.122	0.557	0.464	0.630	0.507	0.290	0.370
		(0.111)	(0.203)	(0.11)	(0.119)	(0.133)	(0.102)	(0.086)	(0.084)
		100%	91%	100%	100%	100%	100%	100%	100%
	(20,400)	0.561	1.179	0.567	0.458	0.645	0.521	0.330	0.381
		(0.081)	(0.164)	(0.08)	(0.072)	(0.089)	(0.07)	(0.051)	(0.058)