FUNCTIONAL SUFFICIENT DIMENSION REDUCTION THROUGH AVERAGE FRÉCHET DERIVATIVES

Abstract

Sufficient dimension reduction (SDR) embodies a family of methods that aim for reduction of dimensionality without loss of information in a regression setting. In this article, we propose a new method for nonparametric function-on-function SDR, where both the response and the predictor are a function. We first develop the notions of functional central mean subspace and functional central subspace, which form the population targets of our functional SDR. We then introduce an average Fréchet derivative estimator, which extends the gradient of the regression function to the operator level and enables us to develop estimators for our functional dimension reduction spaces. We show the resulting functional SDR estimators are unbiased and exhaustive, and more importantly, without imposing any distributional assumptions such as the linearity or the constant variance conditions that are commonly imposed by all existing functional SDR methods. We establish the uniform convergence of the estimators for the functional dimension reduction spaces, while allowing both the number of Karhunen–Loève expansions and the intrinsic dimension to diverge with the sample size. We demonstrate the efficacy of the proposed methods through both simulations and two real data examples.

1. Introduction.

Sufficient dimension reduction (SDR) embodies a family of methods in regressions that seek a low-dimensional representation of the high-dimensional data while minimizing the loss of regression information. Since the pioneering work of sliced inverse regression (Li (1991)), SDR has enjoyed a rapid development in the past three decades, and has been widely used in a variety of applications, including biology, finance, medical science, among others. In its full generality, SDR aims at the entire conditional distribution of Y|X, where Y is a response variable and X a p-dimensional predictor vector, and (X, Y) has a joint distribution. It seeks a lower-dimensional representation R(X) such that Y ╨ X | R(X). In plenty of applications, the regression interest focuses on the conditional mean E(Y|X) only. In that case, Cook and Li (2002) developed the notion of sufficient mean dimension reduction, by seeking R(X) such that Y ╨ E(Y|X) | R(X), or equivalently, E(Y|X) = E{Y|R(X)}. The reduction R(X) usually takes the form of linear combinations of X, that is, R(X) = β^⊤X, where β is a p × q matrix with q ≤ p. Sufficient reduction or sufficient mean reduction then pursues the minimum subspace spanned by β, which uniquely exists under very mild conditions (Yin, Li and Cook (2008)). Such a space is called the central subspace or the central mean subspace, and its dimension q is called the intrinsic dimension. There have been a large body of methods proposed for SDR, usually in a nonparametric fashion. Depending on their estimation strategies, broadly speaking, those methods can be grouped into two categories. One category utilizes the first and second moments of inverse regression X|Y, which under certain distributional assumptions, contain useful information about R(X); see, for example, Cook and Weisberg (1991), Li (1991), Li, Zha and Chiaromonte (2005), Li and Wang (2007), among others. The other category utilizes the gradient of the forward regression mean E(Y|X); see, for example, Hardle (1989), Xia (2007), Xia et al. (2002), Yin and Li (2011), Fukumizu and Leng (2014), among others. See Li (2018a) for a comprehensive review of SDR.

In this article, we target the problem of sufficient reduction and sufficient mean reduction when both the response and the predictor are a function. Function-on-function regression is receiving increasing attention in recent years, and is being widely used in applications such as environmental science, neuroimaging analysis, and e-commerce (Kim et al. (2018), Luo and Qi (2017), Müller and Yao (2008), Reimherr, Sriperumbudur and Taoufik (2018), Sun et al. (2018), Luo and Qi (2019)). We aim to relax the parametric or semiparametric model assumptions, and propose a new method of nonparametric function-on-function SDR. We first develop the notions of functional central mean subspace and functional central subspace, which are the population targets of our functional SDR inquiries. Motivated by the fact that the gradient of the regression mean function E(Y|X) lies in the central mean or central subspace (Xia et al. (2002), Xia (2007)), we extend the idea to the functional setting, and show that the Riesz representation of the Fréchet derivative of the regression functional is located in the functional central mean or central subspace. We then propose the corresponding functional dimension reduction estimators based on the average Fréchet derivative, and show that the resulting estimators are both unbiased and exhaustive. Moreover, our proposal leads naturally to a procedure for predicting any functional of the response after dimension reduction. Theoretically, we establish the uniform convergence for the estimated bases of the dimension reduction spaces, while allowing both the number of Karhunen–Loève expansions and the intrinsic dimension to diverge with the sample size.

Our proposal is closely related to but also clearly different from a number of lines of research on sufficient dimension reduction and functional regression. We next review the relevant literature and discuss the connections and differences with our proposal.

First, our proposal extends the gradient-based dimension reduction methods such as Hardle (1989), Xia (2007), Xia et al. (2002), Yin and Li (2011), Fukumizu and Leng (2014) from the random variable setting to the random function setting. However, such an extension is far beyond routine, and our new solution is utterly different, in both computation and theory, from the existing ones. On the computation side, most existing gradient-based methods approximated the high-dimensional gradients using iterative least squares, and the resulting computation can be intensive (Xia (2007), Xia et al. (2002), Yin and Li (2011)). To address this issue, we establish the interchangeability between the Fréchet derivative and the reproducing kernel Hilbert space (RKHS), which in effect provides a closed form of the Fréchet derivative. Consequently, our algorithm requires only spectral decompositions of linear operators and no iterative optimization, and hence is computationally much simpler. On the theory side, in the random variable setting, the variable dimension p and the intrinsic dimension q after dimension reduction are usually treated as fixed. Only recently, Lin et al. (2017) and Lin, Zhao and Liu (2019) established the consistency of sliced inverse regression with a diverging intrinsic dimension q. By contrast, in the random function setting, both the response and the predictor are random elements in a potentially infinite-dimensional Hilbert space. We need to handle a diverging number of Karhunen–Loève (KL) expansions from the random functions, and this portion of asymptotic analysis requires a more complex treatment than in the classical setting. Besides, we allow the intrinsic dimension q to diverge with the sample size.

Second, our work is also related to a family of proposals that extended the inverse regression-based SDR methods to the functional setting. The first generalization was introduced by Ferré and Yao (2003, 2005), in which they extended the predictor space from an Euclidean space to a Hilbert space, then proposed a functional slice inverse regression method. Hsing and Ren (2009), Jiang, Yu and Wang (2014), Wang, Lin and Zhang (2013), Wang et al. (2015), Yao, Lei and Wu (2015) further extended and developed a series of inverse regression-based estimators in the functional setting. All these methods involved a scalar response Y and a functional predictor X, targeted sufficient reduction of the full conditional distribution Y|X, and were based on some moments of the inverse regression X|Y. Our proposal on one hand extends SDR to the setting where not only the predictor but also the response is a function. More importantly, our method is built on the derivative of the forward regression, instead of the inverse regression. A critical implication of this difference is that, our functional SDR estimators achieve the unbiasedness and exhaustiveness without imposing distributional assumptions such as the linearity condition or the constant variance condition that are commonly imposed by all existing inverse moments-based functional SDR methods. As pointed out by Ma and Zhu (2012, 2013) under the random variable setting, those distributional assumptions can be restrictive, and our proposal is the first unbiased and exhaustive solution that relaxes those conditions under the random function setting.

More recently, Li and Song (2017) proposed a nonlinear SDR method for functional data. Our proposal is similar, in that we both target SDR for functional data, and both methods are built on some linear operators. However, the two also differ considerably in terms of methodology and theory. On the methodology side, Li and Song (2017) focused on nonlinear reduction, rather than linear reduction. Although nonlinear reduction is more flexible in the sense that its reduced predictors do not have to be linear, the linear framework of our proposal is still pivotally important. Linear SDR is usually easier to interpret, and is better connected with other modeling techniques, since it preserves the original coordinates of the predictors; see Li ((2018a), Chapter 14) for more comparison between linear and nonlinear SDR. Moreover, our estimator can not be directly induced by that of Li and Song (2017). On the theory side, Li and Song (2017) required the constant variance assumption to show their generalized sliced average variance estimator is unbiased, which can be restrictive. By contrast, our estimator does not impose such a condition. Furthermore, our asymptotic analysis requires a different set of tools from that of Li and Song (2017). Specifically, we employ the leading KL coefficients to approximate the sample covariance operators to accelerate the computation, which was not considered in Li and Song (2017). As a result, we need to take into account this extra layer of approximation when proving the consistency. More importantly, Li and Song (2017) treated the intrinsic dimension q as fixed, while we allow q to diverge, which leads to a completely different regime of asymptotics.

In summary, our proposal is profoundly different from the existing sufficient dimension reduction methods. The extension from random variables to random functions, and the change from inverse regression to forward regression-based estimation are both far from straightforward. It leads to a new set of methodological and theoretical results, and makes a useful addition to the toolbox of dimension reduction and functional data analysis.

The rest of the article is organized as follows. Section 2 introduces the functional central mean and central subspaces. Section 3 develops the average Fréchet derivative estimator for functional SDR, and establishes the population properties including the unbiasedness and exhaustiveness. Section 4 develops the estimation procedure, both at the operator level and under a coordinate system. Section 5 derives the asymptotic properties. Section 6 presents the simulations and two real data examples. Section 7 concludes the paper with a discussion. The Supplementary Material (Lee and Li (2022)) collects additional results and proofs.

2. Functional dimension reduction subspaces.

In this section, we first formally define the functional central mean and central subspaces, which are the population targets of functional SDR. We then introduce a series of linear operators useful for SDR estimation.

2.1. Functional central mean subspace and central subspace.

Let (Ω, ℱ, P) be a probability space. Let X : Ω → Ω_X, Y : Ω → Ω_Y be the Ω_X-valued and Ω_Y-valued random elements, where Ω_X and Ω_Y are Hilbert spaces of functions on an interval $T\subseteq ℝ$, and their inner products are ${\langle \cdot ,\cdot \rangle }_{{\Omega }_X}$ and ${\langle \cdot ,\cdot \rangle }_{{\Omega }_Y}$. Let P_X = P о X⁻¹ and P_Y = P о Y⁻¹ be the distributions of X and Y. We first seek sufficient reduction for the mean of the conditional distribution Y|X.

Definition 1. Suppose 𝒮 is a linear, closed subspace of Ω_X, and P_𝒮 : Ω_X → 𝒮 is the projection onto 𝒮. If, for all ψ ∈ Ω_Y,

$$E\left({\langle \psi ,Y\rangle }_{{\Omega }_Y}\mid X\right)=E\left({\langle \psi ,Y\rangle }_{{\Omega }_Y}\mid P_{𝒮}X\right),$$ (1)

we call 𝒮 a functional mean dimension reduction subspace. Let 𝒯 denote the collection of all 𝒮 satisfying (1). If 𝒮_E(Y|X) = ∩{𝒮 : 𝒮 ∈ 𝒯} is in 𝒯, we call 𝒮_E(Y|X) the functional central mean subspace.

We next seek sufficient reduction for the entire conditional distribution Y|X. The idea follows that of Xia (2007), Yin and Li (2011), in that one can extend the estimation of central mean subspace to that of central subspace through a class of functions that characterize the conditional distribution Y | X. More specifically, note that the collection of conditional means {E{g(Y)|X} : g ∈ ℋ_Y} characterize the full information of Y | X, as long as the class of functions ℋ_Y is sufficiently rich. This implies, if there is a functional dimensional reduction space for the conditional mean E{g(Y)|X} for all g ∈ ℋ_Y, then it is also a functional dimension reduction space for Y | X. We formalize this idea below.

Definition 2. Let ℋ_Y be a dense subset of L₂(P_Y) modulo constants. Suppose 𝒮 is a linear, closed subspace of Ω_X, and P_𝒮 : Ω_X → 𝒮 is the projection onto 𝒮. If, for all g ∈ ℋ_Y,

$$E\{g(Y)\mid X\}=E\left\{g(Y)\mid P_{𝒮}X\right\},$$ (2)

we call 𝒮 a functional dimension reduction subspace. Let 𝒯 denote the collection of all 𝒮 satisfying (2). If 𝒮_Y|X = ∩{𝒮 : 𝒮 ∈ 𝒯} is in 𝒯, we call 𝒮_Y|X the functional central subspace.

Here we say ℋ_Y is dense in L₂(P_Y) modulo constants if, for any g ∈ L₂(P_Y), there exists a sequence ${\left\{g^k\right\}}_{k\in \mathbb{N}}$ of ℋ_Y, such that E{g^k(X) − g(X)}² → 0, as k → ∞, where $\mathbb{N}=\left\{0,1,2,\dots \right\}$ denotes the collection of natural numbers. We also note that, if ℋ_Y is a dense subset of L₂(P_Y), ℋ_Y is characteristic (Fukumizu, Bach and Jordan (2009)). Therefore, (2) is equivalent to Y ╨ X | P_𝒮X. This justifies why (2) can characterize the conditional distribution Y | X. It also shows that 𝒮_Y|X is independent of the selection of ℋ_Y.

In both definitions of 𝒮_E(Y|X) and 𝒮_Y|X, we are seeking the smallest subspace in 𝒯 that satisfies (1) or (2), so to achieve maximal dimension reduction. Toward that goal, we take the intersection of all subspaces in 𝒯, and assume such an intersection still belongs to 𝒯. This actually holds under some mild conditions, as we show next. We introduce the concept of M-set in an Hilbert space.

Definition 3. Let Ω₁ and Ω₂ denote two generic Hilbert spaces, and M a subset of the product space Ω₁ × Ω₂. If, for any two pairs of points (ω₁, ω₂), $\left({\omega }_1^{*},{\omega }_2^{*}\right)\in M$, there exist a sequence of pairs of points ${\left\{\left({\omega }_1^j,{\omega }_2^j\right)\right\}}_{j=1}^J\in {\Omega }_1\times {\Omega }_2$, with $\left({\omega }_1^1,{\omega }_2^1\right)=\left({\omega }_1,{\omega }_2\right)$ and $\left({\omega }_1^J,{\omega }_2^J\right)=\left({\omega }_1^{*},{\omega }_2^{*}\right)$, such that, (a) $\left({\omega }_1^j,{\omega }_2^j\right)\in M$, for j = 2, …, J − 1, and (b) ${\omega }_1^{j+1}={\omega }_1^j$ or ${\omega }_2^{j+1}={\omega }_2^j$, for j = 1, …, J − 1, then we call M an M-set in Ω₁ × Ω₂.

Yin, Li and Cook (2008) introduced the notion of M-set for the random variable setting, and Definition 3 extends it to the random function setting. The conditions in Definition 3 require that any two points in an M-set can be connected via a “stairway”, and that only the corner points need to belong to the M-set. These are fairly mild conditions; see, for example, any convex and open subset of Ω₁ × Ω₂ is an M-set. Let supp(X) denote the support of X. For linear and closed subspaces 𝒮₁ and 𝒮₂ of Ω_X, and g a member in $\left\{P_{{𝒮}_1\cap {𝒮}_2}f:f\in \text{supp}(X)\right\}$, denote $\Omega \left({𝒮}_1,{𝒮}_2\right)=\left\{\left(P_{{𝒮}_1}f,P_{{𝒮}_2}f,P_{{𝒮}_1\cap {𝒮}_2}f\right):f\in \text{supp}(X)\right\}$, and $\Omega \left({𝒮}_1,{𝒮}_2,g\right)=\left\{\left(P_{{𝒮}_1}f,P_{{𝒮}_2}f\right):\left(P_{{𝒮}_1}f,P_{{𝒮}_2}f,g\right)\in \Omega \left({𝒮}_1,{𝒮}_2\right)\right\}$. Let 𝒯 be the collection of all subspaces satisfying (1) or (2). The next theorem justifies the existence of 𝒮_E(Y|X) and 𝒮_Y|X.

Theorem 1. Suppose, for any 𝒮₁ and 𝒮₂ in 𝒯, Ω(𝒮₁, 𝒮₂, g) is an M-set in 𝒮₁ × 𝒮₂, for all $g\in \left\{P_{{𝒮}_1\cap {𝒮}_2}f:f\in \text{supp}(X)\right\}$.

1. If the subspaces in 𝒯 satisfy (1), then there exists a unique intersection, 𝒮_E(Y|X) = ∩{𝒮 : 𝒮 ∈ 𝒯} ⊆ Ω_X, satisfying that 𝒮_E(Y|X) is in 𝒯, and 𝒮_E(Y|X) ⊆ 𝒮, for all 𝒮 ∈ 𝒯;

2. If the subspaces in 𝒯 satisfy (2), then there exists a unique intersection, 𝒮_Y|X = ∩{𝒮 : 𝒮 ∈ 𝒯 } ⊆ Ω_X, satisfying that 𝒮_Y|X is in 𝒯, and 𝒮_Y|X ⊆ 𝒮, for all 𝒮 ∈ 𝒯.

We also note that, the sufficient predictors in Definitions 1 and 2 are linear mappings on Ω_X. Consider full reduction as an example, and let 𝒮 = Span{ϕ_i ∈ Ω_X : i = 1…, q}. Then

$$Y╨X\mid {\langle X,{\varphi }_1\rangle }_{{\Omega }_X},\dots {\langle X,{\varphi }_q\rangle }_{{\Omega }_X}.$$

For this reason, our functional SDR is linear dimension reduction. By contrast, Li and Song (2017) studied nonlinear dimension reduction, by relaxing the linear constraint and allowing the sufficient predictors to be nonlinear mappings, in that,

$$Y╨X\mid f_1(X),\dots ,f_q(X),$$

where f₁, …, f_q are elements of the RKHS ℋ_X. This key difference leads to two sets of utterly different methodology and theory for our proposal and for Li and Song (2017), as detailed in Sections 1 and 7.

We next give some concrete examples of 𝒮_E(Y|X) and 𝒮_Y|X. For a positive definite kernel function ${\kappa }_T(\cdot ,\cdot ):ℝ\times ℝ\to ℝ$, let Ω_T denote the RKHS induced by κ_T, which can be constructed by ${\Omega }_T=\overline{\text{Span}}\left\{{\kappa }_T(\cdot ,t):t\in ℝ\right\}$, where Span is the closure of the space spanned by the set of functions. Next, consider the Brownian motion kernel, κ_T(t₁, t₂) = min(t₁, t₂), and let (α_j, β_j) denote the j th pair of eigenvalue and eigenfunction of κ_T, with α_j = 1/{(j − 0.5)π}², and β_j(t) = sin{(j − 0.5)πt}, $j\in \mathbb{N}$ (see, e.g., Amini and Wainwright (2012)). We then construct the predictor function X(t) and the error function ϵ(t) using the leading pairs of eigenvalues and eigenfunctions, where

$$X(t)=\sum _{j=1}^J{a}_j\sqrt{{\alpha }_j}{\beta }_j(t),ϵ(t)=\sum _{k=1}^K{b}_k\sqrt{{\alpha }_k}{\beta }_k(t),\text{\hspace{1em} for some integer values }J,K,$$

E(a_j) = E(b_k) = 0, j = 1, …, J, k = 1, …, K, and (a₁, …, a_J)^⊤ and (b₁, …, b_K)^⊤ are independent. By this construction, both ϵ(t) and X(t) are elements in Ω_T. Besides, for any ϕ ∈ Ω_T, $E{\langle \varphi ,ϵ(t)\rangle }_{{\Omega }_T}={\sum }_{k=1}^K\sqrt{{\alpha }_k}E\left(b_k\right){\langle f,b_k(t)\rangle }_{{\Omega }_T}=0$. Moreover, because there are one-to-one correspondences between X(t) and (a₁, …, a_J)^⊤, and between ϵ(t) and (b₁, …, b_K)^⊤, we have ϵ(t) ╨ X(t). Following this construction, we consider the following examples.

Example 1. Suppose there exist ϕ₁, ϕ₂ ∈ Ω_T, such that X(t) and Y(t) satisfy that

$$Y(t)=m_1\left({\langle {\varphi }_1,X\rangle }_{{\Omega }_T}\right)\text{sin}(0.5\pi t)+m_2\left({\langle {\varphi }_2,X\rangle }_{{\Omega }_T}\right)\text{sin}(1.5\pi t)+\sigma \epsilon (t),$$

where m₁(·) and m₂(·) are mappings from $ℝ$ to $ℝ$, σ is a positive constant, and ${\langle \varphi ,\psi \rangle }_{{\Omega }_T}=\left\{d\varphi (t)/dtd\psi (t)/dt\right\}dt$, for any ϕ, ψ ∈ Ω_T. Then Y is an Ω_T-valued random element, and 𝒮_E(Y|X) = Span{ϕ₁, ϕ₂}.

Example 2. Suppose there exist ϕ₁, ϕ₂, ϕ₃ ∈ Ω_T, such that X(t) and Y(t) satisfy that

$$Y(t)=m_1\left({\langle {\varphi }_1,X\rangle }_{{\Omega }_T}\right)\text{sin}(0.5\pi t)+m_2\left({\langle {\varphi }_2,X\rangle }_{{\Omega }_T}\right)\text{sin}(1.5\pi t)+m_3\left({\langle {\varphi }_3,X\rangle }_{{\Omega }_T}\right)\epsilon (t),$$

where m₁(·) to m₃(·) are mappings from $ℝ$ to $ℝ$, and ${\langle \varphi ,\psi \rangle }_{{\Omega }_T}=\int \left\{d\varphi (t)/dtd\psi (t)/dt\right\}dt$, for any ϕ, ψ ∈ Ω_T. Then Y is an Ω_T-valued random element, and 𝒮_Y|X = Span{ϕ₁, ϕ₂, ϕ₃}.

Finally, we note that it is straightforward to show 𝒮_E(Y|X) ⊆ 𝒮_Y|X. This echos the classical result in the random variable setting. In Example 2, 𝒮_E(Y|X) is a proper subset of 𝒮_Y|X.

2.2. Functional regression operator.

By Definitions 1 and 2, the subspaces of interest are defined via the conditional expectation $E\left({\langle \psi ,Y\rangle }_{{\Omega }_X}\mid X\right)$ or E{g(Y)|X}, that is, the regression functionals that link the predictor function X and the response function Y. In order to characterize such a regression relationship, we introduce the functional regression operator as an extension of regression coefficient to both functional and nonlinear settings.

We first define the nested kernel and the associated RKHS. A positive definite kernel ${\kappa }_X:{\Omega }_X\times {\Omega }_X\to ℝ$ can be constructed via the inner product ${\langle \cdot ,\cdot \rangle }_{{\Omega }_X}$, if there exists a mapping $\rho :{ℝ}^3\to ℝ$, such that

$${\kappa }_X\left({\varphi }_1,{\varphi }_2\right)=\rho \left({\langle {\varphi }_1,{\varphi }_1\rangle }_{{\Omega }_X},{\langle {\varphi }_1,{\varphi }_2\rangle }_{{\Omega }_X},{\langle {\varphi }_2,{\varphi }_2\rangle }_{{\Omega }_X}\right),\text{\hspace{1em} for any }{\varphi }_1,{\varphi }_2\in {\Omega }_X.$$ (3)

Because κ_X is uniquely characterized by Ω_X, it is called a nested kernel. Examples of nested kernels include the radial basis function kernel ${\kappa }_X\left({\varphi }_1,{\varphi }_2\right)=\text{exp}\left(-\gamma {\Vert {\varphi }_1-{\varphi }_2\Vert }_{{\Omega }_X}^2\right)$, the polynomial kernel ${\kappa }_X\left({\varphi }_1,{\varphi }_2\right)={\left(1+{\langle {\varphi }_1,{\varphi }_2\rangle }_{{\Omega }_X}\right)}^{\gamma }$, among others. Then given κ_X, a nested RKHS ℋ_X can be induced by this kernel. Similarly, a positive definite kernel ${\kappa }_Y:{\Omega }_Y\times {\Omega }_Y\to ℝ$, and a nested RKHS ℋ_X can be constructed via the inner product ${\langle \cdot ,\cdot \rangle }_{{\Omega }_Y}$.

We next introduce some background and notation. Let ℋ, ℋ′ be two generic Hilbert spaces, and A a linear mapping from ℋ to ℋ′. Define the norm of A as ‖A‖ = sup{‖Af‖_ℋ′ : f ∈ ℋ, ‖f‖_ℋ =1}, where ‖ · ‖_ℋ and ‖ · ‖_ℋ′ are the norms of ℋ and ℋ′, respectively. An operator is said to be bounded, if its norm is finite. A linear operator A : ℋ → ℋ′ is said to be Hilbert-Schmidt, if ${\sum }_{k\in \mathbb{N}}{\Vert A{f}^k\Vert }_{{\mathscr{H}}^{\prime }}^2<\infty$ for an orthonormal basis ${\left\{f^k\right\}}_{k\in \mathbb{N}}$ of ℋ. Define the Hilbert-Schmidt norm of A as $\Vert A{\Vert }_{\text{HS}}={\left({\sum }_{k\in \mathbb{N}}{\Vert A{f}^k\Vert }_{{\mathscr{H}}^{\prime }}^2\right)}^{1/2}$. Let B(ℋ, ℋ′) be the collection of all linear bounded operators from ℋ to ℋ′, and B₂(ℋ, ℋ′) the collection of all Hilbert-Schmidt operators from ℋ to ℋ′. Note that B(ℋ, ℋ′) is a Banach space endowed with the operator norm ‖ · ‖, and B₂(ℋ, ℋ′) is a Hilbert space with its inner product defined as ${\langle A_1,A_2\rangle }_{\text{HS}}={\sum }_{k\in \mathbb{N}}{\langle A_1{f}^k,A_2{f}^k\rangle }_{{\mathscr{H}}^{\prime }}$, for any A₁, A₂ ∈ B₂(ℋ, ℋ′). Because ‖A‖ ≤ ‖A‖_HS, it holds that B₂(ℋ, ℋ′) ⊆ B(ℋ, ℋ′) (Weidmann (1980)). We abbreviate B(ℋ) = B(ℋ, ℋ) and B₂(ℋ) = B₂(ℋ, ℋ) whenever appropriate. Furthermore, let ker(A), ran(A), $\overline{\text{ran}}(A)$, A* denote the null space, the range, the closure of the range, and the adjoint of an operator A, respectively.

We now develop a series of linear operators. We begin with a regularity condition. It ensures the square-integrability of the sample path of Y and that of every function in ℋ_X and ℋ_Y. The first part is a standard moment condition; see, for example, Yao, Lei and Wu (2015). The second part holds for all bounded kernels. To avoid digression, we defer a detailed discussion of this condition, along with all other conditions in this article, to Section S1 of the Supplementary Material (Lee and Li (2022)).

Assumption 1. Suppose $E\Vert Y{\Vert }_{\Omega \text{Y}}^2<\infty$, E{κ_X(X, X)} < ∞, and E{κ_Y(Y, Y)} < ∞. In addition, ℋ_X and ℋ_Y are dense in L₂(P_X) and L_Y(P_Y) modulo constants, respectively.

Let m_X denote the mean element of ℋ_X, such that ${\langle f,m_X\rangle }_{{\mathscr{H}}_X}=E{\langle f,{\kappa }_X(\cdot ,X)\rangle }_{{\mathscr{H}}_X}=E\left\{f(X)\right\}$ for any f ∈ ℋ_X. Similarly, let μ_Y and m_Y denote the mean elements of Ω_Y and ℋ_Y, respectively. We next define a number of covariance operators:

$${\Lambda }_{XY}:{\Omega }_Y\to {\mathscr{H}}_X,{\langle f,{\Lambda }_{XY}\psi \rangle }_{{\mathscr{H}}_X}=E\left\{{\langle f,{\kappa }_X(\cdot ,X)-m_X\rangle }_{{\mathscr{H}}_X}{\langle \psi ,Y-{\mu }_Y\rangle }_{{\Omega }_Y}\right\},$$

$${\Lambda }_{YY}:{\Omega }_Y\to {\Omega }_Y,{\langle {\psi }^{\prime },{\Lambda }_{YY}\psi \rangle }_{{\Omega }_Y}=E\left\{{\langle {\psi }^{\prime },Y-{\mu }_Y\rangle }_{{\Omega }_Y}{\langle \psi ,Y-{\mu }_Y\rangle }_{{\Omega }_Y}\right\},$$

$${\Sigma }_{XX}:{\mathscr{H}}_X\to {\mathscr{H}}_X,{\langle f^{\prime },{\Sigma }_{XX}f\rangle }_{{\mathscr{H}}_X}=E\left\{{\langle f^{\prime },{\kappa }_X(\cdot ,X)-m_X\rangle }_{{\mathscr{H}}_X}{\langle f,{\kappa }_X(\cdot ,X)-m_X\rangle }_{{\mathscr{H}}_X}\right\},$$

$${\Sigma }_{XY}:{\mathscr{H}}_Y\to {\mathscr{H}}_X,{\langle f,{\Sigma }_{XY}g\rangle }_{{\mathscr{H}}_Y}=E\left\{{\langle f,{\kappa }_X(\cdot ,X)-m_X\rangle }_{{\mathscr{H}}_X}{\langle g,{\kappa }_Y(\cdot ,Y)-m_Y\rangle }_{{\mathscr{H}}_Y}\right\},$$

$${\Sigma }_{YY}:{\mathscr{H}}_Y\to {\mathscr{H}}_Y,{\langle g^{\prime },{\Sigma }_{YY}g\rangle }_{{\mathscr{H}}_Y}=E\left\{{\langle g^{\prime },{\kappa }_Y(\cdot ,Y)-m_Y\rangle }_{{\mathscr{H}}_Y}{\langle g,{\kappa }_Y(\cdot ,Y)-m_Y\rangle }_{{\mathscr{H}}_Y}\right\},$$

for any ψ, ψ′ ∈ Ω_Y, f, f′ ∈ ℋ_X, and g, g′ ∈ ℋ_Y. Note that ${\Sigma }_{YX}={\Sigma }_{XY}^{*}$. The next lemma justifies the existence of the mean elements and covariance operators we define above.

Lemma 1. Suppose Assumption 1 holds. Then there exist the mean elements m_X, μ_Y, and m_Y, and the covariance operators Λ_XY, Λ_YY, Σ_XX, Σ_XY, Σ_YY.

Next, we define two functional regression operators,

$$m_{E(Y\mid X)}={\Sigma }_{XX}^{†}{\Lambda }_{XY},m_{Y\mid X}={\Sigma }_{XX}^{†}{\Sigma }_{XY},$$

where ${\Sigma }_{XX}^{†}:\text{ran}\left({\Sigma }_{XX}\right)\to \overline{\text{ran}}\left({\Sigma }_{XX}\right)$ is the Moore–Penrose inverse of Σ_XX. That is, for any f ∈ ran(Σ_XX), there exist g ∈ ker(Σ_XX) and $h\in \overline{\text{ran}}\left({\Sigma }_{XX}\right)$ such that f = Σ_XX(g + h), and the Moore–Penrose inverse ${\Sigma }_{XX}^{†}$ maps f ∈ Σ_XX to $h\in \overline{\text{ran}}\left({\Sigma }_{XX}\right)$ (Li (2018b)). Because they resemble the notion of classical regression coefficients, we refer m_E(Y|X) and m_Y|X as the functional regression operators. We next introduce another regularity condition to ensure such definitions. See Section S1 of the Supplementary Material (Lee and Li (2022)) for more discussion on this condition.

Assumption 2. Suppose ker(Σ_XX) = {0}, ran(Λ_XY) ⊆ ran(Σ_XX), and ran(Σ_XY) ⊆ ran(Σ_XX). In addition, m_E(Y|X) ∈ B₂(Ω_Y, ℋ_X), and m_Y|X ∈ B₂(ℋ_Y, ℋ_X).

Assumption 2 ensures that the mappings of Λ_XY and Σ_XY are both in that of Σ_XX, which in turns ensures that the two operators m_E(Y|X) and m_Y|X are well defined. Moreover, it also ensures that the Moore–Penrose inverse ${\Sigma }_{XX}^{†}$ is well defined.

The next proposition shows that the regression functionals $E\left({\langle \psi ,Y\rangle }_{{\Omega }_Y}\mid X\right)$ and $E\left\{{\langle g\text{,}{\kappa }_Y(\cdot ,Y)\rangle }_{\mathscr{H}Y}\mid X\right\}$ can be induced by the regression operators m_E(Y|X) and m_Y|X. Its proof is similar to Li and Song ((2017), Proposition 1) and is omitted.

Proposition 1. Suppose Assumptions 1 and 2 hold. Then for any ψ ∈ Ω_Y and g ∈ ℋ_Y, there exist constants c_ψ and c_g, such that

$$E\left({\langle \psi ,Y-{\mu }_Y\rangle }_{{\Omega }_Y}\mid X\right)=m_{E(Y\mid X)}\psi +c_{\psi },and$$

$$E\left\{{\langle g,{\kappa }_Y(\cdot ,Y)-m_Y\rangle }_{{\mathscr{H}}_Y}|X\right\}=m_{Y\mid X}g+c_g.$$

3. Average Fréchet derivative estimators.

In this section, we first establish the connection between the Fréchet derivative of the regression functional and the functional central mean or central subspace. Built on this connection, we introduce the average Fréchet derivative estimators, and develop their population-level properties.

3.1. Fréchet derivative of regression functionals.

We first derive a key property for our proposal: the Riesz representation of the Fréchet derivative of the regression functional is located within the dimension reduction spaces of interest. Let $M_{\psi }(\cdot )=E\left({\langle \psi ,Y\rangle }_{{\Omega }_Y}\mid X=\cdot \right)$ for any ψ ∈ Ω_Y, and M_g(·) = E{g(Y) | X = ·} for any g ∈ ℋ_Y, both of which are mappings from ${\Omega }_X\to ℝ$. Let ∂_xM_ψ(x, ·) and ∂_xM_g(x, ·) denote the Fréchet derivative of M_ψ and M_g, respectively, at any x ∈ Ω_X. If $A:\mathscr{H}\to ℝ$ is a bounded linear mapping, then, by Riesz representation theorem, there exists a unique g ∈ ℋ, such that Af = 〈g, f〉_ℋ for all f ∈ ℋ. In other words, g is the Riesz representation of A in ℋ (Conway (2010)).

Theorem 2. If M_ψ and M_g are Fréchet differentiable for any ψ ∈ Ω_Y and g ∈ ℋ_Y, then:

1. for any ψ ∈ Ω_Y, the Riesz representation of ∂_xM_ψ(x, ·) is in 𝒮_E(Y|X);

2. for any g ∈ ℋ_Y, the Riesz representation of ∂_xM_g(x, ·) is in 𝒮_Y|X.

Its proof is given in Section S4 of the Supplementary Material (Lee and Li (2022)). Theorem 2 shares the same spirit as the gradient-based SDR methods such as Xia et al. (2002), Yin and Li (2011), but is stated in a more general setting. It shows that the Riesz representations of the Fréchet derivatives ∂_xM_ψ(x, ·) and ∂_xM_g(x, ·) are elements of the functional dimension reduction spaces. Next, we show that such representations have closed forms.

We introduce an additional assumption, which is mild and is satisfied for many standard kernels κ_X, with the ρ function in (3) properly chosen, for example, the radial basis kernel and the polynomial kernel. In Section S1 of the Supplementary Material (Lee and Li (2022)), we further give the sufficient condition under which this assumption holds, and the explicit form of the derivative of κ_X for two commonly used kernels, the radial basis kernel and the polynomial kernel.

Assumption 3. κ_X(·, ·) is continuously differentiable.

We next establish the interchangeability between the Fréchet derivative and the reproducing kernel Hilbert space for the general case. Its proof is given in Section S4 of the Supplementary Material (Lee and Li (2022)).

Proposition 2. Suppose Assumption 3 holds. Then, for any f ∈ ℋ_X such that $f(x)=\langle f,{\kappa }_X(x,\cdot )\rangle {\Omega }_X$, x ∈ Ω_X, the Fréchet derivative of f(x) at x, ${\partial }_{x}f(x,\cdot ):{\Omega }_X\to ℝ$, u ↦ ∂_x f (x, u) is equal to ${\partial }_{x}f(x,u)={\langle f,{\partial }_x{\kappa }_X(x,u)\rangle }_{{\mathscr{H}}_X}$.

Steinwart and Christmann (2008) established the differentiability of RKHS in the random variable setting. Proposition 2 extends their result to the functional setting. See also Fukumizu and Leng (2014) for a simpler version of this result in the random variable setting. Note that the regression functionals M_ψ(·) and M_g(·) are members in ℋ_X, and thus by Proposition 2, their Fréchet derivatives ∂_xM_ψ(x, ·) and ∂_xM_g(x, ·) can be calculated using the derivative of κ_X. By Proposition 1, this further implies that, via the mappings of the regression operators, we can explicitly compute ∂_xM_ψ(x, ·) and ∂_xM_g(x, ·), as well as their Riesz representations. The next theorem summarizes the above statement. Hereafter, we write ∂_xκ(x, u) as {∂_xκ(x)} о u, and ∂_xκ (x) : Ω_X → ℋ_X is a member in B(Ω_X, ℋ_X).

Theorem 3. Suppose Assumptions 1 to 3 hold. Then, for any x ∈ Ω_X,

$$(a)\overline{\text{ran}}\left[{\left\{{\partial }_x{\kappa }_X(x)\right\}}^{*}{m}_{E(Y\mid X)}\right]\subseteq {𝒮}_{E(Y\mid X)};(\text{b})\overline{\text{ran}}\left[{\left\{{\partial }_x{\kappa }_X(x)\right\}}^{*}{m}_{Y\mid X}\right]\subseteq {𝒮}_{Y\mid X}.$$

Theorem 3 shows that $\overline{\text{ran}}\left[{\left\{{\partial }_x{\kappa }_X(x)\right\}}^{*}{m}_{E(Y\mid X)}\right]$ and $\overline{\text{ran}}\left[{\left\{{\partial }_X{\kappa }_X(x)\right\}}^{*}{m}_{Y\mid X}\right]$ are subspaces of the functional central mean and functional central subspaces, respectively, which provides the essential foundation for our functional SDR estimation. In particular, this result does not require any distributional or structural assumptions, such as the linearity condition or the constant variance condition, that are required in all existing inverse momentbased functional SDR methods (Ferré and Yao (2003), Hsing and Ren (2009), Jiang, Yu and Wang (2014), Wang, Lin and Zhang (2013), Yao, Lei and Wu (2015)). To gain further insight about Theorem 3, we observe that {∂_xκ_X(x)}*m_Y|X has an intuitive explanation that can be linked to the classical linear regression coefficient. Consider a linear kernel ${\kappa }_X={\langle \varphi ,{\varphi }^{\prime }\rangle }_{{\Omega }_X}$ for any ϕ, ϕ′ ∈ Ω_X. Suppose ${\Omega }_X={ℝ}^p$, ${\Omega }_Y=ℝ$, Y and X are centered and satisfy that E(Y|X = x) = 〈β, x〉 =β^⊤x with $\beta \in {ℝ}^p$. Then for any $x^{\prime }\in {ℝ}^p$, we have 〈{∂_xκ_X(x)}*m_E(Y|X), x′〉 = 〈β^⊤x, {∂_xκ_X (x)} о x′〉, which is equal to ${\langle {\beta }^{\top }x,{\left(x^{\prime }\right)}^{\top }x\rangle }_{{\mathscr{H}}_X}={\beta }^{\top }{x}^{\prime }$. This implies that {∂_xκ_X(x)}*m_E(Y|X) = β.

3.2. Average Fréchet derivative.

Built on Theorem 3, let B = {∂_xκ_X(x)}*m_E(Y|X) or {∂_xκ_X(x)}*m_Y|X. Because $\overline{\text{ran}}(B)=\overline{\text{ran}}\left(B{B}^{*}\right)$, we can define two mappings that map X in Ω_X to the random operators F_E(Y|X)(X) and F_Y|X(X) in B(Ω_X), respectively, via

$$F_{E(Y\mid X)}(x)={\left\{{\partial }_x{\kappa }_X(x)\right\}}^{*}{m}_{E(Y\mid X)}{m}_{E(Y\mid X)}^{*}\left\{{\partial }_x{\kappa }_X(x)\right\},$$

$$F_{Y\mid X}(x)={\left\{{\partial }_x{\kappa }_X(x)\right\}}^{*}{m}_{Y\mid X}{m}_{Y\mid X}^{*}\left\{{\partial }_x{\kappa }_X(x)\right\}.$$

Note that the adjoints of m_E(Y|X) and m_Y|X are defined on ran(Σ_XX) instead of ℋ_X. Nonetheless, because m_E(Y|X) and m_Y|X are both bounded, their domains can be extended to $\overline{\text{ran}}\left({\Sigma }_{XX}\right)$, which, by Assumption 2, is equal to ℋ_X. Next, we establish the existence of the expectations of F_E(Y|X)(X) and F_Y|X(X). We first need a condition on the moments of the Fréchet derivative of κ_X. In Section S1 of the Supplementary Material (Lee and Li (2022)), we give further results based on which we can show this moment condition holds. We also derive the explicit bounds of E‖∂_xκ_X(X)‖⁴ for the radial basis kernel and the polynomial kernel.

Assumption 4. E‖∂_xκ_X(X)‖⁴ < ∞.

The next proposition establishes the existence of the expectations of F_E(Y|X)(X) and F_Y|X(X). Its proof immediately follows by the representation theorem, and is omitted.

Proposition 3. Suppose Assumptions 1 to 4 hold. Then the mean elements of F_E(Y|X)(X) and F_Y|X(X) in B(Ω_X), denoted by E{F_E(Y|X)} and E(F_Y|X), respectively, uniquely exist via the following relations: for any ϕ, ϕ′ ∈ Ω_X,

$${\langle \varphi ,E\left\{F_{E(Y\mid X)}\right\}{\varphi }^{\prime }\rangle }_{{\Omega }_X}=E\left[{\langle \varphi ,\left\{F_{E(Y\mid X)}(X)\right\}{\varphi }^{\prime }\rangle }_{{\Omega }_X}\right],$$

$${\langle \varphi ,E\left(F_{Y\mid X}\right){\varphi }^{\prime }\rangle }_{{\Omega }_X}=E\left[{\langle \varphi ,\left\{F_{Y\mid X}(X)\right\}{\varphi }^{\prime }\rangle }_{{\Omega }_X}\right].$$

The two operators, E{F_E(Y|X)} and E(F_Y|X), share the same spirit as the average derivative estimator of Hardle (1989) in the classical random variable SDR setting, and are generalized to the functional setting. For this reason, we refer E{F_E(Y|X)} and E(F_Y|X) as the average Fréchet derivatives (AFD).

In SDR for random variables, we say a basis matrix is an unbiased estimator for the central subspace if the spanning space of this basis matrix is located within the central subspace. In addition, we say the basis matrix is exhaustive if the spanning space recovers the entire central subspace (Li and Wang (2007)). The same definitions apply to the central mean subspace. We now extend these concepts to the functional setting.

Definition 4. Let 𝒮 be a subspace of Ω_X.

1. We say 𝒮 is unbiased for 𝒮_E(Y|X) if 𝒮 ⊆ 𝒮_E(Y|X), and exhaustive for 𝒮_E(Y|X) if 𝒮 = 𝒮_E(Y|X);

2. We say 𝒮 is unbiased for 𝒮_Y|X if 𝒮 ⊆ 𝒮_Y|X, and exhaustive for 𝒮_Y|X if 𝒮 = 𝒮_Y|X.

Next, we show that E{F_E(Y|X)} and E(F_Y|X) are unbiased and exhaustive estimators for 𝒮_E(Y|X) and 𝒮_Y|X, respectively. We add another condition, and discuss it in detail in Section S1 of the Supplementary Material (Lee and Li (2022)).

Assumption 5. The supports, $\left(P_{{𝒮}_{E(Y\mid X)}}X\right)$ and $\text{supp}\left(P_{{𝒮}_{Y\mid X}}X\right)$, are convex.

Theorem 4. Suppose Assumptions 1 to 4 hold. Then:

1. Unbiasedness: $\overline{\text{ran}}\left[E\left\{F_{E(Y\mid X)}\right\}\right]\subseteq {𝒮}_{E(Y\mid X)}$, and $\overline{\text{ran}}\left[E\left(F_{Y\mid X}\right)\right]\subseteq {𝒮}_{Y\mid X}$.

2. Exhaustiveness: $\overline{\text{ran}}\left[E\left\{F_{E(Y\mid X)}\right\}\right]={𝒮}_{E(Y\mid X)}$, and $\overline{\text{ran}}\left[E\left\{F_{Y\mid X}\right\}\right]={𝒮}_{Y\mid X}$, if Assumption 5 further holds.

Theorem 4 suggests that we can use the space spanned by E{F_E(Y|X)} or E(F_Y|X) to estimate the functional central mean subspace or the functional central subspace. In other words, we can use the leading eigenfunctions of these operators to estimate the bases of 𝒮_E(Y|X) and 𝒮_Y|X. The next corollary summarizes this observation, which provides the estimators for our functional SDR spaces at the population level. To ensure that we can compute the two functional spaces in Theorem 4, $\overline{\text{ran}}\left[E\left(F_{E(Y\mid X)}\right)\right]$ and $\overline{\text{ran}}\left[E\left(F_{Y\mid X}\right)\right]$, at the sample level, we let the rank of E{F_E(Y|X)} and E{F_Y|X} equal q, while we allow q to diverge.

Corollary 1. Suppose the conditions in Theorem 4 hold.

1. If ${\varphi }_1^{\prime },\dots ,{\varphi }_q^{\prime }$ are the solutions to the sequential optimization problem

   $$\begin{gathered} \text{max}\left[{\langle \varphi ,E\left\{F_{E(Y\mid X)}\right\}\varphi \rangle }_{{\Omega }_X}:\varphi \in {\Omega }_X\right] \\ \mathit{\text{subject}}to\Vert \varphi {\Vert }_{{\Omega }_X}=1,{\langle \varphi ,{\varphi }_1^{\prime }\rangle }_{{\Omega }_X}=\cdots ={\langle \varphi ,{\varphi }_q^{\prime }\rangle }_{{\Omega }_X}=0, \end{gathered}$$

   then ${𝒮}_{E(Y\mid X)}=\text{Span}\left\{{\varphi }_1^{\prime },\dots ,{\varphi }_q^{\prime }\right\}$.

2. If ${\varphi }_1^{\prime },\dots ,{\varphi }_q^{\prime }$ are the solutions to the sequential optimization problem

   $$\begin{gathered} \text{max}\left[{\langle \varphi ,E\left\{F_{Y\mid X}\right\}\varphi \rangle }_{{\Omega }_X}:\varphi \in {\Omega }_X\right] \\ \mathit{\text{subject}}to\Vert \varphi {\Vert }_{{\Omega }_X}=1,{\langle \varphi ,{\varphi }_1^{\prime }\rangle }_{{\Omega }_X}=\cdots ={\langle \varphi ,{\varphi }_q^{\prime }\rangle }_{{\Omega }_X}=0, \end{gathered}$$

   then ${𝒮}_{Y\mid X}=\text{Span}\left\{{\varphi }_1^{\prime },\dots ,{\varphi }_q^{\prime }\right\}$.

4. Sample estimation.

In this section, we develop the functional SDR estimation procedure, first at the operator level, then under a coordinate system. We also develop a postreduction prediction method based on the proposed dimension reduction framework.

4.1. Estimation at the operator level.

Let ${\left\{{\left(X^k,Y^k\right)}^{\top }\right\}}_{k=1}^n$ denote i.i.d. samples of (X, Y)^⊤. We estimate the mean elements μ_Y, m_X and m_Y of Ω_Y, ℋ_X and ℋ_Y by ${\widehat{\mu }}_Y=E_n(Y)$, ${\widehat{m}}_X=E_n\left\{{\kappa }_X(\cdot ,X)\right\}$, and ${\widehat{m}}_Y=E_n\left\{{\kappa }_Y(\cdot ,Y)\right\}$, respectively, where E_n(·) is the mean operator such that $E_n(\omega )=n^{-1}{\sum }_{k=1}^n{\omega }^k$ for the samples (ω¹, …, ωⁿ). We estimate the three key covariance operators by

$${\widehat{\Lambda }}_{XY}=E_n\left[\left\{{\kappa }_X(\cdot ,X)-{\widehat{m}}_X\right\}\otimes \left(Y-{\widehat{\mu }}_Y\right)\right],$$

$${\widehat{\Sigma }}_{XY}=E_n\left[\left\{{\kappa }_X(\cdot ,X)-{\widehat{m}}_X\right\}\otimes \left\{{\kappa }_Y(\cdot ,Y)-{\widehat{m}}_Y\right\}\right],$$

$${\widehat{\Sigma }}_{XX}=E_n\left[\left\{{\kappa }_X(\cdot ,X)-{\widehat{m}}_X\right\}\otimes \left\{{\kappa }_X(\cdot ,X)-{\widehat{m}}_X\right\}\right],$$

where ⊗ represents the tensor product. Besides, we estimate Λ_YY and Σ_YY by ${\widehat{\Lambda }}_{YY}=E_n\left\{\left(Y-{\widehat{\mu }}_Y\right)\otimes \left(Y-{\widehat{\mu }}_Y\right)\right\}$, and ${\widehat{\Sigma }}_{YY}=E_n\left[\left\{{\kappa }_Y(\cdot ,Y)-{\widehat{m}}_Y\right\}\otimes \left\{{\kappa }_Y(\cdot ,Y)-{\widehat{m}}_Y\right\}\right]$, respectively. For any ψ, ψ′ ∈ Ω_Y, we have ${\langle \psi ,{\widehat{\Lambda }}_{YY}{\psi }^{\prime }\rangle }_{{\Omega }_Y}={\text{cov}}_n\left\{{\langle \psi ,Y\rangle }_{{\Omega }_Y},{\langle {\psi }^{\prime },Y\rangle }_{{\Omega }_Y}\right\}$, and the rest of the sample covariance operators have similar properties, where cov_n(·, ·) denotes the sample covariance between the designated quantities.

In the next section, we show that these sample covariance operators can be represented as functions of Gram kernel matrices, which often have fast decaying eigenvalues and can be well approximated by the spectral decomposition associated with only the leading eigenvalues (Lee and Huang (2007)). This suggests that using a reduced set of bases of RKHS can improve the estimation efficiency both statistically and computationally. We adopt this idea in our functional SDR estimation. Let ${\left\{\left({\lambda }^k,{\psi }^k\right)\right\}}_{k\in \mathbb{N}}$, ${\left\{\left(a^k,f^k\right)\right\}}_{k\in \mathbb{N}}$, and ${\left\{\left(b^k,g^k\right)\right\}}_{k\in \mathbb{N}}$ denote the pairs of eigenvalues and eigenfunctions of Λ_YY, Σ_XX, and Σ_YY, such that

$${\Lambda }_{YY}=\sum _{k\in \mathbb{N}}{\lambda }^k\left({\psi }^k\otimes {\psi }^k\right),{\Sigma }_{XX}=\sum _{k\in \mathbb{N}}{a}^k\left(f^k\otimes f^k\right),{\Sigma }_{YY}=\sum _{k\in \mathbb{N}}{b}^k\left(g^k\otimes g^k\right).$$

Let ${\left\{{\theta }^k,c^k,d^k\right\}}_{k\in \mathbb{N}}$, denote the Karhunen–Loève coefficients satisfying that,

$$Y-{\mu }_Y=\sum _{k\in \mathbb{N}}{\theta }^k{\psi }^k,{\kappa }_X(\cdot ,X)-m_X=\sum _{k\in \mathbb{N}}{c}^k{f}^k,{\kappa }_Y(\cdot ,Y)-m_Y=\sum _{k\in \mathbb{N}}{d}^k{g}^k.$$

That is, ${\theta }^k={\langle Y-{\mu }_Y,{\psi }^k\rangle }_{{\Omega }_Y}$, $c^k={\langle {\kappa }_X(\cdot ,X)-m_X,f^k\rangle }_{{\mathscr{H}}_X}$ and $d^k={\langle {\kappa }_Y(\cdot ,Y)-m_Y,g^k\rangle }_{{\mathscr{H}}_Y}$. By definition, the covariance operators Λ_XY, Σ_XX, and Σ_XY can be written as

$$\begin{gathered} {\Lambda }_{XY}=\sum _{k,\ell \in \mathbb{N}}\text{cov}\left(c^k,{\theta }^{\ell }\right)\left(f^k\otimes {\psi }^k\right),{\Sigma }_{XY}=\sum _{k,\ell \in \mathbb{N}}\text{cov}\left(c^k,d^{\ell }\right)\left(f^k\otimes g^{\ell }\right), \\ {\Sigma }_{XX}=\sum _k\text{var}\left(c^k\right)\left(f^k\otimes f^k\right). \end{gathered}$$ (4)

Similarly, at the sample level, let ${\left\{\left({\widehat{\lambda }}^k,{\widehat{\psi }}^k\right)\right\}}_{k\in \mathbb{N}}$, ${\left\{\left({\widehat{a}}^k,{\widehat{f}}^k\right)\right\}}_{k\in \mathbb{N}}$ and ${\left\{\left({\widehat{b}}^k,{\widehat{g}}^k\right)\right\}}_{k\in \mathbb{N}}$ denote the pairs of eigenvalues and eigenfunctions of ${\widehat{\Lambda }}_{YY}$, ${\widehat{\Sigma }}_{XX}$ and ${\widehat{\Sigma }}_{YY}$. Let ${\left\{{\widehat{\theta }}^k,{\widehat{c}}^k,{\widehat{d}}^k\right\}}_{k\in \mathbb{N}}$ denote the estimated KL coefficients, ${\widehat{\theta }}^k={\langle Y-{\widehat{\mu }}_Y,{\widehat{\psi }}^k\rangle }_{{\Omega }_Y}$, ${\widehat{c}}^k={\langle {\kappa }_X(\cdot ,X)-{\widehat{m}}_X,{\widehat{f}}^k\rangle }_{{\mathscr{H}}_X}$ and ${\widehat{d}}^k={\langle {\kappa }_Y(\cdot ,Y)-{\widehat{m}}_Y,{\widehat{g}}^k\rangle }_{{\mathscr{H}}_Y}$. To estimate the covariance operators in (4), we then truncate and focus on the leading d terms, and obtain the estimators via

$$\begin{gathered} {\widehat{\Lambda }}_{XY}^d=\sum _{k,\ell =1}^d{\text{cov}}_n\left({\widehat{c}}^k,{\widehat{\theta }}^{\ell }\right)\left({\widehat{f}}^k\otimes {\widehat{\psi }}^{\ell }\right),{\widehat{\Sigma }}_{XY}^d=\sum _{k,\ell =1}^d{\text{cov}}_n\left({\widehat{c}}^k,{\widehat{d}}^{\ell }\right)\left({\widehat{f}}^k\otimes {\widehat{g}}^{\ell }\right), \\ {\widehat{\Sigma }}_{XX}^d=\sum _{k=1}^d{\text{var}}_n\left({\widehat{c}}^k\right)\left({\widehat{f}}^k\otimes {\widehat{f}}^k\right), \end{gathered}$$ (5)

where var_n(ω) = cov_n(ω, ω).

Correspondingly, we estimate the functional regression operators m_E(Y|X) and m_Y|X by

$${\widehat{m}}_{E(Y\mid X)}^d={\left({\widehat{\Sigma }}_{XX}^d+ϵI\right)}^{-1}{\widehat{\Lambda }}_{XY}^d,{\widehat{m}}_{Y\mid X}^d={\left({\widehat{\Sigma }}_{XX}^d+ϵI\right)}^{-1}{\widehat{\Sigma }}_{XY}^d,$$ (6)

respectively, where ϵ is a ridge parameter, and I : ℋ_X → ℋ_X is the identify mapping. Then, by Corollary 1, we solve the optimization problem

$$\text{max}\left[{\langle \varphi ,E_n\left\{{\widehat{F}}^d(X)\right\}\varphi \rangle }_{{\Omega }_X}:\varphi \in {\Omega }_X\right],$$ (7)

where ${\widehat{F}}^d(x)={\widehat{F}}_{E(Y\mid X)}^d(x)={\left\{{\partial }_x{\kappa }_X(x)\right\}}^{*}{\widehat{m}}_{E(Y\mid X)}^d{\left({\widehat{m}}_{E(Y\mid X)}^d\right)}^{*}\left\{{\partial }_x{\kappa }_X(x)\right\}$ when estimating 𝒮_E(Y|X), and ${\widehat{F}}^d(x)={\widehat{F}}_{Y\mid X}^d(x)={\left\{{\partial }_x{\kappa }_X(x)\right\}}^{*}{\widehat{m}}_{Y\mid X}^d{\left({\widehat{m}}_{Y\mid X}^d\right)}^{*}\left\{{\partial }_x{\kappa }_X(x)\right\}$ when estimating 𝒮_Y|X. That is, we estimate 𝒮_E(Y|X) or 𝒮_Y|X by $\text{Span}\left\{{\widehat{\varphi }}_k:k=1,\dots ,q\right\}$, where ${\widehat{\varphi }}_k$ is the eigenfunction corresponding to the kth largest eigenvalue of $E_n\left\{{\widehat{F}}^d(X)\right\}$ in (7). Lastly, we estimate the sufficient predictors ${\langle {\varphi }_k,X\rangle }_{{\Omega }_X}$ by ${\langle {\widehat{\varphi }}_k,X\rangle }_{{\Omega }_X}$, for k = 1, …, q.

4.2. Estimation under a coordinate system.

In practice, a function can only be observed at a finite set of points. For simplicity, we focus on the balanced setting where all the subjects are observed under a common set of time points, and the observed data are {[X^k(t), Y^k(t)]^⊤ : t = t₁, …, t_m, k = 1, …, n}. Meanwhile, our method can be straightforwardly extended to the unbalanced setting with only minor modifications. We next develop an estimation procedure under a chosen coordinate system given the observed data.

We first introduce the notion of coordinate representation here, and give more details in Section S2.1 of the Supplementary Material (Lee and Li (2022)). Let ${\lfloor f\rfloor }_{𝓑}\in {ℝ}^m$ denote the coordinate of f in a generic Hilbert space ℋ with respect to its bases 𝓑. Moreover, let _𝓑′⌊A⌋_𝓑 denote the coordinate of a linear operator A : ℋ → ℋ′ with respect to 𝓑 and 𝓑′, where ℋ′ is another Hilbert space spanned by 𝓑′. For convenience, we write ⌊f⌋_𝓑 as ⌊f⌋, and _𝓑′⌊A⌋_𝓑 as ⌊A⌋ when there is no confusion.

We divide our coordinate-level estimation procedure into four major steps.

In Step 1, we first construct the RKHS, Ω_X and Ω_Y, which are Hilbert spaces of functions on an interval $T\subseteq ℝ$. We begin with a positive definite kernel function, ${\kappa }_T:ℝ\times ℝ\to ℝ$. There are many choices, for example, the Brownian motion kernel, or the radial basis kernel. Suppose Ω_X = Ω_Y is the linear span of a sequence of functional basis ${\left\{{\kappa }_T\left(\cdot ,t_S\right)\right\}}_{S=1}^m$.

Next, we derive the coordinate representation of the observed data X^k(t) and Y^k(t). Let X^k(τ) = {X^k(t₁), …, X^k(t_m)}^⊤, and Y^k(τ) = {Y^k(t₁), …, Y^k(t_m)}^⊤, for the kth subject, k = 1, …, n, and ${\left(K_T\right)}_{s,u=1}^m={\kappa }_T\left(t_s,t_u\right)$ be the m × m Gram matrix of T with the spectral decomposition, $K_T=U_T{D}_T{U}_T^{\top }$. It is then straightforward to obtain that

$${𝓑}_T(\cdot )=D_T^{-1/2}{U}_T^{\top }{\left[{\kappa }_T\left(\cdot ,t_1\right),\dots ,{\kappa }_T\left(\cdot ,t_m\right)\right]}^{\top }$$

is an orthonormal basis of Ω_X and Ω_Y. Therefore, we can write X^k(τ) and Y^k(τ) as $X^k(\tau )=U_T{D}_T^{1/2}{\lfloor X^k\rfloor }_{{𝓑}_T}$, and $Y^k(\tau )=U_T{D}_T^{1/2}{\lfloor Y^k\rfloor }_{{𝓑}_T}$, which further leads to the their coordinate representations,

$${\lfloor X^k\rfloor }_{{𝓑}_T}=D_T^{-1/2}{U}_T^{\top }{X}^k(\tau ),{\lfloor Y^k\rfloor }_{{𝓑}_T}=D_T^{-1/2}{U}_T^{\top }{Y}^k(\tau ).$$ (8)

Note that if K_T is not of a full rank, we simply use its leading nonzero eigenvalues and corresponding eigenfunctions to construct 𝓑_T.

In Step 2, we first construct the nested RKHS, ℋ_X and ℋ_Y, which are Hilbert spaces of functions on X and Y, respectively. We choose positive definite nested kernels κ_X and κ_Y, using, for example, the radial basis function kernel, or the Laplacian kernel. We then follow the coordinates derived in (8), and estimate the inner products by ${\langle X^k,X^{\ell }\rangle }_{{\Omega }_X}={\lfloor X^k\rfloor }_{{𝓑}_T}^{\top }{\lfloor X^{\ell }\rfloor }_{{𝓑}_T}$, and ${\langle Y^k,Y^{\ell }\rangle }_{{\Omega }_X}={\lfloor Y^k\rfloor }_{{𝓑}_T}^{\top }{\lfloor Y^{\ell }\rfloor }_{{𝓑}_T}$. Let ℋ_X and ℋ_Y be the linear span of ${\left\{{\kappa }_X\left(\cdot ,X^k\right)-E_n{\kappa }_X(\cdot ,X)\right\}}_{k=1}^n$ and ${\left\{{\kappa }_Y\left(\cdot ,Y^k\right)-E_n{\kappa }_Y(\cdot ,Y)\right\}}_{k=1}^n$, respectively.

Next, we construct the orthonormal bases of ℋ_X and ℋ_Y, following a similar way as in Step 1. That is, let ${\left(K_X\right)}_{k,\ell =1}^n={\kappa }_X\left(X^k,X^{\ell }\right)$ and ${\left(K_Y\right)}_{k,\ell =1}^n={\kappa }_Y\left(Y^k,Y^{\ell }\right)$ be the n × n Gram matrix of X and Y, respectively, and G_X = Q_nK_XQ_n and G_Y = Q_nK_YQ_n be their centered version, with $Q_n=I_n-n^{-1}{1}_n^{\top }{1}_n$ and 1_n being the n-dimensional vector of ones. We then perform the spectral decomposition, $G_X=U_X{D}_X{U}_X^{\top }+U_X^0{D}_X^0{\left(U_X^0\right)}^{\top }$, and $G_Y=U_Y{D}_Y{U}_Y^{\top }+U_Y^0{D}_Y^0{\left(U_Y^0\right)}^{\top }$, where $U_X{D}_X{U}_X^{\top }$ and $U_Y{D}_Y{U}_Y^{\top }$ correspond to the first d eigenvalues, and $U_X^0{D}_X^0{\left(U_X^0\right)}^{\top }$ and $U_Y^0{D}_Y^0{\left(U_Y^0\right)}^{\top }$ correspond to the last n − d eigenvalues. Therefore, we construct the orthonormal basis of ℋ_X and ℋ_Y, respectively, via

$$\begin{gathered} {𝒞}_X(\cdot )=D_X^{-1/2}{U}_X^{\top }{\left[{\kappa }_X\left(\cdot ,X^1\right)-{\widehat{m}}_X,\dots ,{\kappa }_X\left(\cdot ,X^n\right)-{\widehat{m}}_X\right]}^{\top }, \\ {𝒞}_Y(\cdot )=D_Y^{-1/2}{U}_Y^{\top }{\left[{\kappa }_Y\left(\cdot ,Y^1\right)-{\widehat{m}}_Y,\dots ,{\kappa }_Y\left(\cdot ,Y^n\right)-{\widehat{m}}_Y\right]}^{\top }. \end{gathered}$$ (9)

In Step 3, we derive the coordinate representation of the truncated sample covariance operators in (5), given the constructed Ω_X, Ω_Y, ℋ_X and ℋ_Y, and their orthonormal bases. Specifically, we perform the spectral decomposition on $E_n{\left({\lfloor Y^k-{\widehat{\mu }}_Y\rfloor }_{{𝓑}_T}{\lfloor Y^k-{\widehat{\mu }}_Y\rfloor }_{{𝓑}_T}\right]}^{\top })$, and denote its first d eigenvalues and eigenfunctions as ${\left\{\left({\widehat{\lambda }}^k,{\lfloor {\widehat{\psi }}^k\rfloor }_{{𝓑}_T}\right)\right\}}_{k=1}^d$. Let ${𝓑}_Y=\left\{{\widehat{\psi }}^k:k=1,\dots ,d\right\}$. We then compute the leading sample KL coefficients by, for k = 1, …, d, ℓ = 1, …, n,

$$\begin{gathered} {\widehat{\theta }}^{k,\ell }={\lfloor Y^{\ell }-{\widehat{\mu }}_Y\rfloor }_{{𝓑}_T}^{\top }{\lfloor {\widehat{\psi }}^k\rfloor }_{{𝓑}_T},{\widehat{c}}^{k,\ell }=e_k^{\top }\left\{{𝒞}_X\left(X^{\ell }\right)-E_n{𝒞}_X(X)\right\}, \\ {\widehat{d}}^{k,\ell }=e_k^{\top }\left\{{𝒞}_Y\left(Y^{\ell }\right)-E_n{𝒞}_Y(Y)\right\}, \end{gathered}$$

where e_k is the d-dimensional vector with its kth position being one and the rest being zero. Stack the sample KL coefficients θ^k,ℓ, ${\widehat{c}}^{k,\ell }$, and ${\widehat{d}}^{k,\ell }$ vertically, and form the n-dimensional vectors ${\widehat{\mathrm{\Theta }}}^k={\left({\widehat{\theta }}^{k,1},\dots ,{\widehat{\theta }}^{k,n}\right)}^{\top }$, ${\widehat{C}}^k={\left({\widehat{c}}^{k,1},\dots ,{\widehat{c}}^{k,n}\right)}^{\top }$, and ${\widehat{D}}^k={\left({\widehat{d}}^{k,1},\dots ,{\widehat{d}}^{k,n}\right)}^{\top }$, for k = 1, …, d. Next stack ${\widehat{\mathrm{\Theta }}}^k$, ${\widehat{C}}^k$ and ${\widehat{D}}^k$ horizontally, and form the n × d matrices $\widehat{\mathrm{\Theta }}=\left({\widehat{\mathrm{\Theta }}}^1,\dots ,{\widehat{\mathrm{\Theta }}}^d\right)$, $\widehat{C}=\left({\widehat{C}}^1,\dots ,{\widehat{C}}^d\right)$ and $\widehat{D}=\left({\widehat{D}}^1,\dots ,{\widehat{D}}^d\right)$. Then we obtain the coordinate representation of the truncated sample covariance operators in (5) as

$$\begin{gathered} {𝒞}_X{\lfloor {\widehat{\Lambda }}_{XY}^d\rfloor }_{{𝓑}_Y}=n^{-1}{\widehat{C}}^{\top }\widehat{\mathrm{\Theta }},{𝒞}_X{\lfloor {\widehat{\Sigma }}_{XY}^d\rfloor }_{{𝒞}_Y}=n^{-1}{\widehat{C}}^{\top }D, \\ {𝒞}_X{\lfloor {\widehat{\Sigma }}_{XX}^d\rfloor }_{{𝒞}_X}=n^{-1}{\widehat{C}}^{\top }\widehat{C}. \end{gathered}$$ (10)

In Step 4, we derive the coordinate representation of ${\widehat{F}}^d(X)$ in (7) as

$${𝓑}_T{\lfloor {\widehat{F}}^d(X)\rfloor }_{{𝓑}_T}=\{\begin{array}{l}{\lfloor {\partial }_x{\kappa }_X(X)\rfloor }^{\top }{\left\{\lfloor {\widehat{\Sigma }}_{XX}^d\rfloor +ϵI_d\right\}}^{-1}\lfloor {\widehat{\Lambda }}_{XY}^d\rfloor {\lfloor {\widehat{\Lambda }}_{XY}^d\rfloor }^{\top }{\left\{\lfloor {\widehat{\Sigma }}_{XX}^d\rfloor +ϵI_d\right\}}^{-1}\lfloor {\partial }_x{\kappa }_X(X)\rfloor \hfill \\ \text{\hspace{1em} for }{𝒮}_{E(Y\mid X)},\hfill \\ {\lfloor {\partial }_x{\kappa }_X(X)\rfloor }^{\top }{\left\{\lfloor {\widehat{\Sigma }}_{XX}^d\rfloor +ϵI_d\right\}}^{-1}\lfloor {\widehat{\Sigma }}_{XY}^d\rfloor {\lfloor {\widehat{\Sigma }}_{XY}^d\rfloor }^{\top }{\left\{\lfloor {\widehat{\Sigma }}_{XX}^d\rfloor +ϵI_d\right\}}^{-1}\lfloor {\partial }_x{\kappa }_X(X)\rfloor \hfill \\ \text{\hspace{1em} for }{𝒮}_{Y\mid X},\hfill \end{array}$$ (11)

where ⌊∂_xκ_X(X)⌋ is the coordinate of the Fréchet derivative, which we derive explicitly for the radial basis kernel and the polynomial kernel in Section S2.2 of the Supplementary Material (Lee and Li (2022)). We take the value of ϵ to be 10⁻⁴ × [D_X]_1,1. Finally, we compute the sample average $E_n\left\{{}_{{𝓑}_T}{⌊{\widehat{F}}^d(X)⌋}_{{𝓑}_T}\right\}$, perform its spectral decomposition, and obtain the eigenfunctions $\lfloor {\widehat{\varphi }}_1\rfloor ,\dots ,\lfloor {\widehat{\varphi }}_q\rfloor$ associated with the leading q eigenvalues. We estimate 𝒮_E(Y|X) or 𝒮_Y|X by $\text{Span}\left\{{\widehat{\varphi }}_1,\dots ,{\widehat{\varphi }}_q\right\}$, where ${\widehat{\varphi }}_i={\lfloor {\widehat{\varphi }}_i\rfloor }^{\top }\left\{{𝓑}_T(\cdot )\right\}$, i = 1, …, q..

We summarize the above estimation procedure in Algorithm 1.

Our estimation method requires specifying kernel functions for κ_T, κ_X, and κ_Y. We study the effect of kernel choices in Section S3 of the Supplementary Material (Lee and Li (2022)). In general, we have found that the estimated sufficient predictors are not overly sensitive to the choices of kernel functions.

In addition, we need to determine the number of leading KL coefficients d, and the intrinsic dimension q of the functional central subspace or central mean subspace. We adopt the commonly used strategy in principal component analysis (PCA) to select d and q, such that a certain proportion of total variation has been accounted for. Specifically, let tr(·) and λ_k(·) denote the trace and the kth largest eigenvalue of a designated symmetric matrix, and p_d and p_q denote some prespecified proportion, say, 90% or 95%. We then determine d and q by

$$\begin{gathered} d=\text{min}\left\{d^{\prime }\in \mathbb{N}:\frac{{\sum }_{k=1}^{d^{\prime }}{\lambda }_k\left(\lfloor {\widehat{\Sigma }}_{XX}\rfloor \right)}{\text{tr}\left(\lfloor {\widehat{\Sigma }}_{XX}\rfloor \right)}\ge p_d\text{ and }\frac{{\sum }_{k=1}^{d^{\prime }}{\lambda }_k\left(\lfloor {\widehat{\Sigma }}_{YY}\rfloor \right)}{\text{tr}\left(\lfloor {\widehat{\Sigma }}_{YY}\rfloor \right)}\ge p_d\right\}, \\ q=\text{min}\left\{q^{\prime }\in \mathbb{N}:\frac{{\sum }_{k=1}^{q^{\prime }}{\lambda }_k\left(E_n\lfloor {\widehat{F}}^d(X)\rfloor \right)}{\text{tr}\left(E_n\lfloor {\widehat{F}}^d(X)\rfloor \right)}\ge p_q\right\}. \end{gathered}$$ (12)

4.3. Postreduction prediction.

In function-on-function regression, in addition to finding the reduced-dimensional representation, it is of equal interest to predict the response function Y, or any of its functional g(Y) in ℋ_Y, given the predictor function X. Our proposed dimension reduction framework leads naturally to a procedure for such prediction tasks.

Let ${\widehat{\varphi }}_i$, i = 1, …, q, denote the estimates from Step 5 of Algorithm 1. Then, for any x ∈ Ω_X, its projections onto $\text{Span}\left\{{\widehat{\varphi }}_k:k=1,\dots ,q\right\}$ is $x_q={\sum }_{i=1}^q\left({\lfloor x\rfloor }^{\top }\lfloor {\widehat{\varphi }}_i\rfloor {\widehat{\varphi }}_i\right)$, where $\lfloor x_q\rfloor ={\left\{{\lfloor x\rfloor }^{\top }\lfloor {\widehat{\varphi }}_1\rfloor ,\dots ,{\lfloor x\rfloor }^{\top }\lfloor {\widehat{\varphi }}_q\rfloor \right\}}^{\top }$. Therefore, for any x, x′ ∈ Ω_X, we can define a new kernel ${\kappa }_X^q$ for the reduced predictors x_q via

$${\kappa }_X^q\left(x,x^{\prime }\right)=\rho \left({\langle x_q,x_q\rangle }_{{\Omega }_X},{\langle x_q,x_q^{\prime }\rangle }_{{\Omega }_X},{\langle x_q^{\prime },x_q^{\prime }\rangle }_{{\Omega }_X}\right)=\rho ({\lfloor x_q\rfloor }^{\top }\lfloor x_q\rfloor ,{\lfloor x_q\rfloor }^{\top }\lfloor x_q^{\prime }\rfloor ,{\lfloor x_q^{\prime }\rfloor }^{\top }\lfloor x_q^{\prime }\rfloor ),$$

where ρ is from (3). Based on ${\kappa }_X^q$, we calculate the Gram matrix $K_X^q={\left[{\kappa }_X^q\left(X^k,X^{\ell }\right)\right]}_{k,\ell =1}^n.$. Suppose we aim to estimate the conditional mean of g(Y) given X = x, that is, E {g(Y) | X = x}, for a given g in ℋ_Y. We first compute the coordinates of g with respect to 𝒞_Y by ${\lfloor g\rfloor }_{{𝒞}_Y}=D_Y^{-1/2}{U}_Y^{\top }{\left\{g\left(Y^1\right),\dots ,g\left(Y^n\right)\right\}}^{\top }$. We then estimate E {g(Y) | x} by

$$\widehat{E}\{g(Y)\mid x\}={\left[\left\{{\left(D_X^q\right)}^{-1/2}{\left(U_X^q\right)}^{\top }\right\}{U}_Y{D}_Y^{1/2}{\lfloor g\rfloor }_{{𝒞}_Y}\right]}^{\top }{𝒞}_X^q(x),$$

where $D_X^q$ and $U_X^q$ are from the spectral decomposition $Q_n{K}_X^q{Q}_n=U_X^q{D}_X^q{\left(U_X^q\right)}^{\top }$ corresponding to the leading d eigenvalues, and ${𝒞}_X^q(\cdot )$ is the basis with respect to the new kernel ${\kappa }_X^q$, that is ${𝒞}_X^q(\cdot )=D_X^{-1/2}{\left(U_X^q\right)}^{\top }{\left[{\kappa }_X^q\left(\cdot ,X^1\right)-E_n{\kappa }_X^q(\cdot ,X),\dots ,{\kappa }_X^q\left(\cdot ,X^n\right)-E_n{\kappa }_X^q(\cdot ,X)\right]}^{\top }$.

A special case that is of particular interest is to predict Y given X. Consider the mapping, $g^s:{\Omega }_Y\to ℝ$, $y\mapsto g^s(y)={\lfloor y\rfloor }_{{𝓑}_T,s}$, that is, the sth coordinate of y, for each s = 1, …, m. Therefore, we estimate $E\left\{{\lfloor Y\rfloor }_{{𝓑}_T,s}\mid x\right\}$ by $\widehat{E}\left\{{\lfloor Y\rfloor }_{{𝓑}_T,s}\mid x\right\}={\left[\left\{{(D_X^q)}^{-1/2}{(U_X^q)}^{\top }\right\}{\left({\lfloor Y^1\rfloor }_{{𝓑}_T,s},\dots ,{\lfloor Y^n\rfloor }_{{𝓑}_T,s}\right)}^{\top }\right]}^{\top }{𝒞}_X^q(x)$, and estimate E{Y(t) | x} by

$$\widehat{E}\{Y(t)\mid x\}=\widehat{E}\left[{\lfloor Y\rfloor }_{{𝓑}_T}^{\top }\left\{{𝓑}_T(t)\right\}\mid x\right]={\left\{\widehat{E}\left({\lfloor Y\rfloor }_{{𝓑}_T}\mid x\right)\right\}}^{\top }\left\{{𝓑}_T(t)\right\},$$

where $\widehat{E}\left({\lfloor Y\rfloor }_{{𝓑}_T}\mid x\right)={\left[\widehat{E}\left\{{\lfloor Y\rfloor }_{{𝓑}_T,1}\mid x\right\},\dots ,\widehat{E}\left\{{\lfloor Y\rfloor }_{{𝓑}_T,m}\mid x\right\}\right]}^{\top }$.

5. Asymptotic properties.

In this section, we establish the consistency and convergence rate of our functional SDR estimator for the functional central subspace 𝒮_Y|X. The result for the functional central mean subspace 𝒮_E(Y|X) can be obtained in a similar fashion, and is thus omitted. We first show the convergence of the truncated sample covariance operators ${\widehat{\Sigma }}_{XX}^d$ and ${\widehat{\Sigma }}_{XY}^d$ in (5). We next show the convergence of the sample functional regression operator ${\widehat{m}}_{Y\mid X}^d$ in (6). We then establish the convergence of the sample average Fréchet derivative of the regression function, $E_n\left\{{\widehat{F}}_{Y\mid X}^d(X)\right\}$, which leads to the uniform convergence of our functional SDR estimator for 𝒮_Y|X. We assume the trajectory of the random functions {X(t), Y(t)}^⊤ is fully observed on t ∈ T, and briefly comment on the setting when {X(t), Y(t)}^⊤ is partially observed toward the end. We also derive the consistency and convergence rate of the estimator based on the un-truncated sample covariance operators in Section S2.3 of the Supplementary Material (Lee and Li (2022)). Unless otherwise stated, all proofs are relegated to Section S4 of the Supplementary Material (Lee and Li (2022)).

We begin with some supporting lemmas. The first lemma is about the perturbation of self-adjoint operators. Given $d\in \mathbb{N}$, let λ₁ ≥ ⋯ ≥ λ_d+1 denote the top d + 1 eigenvalues of a self-adjoint operator Σ, and ν_d(Σ) denote the minimum distance between these eigenvalues, ν_d(Σ) = min{λ^k − λ^k+1: k = 1, …, d}.

Lemma 2. Let Σ and $\widehat{\Sigma }$ be self-adjoint operators in B(ℋ), and λ₁ > λ₂ > ⋯ and ${\widehat{\lambda }}_1\ge {\widehat{\lambda }}_2\ge \dots$ be their sequences of eigenvalues, respectively. If $v_d(\Sigma )>2\Vert \widehat{\Sigma }-\Sigma \Vert$, then ${\text{max}}_{k=1,\dots ,d}\left|{\widehat{\lambda }}^k-{\lambda }^k\right|\le 4v_d^{-1}(\Sigma )\Vert \widehat{\Sigma }-\Sigma \Vert$.

The next lemma provides the convergence rates of the relevant sample operators.

Lemma 3. Suppose Assumption 1 holds. Then the orders of magnitude of the terms,

$$\begin{gathered} {\Vert {\widehat{m}}_X-m_X\Vert }_{{\mathscr{H}}_X},{\Vert {\widehat{m}}_Y-m_Y\Vert }_{{\mathscr{H}}_Y},{\Vert E_n\left\{{\kappa }_X(\cdot ,X)-m_X\right\}\otimes \left\{{\kappa }_X(\cdot ,X)-m_X\right\}-{\Sigma }_{XX}\Vert }_{\text{HS}}, \\ {\Vert E_n\left\{{\kappa }_Y(\cdot ,Y)-m_Y\right\}\otimes \left\{{\kappa }_Y(\cdot ,Y)-m_Y\right\}-{\Sigma }_{YY}\Vert }_{\text{HS}}, \\ {\Vert E_n\left\{{\kappa }_X(\cdot ,X)-m_X\right\}\otimes \left\{{\kappa }_Y(\cdot ,Y)-m_Y\right\}-{\Sigma }_{XY}\Vert }_{\text{HS}}, \end{gathered}$$

are O_P (n^−1/2).

Next, we establish the convergence of the truncated sample covariance operators ${\widehat{\Sigma }}_{XX}^d$ and ${\widehat{\Sigma }}_{XY}^d$ in (5). Since our interest is on 𝒮_Y|X, not 𝒮_E(Y|X), we do not discuss the property of ${\widehat{\Lambda }}_{XY}^d$ here. We introduce two intermediate operators, ${\Sigma }_{XX}^d$ and ${\Sigma }_{XY}^d$, which are defined as

$${\Sigma }_{XX}^d=\sum _{k=1}^d\text{var}\left(c^k\right)\left(f^k\otimes f^k\right),{\Sigma }_{XY}^d=\sum _{k,\ell =1}^d\text{cov}\left(c^k,d^{\ell }\right)\left(f^k\otimes g^{\ell }\right).$$

Note that we allow d to grow with the sample size, but denote it as d instead of d_n to simplify the notation. Moreover, for three positive sequences {a_n}, {b_n} and {c_n}, denote a_n ≺ b_n if a_n = o(b_n), a_n ⪯ b_n if a_n = O(b_n), and a_n ∧ b_n = b_n, or a_n ∨ b_n = a_n, if b_n ⪯ a_n. Denote a_n ≍ b_n if both a_n ≍ b_n and b_n ⪯ a_n. Similarly, denote a_n ∧ b_n ∧ c_n = (a_n ∧ b_n) ∧ c_n, and a_n ∨ b_n ∨ c_n = (a_n ∨ b_n) ∨ c_n.

Lemma 4. Suppose Assumption 1 holds, and ξ ≡ d²n^−1/2{ν_d(Σ_XX)∧ν_d(Σ_YY)}⁻¹ ⪯ 1. Then ${\Vert {\widehat{\Sigma }}_{XX}^d-{\Sigma }_{XX}^d\Vert }_{\text{HS}}=O_P(\xi )$, and ${\Vert {\widehat{\Sigma }}_{XY}^d-{\Sigma }_{XY}^d\Vert }_{\text{HS}}=O_P(\xi )$.

Next, we establish the convergence of the sample functional regression operator ${\widehat{m}}_{Y\mid X}^d$. We require two additional smoothness assumptions. We discuss the two assumptions in detail in Section S1 of the Supplementary Material (Lee and Li (2022)).

Assumption 6. There exist β₁ > 0 and m⁰ ∈ B₂(ℋ_Y, ℋ_X), such that $m_{Y\mid X}={\Sigma }_{XX}^{{\beta }_1}{m}^0$.

Assumption 7. Let a^k and b^k denote the kth eigenvalues of Σ_XX and Σ_YY, respectively. There exists β₂ > 0, such that ${\sum }_{k\ge d+1}{a}^k⪯d^{-{\beta }_2}$, and ${\sum }_{k\ge d+1}{b}^k⪯d^{-{\beta }_2}$.

Theorem 5. Suppose Assumptions 1, 2, 6, 7 hold, and that ϵ ≺ 1. Then,

$${\Vert {\widehat{m}}_{Y\mid X}^d-m_{Y\mid X}\Vert }_{\text{HS}}=O_P\left\{{ϵ}^{-1}\left(\xi \vee d^{-{\beta }_2}\right)+{ϵ}^{{\beta }_1}\right\}.$$

Next, we present a lemma regarding the convergence of linear operators in the form of B*B. We then establish the convergence of the sample average Fréchet derivative estimator.

Lemma 5. Let {B_n}_n≥1 and B be random elements in B₂(ℋ, 𝒦). Suppose ∥B_n – B∥_HS = O_P (a_n). Then ${\Vert B_n^{*}{B}_n-{\stackrel{}{B}}^{*}B\Vert }_{\text{HS}}=O_P\left(a_n\right)$.

Theorem 6. Suppose Assumptions 1 to 7 hold, and that ϵ ≺ 1. Then,

$${\Vert E_n\left\{{\widehat{F}}_{Y\mid X}^d(X)\right\}-E\left(F_{Y\mid X}\right)\Vert }_{\text{HS}}=O_P\left\{{ϵ}^{-1}\left(\xi \vee d^{-{\beta }_2}\right)+{ϵ}^{{\beta }_1}\right\}.$$

We are now ready to establish the uniform convergence of the estimated bases $\left\{{\widehat{\varphi }}_1,\dots ,{\widehat{\varphi }}_q\right\}$ for 𝒮_Y|X. The proof follows Lemma 2 and Theorem 6, and is omitted.

Theorem 7. Let ${\left\{\left({\lambda }_k^0,{\varphi }_k^0\right)\right\}}_{k=1}^q$ denote the eigenvalues and eigenfunctions of E(F_Y|X), with ${\lambda }_1^0>\cdots >{\lambda }_q^0>0$. Suppose the same conditions in Theorem 6 hold. Then,

$$\underset{k=1,\dots ,q}{\text{max}}{\Vert {\widehat{\varphi }}_k-{\varphi }_k^0\Vert }_{{\Omega }_X}=O_P\left[v_q^{-1}\left\{E\left(F_{Y\mid X}\right)\right\}\left\{{ϵ}^{-1}\left(\xi \vee d^{-{\beta }_2}\right)+{ϵ}^{{\beta }_1}\right\}\right].$$

We make some remarks about Theorems 5 to 7. First, we note that Li and Song (2017) also studied the convergence of the functional regression operator. Our Theorem 5 further extends their result to the setting where the sample covariance operators are based on the truncated bases. In addition, Li and Song (2017) obtained the consistency of their nonlinear basis functions, whereas our Theorem 7 studied the linear basis functions ϕ_k, which involves a different asymptotic analysis.

Second, and more importantly, in all our theorems, we allow the intrinsic dimension q to diverge, while Li and Song (2017) only considered the setting of a fixed q. This difference has profound implications, and makes our asymptotic analysis far from a straightforward extension of Li and Song (2017). Actually, Theorem 7 suggests that q affects the convergence rate of the basis estimates through ν_q {E(F_Y|X)}, that is, the minimal gap between the eigenvalues of E(F_Y|X). To gain further insight, we note that, by definition, the squared Hilbert-Schmidt norm of E(F_Y|X) is

$$\sum _{k,\ell =1}^{\infty }{\langle {\varphi }_k,E\left(F_{Y\mid X}\right){\varphi }_{\ell }\rangle }_{{\Omega }_X}^2=\sum _{k,\ell =1}^{\infty }{\left(E\left[{({\varphi }_k,\left\{F_{Y\mid X}(X)\right\}{\varphi }_{\ell }\rangle }_{{\Omega }_X}\right]\right)}^2,$$

where ${\left\{{\varphi }_k\right\}}_{k=1}^{\infty }$ are a set of orthonormal bases in Ω_X. Following the proof of Theorem 6, the right-hand side of the above equation is bounded. This implies that E(F_Y|X) is Hilbert-Schmidt, which further indicates that its eigenvalues decay to zero. Therefore, Theorem 7 suggests that we need to restrict this decaying rate. We also comment that, Lin, Zhao and Liu (2019) has recently shown that the convergence rate of sliced inverse regression (SIR) relies on the smallest eigenvalue of the SIR matrix. Although we consider a very different problem than Lin, Zhao and Liu (2019), both works have suggested that the eigen-structure of the key quantity of interest, the SIR matrix in their work, and the average Fréchet derivative in ours, plays an important role in the consistency of the estimated basis.

Third, we examine more closely the convergence rate in Theorem 7. For a given d, we denote the rate of the regularization parameter ϵ in (6) as ϵ(d), and the resulting convergence rate in Theorem 7 as ζ(d). We study the rate of the ϵ(d) under which the best convergence rate ζ(d) is achieved. We have the following result.

Corollary 2. Suppose the same conditions in Theorem 7 hold. Then

$$ϵ(d)={\left(\xi \vee d^{-{\beta }_2}\right)}^{1/\left(1+{\beta }_1\right)},\zeta (d)=v_q^{-1}\left(E{F}_{Y\mid X}\right){\left(\xi \vee d^{-{\beta }_2}\right)}^{{\beta }_1/\left(1+{\beta }_1\right)}.$$

Its proof is by direct calculation and is omitted. To better understand the rates in Corollary 2, suppose d = n^a, $\left\{v_d\left({\Sigma }_{XX}\right)\wedge v_d\left({\Sigma }_{YY}\right)\right\}=n^{-b}$, and ν_q {E(F_Y|X)} = n^−c, for some constants a, b, c > 0. The value of a reflects the growing rate of the number of KL expansions, b restricts the decaying rate of the eigenvalues of Σ_XX and Σ_YY, and c controls the decaying rate of the eigenvalues of E(F_Y|X). Combined with Corollary 2, we have $ϵ\left(a,b,{\beta }_1,{\beta }_2\right)=n^{-r\left(a,b,{\beta }_1,{\beta }_2\right)}$, and $\zeta \left(a,b,{\beta }_1,{\beta }_2\right)=n^{-{\beta }_{1}r\left(a,b,{\beta }_1,{\beta }_2\right)+c}$, where r(a, b, β₁, β₂) = {(1/2 − 2a − b)∧(aβ₂)}/(1 + β₁). This further implies that, when (2 + β2)a + b = 1/2, we obtain the best rates,

$$ϵ\left(b,{\beta }_1,{\beta }_2\right)=n^{-{\beta }_2(1/2-b)/\left\{\left(1+{\beta }_1\right)\left(2+{\beta }_2\right)\right\}},\zeta \left(b,{\beta }_1,{\beta }_2\right)=n^{-{\beta }_1{\beta }_2(1/2-b)/\left\{\left(1+{\beta }_1\right)\left(2+{\beta }_2\right)\right\}+c}.$$

There is a clear interpretation regarding how these best rates are affected by β₁, β₂ as defined in Assumptions 6 and 7, and also the value of b. Specifically, β₁ controls the smoothness of the functional regression operator, and a larger value of β₁ encourages a smoother relation between X and Y, which leads to a larger penalty from ϵ, and a faster convergence rate of ζ. Meanwhile, β₂ regulates the behavior of the tail eigenvalues of Σ_XX and Σ_YY, and a larger β₂ encourages a faster decaying of those tail eigenvalues, which leads to a smaller penalty from ϵ, and a faster rate of ζ. The quantity b restricts the decaying rate of the gaps from the leading eigenvalues of Σ_XX and Σ_YY, and a smaller b implies a slower decaying rate, which means a smaller ϵ is required to stabilize the inversion of ${\Sigma }_{XX}^d$, and a faster rate of ζ.

Finally, we have so far studied the asymptotics under the setting when the random functions are fully observed. We remark that all the results can be extended to the setting when the random function are only partially observed. In this case, the sample mean and covariance operators in Lemma 3 are to have a slower convergence rate than n^−1/2. Suppose the rate is n^−1/2+δ, with δ between [0, 1/2), and the denser the observed time points are for the functions, the closer δ is to 0. The specific value of δ depends on the actual sampling schedule of how functional data are collected; see Wang, Chiou and Muller (2016) for more details. Based on the rate of n^−1/2+δ, we can extend the results in Theorems 5 to 7 to the setting of partially observed functions. We skip the details, since this extension is relatively straightforward.

6. Numerical studies.

In this section, we first carry out simulations to examine the empirical performance of our proposed functional SDR estimators. We then illustrate our method with two real data examples, the classical Canadian weather dataset, and a more recent bike sharing dataset. We report the additional simulation results that compare our method with Li and Song (2017), and study the effect of diverging intrinsic dimensionality in Section S3 of the Supplementary Material (Lee and Li (2022)).

6.1. Simulations.

We consider two sets of simulation models: in the first set, the response function is associated with the predictor function via the conditional mean only, and in the second set, with not only the conditional mean but also with the conditional variance. Let α_j = 1/{(j − 0.5)π}² and β_j(t) = sin {(j − 0.5)πt} denote the jth eigenvalue and eigenfunction of the Brownian motion kernel, for j = 1, …, 100. We independently sample the predictor function X(t) and the error function ε(t) from ${\sum }_{j=1}^{100}{a}_j\sqrt{{\alpha }_j}{\beta }_j(t)$, where a_j’s are i.i.d. standard normal variables. Let ϕ₁(t) = β₄(t), and ϕ₂(t) = β₅(t). We then generate the response function Y(t) as

• I-1: $Y(t)={\beta }_1(t)\times {\langle X(t),{\varphi }_1(t)\rangle }_{{\Omega }_X}+{\beta }_2(t)\times {\langle X(t),{\varphi }_2(t)\rangle }_{{\Omega }_X}+\epsilon (t);$

• I-2: $Y(t)={\beta }_1(t)\times \text{sin}\left\{{\langle X(t),{\varphi }_1(t)\rangle }_{{\Omega }_X}\right\}+{\beta }_2(t)\times \text{sin}\left\{{\langle X(t),{\varphi }_2(t)\rangle }_{{\Omega }_X}\right\}+\epsilon (t);$

• II-1: $Y(t)={\beta }_1(t)\times {\langle X(t),{\varphi }_1(t)\rangle }_{{\Omega }_X}+{\langle X(t),{\varphi }_2(t)\rangle }_{{\Omega }_X}\times \epsilon (t);$

• II-2: $Y(t)={\beta }_1(t)\times \text{sin}\left\{{\langle X(t),{\varphi }_1(t)\rangle }_{{\Omega }_X}\right\}+\text{sin}\left\{{\langle X(t),{\varphi }_2(t)\rangle }_{{\Omega }_X}\right\}\times \epsilon (t).$

By construction, for models I-1 and I-2, 𝒮_E(Y|X) = 𝒮_{Y |X} = Span{ϕ₁(t), ϕ₂(t)}, while for models II-1 and II-2, 𝒮_E(Y|X) = Span{ϕ₁(t)} and 𝒮_Y|X = Span{ϕ₁(t), ϕ₂(t)}. We then take the same 50 equally spaced points {t_k1, …, t_k50} between [0, 1] as the observed time points, for all k = 1, …, n. We consider different values of the sample size n = {100, 250, 500}. In our implementation of the functional SDR, we use a Brownian motion kernel for κ_T, and a radial basis kernel for κ_X and κ_Y.

We first evaluate the accuracy of recovering 𝒮_E(Y|X) and 𝒮_Y|X. Toward that end, we generate n i.i.d. pairs of random functions {X(t), Y(t)} for each model, then divide them into a training set and a testing set, each with n/2 observations. We then apply our methods to the training data to estimate the basis functions for the functional dimension reduction subspaces. We next compute the top q sufficient predictors using the testing data. We first use the true value of q, then examine the estimation of q using (12) later. To evaluate how effective the estimated sufficient predictors approximate the true ones, we compute the trace of the average squared multivariate correlation coefficient matrix (MCC, Ferré and Yao (2005)),

$$\text{MCC}(Z,\widehat{Z})=(q^{-1}\text{tr}[{\left\{{\text{cov}}_n(Z,Z)\right\}}^{-1/2}\left\{{\text{cov}}_n(Z,\widehat{Z})\right\}{\times {\left\{{\text{cov}}_n(\widehat{Z},\widehat{Z})\right\}}^{-1}\left\{{\text{cov}}_n(\widehat{Z},Z)\right\}{\left\{{\text{cov}}_n(Z,Z)\right\}}^{-1/2}])}^{1/2},$$

Where $Z={\left({\langle {\varphi }_1,X\rangle }_{{\Omega }_X},\dots ,{\langle {\varphi }_1,X\rangle }_{{\Omega }_X}\right)}^{\top }$, and $\widehat{Z}={\left({\langle {\widehat{\varphi }}_1,X\rangle }_{{\Omega }_X},\dots ,{\langle {\widehat{\varphi }}_1,X\rangle }_{{\Omega }_X}\right)}^{\top }$ are the true and estimated sufficient predictors, respectively. The value of MCC is between 0 and 1, with 1 indicating a perfect recovery. Besides, it is calculated based on the testing samples, and hence avoids the overfitting issue. Figure 1 reports the box plots of this criterion for the four models based on 80 data replications, for both the functional central mean subspace (fCMS) and the functional central subspace (fCS) estimation. It is seen that, for models I-1 and I-2, both estimators perform well, since the conditional mean contains all the relevant regression information. For models II-1 and II-2, the functional central subspace estimator performs consistently better than the functional central mean subspace estimator, because the conditional variance contains additional relevant regression information that the functional central mean subspace misses. Moreover, the performance improves as the sample size n increases, which agrees with our asymptotic theory.

Fig. 1.

Box plots of the average squared multivariate correlation coefficients between the true and estimated sufficient predictors, for the functional central mean subspace estimator (fCMS), and the functional central subspace estimator (fCS), and the four models, respectively. Each panel consists of 3 boxes for three sample sizes n= 100,250,500.

Next, we evaluate the performance of selecting the intrinsic dimension q using (12). We choose the top q eigenfunctions that account for 90% total variation in the estimated average Fréchet derivative. For models I-1 and I-2, both 𝒮_E(Y|X) and 𝒮_Y|X have the intrinsic dimension q = 2. For models II-1 and II-2, 𝒮_E(Y|X) has the intrinsic dimension q = 1, and 𝒮_Y|X has the intrinsic dimension q = 2. Table 1 reports the estimated q averaged over 80 data replications. It is seen that, the estimated q is slightly larger than the true intrinsic dimension, especially for models II-1 and II-2. This is acceptable from a dimension reduction perspective, as a slightly larger estimated q implies that the method recovers some additional redundant information, while a smaller estimated q means the method misses some important information. Moreover, the estimation of q becomes more accurate as the sample size n increases.

Table 1.

The estimated intrinsic dimension q

Model		fCMS			fCS
	n	100	250	500	100	250	500
I-1	mean	3.00	2.48	2.01	3.99	3.01	2.14
	s.d.	0.32	0.50	0.11	0.11	0.11	0.35
I-2	mean	3.58	3.00	2.58	4.03	3.92	3.02
	s.d.	0.50	0.23	0.50	0.16	0.27	0.16
II-1	mean	4.00	3.86	3.27	4.08	3.94	3.01
	s.d.	0.00	0.35	0.45	0.27	0.24	0.11
II-2	mean	4.00	3.95	3.39	4.41	4.01	3.88
	s.d.	0.00	0.22	0.49	0.50	0.11	0.33

Finally, we study the performance of postreduction prediction. We compare our method with three state-of-the-art parametric function-on-function regression methods: the functional linear regression (LIN, Yao, Müller and Wang (2005)), the functional linear regression by signal compression (LQ, Luo and Qi (2017)), and the functional additive regression (ADD, Müller and Yao (2008)). Both LIN and ADD are implemented in the R package PACE, and LQ is implemented in the R package FRegSigCom. Since the competing methods were designed to predict the mean function of the response function, it is more meaningful to focus our comparison on the conditional mean models I-1 and I-2. To evaluate the predictive performance, we fit the model using the training samples, then obtain the predicted response function $\widehat{Y}(x,t)=\widehat{E}\{Y(t)\mid x\}$ using the testing samples. We then compute the mean squared prediction error (MSPE),

$$\text{MSPE}=\sum _{k=1}^{n_t}\sum _{s=1}^m{\left\{Y^k\left(t_s\right)-\widehat{Y}\left(X^k,t_s\right)\right\}}^2/\left(n_{t}m\right),$$

where ${\left(X^1,Y^1\right)}^{\top },\dots ,{\left(X^{n_t},Y^{n_t}\right)}^{\top }$ are the testing samples. Table 2 reports the results based on 80 data replications. It is seen that, LQ has the best predictive performance for model I-1, while our prediction method based on the functional central subspace 𝒮_Y|X is the second best. This is not surprising though, since their method was specifically designed for a linear function-on-function regression model. Meanwhile, our method based on 𝒮_Y|X performs the best for model I-2, where there is a clear nonlinear association between the predictor and response functions.

Table 2.

The mean squared prediction error for the prediction methods based on the functional central mean subspace (fCMS), the functional central subspace (fCS), the functional linear regression (LIN), the functional linear regression by signal compression (LQ), and the functional additive regression (ADD)

Model		fCMS	fCS	LIN	LQ	ADD
I-1	mean	0.715	0.695	2.584	0.540	2.618
	s.d.	0.159	0.121	0.151	0.042	0.147
I-2	mean	0.610	0.603	1.396	0.666	1.411
	s.d.	0.063	0.058	0.081	0.057	0.088

6.2. Canadian weather data.

We first illustrate our methods with the classical Canadian weather data, which is often used as a benchmark for function-on-function regression analysis (Ramsay and Silverman (2005)). The dataset consists of the average daily temperature and logarithm of the daily precipitation over 35 years, from 1960 to 1994, for 35 different locations in Canada. The data is available from the R package fda. Figure 2(a)–(b) show the observed time series for the daily temperature and precipitation in the logarithmic scale. The analysis goal is to study the association between the temperature and precipitation, and to use the temperature to predict the precipitation.

Fig. 2.

Canadian weather data.

We apply the proposed functional SDR methods to this data. For 𝒮_E(Y|X), the top 3 sufficient predictors explain 45.1%, 33.6% 15.2% of total variation, respectively, and 93.9% accumulatively. For 𝒮_Y|X, the top 3 sufficient predictors explain 48.5%, 37.3%, 9.2% of total variation, respectively, and 94.9% accumulatively. Figure 2(c)–(d) show these estimated bases, which look similar for 𝒮_E(Y|X) and 𝒮_Y|X in this example. From the plots, it is seen that the first basis function ${\widehat{\varphi }}_1(t)$ weighs more at the beginning and toward the end of the year, while the second basis function ${\widehat{\varphi }}_2(t)$ weighs more in the middle of the year.

Next, we apply the postreduction prediction method to estimate the conditional mean of the precipitation function in logarithm given the top 3 estimated bases from the temperature function. Figure 3(a) shows the scatterplot of the first sample KL coefficient of the logarithmic daily precipitation, as well as its estimated regression function, that is, the conditional mean, versus the first and second estimated sufficient predictors. We also interpolate the regression surface for a clearer visualization. From the plot, we see that the sample KL coefficient is well approximated by its regression function. We also observe an increasing trend of the conditional mean as the sufficient predictors increase.

Fig. 3.

The first sample KL coefficient of the response function (black dots) and its regression function (colored mesh) versus the first two sufficient predictors.

Finally, we perform an out-of-sample prediction analysis. That is, we randomly draw 30 out of 35 observations as the training samples, and use the rest as the testing samples. We record the mean squared prediction error based on the testing data, and compare the five methods studied in Section 6.1. We repeat this process 100 times. Table 3 reports the results averaged over such 100 replications. It is seen that our method based on the functional central subspace achieves the best prediction accuracy.

Table 3.

The mean squared prediction error for the prediction methods based on the functional central mean subspace (fCMS), the functional central subspace (fCS), the functional linear regression (LIN), the functional linear regression by signal compression (LQ), and the functional additive regression (ADD)

Dataset		fCMS	fCS	LIN	LQ	ADD
Canadian weather	mean	0.0865	0.0861	0.1134	0.1230	0.1192
	s.d.	0.0301	0.0300	0.0401	0.0589	0.0492
Bike sharing	mean	0.1357	0.1361	0.1995	0.1775	0.1458
	s.d.	0.0421	0.0415	0.0732	0.0630	0.0545

6.3. Bike sharing data.

We next illustrate our methods with a more recent business application, the bike sharing data (Fanaee-T and Gama (2014)). The dataset consists of the hourly counts of total rental bikes for both casual and registered users, and weather related information, such as the hourly temperature, precipitation, wind speed, and humidity. The observations were recorded from the Capital Bike Share system in Washington, DC every day from January 1, 2011 to December 31, 2012. The data is available from https://archive.ics.uci.edu/ml/datasets/Bike+Sharing+Dataset. One of the analysis goals is to understand how the bike rentals are affected by the temperature on Saturdays. Therefore, we treat the hourly bike rentals as the response function and the hourly temperature as the predictor function. After removing 3 weeks of observations with many missing values, we obtain 102 weeks of pairs of curves, which are shown in Figure 4(a)–(b).

Fig. 4.

Bike sharing data for 102 Saturdays.

We apply the proposed functional SDR methods to this data. For the functional central mean subspace 𝒮_E(Y|X), the top 3 sufficient predictors explain 62.8%, 18.4% 14.6% of total variation, respectively, and 95.9% accumulatively. For the functional central subspace 𝒮_Y|X, the top 3 sufficient predictors explain 69.3%, 16.2%, 10.9% of total variation, respectively, and 96.4% accumulatively. Figure 4(c)–(d) show these estimated bases. It is seen that, the first basis ${\widehat{\varphi }}_1(t)$ suggests that the temperature between 10 am and midnight has more impact on the bike rentals than the rest of the day, while the second and third bases suggest that early morning tends to affect more on the bike rentals. Moreover, we observe that, although the estimated bases of the 𝒮_E(Y|X) and 𝒮_Y|X are not numerically identical, they capture very similar patterns. This suggests that the leading estimated bases are most likely related to the change of the conditional mean of the hourly rental counts. Also, we see that the second and third estimated bases from the two estimations roughly span the same space, with the second estimated basis for 𝒮_E(Y|X) corresponding to the sign-flipped third estimated basis for 𝒮_Y|X, and the third estimated basis for 𝒮_E(Y|X) corresponding to the second estimated basis for 𝒮_Y|X. This order switching between the estimated bases is likely due to the closeness of the second and third largest eigenvalues.

Next, we apply the postreduction prediction method to estimate the conditional mean of the bike rental function given the top 3 estimated bases from the temperature function. Figure 3(b) shows the scatterplot of the first sample KL coefficient of the hourly rental, and its estimated regression function, versus the first and second estimated sufficient predictors. We again interpolate the regression surface. We see that the estimated regression function well approximates the estimated KL coefficient. Besides, as the first two sufficient predictors increase, the conditional mean of the first KL of the bike rental decreases.

Finally, we perform the out-of-sample prediction analysis, by randomly drawing 90 observations out of 102 as the training samples, and then using the rest as the testing samples. Table 3 reports the results averaged over 100 replications. For this example, our method based on the functional central mean subspace achieves the best prediction accuracy.

7. Discussion.

In this section, we reiterate and further clarify some key differences between our proposal and the existing gradient-based SDR and functional SDR solutions. We first compare to the gradient-based SDR methods for the random variable case.

1. The gradient-based linear SDR methods such as Härdle and Stoker (1989), Xia (2007), Xia et al. (2002), Yin and Li (2011) targeted the central mean or central subspaces that are finite-dimensional subspaces of ${ℝ}^p$. In comparison, our functional central mean and central subspaces are defined as the projections of Hilbert space-valued random predictors, which themselves can be Hilbert space-valued and infinite-dimensional.

2. A major step in the gradient-based methods is to estimate the gradient itself. Nevertheless, this step has imposed great challenges under the functional setting. Härdle and Stoker (1989) estimated the gradient of the density function; however, the density function does not always exist for random functions. Alternatively, Xia (2007), Xia et al. (2002), Yin and Li (2011) estimated the gradient of the regression functions, which requires to solve a high-dimensional local linear regression. Extending the local linear regression to the functional setting is feasible, but is analytically complicated. In contrast, we build our estimator on the Fréchet derivative of the regression functionals, which has a closed form. To achieve this, we rely on a key property of RKHS, in which we establish the interchangeability between the nested kernel function and the Fréchet derivative; see Proposition 2. This simplification in estimating the Fréchet derivative is useful in both estimation at the population and sample levels, as well as in deriving the consistency results.

3. We have developed a number of useful properties regarding the Fréchet differentiability of the nested RKHS. For instance, we derive the conditions under which the nested kernel function is continuously Fréchet differentiable in Proposition S1. To show that the Fréchet derivative of the regression functionals have tractable forms, we illustrate with some popular kernel functions and derive their Fréchet derivatives in Proposition S2 at the population level, and in Proposition S5 at the coordinate level. Moreover, in Lemma S3 we derive a property on the interchangeability of partial Fréchet differentiability on the nested RKHS, which allows us to compute the moments of the Fréchet derivative; see Proposition S4. These results are sufficiently general to be applied to problems beyond the SDR framework, and are useful for functional data analysis involving the Fréchet derivative.

4. We remark that, a possible alternative for our functional SDR is to discretize the response function as Y(t₁), …, Y(t_m) for the observed time points t₁, …, t_m, then perform dimension reduction by treating Y as an m-dimensional random vector (Hsing (1999), Li, Wen and Zhu (2008)). This solution does not require to specify a working RKHS. However, a major advantage of our approach is that we are able to borrow information across the entire function to compute the KL coefficients, then to recover the entire function using only a few leading coefficients. This approach greatly alleviates the curse of dimensionality, as the working space is of the same dimension as the number of extracted KL coefficients d, rather than the number of observed time points m, and m is usually much larger than d. This is also reflected in our asymptotic convergence rates, which depend on d, not m.

We next compare to the functional SDR methods, particularly, Li and Song (2017).

1. Although Li and Song (2017) has shown the existence of the nonlinear functional subspace, our Theorem 1 on the existence of the functional central mean and central subspaces can not be directly inferred by their result. To establish the existence, we introduce the concept of M-set in the functional space, a result that is not available in their work.

2. A major difference between our work and Li and Song (2017) is that, their estimator was based on the inverse regression of X | Y, whereas our estimator is built on the forward regression of Y | X. The inverse regression approach has been taken by many other functional SDR estimators, for example, Hsing and Ren (2009), Jiang, Yu and Wang (2014), Wang, Lin and Zhang (2013), Wang et al. (2015), Yao, Lei and Wu (2015). Such a difference leads to completely different estimation approaches, theoretical analyses, as well as required conditions. For instance, the generalized sliced average variance estimator of Li and Song (2017) and all other inverse regression based functional SDR methods required some version of the linearity or constant variance conditions to establish the unbiasedness or exhaustiveness. In contrast, our method achieves the unbiasedness and exhaustiveness without imposing such distributional assumptions.

3. Another major difference is that Li and Song (2017) targeted nonlinear SDR, whereas we aim at linear SDR, as we have clarified after Theorem 1. Each has its own strength and limitation; see Li ((2018a), Chapter 14) for more comparison. However, simply using a linear kernel in Li and Song (2017) would not produce the same result as our method, due to different approaches of inverse and forward regressions the two methods adopt.

4. Compared to the estimator of Li and Song (2017) that was solely built on the functional regression operator, our estimator requires both the functional regression operator, and another key element, the Fréchet derivative. Consequently, we have developed a different set of tools regarding the Fréchet differentiability of the nested RKHS, None of those results are available in Li and Song (2017).

5. A key novelty of our asymptotic analysis is that we allow the number of sufficient predictors to diverge with the sample size. Li and Song ((2017), Corollary 3) has shown the pointwise consistency, but assumed a fixed number of sufficient predictors. In comparison, in Theorem 7, we have established the uniform consistency with a diverging number of sufficient predictors, which is considerably more challenging than Li and Song (2017). We require additional technical tools such as Lemma 2 to deal with the perturbation of the sample linear operators. Moreover, our asymptotic framework needs to deal with the consistency of the Fréchet derivative estimation; for example, in the proof of Theorem 6, we derive the rate of convergence for the Fréchet derivative, based on which we are able to establish the consistency of our average Fréchet derivative estimator.

In summary, our proposal is far from an incremental extension of the existing SDR methods. We believe it is important and useful for both fields of sufficient dimension reduction and functional data analysis in general.