Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2023 Mar 28.
Published in final edited form as: J Comput Graph Stat. 2022 Mar 28;31(3):835–845. doi: 10.1080/10618600.2022.2037434

Estimation and model selection for nonparametric function-on-function regression

Zhanfeng Wang 1, Hao Dong 2, Ping Ma 3, Yuedong Wang 4
PMCID: PMC9802009  NIHMSID: NIHMS1798405  PMID: 36594047

Abstract

Regression models with a functional response and functional covariate have received significant attention recently. While various nonparametric and semiparametric models have been developed, there is an urgent need for model selection and diagnostic methods. In this article, we develop a unified framework for estimation and model selection in nonparametric function-on-function regression. We propose a general nonparametric functional regression model with the model space constructed through smoothing spline analysis of variance (SS ANOVA). The proposed model reduces to some of the existing models when selected components in the SS ANOVA decomposition are eliminated. We propose new estimation procedures under either L1 or L2 penalty and show that the combination of the SS ANOVA decomposition and L1 penalty provides powerful tools for model selection and diagnostics. We establish consistency and convergence rates for estimates of the regression function and each component in its decomposition under both the L1 and L2 penalties. Simulation studies and real examples show that the proposed methods perform well. Technical details and additional simulation results are available in online supplementary materials.

Keywords: Convergence rate, Regularization, Reproducing kernel Hilbert space, Smoothing spline ANOVA

1. Introduction

With the advance of modern technology, it becomes increasingly common that data are in the form of functions. Rapid developments of new statistical methods for this new type of data has created the field of functional data analysis (FDA) (Yao et al. 2005 b; Ramsay and Silverman 2006; Ferraty and Vieu 2006; Hsing and Eubank 2015; Kokoszka and Reimherr 2017; Lin et al. 2018; Lin and Yao 2019). Many semiparametric and nonparametric methods have been proposed for regression with functional response and/or covariates. Nevertheless, there is a lack of flexible model selection and diagnostics methods for functional regression (Wang et al. 2015; Morris 2015). Ling and Vieu (2018) suggested model building and testing using nonparametric methods as two important future research areas.

In this paper, we consider function-on-function regression where both the response Y and covariate X are functions. Denote the functional observations by {(Yi(t), Xi(t)), i = 1, …, n; t∈T}, where T is an arbitrary set. We want to investigate the relationship between X and Y. Ramsay and Silverman (2006) proposed the concurrent linear model (CLM),

Yi(t)=α(t)+β(t)Xi(t)+ϵi(t),i=1,⋯,n, (1)

where α(t) and β(t) are unknown functions to be estimated, ϵi(t), independent of Xi(t), are i.i.d. random errors. Model (1) assumes that the value of Y at t depends on X at the same point t only. Ramsay and Silverman (2006) also proposed the functional linear model (FLM),

Yi(t)=α(t)+∫Tβ(s,t)Xi(s)ds+ϵi(t),i=1,⋯,n, (2)

where Y(t) depends on the whole function of X(·), and α(t) and β(s,t) are unknown functions to be estimated. Both models (1) and (2) assume that Y is linearly dependent on X, which could be restrictive for some applications. While many nonlinear models have been proposed for scalar-on-function regression (Reiss et al. 2017; Ling and Vieu 2018), nonlinear function-on-function regression has received considerably less attention (Morris 2015; Reimherr et al. 2018). Müller and Yao (2008) considered an additive model where functional principal component analysis (FPCA) scores were used as predictors.

Different from aforementioned linear and nonlinear models, nonparametric methods, which provide more flexible regression relationship between the response and covariates, have been considered by many authors Yuan and Cai 2010; Ferraty et al. 2011, 2012; Lian 2007; Kardi et al. 2016; Ling and Vieu 2018. In particular, the nonparametric concurrent model (NCM) (Zhang et al. 2013; Scheipl et al. 2015; Kim, Maity and Staicu 2018) assumes that

Yi(t)=g(t,Xi(t))+ϵi(t),i=1,⋯,n, (3)

where g is an unknown bivariate function; and an extension of functional linear model (EFLM) (Ma and Zhu 2016; Kim, Staicu, Maity, Carroll and Ruppert 2018; Reimherr et al. 2018) assumes that

Yi(t)=∫Tg(t,s,Xi(s))ds+ϵi(t),i=1,⋯,n, (4)

where g is an unknown function of three variables. In practice, Y(t) may depend on both X(t) and the whole function of X (see examples in Section 6). Furthermore, data-driven model diagnostic tools are still lacking. It is very difficult for practitioners to decide how to choose an adequate model for their data. Hypothesis testing could be used for the model selection (Xing et al., 2020). However, rigorous tests are only available for simple settings. In addition, since these models were fitted using different methods, the convergence rate of the estimated functions have not been analyzed and compared carefully.

To establish a unified framework for estimation, model selection, and theoretical study, we consider the following nonparametric functional regression model,

Yi(t)=∫Tg(t,s,Xi(t),Xi(s))ds+ϵi(t),i=1,⋯,n, (5)

where g:T×T×R×R→R is an unknown function of four variables to be estimated, R is the range of the functional covariate X, and ϵi(t)s are i.i.d random error functions in L2 with mean zero and finite ∫TE(ϵi2(t))dt. The proposed method can be extended to other nonparametric functional models, such as scalar-on-function and function-on-scalar regression models.

Model (5) is a flexible function-on-function model that has not been studied in the literature. Different from the methods used in Scheipl et al. (2015), Ma and Zhu (2016), Kim, Maity and Staicu (2018), and Reimherr et al. (2018), in this paper we construct model space for the multivariate function g through smoothing spline analysis of variance (SS ANOVA) based on the decomposition of tensor product of reproducing kernel Hilbert spaces (RKHS). Model (5) reduces to existing models when selected components in the SS ANOVA decomposition are eliminated. We propose penalized least squares methods for estimating g under either L1 or L2 penalty. We establish the representer theorem and develop computational methods for solving both L1 and L2-regularized least squares problems. The SS ANOVA model with L1 penalty provides a systematic approach for model selection and diagnostics of existing models. We establish a coherent framework to study the convergence rates of the penalized least squares estimates as well as each component of the SS ANOVA model under both the L1 and L2 penalties. We note that Sun et al. (2018) constructed SS ANOVA for the bivariate function β(s,t) in the FLM (2). However, there is an error in their derivation of the representer theorem. We will use a different approach to derive the representer theorem and our estimate is different from that in Sun et al. (2018).

Our methodological contribution for nonparametric function-on-function regression is to develop new estimation procedures under either a L2 or a joint L1 and L2 penalty. Note that through eliminating selected components in a SS ANOVA model, our model (5) can be reduced to models (1)-(4) and other special cases. In practice, researchers may use a reduced model appropriate for their data rather than the general model (5). However, coherent model selection and model diagnostics tools for function-on-function regression are still lacking. Model selection and diagnostics have to be conducted manually, which is laborious, time-consuming, and error prone. The lack of model selection and diagnostics hinders the wide application of the aforementioned models. Our work is the first to provide model selection and diagnostic tools through the combination of SS ANOVA and a joint L1 and L2 penalty. In particular, model fitting, model selection and diagnostics can be conducted simultaneously and substantially alleviate the cost of manual model selection and diagnostics.

Our theoretical contribution is to establish a coherent framework to study the convergence rates of the penalized least square estimates and components in the SS ANOVA model. In the literature, the convergence of the estimators and components are conducted through two technical routes, dimensionless approach (Gu and Qiu 1993) and tensor product approach (Lin 2000). In this paper, we reconcile these two methods into one coherent framework. As far as we know, these convergence rates of the estimators and components are the first ones established for function-on-function regression models in a general framework. Moreover, the convergence rates of the estimators and components under a joint L1 and L2 penalty are still lacking in the literature even for simple SS ANOVA models. We bridge this gap by establishing the first convergence rates for function-on-function regression models in a general framework. Equipped with all these convergence rates, our theoretical work lays out the first full-fledged analysis framework in this line of research.

The remainder of the paper is organized as follows. In Section 2, we present SS ANOVA model for the function g in model (5), and the estimation procedure under L2 penalty. In Section 3, we present model selection method through SS ANOVA and estimation under L1 penalty. In Section 4, consistence and convergence rates of estimators of the overall regression function and each component in the SS ANOVA decomposition are obtained. Numerical studies and real examples are given in Sections 5 and 6. We conclude in Section 7. Additional technical details, proofs, and codes are included in the Supplementary Materials.

2. Model Space and Estimation

We introduce the SS ANOVA model in Section 2.1 and present the estimation procedure with L2 penalty in Section 2.2.

2.1. SS ANOVA Models

Different from the methods used in Scheipl et al. (2015), Ma and Zhu (2016), Kim, Maity and Staicu (2018), and Reimherr et al. (2018), we use tensor product of RKHS’s and the SS ANOVA decomposition to build a model space for the regression function g in (5). Denote H1(T), H2(T), H3(R), and H4(R) as RKHS’s of functions on domains T and R respectively. The choices of these RKHS’s depend on the domain of these functions, prior knowledge, and the purpose of study (Wang, 2011). For example, for functions on a compact interval, without loss of generality denote the interval as [0,1], we may consider the Sobolev space

W2m[0,1]={f:f,f′,⋯,f(m−1)are absolutely continuous,∫01(f(m))2dx<∞}. (6)

We consider an SS ANOVA decomposition of g in the tensor product RKHS H1(T)⊗H2(T)⊗H3(R)⊗H4(R) (Wang 2011; Gu 2013):

g(t,s,X(t),X(s))=μ+g1(t)+g2(s)+g3(X(t))+g4(X(s))+g12(t,s)+g13(t,X(t))+g14(t,X(s))+g23(s,X(t))+g24(s,X(s))+g34(X(t),X(s))+g123(t,s,X(t))+g124(t,s,X(s))+g134(t,X(t),X(s))+g234(s,X(t),X(s))+g1234(t,s,X(t),X(s)),

where μ is the grand mean, g1 and other single subscript g are main effects, double subscript g are two-way interactions, and so on. For identifiability of model (5), we need side conditions,

g2(s)=g12(t,s)=g23(s,X(t))=g123(t,s,X(t))=g134(t,X(t),X(s))=0.

Thus, g has a decomposition as follows,

g(t,s,X(t),X(s))=μ+g1(t)+g3(X(t))+g4(X(s))+g13(t,X(t))+g14(t,X(s))+g24(s,X(s))+g34(X(t),X(s))+g124(t,s,X(s))+g234(s,X(t),X(s))+g1234(t,s,X(t),X(s)). (7)

The SS ANOVA decomposition (7) builds a hierarchical structure for function g and handles the side conditions in a natural manner. In addition to having nice interpretation as main effects and interactions, the SS ANOVA decomposition facilitates diagnostics and model selection. It is easy to see that models (1)-(3) are special cases of (7) with certain components equal zero. For example, the NCM (4) and EFLM (3) are special cases with

g4=g14=g24=g34=g124=g234=g1234=0, (8)

and

g3=g13=g34=g234=g1234=0, (9)

respectively. Conditions for the CLM (1) and FLM (2) are presented in the Supplemental Materials. Therefore, checking whether components in these conditions equal zero provides a diagnostic tool for existing models. In addition, we can fit existing models by removing components in these conditions. Thus the proposed estimation methods provide a unified framework for fitting some of the existing models with theoretical guarantees.

We may model g using any subset of components in the SS ANOVA decomposition (7). Given an SS ANOVA model, we can regroup components and rewrite the model space for g as Section 4.5 of Wang (2011)

M=H0⊕H1⊕⋯⊕Hq, (10)

where H0 is a finite dimensional space with an orthonormal basis {ϕ1,…,ϕm},H1,⋯,Hq are orthogonal RKHS’s with reproducing kernels (RK) R1,…,Rq, respectively, and q is the number of components in the SS ANOVA model. We note that different regrouping may be used for different purposes. In particular, we may consider different groups of components for model selection with L1 penalty (see Section 3).

2.2. Penalized Least Squares with L2 Penalty

We assume that Yi(t)’s are stochastic processes that belongs to L2(T). We estimate g as a minimizer of the following penalized least squares (PLS):

1n∑i=1n∫T{Yi(t)−∫Tg(t,s,Xi(t),Xi(s))ds}2dt+λ2∑j=1q1θj‖Pjg‖2, (11)

where Pj is a projection operator onto Hj, λ and θj are smoothing parameters, and ∣∣η∣∣ is an induced norm in M.

We shall now present the solution to the PLS (11). Let H1∗=H1⊕⋯⊕Hq, and define a new inner product in H1∗ as

〈f,g〉∗=∑j=1q1θj〈Pjf,Pjg〉, (12)

where ⟨·,·⟩ is the inner product in M. Under the new inner product, the RK of M is R = R0 + R1, where

R0(t,s,x,z,t′,s′,x′,z′)=Rt,s,x,z0(t′,s′,x′,z′)=∑k=1mϕk(t,s,x,z)ϕk(t′,s′,x′,z′),R1(t,s,x,z,t′,s′,x′,z′)=Rt,s,x,z1(t′,s′,x′,z′)=∑j=1qθjRj(t,s,x,z,t′,s′,z′).

Let {vk(t),k = 1,2,…} be an orthonormal basis of L2(T) where the first n basis functions are the empirical functional principal components (EFPC) of Y1,…,Yn. See Hsing and Eubank (2015) for principal component analysis of stochastic processes defined on a general compact metric space. Our result is presented in the following representer theorem, which proof is in the Supplemental Materials.

Theorem 2.1 (Representer Theorem). (a) The solution to the PLS (11) is

g^(t,s,x,z)=∑k=1mdkϕk(t,s,x,z)+∑i=1n∑j=1ncijξij(t,s,x,z), (13)

where ξij(t,s,x,z)=∫T∫TRt,s,x,z1(t′,s′,Xi(t′),Xi(s′))vj(t′)dt′ds′.

(b) Consider the following standard SS ANOVA model with a scalar response variable,

Yij=Lijg+ϵij, (14)

where Yij = ⟨Yi, vj⟩, g∈M, Lijg=∫T∫Tg(t,s,Xi(t),Xi(s))vj(t)dsdt which is assumed to be a bounded linear functional, and ϵij are iid random errors with mean zero. Then the estimate g^ in (11) is the same as the solution to the following PLS

1n∑i=1n∑j=1n{Yij−Lijg}2+λ2‖P1∗g‖∗2, (15)

where P1∗ is the projection operator onto H1∗ with the inner product ⟨f,g⟩ in (12), and ∣·∣ is the induced norm in H1∗ by ⟨·,·⟩*.

The representer theorem states that the task of fitting a function-on-function regression model reduces to the task of fitting an SS ANOVA model with a scalar response variable. We now present the computation details. Note that R = R0 + R1 and it can be shown that

〈g,∫T∫TRt,s,Xi(t),Xi(s)0vj(t)dsdt〉=∑k=1maijkdk,〈g,∫T∫TRt,s,Xi(t),Xi(s)1vj(t)dsdt〉=∑k=1n∑l=1ncklbijkl,

where

aijk=∫T∫Tϕk(t,s,Xi(t),Xi(s))vj(t)dsdt,bijkl=∫T∫Tξkl(t,s,Xi(t),Xi(s))vj(t)dsdt.

Let Yj = (Y1j…, Ynj)⊤, Y = (Y1⊤, …, Yn⊤)⊤, d = (d1, …, dm)⊤, c = (c11, c21, …, cnn)⊤, T be an n2 × m matrix with the (i + (j−1)n,k)th element as aijk, and Σ be an n2 × n2 matrix with the (i + (j−1)n,k + (l−1)n)th element as bijkl. Then the PLS (15) reduces to

1n∑i=1n∑j=1n(Yij−∑k=1maijkdk−∑k=1n∑l=1ncklbijkl)2+λ2∑i=1n∑j=1n∑k=1n∑l=1ncijbijklckl=1n‖Y−Td−Σc‖2+λ2cTΣc. (16)

Following the above discussion, one may use existing software to compute the estimate. Details for solving the PLS (16) with data-driven smoothing parameters selection methods such as the generalized cross-validation (GCV) and restricted maximum likelihood (REML), also known as generalized maximum likelihood (GML), can be found in Wang (2011). The recent development in selecting smoothing parameters can be found in Sun et al. (2020). Since the reduced models have n2 observations, existing procedures may be infeasible when n is large. One may consider an approximate solution by using the first p(p⪡n) EFPCs. Furthermore, to save computational time, we propose a backfitting procedure to estimate smoothing parameters. Details about the backfitting procedure can be found in the Supplemental Materials.

3. Model Selection

In this section, we consider penalized least squares with L1 penalty for model selection and diagnostics. Given an SS ANOVA model, we can regroup and rewrite the model space in the form (10). Different regrouping may be considered for different purposes. For estimation with L2 penalty, we usually set H0 as the space collecting all functions that are not penalized, and other spaces contain main effects and interactions subject to penalties. For model selection, we are interested in whether certain components in the SS ANOVA decomposition can be eliminated. We may use the same regrouping as (10) with L1 penalties to all components including functions in the space H0 except for the constant functions. For diagnostics of a specific model, we are interested in whether a group of components in the SS ANOVA decomposition can be eliminated. For example, the NCM is equivalent to all components in (8) equal zero. We may regroup such that one subspace collects all components in (8). Consequently, the L1 penalty to functions in this subspace encourages all components in (8) to be simultaneously zero. For generality, for a given SS ANOVA model, we rewrite the regrouped model space as

M=H~0⊕H~1⊕⋯⊕H~q~, (17)

where H~0 is a finite dimensional space with an orthonormal basis {ϕ~1,⋯,ϕ~m~}, and H~1,⋯,H~q~ are orthogonal RKHS’s with RKs R~1,⋯,R~q~, respectively. Note that, as discussed above, each space Hj may include parametric components in the space H0 and multiple components of H1,⋯,Hq.

Denote the projection of g onto H0 as ∑k=1m~dkϕ~k(t,s,x,z) and the RK of M as R~. Following the same arguments in Section 2.2, we estimate g via the following PLS with L1 penalties:

1n∑i=1n∑j=1n{Yij−〈g,∫T∫TR~t,s,Xi(t),Xi(s)vj(t)dsdt〉}2+λ1∑k=1m~w1k∣dk∣+λ2∑v=1q~w2v‖P~vg‖, (18)

where P~v is the projection operator onto H~v, λ1 and λ2 are tuning parameters, and 0 ≤ w1k,w2v < ∞ are pre-specified weights that can be selected based on some initial estimates. One may set w11 = 0 when ϕ~1=1 to avoid penalty to the constant functions. Note that if one subspace, say H1, collects all components in (8), then ‖P~1g‖=(‖g4‖2+‖g14‖2+‖g24‖2+‖g34‖2+‖g124‖2+‖g234‖2+‖g1234‖2)1∕2 is a group penalty that encourages all components in (8) to be simultaneously zero.

As in Zhang et al. (2011), instead of (18), Lemma 3.1 indicates that we can solve the following equivalent but more convenient minimization problem:

1n∑i=1n∑j=1n{Yij−〈g,∫T∫TR~t,s,Xi(t),Xi(s)vj(t)dsdt〉}2+λ1∑k=1m~w1k∣dk∣+τ0∑v=1q~w2vθv−1‖P~vg‖2+τ1∑v=1q~w2vθv, (19)

subject to θν ≥ 0 for 1≤v≤q~, where λ1, τ0, and τ1 are tuning parameters.

Lemma 3.1. Let θ=(θ1,⋯,θq~)T, and τ1=λ22∕4τ0. If g^ minimizes (18), set θ^v=τ01∕2τ1−1∕2‖P~vg^‖, then (θ, g^) minimizes (19). On the other hand, if (θ, g^) minimizes (19), then g^ minimizes (18).

We choose to solve (19) since it is similar to the PLS with L2 penalty in (16). The additional penalties on θv’s control the sparsity of components in the SS ANOVA model for the purpose of model selection. Specifically, θν = 0 implies ‖P~vg‖=0. It is easy to show that the solution to (19) is

g(t,s,x,z)=∑k=1m~dkϕ~k(t,s,x,z)+∑v=1q~w2v−1θv∑i=1n∑j=1ncijξ~ij(t,s,x,z), (20)

where ξ~ij(t,s,x,z)=∫T∫TR~v(t,s,x,z,t′,s′,Xi(t′),Xi(s′))vj(t′)ds′dt′ and R~v is the RK in H~v. Hence, (19) reduces to

1n∑i=1n∑j=1n(Yij−∑k=tm~a~ijkdk−∑v=1q~w2v−1θv∑k=1n∑l=1ncklb~ijkl)2+λ1∑k=1m~w1k∣dk∣+τ0∑v=1q~w2v−1θv∑i=1n∑j=1n∑k=1n∑l=1ncijb~ijklckl+τ1∑v=1q~w2vθv, (21)

where

a~ijk=∫T∫Tϕ~k(t,s,Xi(t),Xi(s))vj(t)dsdt,b~ijkl=∫T∫Tξ~kl(t,s,Xi(t),Xi(s))vj(t)dsdt.

Let d = (d1,…,dm~)⊤, c = (c11,c21,…,cnn)⊤, w2 = (w21,…,w2q~)T, T be an n2×m~ matrix with the (i + (j−1)n,k)-th element as a~ijk, Σν be an n2 × n2 matrix with the (i + (j−1)n,k + (l−1)n)-th element as b~ijkl, and Σ=∑v=1q~w2v−1θvΣv. Then (21) is simplified as

1n‖Y−Td−Σc‖2+λ1∑k=1m~w1k∣dk∣+τ0cTΣc+τ1w2Tθ (22)

subject to θν ≥0, ν = 1,…,q~.

We use a backfitting algorithm to solve (22). With fixed d and θ, (22) reduces to (16) with a working response Y = Y − Td. Therefore, we can update c using existing methods with τ0 chosen by the GCV or the REML method. With fixed c and τ0, noting that Σ is linear in θ, (22) reduces to the standard LASSO for a linear regression model subject to constraints on θ. With fixed θ and τ1, we apply an LASSO regularization procedure to update d with λ1 chosen by k-fold cross-validation. Then with fixed d and λ1, we apply a constrained quadratic programming procedure to update θ. The selection of τ1 is equivalent to finding M such that w2Tθ≤M. We apply the k-fold cross-validation or the BIC method to select M. The complete algorithm can be found in the Supplemental Materials.

4. Statistical Properties

In this section, we study consistence and convergence rate of the regression function estimates g^ and gˇ as well as their components in the SS ANOVA decompositions under both the L1 and L2 penalties.

4.1. Loss Function and Regularity Conditions

We will first define a loss function similar to that in Gu (2013). Denote the least squares as l(g)=∫T{Y(t)−∫Tg(t,s,X(t),X(s))ds}2dt=∑j=1∞Qj(g,Y) where Qj(g,Y)={〈Y,vj〉−〈g,∫T∫TRt,s,x(t),x(s)vj(t)dsdt〉M}2. Let D be the Fr e chet derivative with respect to g, uj(g; Y) = DQj(g; Y), wj(g; Y) = D2Qj(g; Y), u(g;Y)=∑j=1nuj(g;Y), and w(g;Y)=∑j=1nwj(g;Y). It is easy to check that the first and second Fr e chet derivatives of Qj(g,Y) at directions h1 and h2 are

uj(g;Y)(h1)=−2{〈Y,vj〉−〈g,∫T∫TRt,s,x(t),x(s)vj(t)dsdt〉M}×〈h1,∫T∫TRt,s,x(t),x(s)vj(t)dsdt〉M,wj(g;Y)(h1,h2)=2〈h1,∫T∫TRt,s,x(t),x(s)vj(t)dsdt〉M×〈h2,∫T∫TRt,s,x(t),x(s)vj(t)dsdt〉M.

Let g0 be the true function of g and wj(g;Y) = wj(g;Y)(g, g). We denote

V(g1,g2)=2E[g1g2E{D2l(g0)}]=2E[g1g2E{∑j=1∞wj(g0;Y)}]

as a quadratic functional, and define the loss function

V(g)=V(g,g)=2E[g2E{∑j=1∞wj(g0;Y)}]. (23)

Note that 〈g0,∫T∫TRt,s,x(t),x(s)vj(t)dsdt〉M=0 when j ≥ n + 1. Consequently

V(g)=2E[g2E{w(g0;Y)}]. (24)

We shall show that the loss function V(g) is equivalent to the integrated mean square loss function E(g2) under some conditions. We denote J(g1,g2)=〈P1∗g1,P1∗g2〉∗ as another quadratic functional, and define the penalty function J(g)=J(g,g)=〈P1∗g,P1∗g〉∗=‖P1∗g‖∗2. A quadratic functional V is said to be completely continuous with respect to another quadratic functional J, if for any ϕ > 0, there exist a finite number of linear functionals L1, …,Lk such that Ljf = 0, j = 1,…,k, implies that V(g)≤ϕJ(g). To derive the convergence rate, we need the following conditions.

Condition 1. V is completely continuous with respect to J=‖P1∗g‖∗2.

Condition 1 is satisfied when c1 ≤ w(g0; y) ≤ c2 holds for some positive constants c1 and c2. From Theorem 3.1 of Weinberger (1974), under Condition 1, there exist eigenvalues ρk and eigenfunctions ψk such that V(ψk, ψj) = δk,j and J(ψk, ψj) = ρkδk,j where δk,j is the Kronecker delta and 0 ≤ ρk ↑ ∞.

Define a quadratic functional as

1n∑i=1nu(g0i;Yi)gi+12V(g−g0)+λ2J(g), (25)

where gi = g(t,s,Xi(t),Xi(s)) and g0i = g0(t,s,Xi(t),Xi(s)). By the Fourier series expansion with basis {ψk}, we have g=∑kζkψk and g0=∑kζk,0ψk. It is easy to show that the minimizer g~ of (25) has Fourier coefficients

ζ~k=(βk+ζk,0)∕(1+λρk),whereβk=−n−1∑i=1nu(g0i;Yi)ψk(t,s,Xi(t),Xi(s)).

We need the following conditions for the eigenvalues ρk and the eigenfunctions ψk.

Condition 2. For k large enough and some β > 0, the eigenvalues ρk satisfy ρk > βkr with r > 1 and.

Condition 3. For any k and j, Var {ψk(X)ψj(X)w(g0(X),Y)} ≤ c3 with some c3< ∞, where ψk (X) = ψk(t,s,X(t),X(s)) and g0(X) = g0(t,s,X(t),X(s)).

Condition 4. For g in a convex set B0 around g0 containing ĝ and ∑kρkζk,02<∞, c4w(g0, Y) ≤ w(g, Y) ≤ c5w(g0, Y) holds uniformly for some 0 < c4 < c5 < ∞.

Conditions 1-4 are common assumptions for convergence rate analysis of the SS ANOVA estimates, which were also made in Gu (2013). The ∑kρkζk,02<∞in

Condition 2 is a special case in Theorem 9.15 with p = 1 in Gu (2013), and states that the growth rate of the eigenvalues ρk is at kr, which controls how fast λ approaches zero. Condition 3 requires the fourth moment of ψi is bounded. Condition 4 bounds w(g, Y) at g in a convex set B0 around g0.

4.2. Convergence Results Under L2 Penalty

In order to present our theorem, we need the Op notation. For a sequence of random variables, {An}, and a sequence of constants, {an}, the notation An = Op(an), means that {An / an} is stochastically bounded (or bounded in probability). That is, for any τ > 0, there exit a constant K(τ) and an integer n(τ) such that if n ≥ n(τ), then

P(∣An∕an∣≤K(τ))≥1−τ.

More details and examples of this can be found in Section 1.2 of Serfling (1980).

Theorem 4.1. Assume that g0∈M. Under Conditions 1-4, as λ → 0 and nλ2/r → ∞,

(V+λJ)(g^−g0)=Op(n−1λ−1∕r+λ).

For the penalized least square estimation, we transform the integration in the objection function (11) to a summation criterion by using EFPCs of Y(t). Different from the proofs of consistence and convergence rate of function estimation in Gu (2013), we need to deal with an infinite summation for EFPC decomposition of Y(t). The proof of Theorem 4.1 is presented in the Supplemental Materials. In addition, we derive convergence rate for each term in the SS ANOVA model as follows.

To study the convergence rate of each component in the decomposition of g, as Lin (2000), we consider the special case with T=R=[0,1] and g∈⊗4W2m([0,1]) where ⊗kW2m([0,1]) denotes the tensor product of k Sobolev spaces W2m([0,1]) defined in (6). Then the main effects lies in W2m([0,1]) and the kth order interactions lies in a tensor product space of ⊗kW2m([0,1]). From the common norm ‖⋅‖W2m, in W2m([0,1]), we can deduce a norm ‖⋅‖⊗qW2m in the tense product space of ⊗qW2m([0,1]). For example with ⊗qW20([0,1]), the norm is ‖g(z)‖⊗qW202=∫g2(z)dz. Following Lin (2000), under the condition c1 ≤ w(g0;y) ≤ c2 for some positive constants c1 and c2, we can show that with m > 0 the norms ‖⋅‖⊗qW2m2 and ∣⋅∣⊗qW2m2=V(⋅)+‖P1∗⋅‖∗2 are equivalent on the tensor product space ⊗qW2m([0,1]), and when m = 0 the norms ‖⋅‖⊗qW202 and ∣⋅∣⊗qW202=V(⋅) are equivalent on the tensor product space ⊗qW20([0,1]), denoted by ‖⋅‖⊗qW2m2∼∣⋅∣⊗qW2m2 and ‖⋅‖⊗qW202∼∣⋅∣⊗qW202=V(⋅). From the SS ANOVA decomposition (7), it is easy to see that

‖g‖⊗qW2m2=μ2+‖g1‖W2m2+‖g3‖W2m2+‖g4‖W2m2+‖g13‖⊗2W2m2+‖g14‖⊗2W2m2+‖g24‖⊗2W2m2+‖g34‖⊗2W2m2+‖g124‖⊗3W2m2+‖g234‖⊗3W2m2+‖g1234‖⊗4W2m2.

Thence, we have

∣g∣⊗qW2m2∼μ2+‖g1‖W2m2+‖g3‖W2m2+‖g4‖W2m2+‖g13‖⊗2W2m2+‖g14‖⊗2W2m2+‖g24‖⊗2W2m2+‖g34‖⊗2W2m2+‖g124‖⊗3W2m2+‖g234‖⊗3W2m2+‖g1234‖⊗4W2m2. (26)

Since ∣g∣⊗qW202=V(g), we have ∣g^−g0∣⊗qW202=Op(n−1λ−1∕r+λ). Combined with (26), we have the following corollary.

Corollary 4.1. Assume that g∈⊗4W2m([0,1]), the conditions in Theorem 4.1 hold, and c1 ≤ w(g0;y) ≤ c2 holds for some positive constants c1 and c2. As λ → 0 and nλ2/r ↑ ∞, we have

‖g^I−g0I‖W202=Op(n−1λ−1∕r+λ),I=1,3,4,(1,3),(1,4),(2,4),(3,4),(1,2,4),(2,3,4),(1,2,3,4), (27)

where indices inside each index set I indicate the component in the SS ANOVA decomposition.

Corollary 4.1 provides convergence rates of the component estimators in the SS ANOVA decomposition under the integrated mean square loss function. If we set the tuning parameter λ as λ=[n(logn)−a]−m2m+1 where a > 0, it satisfies that » → 0 and nλ2/r → ∞. Hence, we have for n larger enough,

n−1λ−1∕r=n−(1−m(2m+1)r)(logn)−am(2m+1)r<[n(logn)−a]−m2m+1.

From Theorem 4.1 and Corollary 4.1, it follows that

Op(n−1λ−1∕r+λ)=Op([n(logn)−a]−m2m+1).

Furthermore, when r ≥ max{2,2m}, we set λ=[n(logn)−a]−2m2m+1, then λ → 0 and nλ2/r → ∞. We have faster convergence rates for g^ and its components as

Op(n−1λ−1∕r+λ)=Op([n(logn)−a]−2m2m+1),

which is similar to the convergence result in Lin (2000).

4.3. Convergence Results Under L1 penalty

We consider model (5) with g∈M where the model space is given in (17). Write g = g(0) + g(1) where g(0)∈H0 and g(1)∈H1⊕⋯⊕Hq~. Then

wj(g;Y)(h1,h2)=2〈h1,∫T∫TR~t,s,x(t),x(s)vj(t)dsdt〉M×〈h2,∫T∫TR~t,s,x(t),x(s)vj(t)dsdt〉M=2〈h1,∫T∫TR~t,s,x(t),x(s)0vj(t)dsdt〉M×〈h2,∫T∫TR~t,s,x(t),x(s)0vj(t)dsdt〉M+2〈h1,∫T∫TR~t,s,x(t),x(s)1vj(t)dsdt〉M×〈h2∫T∫TR~t,s,x(t),x(s)1vj(t)dsdt〉M=wj0(g(0);Y)+wj1(g(1);Y),

where R~0 and R~1 are the RKs of H0 and H1⊕⋯⊕Hq~. The cross term is zero because H0 and H1⊕⋯⊕Hq~ are orthogonal to each other. Let

w0(g(0);Y)=∑j=1nwj0(g(0);Y)andw1(g(1);Y)=∑j=1nwj1(g(1);Y).

Define V*(g) = V0(g(0)) + V1(g(1)) and J*(g) = J0(g(0)) + J1(g(1)), where

V0(g(0))=2E[g(0)2E{w0(g(0);Y)}],V1(g(1))=2E[g(1)2E{w1(g(1);Y)}],J0(g(0))=∑k=1m~∣∣dkϕk∣∣=∑k=1m~∣dk∣,andJ1(g(1))=∑j=1q~∣∣P~vg‖.Letgˇanddˇk(k=1,…,m~)

be the minimizer of (18), and λˇ=max{λ1,λ2}. Then, we have the following convergence results.

Theorem 4.2. Assume that g0∈M. Under Conditions 1- 4 with g^ in Condition 4 being replaced by gˇ, as λˇ→0 and nλˇ2∕r→∞,

(V∗+λˇJ∗)(gˇ−g0)=Op(n−1∕2λˇ−1∕2r+λˇ1∕2).

Corollary 4.2. Assume conditions in Theorem 4.2 hold and c~1≤w(g0;y)≤c~2 holds for some positive constants c~1 and c~2, as λˇ→0 and nλˇ2∕r→∞, we have

∣dˇk−d0,k∣=Op(n−1∕2λˇ−1∕2r+λˇ1∕2)k=1,⋯,m~,‖P~vgˇ−P~vg0‖W02=Op(n−1∕2λˇ−1∕2r+λˇ1∕2)v=1,⋯,q~,

where {d0,k} are coefficients of parametric functions in the true function g0. Proofs of Theorem 4.2 and Corollary 4.2 are given in the Supplemental Materials.

Remark The convergence rate in Theorem 4.2 and the component convergence rate in Corollary 4.2 are the square root of the rate in Theorem 4.1 and Corollary 4.1. This is due to the fact that the square of L2 norm was used in the Section 4.2 while the L2 norm was used in the Section 4.3.

5. Simulation Results

Simulation studies are conducted to evaluate the performance of the proposed model selection and estimation methods. We generate data based on model (5) with T=R=[0,1]. For convenience of presentation, let f(t)=∫01g(t,s,X(t),X(s))ds. We consider a factorial design with the following three choices of f.

M1: f(t) = 1 + 0.5 cos(2πt) + 3X(t) + 3(X(t) − 0.5)(2t − 1)2,

M2: f(t)=1+3∫01(2X(s)−1)ds+3(t−0.5)2+10∫01(X(s)−0.5)((2s−1)2−1∕3)ds,

M3: f(t)=1+5(2X(t)−1)3+2∫01(2X(s)−1)ds+5(t−0.5)2.

M1 is an NCM model with g1(t) = 0.5cos(2πt), g3(X(t)) = 3X(t), and g13 = 3(X(t) − 0.5)(2t − 1)2. M2 is an EFLM model with g1(t) = 3(t − 0.5)2, g4(X(s)) = 3(2X(s) − 1), and g24 = 10(X(s) − 0.5)((2s − 1)2 − 1/3).

M3 is neither an NCM nor an EFLM model with g1(t) = 5(t − 0.5)2, g3(X(t)) = 5(2X(t) − 1)3, and g4(X(s)) = 2(2X(s) − 1). All other terms in the SS ANOVA decomposition not mentioned above are set to be zero. Two choices of sample size n = 40 and n = 80; and two choices of error standard deviation: σ = 0.2 and σ = 0.5. For M1, we generate Xi(t) = ai1(cos{2π(t + ai2)} + 1)/2 where ai1, ai2∼iidUniform(0,1). For M2 and M3, Xi(t)=exp(Xi∗(t))∕(1+exp(Xi∗(t))) where Xi∗(t)∼iidGP(0,k), and GP(0, k) is the Gaussian process with mean 0 and Mat e rn kernel k(t, s) = (1 + 8∣t − s∣) exp (−8∣ ul − vl∣). We generate observations of Yi(t) on 20 equally spaced grid points in [0,1]. All simulation are repeated 100 times. In all simulations in this section and real data examples in the next section, we compute the approximate estimates using the first p = 10 EFPCs and estimate smoothing parameters using the GCV method.

We assume that g∈M=⊗4W22([0,1]). It is known that W22[0,1]=H(0)⊕H(1)⊕H(2) which corresponds to constant, linear, and smooth (nonparametric) functions (Wahba, 1990; Wang, 2011). Then

⊗4W22([0,1])=(H1(0)⊕H1(1)⊕H1(2))⊗(H2(0)⊕H2(1)⊕H2(2))⊗(H3(0)⊕H3(1)⊕H3(2))⊗(H4(0)⊕H4(1)⊕H4(2))=H∅⊕∑k=14∑j=12Hk(j)⊕∑1≤k1<k2≤4∑j1,j2=12Hk1k2(j1j2)⊕∑1≤k1<k2<k3≤4∑j1,j2,j3=12Hk1k2k3(j1j2j3)⊕∑j1,j2,j3,j4=12H1234(j1j2j3j4), (28)

where H∅ contains constant functions; Hk(1) and Hk(2) contain the linear and smooth main effects of the kth variable; Hk1k2(11), Hk1k2(12), Hk1k2(21), and Hk1k2(22) contain the linear-linear, linear-smooth, smooth-linear, and smooth-smooth two-way interactions between the k1th and the k2th variables; and so on. When fitting the SS ANOVA model under the L2 penalty, we regroup the model space M with

H0=H∅⊕∑k∈I1Hk(1)⊕∑(k1,k2)∈I2Hk1k2(11)⊕∑(k1,k2,k3)∈I3Hk1k2k3(111)⊕H1234(1111)

collecting all parametric components for I1 ={1,3,4}, I2 ={(1,3),(1,4), (2,4),(3,4)}, I3 ={(1,2,4), (2,3,4)}, and each of the remaining subspaces constitute to H1 to Hq with q = 10 in (10). Specifically, H1 to H3 collect smooth main effects with H1=H1(2), H2=H3(2), and H3=H4(2); Hl=Hk1k2(12)⊕Hk1k2(21)⊕Hk1k2(22) for l = 4,5,6,7 collects the linear-smooth, smooth-linear, and smooth-smooth two-way interactions between the k1th and the k2th variables for (k1, k2) ∈ I2; Hl for l = 8, 9 collects three-way interactions that involving nonparametric components for variables in l3, and H10 collects four-way interactions that involving nonparametric components. Notice that subspaces containing g2, g12, g23, g123 and g134 are removed for identifiablity. When fitting the SS ANOVA model with L1 penalty, we use the same regrouping for comparison with the L2 estimates. That is, we set Hl=Hl for l = 0,1,⋯,10 and enforce L1 penalties to all elements in the subspace H0.

For comparison, we also fit the NCM (4) and EFLM (3) with reduced SS ANOVA models under conditions (8) and (9) respectively. We evaluate the performance using the following integrated mean square errors (IMSE),

IMSE=∫01(f^(t)−f(t))2dt.

We apply both the L2 and L1 penalized methods to fit the SS ANOVA model with all main effects and two-way interactions, and apply the L2 penalized method to fit the NCM and EFLM. Table 1 presents IMSEs under different simulation settings. The IMSEs are generally smaller when the error variance is smaller or the sample size is larger. In general fitting correct models leads to smaller IMSEs. The proposed general model has comparable IMSEs with NCM under M1 and comparable IMSEs with EFLM under M2. Under M3 when neither NCM nor EFLM holds, the general model has much smaller IMSEs. IMSEs under L1 penalty are larger than that under L2, confirming the theoretical results in Theorems 4.1 and 4.2 that the PLS estimate under L1 has a slower convergence rate.

Table 1.

Averages and standard deviations (in parentheses) of the integrated mean square error (IMSE) for M1, M2 and M3 with sample sizes n = 40 and n = 80 and error standard deviations σ = 0.2 and σ = 0.5.

n σ Method M1 M2 M3
40 0.2 L 2 0.006(0.001) 0.003(0.001) 0.006(0.001)
L 1 0.080(0.019) 0.011(0.008) 0.019(0.006)
NCM 0.003(0.007) 0.321(0.039) 0.152(0.019)
EFLM 0.011(0.002) 0.002(0.001) 0.096(0.049)
40 0.5 L 2 0.017(0.006) 0.010(0.003) 0.017(0.004)
L 1 0.080(0.032) 0.022(0.010) 0.034(0.007)
NCM 0.016(0.036) 0.323(0.039) 0.158(0.019)
EFLM 0.034(0.011) 0.006(0.003) 0.172(0.064)
80 0.2 L 2 0.005(0.001) 0.003(0.000) 0.005(0.001)
L 1 0.081(0.012) 0.004(0.001) 0.012(0.004)
NCM 0.003(0.017) 0.324(0.025) 0.155(0.013)
EFLM 0.009(0.002) 0.002(0.000) 0.093(0.012)
80 0.5 L 2 0.014(0.007) 0.006(0.002) 0.013(0.002)
L 1 0.086(0.021) 0.012(0.005) 0.021(0.006)
NCM 0.021(0.050) 0.329(0.025) 0.158(0.013)
EFLM 0.022(0.009) 0.004(0.002) 0.146(0.018)
 

For the purpose of selection, we are interested in the non-zero entries in d and θ. To evaluate the model selection performance, we compute three criteria: specificity (SPE), sensitivity (SEN) and F1 scores:

SPE=TNTN+FP,SEN=TPTP+FN,F1=2TP2TP+FN+FP,

where TP, TN, FP and FN are the numbers of true positives, true negatives, false positives and false negatives. Here we consider the non-zero parametric or nonparametric components in (19) as true positives and the corresponding non-zero entries in the estimates d and θ as identified positives.

Table 2 presents sensitivities, specificities, and F1 scores. Overall, the proposed method performed very well under all simulation settings. The selection performance improves as sample size increases or error variance decreases.

Table 2.

Averages and standard deviations (in parentheses) of specificity (SPE), sensitivity (SEN), and F1 score for M1, M2 and M3 models with sample sizes n = 40 and n = 80 and error stadard deviations σ = 0.2 and σ = 0.5.

n σ SPE SEN F 1
40 0.2 M1 0.929(0.064) 0.997(0.033) 0.849(0.126)
M2 0.928(0.053) 0.977(0.085) 0.828(0.093)
M3 0.872(0.065) 1.000(0.000) 0.806(0.086)
0.5 M1 0.895(0.049) 0.997(0.033) 0.777(0.084)
M2 0.932(0.054) 0.98(0.080) 0.841(0.107)
M3 0.842(0.068) 1.000(0.000) 0.768(0.082)
80 0.2 M1 0.944(0.061) 1.000(0.000) 0.881(0.122)
M2 0.938(0.067) 1.000(0.000) 0.870(0.130)
M3 0.972(0.039) 1.000(0.000) 0.951(0.065)
0.5 M1 0.907(0.065) 1.000(0.000) 0.808(0.115)
M2 0.920(0.070) 0.997(0.033) 0.833(0.127)
M3 0.908(0.049) 1.000(0.000) 0.850(0.070)
 

6. Real examples

In this section, we illustrate the proposed method using four real data examples.

Example 1. (Canada weather). Weekly temperature and precipitation were collected from 35 Canadian weather observation stations, and the goal is to study the relationship between temperature (X(t)) and precipitation (Y(t)). Ramsay et al. (2010) fitted the CLM and FLM models.

Example 2. (Gait curve). The gait curve data consist of movement cycle curve of hip and knee angles in degrees of 39 boys) Ramsay and Silverman, 2006). Relationship of knee angle curve (Y(t)) and hip angle curve (X(t)) is studied. We fit the proposed model and check the NCM model.

Example 3. (Growth curve). The growth curve data consist of heights of 39 boys and 54 girls from age 1 to 18 (Jones and Bayley, 1941). Regarding the first derivative of growth curve as the response Y(t) and the growth curve as the covariate X(t), Verzelen et al. (2012) studied the dynamics of girl growth by a nonlinear dynamic systems. We fit the proposed model and check the NCM model in Verzelen et al. (2012). We use functions in the fda R package, smooth.basis and deriv.fd, to smooth the growth curves and then compute their first derivatives.

Example 4. (Stock price). We collect weekly stock prices of GE (General Electric), IBM (International Business Machines), and JPM (JP Morgan Chase & Co.) from 1996 to 2015. The goal is to study the relationship between two stock prices. For illustration, we build two models to investigate how weekly GE prices or IBM prices (Y(t)) depend on weekly JPM price (X(t)).

Data in Example 1-3 can be found in the fda R package, and stock data in Example 4 will be provided with code in the supplements. We transform the domains and ranges for all real data examples such that T=R=[0,1]. We consider model (5) for all real data examples with g∈M=⊗4W22([0,1]) and the same SS ANOVA regrouping as discussed in Section 5 including all high-order interactions.

We first apply our model selection method with L1 penalty. We avoid the selection of the constant function by setting w11 = 0 where ϕ~1=1. Table 3 lists the selected components. For Gait data, the selected model reduces to an NCM model which indicates that the knee angle depends on the current time and current hip angle only. The other selected models contain components that are functions of X(t) and X(s), indicating that Y(t) depends on both current values and the whole function.

Table 3.

Model selection results from the PLS with L1 penalty. The selected and removed components are denoted by 1s and 0s respectively.

Function Components
Data g 1 g 3 g4 g13 g14 g34 g24 g124 g234 g1234
Weather 1 0 1 0 1 1 1 0 0 0
Gait 1 1 0 1 0 0 0 0 0 0
Growth 1 1 0 1 1 0 1 0 0 1
GE/JPM 1 1 0 0 0 0 0 1 1 1
IBM/JPM 0 1 0 1 0 0 0 0 1 0
 

After selection with the L1 penalty, we then use the PLS with L2 penalty to estimate the selected models. Figure 1 presents the 3D-plots for Gait and Weather data. Note that the selected model for Gait data is an NCM model, which only depends on current time t and x. Therefore, to visualize the estimate of function g(t,s,x,z) and two-way interaction g13, we construct 3D-plots of g against (t, x) and g13 against (t, x). For Weather data, we construct 3D-plots of g(t,s,x,z) against (t, s) with x and z fixed at their mean values, and 3D-plots of g(t,s,x,z) against (x, z) with t and s set to be the value of the tenth observation time. The 3D plots show complex interactions between variables.

Fig. 1.

Fig. 1

3D-plots for estimates of the function g(t,s,x,z) or interaction. Gait data: (a) g as a function of t and x, (b) the interaction g13 as a function of t and x; Weather data: (c) g as a function of t and s with x and z fixed at their mean values, (d) g as a function of x and z with t and s set to be the value of the tenth observation time.

To evaluate the prediction performance, we consider three approaches to fit model (5): PLS with L2 penalty (denoted as L2), PLS with L1 penalty (denoted as L1), and PLS with L2 penalty after selection with the L1 penalty (denoted as L1 + L2). For comparison, we consider EFLM and NCM with PLS subject to the L2 penalty. We use 10-fold cross validation to show performance of prediction from the 5 fitted methods and average the prediction errors on the test fold. For each real data example, we compute the mean square prediction error (MSPE)

MSPE=1n∑j=110∑j∈jth fold∫01(Yi(t)−Y^i(−j)(t))2dt,

where Y^i(−j)(t) is the prediction of Yi(t) without using data in the jth fold.

MSPE’s are listed in Table 4. For the Weather data, the EFLM has the smallest MSPE. For the Gait data, the NCM has the smallest MSPE, which agrees with model selection result. For the Growth and Stock price data, the propose general model has smaller MSPEs. Overall, the L1 + L2 approach performs well.

Table 4.

Mean square prediction error (MSPE) and their standard deviations (in parentheses) based on 10-fold cross validation for each real data set.

Weather Gait Grouth GE/JPM IBM/JPM
L1+L2 0.537(0.737) 0.106(0.029) 0.204(0.08) 0.470(0.445) 0.509(0.341)
L1 0.592(0.373) 0.261(0.034) 0.245(0.097) 0.553(0.561) 0.552(0.375)
L2 0.540(0.694) 1.797(1.083) 0.620(0.053) 0.496(0.477) 0.561(0.371)
EFLM 0.371(0.502) 1.043(0.592) 0.659(0.103) 0.64(0.663) 0.577(0.362)
NCM 0.669(0.512) 0.079(0.017) 0.208(0.081) 0.588(0.473) 0.516(0.353)
 

7. Conclusion and Discussion

In this paper, we developed a unified framework for estimation, model selection and theoretical study for function-on-function nonparametric regression. We proposed a flexible function-on-function nonparametric regression model that includes concurrent linear model, function linear model, extensions of function linear model, and nolinear concurrent model as special cases. We modeled the regression function using SS ANOVA decomposition of a tensor product of RKHS’s. We fitted the model using penalized least squares with L1 or L2 penalty. The L1 regularization can be used to select important components in an SS ANOVA model and check the adequacy of existing models. Consistency and convergence rates of regression function estimation were obtained. We note that the proposed estimation and model selection methods apply to existing models which are special cases of the proposed general model, and the theoretical results fill the gaps in the literature.

It is not difficult to extend the proposed methods to fit nonparametric scalar-on-function and function-on-scalar regression models. For simplicity, we considered the case where functional data were observed at equally spaced points in our simulation. When functional data are observed at irregular points, we can estimate each function first using any existing smoothing methods, and then apply the proposed method (Yao et al. 2005 a; Zhang and Chen 2007). Future research topics include extending the proposed model and methods to more complex data such as repeated measures, developing methods and theories with sparse and irregular functional data, and developing computational methods for big data.

Supplementary Material

Supp 1

Acknowledgement

The first two authors contributed equally to this work. Yuedong Wang is the corresponding author. The authors thank the editor, the associate editor, and the two referees for their constructive comments and suggests that have led to significant improvement in this article. Zhanfeng Wang’s research was partially supported by research grants from National Natural Science Foundation of China (No. 11971457), and Anhui Provincial Natural Science Foundation (No. 1908085MA06). Ping Ma’s research was partially supported by U.S. National Science Foundation under grants DMS-1903226, DMS-1925066, DMS-2124493, the U.S. National Institute of Health under grant R01GM122080. Yuedong Wang’ s research was partially supported by U.S. National Science Foundation under grant DMS-1507620, the U.S. National Institute of Health under grant R01DK130067.

Footnotes

SUPPLEMENTAL MATERIALS

Supplementary Materials: Relationship between the proposed model and some existing models, backfitting and model selection algorithm and some proofs. (.pdf file)

Computational R code and data set: Computational R code for simulation and real examples. This file also contains the data set used in one of the real examples. (.zip file)

Contributor Information

Zhanfeng Wang, International Institute of Finance, The School of Management, University of Science and Technology of China.

Hao Dong, Department of Statistics and Applied Probability, University of California, Santa Barbara.

Ping Ma, Department of Statistics, University of Georgia.

Yuedong Wang, Department of Statistics and Applied Probability, University of California, Santa Barbara.

References

  1. Ferraty F, Laksaci A, Tadj A and Vieu P (2011), ‘Kernel regression with functional response’, Electronic Journal of Statistics 5, 159–171. [Google Scholar]
  2. Ferraty F, Van Keilegom I and Vieu P (2012), ‘Regression when both response and predictor are functions’, Journal of Multivariate Analysis 109, 10–28. [Google Scholar]
  3. Ferraty F and Vieu P (2006), Nonparametric Functional Data Analysis, Springer, New York. [Google Scholar]
  4. Gu C (2013), Smoothing spline ANOVA models, Vol. 297, Springer Science & Business Media. [Google Scholar]
  5. Gu C and Qiu C (1993), ‘Smoothing spline density estimation: Theory’, The Annals of Statistics pp. 217–234. [Google Scholar]
  6. Hsing T and Eubank R (2015), Theoretical Foundations of Functional Data Analysis, with an Introduction to Linear Operators, Wiley, West Sussex. [Google Scholar]
  7. Jones H and Bayley N (1941), ‘The berkeley growth study’, Child Development 12, 167–173. [Google Scholar]
  8. Kardi H, Duflos E, Preux P, Canu S, Rakotomamonjy A and Audiffren J (2016), ‘Operator-valued kernels for learning from functional response data’, Journal of Machine Learning Research 17, 1–54. [Google Scholar]
  9. Kim JS, Maity A and Staicu A-M (2018), ‘Additive nonlinear functional concurrent model’, Statistics and its interface 11(4), 669. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Kim JS, Staicu AM, Maity A, Carroll RJ and Ruppert D (2018), ‘Additive function-on-function regression’, Journal of Computational and Graphical Statistics 27, 234–244. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Kokoszka P and Reimherr M (2017), Introduction to Functional Data Analysis, Chapman and Hall/CRC, Boca Raton. [Google Scholar]
  12. Lian H (2007), ‘Nonlinear functional models for functional responsesa in reproducing kernel hilbert spaces’, The Canadian Journal of Statistics 35, 597–606. [Google Scholar]
  13. Lin Y (2000), ‘Tensor product space anova models’, The Annals of Statistics 28, 734–755. [Google Scholar]
  14. Lin Z, Müller H and Yao F (2018), ‘Mixture inner product spaces and their application to functional data analysis’, The Annals of Statistics 46, 370–400. [Google Scholar]
  15. Lin Z and Yao F (2019), ‘Intrinsic riemannian funcitonal data analysis’, The Annals of Statistics 47, 3533–3577. [Google Scholar]
  16. Ling N and Vieu P (2018), ‘Nonparametric modelling for functional data: selected survey and tracks for future’, Statistics: A Journal of Theoretical and Applied Statistics 52, 1–16. [Google Scholar]
  17. Ma H and Zhu Z (2016), ‘Continuously dynamic additive models for functional data’, Journal of Machine learning Research 150, 1–13. [Google Scholar]
  18. Müller H-G and Yao F (2008), ‘Functional additive models’, Journal of the American Statistical Association 103, 1534–1544. [Google Scholar]
  19. Morris JS (2015), ‘Functional regression’, Annual Review of Statistics and Its Application 2, 321–359. [Google Scholar]
  20. Ramsay JO, Hooker G and Graves S (2010), Functional Data Analysis with R and Matlab, Springer, New York. [Google Scholar]
  21. Ramsay JO and Silverman BW (2006), Functional Data Analysis, 2nd edn, Springer, New York. [Google Scholar]
  22. Reimherr M, Sriperumbudur B and Taoufik B (2018), ‘Optimal prediction for additive function-on-function regression’, Electronic Journal of Statistics 12, 4571–4601. [Google Scholar]
  23. Reiss PT, Goldsmith J, Shang H and Ogden RT (2017), ‘Methods for scalar-on-function regression’, International Statistical Review 85, 228–249. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Scheipl F, Staicu AM and Greven S (2015), ‘Functional additive mixed models’, Journal of Computational and Graphical Statistics 24, 477–501. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Serfling RJ (1980), Approximation Theorems of Mathematical Statistics, John Wiley & Sons. [Google Scholar]
  26. Sun X, Du P, Wang X and Ma P (2018), ‘Optimal penalized function-on-function regression under a reproducing kernel hilbert space framework’, Journal of the American Statistical Association 113, 1601–1611. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Sun X, Zhong W and Ma P (2020), ‘An asymptotic and empirical smoothing parameters selection method for smoothing spline ANOVA models in large samples’, Biometrika 108, 1–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Verzelen N, Tao W and Müller H-G (2012), ‘Inferring stochastic dynamics from functional data’, Biometrika 99(3), 533–550. [Google Scholar]
  29. Wahba G (1990), Spline Models for Observational Data, Vol. 59 of CBMS-NSF Regional Conference Series in Applied Mathematics, SIAM, Philadelphia. [Google Scholar]
  30. Wang JL, Chiou JM and Müller HG (2015), ‘Functional data analysis’, Annu. Rev. Statist 3, 257–295. [Google Scholar]
  31. Wang Y (2011), Smoothing splines: methods and applications, CRC Press. [Google Scholar]
  32. Weinberger H (1974), Variational methods for eigenvalue approximation, CBMS-NSF regional conference series in applied mathematics, vol 15. SIAM, Philadelphia. [Google Scholar]
  33. Xing X, Liu M, Ma P and Zhong W (2020), ‘Minimax nonparametric parallelism test’, Journal of Machine Learning Research 21, 1–47. [PMC free article] [PubMed] [Google Scholar]
  34. Yao F, Müller H-G and Wang J-L (2005 a), ‘Functional data analysis for sparse longitudinal data’, Journal of the American Statistical Association 100(470), 577–590. [Google Scholar]
  35. Yao F, Müller H-G and Wang J-L (2005 b), ‘Functional linear regression analysis for longitudinal data’, The Annals of Statistics 33(6), 2873–2903. [Google Scholar]
  36. Yuan M and Cai TT (2010), ‘A reproducing kernel Hilbert space approach to functional linear regression’, The Annals of Statistics 38(6), 3412–3444. [Google Scholar]
  37. Zhang HH, Cheng G and Liu Y (2011), ‘Linear or nonlinear? automatic structure discovery for partially linear models’, Journal of the American Statistical Association 106(495), 1099–1112. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Zhang J-T and Chen J (2007), ‘Statistical inferences for functional data’, The Annals of Statistics 35(3), 1052–1079. [Google Scholar]
  39. Zhang X, Park B and Wang J (2013), ‘Time-varying additive models for longitudinal data’, Journal of the American Statistical Association 108, 983–998. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supp 1

RESOURCES