Variable Selection in Function-on-Scalar Regression

Abstract

For regression models with functional responses and scalar predictors, it is common for the number of predictors to be large. Despite this, few methods for variable selection exist for function-on-scalar models, and none account for the inherent correlation of residual curves in such models. By expanding the coefficient functions using a B-spline basis, we pose the function-on-scalar model as a multivariate regression problem. Spline coefficients are grouped within coefficient function, and group-minimax concave penalty (MCP) is used for variable selection. We adapt techniques from generalized least squares to account for residual covariance by “pre-whitening” using an estimate of the covariance matrix, and establish theoretical properties for the resulting estimator. We further develop an iterative algorithm that alternately updates the spline coefficients and covariance; simulation results indicate that this iterative algorithm often performs as well as pre-whitening using the true covariance, and substantially outperforms methods that neglect the covariance structure. We apply our method to two-dimensional planar reaching motions in a study of the effects of stroke severity on motor control, and find that our method provides lower prediction errors than competing methods.

1 Introduction

Regression models with functional responses and scalar predictors are routinely encountered in practice. These models face a challenge that also arises for traditional models: how to identify the important predictors among a potentially large collection. Functional-response models face the additional challenges of high dimensionality and residual correlation. There are few methods for variable selection in this class of models and none of them properly account for the correlation structure in the residuals. The purpose of this article is to address the current lack of methods for variable selection that account for residual correlation in function-on-scalar regression problems.

Our work is motivated by two-dimensional planar reaching data. As an assessment of upper extremity motor control, stroke patients and healthy controls made repeated reaching movements from a central point to eight targets arranged on a circle. The dataset consists of 57 subjects, including 33 patients suffering a unilateral stroke (meaning only one arm is affected) and 24 healthy controls, and contains motions made with both the dominant and non-dominant hands to each of the eight targets. Our analytic goal is to explore the effects of the potential predictors of motor control on these motions and to identify the most essential ones using variable selection. Among the potential predictors, the Fugl-Meyer score is a quantity that measures the severity of arm motor impairment (Fugl-meyer et al., 1975). It ranges from 0 to 66 with smaller values indicating more severe impairment and 66 indicating healthy function. Other potentially important predictors include target direction, whether the hand used was the dominant or non-dominant, and whether the hand used was contralesional (directly affected by the stroke) or ipsilesional (indirectly affected or unaffected).

Figure 1 shows the observed reaching motions for three subjects: a stroke patient with contralesional dominant hand in the left column; a stroke patient with contralesional non-dominant hand in the center column and a heathy control in the right column. Reaching motions made by contralesional hand display deviation from straight paths from the starting point to each target; these deviations may be consistent for contralesional dominant or non-dominant hands. While deviation from straightness is not obvious in the ipsilesional arm, other effects, like over-reach, are observed. The potential for differential effects of stroke severity on reaching motions indicates the importance of allowing interactions between predictors of interest.

Figure 1.

Observed reaching motions for three subjects. The top row shows the dominant hand and the bottom row shows the non-dominant hand of three subjects. The left column is a subject with a contralesional dominant hand. The center column is a subject with a contralesional non-dominant hand. The right column is a healthy control subject. Dashed lines are the straight paths to the eight targets terminating at the target location.

The observed data are horizontal and vertical coordinates of the hand position for each reaching motion as functions of time. We construct function-on-scalar regression models for the two outcome functions separately. Given scalar predictors x_ij, i = 1, ··· , n, j = 1, ··· , p and functional responses $y_i\left(t\right),i=1,\cdots ,n,t\in \mathcal{T}$, where $\mathcal{T}$ is some compact finite interval in $\mathbb{R}$, the linear function-on-scalar regression model is

$$y_i\left(t\right)={\beta }_0\left(t\right)+\sum _{j=1}^p{x}_{ij}{\beta }_j\left(t\right)+{ϵ}_i\left(t\right),i=1,\cdots ,n,t\in \mathcal{T}$$ (1)

where β_j(·), j = 0, ··· , p are the p + 1 coefficient functions and ε_i(·) ~ (0, Σ) is the error function drawn from a continuous stochastic process with expectation zero and covariance function $\Sigma (s,t)=\mathrm{cov}({ϵ}_i\left(s\right),{ϵ}_i\left(t\right)),s,t\in \mathcal{T}$.

A common model fitting framework for function-on-scalar regression is outlined by Chapter 13 of Ramsay and Silverman (2005), in which the coefficient functions β_j(·) are expanded using some set of basis functions and basis coefficients are estimated using ordinary least squares. The imposition of quadratic roughness penalties to enforce smoothness of the estimated coefficient functions is also common. Reiss et al. (2010) developed a fast automatic method for choosing tuning parameters in this model and accounted for correlated errors using generalized least squares. Goldsmith and Kitago (2015) develop a Bayesian approach that jointly models coefficient functions and the covariance structure, and applied their methods to the stroke kinematics dataset considered here.

When p is large, many scalar predictors may have no effect on the functional response and the corresponding coefficient functions would equal zero over all time points. In order to accurately identify the important predictors, we apply variable selection techniques when estimating the coefficient functions in Model (1). Since the coefficient functions are expanded using basis functions, the shape of each coefficient function is determined by a distinct group of basis coefficients. We therefore apply variable selection at the group level to include or exclude the vector of basis coefficients. The group lasso, proposed by Yuan and Lin (2006), is an extension of the classic lasso (Tibshirani, 1994) to the problem of selecting grouped variables. The lasso is known to induce biases in the included variables, so two alternative penalties, the smoothy clipped absolute deviation (SCAD) penalty (Fan and Li, 2001) and the minimax concave penalty (MCP) (Zhang, 2010), were proposed. These achieve consistency and asymptotic unbiasedness, and have been extended to grouped variable selection problem (Wang et al. (2007); Breheny and Huang (2013)).

Few approaches that consider variable selection in the context of functional regression models have been proposed in current literature. Wang et al. (2007) developed a penalized estimation procedure using group SCAD for variable selection in function-on-scalar regression assuming errors ε_i(·) are uncorrelated over their domain; this assumption is clearly violated in practice. Barber et al. (2015) presented Function-on-Scalar LASSO (FS-LASSO), a framework which extends the group LASSO to function-on-scalar regression; theory is developed for cases in which predictors are observed over dense or sparse grids. However, the bias for non-zero coefficients introduced by LASSO was not addressed, and the method does not account for correlation among residual curves. Gertheiss et al. (2013) proposed a variable selection procedure for generalized scalar-on-function linear regression models, in which the predictors are in the form of functions and responses are scalar; though they also consider regression models for functional data, the structure of their models is very different from the one considered here.

The main contributions of this paper is a method for variable selection in function-on-scalar regression that accounts for residual correction using tools from generalized least squares. We develop theory for this method and demonstrate its effectiveness in simulations that mimic our real-data application; direct comparisons with the method of Wang et al. (2007) and Barber et al. (2015) indicate superior performance of our proposed method for variable selection and prediction.

The rest of the article is organized as follows. In Section 2, we describe an estimation procedure for function-on-scalar regression models with errors that are uncorrelated over t using grouped variable selection methods. We then introduce our methods for the estimation of function-on-scalar regression models with correlated errors, including the development of an iterative method that refines the estimation of the error covariance and the variable selection. Simulations that resemble our motivating data examine and compare the numerical performance of competing methods in Section 3. An application of our method to the reaching motion data is given in Section 4. Finally, we present concluding remarks in Section 5. Our method is implemented in the user-friendly fosr.vs() function in the refund package (Ciprian Crainiceanu et al., 2014), available on CRAN, and code for our simulations is included in the supplementary material.

2 Methodology

2.1 Estimation for models with i.i.d. errors

Suppose {ϕ₁(·), ··· , ϕ_K(·)} is a set of pre-specified basis functions. The coefficient functions β_j(·), j = 0, ··· , p can be expanded as

$${\beta }_j(\cdot )=\sum _{k=1}^K{b}_{jk}{\varphi }_k(\cdot ).$$ (2)

Hence, Model (1) is expressed as

$$y_i\left(t\right)=\sum _{k=1}^K{b}_{0k}{\varphi }_k\left(t\right)+\sum _{j=1}^p{x}_{ij}\left(\sum _{k=1}^K{b}_{jk}{\varphi }_k\left(t\right)\right)+{ϵ}_i\left(t\right)$$ (3)

The problem is thereby reduced to estimating the basis coefficients {b_jk}_{j=0,··· , p; k=1,··· ,K}.

In practice, functions are observed on a discrete grid. For simplicity, we assume that the grid, denoted {t₁, ··· , t_D}, is shared across subjects. Let Y be the n × D matrix whose rows are vector-valued functional responses; Φ be the D × K matrix whose columns correspond to the K basis functions evaluated at {t₁, ··· , t_D}; and B be the (p + 1) × K matrix with jth row being the vector of basis coefficients for β_j(·). Then Model (3) can be expressed as

$$\mathit{Y}=XB{\Phi }^T+\mathit{E}$$ (4)

where X is the n×(p+1) design matrix and E is the n×D matrix containing vector-valued error functions.

Model (4) can be posed as a standard linear model in the following way. Let vec(Y^T) be the vector formed by concatenating the rows of Y, and note that $\mathrm{vec}\left({\left(XB{\Phi }^T\right)}^T\right)=(\mathit{X}\otimes \Phi )\mathrm{vec}\left({\mathit{B}}^T\right)$ where $\otimes$ represents the Kronecker product of two matrices. Then

$$\mathrm{vec}\left({\mathit{Y}}^T\right)=(\mathit{X}\otimes \Phi )\mathrm{vec}\left({\mathit{B}}^T\right)+\mathrm{vec}\left({\mathit{E}}^T\right)$$ (5)

and vec(B^T) can be estimated using least squares. An estimate of B̂ is obtained by rearranging vec (B̂^T).

To accurately identify the zero coefficient functions, we apply variable selection techniques when estimating vec(B^T) in Model (5). Let B_j be the vector of coefficients associated with the jth coefficient function β_j(·), specifically the jth row of B. Note that the “zeroth” row of B corresponds to the intercept function β₀(t), which we do not penalize. Setting the entire β_j(·) function to 0 is equivalent to setting all the entries of B_j to zero. Therefore, we apply variable selection techniques at the group level.

Variable selection can be achieved by penalizing the estimates of the coefficients. The general form of a group penalty is ${\sum }_{j=1}^p{p}_{\lambda ,\gamma }\left(\Vert {\mathit{B}}_j\Vert \right)$, where p_λ,γ(·) is the penalty function for the specific method and λ and γ are the tuning parameters. Therefore, the penalized estimator is obtained by minimizing

$$\frac{1}{2}{\Vert \mathrm{vec}\left({\mathit{Y}}^T\right)-(\mathit{X}\otimes \Phi )\mathrm{vec}\left({\mathit{B}}^T\right)\Vert }^2+nD\sum _{j=1}^p{p}_{\lambda ,\gamma }\left(\Vert {\mathit{B}}_j\Vert \right).$$ (6)

We use group MCP to perform variable selection; the penalty has the form

$$p_{\mathrm{mcp}}\left(\Vert {\mathit{B}}_j\Vert \right)=\{\begin{array}{cc}\lambda \Vert {\mathit{B}}_j\Vert -\frac{{\Vert {\mathit{B}}_j\Vert }^2}{2\gamma }\hfill & \text{if}\Vert {\mathit{B}}_j\Vert \le \gamma \lambda ,\hfill \\ \frac{1}{2}\gamma {\lambda }^2\hfill & \text{if}\Vert {\mathit{B}}_j\Vert >\gamma \lambda \hfill \end{array}\phantom{\}}$$

where λ and γ are tuning parameters. When ∥B_j∥ is small, the MCP penalty behaves exactly as lasso, but as ∥B_j∥ increases the amount of penalization is reduced until there is no penalization at all, thereby avoiding bias in the estimate of large coefficients.

In terms of tuning parameter selection, γ is set to be 3 as recommended in Zhang (2010) and λ is chosen by cross-validation. Another parameter to be determined is K, the number of basis functions used in the expansion of the coefficient functions. In the following implementations of our method, a cubic B-spline basis with 10 basis functions was used. However, since we do not explicitly penalize the roughness of the estimated coefficient functions, the exact choice of K will vary from application to application and should be chosen with care.

2.2 Estimation for models with correlated errors

The estimation framework discussed in Section 2.1 assumes that errors are independent and identically distributed over the entire domain, and is similar to the framework of Wang et al. (2007). In most cases, however, within-function errors are correlated. Let Σ denote the D × D covariance matrix for discretely observed data. For estimation of the Model (4) with correlated errors, we use techniques from generalized least squares. If Σ is known, one can “pre-white” both sides of (4) with the lower triangular matrix L obtained by Cholesky decomposition of Σ, i.e., Σ = LL^T, to construct a new model

$${\mathit{Y}}^{\ast }=XB{\Phi }^{\ast T}+{\mathit{E}}^{\ast }$$ (7)

where Y* = Y(L⁻¹)^T, Φ* = L⁻¹Φ and the error E* = E(L⁻¹)^T is independent. Similarly, parameters in model (7) can be estimated by minimizing

$$\frac{1}{2}{\Vert \mathrm{vec}\left({\mathit{Y}}^{\ast T}\right)-(\mathit{X}\otimes {\Phi }^{\ast })\mathrm{vec}\left({\mathit{B}}^T\right)\Vert }^2+nD\sum _{j=1}^p{p}_{\lambda ,\gamma }\left(\Vert {\mathit{B}}_j\Vert \right).$$ (8)

For a given Σ, the minimizer of (8) can be obtained using existing software by pre-whitening as described; our implementation is publicly available and uses the grpreg function in the grpreg package (Breheny and Huang, 2013).

The covariance matrix Σ is unknown in practice and it is necessary to obtain an estimate $\widehat{\Sigma }$ of Σ and to use this estimate to pre-whiten data. To obtain this estimate, we first fit Model (5) using ordinary least squares under the assumption of independence; this provides an unbiased estimate B̂ of the coefficient matrix B. From this model fit, we obtain the estimated residual matrix Ê = Y – XB̂Φ^T. Using Ê, we consider two approaches for estimating Σ. The first, which we refer as the raw estimate, is constructed using a method-of-moments approach based on the residual matrix. The second approach uses functional principal component analysis (Yao et al., 2005). Here, the off-diagonal elements of the raw covariance are smoothed and an eigen-decomposition of the resulting matrix is obtained. Our estimate is

$$\widehat{\Sigma }=\sum _{l=1}^L{\widehat{\nu }}_l{\widehat{\psi }}_l{\widehat{\psi }}_l^T+{\widehat{\sigma }}^2\mathit{I}$$ (9)

where ${\widehat{\psi }}_1,\cdots ,{\widehat{\psi }}_L$ are the estimated eigenfunctions over the grid {t₁, ··· , t_D}, ${\widehat{\nu }}_l,l=1,2,\cdots ,L$ are the corresponding eigenvalues, ${\widehat{\sigma }}^2$ is the estimated measurement error variance and I is the identity matrix. The truncation level L is determined by the cumulative proportion of variability explained by eigenfunctions. This approach separates Σ into a smooth covariance over the observed grid and an additional uncorrelated measurement error process. Although we focus on these methods for estimating Σ, others that provide consistent estimators can be substituted.

2.3 Oracle properties of generalized group MCP estimator

We now discuss the theoretical properties of the method described in Section 2.2. Without loss of generality, we assume ${\beta }_0\left(t\right)=0,\forall t\in \mathcal{T}$. Hence, $\mathit{X}={({\mathit{x}}_1^T,\cdots ,{\mathit{x}}_n^T)}^T$ is an n × p matrix with the ith row being ${\mathit{x}}_i^T=(x_{i1},\cdots ,x_{ip})$. We also assume the true coefficient functions β_j(t) are in the space spanned by the set of basis functions Φ. Additionally, we assume the first s groups of coefficients, ${\mathit{B}}_{+}={({\mathit{B}}_1^T,\cdots ,{\mathit{B}}_s^T)}^T$, are nonzero and the remaining p – s groups of coefficients, ${\mathit{B}}_0={({\mathit{B}}_{s+1}^T,\cdots ,{\mathit{B}}_p^T)}^T$, are zero. Let x_i+ denote the vector associated with B₊ and x_i0 denote the one associated with B₀. Therefore, we have $\mathit{B}={({\mathit{B}}_{+}^T,{\mathit{B}}_0^T)}^T$ and ${\mathit{x}}_i^T=({\mathit{x}}_{i+}^T,{\mathit{x}}_{i0}^T)$, where ${\mathit{x}}_{i+}^T=(x_{i1},\cdots ,x_{is})$ and ${\mathit{x}}_{i0}^T=(x_{i(s+1)},\cdots ,x_{ip})$. We further assume that the tuning parameter γ of the penalty is fixed. The additional conditions required for the theorems are

1. $\underset{n\to \infty }{\lim }\frac{1}{nD}{\sum }_{i=1}^n{({\mathit{x}}_i^T\otimes \Phi )}^T({\mathit{x}}_i^T\otimes \Phi )$ is a positive definite matrix;

2. λ_n → 0 and $\sqrt{n}{\lambda }_n\to \infty$ as n → ∞;

3. there exists an estimate $\widehat{\Sigma }$ to Σ for which each elements is $\sqrt{n}$-consistent;

4. Σ is non-singular.

Then we have the following results:

Theorem 1 (Estimation consistency)

Under assumptions 1-4, there exists a local minimizer $\widehat{\mathit{B}}$ of

$$Q\left(\mathit{B}\right)=\frac{1}{2}{\left[\mathrm{vec}\left({\mathit{Y}}^T\right)-(\mathit{X}\otimes \Phi )\mathrm{vec}\left({\mathit{B}}^T\right)\right]}^T{({\mathit{I}}_n\otimes \widehat{\Sigma })}^{-1}\left[\mathrm{vec}\left({\mathit{Y}}^T\right)-(\mathit{X}\otimes \Phi )\mathrm{vec}\left({\mathit{B}}^T\right)\right]+nD\sum _{j=1}^p{p}_{{\lambda }_n,\gamma }\left(\Vert {\mathit{B}}_j\Vert \right)$$

such that $\Vert \mathrm{vec}\left({\widehat{\mathit{B}}}^T\right)-\mathrm{vec}\left({\mathit{B}}^T\right)\Vert =O_p\left(n^{-1∕2}\right)$.

Theorem 2 (Oracle property)

Under assumptions 1-4, the $\sqrt{n}$-consistent local minimizer $\widehat{\mathit{B}}={({\widehat{\mathit{B}}}_{+}^T,{\widehat{\mathit{B}}}_0^T)}^T$ satisfies

• (1)Sparsity: B̂₀ = 0, with probability tending to 1;
• (2)
  Asymptotic normality:

  $$\sqrt{n}(\mathrm{vec}\left({\widehat{\mathit{B}}}_{+}^T\right)-\mathrm{vec}\left({\mathit{B}}_{+}^T\right))\stackrel{\mathcal{D}}{\to }\mathcal{N}\left(0,{\left(\underset{n\to \infty }{\lim }\frac{1}{n}\sum _{i=1}^n{({\mathit{x}}_{i+}^T\otimes \Phi )}^T{\Sigma }^{-1}({\mathit{x}}_{i+}^T\otimes \Phi )\right)}^{-1}\right).$$

The proof of these theorems is provided in Appendix A.

2.4 Iterative algorithm for models with correlated errors

The method described in Section 2.2 uses ordinary least squares to estimate basis coefficients and obtains an estimate $\widehat{\Sigma }$ of the covariance Σ; this estimate is then used to pre-whiten the data prior to the application of variable selection techniques. However, re-estimating the covariance after variable selection may give a refined estimate which can, in turn, be used to pre-whiten the data. This intuition suggests an iterative algorithm:

1. Fit a model using ordinary least squares to obtain an initial estimate B̂⁽⁰⁾;

2. Compute residuals and obtain an estimate ${\widehat{\Sigma }}^{\left(0\right)}$ of Σ;

3. For k > 0, iterate the following steps until convergence:

   • (a)
     Pre-whiten using the covariance ${\widehat{\Sigma }}^{(k-1)}$;
   • (b)
     Minimize (8) to obtain B̂^(k);
   • (c)
     Use B̂^(k) to construct fitted values and residual curves, and use these to construct ${\widehat{\Sigma }}^{\left(k\right)}$

.

Various criteria of convergence can be used to monitor convergence of this iterative algorithm; one possible criterion is ${\Vert {\widehat{\mathit{B}}}^{(k+1)}-{\widehat{\mathit{B}}}^{\left(k\right)}\Vert }^2<\delta$, which we use in our implementations. This iterative method will be compared to the one-step approach of Section 2.2 in simulations.

3 Simulation

We conducted simulation studies to examine the properties of the proposed approach. Specifically, we constructed 500 training samples, each consisting of 100 random curves, and 1 test sample containing 1000 random curves. All curves are generated from the model

$$y_i\left(t\right)=\sum _{j=1}^{20}{x}_{ij}{\beta }_j\left(t\right)+{ϵ}_i\left(t\right)$$

where $x_{ij}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathrm{N}(0,10)$, β₁(t), β₂(t), β₃(t) are non-zero functions, and the remaining coefficient functions are zero. All functions are observed on a equally spaced grid of length 25. Errors ε_i(t_d) are generated from a multivariate Gaussian distribution with mean zero and covariance Σ = G + I where G is the error covariance and I is the identity matrix. the non-zero coefficient functions β₁(t), β₂(t) and β₃(t) are derived from the motivating data in the following way. Focusing on y position curves for reaching motions made to the target at 0 degrees, we estimated motions made by healthy controls, moderately affected stroke patients, and severely affected patients (stroke severity was defined by thresholding the Fugl-Meyer score). These estimated motions were the non-zero coefficients, and are shown in the middle panel of Figure 2. The error covariance G was constructed using an FPCA decomposition of residual curves after subtracting the group-specific means.

Figure 2.

Estimates of zero functions (left) and non-zero functions (middle) obtained using the iterative approach with FPCA-based covariance matrix estimate using PVE=0.99 across all simulated datasets. The true functions are overlaid (bold curves). The right panel shows the both MSE (solid) and squared bias (dashed) as functions of time for all the coefficient functions.

Four implementations of our proposed method are considered: one-step approaches as described in Section 2.2 using raw and FPCA-based covariance matrix estimates, and iterative approaches as described in Section 2.4 using raw and FPCA-based covariance matrix estimates. For the FPCA-based covariance matrix estimate, we used two different values, 0.5 and 0.99, as the cumulative proportion of variance explained (PVE) threshold to determine L. For comparison, we include an approach that pre-whitens using true covariance matrix, as well as ordinary least squares, a variational Bayes method that includes a smoothness penalty (Goldsmith and Kitago, 2015), the FS-LASSO method that uses group LASSO but does not account for residual correlation or biases due to the LASSO penalty, and a group MCP method that assumes uncorrelated error curves, analogously to Wang et al. (2007).

Table 1 reports the true positive (TP) and true negative (TN) rates of the estimates of both zero and non-zero coefficient functions. We define functions estimated to be non-zero as “positive” while functions estimated to be zero as “negative”. Our iterative approach using a FPCA-based covariance matrix estimate with PVE=0.99 outperforms most competing approaches in terms of correctly identifying the zero functions; its performance is comparable to the approach that uses the true covariance matrix. The approaches using PVE=0.5 perform less well because the estimate of the covariance matrix omits important structure. Our proposed methods substantially outperform the method that assumes uncorrelated errors in accurately identifying zero functions. FS-LASSO has the highest true negative rate but the lowest true positive rate for β₁(t), potentially indicating a tendency to over-shrink coefficients to zero. All methods are able to identify β₂(t) and β₃(t) as non-zero.

Table 1.

true positive (TP) and true negative (TN) rates of estimated coefficient functions, where FN is false negative and FP is false positive. They are estimated across all the training samples.

	$\frac{TN}{FP+TN}$	$\frac{TP}{TP+FN}\left({\beta }_1\right)$	$\frac{TP}{TP+FN}\left({\beta }_2\right)$	$\frac{TP}{TP+FN}\left({\beta }_3\right)$
FS-LASSO	0.953	0.850	1.000	1.000
MCP Assuming Independent Errors	0.567	1.000	1.000	1.000
One-step with Raw Matrix	0.370	1.000	1.000	1.000
Iterative with Raw Matrix	0.813	0.962	1.000	1.000
One-step with FPCA-based Matrix (PVE = 0.5)	0.755	0.996	1.000	1.000
Iterative with FPCA-based Matrix (PVE = 0.5)	0.779	0.996	1.000	1.000
One-step with FPCA-based Matrix (PVE = 0.99)	0.863	0.986	1.000	1.000
Iterative with FPCA-based Matrix (PVE = 0.99)	0.915	0.956	1.000	1.000
Pre-whiten with True Σ	0.913	0.964	1.000	1.000

Estimates of zero and non-zero coefficient functions obtained using the iterative algorithm with FPCA-based covariance matrix estimate using PVE=0.99 are shown in the left and middle panels of Figure 2, respectively. Because their coefficients are relatively large, the estimate of β₂(·) and β₃(·) are approximately unbiased owing to the structure of the penalty. For β₁(·), coefficients are shrunk toward and sometimes set equal to zero. We show the mean squared error $\mathrm{MSE}=E{({\beta }_j\left(t\right)-{\widehat{\beta }}_j\left(t\right))}^2$ and squared bias $E{({\beta }_j\left(t\right)-{\overline{\widehat{\beta }}}_j\left(t\right))}^2$ as functions of t in the right panel of Figure 2, where ${\overline{\widehat{\beta }}}_j\left(t\right)$ is the average curve across all the simulation datasets. For β₁(·), both the MSE and squared bias curves present a sinusoidal shape, which is driven by the sinusoidal shape of the coefficient function itself and by the shrinkage to zero. There is an increasing trend in general as t increases for the MSE of β₂(·) and β₃(·), which is mostly caused by the increased variability of curves at the end of the distribution as the biases are relatively small. This plot further emphasizes the lack of bias for large coefficients stemming from the use of the group MCP penalty, especially in the case of β₃(·).

The left and middle columns of Figure 3 display the root mean integrated squared error (RMISE), $\sqrt{{\int }_0^1{({\beta }_j\left(t\right)-{\widehat{\beta }}_j\left(t\right))}^{2}dt}$ for zero and non-zero functions, respectively; in the top row, the FPCA-based covariance estimate is based on PVE=0.99 and in the bottom row based on PVE=0.5. The iterative approach with FPCA-based covariance matrix estimate compares favorably to other approaches, reinforcing the results from Table 1. Indeed, the RMISE of our iterative method is comparable to pre-whitening using the true covariance for both zero and non-zero functions. Although FS-LASSO is comparable for zero functions, it has substantially higher RMISE for non-zero functions. Prediction errors on the test sample are shown in the right panel of Figure 3. These errors reflect a combination of RMISEs for zero and non-zero functions, and display similar patterns: our proposed methods, in particular when using the FPCA-based estimate of the covariance, have excellent numerical performance. Although there is a slight decline in performance when PVE=0.5, the proposed method still outperforms OLS, FS-LASSO and the method that assumes uncorrelated errors.

Figure 3.

The top row shows the comparison among the algorithms when PVE = 0.99 while the second row shows the comparison when PVE = 0.5. The three columns show RMISE for zero functions (left) and non-zero functions (middle); and prediction error (right).

Additional simulations that generate uncorrelated errors are presented in detail in Appendix B. In this case, there is no noticeable disadvantage to using our proposed approach, which outperforms competing methods in prediction error.

4 Application

We now apply our iterative algorithm using the FPCA-based covariance matrix estimate described in Section 2.4 to our motivating dataset. The X and Y position functions are the outcomes of interest, and potential predictors include the Fugl-Meyer score, whether the hand was dominant or non-dominant, whether the hand was contralesional or ipsilesional, target direction (as a categorical predictor) and the interactions between these variables. In general, not a lot of correlation is observed between the residuals from models for X and Y position functions (Goldsmith and Kitago, 2015). It is not a great reason to consider a bivariate analysis. We thus analyze the X and Y position functions separately, using the same models and steps.

First, we perform a cross validation analysis to evaluate the algorithm in terms of prediction error on the motivating data. Training and test sets are generated in the following way. For each subject and each hand, we randomly select one motion to each of the eight target directions. These motions are partitioned so that four are in the training set and four are in the test set. Previous work on this dataset (Goldsmith and Kitago, 2015) indicates little or no correlation between motions to different targets made by the same subject, and so our training and test sets are approximately independent even though they contain data from the same subjects. This procedure results in 452 curves in the training set and 452 curves in the test set; an example is shown in Figure 4.

Figure 4.

One training sample (left) and one test sample (middle) generated from the planar reaching data. Highlighted curves are from one subject and show how each subject contributes to the training and test sets. Violin plots (right) of cross validation errors using the variational Bayes approach and iterative algorithm.

A function-on-scalar regression model is then constructed on the training sample, and prediction errors are obtained for the test sample. Four predictors of interest, the target direction (a categorical variable with eight levels), Fugl-Meyel score (a continuous variable), hand used (dominant/non-dominant) and arm affectedness (contralesional/ipsilesional), are considered in these models. In addition to main effects, all the possible interactions are included to maximize flexibility and scientific interpretation. Thus, the model has 64 coefficient functions to estimate. Rather than the typical design that assigns a reference level for each categorical predictor, a constraint is imposed to the construction of design matrix so that target-specific interpretations are available. This design matrix is equivalent to building the following model for each target:

$$\begin{array}{cc}\hfill y\left(t\right)=& {\beta }_0\left(t\right)+{\beta }_1\left(t\right)\ast \mathrm{Ips}.\mathrm{Non}.+{\beta }_2\left(t\right)\ast \mathrm{Con}.\mathrm{Dom}.+{\beta }_3\left(t\right)\ast \mathrm{Con}.\mathrm{Non}.+{\beta }_4\left(t\right)\ast \text{Fugl-Meyer}\hfill \\ \hfill & +{\beta }_5\left(t\right)\ast \text{Fugl-Meyer}\ast \mathrm{Ips}.\mathrm{Non}.+{\beta }_6\left(t\right)\ast \text{Fugl-Meyer}\ast \mathrm{Con}.\mathrm{Dom}.\hfill \\ \hfill & +{\beta }_7\left(t\right)\ast \text{Fugl-Meyer}\ast \mathrm{Con}.\mathrm{Non}.+ϵ\left(t\right)\hfill \end{array}$$ (10)

where we use the ipsilesional (unaffected) dominant hand of a healthy control as the reference β₀(t). Coefficients β₁(t), β₂(t) and β₃(t) compare ipsilesional nondominant, contralesional dominant, and contralesional nondominant to the reference, respectively. The effect of increasing motor impairment in the ipsilesional dominant arm is estimated by β₅(t), while differences in the effect of increasing motor impairment comparing other groups to baseline are given by β₆(t), β₇(t) and β₈(t).

The complete procedure described above, consisting of generating training and test sets, fitting the full model to the training set, and producing predictions for the test set, is repeated 100 times. We fit the model using 5, 10, 15 and 20 basis functions, and found that K = 15 gave the smallest cross-validated prediction errors. The right panel of Figure 4 presents the prediction errors obtained using our iterative algorithm with FPCA-based covariance matrix estimate; we compare to the variational Bayes approach (without variable selection but with a standard second-derivative penalty). Our iterative algorithm decreases mean prediction error by around 10% (X direction: 163.8 vs. 144.8; Y direction: 143.6 vs. 132.9) compared to the variational Bayes approach. In addition, the iterative algorithm seems to be more stable than the variational Bayes approach as it has fewer outliers and lower median prediction error.

We next conduct our analysis without splitting data into training and test sets. The function-on-scalar regression model is estimated using one motion for each subject and hand to each target with motions drawn randomly for each target and hand. We repeat this analysis 100 times, and Table 2 presents the proportion of times selected by the algorithm for each of the 64 coefficient functions. Each row of Table 2 corresponds to coefficients β₀(t), β₁(t), ··· , β₇(t) in Model (10) for a specific target. For instance, the value 0.24 in the third entry of the first row indicates that, in 24 of 100 datasets, there was an estimated difference between contralesional and ipsilesional dominant hands when reaching to the target at 0°.

Table 2.

Proportions of 64 coefficient functions being selected, obtained from models with X trajectories (top) and Y trajectories (bottom).

Target Direction	Fugl-Meyer = 66				Δ Fugl-Meyer = −1
	Ips.Dom.	Ips.Non.	Con.Dom.	Con.Non.	Ips.Dom.	Ips.Non.	Con.Dom.	Con.Non.
0°	1.00	0.21	0.24	0.35	0.41	0.38	0.58	0.34
45°	1.00	0.20	0.03	0.15	0.16	0.08	0.46	0.37
90°	0.31	0.65	0.31	0.33	0.30	0.17	0.23	0.60
135°	1.00	0.22	0.16	0.24	0.11	0.48	0.64	0.57
180°	1.00	0.35	0.13	0.40	0.18	0.38	0.48	0.35
225°	1.00	0.18	0.16	0.33	0.08	0.22	0.45	0.36
270°	0.83	0.37	0.19	0.34	0.14	0.35	0.52	0.33
315°	1.00	0.41	0.20	0.33	0.28	0.35	0.57	0.67

Target Direction	Fugl-Meyer = 66				Δ Fugl-Meyer = −1
	Ips.Dom.	Ips.Non.	Con.Dom.	Con.Non.	Ips.Dom.	Ips.Non.	Con.Dom.	Con.Non.
0°	0.34	0.10	0.14	0.11	0.13	0.18	0.66	0.20
45°	1.00	0.02	0.02	0.05	0.01	0.20	0.36	0.12
90°	1.00	0.07	0.16	0.15	0.08	0.21	0.17	0.43
135°	1.00	0.05	0.22	0.29	0.05	0.40	0.79	0.43
180°	0.40	0.13	0.11	0.31	0.15	0.25	0.79	0.27
225°	1.00	0.05	0.03	0.11	0.01	0.12	0.33	0.21
270°	1.00	0.03	0.15	0.14	0.06	0.24	0.13	0.22
315°	1.00	0.03	0.22	0.18	0.06	0.36	0.52	0.54

Large numbers in the table suggest consistent non-zero effects or differences in effect across datasets. Targets at 90° and 270° may have zero effects in the X trajectories, since for those targets the X position is roughly constant over time. The same is true for Targets at 0° and 180° for the Y trajectories. The results in Table 2 indicate relatively few differences between ipsilesional and contralesional dominant arms for very mild strokes (Fugl-Meyer = 66), and some differences between the non-dominant arms and the ipsilesional dominant arm. An effect of increasing stroke severity is relatively rarely found for the ipsilensional arms but, as expected, is much more frequently found for the contralesional arms. The conclusions are further reinforced by Figure 5, where the predicted motions of subjects with different combinations of Fugl-Meyel Score(66/26), hand used (dominant/non-dominant) and arm affectedness (contralesional/ipsilesional) are presented.

Figure 5.

Predicted reaching motions for eight subjects with different combinations of Fugl-Meyel Score(66/26), hand used (dominant/non-dominant) and arm affectedness (contralesional/ipsilesional). Motions to different targets are distinguished by colors.

5 Discussion

We proposed a model fitting framework that performs variable selection in the context of function-on-scalar regression allowing within-function correlation. This work was motivated by two-dimensional planar reaching data gathered to understand the mechanisms of motor deficit following stroke. We developed an iterative algorithm that alternatively estimates the coefficient functions and covariance structure. Our method relies on a reasonable estimate of the covariance structure, and in our simulations and application we found that an estimation procedure based on FPCA works well. Results from the simulation studies demonstrate the effectiveness of our proposed method in identifying the true zero functions. Indeed, our proposed method has performance comparable to performing variable selection using the true covariance. The application to the motivating data indicates our proposed iterative algorithm makes a significant improvement in terms of decreasing prediction errors and identifying true zero functions.

Future extension of our methodology may take several directions. Quadratic roughness penalties are often applied to enforce smoothness of the coefficient functions in spline-based estimation frameworks. It would be worthwhile to incorporate an explicit roughness penalty in addition to the variable selection penalty to reduce sensitivity to the size of the basis expansion. Motivated by our application (in which repeated motions are made to each target by each subject), the development of methods that account for subject- and target-specific random effects is necessary.

Appendices to: Variable Selection in Function-on-Scalar Regression

Yakuan Chen, Jeff Goldsmith and Todd Ogden

A Proof of the theorems

These proofs follow the same general strategy of Zeng and Xie (2014) and Peng and Lu (2012).

For convenience of notation, we denote vec(Y^T) as w, X $\otimes$ Φ as Z, vec(B^T) as θ and vec(E^T) as ε. Then Model (5) can be rewritten as

$$\mathit{w}=\mathit{Z}\theta +ϵ$$

where $\mathit{w}={({\mathit{w}}_1^T,\cdots ,{\mathit{w}}_n^T)}^T$ is a vector of length nD with w_i = (w_i1, ··· , w_iD)^T, i = 1, ··· , n; $\mathit{Z}={({\mathit{Z}}_1^T,\cdots ,{\mathit{Z}}_n^T)}^T$ is a nD × pK matrix with Z_i = (Z_i1, ··· , Z_ip), i = 1, ··· , n, where Z_ij, j = 1 ··· , p is a D × K matrix; and $\theta ={({\theta }_1^T,\cdots ,{\theta }_p^T)}^T$ is a vector of length K_p with θ_j = (θ_j1, ··· , θ_jK)^T, j = 1, ··· , p. Without loss of generality, we assume the first s groups of coefficients, ${\theta }_{+}={({\theta }_1^T,\cdots ,{\theta }_s^T)}^T$, are nonzero and the rest p – s groups of coefficients, ${\theta }_0={({\theta }_{s+1}^T,\cdots ,{\theta }_p^T)}^T$, are zeros. Let Z_i+ denote the design matrix associated with θ₊ and Z_i0 denote the one associated with θ₀ for i = 1, ··· , n. Therefore, we have $\theta ={({\theta }_{+}^T,{\theta }_0^T)}^T$ and Z_i = (Z_i+|Z_i0), where Z_i+ = (Z_i1, ··· , Z_is) and Z_i0 = (Z_i(s+1), ··· , Z_ip). Also, $ϵ={({ϵ}_1^T,\cdots ,{ϵ}_n^T)}^T\sim \mathcal{N}(0,\mathit{V})$ where V is a nD × nD block diagonal matrix with diagonal elements, the D × D matrix Σ, i.e. $\mathit{V}={\mathit{I}}_n\otimes \Sigma$ and ${ϵ}_i\sim \mathcal{N}(0,\Sigma )$. Let ${\theta }^{\ast }={({\theta }_{+}^{\ast T},{\theta }_0^{\ast T})}^T$ denote the true values of the parameters. Everything is regarded as fixed unless specified otherwise. The conditions required for the proofs are

1. $\underset{n\to \infty }{\lim }\frac{1}{nD}{\sum }_{i=1}^n{\mathit{Z}}_i^T{\mathit{Z}}_i$ is a positive definite matrix;

2. λ_n → 0 and $\sqrt{n}{\lambda }_n\to \infty$ as n → ∞;

3. there exists an estimate $\widehat{\Sigma }$ of Σ for which each element is $\sqrt{n}$-consistent;

4. Σ is non-singular.

Proof of Theorem 1

The penalized least square objective function for estimating θ is

$$\begin{array}{cc}\hfill Q\left(\theta \right)=& \frac{1}{2}{(\mathit{w}-\mathit{Z}\theta )}^T{\widehat{\mathit{V}}}^{-1}(\mathit{w}-\mathit{Z}\theta )+nD\sum _{j=1}^p{p}_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j\Vert \right)\hfill \\ \hfill =& \frac{1}{2}{(\mathit{w}-\mathit{Z}\theta )}^T{({\mathit{I}}_n\otimes \widehat{\Sigma })}^{-1}(\mathit{w}-\mathit{Z}\theta )+nD\sum _{j=1}^p{p}_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j\Vert \right)\hfill \\ \hfill =& \frac{1}{2}\sum _{i=1}^n{({\mathit{w}}_i-{\mathit{Z}}_i\theta )}^T{\widehat{\Sigma }}^{-1}({\mathit{w}}_i-{\mathit{Z}}_i\theta )+nD\sum _{j=1}^p{p}_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j\Vert \right)\hfill \\ \hfill =& L\left(\theta \right)+nD\sum _{j=1}^p{p}_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j\Vert \right).\hfill \end{array}$$

Let's consider a ball $B=\{{\theta }^{\ast }+n^{-1∕2}\mathit{u}:\Vert \mathit{u}\Vert \le C\}$ where C is a constant. Since B is a compact set and Q(θ) is a continuous function on B, there exists a minimum of Q(θ) on B. If Q(θ) > Q(θ*) for every θ on the boundary of B, then there exists a local minimizer inside the ball B.

Therefore it suffices to show that for any given ε > 0, there exists a constant C such that

$$P\left\{\underset{\Vert u\Vert =C}{\inf }Q({\theta }^{\ast }+n^{-1∕2}\mathit{u})>Q\left({\theta }^{\ast }\right)\right\}>1-ϵ.$$ (A.1)

This implies with probability at least 1 – ε that there exists a local minimizer in the ball B. Equivalently, for any given ε > 0, there exists a constant C such that $P\{n^{-1∕2}\Vert \widehat{\theta }-{\theta }^{\ast }\Vert <C\}\ge 1-ϵ$, where $\widehat{\theta }$ is the local minimizer that satisfies $\Vert \widehat{\theta }-{\theta }^{\ast }\Vert =O_p\left(n^{-1∕2}\right)$.

Since p_λn,γ(0) = 0, we have

$$\begin{array}{cc}\hfill Q({\theta }^{\ast }+n^{-1∕2}\mathit{u})-Q\left({\theta }^{\ast }\right)& \ge L({\theta }^{\ast }+n^{-1∕2}\mathit{u})-L\left({\theta }^{\ast }\right)+nD\sum _{j=1}^s{p}_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j^{\ast }+n^{-1∕2}{\mathit{u}}_j\Vert \right)-nD\sum _{j=1}^s{p}_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j^{\ast }\Vert \right)\hfill \\ \hfill & =\frac{1}{2n}\sum _{i=1}^n{\mathit{u}}^T{\mathit{Z}}_i^T{\widehat{\Sigma }}^{-1}{\mathit{Z}}_i\mathit{u}-\frac{1}{\sqrt{n}}\sum _{i=1}^n{\mathit{u}}^T{\mathit{Z}}_i^T{\widehat{\Sigma }}^{-1}{ϵ}_i+nD\sum _{j=1}^s\left[p_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j^{\ast }+n^{-1∕2}{\mathit{u}}_j\Vert \right)-p_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j^{\ast }\Vert \right)\right]\hfill \\ \hfill & \triangleq I_1+I_2.\hfill \end{array}$$

First,

$$\begin{array}{cc}\hfill I_1=& \frac{1}{2n}\sum _{i=1}^n{\mathit{u}}^T{\mathit{Z}}_i^T{\widehat{\Sigma }}^{-1}{\mathit{Z}}_i\mathit{u}-\frac{1}{\sqrt{n}}\sum _{i=1}^n{\mathit{u}}^T{\mathit{Z}}_i^T{\widehat{\Sigma }}^{-1}{ϵ}_i\hfill \\ \hfill =& \frac{D}{2}{\mathit{u}}^T\left[\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_i^T{\widehat{\Sigma }}^{-1}{\mathit{Z}}_i\right]\mathit{u}-\sqrt{D}{\mathit{u}}^T\left[\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_i^T{\widehat{\Sigma }}^{-1}{ϵ}_i\right]\hfill \\ \hfill =& \frac{D}{2}{\mathit{u}}^T\left[\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_i^T{\Sigma }^{-1}{\mathit{Z}}_i\right]\mathit{u}+\frac{D}{2}{\mathit{u}}^T\left[\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_i^T({\widehat{\Sigma }}^{-1}-{\Sigma }^{-1}){\mathit{Z}}_i\right]\mathit{u}-\sqrt{D}{\mathit{u}}^T\left[\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_i^T{\Sigma }^{-1}{ϵ}_i\right]-\sqrt{D}{\mathit{u}}^T\left[\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_i^T({\widehat{\Sigma }}^{-1}-{\Sigma }^{-1}){ϵ}_i\right]\hfill \end{array}$$

Let ν₁ ≥ ··· ≥ ν_D > 0 and Q_Σ denote, respectively, the eigenvalues and the matrix formed by aligning the eigenvectors of Σ in columns; ${\zeta }_1\ge \cdots \ge {\zeta }_{pK}>0$ and Q_Z denote, respectively, the eigenvalues and the matrix formed by aligning the eigenvectors of of $\underset{n\to \infty }{\lim }\frac{1}{nD}{\sum }_{i=1}^n{\mathit{Z}}_i^T{\mathit{Z}}_i$ in columns. Under conditions (1), (3) and (4) and by the definition of eigen decomposition,

$$\begin{array}{cc}\hfill \underset{n\to \infty }{\lim }\frac{D}{2}{\mathit{u}}^T\left[\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_i^T{\Sigma }^{-1}{\mathit{Z}}_i\right]\mathit{u}=& \underset{n\to \infty }{\lim }\frac{D}{2}{\mathit{u}}^T\left[\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_i^T{\mathit{Q}}_{\Sigma }\left(\begin{array}{ccc}\hfill \frac{1}{{\nu }_D}\hfill & \hfill \hfill & \hfill 0\hfill \\ \hfill \hfill & \hfill \ddots \hfill & \hfill \hfill \\ \hfill 0\hfill & \hfill \hfill & \hfill \frac{1}{{\nu }_1}\hfill \end{array}\right){\mathit{Q}}_{\Sigma }^{-1}{\mathit{Z}}_i\right]\mathit{u}\hfill \\ \hfill & \ge \underset{n\to \infty }{\lim }\frac{D}{2}{\mathit{u}}^T\left[\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_i^T{\mathit{Q}}_{\Sigma }\left(\begin{array}{ccc}\hfill \frac{1}{{\nu }_1}\hfill & \hfill \hfill & \hfill 0\hfill \\ \hfill \hfill & \hfill \ddots \hfill & \hfill \hfill \\ \hfill 0\hfill & \hfill \hfill & \hfill \frac{1}{{\nu }_1}\hfill \end{array}\right){\mathit{Q}}_{\Sigma }^{-1}{\mathit{Z}}_i\right]\mathit{u}\hfill \\ \hfill =& \frac{D}{2{\nu }_1}{\mathit{u}}^T\left[\underset{n\to \infty }{\lim }\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_i^T{\mathit{Z}}_i\right]\mathit{u}\hfill \\ \hfill =& \frac{D}{2{\nu }_1}{\mathit{u}}^T{\mathit{Q}}_Z\left(\begin{array}{ccc}\hfill {\zeta }_1\hfill & \hfill \hfill & \hfill 0\hfill \\ \hfill \hfill & \hfill \ddots \hfill & \hfill \hfill \\ \hfill 0\hfill & \hfill \hfill & \hfill {\zeta }_{pK}\hfill \end{array}\right){\mathit{Q}}_Z^{-1}\mathit{u}\hfill \\ \hfill \ge & \frac{D}{2{\nu }_1}{\mathit{u}}^T{\mathit{Q}}_Z\left(\begin{array}{ccc}\hfill {\zeta }_{pK}\hfill & \hfill \hfill & \hfill 0\hfill \\ \hfill \hfill & \hfill \ddots \hfill & \hfill \hfill \\ \hfill 0\hfill & \hfill \hfill & \hfill {\zeta }_{pK}\hfill \end{array}\right){\mathit{Q}}_Z^{-1}\mathit{u}\hfill \\ \hfill =& \frac{D{\zeta }_{pK}}{2{\nu }_1}{\mathit{u}}^T\mathit{u}.\hfill \end{array}$$

Therefore, $\frac{D}{2}{\mathit{u}}^T\left[\frac{1}{nD}{\sum }_{i=1}^n{\mathit{Z}}_i^T{\Sigma }^{-1}{\mathit{Z}}_i\right]\mathit{u}\ge O_p\left(1\right){\Vert \mathit{u}\Vert }^2$.

Because $\widehat{\Sigma }-\Sigma =O_p\left(n^{-1∕2}\right)$,

$$\frac{D}{2}{\mathit{u}}^T\left[\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_i^T({\widehat{\Sigma }}^{-1}-{\Sigma }^{-1}){\mathit{Z}}_i\right]\mathit{u}=O_p\left(n^{-1∕2}\right)\frac{D}{2}{\mathit{u}}^T\left[\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_i^T{\mathit{Z}}_i\right]\mathit{u}=O_p\left(n^{-1∕2}\right){\Vert \mathit{u}\Vert }^2.$$

Additionally, since ${ϵ}_i\sim \mathcal{N}(0,\Sigma )$,

$$\sqrt{D}{\mathit{u}}^T\left[\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_i^T{\Sigma }^{-1}{ϵ}_i\right]\stackrel{\mathcal{D}}{\to }\sqrt{D}\cdot \mathcal{N}\left(0,u^T\left[\underset{n\to \infty }{\lim }\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_i^T{\Sigma }^{-1}{\mathit{Z}}_i\right]\mathit{u}\right),$$

which implies $\sqrt{D}{\mathit{u}}^T\left[\frac{1}{\sqrt{nD}}{\sum }_{i=1}^n{\mathit{Z}}_i^T{\Sigma }^{-1}{ϵ}_i\right]=O_p\left(1\right)\Vert \mathit{u}\Vert$.

Because $\widehat{\Sigma }-\Sigma =O_p\left(n^{-1∕2}\right)$,

$$\sqrt{D}{\mathit{u}}^T\left[\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_i^T({\widehat{\Sigma }}^{-1}-{\Sigma }^{-1}){ϵ}_i\right]=O_p\left(n^{-1∕2}\right)\sqrt{D}{\mathit{u}}^T\left[\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_i^T{ϵ}_i\right]=O_p\left(n^{-1∕2}\right)\Vert \mathit{u}\Vert .$$

Hence,

$$I_1\ge O_p\left(1\right)({\Vert \mathit{u}\Vert }^2-\Vert \mathit{u}\Vert )+O_p\left(n^{-1∕2}\right)({\Vert \mathit{u}\Vert }^2-\Vert \mathit{u}\Vert ).$$

Then I₁ ≥ 0 in ∥u∥ = C by choosing a sufficiently large C.

Moreover,

$$\begin{array}{cc}\hfill I_2=& nD\sum _{j=1}^s\left[p_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j^{\ast }+n^{-1∕2}{\mathit{u}}_j\Vert \right)-p_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j^{\ast }\Vert \right)\right]\hfill \\ \hfill =& nD\sum _{j=1}^s\left[n^{-1∕2}\frac{\partial p_{{\lambda }_n,\gamma }{\left(\Vert {\theta }_j^{\ast }\Vert \right)}^T}{\partial {\theta }_j^{\ast }}{\mathit{u}}_j+\frac{1}{2}{n}^{-1}{\mathit{u}}_j^T\frac{{\partial }^2{p}_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j^{\ast }\Vert \right)}{\partial {\theta }_j^{\ast }\partial {\theta }_j^{\ast T}}{\mathit{u}}_j\{1+o\left(1\right)\}\right],\hfill \end{array}$$

Since λ_n → 0 as n → ∞, for sufficiently large n, $\frac{\partial p_{{\lambda }_n,\gamma }{\left(\Vert {\theta }_j^{\ast }\Vert \right)}^T}{\partial {\theta }_j^{\ast }}=0$ and $\frac{{\partial }^2{p}_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j^{\ast }\Vert \right)}{\partial {\theta }_j^{\ast }\partial {\theta }_j^{\ast T}}=0$ for ${\theta }_j^{\ast }$ that satisfies $\Vert {\theta }_j^{\ast }\Vert \ne 0$ by the definition of the MCP function.

Hence, by choosing a sufficiently large C, I₁ + I₂ ≥ 0 when n is large enough. Then we have Q(θ* + n^−1/2u) > Q(θ*) with arbitrarily large probability 1 – ε, i.e., Inequality A.1 holds. Based on the discussions above, there exists a local minimizer $\widehat{\theta }$ such that

$$\Vert \widehat{\theta }-{\theta }^{\ast }\Vert =O_p\left(n^{-1∕2}\right).$$

Proof of Theorem 2

The partial derivative of Q(θ) with respect to θ_j is

$$\begin{array}{cc}\hfill \frac{\partial Q\left(\theta \right)}{\partial {\theta }_j}& =-\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\widehat{\Sigma }}^{-1}({\mathit{w}}_i-{\mathit{Z}}_i\theta )+nD\frac{\partial p_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j\Vert \right)}{\partial {\theta }_j}\hfill \\ \hfill & \triangleq J_1+J_2\hfill \end{array}$$

First,

$$\begin{array}{cc}\hfill J_1=& -\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\widehat{\Sigma }}^{-1}({\mathit{w}}_i-{\mathit{Z}}_i{\theta }^{\ast })+\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\widehat{\Sigma }}^{-1}{\mathit{Z}}_i(\theta -{\theta }^{\ast })\hfill \\ \hfill =& -\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\Sigma }^{-1}({\mathit{w}}_i-{\mathit{Z}}_i{\theta }^{\ast })-\sum _{i=1}^n{\mathit{Z}}_{ij}^T\left({\widehat{\Sigma }}^{-1}-{\Sigma }^{-1}\right)({\mathit{w}}_i-{\mathit{Z}}_i{\theta }^{\ast })+\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\Sigma }^{-1}{\mathit{Z}}_i(\theta -{\theta }^{\ast })+\sum _{i=1}^n{\mathit{Z}}_{ij}^T\left({\widehat{\Sigma }}^{-1}-{\Sigma }^{-1}\right){\mathit{Z}}_i(\theta -{\theta }^{\ast })\hfill \end{array}$$

Since ${ϵ}_i={\mathit{w}}_i-{\mathit{Z}}_i{\theta }^{\ast }\sim \mathcal{N}(0,\Sigma )$ and $\widehat{\Sigma }-\Sigma =O_p\left(n^{-1∕2}\right)$,

$$\begin{array}{cc}\hfill \frac{1}{\sqrt{n}}\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\Sigma }^{-1}({\mathit{w}}_i-{\mathit{Z}}_i{\theta }^{\ast })& =\sqrt{D}\left[\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\Sigma }^{-1}({\mathit{w}}_i-{\mathit{Z}}_i{\theta }^{\ast })\right]\hfill \\ \hfill & \stackrel{\mathcal{D}}{\to }\sqrt{D}\cdot \mathcal{N}\left(0,\underset{n\to \infty }{\lim }\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\Sigma }^{-1}{\mathit{Z}}_{ij}\right).\hfill \end{array}$$

Hence, ${\sum }_{i=1}^n{\mathit{Z}}_{ij}^T{\Sigma }^{-1}({\mathit{w}}_i-{\mathit{Z}}_i{\theta }^{\ast })=O_p\left(\sqrt{n}\right)$.

In addition,

$$\sum _{i=1}^n{\mathit{Z}}_{ij}^T\left({\widehat{\Sigma }}^{-1}-{\Sigma }^{-1}\right)({\mathit{w}}_i-{\mathit{Z}}_i{\theta }^{\ast })=O_p\left(n^{-1∕2}\right)\sqrt{nD}\left[\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_{ij}^T({\mathit{w}}_i-{\mathit{Z}}_i{\theta }^{\ast })\right]=O_p\left(1\right).$$

For any θ_n that satisfies ∥θ_n – θ*∥ = O_p(n^−1/2), under condition (1),

$$\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\Sigma }^{-1}{\mathit{Z}}_i({\theta }_n-{\theta }^{\ast })=nD\left[\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\Sigma }^{-1}{\mathit{Z}}_i\right]({\theta }_n-{\theta }^{\ast })=O_p\left(\sqrt{n}\right),$$

where $\frac{1}{nD}{\sum }_{i=1}^n{\mathit{Z}}_{ij}^T{\Sigma }^{-1}{\mathit{Z}}_i=O_p\left(1\right)$ has been shown in the proof of Theorem 1.

Also,

$$\sum _{i=1}^n{\mathit{Z}}_{ij}^T\left({\widehat{\Sigma }}^{-1}-{\Sigma }^{-1}\right){\mathit{Z}}_i({\theta }_n-{\theta }^{\ast })=O_p\left(n^{-1∕2}\right)nD\left[\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\mathit{Z}}_i\right]({\theta }_n-{\theta }^{\ast })=O_p\left(1\right).$$

Therefore, we have $J_1=O_p\left(\sqrt{n}\right)$.

Furthermore,

$$J_2=nD{\left(\frac{\partial p_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j\Vert \right)}{\partial {\theta }_{j1}},\cdots ,\frac{\partial p_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j\Vert \right)}{\partial {\theta }_{jK}}\right)}^T=nD{\left(p_{{\lambda }_n,\gamma }^{\prime }\left(\Vert {\theta }_j\Vert \right)\frac{{\theta }_{j1}}{\Vert {\theta }_j\Vert },\cdots ,p_{{\lambda }_n,\gamma }^{\prime }\left(\Vert {\theta }_j\Vert \right)\frac{{\theta }_{jK}}{\Vert {\theta }_j\Vert }\right)}^T.$$

Since $\sqrt{n}{\lambda }_n\to \infty$ as n → ∞, there exists a C such that $C∕\sqrt{n}<\gamma {\lambda }_n$ for any n. When $\Vert {\theta }_j\Vert <\gamma {\lambda }_n$, we have $p_{{\lambda }_n,\gamma }^{\prime }\left(\Vert {\theta }_j\Vert \right)={\lambda }_n-\frac{\Vert {\theta }_j\Vert }{\gamma }>0$. Hence, J₂ = O_p(n) and the sign of $\frac{\partial Q\left(\theta \right)}{\partial {\theta }_{jk}}$ is determined by θ_jk when n is large enough.

This means as n → ∞, with probability tending to 1, for any θ satisfying ∥θ – θ*∥ ≤ $C∕\sqrt{n}$ and constant C,

$$\begin{array}{c}\hfill \frac{\partial Q\left(\theta \right)}{\partial {\theta }_{jk}}>0,\text{for}0<{\theta }_{jk}\le \Vert {\theta }_j\Vert <C∕\sqrt{n},\hfill \\ \hfill \frac{\partial Q\left(\theta \right)}{\partial {\theta }_{jk}}<0,\text{for}-C∕\sqrt{n}<-\Vert {\theta }_j\Vert \le {\theta }_{jk}<0,\hfill \end{array}$$

which implies

$$Q\left({({\theta }_{+}^T,0)}^T\right)=\underset{\theta :\Vert \theta -{\theta }^{\ast }\Vert \le C∕\sqrt{n}}{\inf }Q\left({({\theta }_{+}^T,{\theta }_0^T)}^T\right).$$

In other words, with probability approaching 1, Q(θ) reaches its minimum when θ_j = 0, j > s. In the proof of Theorem 1, we have shown that there exists a constant C such that $\Vert \widehat{\theta }-{\theta }^{\ast }\Vert <C∕\sqrt{n}$. Therefore, ${\widehat{\theta }}_0=0$ with probability approaching 1.

Next we will prove the asymptotic normality of ${\widehat{\theta }}_{+}$. ${\widehat{\theta }}_{+}={({\widehat{\theta }}_1^T,\cdots ,{\widehat{\theta }}_s^T)}^T$ is the minimizer of ${Q\left(\theta \right)\mid }_{{\theta }_0=0}$ i.e. $\frac{\partial Q\left(\theta \right)}{\partial {\theta }_{+}}\mid {}_{\theta ={({\widehat{\theta }}_{+}^T,0)}^T}=0$.

By expanding the equation above and dividing both sides of the equation by $\sqrt{nD}$, we have, for j = 1, ··· , s,

$$\begin{array}{cc}\hfill 0=& -\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\widehat{\Sigma }}^{-1}\left({\mathit{w}}_i-\sum _{j=1}^s{\mathit{Z}}_{ij}{\widehat{\theta }}_j\right)+\sqrt{nD}\frac{\partial p_{{\lambda }_n,\gamma }\left(\Vert {\widehat{\theta }}_j\Vert \right)}{\partial {\widehat{\theta }}_j}\hfill \\ \hfill =& -\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\widehat{\Sigma }}^{-1}\left({\mathit{w}}_i-\sum _{j=1}^s{\mathit{Z}}_{ij}{\theta }_j^{\ast }\right)+\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_{ij}^T{\widehat{\Sigma }}^{-1}\left[\sum _{j=1}^s{\mathit{Z}}_{ij}({\widehat{\theta }}_j-{\theta }_j^{\ast })\right]+\sqrt{nD}\frac{\partial p_{{\lambda }_n,\gamma }\left(\Vert {\widehat{\theta }}_j\Vert \right)}{\partial {\widehat{\theta }}_j}\hfill \end{array}$$

By the definition of MCP function, $p_{{\lambda }_n,\gamma }^{\left(m\right)}\left(\Vert {\theta }_i\Vert \right)=0$ when m > 2, therefore,

$$\sqrt{nD}\frac{\partial p_{{\lambda }_n,\gamma }\left(\Vert {\widehat{\theta }}_j\Vert \right)}{\partial {\widehat{\theta }}_j}=\sqrt{nD}\left(\frac{\partial p_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j^{\ast }\Vert \right)}{\partial {\theta }_j^{\ast }}+\frac{{\partial }^2{p}_{{\lambda }_n,\gamma }\left(\Vert {\theta }_j^{\ast }\Vert \right)}{\partial {\theta }_j^{\ast }\partial {\theta }_j^{\ast T}}({\widehat{\theta }}_j-{\theta }_j^{\ast })\right).$$

Since λ_n → 0 as n → ∞, we have $\Vert {\theta }_j^{\ast }\Vert >\gamma {\lambda }_n$, j = 1, ··· , s when n is large enough. Then for sufficiently large n, $p_{{\lambda }_n,\gamma }^{\prime }\left(\Vert {\theta }_j^{\ast }\Vert \right)=0$ and $p_{{\lambda }_n,\gamma }^{″}\left(\Vert {\theta }_j^{\ast }\Vert \right)=0$, which indicates $\sqrt{nD}\frac{\partial p_{{\lambda }_n,\gamma }\left(\Vert {\widehat{\theta }}_j\Vert \right)}{\partial {\widehat{\theta }}_j}$ is negligible.

Therefore, by rearranging the equation above and combining all the s equations, we have

$$\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_{i+}^T{\widehat{\Sigma }}^{-1}{\mathit{Z}}_{i+}({\widehat{\theta }}_{+}-{\theta }_{+}^{\ast })=\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_{i+}^T{\widehat{\Sigma }}^{-1}({\mathit{w}}_i-{\mathit{Z}}_{i+}{\theta }_{+}^{\ast }).$$

Since $\widehat{\Sigma }-\Sigma =O_p\left(n^{-1∕2}\right)$, by Slutsky's theorem, we have

$${\mathit{Z}}_{i+}^T{\widehat{\Sigma }}^{-1}({\mathit{w}}_i-{\mathit{Z}}_{i+}{\theta }_{+}^{\ast })\stackrel{\mathcal{D}}{\to }\mathcal{N}(0,{\mathit{Z}}_{i+}^T{\Sigma }^{-1}{\mathit{Z}}_{i+}).$$

Additionally, since ε₁, ··· , ε_n are i.i.d. and $\underset{n\to \infty }{\lim }\frac{1}{nD}{\sum }_{i=1}^n{\mathit{Z}}_i^T{\mathit{Z}}_i$ is positive definite,

$$\frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_{i+}^T{\widehat{\Sigma }}^{-1}({\mathit{w}}_i-{\mathit{Z}}_{i+}{\theta }_{+}^{\ast })\stackrel{\mathcal{D}}{\to }\mathcal{N}\left(0,\underset{n\to \infty }{\lim }\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_{i+}^T{\Sigma }^{-1}{\mathit{Z}}_{i+}\right).$$

Hence,

$$\begin{array}{cc}\hfill \frac{1}{\sqrt{nD}}\sum _{i=1}^n{\mathit{Z}}_{i+}^T{\widehat{\Sigma }}^{-1}{\mathit{Z}}_{i+}({\widehat{\theta }}_{+}-{\theta }_{+}^{\ast })\stackrel{\mathcal{D}}{\to }\mathcal{N}\left(0,\underset{n\to \infty }{\lim }\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_{i+}^T{\Sigma }^{-1}{\mathit{Z}}_{i+}\right),& \hfill \\ \hfill \left[\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_{i+}^T{\widehat{\Sigma }}^{-1}{\mathit{Z}}_{i+}\right]\sqrt{nD}({\widehat{\theta }}_{+}-{\theta }_{+}^{\ast })\stackrel{\mathcal{D}}{\to }\mathcal{N}\left(0,\underset{n\to \infty }{\lim }\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_{i+}^T{\Sigma }^{-1}{\mathit{Z}}_{i+}\right)& \hfill \end{array}$$

By Slutsky's theorem again,

$$\sqrt{nD}({\widehat{\theta }}_{+}-{\theta }_{+}^{\ast })\stackrel{\mathcal{D}}{\to }\mathcal{N}\left(0,{\left(\underset{n\to \infty }{\lim }\frac{1}{nD}\sum _{i=1}^n{\mathit{Z}}_{i+}^T{\Sigma }^{-1}{\mathit{Z}}_{i+}\right)}^{-1}\right).$$

i.e.,

$$\sqrt{n}({\widehat{\theta }}_{+}-{\theta }_{+}^{\ast })\stackrel{\mathcal{D}}{\to }\mathcal{N}\left(0,{\left(\underset{n\to \infty }{\lim }\frac{1}{n}\sum _{i=1}^n{\mathit{Z}}_{i+}^T{\Sigma }^{-1}{\mathit{Z}}_{i+}\right)}^{-1}\right).$$

B Simulation results for data with i.i.d. errors

Similar simulation studies were conducted for the case that within-function errors are uncorrelated. Datasets were constructed in the same way as what we did for the correlated case in Section 3 except that the errors were generated from a Gaussian distribution with covariance Σ = I. Likewise, we implemented four of our proposed method: one-step approaches using raw and FPCA-based covariance matrix estimates, and iterative approaches using raw and FPCA-based covariance matrix estimates. Two different values for PVE, 0.5 and 0.99, were used in the approaches involving FPCA-based covariance matrix estimate. In addition, the approach that pre-whitens using true covariance matrix, as well as ordinary least squares, the variational Bayes approach, FS-LASSO and a method that assumes uncorrelated error curves, were included for comparison.

Table A.1 reports the true positive (TP) and true negative (TN) rates of the estimates of both zero and non-zero coefficient functions. Our iterative approaches using FPCA-based covariance matrix estimate outperform all competing approaches in terms of correctly identifying the zero functions. In this case, there is no sign of a substantial decline in performance for the approaches using PVE=0.5 compared with the ones using PVE=0.99. Most methods are capable of identifying β₁(t), β₂(t) and β₃(t) as non-zero functions.

Estimates of zero and non-zero coefficient functions obtained using the iterative algorithm with FPCA-based covariance matrix estimate using PVE=0.99, as well as the mean squared error and squared bias are shown in Figure A.1. As indicated in Table A.1, none of the estimates of β₁(·), β₂(·) and β₃(·) are set equal to zero. Driven by the sinusoidal shape of the coefficient function itself and by the shrinkage to zero, the squared bias curve of β₁(·) presents a sinusoidal shape. On the contrary, the estimate of β₂(·) and β₃(·) are approximately unbiased owing to the structure of the penalty since their coefficients are relatively large. Due to the increased variability of curves at both ends of the distribution, large MSE is observed at both ends of the curves for all three non-zero coefficient functions.

Table A.1.

true positive (TP) and true negative (TN) rates of estimated coefficient functions, where FN is false negative and FP is false positive. They are estimated across all the training samples.

	$\frac{TN}{FP+TN}$	$\frac{TP}{TP+FN}\left({\beta }_1\right)$	$\frac{TP}{TP+FN}\left({\beta }_2\right)$	$\frac{TP}{TP+FN}\left({\beta }_3\right)$
FS-LASSO	0.908	0.990	1.000	1.000
MCP Assuming Independent Errors	0.943	1.000	1.000	1.000
One-step with Raw Matrix	0.371	1.000	1.000	1.000
Iterative with Raw Matrix	0.822	1.000	1.000	1.000
One-step with FPCA-based Matrix (PVE = 0.5)	0.944	1.000	1.000	1.000
Iterative with FPCA-based Matrix (PVE = 0.5)	0.950	1.000	1.000	1.000
One-step with FPCA-based Matrix (PVE = 0.99)	0.945	1.000	1.000	1.000
Iterative with FPCA-based Matrix (PVE = 0.99)	0.951	1.000	1.000	1.000
Pre-whiten with True Σ	0.943	1.000	1.000	1.000

Figure A.1.

Estimates of zero functions (left) and non-zero functions (middle) obtained using the iterative approach with FPCA-based covariance matrix estimate using PVE=0.99 across all simulated datasets. The true functions are overlaid (bold curves). The right panel shows the both MSE (solid) and squared bias (dashed) as functions of time for all the coefficient functions.

RMISE for estimated zero and non-zero functions, as well as the prediction errors on the test sample are presented in Figure A.2. Comparisons based on PVE=0.99 and PVE=0.5 are shown in the top and bottom rows, respectively. Our iterative algorithm with FPCA-based covariance matrix estimate, in particular when PVE=0.99, compares favorably to other approaches, reinforcing the results from Table A.1. In terms of both RMISE and prediction error, it is comparable to the method assuming independent error and the one that pre-whitens using true Σ.

Figure A.2.

The top row shows the comparison among the algorithms when PVE = 0.99 while the second row shows the comparison when PVE = 0.5. The three columns show RMISE for zero functions (left) and non-zero functions (middle); and prediction error (right).