Structured penalties for functional linear models—partially empirical eigenvectors for regression

Abstract

One of the challenges with functional data is incorporating geometric structure, or local correlation, into the analysis. This structure is inherent in the output from an increasing number of biomedical technologies, and a functional linear model is often used to estimate the relationship between the predictor functions and scalar responses. Common approaches to the problem of estimating a coefficient function typically involve two stages: regularization and estimation. Regularization is usually done via dimension reduction, projecting onto a predefined span of basis functions or a reduced set of eigenvectors (principal components). In contrast, we present a unified approach that directly incorporates geometric structure into the estimation process by exploiting the joint eigenproperties of the predictors and a linear penalty operator. In this sense, the components in the regression are ‘partially empirical’ and the framework is provided by the generalized singular value decomposition (GSVD). The form of the penalized estimation is not new, but the GSVD clarifies the process and informs the choice of penalty by making explicit the joint influence of the penalty and predictors on the bias, variance and performance of the estimated coefficient function. Laboratory spectroscopy data and simulations are used to illustrate the concepts.

1. Introduction

The coefficient function, β, in a functional linear model (fLM) represents the linear relationship between responses, y, and predictors, x, either of which may appear as a function. We consider the special case of scalar-on-function regression, formally written as y = ∫_I x(t)β(t) dt + ε, where x is a random function, square integrable on a closed interval I ⊂ , and ε a vector of random i.i.d. mean-zero errors. In many instances, one has an approximate idea about the informative structure of the predictors, such as the extent to which they are smooth, oscillatory, peaked, etc. Here we focus on analytical framework for incorporating such information into the estimation of β.

The analysis of data in this context involves a set of n responses ${\{y_i\}}_{i=1}^n$ corresponding to a set of predictor curves ${\{x_i\}}_{i=1}^n$, each arising as a discretized sampling of an idealized function; i.e., x_i ≡ (x_i(t₁), …, x_i(t_p)), for some, t₁ < · · · < t_p, of I. In particular, the concept of geometric or spatial structure implies an order relation among the index parameter values. We assume the predictor functions have been sampled equally and densely enough to capture geometric structure of the type typically attributed to functions in (subspaces of) L²(I). For this, it will be assumed that p > n although this condition is not necessary for our discussion.

Several methods for estimating β are based on the eigenfunctions associated with the auto-covariance operator defined by the predictors [16, 32]. These eigenfunctions provide an empirical basis for representing the estimate and are the basis for the usual ordinary least-squares and principal-component estimates in multivariate analysis. The book by Ramsay and Silverman [38] summarize a variety of estimation methods that involve some combination of the empirical eigenfunctions and smoothing, using B-splines or other technique, but none of these methods provide an analytically tractable way to incorporate presumed structure directly into the estimation process. The approach presented here achieves this by way of a penalty operator, , defined on the space of predictor functions.

The joint influence of the penalty and predictors on the estimated coefficient function is made explicit by way of the generalized singular value decomposition (GSVD) for a matrix pair. Just as the ordinary SVD provides the ingredients for an ordinary least squares estimate (in terms of the empirical basis), the GSVD provides a natural way to express a penalized least-squares estimate in terms of a basis derived from both the penalty and the predictors. We describe this in terms of the n × p matrix of sampled predictors, X, and an m × p discretized penalty operator, L. The general formulation is familiar as we consider estimates of β that arise from a squared-error loss with quadratic penalty:

$${\stackrel{\sim }{\beta }}_{\alpha ,L}=\underset{\beta }{argmin}\{{\left|\right|y-X\beta \left|\right|}_{{\mathbb{R}}^n}^2+\alpha {\left|\right|L\beta \left|\right|}_{L^2}^2\}.$$ (1)

What distinguishes our presentation from others using this formulation is an emphasis on the joint spectral properties of the pair (X, L), as arise from the GSVD. We investigate the analytical role played by L in imposing structure on the estimate and focus on how the structure of L’s least-dominant singular vectors should be commensurate with the informative structure of β.

In a Bayesian view, one may think of L as implementing a prior that favors a coefficient function lying near a particular subspace; this subspace is determined jointly by X and L. We note, however, that informative priors must come from somewhere and while they may come from expectations regarding smoothness, other information often exists—including pilot data, scientific knowledge or laboratory and instrumental properties. Our presentation aims to elucidate the role of L in providing a flexible means of implementing informative priors, regardless of their origin.

The general concept of incorporating “structural information” into regularized estimation for functional and image data is well established [2, 12, 36]. Methods for penalized regression have adopted this by constraining high-dimensional problems in various “structured” ways (sometimes with use of an L¹ norm): locally-constant structure [49, 46], spatial smoothness [20], correlation-based constraints [52], and network-dependence structure described via a graph [26]. These general penalties have been motivated by a variety of heuristics: Huang et al. [24] refer to the second-difference penalty as an “intuitive choice”; Hastie et al. [20] refer to a “structured penalty matrix [which] imposes smoothness with regard to an underlying space, time or frequency domain”; Tibshirani and Taylor [50] note that the rows of L should “reflect some believed structure or geometry in the signal”; and the penalties of Slawski et al. [46] aim to capture “a priori association structure of the features in more generality than the fused lasso.”

The most common penalty is a (discretized) derivative operator, motivated by the heuristic of penalizing roughness (see [21, 38]). Our perspective on this is more analytical: since the eigenfunctions of the second-derivative operator = (with zero boundary conditions on [0, 1]) are of the form ϕ(t) = sin(kπt), with eigenvalues k²π² (k = 1, 2, …), implements the assumption that the coefficient function is well represented by low-frequency trigonometric functions. This is in contrast to ridge regression (L = I) which imposes no geometric structure. Although not typically viewed this way, the choice of = , or any differential operator, implies a favored basis for expansion of the estimate.

A purely empirical basis comprised of a few dominant right singular vectors of X is a common and theoretically tractable choice. This is the essence of principal component regression (PCR) and these vectors also form the basis for a ridge estimate. Although this empirical basis does not technically impose local spatial structure (no order relation among the index parameter values is used), it may be justified by arguing that a few principal component vectors capture the “greatest part” of a set of predictors [17]. Properties of this approach for signal regression is the focus of [7] and [16]. The functional data analysis framework of Ramsay and Silverman [38] provides two formulations of PCR. One in which the predictor curves are themselves smoothed prior to construction of principal components (chap. 8) and another that incorporates a roughness penalty into the construction of principal components (chap. 9), as originally proposed in [45]. In a related presentation on signal regression, Marx and Eilers [30] proposed a penalized B-spline approach in which predictors are transformed using a basis external to the problem (B-splines) and the estimated coefficient function is derived using the transform coefficients. Combining ideas from [30] and [21], the “smooth principal components” method of [8] projects predictors onto the dominant eigenfunctions to obtain an estimate then uses B-splines in a procedure that smooths the estimate. Reiss and Ogden [40] provide a thorough study on several of these methods and propose modifications that include two versions of PCR using B-splines and second-derivative penalties: FPCR_C applies the penalty to the construction of the principal components (cf. [45]), while FPCR_R incorporates the penalty into the regression (cf. [38]).

In the context of nonparametric regression (X = I) the formulation (1) plays a dominant role for smoothing [54]. Related to this, Heckman and Ramsay [22] proposed a differential equations model-based estimate of a function μ whose properties are determined by a linear differential operator chosen from a parameterized family of differential equations, Lμ = 0. In this context, however, the GSVD is irrelevant since X does not appear and the role of L is relatively transparent.

Algebraic details on the GSVD as it relates to penalized least-squares are given in section 3 with analytic expressions for various properties of the estimation process are described in section 3.2. Intuitively, smaller bias is obtained by an informed choice of L (the goal being small Lβ). The affect of such a choice on the variance is described analytically. Section 4 describes several classes of structured penalties including two previously-proposed special cases that were justified by numerical simulations. The targeted penalties of subsection 4.2 are studied in more detail in section 5 including an analysis of the mean squared error for a family of penalized estimates which encompasses the ridge, principal-component and James-Stein estimates.

The assumptions on L here are increasingly restrictive to the point where the estimates are only minor extensions of these well-studied estimates. The goal, however, is to analytically describe the substantial gains achievable by even mild extensions of these established methods.

In applications the selection of the tuning parameter, α in (1), is important and so Section 6 describes our application of REML-based estimation for this. Numerical illustrations are provided in section 7: the simulation in subsection 7.1 is motivated by Reiss and Ogden’s study of fLMs [40]; 7.2 presents a simulation using experimentally-derived Raman spectroscopy curves in which the “true” β has naturally-occurring (laboratory) structure; and section 7.3 presents an application based on experimentally collected spectroscopy curves representing varied biochemical (nanoparticle) concentrations. An appendix looks at the simulation studied by Hall and Horowitz [16]. We begin in section 2 with a brief setup for notation and an introductory example. Note that for any L ≠ I, the estimated β is not given in terms of the ordinary empirical singular vectors (of X), but rather in terms of a “partially empirical” basis arising from a simultaneous diagonalization of X′X and L′L via the GSVD. Hence, for brevity, we refer to β̃_α,L as a PEER (partially empirical eigenvector for regression) estimate whenever L ≠ I.

2. Background and simple example

Let β represent a linear functional on L²(I) defining a linear relationship y = ∫_I x(t)β(t) dt + ε (observed with error, ε) between a response, y, and random predictor function, x ∈ L²(I). We assume a set of n scalar responses ${\{y_i\}}_{i=1}^n$ corresponding to the set of n predictors, ${\{x_i\}}_{i=1}^n$, each discretely sampled at common locations in I. Denote by X the n × p matrix whose ith row is a p-dimensional vector, x_i, of discretely sampled functions, and columns that are centered to have mean 0. The notation 〈·, ·〉 will be used to denote the inner product on either L²(I) or , depending on the context.

The empirical covariance operator is $K=\frac{1}{n}{X}^{\prime }X$, but for functional predictors, typically p > n or else K is ill-conditioned or rank deficient. In this case, there are either infinitely many least-squares solutions, β̂ ≡ arg min_β||y − Xβ||², or else any such solution is highly unstable and of little use. The least-squares solution having minimum norm is unique, however, and it can be obtained directly by the singular value decomposition (SVD): X = UDV′ where the left and right singular vectors, u_k and v_k, are the columns of U and V, respectively, and D =[D₁ 0], where $D_1=\text{diag}{\{{\sigma }_k\}}_{k=1}^n$, typically ordered as σ₁ ≥ σ₂ ≥ · · · ≥ σ_r > 0 (r = rank(X)). In terms of the SVD of X, the minimum-norm solution is ${\widehat{\beta }}_{+}=X^{†}y={\sum }_{{\sigma }_k\ne 0}(1/{\sigma }_k)u_k^{\prime }y{v}_k$, where X^† denotes the Moore-Penrose inverse of X: X^† = VD^†U′, where D^† = diag{1/σ_k if σ_k ≠ 0; 0 if σ_k = 0}. The orthogonal vectors that form the columns of V are the eigenvectors of X′X and sometimes referred to as a Karhunen-Lòeve (K-L) basis for the row space of X.

The solution β̂₊ is Marquardt’s generalized inverse estimator whose properties are discussed in [29]. For functional data, β̂₊ is an unstable, meaningless solution. One obvious fix is to truncate the sum to d < r terms so that ${\{{\sigma }_k\}}_{k=1}^d$ is bounded away from zero. This leads to the truncated singular value or principal component regression (PCR) estimate: ${\stackrel{\sim }{\beta }}_{\text{PCR}}\equiv {\stackrel{\sim }{\beta }}_{\text{PCR}}^d=V_d{D}_d^{-1}{U_d}^{\prime }y$ where here, and subsequently, we use the notation A_d ≡ col[a₁, …, a_d] to denote the first d columns of a matrix A.

When L = I, the minimizer in (1) is the ridge penalty due to A. E. Hoerl [23]

$${\stackrel{\sim }{\beta }}_{\alpha ,I}={(X^{\prime }X+\alpha I)}^{-1}{X}^{\prime }y=\sum _{k=1}^r\left(\frac{{\sigma }_k^2}{{\sigma }_k^2+\alpha }\right)\frac{1}{{\sigma }_k}{u}_k^{\prime }y{v}_k,$$ (2)

or, ${\stackrel{\sim }{\beta }}_{\alpha ,I}=V_r{F}^{\alpha }{D}_r^{†}{U_r}^{\prime }y$, where $F^{\alpha }=\text{diag}\{\frac{{\sigma }_k^2}{{\sigma }_k^2+\alpha }\}$. The factor F^α acts to counterweight, rather than truncate, the terms $\frac{1}{{\sigma }_k}$ as they get large. This is one of many possible filter factors which address problems of ill-determined rank (for more, see [12, 19, 33]). Weighted (or generalized) ridge regression replaces L = I with a diagonal matrix whose entries downweight those terms corresponding to the most variation [23]. Other “generalized ridge” estimates replace L = I by a discretized second-derivative operator, L = . Indeed, the Tikhonov-Phillips form of regularization (1) has a long history in the context of differential equations [51, 36] and image analysis [15, 33] with emphasis on numerical stability; see also [28]. In a linear model context, the smoothing imposed by this penalty was mentioned by Hastie and Mallows [21], discussed in Ramsay and Silverman [38], and used (on the space of spline-transform coefficients) by Marx and Eilers [31], among others. The following simple example illustrates basic behavior for some of these penalties alongside an idealized PEER penalty.

2.1. A simple example

We consider a set of n = 50 bumpy predictor curves {x_i} discretely sampled at p = 250 locations, as displayed in gray in the last panel of Figure 1. The true coefficient function, β, is displayed in black in this same panel. The responses are defined as y_i = 〈x_i, β〉 + ε_i (ε_i normal, uncorrelated mean-zero errors), and hence depend on the amplitudes of β’s three bumps centered at locations t = 45, 110, 210.

Fig 1. Partial sums of penalized estimates.

The first five odd-numbered partial sums from (7) for three penalties: 2nd-derivative (dashed), ridge (gray), targeted PEER (black; see text in sections 2.1 and 7.1). The last panel displays β (black) and 15 predictors, x_i (gray), from the simulation.

A detailed simulation with complete results are provided in section 7.1. Here we simply illustrate the estimation process for L = I, as in (2), in comparison with L = and an idealized PEER penalty. The latter is constructed using a visual inspection of the predictors and lightly penalizes the subspace spanned by such structure, specifically, bumps centered at all visible locations (approximately t = 15, 45, 80, 110, 160, 210, 240).

The first five panels serve to emphasize the role played by the structure of basis vectors that comprise the series expansion in (2) (in terms of ordinary singular vectors) versus the analogous expansion (see (7)) in terms of generalized singular vectors. In particular, Figure 1 shows several partial sums of (7) for these three penalties. The ridge process (gray) is, naturally, dominated by the right singular vectors of X which become increasingly noisy in successive partial sums. The second-derivative penalized estimate (dashed) is dominated by low-frequency structure, while the targeted PEER estimate converges quickly to the informative features.

In this toy example, visual structure (spatial location) is used to define a regularization process that easily outperforms uninformed methods of penalization. Less visual examples where the penalty is defined by a set of laboratory-derived structure (in Raman spectroscopy curves) is given in sections 7.2 and 7.3; see Figure 2. In that setting, and in general, the role played by L is appropriately viewed in terms of a preferred subspace in determined by its singular vectors. Algebraic details about how structure in the estimation process is determined jointly by X and L ≠ I are described next.

Fig 2.

Nine pure COIN spectra, P₁, …, P₉, and a coefficient function, β (each shifted for display). β arises as a solution to the fLM in which y denotes concentrations of P₉ in an in silico mixture of 50 COIN spectra, x_i (light gray). This β is used in the simulation study of Section 7.2.

3. Penalized least squares and the GSVD

Of the many methods for estimating a coefficient function discussed in the Introduction, nearly are all aimed at imposing geometric or “functional” structure into the process via the use of basis functions in some manner. An alternative to choosing a basis outright is to exploit the structure imposed by an informed choice of penalty operator. The basis, determined by a pair (X, L), can be tailored toward structure of interest by the choice of L. When this is carried out in the least-squares setting of (1), the algebraic properties of the GSVD explicitly reveal how the structure of the estimate is inherited from the spectral properties of (X, L).

3.1. The GSVD

For a given linear penalty L and parameter α > 0, the estimate in (1) takes the form

$${\stackrel{\sim }{\beta }}_{\alpha ,L}={(X^{\prime }X+\alpha L^{\prime }L)}^{-1}{X}^{\prime }y.$$ (3)

This cannot be expressed using the singular vectors of X alone, but the generalized singular value decomposition of the pair (X, L) provides a tractable and interpretable series expansion. The GSVD appears in the literature in a variety of forms and notational conventions. Here we provide the necessary notation and properties of the GSVD for our purposes (see, e.g., [19]) but refer to [4, 13, 35] for a complete discussion and details about its computation. See also the comments of Bingham and Larntz [3].

Assume X is an n×p matrix (n ≤ p) of rank n, L is an m×p matrix (m ≤ p) of rank m. We also assume that n ≤ m ≤ p ≤ m + n, and the rank of the (n + m) × p matrix Z: = [X′L′]′ is p. A unique solution is guaranteed if the null spaces of X and L intersect trivially: Null(L) ∩ Null(X) = {0}. This is not necessary for implementation, but it is natural in our applications and simplifies the notation. In addition, the condition p > n is not required, but rather than develop notation for multiple cases, this will be assumed.

Given X and L, the following matrices exist and form the decomposition below: an n × n matrix U and an m × m matrix V, each with orthonormal columns, U′U = I, V′V = I; diagonal matrices S (n × n) and M (m × m); and a nonsingular p × p matrix W such that

$$\begin{array}{lll}X=U\underset{\_}{S}{W}^{-1},\hfill & \underset{\_}{S}=\left[\begin{array}{ll}0\hfill & S\hfill \end{array}\right],\hfill & S=\text{diag}\{{\sigma }_k\}\hfill \\ L=V\underset{\_}{M}{W}^{-1}\hfill & \underset{\_}{M}=\left[\begin{array}{ll}M\hfill & 0\hfill \end{array}\right],\hfill & M=\text{diag}\{{\mu }_k\}.\hfill \end{array}$$ (4)

Here, S and M are of the form $S=\left[\begin{array}{cc}{S}_1& 0\\ 0& I_{p-m}\end{array}\right]$ and $M=\left[\begin{array}{cc}{I}_{p-n}& 0\\ 0& M_1\end{array}\right]$, whose submatrices S₁ and M₁ have l:= n + m − p diagonal entries ordered as

$$\begin{array}{c}0\le {\sigma }_1\le {\sigma }_2\le \cdots \le {\sigma }_{\ell }\le 1\\ 1\ge {\mu }_1\ge {\mu }_2\ge \cdots \ge {\mu }_{\ell }\ge 0\end{array}\text{where}{\sigma }_k^2+{\mu }_k^2=1,k=1,\dots ,\ell ,$$ (5)

These matrices satisfy

$$W^{\prime }{X}^{\prime }XW=\left[\begin{array}{ccc}0& 0& 0\\ 0& S_1^2& 0\\ 0& 0& I\end{array}\right]={\underset{\_}{S}}^{\prime }\underset{\_}{S},W^{\prime }{L}^{\prime }LW=\left[\begin{array}{ccc}I& 0& 0\\ 0& M_1^2& 0\\ 0& 0& 0\end{array}\right]={\underset{\_}{M}}^{\prime }\underset{\_}{M},$$ (6)

with S′S + M′M = I.

Denote the columns of U, V and W by u_k, v_k and w_k, respectively. When L = I, the the generalized singular vectors u_k and v_k are those in the ordinary SVD of X (as denoted in Section 2) but their ordering is reversed. In this case, the corresponding ordinary singular values are equal to γ_k:= σ_k/μ_k for μ_k > 0. When L ≠ I, however, the GSVD vectors and values of the pair (X, L) differ from those in the SVD of X. By the convention for ordering the GS values and vectors, the last few columns of W span the subspace Null(L) (or, if Null(L) is empty, they correspond to the smallest GS values, μ_k). We set d = dim(Null(L)) and note that μ_k = 0 for k > n − d. In the special case that m = p and L is a p × p nonsingular matrix, we have L = V MW⁻¹ and XL⁻¹ = U(S M^†)V′, which connects the SVD of XL⁻¹ to the GSVD of (X, L). In general, we define the n × m matrix $\underset{\_}{\mathrm{\Gamma }}:=\underset{\_}{S}{\underset{\_}{M}}^{†}=[\begin{array}{cc}0& S_1{M}_1^{†}\end{array}]$ and the l × l diagonal matrix $\mathrm{\Gamma }:=S_1{M}_1^{†}$ with entries γ_k = σ_k/μ_k for μ_k > 0, and γ_k = 0 for μ_k = 0.

Now, equation (6) and some algebra gives (X′X + αL′L)⁻¹ = W(S′S + αM′M)⁻¹W′, and so β̃_α,L = W (S′S + αM′M)⁻¹S′U′y. A consequence of the ordering adopted for the GS values and vectors is that the first p−n columns of W don’t enter into the expression for β̃_α,L; see equation (4). Therefore, we will re-index the columns of W to reflect this and also so that the indexing coincides with that established for the GS values and vectors in (5). That is, denote the columns of W as follows: W:= col[w₍₁₎, …, w_(p−n) | w₁, …, w_n−d | w_n−d+1, …, w_n]. Therefore, the L-penalized estimate can be expressed as a series in terms of the GSVD as

$${\stackrel{\sim }{\beta }}_{\alpha ,L}=\sum _{k=1}^{n-d}\left(\frac{{\sigma }_k^2}{{\sigma }_k^2+\alpha {\mu }_k^2}\right){\frac{1}{\sigma }}_k{u}_k^{\prime }y{w}_k+\sum _{k=n-d+1}^n{u}_k^{\prime }y{w}_k.$$ (7)

This GSV expansion corresponds to a new basis for the estimation process: the estimate is expressed in terms of GS vectors {w_k} determined jointly by X and L; cf. the ridge estimate in (2).

For brevity, set o:= n − d and recall that A_o denotes the first n − d columns of a matrix A. Now denote by A_φ the last d columns of A. In particular, the range of W_φ is Null(L). Using this notation,

$$W=\text{col}[w_{(1)},\dots ,w_{(p-n)}\mid w_1,\dots ,w_{n-d}\mid w_{n-d+1},\dots ,w_n]\equiv [W_{(\ast )}\mid W_{\text{o}}\mid W_{\phi }]$$ (8)

and equation (7) may be written concisely as

$${\stackrel{\sim }{\beta }}_{\alpha ,L}=W_o{F}^{\alpha }{\mathrm{\Gamma }}_o^{†}{U_o}^{\prime }y+W_{\phi }{U_{\phi }}^{\prime }y,$$ (9)

where $F^{\alpha }=\text{diag}{\left\{\frac{{\sigma }_k^2}{{\sigma }_k^2+\alpha {\mu }_k^2}\right\}}_{k=1}^{n-d}$.

In summary, the utility of a penalty L depends on whether the true coefficient function shares structural properties with this GSVD basis, ${\{w_k\}}_{k=1}^n$. With regard to this, the importance of the parameter α may be reduced by a judicious choice of L since the terms in (7) corresponding to the vectors {w_k: μ_k = 0} are independent of the parameter α [53].

As we’ll see, bias enters the estimate to the extent that the vectors {w_k: μ_k ≠ 0 } appear in the expansion (7). The portion of β̃_α,L that extends beyond the subspace Null(L) is constrained by a sphere (of radius determined by α); this portion corresponds to bias. Hence, L may be chosen in such a way that the bias and variance of β̃_α,L arises from a specific type of structure, potentially decreasing bias without increasing complexity of the model. As a common example, L = introduces a bias toward smoothness with structure imposed by the low-frequency trigonometric functions that correspond to its smallest eigenfunctions.

3.2. Bias and variance and the choice of penalty operator

Begin by observing that the penalized estimate β̃_α,L in (3) is a linear transformation of any solution to the normal equations. Indeed, define $X^{\#}=X_{\alpha ,L}^{\#}={(X^{\prime }X+\alpha L^{\prime }L)}^{-1}{X}^{\prime }$ and note that if β̂ denotes any solution to X′Xβ = X′y, then β̃_α,L = X^#X β̂ +X^#ε. The resolution operator X^#X reflects the extent to which the estimate in (7) is linearly transformed relative to an exact solution. In particular, E(β̃_α,L) = X^#Xβ. Additionally, we have bias(β̃_α,L) = (I − X^#X)β = α(X′X + αL′L)⁻¹L′Lβ, and so ||bias(β̃_α,L)|| ≤ ||α(X′X + αL′L)⁻¹L′||||Lβ||. Hence bias can be controlled by the choice of L, with an estimate being unbiased whenever Lβ = 0. There is a tradeoff, of course, and equation (11) below quantifies the effect on the variance as determined by W_φ (i.e., ${\{w_k\}}_{k=n-d+1}^n$) if Null(L) is chosen to be too large.

More generally, the decompositions in (4) lead to an expression for the resolution matrix as X^#X = W(S′S + αM′M)⁻¹S′SW⁻¹, and

$$\begin{array}{l}I-X^{\#}X=\alpha W{({\underset{\_}{S}}^{\prime }\underset{\_}{S}+\alpha {\underset{\_}{M}}^{\prime }\underset{\_}{M})}^{-1}{\underset{\_}{M}}^{\prime }\underset{\_}{M}{W}^{-1}\\ =\alpha W{\left(\left[\begin{array}{ccc}0& 0& 0\\ 0& S_1^2& 0\\ 0& 0& I\end{array}\right]+\alpha \left[\begin{array}{ccc}I& 0& 0\\ 0& M_1^2& 0\\ 0& 0& 0\end{array}\right]\right)}^{-1}\left[\begin{array}{ccc}I& 0& 0\\ 0& M_1^2& 0\\ 0& 0& 0\end{array}\right]W^{-1}\\ =W\left[\begin{array}{ccc}I& 0& 0\\ 0& \alpha {(S_1^2+\alpha M_1^2)}^{-1}{M}_1^2& 0\\ 0& 0& 0\end{array}\right]W^{-1}.\end{array}$$

For notational convenience, define W̃:= W′⁻¹ (note, W̃ plays a role analogous to V ≡ V′⁻¹ in the SVD). The bias of β̃_α,L can be expressed as

$$\text{bias}({\stackrel{\sim }{\beta }}_{\alpha ,L})=(I-X^{\#}X)\beta =\sum _{k=1}^{n-d}\frac{\alpha {\mu }_k^2}{{\sigma }_k^2+\alpha {\mu }_k^2}{w}_k{{\stackrel{\sim }{w}}_k}^{\prime }\beta +\sum _{j=1}^{p-n}{w}_{(j)}{\stackrel{\sim }{w}}_{(j)}^{\prime }\beta$$ (10)

where w̃_k is the kth column of W̃, and the w_(j)’s come from the first p − n columns of W; see (8).

A counterpart is an expression for the variance in terms of the GSVD. Let Σ denote the covariance for ε. Then var(β̃_α,L) = var(X^#Xβ+X^#ε) = X^#Σ(X^#)′. Assuming Σ = σ_εI, this simplifies to

$$var({\stackrel{\sim }{\beta }}_{\alpha ,L})={\sigma }_{\epsilon }^2{X}^{\#}{(X^{\#})}^{\prime }={\sigma }_{\epsilon }^2\left(\sum _{k=1}^{n-d}\frac{{\sigma }_k^2}{({\sigma }_k^2+\alpha {\mu }_k^2)}{w}_k{w}_k^{\prime }+\sum _{k=n-d+1}^n{w}_k{w}_k^{\prime }\right).$$ (11)

An interesting perspective of the bias-variance tradeoff is provided by the relationship between the GS-values in (5) and their role in equations (10) and (11). Moreover, these lead to an explicit expression for the mean squared error (MSE) of a PEER estimate. Since E(β̃_α,L) = X^#Xβ,

$$\begin{array}{l}\text{MSE}({\stackrel{\sim }{\beta }}_{\alpha ,L})=E({\left|\right|\beta -{\stackrel{\sim }{\beta }}_{\alpha ,L}\left|\right|}^2)=E({\left|\right|\beta \left|\right|}^2+{\left|\right|{\stackrel{\sim }{\beta }}_{\alpha ,L}\left|\right|}^2-2\langle \beta ,{\stackrel{\sim }{\beta }}_{\alpha ,L}\rangle )\\ ={\left|\right|\beta -X^{\#}X\beta \left|\right|}^2+{\sigma }_{\epsilon }^2\text{trace}(X^{\#}{X^{\#}}^{\prime })\\ ={\left|\right|\text{bias}({\stackrel{\sim }{\beta }}_{\alpha ,L})\left|\right|}^2+{\sigma }_{\epsilon }^2\sum _{k=1}^{n-d}\frac{{\sigma }_k^2}{{({\sigma }_k^2+\alpha {\mu }_k^2)}^2}{\left|\right|w_k\left|\right|}^2.\end{array}$$

As a final remark, recall that one perspective on ridge estimation defines fictitious data from an orthogonal “experiment,” represented by an L, and expresses I as I = L′L [29]. Regardless of orthogonality this applies to any penalized estimate and L may similarly be viewed as augmenting the data, influencing the estimation process through its eigenstructure; the response, y, is set to zero for these supplementary “data”. In this view, equation (3) can be written as Zβ = y where $Z=\left[\begin{array}{c}X\\ \sqrt{\alpha }L\end{array}\right]$ and $\underset{\_}{y}=\left[\begin{array}{c}y\\ 0\end{array}\right]$. This formulation proves useful in section 5.3 when assuring that the estimation process is stable with respect to perturbations in X and the choice of L.

4. Structured penalties

A structured penalty refers to a second term in (1) that involves an operator chosen to encourage certain functional properties in the estimate. A prototypical example is a derivative operator which imposes smoothness via its eigenstructure. Here we describe several examples of structured penalties, including two that were motivated heuristically and implemented without regard to the spectral properties that define their performance. Sections 3.2 and 5.3 provide a complete formulation of their properties as revealed by the GSVD.

4.1. The penalty of C. Goutis

The concept of using a penalty operator whose eigenstructure is targeted toward specific properties in the predictors appears implicitly in the work of C. Goutis [14]. This method aimed to account for the “functional nature of the predictors” without oversmoothing and, in essence, considered the inverse of a smoothing penalty. Specifically, if Δ denotes a discretized second-derivative operator (with some specified boundary conditions), the minimization in (1) was replaced by ${min}_{\beta }\{{\left|\right|Y-X{\mathrm{\Delta }}^{\prime }\mathrm{\Delta }\beta \left|\right|}_{{\mathbb{R}}^n}^2+\alpha {\left|\right|\mathrm{\Delta }\beta \left|\right|}_{L^2}^2\}$. Here, the term XΔ′ Δβ can be viewed as the product of XΔ′ (derivatives of the predictor curves) and Δβ (derivative of β). Defining γ: = Δ′Δβ and seeking a penalized estimate of γ leads to

$$\begin{array}{l}\stackrel{\sim }{\gamma }={(X^{\prime }X+\alpha {({\mathrm{\Delta }}^{\prime }\mathrm{\Delta })}^{†})}^{-1}{X}^{\prime }y\\ =\underset{\gamma }{argmin}\{{\left|\right|y-X\gamma \left|\right|}^2+\alpha \langle \gamma ,{({\mathrm{\Delta }}^{\prime }\mathrm{\Delta })}^{†}\gamma \rangle \}.\end{array}$$ (12)

In [14], the properties of γ̃ were conjectured to result from the eigenproperties of (Δ′Δ)^†. This was explored by ignoring X and plotting some eigenvectors of (Δ′Δ)^†. The properties of this method become transparent, however, when formulated in terms of the GSVD. That is, let L:= ((Δ′Δ)^†)^1/2 and note the functions that define γ̂ are influenced most by the highly oscillatory eigenvectors of L which correspond to its smallest eigenvalues; see equations (5) and (7).

This approach was applied in [14] only for prediction and has drawbacks in producing an interpretable estimate, especially for non-smooth predictor curves. The general insight is valid, however, and modifications of this penalty can be used to produce more stable results. The operator (Δ′Δ)^† essentially reverses the frequency properties of the eigenvectors of Δ and is an extreme alternative to this smoothing penalty. An eigenanalysis of the pair (X, L), however, suggests penalties that may be more suited to the problem. This is illustrated in Section 7.

4.2. Targeted penalties

Given some knowledge about the relevant structure, a penalty can be defined in terms of a subspace containing this structure. For example, suppose $\beta \in \mathcal{Q}:=\text{span}{\{q_j\}}_{j=1}^d$ in L²(I). Set $Q={\sum }_{j=1}^d{q}_j\otimes q_j$ and consider the orthogonal projection = QQ^†. (Here, q ⊗ q denotes the rank-one operator f ↦ 〈f, q〉q, for f ∈ L²(I).) For L = I − , then β ∈ Null(L) and β̃_α,L is unbiased. The problem may still be underdetermined so, more pragmatically, define a decomposition-based penalty

$$L\equiv L_{\mathcal{Q}}=a(I-P_{\mathcal{Q}})+{bP}_{\mathcal{Q}}$$ (13)

for some a, b ≥ 0. Heuristically, when a > b > 0 the effect is to move the estimate towards Q by preferentially penalizing components orthogonal to ; i.e., assign a prior favoring structure contained in the subspace . To implement the tradeoff between the two subspaces, we view a and b as inversely related, ab = const. The analytical properties of estimates that arise from this are developed in the next section and illustrated numerically in Section 7. For example, bias is substantially reduced when β ⊂ , and equation (18) quantifies the tradeoff with respect to variance when the prior is chosen poorly.

More generally, one may penalize each subspace differently by defining L = α₁(I − ) (I − ) + α₂ , for some operators and . This idea could be carried further: for any orthogonal decomposition of L²(I) by subspaces , …, , let P_j be the projection onto . Then the multi-space penalty $L={\sum }_{j=1}^J{\alpha }_j{P}_j$ leads to the estimate

$$\stackrel{\sim }{\beta }=\underset{\beta }{argmin}\{{\left|\right|y-X\beta \left|\right|}^2+\sum _{j=1}^J{\alpha }_j{\left|\right|P_j\beta \left|\right|}^2\}.$$

This concept was applied in the context of image recovery (where X represents a linear distortion model for a degraded image y) by Belge et al. [1].

The examples here illustrate ways in which assumptions about the structure of a coefficient function can be incorporated directly into the estimation process. In general, any estimation of β imposes assumptions about its structure (either implicitly or explicitly) and section 3.2 shows that the bias-variance tradeoff involves a choice on the type of bias (spatial structure) as well as the extent of bias (regularization parameter(s)).

5. Some analytical properties

Any direct comparison between estimates using different penalty operators is confounded by the fact there is no simple connection between the generalized singular values/vectors and the ordinary singular values/vectors. Therefore, we first consider the case of targeted or projection-based penalties (13). Within this class, we introduce a parameterized family of estimates that are comprised of ordinary singular values/vectors. Since the ridge and PCR estimates are contained in (or a limit of) this family, a comparison with some targeted PEER estimates is possible. For more general penalized estimates, properties of perturbations provide some less precise relationships; see proposition 5.6.

5.1. Transformation to standard form

We have reason to consider decomposition-based penalties (13) in which L is invertible. In this case, an expression for the estimate does not involve the second term in (9), and decomposing the first term into two parts will be useful. For this, we find it convenient to use the standard-form transformation due to Elden [11] in which the penalty L is absorbed into X. This transformation also provides a computational alternative to the GSVD which, for projection-based penalties in particular, can be less computationally expensive; see, e.g., [25]. By this transformation of X, a general PEER estimate (L ≠ I) can be expressed via a ridge-regression process.

Define the X-weighted generalized inverse of L and the corresponding transformed X as

$$L_X^{†}:=(I-{[X(I-L^{†}L)]}^{†}X)L^{†}\text{and}\mathbb{X}:=X{L}_X^{†};$$

see [11, 19]. In terms of the GSVD components (4), the transformed X is = UΓV ′. In particular, the diagonal elements of $\mathrm{\Gamma }=S_1{M}_1^{†}$ are the ordinary singular values of , but in reversed order.

Now define β̃_φ:= [X(I − L^†L)]^†y, the component of the regularized solution β̃_α,L that is in Null(L). The PEER estimate can be obtained from a ridge-like penalization process with respect to as follows. Defining a ridge estimate in the transformed space as

(14)

then the PEER estimate is recovered as

Note that the transformed estimate as given in terms of the GSVD factors is: = V FΓ^†U′y, where $F=\text{diag}\{{\gamma }_k^2/({\gamma }_k^2+\alpha )\}$.

In what follows we consider invertible L in which case $L_X^{†}=L^{-1}$, [X(I − L^†L)]^† = 0, and = y. In particular, β̃_α,L = L⁻¹ . For the penalty (13) of the form L = a(I − ) + b , then $L^{-1}=\frac{1}{a}(I-P_{\mathcal{Q}})+\frac{1}{b}{P}_{\mathcal{Q}}$, and so $\mathbb{X}=\frac{1}{a}X(I-P_{\mathcal{Q}})+\frac{1}{b}X{P}_{\mathcal{Q}}$. The regularization parameter, previously denoted by α, can be absorbed into the values a and b, so we will denote this PEER estimate β̃_α,L simply as β̃_a,b.

Remark 5.1

When $a=b=\sqrt{\alpha }$, this is simply a ridge estimate: β̃_a,b = β̃_α,I. Therefore, the best performance among this family of estimates is as least as good as the performance of ridge, regardless of the choice of .

5.2. SVD targeted penalties

Consider the special case in which is the span of the d largest right singular vectors of an n ×p matrix X of rank n. Let $X=U\left[\begin{array}{cc}0& D\end{array}\right]V^{\prime }$ be an ordinary singular value decomposition where D is a diagonal matrix of singular values. For consistency with the GSVD notation, these will be ordered as 0 ≤ σ₁ ≤ · · · ≤ σ_n. Hence the last d columns of V correspond to the d largest singular values of X. For the rest of this section, we adopt the convention for indexing the columns of V as use for W in (8). In particular, Q = V_φ.

We are interested interested in the penalty L = a(I − ) + b , where d = dim(Null(I − )). Similar to before, set o = n − d and define o × o and d × d submatrices, D_o and D_φ, of D as

$$D=\left[\begin{array}{cc}{D}_{\text{o}}& 0\\ 0& D_{\phi }\end{array}\right];\text{also}\text{set}\mathrm{\Lambda }=\left[\begin{array}{cc}{aI}_{\text{o}}& 0\\ 0& {bI}_d\end{array}\right].$$ (15)

Here, = V_φV_φ′ and (I − ) = V_oV_o′ and so,

$$\begin{array}{l}\mathbb{X}=\frac{1}{a}{\mathit{UDV}}^{\prime }(V_{\text{o}}{V_{\text{o}}}^{\prime })+\frac{1}{b}{\mathit{UDV}}^{\prime }(V_{\phi }{V_{\phi }}^{\prime })=\frac{1}{a}UD\left[\begin{array}{c}{V_{\text{o}}}^{\prime }\\ 0\end{array}\right]+\frac{1}{b}UD\left[\begin{array}{c}0\\ {V_{\phi }}^{\prime }\end{array}\right]\\ =U\left[\begin{array}{c}\frac{1}{a}{D}_{\text{o}}{V_{\text{o}}}^{\prime }\\ 0\end{array}\right]+U\left[\begin{array}{c}0\\ \frac{1}{b}{D}_{\phi }{V_{\phi }}^{\prime }\end{array}\right]=U(D{\mathrm{\Lambda }}^{-1})V^{\prime }.\end{array}$$

This decomposition implies that the ridge estimate in (14) in the transformed space is of the following form: setting G = DΛ⁻¹, denoting its diagonal entries by {γ_k}, and defining $F=\text{diag}\{{\gamma }_k^2/({\gamma }_k^2+1)\}$ gives = V FG^†U′y. Now,

$$L^{-1}V=\frac{1}{a}{V}_{\text{o}}{V_{\text{o}}}^{\prime }V+\frac{1}{b}{V}_{\phi }{V_{\phi }}^{\prime }V=\left[\begin{array}{c}{V}_{\text{o}}\\ V_{\phi }\end{array}\right]\left[\begin{array}{cc}\frac{1}{a}{I}_{\text{o}}& 0\\ 0& \frac{1}{b}{I}_d\end{array}\right]=V{\mathrm{\Lambda }}^{-1}$$

and so β̃_a,b = L⁻¹ = L⁻¹(V FG^†U′y) = VΛ⁻¹FΛD⁻¹U′y. By the decomposition (15),

$${\stackrel{\sim }{\beta }}_{a,b}=V_{\text{o}}{F}_{\text{o}}{D}_{\text{o}}^{-1}{U_{\text{o}}}^{\prime }y+V_{\phi }{F}_{\phi }{D}_{\phi }^{-1}{U_{\phi }}^{\prime }y.$$

This shows that the estimate decomposes as follows.

Theorem 5.2

Let be the span of the largest d right singular vectors of X. Set L = a(I − ) + b . Then, in terms of the notation above, the estimate β̃_a,b decomposes as

$${\stackrel{\sim }{\beta }}_{a,b}=\sum _{k=1}^{n-d}\left(\frac{{\sigma }_k^2}{{\sigma }_k^2+a^2}\right){\frac{1}{\sigma }}_k{u}_k^{\prime }y{v}_k+\sum _{k=n-d+1}^n\left(\frac{{\sigma }_k^2}{{\sigma }_k^2+b^2}\right){\frac{1}{\sigma }}_k{u}_k^{\prime }y{v}_k,$$ (16)

where the left and right terms are independent of b and a, respectively.

Similar arguments can be used to decompose an estimate for arbitrary :

$${\stackrel{\sim }{\beta }}_{a,b}=\sum _{k=1}^{n-d}\left(\frac{{\sigma }_k^2}{{\sigma }_k^2+a^2{\mu }_k^2}\right){\frac{1}{\sigma }}_k{u}_k^{\prime }y{w}_k+\sum _{k=n-d+1}^n\left(\frac{{\sigma }_k^2}{{\sigma }_k^2+b^2{\mu }_k^2}\right){\frac{1}{\sigma }}_k{u}_k^{\prime }y{w}_k.$$ (17)

In this case, however, all terms are dependent on both a and b. Indeed, using notation as in (9) one can decompose $\mathbb{X}=\frac{1}{a}{U}_{\text{o}}{\mathrm{\Gamma }}_{\text{o}}{V}_{\text{o}}^{\prime }+\frac{1}{b}{U}_{\phi }{\mathrm{\Gamma }}_{\phi }{V}_{\phi }^{\prime }$ and obtain = V FΓ^†U′y. However, L⁻¹V = WM^†, and the non-orthogonal terms provided by W do not decompose the estimate into terms from the orthogonal sum ⊕ .

The following corollary, along with Remark 5.1, records the manner in which (16) is a family of penalized estimates, parameterized by a, b ≥ 0 and d ∈ {1, …, n}, that extends some standard estimates.

Corollary 5.3

Under the conditions in Theorem 5.2,

1. when a > b > 0, β̃_a,b is a sum of weighted ridge estimates on and ;

2. when a > 0 and b = 0, β̃_a,0 is given by (9), which is a sum of PCR and ridge estimates on and , respectively;

3. for each d, the PCR estimate ${\stackrel{\sim }{\beta }}_{\mathit{PCR}}^d$ is the limit of β̃_a,0 as a → ∞.

In item 2, this estimate is similar to PCR except that a ridge penalty is placed on the least-dominant singular vectors. Under the assumptions here, w_k ≡ v_k are the ordinary singular vectors of X and the ordinary singular values appear as γ_k = σ_k/μ_k, for μ_k > 0. In the second term of (9), the singular vectors are in the null space of L (since b = 0), and so μ_k = 0 and σ_k = 1, for k = n − d + 1, …, p. Regarding item 3, although a PCR estimate is not obtained from equation (3) for any L, it is a limit of such estimates.

Other decompositions may be obtained simply by using a permutation, such as Q = ΠV, for some n × n permutation matrix Π. Stein’s estimate, β̃_α,S, also fits into this framework as follows. When X′X is nonsingular, then β̃_α,S = (X′X+αX′X)⁻¹X′y (see, e.g., the class ‘STEIN’ in [10]), and X′X = V D′DV ′. Hence this estimate arises from the penalty L_S = DV ′. This is a re-weighted version of L = a(I − ) where d = n, Q = V and the parameter a is replaced by the matrix D. The result is a constant filter factor F = diag{1/(1 + α)}. Using d < n and Q = V_d is a natural extension of this idea. More generally, may be enriched with functions that span a wider range of structure potentially relevant to the estimate. This concept is illustrated in Section 7.3 where instead of V_d, we use a d-dimensional set of experimentally-derived “template” spectra supplemented with their derivatives to define .

As an aside, we note that in a different approach to regularization one can define a general family of estimates arising from the SVD by way of β̃_h,ϕ = VΣ_hU ′y, where ${\mathrm{\sum }}_h=\text{diag}\{\frac{{\sigma }_k}{h}\varphi (\frac{{\sigma }_k^2}{h^2})\}$, and ϕ: → is an arbitrary continuous function [33]. A ridge estimate is obtained for ϕ(t) = 1/(1 + t), and PCR obtained for ϕ(t) = 1/t if t > 1, ϕ(t) = 0 if t ≤ 1 (an L²-limit of continuous functions). This is similar to item 3 in Corollary 5.3, but the family of estimates β̃_h,ϕ is formulated in terms of functional filter factors rather than explicit penalty operators. Related to this is the fact that the optimal (with respect to MSE) estimate using SVD filter factors is, in the case C = σ_εI, expressed as β̃_OH = V FD^†V ′y, where $F=\text{diag}\{{\sigma }_k^2/({\sigma }_k^2+{\sigma }_{\epsilon }^2{(v_k^{\prime }\beta )}^{-2})\}$; see the “ideal filter” of O’Brien and Holt [34]. In fact, it’s easy to check that this optimal estimate can be obtained as β̃_OH = β̃_α,L for some L ≠ I. Since β̃_OH involves knowledge of β, it is not directly obtainable but it points to the optimality of a PEER estimate.

5.3. The MSE of some penalized estimates

Theorem 5.2 is used here to show that β̃_a,b can have smaller MSE than the ridge or PCR estimates for a wide range of values of a and/or b. The MSE is potentially decreased further when L is defined by a more general . In that case, a general statement is difficult to formulate but Proposition 5.6 confirms that any improvement in MSE is robust to perturbations in L (e.g., general ) and errors in x.

An immediate consequence of Theorem 5.2 is that the mean squared error for an estimate in this family (16) decomposes into easily-identifiable terms for the bias and variance:

$$\text{MSE}({\stackrel{\sim }{\beta }}_{a,b})=\sum _{k=1}^{n-d}{\left(\frac{a^2}{{\sigma }_k^2+a^2}\right)}^2{(v_k^{\prime }\beta )}^2+\sum _{k=n-d+1}^n{\left(\frac{b^2}{{\sigma }_k^2+b^2}\right)}^2{(v_k^{\prime }\beta )}^2+\sum _{j=1}^{p-n}{(v_{(j)}^{\prime }\beta )}^2+{\sigma }_{\epsilon }^2\left(\sum _{k=1}^{n-d}\frac{{\sigma }_k^2}{{({\sigma }_k^2+a^2)}^2}+\sum _{k=n-d+1}^n\frac{{\sigma }_k^2}{{({\sigma }_k^2+b^2)}^2}\right).$$ (18)

The influence of b = 0 on the estimate is now clear: when the numerical rank of X is small relative to d, the σ_k’s in the last term decrease and the contribution to the variance from this term increases—the estimate fails for the same reason that ordinary least-squares fails. Any nonzero b stabilizes the estimate in the same way that a nonzero α stabilizes a standard ridge estimate; the decomposition (16) merely re-focuses the penalty. This is illustrated in Section 7 (Table 1) and in the Appendix (Table 4). Although there are three parameters to consider, the MSE of β̃_a,b is relatively insensitive to b > 0 for sufficiently large d. This could be optimized (similar to efforts to optimize the number of principal components) but here we assume approximate knowledge regarding , hence d. Relationships between ridge, PCR and PEER estimates in this family {β̃_a,b}_a,b>0 can be quantified more specifically as follows.

Table 1.

Estimation errors (MSE) for simulation with selected bump locations. Sample size is n = 50

R2	S/N	LV	LU	PCR	ridge		+ a I
0.8	10	4.00	13.81	9.38	34.39	359.83	76.31
0.8	5	3.72	15.46	21.50	40.02	246.17	72.64
0.8	2	4.40	12.96	57.89	58.22	126.75	59.35
0.6	10	9.60	21.60	14.10	50.50	497.70	113.60
0.6	5	10.22	21.65	26.02	50.68	338.70	87.58
0.6	2	11.75	23.18	63.50	67.94	181.75	78.45

Table 4.

Estimation errors (MSE) for the simulation with localized frequencies

n	R2	S/N	LF	LG	PCR	ridge		+ a I
50	0.8	10	42.42	77.31	123.60	132.50	1051.20	568.45
50	0.8	5	41.55	75.41	128.75	143.48	447.64	184.07
200	0.8	10	8.28	13.48	33.44	65.78	453.54	169.41
200	0.8	5	8.56	13.08	36.36	87.59	100.15	76.24
50	0.6	10	106.89	200.08	173.56	173.94	1098.20	631.05
50	0.6	5	109.51	178.05	178.15	196.62	612.12	259.92
200	0.6	10	25.30	38.73	58.90	98.13	847.59	350.46
200	0.6	5	22.08	33.79	59.92	119.48	240.09	127.52

Proposition 5.4

Suppose β ∈ and fix α > 0. Then for any $a>\sqrt{\alpha }$, the ridge estimate satisfies

$$\mathit{MSE}({\stackrel{\sim }{\beta }}_{\alpha ,I})\equiv \mathit{MSE}({\stackrel{\sim }{\beta }}_{\sqrt{\alpha },\sqrt{\alpha }})>\mathit{MSE}({\stackrel{\sim }{\beta }}_{a,\sqrt{\alpha }}).$$

Proof

This follows from the fact that if β ∈ , then V_o′β = 0 and so the first term in (18) is zero. Therefore, the contribution to the MSE by the fourth term is decreased whenever $a>\sqrt{\alpha }$.

If β is exactly a sum of the d dominant right singular vectors, A PCR estimate using d terms may perform well, but in general it is not optimal:

Proposition 5.5

If β ∈ , a sufficient condition for the PCR estimate to satisfy

$$\mathit{MSE}({\stackrel{\sim }{\beta }}_{\mathit{PCR}}^d)\equiv \mathit{MSE}({\stackrel{\sim }{\beta }}_{\infty ,0})>\mathit{MSE}({\stackrel{\sim }{\beta }}_{\infty ,b})$$

is

$${\sigma }_{\epsilon }^2\left(\sum _{k=n-d+1}^n\frac{1}{{\sigma }_k^2}+\frac{2d}{b^2}\right)>{\left|\right|{V_{\phi }}^{\prime }\beta \left|\right|}^2.$$ (19)

Note that the left side of (19) increases without bound as σ_k → 0. Since ${\left|\right|{V_{\phi }}^{\prime }\beta \left|\right|}^2={\sum }_{k=n-d+1}^n{(v_k^{\prime }\beta )}^2$, and since the premise of PCR is that $v_k^{\prime }\beta$ decreases with decreasing σ_k, this sufficient condition is entirely plausible.

Proof

If β ∈ , then the first and third terms in (18) are zero and the MSE of ${\stackrel{\sim }{\beta }}_{\text{PCR}}^d$ consists of the second and last terms of (18):

$$\text{MSE}({\stackrel{\sim }{\beta }}_{\text{PCR}}^d)=\sum _{k=1}^{n-d}{(v_k^{\prime }\beta )}^2+{\sigma }_{\epsilon }^2\sum _{k=n-d+1}^n\frac{1}{{\sigma }_k^2}.$$

In particular, a sufficient condition for this to exceed MSE(β̃_∞,b) is for the variance term to exceed the second and last terms of (18):

$${\sigma }_{\epsilon }^2\sum _{k=n-d+1}^n\frac{1}{{\sigma }_k^2}>{\sigma }_{\epsilon }^2\sum _{k=n-d+1}^n\frac{{\sigma }_k^2}{{({\sigma }_k^2+b^2)}^2}+\sum _{k=n-d+1}^n{\left(\frac{b^2}{{\sigma }_k^2+b^2}\right)}^2{(v_k^{\prime }\beta )}^2.$$

One can check that this is satisfied when (19) holds.

A comment by Bingham and Larntz [3] on Dempster et al.’s intensive simulation study of ridge regression in [10] notes that “it is not at all clear that ridge methods offer a clear-cut improvement over [ordinary] least squares except for particular orientations of β relative to the eigenvectors of X′X.” Equation (18) repeats this observation relating these two classical methods as well as the minor extensions contained in (16). If, on the other hand, the orientation of β relative to the v_k’s is not favorable, i.e., if β is nowhere near the range of V, then a PEER estimate as in (17) is more desirable than the estimate in (16) (assuming sufficient information is available to form ).

In summary, the family of estimates {β̃_a,b}_a,b>0 in (16) represents a hybrid of ridge and PCR estimation. This family—based on the ordinary singular vectors of X—is introduced here to provide a framework within which these two familiar estimates can be compared to (slightly) more general PEER estimates. Direct analytical comparison between general PEER estimates is more difficult since there’s no simple relationship between the generalized singular vectors for two different L (including L = I versus L ≠ I). However, it is important that the estimation process be stable with respect to changes in L and/or X. I.e., in going from an estimate in (16) to one in (17), the performance of the estimate should be predictably altered. Given an estimate in Proposition 5.4, if is modified and/or X is observed with error, the MSE of the corresponding estimate, ${\stackrel{\sim }{\beta }}_{\alpha ,L}^E$, should be controlled: for sufficiently small perturbation E, the corresponding estimate $\text{MSE}({\stackrel{\sim }{\beta }}_{\alpha ,L}^E)$ should be close to MSE(β̃_α,I). This “stability” is true in general. To see this recall $Z={\left[\begin{array}{cc}{X}^{\prime }& \sqrt{\alpha }{L}^{\prime }\end{array}\right]}^{\prime }$, (of rank p) and $\underset{\_}{y}={\left[\begin{array}{cc}{y}^{\prime }& 0\end{array}\right]}^{\prime }$. Then another way to represent the estimate (3) is β̃_α,L = Z^†y. Let $E={\left[\begin{array}{cc}{E_1}^{\prime }& {E_2}^{\prime }\end{array}\right]}^{\prime }$ for some n×p and m×p matrices E₁ and E₂. Set Z_E = Z + E and denote the perturbed estimate by ${\stackrel{\sim }{\beta }}_{\alpha ,L}^E=Z_E^{†}y$. By continuity of the generalized inverse (e.g., [4], Section 1.4), ${lim}_{\left|\right|E\left|\right|\to 0}{Z}_E^{†}=Z^{†}$ if and only if lim_||E||→0 rank(Z_E) = rank(Z). Therefore, provided the rank of Z is not changed by E,

$$\underset{\left|\right|E\left|\right|\to 0}{lim}\left|\right|{\stackrel{\sim }{\beta }}_{\alpha ,L}-{\stackrel{\sim }{\beta }}_{\alpha ,L}^E\left|\right|\le \underset{\left|\right|E\left|\right|\to 0}{lim}\left|\right|Z^{†}-Z_E^{†}\left|\right|\left|\right|y\left|\right|=0,$$

and hence $\text{MSE}({\stackrel{\sim }{\beta }}_{\alpha ,L}^E)\to \text{MSE}({\stackrel{\sim }{\beta }}_{\alpha ,L})$ as ||E|| → 0. A more specific bound on the difference of estimates can be obtained under the condition ||Z^†||||E||< 1 which implies that $\left|\right|Z_E^{†}\left|\right|<\frac{\left|\right|Z^{†}\left|\right|}{1-\left|\right|Z^{†}\left|\right|\left|\right|E\left|\right|}$. This can be used to obtain the following bound.

Proposition 5.6

Assume ||Z^†||||E|| < 1 and let r = y − Zβ̃_α,L. Then

$$\left|\right|{\stackrel{\sim }{\beta }}_{\alpha ,L}-{\stackrel{\sim }{\beta }}_{\alpha ,L}^E\left|\right|\le \frac{\left|\right|Z^{†}\left|\right|\left|\right|E\left|\right|}{1-\left|\right|Z^{†}\left|\right|\left|\right|E\left|\right|}\left(\left|\right|{\stackrel{\sim }{\beta }}_{\alpha ,L}\left|\right|+\left|\right|Z^{†}\left|\right|\left|\right|r\left|\right|\right).$$

See [4] and [18].

6. Tuning parameter selection

Despite our focus on the GSVD, the computation of a PEER estimate in (1) does not, of course, require that this decomposition be computed. Rather, the role of the GSVD has been to provide analytical insight into the role a penalty operator plays in the estimation process. For computation, on the other hand, we have chosen to use a method in which the tuning parameter, α, is estimated as part of the coefficient-function estimation process.

Because the choice of tuning parameter is so important, many selection criteria have been proposed, including generalized cross-validation (GCV) [9], AIC and its finite sample corrections [55]. As an alternative to GCV and AIC, a recently-proven equivalence between the penalized least squares estimation and a linear mixed model (LMM) representation [6] can be used. In particular, the best linear unbiased predictor (BLUP) of the response y is composed of the best linear unbiased estimator of the fixed effects and BLUP of the random effects for the given values of the random component variances (see [47] and [6]). Within the LMM framework, restricted maximum likelihood (REML) can be used to estimate the variance components and thus the choice of the tuning parameter, α, which is equal to the ratio of the error variance and the random effects variance [42]. REML-based estimation of the tuning parameter has been shown to perform at least as well as the other criteria and, under certain conditions, it seen to be less variable than GCV-based estimation [41]. In our case, the penalized least-squares criterion (1) is equivalent to

$${\stackrel{\sim }{\beta }}_{\alpha ,L}=\underset{\beta }{argmin}\{{\left|\right|y-X_{\mathit{unp}}{\beta }_{\mathit{unp}}-X_{\mathit{pen}}{\beta }_{\mathit{pen}}\left|\right|}^2+\alpha {\left|\right|L\beta \left|\right|}^2\}$$ (20)

where $\beta ={[{\beta }_{\mathit{unp}}^{\prime }{\beta }_{\mathit{pen}}^{\prime }]}^{\prime }$, the X_unp corresponds to the unpenalized part of the design matrix, and X_pen to the penalized part.

For simplicity of presentation, we describe the transformation with an invertible L. However, a generalized inverse can be used in case L is not of full rank; see equation (14). Also, to facilitate a straightforward use of existing linear mixed model routines in widely available software packages (e.g., R [37] or SAS software [43]), we transform the coefficient vector β using the inverse of the matrix L. Let X^★ = XL⁻¹ and β^★ = Lβ. Then equation (20) can be modified as follows

$${\stackrel{\sim }{\beta }}_{\alpha ,L}^{★}=\underset{\beta }{argmin}\{{\left|\right|y-X^{★}{\beta }^{★}\left|\right|}^2+\alpha {\left|\right|{\beta }^{★}\left|\right|}^2\}.$$

This REML-based estimation of tuning parameters is used in the application of Section 7.3.

For estimation of the parameters a, b and α involved in the decomposition-based penalty of equation (16), we view a and b as weights in a tradeoff between the subspaces and assume ab = const. In the current implementation, we use REML to estimate α for a fixed value of a, and do a grid search over the a values to jointly select the tuning parameters which maximize the REML criterion.

7. Numerical examples

To illustrate algebraic properties given in Section 5, we consider PEER estimation alongside some familiar methods in several numerical examples. Section 7.1 elaborates on the simple example in Section 2.1. These mass spectrometry-like predictors are mathematically synthesized in a manner similar to the study of Reiss and Ogden [40] (see also a numerical study in [48]). Here, β is also synthesized to represent a spectrum, or specific set of bumps. In contrast, Section 7.3 presents a real application to Raman spectroscopy data in which a set of spectra {x_i} and nanoparticle concentrations {y_i} are obtained from sets of laboratory mixtures. This laboratory-based application is preceded in section 7.2 by a simulation that uses these same Raman spectra. In both Raman examples, targeted penalties (13) are defined using discretized functions q_j chosen to span specific subspaces, $\mathcal{Q}=\text{span}{\{q_j\}}_{j=1}^d$. As before, let Q = col[q₁, …, q_d] and = QQ^†.

Each section displays the results from several methods, including derivative-based penalties. Implementing these requires a choice of discretization scheme and boundary conditions which define the operator. We use where = [d_i,j] is a square matrix with entries d_i,i = 1, d_i,i+1 = −1 and d_i,j = 0 otherwise. In addition to some standard estimates, sections 7.3 and 7.2 also consider FPCR_R, a functional PCR estimate described in [40]. This approach extends the penalized B-spline estimates of [8] and assumes β = Bη where B is an p × K matrix whose columns consist of K B-spline functions and η is a vector of B-spline coefficients. The estimation process takes place in the coefficient space using the penalty L = applied to η. The FPCR_R estimate further assumes β = BV_d η (V_d as defined in section 2).

Estimation error is defined as mean squared error (MSE) ||β − β̃_α,L||², and the prediction error defined similarly as Σ_i |y_i − ỹ_i|², where ỹ_i = 〈x_i, β̃〉. Each simulation incorporates response random errors, ${\epsilon }_i\sim N(0,{\sigma }_{\epsilon }^2)$, added to the ith true response, $y_i^{\text{true}}=\langle x_i,\beta \rangle$. Letting $S_Y^2$ denote the sample variance in the set ${\{y_i^{\text{true}}\}}_{i=1}^n$, the response random errors, ε_i, are chosen such that $R^2:=S_Y^2/[S_Y^2+{\sigma }_{\epsilon }^2]$(the squared multiple correlation coefficient of the true model) takes values 0.6 and 0.8. In sections 7.1 and 7.2, tuning parameters are chosen by a grid search. In section 7.3, tuning parameters are chosen using REML, as described in section 6.

7.1. Bumps simulation

Here we elaborate on the simple example of section 2.1. This simulation involves bumpy predictor curves x_i(t) with a response y_i that depends on the amplitudes x_i(t) at some of the bump locations, t = c_k, via the regression function β. In particular,

$$x_i(t)=\sum _{j\in J_X}{a}_{ij}exp[-b_j(t-c_j)]+e_i(t),\beta (t)=\sum _{j\in J_{\beta }}{a}_{j}exp[-b_j(t-c_j)],$$

for t ∈ [0, 1], where J_X = {2, 6, 10, 14, 20, 26, 30} and J_β = {6, 14, 26}; a_★ are magnitudes, b_★ are spreads, and c_★ are the locations of the bumps. In the first simulation, we set b_j = 10000 and c_j = 0.004(8j − 1), the same for each curve x_i. This mimics, for instance, curves seen in mass spectrometry data. The assumption J_β ⊂ J_X simulates a setting in which the response is associated with a subset of metabolite or protein features in a collection of spectra. The a_ij’s are from a uniform distribution, and a_j = 3, 5, 2 for j = 6, 14, 24, respectively. We consider discretized curves, x_i(t), evaluated at p = 250 points, t_j, j = 1, …, p. The sample size is fixed at n = 50 in each case.

Penalties

We consider a variety of estimation procedures: ridge (L = I), second-derivative ( ), a more general derivative operator ( + a I) and PCR. We also define two decomposition-based penalties (13) formed by specific sub-spaces = span{q_j}_j∈J for q_j of the form q_j(t) = a_j exp[b_j(t−c_j)], with c_j at all locations seen in the predictors, J_V = {2, 6, 10, 14, 20, 26, 30}, or at uniformly-spaced locations, J_U = {2, 4, …, 30}; denote these penalties by L_V and L_U, respectively.

Simulation results

The simulation incorporates two sources of noise: (i) response random errors, ${\epsilon }_i\sim N(0,{\sigma }_{\epsilon }^2)$, added to the ith true response so that R² = 0.6, 0.8; (ii) measurement error, $e_i\sim N_p(0,{\sigma }_e^{2}I)$, added to the ith predictor, x_i. To define a signal-to-noise ratio, S/N, set $S_i^2:={\left|\right|x_i-{\mu }_i\left|\right|}^2/(p-1)$, where μ_i is the mean value of x_i, and set $S_X^2:=1/n{\sum }_i{S}_i^2$. The e_i are chosen so that S/N: = S_X/σ_e = 2, 5, 10.

Figure 1 shows a few partial sums of (7) for estimates arising from three penalties: , L = I and L_V, when R² = 0.8 and S/N = 2. Table 1 gives a summary of estimation errors. The penalty L_V, exploiting known structure, performs well in terms of estimation error. Not surprisingly, a penalty that encourages low-frequency singular vectors, , is a poor choice although + a I easily improves on since the GSVs are more compatible with the relevant structure. PCR performs well with estimation errors that can be several times smaller than those of ridge. The number of terms used in PCR ranges here from 8 (S/N = 10) to 25 (S/N = 2).

Predictably, PCR performance degrades with decreasing S/N, a property that is less pronounced, or not shared, by other estimates. Performances of L_V and L_U illustrate properties described in Section 5.3. As S/N → 0, the ordinary singular vectors of X (on which ridge and PCR rely) decreasingly represent the structure in β. The GS vectors of (X, L_V) and (X, L_U), however, retain structure relevant for representing β.

Table 2 summarizes prediction errors. When S/N is large, performance of PCR is comparable with L_V and L_U, but degrades for low S/N. Here, even +a I provides smaller prediction errors, in most cases, than ridge, or PCR. This illustrates the GS vectors role in (12) and reiterates observations in [14].

Table 2.

Prediction errors for simulation with selected bump locations. Sample size is n = 50. Errors are multiplied by 1000 for display

R2	S/N	LV	LU	PCR	ridge		+ a I
0.8	10	9.0	10.5	10.8	16.6	19.3	12.9
0.8	5	8.4	11.0	12.2	26.7	27.9	17.8
0.8	2	12.9	19.0	53.2	55.7	50.3	40.1
0.6	10	21.4	23.0	23.9	33.0	39.0	26.2
0.6	5	23.9	25.0	29.5	49.2	54.6	34.4
0.6	2	34.4	42.5	90.4	110.4	104.4	77.9

7.2. Raman simulation

We consider Raman spectroscopy curves which represent a vibrational response of laser-excited co-organic/inorganic nanoparticles (COINs). Each COIN has a unique signature spectrum and serves as a sensitive nanotag for immunoassays; see [27, 44]. Each spectrum consists of absorbance values measured at p = 600 wavenumbers. By the Beer-Lambert law, light absorbance increases linearly with a COIN’s concentration and so a spectrum from a mixture of COINs is reasonably modeled by a linear combination of pure COIN spectra. The data here come from experiments that were designed to establish the ability of these COINs to measure the existence and abundance of antigens in single-cell assays.

Let P₁, …, P₁₀ denote spectra from nine pure COINs and one “blank” (no biochemical material), each normalized to norm one. We form in-silico mixtures as follows: $x_i={\sum }_{k=1}^{10}{c}_{i,k}{P}_k,i=1,\dots ,n$, i = 1, …, n, with coefficients {c_i,k} generated from a uniform distribution. Figure 2 shows representative spectra from all nine COINs superimposed on a collection of mixture spectra, ${\{x_i\}}_{i=1}^{50}$. Included in Figure 2 is the β (dashed curve) used to defined the simulation: y_i = 〈x_i, β〉 + ε, ε ~ N(0, σ²).

In this simulation, we have created a coefficient function which, instead of being modeled mathematically, is a curve that exhibits structure of the type found in Raman spectra. Details on the construction of this β are in Appendix 9.1 so here we simply note that it arises as a ridge estimate from a set of in-silico mixtures of Raman spectra in which one COIN, P₉, is varied prominently relative to the others. See Figure 2. Motivation for defining β in this way is based on a view that it seems implausible for us to predict the structure of realistic signal in these data and recreate it using polynomials, Gaussians or other analytic functions.

Regardless of its construction, β defines signal that allows us to compute estimation and prediction error. The performances of five methods are summarized in Table 3. Note that although β was constructed as a ridge estimate (using a different set of in-silico mixtures; see Appendix 9.1), the ridge penalty is not necessarily optimal for recovering β. This is because the strictly empirical eigenvectors associated with the new spectra may contain structure not informative regarding y. Also, in these data, the performance of FPCR_R is adversely affected by a tendency for the estimate to be smooth; cf., Figure 3. The PEER penalty used here is defined by a decomposition-based operator (16) in which is spanned by a 10-dimensional set of pure-COIN spectra (including a blank). The success of such an estimate obviously depends on an informed formation of , but as long as the parameter-selection procedure allows for a = b, then the set of possible estimates includes ridge as well as estimates with potentially lower MSE than ridge; see Proposition 5.4.

Table 3.

Estimation (MSE) and prediction (PE) errors of several penalization methods for the simulation described in Figure 2. Numbers represent the average error from 100 runs. PE errors are multiplied by 1000 for display

	LQ	PCR	ridge	+ a I	FPCRR
MSE	8.91	12.34	13.87	41.69	15.29
PE	0.0071	0.0179	0.0139	0.0131	0.0175

Fig 3.

Three estimates for a coefficient function that relates concentrations of P₁ to its signal in 8-COIN laboratory mixtures. Estimates shown: ridge (β̃_ridge; gray), FPCR_R (β̃_FPCRR; black) and PEER (β̃_Q; blue). For perspective, P₁ is plotted (in red) and the mean-adjusted PEER estimate, β̃_Q + μ (dashed blue); μ is the mean of the mixture spectra ${\{x_i\}}_{i=1}^{36}$ (not shown).

We note that this simulation may be viewed as inherently unfair since the PEER estimate uses knowledge about the relevant structure. However, this is a point worth reemphasizing: when prior knowledge about the structure of the data is available, it can be incorporated naturally into the regression problem.

7.3. Raman application

We now consider spectra representing true antibody-conjugated COINs from nine laboratory mixtures. These mixtures contain various concentrations of eight COINs (of the nine shown in Figure 2). Spectra from four technical replicates in each mixture are included to create a set of n = 36 spectra ${\{x_i\}}_{i=1}^n$. We designate P₁ as the COIN whose concentration within each mixture defines y. Assuming a linear relationship between the spectra, {x_i}, and the P₁-concentrations, {y_i}, we estimate P₁. More precisely, we estimate the structure in P₁ that correlates most with its concentrations, as manifest in this set of mixtures. The fLM is a simplistic model of this relationship between the concentration of P₁ and its functional structure, but the physics of this technology imply it is a reasonable starting point.

We present the results of three estimation methods: ridge, FPCR_R and PEER. In constructing a PEER penalty, we note that the informative structure in Raman spectra is not that of low-frequency or other easily modeled features, but it may be obtainable experimentally. Therefore, we define L as in (13) in which contains the span of COIN template spectra: ${\mathcal{Q}}_1=\text{span}{\{P_k\}}_{k=1}^8$. However, since a single set of templates may not faithfully represent signal in subsequent experiments (with new measurement errors, background and baseline noise etc), we enlarge by adding additional structure related to these templates. For this, set ${\mathcal{Q}}_2=\text{span}{\{P_k^{\prime },P_k^{″}\}}_{k=1}^8$, where $P_k^{\prime }$ denotes the derivative of spectrum P_k. (Note, to form , scale-based approximations to these derivatives are used since raw differencing of non-smooth spectra introduces noise.) Then set = span{ ∪ } and define L = a(I − ) + b .

The regularization parameters in the PEER and ridge estimation processes were chosen using REML, as described in Section 6. For the FPCR_R estimate, we used the R-package refund [39] as implemented in [40].

Since β is not known (the model y = Xβ +ε is only approximate), we cannot report MSEs for these three methods. However, the structure of P₁ is qualitatively known and by experimental design, y is directly associated with P₁. The goal here is that of extracting structure of the constituent spectral components as manifest in a linear model. This application is similar to the classic problem of multivariate calibration [5, 31] which essentially leads to a regression model using an experimentally-designed set of spectra from laboratory mixes.

The structure in the estimate here is expected to reflect the structure in P₁ that is correlated with P₁’s concentrations, y. The estimate is not, however, expected to precisely reconstruct P₁ since P₁ shares structure with the other COIN spectra not associated with y. See Figure 2 where P₁ is plotted alongside the other COIN spectra. Now, Figure 3 shows plots of the PEER, FPCR_R and ridge estimates of the fLM coefficient function. The PEER estimate, β̃_Q, provides an interpretable compromise between ridge, which involves no smoothing, and FPCR_R, which appears to oversmooth. For reference, the P₁ spectrum is also plotted along with a mean-adjusted version of β̃_Q, β̃_Q + μ (dashed line), where μ(t) = (1/36) Σ_i x_i(t), t ∈ [400, 1800].

Finally, we consider prediction for these methods by forming a new set of spectra from different mixture compositions (different concentrations of each COIN) and, additionally, taken from different batches. This “test” set consists of spectra from four technical replicates in each of 15 mixtures forming a set of n = 60 spectra, ${\{x_i^{\text{test}}\}}_{i=1}^n$. As before, P₁ is the COIN whose concentration within each mixture defines the values ${\{y_i^{\text{test}}\}}_{i=1}^n$. For the estimates from each of the three methods (shown in Figure 3) we compute the prediction error: $(1/n){\sum }_i{(y_i^{\text{test}}-\langle x_i^{\text{test}},\stackrel{\sim }{\beta }\rangle )}^2$. The errors for PEER, ridge, and FPCR_R estimates are 0.770, 0.752, 2.139, respectively. The ridge estimate here illustrates how low prediction error is not necessarily accompanied by interpretable structure in the estimate (or low MSE) [7].

8. Discussion

As high-dimensional regression problems become more common, methods that exploit a priori information are increasingly popular. In this regard, many approaches to penalized regression are now founded on the idea of “structured” penalties which impose constraints based on prior knowledge about the problem’s scientific setting. There are many ways in which such constraints may be imposed, and we have focused on the algebraic aspects of a penalization process that imposes spatial structure directly into a regularized estimation. This approach fits into the classic framework of L²-penalized regression but with an emphasis on the algebraic role that a penalty operator plays to impart structure on the estimate.

The interplay between a structured regularization term and the coefficient-function estimate may not be well understood in part because it is not typically viewed in terms of the generalized singular vectors/values, which is fundamental to this investigation. In particular, any penalized estimate of the form (1) with L ≠ I is intrinsically based on GSVD factors in the same way that many common regression methods (such as PCR, ridge, James-Stein, or partial least squares) are intrinsically based on SVD factors. Just as the basics of the ubiquitous SVD are important to understanding these methods, we have aspired to established the basics of the GSVD as it applies to a this general penalized regression setting and to illustrate how the theory underlying this approach can be used inform the choice of penalty operator.

Toward this goal the presentation emphasizes the transparency provided by the partially-empirical eigenvector expansion (7). Properties of the estimate’s variance and bias are determined explicitly by the generalized singular vectors whose structure is determined by the penalty operator. We have restricted attention to additive constraints defined by penalty operators on L² in order to retain the direct algebraic connection between the eigenproperties of the operator pair (X, L) and the spatial structure of β̃_α,L. Intuitively, the structure of the penalty’s least-dominant singular vectors should be commensurate with the informative structure of β. The actual effect a penalty has on the properties of the estimate can be quantified in terms of the GSVD vectors/values.

This perspective differs from popular two-stage signal regression methods in which estimation is either preceded by fitting the predictors to a set of (external) basis functions or is followed by a step that smooths the estimate [8, 21, 30, 38, 40]. Instead, structure (smoothness or otherwise) is imposed directly into the estimation process. The implementation of a penalty that incorporates structure less generic than smoothness (or sparseness) requires some qualitative knowledge about spatial structure that is informative. Clearly this is not possible in all situations, but our presentation has focused on how a functional linear model may provide a rigorous and analytically tractable way to take advantage of such knowledge when it exists.

9. Appendix

9.1. Defining β for the simulation in Section 7.2

This simulation is motivated by an interest in constructing a plausibly realistic β whose structure is naturally derived by the scientific setting involving Raman signatures of nanoparticles. Although one could model a β mathematically using, say, polynomials or Gaussian bumps (cf., Appendix A.2), such a simulation would be detached from the physical nature of this problem. Instead, we construct a coefficient function that genuinely comes from a functional linear model with Raman spectra as predictors.

We first generate in-silico mixtures of COIN spectra as $x_i^o={\sum }_{k=1}^9{c}_{i,k}{P}_k,i=1,\dots ,50$, where c_i,k ~ unif[0, 1]. Designating P₉ as the COIN of interest, we define response values that correspond to the “concentration” of P₉ by setting $y_i^o:=3c_{i,9}$, i = 1, …, n. The factor of 3 imposes a strong association between P₉ and the response.

Now, the example in section 7.2 aims to estimate a coefficient function that truly comes from a solution to a linear model. However, the equation y^o = X^oβ has infinitely many solutions (where X^o is the matrix whose ith row is $x_i^o$), so we must we must regularize the problem to obtain a specific β. For this, we simply use a ridge penalty and designate the resulting solution to be β. This is shown by the dotted curve in Figure 2 and is qualitatively similar to P₉.

We note that the simulation in section 7.2 uses the same set of COINs, but a new set of in-silico mixture spectra (i.e., a new set of {c_i,k} ~ Unif[0, 1]). In addition, a small amount of measurement error was added, as in section 7.1, to each spectrum during the simulation.

9.2. Frequency domain simulation

We display results from a study that mimics the scenario of simulations studied by Hall and Horowitz [16]. We illustrate, in particular, properties of the MSE discussed following equation (18) in section 5.3 relating to b = 0. In fact, we consider the more general scenario in which is not constructed from empirical eigenvectors (as in PCR and ridge), but is defined by a prespecified envelope of frequencies.

In this simulation both β and x_i, i = 1, …, n, are generated as sums of the cosine functions

$$x_i(t)=\sum _{j=1}^{40}{\gamma }_j{Z}_{ij}{\phi }_j(t)+e_i(t),\beta (t)=0.75{\phi }_5(t)+1.5{\phi }_{11}(t)+1{\phi }_{17}(t),$$

t ∈ [0, 1]; here γ_j = (−1)^j+1j^−0.75, Z_ij is uniformly distributed on [−3^1/2, 3^1/2] (E(Z_ij) = 0 and var(Z_ij) = 1), φ₁ ≡ 1 and φ_j(t) = 2^1/2 cos(jπt) for j ≥ 1, and $e_i(t)\sim N(0,{\sigma }_X^2)$, and cov(e_i(t), e_i′ (t′)) = 0 for either i ≠ i′ or t ≠ t′. The response y_i is defined as y_i = 〈β, x_i〉 + ε_i, where ε_i ~ N (0σ²), i.i.d.. The simulations involve discretizations of these curves evaluated at p = 100 equally spaced time points, t_j, j = 1, …, p, that are common to all curves.

Penalties

We consider properties of estimates from a variety of penalties: ridge (L = I), , + aI, and PCR¹. In addition, targeted penalties of the form L = I − , are defined by the specified subspaces = span{φ_j}_j∈J, for φ_j defined above. Specifically, we use J = J_F = {j = 5, …, 17} (a tight envelope of frequencies) to define L_F, and J = J_G = {j = 4, …, 20} (a less focused span of frequencies) to define L_G. The operator + aI simply serves to illustrate the role of higher-frequency singular vectors as discussed in Section 4.1. In the simulations, the coefficient a in + aI was chosen simultaneously with α via a two-dimensional grid search.

Simulation results

Table 4 summarizes estimation results for all six penalties and two sample sizes, n = 50, 200. The prediction results for these estimates are in Table 5. These are reported for S/N = 10, 5 and R² = 0.8, 0.6. The number of terms in the PCR estimate was optimized and ranged from 19 to 25 when R² = 0.8 and decreased with decreasing R². Analogously, one could optimize over the dimension of (to implement a truncated GSVD), but the purpose here is illustrative while in practice a more robust approach would emply a penalty of the form (13).

Table 5.

Prediction errors for the simulation with localized frequencies

n	R2	S/N	LF	LG	PCR	ridge		+ a I
50	0.8	10	0.848	1.134	1.490	1.423	1.292	1.246
50	0.8	5	0.840	1.124	1.427	1.390	1.304	1.222
200	0.8	10	0.200	0.273	0.432	0.497	0.444	0.418
200	0.8	5	0.211	0.276	0.460	0.547	0.466	0.455
50	0.6	10	2.165	2.900	3.051	2.705	2.621	2.472
50	0.6	5	2.171	2.832	3.158	2.938	2.912	2.724
200	0.6	10	0.621	0.801	1.058	1.160	1.044	0.990
200	0.6	5	0.584	0.766	1.062	1.186	1.069	1.014

Errors obtained with ridge and PCR are small, as expected, since the structure of β in this example is consistent with the structure represented in the singular vectors, v_k. Therefore, even though the relationship between the y_i and x_i degrades (indeed, even as R² → 0), these estimates are comprised of vectors that generally capture structure in β since it is strongly represented by the dominant eigenstructure of X. The second-derivative penalty, , produces the worst estimate in each of the scenarios due to oversmoothing. Note + a I improves on , yet it is still not optimal for the range of frequencies in β.

Regarding L_G, the MSE gets worse as S/N increases. Indeed, here is fixed and relatively large and since the σ_k decay faster when S/N is big, this leads to rank deficiency and large variance; see equation (18) (note, this only applies approximately since does not consist of ordinary SVs). In our previous examples, this is stabilized by a b > 0.

The problems of estimation and prediction have different properties [7]; good prediction may be obtained even with a poor estimate, as seen in Table 5. The estimate from L_Da is generally poor relative to others (as measured by the L²-norm), but its prediction error is comparable with other methods and is best among the non-targeted penalization methods. This is consistent with the outcome described by C. Goutis [14] where (derivatives of) the predictor curves contain sharp features and so standard least-squares regularization (OLS, PCR, ridge, etc.) perform worse than a PEER estimate which imposes a greater emphasis on “regularly oscillatory but not smooth components”; see section 4.1.