High-dimensional partial linear model with trend filtering

Abstract

Understanding the links between diet, metabolic changes, and health outcomes is a key focus in nutritional science and broader biological research. Analyzing relationships, such as those between ultra-processed food (UPF) intake and metabolites, offers insights into potential biomarkers for diet-related diseases and public health applications. However, these analyses are challenging due to high-dimensional data structures and complex, often nonlinear associations between covariates and health outcomes. Traditional linear models and conventional nonparametric methods often lack the flexibility to accurately capture such complexities in biological data. To address these challenges, we propose a high-dimensional partial linear regression model that captures both linear and nonlinear effects, combining the interpretability of linear models with the adaptability of nonparametric approaches. Our model leverages trend filtering to handle local smoothness variations effectively and achieves minimax optimal rates, making it suitable for complex biological datasets. We apply this model to data from the Interactive Diet and Activity Tracking in AARP (IDATA) Study, demonstrating its utility in identifying biomarkers associated with UPF intake and illustrating its potential for broader applications in dietary, metabolic, and health-related research.

1. Introduction

Ultra-processed food (UPF), characterized by high levels of synthetic additives, unhealthy fats, and added sugars, is associated with negative health impacts, including obesity and metabolic disorders. Investigating these relationships helps identify biomarkers that reflect dietary effects and guide strategies for improving public health and nutrition.

Studying this relationship is challenging due to the high dimensionality of metabolomic data. Accounting for key confounding variables, such as age (Fu, Chen and Liu, 2024) and obesity-related measurements (Canhada et al., 2020), adds further complexity, as these factors may exhibit nonlinear associations with UPF intake. Similar challenges arise in other fields; for instance, lung cancer research faces similar challenges when associating mutation signatures with DNA methylation, where nonlinear relationships between factors like smoking history or tumor purity and mutation signatures add further complexity (Zhang et al., 2024).

Traditional linear models are often limited in capturing nonlinear associations, as they are designed to model only linear relationships. While nonparametric smoothing methods, such as splines and local polynomial smoothing, can accommodate certain nonlinear relationships, they may lack the flexibility necessary to capture more intricate, highly heterogeneous associations between predictors and responses, ultimately resulting in suboptimal predictive performance. These limitations underscore the need for an innovative modeling approach capable of flexibly capturing such complex associations.

In this paper, we propose a high-dimensional partial linear regression model in which the response variable $y$ is modeled as having a linear relationship with a high-dimensional vector of predictors $\mathbf{x}\in {\mathbb{R}}^p$ and a nonlinear relationship with a one-dimensional predictor $\mathbf{z}\in \mathbb{R}$. More specifically, we consider the following partial linear model:

$$y={\mathbf{x}}^{\prime }{\mathit{\beta }}^0+g_0(\mathbf{z})+\epsilon ,$$

where ${\mathit{\beta }}^0$ is a sparse $p$-dimensional vector of coefficients, $g_0(\cdot )$ is an unknown nonparametric function, $\mathbf{z}$ is a univariate predictor chosen based on prior knowledge or structure selection methods (Zhang, Cheng and Liu, 2011; Huang, Wei and Ma, 2012; Lian, Liang and Ruppert, 2015), and $\epsilon$ is a zero-mean error term.

Partial linear models have been widely used in various fields due to their flexibility in modeling complex relationships by combining the interpretability of linear models with the adaptability of nonparametric components. However, their traditional application has largely been confined to settings where the dimension of ${\mathit{\beta }}^0$ is small or fixed relative to $n$; see, for instance, Engle et al. (1986); Chen (1988); Wahba (1990); Mammen and van de Geer (1997); Härdle, Liang and Gao (2000); Bunea (2004); Xie and Huang (2009). For high-dimensional $p$, partial linear models have also been extended (Müller and Van de Geer, 2015; Ma and Huang, 2016; Zhu, 2017; Yu, Levine and Cheng, 2019; Zhu, Yu and Cheng, 2019; Lv and Lian, 2022; Fu, Huang and Yao, 2024), but these extensions are limited by the assumption that $g_0$ lies within a smooth function class (e.g., Sobolev or Hölder). These methods commonly apply penalized regression approaches, such as LASSO (Tibshirani, 1996) or SCAD (Fan and Li, 2001), to estimate the sparse vector ${\mathit{\beta }}^0$ and use nonparametric smoothing methods, such as splines or local polynomial smoothing, for $g_0$. However, the assumption that $g_0$ belongs to a smooth function class may impose overly restrictive conditions in practical settings, as $g_0$ may exhibit heterogeneous smoothness.

Similar to Müller and Van de Geer (2015) and Yu, Levine and Cheng (2019), our model accommodates cases where $p$ is comparable to or larger than the sample size $n$, but unlike their approach, we do not require $g_0$ to belong to a smooth function class. We assume that $g_0$ belongs to the following function class:

$${𝒱}_k(C)≔\left\{g_0:[0,1]\to \mathbb{R}:\text{TV}(g_0^{(k)})\le C\right\},$$ (1)

where $\text{TV}(\cdot )$ denotes the total variation operator, $g_0^{(k)}$ represents the $k$-th weak derivative of $g_0$, and $C>0$ is a constant. The function class ${𝒱}_k(C)$ allows for heterogeneous smoothness of $g_0$, providing greater flexibility in capturing local behavior than traditional Sobolev or Hölder classes.

To estimate $g_0$, we utilize trend filtering (Steidl, Didas and Neumann, 2006; Kim et al., 2009), which extends the idea of locally adaptive regression splines (Mammen and Van De Geer, 1997). Trend filtering estimates $g_0$ by minimizing an objective function regularized with a total variation penalty. In the univariate setting, the trend filtering estimator is given by

$$\underset{g\in {\mathscr{H}}_n^k}{\text{min}}\frac{1}{2}\sum _{i=1}^n{\left(y_i-g(z_i)\right)}^2+\gamma \text{TV}(g^{(k)}),$$ (2)

where $\gamma \ge 0$ is a tuning parameter and ${\mathscr{H}}_n^k$ is an $n$-dimensional space spanned by spline-like basis functions known as falling factorial basis (see Section 2.1 for details). This method adapts to local smoothness variations more effectively than traditional smoothers and achieves minimax optimal rates over ${𝒱}_k(C)$ (Donoho and Johnstone, 1998; Tibshirani, 2014). Unlike linear smoothers, such as local polynomials or splines, which struggle to adapt to changes in smoothness, trend filtering provides a robust solution for estimating functions with varying smoothness levels.

Recent advancements in trend filtering have significantly expanded its range of applications, including univariate nonparametric regression under strong sparsity (Guntuboyina et al., 2020; Ortelli and van de Geer, 2021), graph trend filtering (Wang et al., 2016; Madrid Padilla et al., 2020), functional trend filtering (Wakayama and Sugasawa, 2023), scalar-on-image regression models (Wang, Zhu and Initiative, 2017), additive models (Sadhanala and Tibshirani, 2019; Petersen and Witten, 2019; Tan and Zhang, 2019), quantile regression models (Madrid Padilla and Chatterjee, 2022), and spatiotemporal models (Madrid Padilla, Madrid Padilla and Wang, 2023; Rahardiantoro and Sakamoto, 2024), among others and with further references therein. Note that while Petersen and Witten (2019) emphasizes practical and algorithmic advances for an additive modeling approach that adaptively selects variables, linearity, and knot locations via trend filtering, our work instead provides a comprehensive theoretical framework under a partial linear model. In particular, we assume a known nonparametric component to accommodate highly heterogeneous smoothness, offering a distinct perspective compared to purely additive methods.

In this paper, we extend trend filtering to the high-dimensional partial linear model setting. The partial linear model integrates parametric and nonparametric components, presenting unique estimation challenges. Jointly estimating these components requires carefully balancing bias and variance to avoid overfitting. These complexities are especially significant in the context of trend filtering, affecting both theoretical analysis and practical implementation. From a theoretical standpoint, we prove that our estimate for ${\mathit{\beta }}^0$ attains the oracle rate $(s\text{log}p)/n$ as if $g_0$ were known, and the convergence rate for $g_0$ exhibits a phase transition between $(s\text{log}p)/n$ and the optimal nonparametric rate $n^{-(2k+2)/(2k+3)}$. This dual-rate result highlights the unique adaptive benefits of partial linear trend filtering, distinguishing it from existing partial linear estimators that do not incorporate trend filtering or from trend filtering methods that have not established such optimal rate results. Notably, the conditions we impose are more relaxed than, or at least comparable to, those typically required in existing partial linear model frameworks (Müller and Van de Geer, 2015; Yu, Levine and Cheng, 2019). For implementation, we develop a blockwise coordinate descent algorithm to enable efficient computation and have made the method accessible through an accompanying R package.

We apply the proposed model to examine the association between UPF intake and high-dimensional metabolite profiles, accounting for potential nonlinear and non-smooth effects of age, BMI, hip circumference, and waist circumference. The dataset originates from the Interactive Diet and Activity Tracking in AARP (IDATA) Study, conducted by the National Cancer Institute (NCI). This study was specifically designed to support research on dietary intake, nutrition, and cancer prevention (Subar et al., 2020).

Building on this dataset, our method utilizes high-dimensional feature selection within a partial linear model framework, incorporating trend filtering to enhance the detection of biomarkers associated with UPF intake. By improving prediction accuracy, this approach not only identifies meaningful metabolites but also contributes to the development of evidence-based public health guidelines and dietary recommendations.

The remainder of the paper is organized as follows. Section 2 introduces the proposed method and examines its statistical properties. In Section 3, we present a comprehensive numerical simulation study, and in Section 4, we analyze the relationship between UPF intake and metabolite profiles, accounting for nonlinear confounding effects. Section 5 concludes with remarks on potential extensions.

1.1. Notations

We begin by introducing the notation that will be used throughout the paper. For $\mathbf{a}={\left(a_1,\dots ,a_p\right)}^{\prime }\in {\mathbb{R}}^p$, denote $\Vert \mathbf{a}{\Vert }_q={\left(\sum _{i=1}^p{\left|a_i\right|}^q\right)}^{\frac{1}{q}}$ for $q\in [1,\infty )$ and $\Vert \mathbf{a}{\Vert }_{\infty }={\text{max}}_{1\le i\le p}\left|a_i\right|$. For two vectors $\mathbf{a},\mathbf{b}\in {\mathbb{R}}^n$, we write $\Vert \mathbf{a}{\Vert }_n^2=\frac{1}{n}{\mathbf{a}}^{\prime }\mathbf{a},⟨\mathbf{a},\mathbf{b}{⟩}_n=\frac{1}{n}{\mathbf{a}}^{\prime }\mathbf{b}$. For a vector $\mathbf{a}\in {\mathbb{R}}^n$ and a function $g:\mathbb{R}\to \mathbb{R}$, let $g(\mathbf{a})={\left(g\left(a_1\right),\dots ,g\left(a_n\right)\right)}^{\prime }$. Given a square matrix $\mathbf{A}=(a_{ij})\in {\mathbb{R}}^{p\times p},{\lambda }_{\text{max}}(\mathbf{A})$ and ${\lambda }_{\text{min}}(\mathbf{A})$ represent its largest and smallest eigenvalues respectively. For a general matrix $\mathbf{A}=\left(a_{ij}\right)\in {\mathbb{R}}^{p\times q},\Vert \mathbf{A}{\Vert }_2$ denotes its spectral norm; $\Vert \mathbf{A}{\Vert }_{\text{max}}={\text{max}}_{ij}\left|a_{ij}\right|,\Vert \mathbf{A}{\Vert }_F=\sqrt{\sum _{i,j}{a}_{ij}^2}$. For $a,b\in \mathbb{R},a\wedge b=\text{min}(a,b),a\vee b=\text{max}(a,b)$. For a set $A,{\mathbb{1}}_A(\cdot )$ is the usual indicator function, and $\left|A\right|$ to be its cardinality. Moreover, $a_n\lesssim b_n\left(a_n\gtrsim b_n\right)$ means there exists some constant $C>0$ such that $a_n\le C{b}_n\left(a_n\ge C{b}_n\right)$ for all $n$; thus $a_n\lesssim b_n\left(a_n\gtrsim b_n\right)$ is equivalent to $a_n=O\left(b_n\right)\left(a_n=\Omega \left(b_n\right)\right);a_n\asymp b_n$ if and only if $a_n\lesssim b_n$ and $b_n\gtrsim a_n;a_n\gg b_n$ means $b_n=o\left(a_n\right)$. We put subscript $p$ on $O$ and $o$ for random variables. For i.i.d. samples $\left\{w_1,\dots ,w_n\right\}$ from a distribution $Q$ supported on some space $𝒲$, denote by $Q_n$ the associated empirical distribution. The $L_2(Q)$ and $L_2(Q_n)$ norms for functions $f:𝒲\to \mathbb{R}$ are: $\Vert f{\Vert }_{L_2(Q)}^2={\int }_{𝒲}{f}^2(w)dQ(w),\Vert f{\Vert }_{L_2\left(Q_n\right)}^2=\frac{1}{n}\sum _{i=1}^n{f}^2\left(w_i\right)$. For simplicity we will abbreviate subscripts and write $\Vert f\Vert ,\Vert f{\Vert }_n$ for $\Vert f{\Vert }_{L_2(Q)},\Vert f{\Vert }_{L_2\left(Q_n\right)}$ respectively, whenever $Q$ is the underlying distribution of the covariates. For a random variable $x\in \mathbb{R}$, we also write $\Vert x\Vert$ for $\sqrt{\mathbb{E}{x}^2}$. Given random variables $z_1,z_2,\dots ,z_n$, the order statistics are denoted by $z_{(1)}\le z_{(2)}\le \cdots \le z_{(n)}$. The sub-Gaussian norm of a random variable $x\in \mathbb{R}$ is defined as $\Vert x{\Vert }_{{\psi }_2}=\text{inf}\left\{t>0:\mathbb{E}\text{exp}\left(x^2/t^2\right)\le 2\right\}$.

2. Trend filtering in high-dimensional partial linear models

2.1. Problem setting and the proposed method

We consider the partial linear regression model:

$$y={\mathbf{x}}^{\prime }{\mathit{\beta }}^0+g_0(z)+\epsilon ,$$

where $\epsilon$ is independent of $(\mathbf{x},z)\in {\mathbb{R}}^{p+1},{\mathit{\beta }}^0\in {\mathbb{R}}^p$ has the support $S=\left\{j:{\beta }_j^0\ne 0\right\}$ with $\left|S\right|=s$, and $g_0:[0,1]\to \mathbb{R}$ is a nonparametric function. Without loss of generality, the support of $z$ is assumed as [0, 1]. It can be relaxed to any compact interval. Let ${\left\{\left(y_i,{\mathbf{x}}_i,z_i\right)\right\}}_{i=1}^n$ be $n$ independent observations of $(y,\mathbf{x},z)$, and denote $\mathbf{y}={\left(y_1,\dots ,y_n\right)}^{\prime },\mathbf{X}={\left({\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\right)}^{\prime },\mathbf{z}={\left(z_1,\dots ,z_n\right)}^{\prime }$. We focus on the high-dimensional setting in which the dimension $p$ can be much larger than the sample size $n$, and assume $\text{TV}\left(g_0^{(k)}\right)\le L_g$ with some constant $L_g>0$ to allow for a large degree of heterogeneous smoothness of $g_0$.

Expanding upon the univariate trend filtering (Tibshirani, 2014) discussed in Section 1, we consider the following $k$th degree partial linear trend filtering estimation,

$$(\widehat{\mathit{\beta }},\widehat{g})\in \underset{\mathit{\beta }\in {\mathbb{R}}^p,g\in {\mathscr{H}}_n^k}{\text{arg}\text{min}}\frac{1}{2}\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }-g(\mathbf{z}){\Vert }_n^2+\lambda \Vert \mathit{\beta }{\Vert }_1+\gamma \text{TV}(g^{(k)}),$$ (3)

where $\lambda ,\gamma \ge 0$ are tuning parameters, and ${\mathscr{H}}_n^k$ is the span of the $k$th degree falling factorial basis functions defined over the ordered input points $z_{(1)}<z_{(2)}<\cdots <z_{(n)}$. The set of basis functions take the form (Tibshirani, 2014; Wang, Smola and Tibshirani, 2014; Tibshirani, 2022),

$$q_i(t)=\prod _{l=1}^{i-1}\left(t-z_{(l)}\right),i=1,\dots ,k+1,$$ (4)

$$q_{i+k+1}(t)=\prod _{l=1}^k\left(t-z_{(i+l)}\right)\mathbb{1}\left\{t>z_{(i+k)}\right\},i=1,\dots ,n-k-1,$$ (5)

where we adopt the convention $\prod _{i=1}^0{c}_i=1$. The above falling factorial basis looks similar to the standard truncated power basis for $k$th degree splines with knots at $z_{(k+1)},\dots ,z_{(n-1)}$. In fact, it is straightforward to verify that the two bases are equal when $k=0,1$, and they span different spaces when $k\ge 2$–the falling factorial functions in (5) are piecewise polynomials with discontinuities in their derivatives of orders $1,2,\dots ,k-1$. Define the matrix $\mathbf{Q}\in {\mathbb{R}}^{n\times n}$ with entries $q_{\ell k}=q_{\ell }(z_k),1\le \ell ,k\le n$. Then for any $g\in {\mathscr{H}}_n^k$, we can write $g(\mathbf{z})=\mathbf{Q}\mathit{\alpha }$ for some $\mathit{\alpha }\in {\mathbb{R}}^n$. The estimation in (3) is thus equivalent to

$$(\widehat{\mathit{\beta }},\widehat{\mathit{\alpha }})\in \underset{\mathit{\beta }\in {\mathbb{R}}^p,\mathit{\alpha }\in {\mathbb{R}}^n}{\text{arg}\text{min}}\frac{1}{2}\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }-\mathbf{Q}\mathit{\alpha }{\Vert }_n^2+\lambda \Vert \mathit{\beta }{\Vert }_1+\gamma k!\sum _{l=k+2}^n\left|{\alpha }_l\right|.$$ (6)

Further representing $\mathbf{Q}\mathit{\alpha }=\mathit{\theta }$ and using the formula for ${\mathbf{Q}}^{-1}$ (Wang, Smola and Tibshirani, 2014), we can reformulate the optimization (6) as

$$(\widehat{\mathit{\beta }},\widehat{\mathit{\theta }})\in \underset{\mathit{\beta }\in {\mathbb{R}}^p,\mathit{\theta }\in {\mathbb{R}}^n}{\text{arg}\text{min}}\frac{1}{2}\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }-\mathit{\theta }{\Vert }_n^2+\lambda \Vert \mathit{\beta }{\Vert }_1+\gamma {‖D^{(\mathbf{z},k+1)}\mathit{\theta }‖}_1,$$ (7)

where $D^{(\mathbf{z},k+1)}\in {\mathbb{R}}^{(n-k-1)\times n}$ is the discrete difference operator of order $k+1$. Formulation (7) shares similarity with the method in Drikvandi (2025), where they apply distinct penalties to two separate parameter blocks by treating some covariates as being of interest and others as nuisance. The key difference between their approach and our approach is: (1) Their approach is focused on high-dimensional estimation and inference under linear models, while our approach centers on the estimation under high-dimensional partial linear models; (2) Their approach uses smooth penalties to shrink parameters of interest and control variance, while ours employs a non-smooth penalty to estimate the nonlinear function and achieve local adaptivity (see next paragraph for more details). When $k=0$,

$$D^{(z,1)}=(\begin{array}{llllll}-1\hfill & 1\hfill & 0\hfill & \cdots \hfill & 0\hfill & 0\hfill \\ 0\hfill & -1\hfill & 1\hfill & \cdots \hfill & 0\hfill & 0\hfill \\ ⋮\hfill & \hfill & \hfill & \hfill & \hfill & \hfill \\ 0\hfill & 0\hfill & 0\hfill & \cdots \hfill & -1\hfill & 1\hfill \end{array})\in {\mathbb{R}}^{(n-1)\times n}.$$ (8)

For $k\ge 1$, the difference operator is defined recursively, that is,

$$D^{(\mathbf{z},k+1)}=D^{(\mathbf{z},1)}\cdot \text{diag}\left(\frac{k}{z_{(k+1)}-z_{(1)}},\cdots ,\frac{k}{z_{(n)}-z_{(n-k)}}\right)\cdot D^{(\mathbf{z},k)}.$$

Here, $D^{(\mathbf{z},1)}$ is defined as the form of (8) with the dimension of $(n-k-1)\times (n-k)$. The problem (7) is a generalized LASSO problem. The sparsity and banded structure of the penalty matrix $D^{(\mathbf{z},k+1)}$ provides significant advantages in solving the optimization problem (7). Once $\widehat{\mathit{\theta }}$ is computed from (7), the estimator $\widehat{g}$ in (3) can be obtained as $\widehat{g}(t)=\sum _{\ell =1}^n{\widehat{\alpha }}_{\ell }{q}_{\ell }(t)$ with $\widehat{\mathit{\alpha }}={\mathbf{Q}}^{-1}\widehat{\mathit{\theta }}$.

Our estimation approach in (3) introduces a doubly penalized least squares estimator, similar to those proposed by Müller and Van de Geer (2015) and Yu, Levine and Cheng (2019). This approach involves two penalties: the first shrinkage penalty induces sparsity on the parametric part, and the second smoothness penalty controls the complexity of the nonparametric part. The primary distinction between our approach and theirs lies in the estimation method of the function $g_0$. While both of their methods employ the smoothing spline technique with a squared ${\ell }_2$ penalty, our method utilizes trend filtering based on an ${\ell }_1$ type penalty which can achieve a finer degree of local adaptivity. Given the structure of the partial linear model, a better estimation of $g_0$ is expected to lead to a better estimation of ${\mathit{\beta }}^0$. Therefore, our method improves over the methods in Müller and Van de Geer (2015) and Yu, Levine and Cheng (2019) when $g_0$ possesses heterogeneous smoothness. The results on degrees of freedom provide a clear rationale for our method’s superior ability to adapt to heterogeneous smoothness.

2.2. Degrees of freedom

We assume $\mathbf{X}$ and $\mathbf{z}$ are fixed with $z_{(1)}<z_{(2)}<\cdots <z_{(n)}$ to examine the degrees of freedom for the proposed partial linear trend filtering method (7). Recall that for given data $\mathbf{y}\in {\mathbb{R}}^n$ with $\mathbb{E}(\mathbf{y})=\mathit{\mu },\text{Cov}(\mathbf{y})={\sigma }^2\mathbf{I}$, the effective degrees of freedom (Stein, 1981; Hastie and Tibshirani, 1990) of $\widehat{\mathit{\mu }}$, as an estimator of $\mathit{\mu }$, is defined as

$$\text{df}\left(\widehat{\mathit{\mu }}\right)=\frac{1}{{\sigma }^2}\sum _{i=1}^n\text{Cov}\left({\widehat{\mu }}_i,y_i\right).$$ (9)

The degrees of freedom measures the complexity of an estimator and plays an important role in model assessment and selection. Since (7) follows the generalized LASSO form, we can apply established results on the generalized LASSO (Tibshirani and Taylor, 2011, 2012) to derive the degrees of freedom for (7).

Proposition 2.1. Consider $(\widehat{\mathit{\beta }},\widehat{\mathit{\theta }})$ from (7). Define the two active sets

$$𝒜=\left\{1\le j\le p:{\widehat{\beta }}_j\ne 0\right\},ℬ=\left\{1\le j\le n-k-1:{\left(D^{(\mathbf{z},k+1)}\widehat{\mathit{\theta }}\right)}_j\ne 0\right\}.$$

Assume the Gaussian partial linear model $\mathbf{y}~𝒩(\mathbf{X}{\mathit{\beta }}^0+g_0(\mathbf{z}),{\sigma }^2\mathbf{I})$.

1. For any fixed $\mathbf{X},\mathbf{z}$ and $\lambda \ge 0,\gamma \ge 0$, the degrees of freedom for the fitting $\mathbf{X}\widehat{\mathit{\beta }}+\widehat{\mathit{\theta }}$ is

   $$\text{df}(\mathbf{X}\widehat{\mathit{\beta }}+\widehat{\mathit{\theta }})=\mathbb{E}\left[\text{dim}\left(\text{Col}\left({\mathbf{X}}_{𝒜}\right)+\text{Null}\left(D_{-ℬ}^{(\mathbf{z},k+1)}\right)\right)\right],$$

   where $\text{Col}\left({\mathbf{X}}_{𝒜}\right)$ is the subspace spanned by the columns of $\mathbf{X}$ that are indexed by $\mathcal{A}$, and $\text{Null}(D_{-ℬ}^{(\mathbf{z},k+1)})$ is the nullspace of the matrix $D^{(\mathbf{z},k+1)}$ after removing the rows indexed by $ℬ$.

2. In addition, denote the first $k+1$ columns and last $n-k-1$ columns of $\mathbf{Q}$ in (6) by ${\mathbf{Q}}_1\in {\mathbb{R}}^{n\times (k+1)},{\mathbf{Q}}_2\in {\mathbb{R}}^{n\times (n-k-1)}$ respectively. Let ${\mathbf{U}\mathbf{U}}^{\prime }$ be the projection operator onto the space orthogonal to $\text{Col}\left({\mathbf{Q}}_1\right)$ where $\mathbf{U}\in {\mathbb{R}}^{n\times (n-k-1)}$ has orthonormal columns. If the matrix $\left[{\mathbf{U}}^{\prime }\mathbf{X},\frac{\lambda }{\gamma k!}{\mathbf{U}}^{\prime }{\mathbf{Q}}_2\right]$ has columns in general position (Tibshirani, 2013), then

$$\begin{gathered} \text{df}(\mathbf{X}\widehat{\mathit{\beta }}+\widehat{\mathit{\theta }})=\mathbb{E}\left[\text{dim}\left(\text{Col}\left({\mathbf{X}}_{𝒜}\right)\right)+\text{dim}\left(\text{Null}\left(D_{-ℬ}^{(\mathbf{z},k+1)}\right)\right)\right] \\ =\mathbb{E}[|𝒜|+|ℬ|]+k+1. \end{gathered}$$

It is known that $\mathbb{E}\left[\text{dim}\left(\text{Col}\left({\mathbf{X}}_{𝒜}\right)\right)\right]$ is the degrees of freedom for $\mathbf{X}\widehat{\mathit{\beta }}$ when $\widehat{\mathit{\beta }}$ is a standard LASSO estimate (Tibshirani and Taylor, 2012), and $\mathbb{E}\left[\text{dim}(\text{Null}(D_{-ℬ}^{(\mathbf{z},k+1)}))\right]$ is the degrees of freedom for $\widehat{\mathit{\theta }}$ if $\widehat{\mathit{\theta }}$ is from univariate trend filtering (Tibshirani, 2014). Part (i) of Proposition 2.1 shows that the degrees of freedom for the partial linear trend filtering (7)—based on the idea of combining LASSO and trend filtering–equals to the expected dimension of the sum of $\text{Col}\left({\mathbf{X}}_{𝒜}\right)$ and $\text{Null}(D_{-ℬ}^{(\mathbf{z},k+1)})$. When these two subspaces have no intersection except $\mathbf{0}$, the sum becomes direct sum so that

$$\text{dim}(\text{Col}({\mathbf{X}}_{𝒜})+\text{Null}(D_{-ℬ}^{(\mathbf{z},k+1)}))=\text{dim}(\text{Col}({\mathbf{X}}_{𝒜}))+\text{dim}(\text{Null}(D_{-ℬ}^{(\mathbf{z},k+1)})).$$

Part (ii) of Proposition 2.1 provides a sufficient condition for the above to hold. The result in Part (ii) admits a more direct interpretation: for an unbiased estimate of the degrees of freedom of $\mathbf{X}\widehat{\mathit{\beta }}+\widehat{\mathit{\theta }}$, we count the number of nonzeros in $\widehat{\mathit{\beta }}$ and the number of changes in the $(k+1)$th discrete derivative of $\widehat{\mathit{\theta }}$, and add them up together with $k+1$. The proof for Proposition 2.1 is provided in Appendix A.

The degrees of freedom allows us to calibrate model complexities for a fair comparison of different methods. Consider the method from Müller and Van de Geer (2015); Yu, Levine and Cheng (2019) based on smoothing splines,

$$(\widehat{\mathit{\beta }},\widehat{g})\in \underset{\mathit{\beta }\in {\mathbb{R}}^p,g\in {𝒢}_n^k}{\text{arg}\text{min}}\frac{1}{2}\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }-g(\mathbf{z}){\Vert }_n^2+\lambda \Vert \mathit{\beta }{\Vert }_1+\gamma {\int }_0^1{\left(g^{((k+1)/2)}(t)\right)}^{2}dt,$$ (10)

where ${𝒢}_n^k$ is the space of $k$th degree natural splines with knots at the input points $z_1,\dots ,z_n$. We refer to it as partial linear smoothing spline. Using $n$ basis functions spanning ${𝒢}_n^k$, the partial linear smoothing spline estimation can be rewritten in a form which is similar to (6) or (7) except that the ${\ell }_1$ penalty for the nonparametric part is replaced with a squared ${\ell }_2$ penalty. We omit the detail as this is standard in the nonparametrics literature. Figure 1 presents a comparison between the partial linear smoothing spline (10) and partial linear trend filtering (3) across various degrees of freedom (by changing the tuning parameters $(\lambda ,\gamma )$). Note that the degrees of freedom for partial linear smoothing spline does not admit a simple form. We hence use the original definition (9) to numerically compute it. In this comparison, we consider the model $g_0(z)=2\text{min}(z,1-z{)}^{0.2}\text{sin}\{2.85\pi /(0.3+\text{min}(z,1-z))\}$ with setting $n=1000,s=4$ and $p=100$. Figure 1 illustrates that, when both methods are applied with the same total degrees of freedom of 11.5, the partial linear smoothing spline fails to adequately capture local smoothness at the function’s boundaries, whereas the partial linear trend filtering method performs more effectively in this regard. Although increasing the degrees of freedom to 39 improves the fit of the partial linear smoothing spline at the boundaries, it leads to oversmoothing in the central regions of the function. These observations highlight that partial linear trend filtering better adapts to varying levels of local smoothness compared to partial linear smoothing splines, providing a key motivation for our study. We provide more empirical comparisons in Sections 3 and 4.

Fig 1.

Comparison of $\widehat{g}(z)$ estimates using partial linear trend filtering (PLTF) and partial linear smoothing splines (PLSS) for two different total degrees of freedom. The total degree of freedom is approximately 11.5 for both PLTF and PLSS for the upper plot, and 11.5 and 39 for the lower plot correspondingly. The grey line denotes the true function.

2.3. Theoretical Properties

In this section, we study the rate of convergence for the proposed estimators $(\widehat{\mathit{\beta }},\widehat{g})$ in (3). We first introduce our technical conditions.

Condition 1. The covariate $\mathbf{x}=\left(x_1,\dots ,x_p\right)$ has sub-Gaussian coordinates:

$$\underset{1\le j\le p}{\text{max}}{‖x_j‖}_{{\psi }_2}\le K_x.$$

Condition 2. The noise $\epsilon$ is sub-Gaussian: $\mathbb{E}(\epsilon )=0,\text{Var}(\epsilon )={\sigma }^2,\Vert \epsilon {\Vert }_{{\psi }_2}\le K_{\epsilon }\sigma$.

Condition 3. $z$ has a continuous distribution supported on [0, 1]. Its density is bounded below by a constant ${\ell }_z>0$.

Condition 4. Define $\mathbf{h}(z)=\left(h_1(z),\dots ,h_p(z)\right)=\mathbb{E}(\mathbf{x}\mid z)$ and $\tilde{\mathbf{x}}=\mathbf{x}-\mathbf{h}(z)$. Assume ${\lambda }_{\text{min}}\left(\mathbb{E}{\widetilde{\mathbf{x}\mathbf{x}}}^{\prime }\right)\ge {\Lambda }_{\mathit{\text{min}}}>0$ and ${\lambda }_{\text{max}}\left(\mathbb{E}\mathbf{h}(z)\mathbf{h}(z{)}^{\prime }\right)\le {\Lambda }_{\mathit{\text{max}}}<\infty$, where ${\Lambda }_{\text{min}}$ and ${\Lambda }_{\text{max}}$ are positive and bounded constants.

Condition 5. ${\text{max}}_{1\le j\le p}\text{TV}(h_j^{(k)})\le L_h$.

Condition 6. $\frac{s^2\text{log}p+s{\text{log}}^{2}p}{n}=o(1)$ and $p\to \infty$, as $n\to \infty$.

Compared to Müller and Van de Geer (2015), Condition 1 relaxes the assumption of $\mathbf{x}$ from being uniformly bounded to sub-Gaussian. Compared to Yu, Levine and Cheng (2019), Condition 1 only requires sub-Gaussianity for marginal distributions of $\mathbf{x}$, instead of the joint distribution. Condition 2 is the same as in Yu, Levine and Cheng (2019), relaxing the errors from being standard normal in Müller and Van de Geer (2015) to sub-Gaussian. For Condition 3, the continuity assumption is very weak, and the lower bound on the density is mainly used to bound the maximum gap between adjacent input points with high probability. See Wang, Smola and Tibshirani (2014); Sadhanala and Tibshirani (2019) for similar assumptions in the context of trend filtering. Condition 4 is common in semiparametric literature (Yu, Mammen and Park, 2011; Müller and Van de Geer, 2015; Yu, Levine and Cheng, 2019). It ensures that there is enough information in the data to identify the parameters in the linear part. Condition 5 is similar to Condition 2.6 in Müller and Van de Geer (2015) and Assumption A. 5 in Yu, Levine and Cheng (2019). This condition enables to obtain the fast rate for $\widehat{\mathit{\beta }}$. Condition 6 is a scaling condition in high dimension. While the condition $(s^2\text{log}p)/n=o(1)$ is stronger than the commonly assumed $(s\text{log}p)/n=o(1)$ in the LASSO literature, it allows for the advantage of avoiding any assumptions about the joint distribution of $\mathbf{x}$. It is possible to only require the weaker condition $(s\text{log}p)/n=o(1)$, if certain distributional assumption (e.g. joint sub-Gaussian) on $\mathbf{x}$ is made. We leave this for a future study.

Our main results consist of two parts, the result regarding $\widehat{g}$ for the nonparametric part, and the result about $\widehat{\mathit{\beta }}$ for the high-dimensional linear part. We now move on to the convergence rate result for $\widehat{g}$.

Theorem 2.1. Assume Conditions 1–4 and 6. Choose $\lambda =c_1\sqrt{\frac{\text{log}p}{n}},\gamma =c_2\left(\frac{s\text{log}p}{n}+n^{-\frac{2k+2}{2k+3}}\right)$ with large enough constants $c_1,c_2>0$. Then, there exist constants $c_3,c_4,n_0>0$ such that any solution $\widehat{g}$ in (3) satisfies

$${‖\widehat{g}-g_0‖}^2\le c_3\left(\frac{s\text{log}p}{n}+n^{-\frac{2k+2}{2k+3}}\right),$$

$${‖\widehat{g}-g_0‖}_n^2\le c_3\left(\frac{s\text{log}p}{n}+n^{-\frac{2k+2}{2k+3}}\right),$$

with probability at least $1-p^{c_4}-n^{c_4}$, as long as $n\ge n_0$. The constants $c_1,c_2,c_3,c_4,n_0$ may depend on $k,L_g,L_h,K_x,K_{ϵ},{\ell }_z,{\Lambda }_{\mathit{\text{min}}},{\Lambda }_{\mathit{\text{max}}},\sigma$.

The expressions for the constants ${\left\{c_i\right\}}_{i=1}^4$, though potentially involved and not sharp, can be derived by keeping track of the explicit forms of constants $\left\{D_i\right\}$ in the proof of Theorem 2.1 (see Appendix B.3 for the details). Given that our focus is on the convergence rate in terms of the parameters $\{n,s,p,k\}$, we do not define ${\left\{c_i\right\}}_{i=1}^4$ explicitly in the theorem. Similar treatments appear in the partial linar model and trend filtering literature (Yu, Levine and Cheng, 2019; Tibshirani, 2014; Sadhanala and Tibshirani, 2019). Viewing ${\left\{c_i\right\}}_{i=1}^4$ as fixed, Theorem 2.1 shows that the integrated squared error and the input-averaged squared error have the same convergence rate, and the rate is determined by the maximum between a sparse estimation rate $(s\text{log}p)/n$ and a nonparametric rate $n^{-(2k+2)/(2k+3)}$. When $g_0$ is sufficiently smooth, belonging to a $k$-order Sobolev or Holder class, a similar rate-switching phenomenon (switching between $(s\text{log}p)/n$ and $n^{-(2k)/(2k+1)}$) has been revealed for partial linear smoothing spline (10) (Müller and Van de Geer, 2015), and the rate is proved to be (nearly) minimax optimal (Yu, Levine and Cheng, 2019). Given that the function class ${𝒱}_k\left(L_g\right)$ from (1) considered in Theorem 2.1 is larger than a $(k+1)$-order Sobolev class, the minimax lower bound derived for $(k+1)$-order Sobolev classes in Yu, Levine and Cheng (2019), i.e. $(s\text{log}(p/s))/n+n^{-(2k+2)/(2k+3)}$, implies that the rate obtained by our partial linear trend filtering (3) is (nearly) minimax optimal. In particular, when $(s\text{log}p)/n=o\left(n^{-(2k+2)/(2k+3)}\right)$, our method $\widehat{g}$ achieves the optimal nonparametric rate $n^{-(2k+2)/(2k+3)}$ that is not attainable by partial linear smoothing spline (see related discussions in Section 1). We proceed to the convergence rate result for $\widehat{\mathit{\beta }}$.

Theorem 2.2. Assume Conditions 1-6, with the same choice of $\lambda ,\gamma$ in Theorem 2.1, any solution $\widehat{\mathit{\beta }}$ in (3) satisfies

$${‖\widehat{\mathit{\beta }}-{\mathit{\beta }}^0‖}_2^2\le c_3\frac{s\text{log}p}{n},$$

$${‖{\mathbf{x}}^{\prime }\left(\widehat{\mathit{\beta }}-{\mathit{\beta }}^0\right)‖}^2\le c_3\frac{s\text{log}p}{n},$$

$${‖\mathbf{X}\left(\widehat{\mathit{\beta }}-{\mathit{\beta }}^0\right)‖}_n^2\le c_3\frac{s\text{log}p}{n},$$

with probability at least $1-p^{c_4}-n^{c_4}$, as long as $n\ge n_0$. The constants $c_1,c_2,c_3,c_4,n_0$ are identical to those in Theorem 2.1.

As discussed after Theorem 2.1, the constants ${\left\{c_i\right\}}_{i=1}^4$ can be explicitly defined through careful bookkeeping in the proof. We do not present the explicit expressions in Theorem 2.2, as we focus on the convergence rate with respect to $\{n,s,p\}$. Treating ${\left\{c_i\right\}}_{i=1}^4$ as fixed, Theorem 2.2 demonstrates that the estimation error, out-of-sample prediction error, and insample prediction error, all have the same convergence rate $(s\text{log}p)/n$. It is well known that this rate is the typical rate that the LASSO achieves in standard high-dimensional sparse linear regressions (Tsybakov, Bickel and Ritov, 2009; Ye and Zhang, 2010; Raskutti, Wainwright and Yu, 2011; Verzelen, 2012), though the constant $c_3$ depends on additional parameters such as ${\ell }_z,{\Lambda }_{\text{min}},{\Lambda }_{\text{max}},L_h$. Therefore, we can conclude that our estimator $\widehat{\mathit{\beta }}$ attains the oracle rate (up to a constant factor) as if the true function $g_0$ were known. Müller and Van de Geer (2015); Yu, Levine and Cheng (2019) showed that the partial linear smoothing spline can achieve the same rate, however, only when $g_0$ lies in Sobolev or Holder classes. In contrast, our method obtains the rate when $g_0$ belongs to a larger class ${𝒱}_k\left(L_g\right)$ that covers more heterogeneously smooth functions. All the proofs related to Theorem 2.1 and Theorem 2.2 are provided in Appendix B and Appendix C.

Algorithm 1:

A BCD algorithm for high-dimensional partial linear trend filtering

Data: $\left\{y_i,{\mathbf{x}}_i,z_i\right\},i=1,\dots ,n$

Fixed (tuning) Parameters: $\lambda ,\gamma$

Predefined Error Thereshold: $ϵ$

1. Set $t=0$ and initialization ${\mathit{\theta }}^{(0)}$

2. For $(t+1)$-th iteration, where $t=0,1,2,\dots$:

(a) Block 1: Let $y_i^{(t)*}=y_i-{\theta }_i^{(t)}$, and update ${\mathit{\beta }}^{(t)}$ by fitting the LASSO:

${\mathit{\beta }}^{(t+1)}=\underset{\mathit{\beta }}{\text{arg}\text{min}}\frac{1}{2}{‖{\mathbf{y}}^{(t)*}-\mathbf{X}\mathit{\beta }‖}_n^2+\lambda \Vert \mathit{\beta }{\Vert }_1$

(b) Block 2: Let $y_i^{(t)**}=y_i-{\mathbf{x}}_i^{\prime }{\mathit{\beta }}^{(t+1)}$, and update ${\mathit{\theta }}^{(t)}$ by fitting the univariate trend filtering:

${\mathit{\theta }}^{(t+1)}=\underset{\mathit{\theta }}{\text{arg}\text{min}}\frac{1}{2}{‖{\mathbf{y}}^{(t)**}-\mathit{\theta }‖}_n^2+\gamma {‖D^{(\mathbf{z},k+1)}\mathit{\theta }‖}_1$

(c) If ${‖\mathbf{X}{\mathit{\beta }}^{(t+1)}+{\mathit{\theta }}^{(t+1)}-\mathbf{X}{\mathit{\beta }}^{(t)}-{\mathit{\theta }}^{(t)}‖}_n^2<ϵ$, then stop the iteration. If not, continue the iteration until it reaches the predefined maximum iteration number

3. Return $\left({\mathit{\beta }}^{(t+1)},{\mathit{\theta }}^{(t+1)}\right)$ at convergence

2.4. Computational details

As described in Section 2.1, to compute $(\widehat{\mathit{\beta }},\widehat{g})$ in (3), we first solve (7) to obtain $(\widehat{\mathit{\beta }},\widehat{\mathit{\theta }})$. Then, $\widehat{g}(t)=\sum _{\ell =1}^n{\widehat{\alpha }}_{\ell }{q}_{\ell }(t)$, where $\widehat{\mathit{\alpha }}={\mathbf{Q}}^{-1}\widehat{\mathit{\theta }},\mathbf{Q}={(q_{\ell }(z_k))}_{1\le \ell ,k\le n}$, and $q_{\ell }$’s are the falling factorial basis defined in (4)-(5). We use a Block Coordinate Descent (BCD) algorithm for solving the optimization (7). The algorithm iterates over two blocks, $\mathit{\beta }$ and $\mathit{\theta }$, by solving a standard LASSO problem and univariate trend filtering respectively. The detailed steps of the algorithm are outlined in Algorithm 1.

The objective function in (7) can be written in the following form:

$$\frac{1}{2}\Vert \mathbf{y}-\mathbf{X}\mathit{\beta }-\mathit{\theta }{\Vert }_n^2+\lambda \Vert \mathit{\beta }{\Vert }_1+\gamma {‖D^{(\mathbf{z},k+1)}\mathit{\theta }‖}_1≔f_0(\mathit{\beta },\mathit{\theta })+f_1(\mathit{\beta })+f_2(\mathit{\theta })=f(\mathit{\beta },\mathit{\theta }),$$ (11)

where $f_0$ is convex and differentiable, and $f_1,f_2$ are convex but nondifferentiable. It is direct to verify that $f$ is continuous on the compact set $\left\{(\mathit{\beta },\mathit{\theta }):f(\mathit{\beta },\mathit{\theta })\le f\left({\mathit{\beta }}^{(1)},{\mathit{\theta }}^{(1)}\right)\right\}$, and $f$ attains its minimum, denoted by $f\left({\mathit{\beta }}^{*},{\mathit{\theta }}^{*}\right)={\text{min}}_{\mathit{\beta },\mathit{\theta }}f(\mathit{\beta },\mathit{\theta })$. This fact combined with Theorem 4.1 and Lemma 3.1 in Tseng (2001) shows that there exists a subsequence $\left\{\left({\mathit{\beta }}^{\left(k_n\right)},{\mathit{\theta }}^{\left(k_n\right)}\right)\right\}$ converging to a stationary point of $f$. Due to the convexity of $f$, it further implies $\left({\mathit{\beta }}^{\left(k_n\right)},{\mathit{\theta }}^{\left(k_n\right)}\right)\to \left({\mathit{\beta }}^{*},{\mathit{\theta }}^{*}\right)$, as $n\to \infty$. Given that $f$ is continuous and $f\left({\mathit{\beta }}^{(t+1)},{\mathit{\theta }}^{(t+1)}\right)\le f\left({\mathit{\beta }}^{(t)},{\mathit{\theta }}^{(t)}\right),\forall t=1,2,\dots$, we can conclude that $f\left({\mathit{\beta }}^{(t+1)},{\mathit{\theta }}^{(t+1)}\right)$ converges to the global minimum $f\left({\mathit{\beta }}^{*},{\mathit{\theta }}^{*}\right)$ as $t\to \infty$.

Our algorithm is implemented in R, utilizing the glmnet package for computing the LASSO in the first block update and the glmgen package for univariate trend filtering in the second block. In practice, we compute solutions over a two-dimensional grid of tuning parameters $(\lambda ,\gamma )$, and use model selection criteria such as cross-validation to select the tuning parameter. This makes our method more computationally demanding than single-parameter cases because the tuning grid scales quadratically with the number of parameter values – an issue extends to other double penalization approaches in the context of partial linear models. To address this issue, we employ efficient warm-start tricks (Friedman, Hastie and Tibshirani, 2010; Ramdas and Tibshirani, 2016), using warm-starts across adjacent $(\lambda ,\gamma )$ values, to speed up the computations of solutions on the two-dimensional grid. Consequently, the computation time remains manageable for both our simulation settings and real data analysis. A similar strategy is adopted to compute partial linear smoothing splines. The R function, stats::smooth.spline, is used to calculate the univariate smoothing spline for the second block update. The implemented BCD algorithms as an R package, plmR, for PLTF and PLSS are publicly available at https://github.com/SangkyuStat/plmR.

3. Simulations

Through empirical experiments, we evaluate the performance of partial linear trend filtering (PLTF) introduced in (3), in comparison to partial linear smoothing splines (PLSS) defined in (10) (Müller and Van de Geer, 2015; Yu, Levine and Cheng, 2019).

3.1. Simulation settings and results

We generate the $p$-dimensional covariates $\mathbf{x}$ and the univariate covariate $z$ in the following way: we first sample $\tilde{\mathbf{x}}=\left({\tilde{x}}_1,\dots ,{\tilde{x}}_{p+1}\right)$ from $𝒩(\mathbf{0},\Sigma )$, where $\Sigma =({\sigma }_{jk})$ with ${\sigma }_{jk}={0.5}^{|j-k|}$ and $j,k=1,\dots ,p+1$; then we set $z=\Phi \left({\tilde{x}}_{25}\right)$ with $\Phi$ being the standard normal’s CDF, $x_j={\tilde{x}}_j$ for $j=1,\dots ,24$, and $x_j={\tilde{x}}_{j+1}$ for $j=25,\dots ,p$. We consider three different partial linear models as follows: for $i=1,2,\dots ,n$,

Model 1 (Smooth function model):

$$y_i=x_{i6}{\beta }_1+x_{i12}{\beta }_2+x_{i15}{\beta }_3+x_{i20}{\beta }_4+\text{sin}\left(2\pi z_i\right)+{ϵ}_i,$$

Model 2 (Heterogeneous smooth function model):

$$y_i=x_{i6}{\beta }_1+x_{i12}{\beta }_2+x_{i15}{\beta }_3+x_{i20}{\beta }_4+e^{3z_i}\text{sin}\left(6\pi z_i\right)/7+{ϵ}_i,$$

Model 3 (Doppler-type function model):

$$y_i=x_{i6}{\beta }_1+x_{i12}{\beta }_2+x_{i15}{\beta }_3+x_{i20}{\beta }_4+\text{sin}\left(4/z_i\right)+{ϵ}_i,$$

where ${ϵ}_i~𝒩\left(0,{\sigma }_{ϵ}^2\right)$. The $g_0$ function displays varying levels of heterogeneous smoothness in the three models. The actual forms of these functions are shown in Figure 2. The values for ${\beta }_j,j=1,\dots ,4$ are $(0.5,1,1,1.5)$. Various values for ${\sigma }_{ϵ}^2$ are used to vary the signal-to-noise ratio for the models. Similar to Müller and Van de Geer (2015); Sadhanala and Tibshirani (2019), we define the total signal-to-noise ratio (tSNR) as

$$\text{tSNR}=\sqrt{\frac{\mathbb{E}{({\mathbf{x}}^{\prime }{\mathit{\beta }}^0+g_0(z))}^2}{{\sigma }_{ϵ}^2}}.$$

We vary the tSNR from 4 to 16 on a logarithmic scale and then calculate the error metrics for each method. Specifically, we consider three different error metrics:

1. ${‖\mathbf{X}\widehat{\mathit{\beta }}+\widehat{g}(\mathbf{z})-(\mathbf{X}{\mathit{\beta }}^0+g_0(\mathbf{z}))‖}_n^2$: mean squared error (MSE) for $\mathbf{X}\widehat{\mathit{\beta }}+\widehat{g}(\mathbf{z})$.

2. ${‖\widehat{\mathit{\beta }}-{\mathit{\beta }}^0‖}_2^2:l_2$-norm squared error for $\widehat{\mathit{\beta }}$.

3. ${‖\widehat{g}(\mathbf{z})-g_0(\mathbf{z})‖}_n^2$: mean squared error for $\widehat{g}$.

The metrics are computed over 150 repetitions of randomly generated datasets for each tSNR value, and the medians of each metric are selected as the final results. We consider $p$ to be 100 and 1000 for low and high-dimensional cases, respectively, with $n$ fixed at 500. We compare partial linear cubic smoothing spline ($k=3$ in (10)) and second degree partial linear trend filtering ($k=2$ in (3)) so that both methods regularize the second derivative of $g$. A similar comparison has been performed in Sadhanala and Tibshirani (2019) under additive models. To ensure fair comparisons, we present results using both optimally tuned parameters and cross-validation (CV)-tuned parameters for $(\lambda ,\gamma )$ in the two methods.

Fig 2.

The function $g_0(z)$ in Models 1 – 3. The function becomes increasingly locally heterogeneous from Model 1 (left) to Model 3 (right).

The simulation results are displayed in Figures 3–5. Figure 3 demonstrates that when the function $g_0$ has homogeneous smoothness, the performance of the PLTF method is comparable, almost the same, to that of the PLSS method, with no significant differences observed. However, as shown in Figure 4, when the function exhibits varying smoothness, the difference between the two methods becomes more pronounced. When the degree of heterogeneous smoothness of $g_0$ increases, as seen in Figure 5, all three errors of PLTF are significantly lower than those of PLSS in both low and high-dimensional cases. This disparity becomes more apparent as the tSNR increases. These simulation results suggest that for functions with homogeneous smoothness, PLTF and PLSS are competitive. However, as the function becomes more heterogeneously smooth, PLTF can significantly outperform PLSS in both low and high-dimensional scenarios, particularly when the tSNR is high. The empirical findings are aligned with those for comparing trend filtering and smoothing splines in the context of univariate nonparametric regressions and additive regression models (Tibshirani, 2014; Sadhanala and Tibshirani, 2019).

Fig 3.

PLTF v.s. PLSS under Model 1, with tSNR ranging from 4 to 16. TF (Optimized or CV) denotes PLTF with (optimally or CV) tuned parameters. SS (Optimized or CV) denotes PLSS with (optimally or CV) tuned parameters.

Fig 5.

PLTF v.s. PLSS under Model 3, with tSNR ranging from 4 to 16. TF (Optimized or CV) denotes PLTF with (optimally or CV) tuned parameters. SS (Optimized or CV) denotes PLSS with (optimally or CV) tuned parameters.

Fig 4.

PLTF v.s. PLSS under Model 2, with tSNR ranging from 4 to 16. TF (Optimized or CV) denotes PLTF with (optimally or CV) tuned parameters. SS (Optimized or CV) denotes PLSS with (optimally or CV) tuned parameters.

We also conduct similar simulations for $n=100$, to evaluate the performance of PLTF and PLSS under smaller sample size. The results show similar patterns to those observed in Figures 3–5. We defer the details to Appendix G.

4. Applications to the IDATA Study

The Interactive Diet and Activity Tracking in AARP (IDATA) Study aimed to improve dietary intake assessments and their links to health outcomes (Subar et al., 2020). The amount of the intake of ultra-processed foods (UPF) was estimated using the NOVA system, which is developed to measure the intake of UPF based on the participants’ diet (Steele et al., 2023). This approach and works already have been established between UPF and biomarkers (Abar et al., 2025). More detailed data collection protocols, information and procedures are available in the Appendix E.

To investigate the relationship between metabolomic profiles and UPF intake, we define the response variable $\mathbf{y}$ as UPF intake measured in grams and the covariates $\mathbf{X}$ as metabolite measurements. We analyze 952 metabolites $(p=952)$ from the serum dataset and 1,045 metabolites $(p=1,045)$ from the urine dataset. UPF intake is well established to differ between females and males (Juul et al., 2018; Sung et al., 2021). Accordingly, we classify participants into two groups: females $(n=365)$ and males $(n=353)$. Previous studies have also shown that certain obesity-related body measurements, such as BMI, waist circumference, and hip circumference (Canhada et al., 2020), as well as age (Fu, Chen and Liu, 2024), exhibit significant nonlinear relationships with UPF intake. Therefore, we consider a univariate $z$ in (7), representing as age, BMI, waist circumference, or hip circumference. Previously, a LASSO-based approach was used for feature selection in this dataset, but it did not account for potential nonlinear relationships (Abar et al., 2025). This motivated our use of a partial linear modeling framework, which provides added flexibility. We then apply the proposed method, PLTF, to identify significant metabolites associated with UPF and construct a prediction model to evaluate its predictive accuracy.

To evaluate predictive accuracy, we compare the proposed PLTF method with PLSS and LASSO. Since our simulation settings reflect the structure of the real data, we maintain consistency by setting $k=2$ for PLTF and $k=3$ for PLSS, regularizing the second derivative of $g$. To assess prediction performance, the dataset is randomly split into 90% for training and 10% for testing. Models are trained on the training dataset using 10-fold cross-validation (CV), and the mean squared error (MSE) is computed on the testing set. This process is repeated 500 times to achieve standard errors within 1% to 2% of the MSE. The prediction errors are reported in Table 1. The prediction results indicate that, in the majority of cases, PLTF outperformed the other models (12 out of 16 cases). Notably, for the serum datasets, PLTF significantly outperformed the other methods, particularly in the female subgroup. In the urine datasets, PLTF generally demonstrated better performance. Even in cases where other models marginally outperformed PLTF, the difference between PLTF and the best-performing model was not statistically significant.

Table 1.

Prediction MSE results for different variables $(z)$ with 500 repetitions for different types of datasets. Numbers in parentheses are the corresponding standard errors. The bolded numbers indicate the smallest error for each case, and it is underlined if it is at least the 1 standard error far from any other models. When it is farther than the 2 standard error, then it is double-underlined

			Age	BMI	Hip	Waist
Serum	Female	PLTF	175.390 (2.289)	174.789 (2.328)	176.591 (2.312)	177.725 (2.360)
		PLSS	178.696 (2.383)	180.286 (2.420)	180.230 (2.390)	180.331 (2.440)
		LASSO	188.370 (2.516)	190.647 (2.555)	190.461 (2.545)	190.115 (2.543)
	Male	PLTF	152.774 (2.012)	150.925 (1.994)	153.823 (1.991)	154.350 (2.055)
		PLSS	152.904 (2.019)	154.373 (2.019)	153.470 (1.996)	155.080 (2.070)
		LASSO	155.738 (2.036)	156.044 (2.045)	156.342 (2.040)	156.619 (2.062)
Urine	Female	PLTF	171.911 (2.361)	169.977 (2.371)	170.057 (2.350)	172.937 (2.469)
		PLSS	172.365 (2.425)	171.910 (2.427)	171.859 (2.421)	172.696 (2.473)
		LASSO	172.343 (2.468)	172.626 (2.494)	173.094 (2.488)	172.584 (2.483)
	Male	PLTF	144.786 (1.894)	143.859 (1.905)	145.162 (1.868)	144.145 (1.920)
		PLSS	144.299 (1.894)	144.790 (1.877)	145.145 (1.856)	144.803 (1.904)
		LASSO	148.020 (1.996)	147.060 (1.997)	147.009 (1.982)	146.509 (2.017)

Next, in Tables 2 and 3, we present the top 10 most commonly selected for each $z$ variable (i.e., age, BMI, waist circumference, and hip circumference) stratified by biospecimen types (serum and urine) and sex (male and female). These metabolites are ranked according to the absolute values of their coefficients. To ensure fair comparisons across coefficients, all covariates are standardized. The full lists of the top 10 selected variables are provided in Tables S1–S4 for different datasets in Appendix F.

Table 2.

Top 10 selected metabolites and their super pathways results for the serum dataset, which are selected at least 3 times for 4 different $z$ variables. + and − in parentheses denote the signs of coefficients for metabolites

		Selected Variables
		Biochemical	Super Pathway
Serum	Female	Diglycerol (+)	Xenobiotics
		4-Allylphenol sulfate (−)	Xenobiotics
		Anthranilate (+)	Amino Acid
		Eicosapentaenoylcholine (−)	Lipid
		1-Stearoyl-2-Adrenoyl-GPC	Lipid
		(18:0/22:4) (+)
		Branched chain 14:0 dicarboxylic acid (−)	Lipid
		X-21807 (−)	Unknown
		X-25523 (−)	Unknown
	Male	Quinate (−)	Xenobiotics
		Saccharin (+)	Xenobiotics
		1-Methylhistidine (+)	Amino Acid
		1-(1-Enyl-palmitoyl)-2-Oleoyl-GPE	Lipid
		(P-16:0/18:1) (−)
		1-Lignoceroyl-GPC (24:0) (−)	Lipid
		X-19183 (−)	Unknown
		X-21442 (−)	Unknown
		X-23655 (−)	Unknown

Table 3.

Top 10 selected metabolites and their super pathways results for the urine dataset, which are selected at least 3 times for 4 different $z$ variables. + and − in parentheses denote the signs of coefficients for metabolites

		Selected Variables
		Biochemical	Super Pathway
Urine	Female	Galactonate (−)	Carbohydrate
		Riboflavin (vitamin B2) (−)	Cofactors and Vitamins
		Saccharin (+)	Xenobiotics
		2,3-Dihydroxypyridine (−)	Xenobiotics
		Glutamine conjugate of C8H12O2 (3) (−)	Partially Characterized Molecules
		X-12096 (+)	Unknown
		X-12753 (−)	Unknown
		X-23680 (+)	Unknown
		X-25936 (+)	Unknown
		X-25952 (+)	Unknown
	Male	Quinate (−)	Xenobiotics
		1,6-Anhydroglucose (+)	Xenobiotics
		N-Acetylcitrulline (+)	Amino Acid
		3-Methoxytyramine (+)	Amino Acid
		Cortisone (−)	Lipid
		Ursocholate (+)	Lipid
		X-13847 (−)	Unknown
		X-23161 (−)	Unknown
		X-25952 (+)	Unknown

The selected variables differ across different subsets, with their super pathways and biological functions also varying. For example, saccharin, which was selected in the male serum dataset and the female urine dataset, is a well-known artificial sweetener commonly found in UPF products. Given that the signs of the coefficients are positive, this aligns with expected findings. Quinate, selected in both the male serum and urine datasets, is commonly present in plant-based foods such as coffee, fruits, and vegetables. This suggests that plant-based food consumption in the male population may have an indirect negative association with UPF intake, as reflected by the negative coefficient which aligns with the results in Playdon et al. (2024).

Several other findings, including the direction of the coefficients, are consistent with existing research on UPF intake. For instance, 4-allylphenol sulfate was identified as a potential biomarker for UPF intake in a randomized, controlled, domiciled feeding trial (O’Connor et al., 2023). Diglycerol can be found in cosmetics or can be esterified to diglycerol esters of fatty acids. Diglycerol esters of fatty acids are used as emulsifiers in food and are a hallmark of UPF. Food additive emulsifiers have been associated with higher risk of some types of cancer risks by Sellem et al. (2024).

Additionally, eicosapentaenoylcholine has been positively correlated with Vitamin D consumption (Leung et al., 2020), and previous research shows that UPF intake is associated with higher prevalence of inadequate Vitamin D intake (Falcão et al., 2019). Other nutrient deficiencies, including Vitamin B2 deficiency, have also been associated with higher UPF intake in women in Schenkelaars et al. (2024). Lastly, a positive association of levoglucosan with UPF intake has been identified in urine datasets (Muli et al., 2024) and it could be a food contaminant from processing or packaging. Several other findings involve metabolites that have not been extensively studied or are unknown. These may represent potential novel biomarkers associated with UPF intake, providing new insights for future research.

5. Discussion

In this paper, we extend trend filtering to high-dimensional partial linear models via doubly penalized least squares, preserving the local adaptivity and efficiency of univariate trend filtering. We analyze degrees of freedom, establish optimal convergence rates, and demonstrate superior performance over standard smoothing splines for functions with varying smoothness. Our method identifies metabolites linked to UPF with reduced prediction error compared to alternatives. The following discussion explores key directions for future research.

When the covariate $\mathbf{z}\in {\mathbb{R}}^d$ has dimension $d>1$, our work extends naturally to an additive function form $g_0(\mathbf{z})=\sum _{j=1}^d{g}_{0j}(z_j)$ with $\sum _{j=1}^d\text{TV}(g_{0j}^{(k)})\le C$ for some constant $C>0$. Utilizing some techniques from additive trend filtering (Sadhanala and Tibshirani, 2019), the method is expected to apply when $d$ is bounded. For high-dimensional cases where $d$ is comparable to or exceeds $n$, a sparse additive form for $g_0(\mathbf{z})$ with a sparsity penalty (e.g., empirical ${\ell }_2$ norm) for each $g_{0j}$ may be employed. While theoretically challenging, recent results in high-dimensional additive trend filtering (Tan and Zhang, 2019) provide useful guidance.

A key question in partial linear models is identifying which covariates exhibit linear versus nonlinear effects. While this paper assumes such structure is known, in practice, prior knowledge is often unavailable. Penalization-based methods for structure selection typically assume the underlying functions belong to Sobolev or Hölder classes (Zhang, Cheng and Liu, 2011; Huang, Wei and Ma, 2012; Lian, Liang and Ruppert, 2015). Future work could investigate how trend filtering aids in discovering model structures with more heterogeneously smooth functions.

Under partial linear models, variable selection and statistical inference for the linear part can be important tasks. It would be particularly valuable to study how recent advances in high-dimensional linear regression models (Zhang and Zhang, 2014; van de Geer et al., 2014; Javanmard and Montanari, 2014; Dezeure et al., 2015; Barber and Candès, 2015; Candes et al., 2018; Dai et al., 2023) can be extended to high-dimensional partial linear trend filtering, in order to perform variable selection and inference with guaranteed error control.