Time-varying coefficient model estimation through radial basis functions

ABSTRACT

In this paper, we estimate the dynamic parameters of a time-varying coefficient model through radial kernel functions in the context of a longitudinal study. Our proposal is based on a linear combination of weighted kernel functions involving a bandwidth, centered around a given set of time points. In addition, we study different alternatives of estimation and inference including a Frequentist approach using weighted least squares along with bootstrap methods, and a Bayesian approach through both Markov chain Monte Carlo and variational methods. We compare the estimation strategies mention above with each other, and our radial kernel functions proposal with an expansion based on regression spline, by means of an extensive simulation study considering multiples scenarios in terms of sample size, number of repeated measurements, and subject-specific correlation. Our experiments show that the capabilities of our proposal based on radial kernel functions are indeed comparable with or even better than those obtained from regression splines. We illustrate our methodology by analyzing data from two AIDS clinical studies.

1. Introduction

Statistical models for longitudinal data are powerful instruments of analysis when experimental units (subjects) are measured repeatedly over time in relation to a response variable along with static or time-dependent covariates. A very important feature of this kind of data that must be taken into account when fitting a statistical model, is the likely presence of serial correlation within repeated measurements on a given subject (observations between subjects are assumed to be independent). Typically, the main purpose of the analysis is to identify and characterize the evolution (mean tendency) of the response variable over time and quantify its association with the covariates. Parametric techniques for longitudinal data analysis have been exhaustively studied in the literature [37,38,42, see for example and references within]. Though useful in many cases, questions about the adequacy of the assumptions of parametric models and the potential impact of model misspecification on the analysis often arise. For instance, one of the basic assumptions associated with parametric techniques, yet not always satisfied, establishes that the mean response must be a known function of both fixed and random effects, indexed by a set of unknown parameters. Thus, for many practical situations, parametric models may be too restrictive or even unavailable.

In order to overcome such difficulties, building on the contributions of [8,24,70], the work of [27] considered a nonparametric model that lets the parameters vary over time. Nonparametric models of this nature allow more flexible functional dependence between the response variable and the covariates, since they are based on time-dependent coefficients (smooth functions of time) instead of fixed unknown parameters. Due to their interpretability and flexibility, these models have been the subject of active research over the last twenty years.

Early popular developments are given in [5,6,14,15,29,35,36,47,62,63,68,71]. For applications and surveys, see [16,54,65,72].

Specifically, consider the longitudinal dataset

$$\mathcal{D}=\left\{(y_{i,j},{\mathit{x}}_{i,j},t_{i,j},):j=1,\dots ,n_i,i=1,\dots ,n\right\},$$

where $y_{i,j}\equiv y_i\left(t_{i,j}\right)$ and ${\mathit{x}}_{i,j}\equiv {\mathit{x}}_i\left(t_{i,j}\right)$ with ${\mathit{x}}_i\left(t\right)=\left(x_{0,i}\right(t),x_{1,i}(t),\dots ,x_{d,i}(t\left)\right)$ are the real-valued response variable and the $(d+1)$ column covariate vector, corresponding to measurement j of subject i, observed at time $t_{i,j}$, n is the number of subjects, $n_i$ is the number of observations associated with subject i, and the total number of observations in the sample is $N=\sum _{i=1}^n{n}_i$. Measurement times are often distinct and irregularly spaced in a fixed interval of finite length. In order to evaluate the mean joint effect of time t and the covariates $\mathit{x}\left(t\right)$ on the outcome $y\left(t\right)$, we use the structured nonparametric model

$$y\left(t\right)=\mathit{x}\left(t{)}^{\mathrm{T}}\mathrm{\text{β}}\right(t)+ϵ(t)$$ (1)

where $\mathrm{\text{β}}\left(t\right)=\left({\mathrm{\text{β}}}_0\right(t),{\mathrm{\text{β}}}_1(t),\dots ,{\mathrm{\text{β}}}_d(t\left)\right)$ is a $(d+1)$ column vector of real-valued nonparametric functions of time t, called dynamic coefficients or dynamic parameters, and $ϵ\left(t\right)$ is a zero-mean stochastic process, independent of $\mathit{x}\left(t\right)$, with covariance function $\gamma (s,t)=Cov\left[ϵ\left(s\right),ϵ\left(t\right)\right]$. The model in equation (1) is referred to as a (fixed-effects) time-varying coefficient model (TVCM). This model provides a parsimonious approach for characterizing time varying association patterns of a set of dynamic predictors on the expected value of a functional response. Notice that this model has a natural interpretation since for a fixed time point t, the TVCM (1) reduces to a multiple linear model with response variable $y\left(t\right)$ and covariate vector $\mathit{x}\left(t\right)$, for which a standard interpretation of the time-varying coefficients ${\mathrm{\text{β}}}_r\left(t\right)$, $r=0,1,\dots ,d$, holds. In our experiments, we take $x_0\left(t\right)\equiv 1$ for all t, which means that ${\mathrm{\text{β}}}_0\left(t\right)$ is the intercept coefficient describing the baseline time-trend. Finally, note that a discretized version of the TVCM can be obtain by simply substituting t by $t_{i,j}$ in Equation (1), considering all data points given in $\mathcal{D}$, in order to highlight the dependence on data when necessary.

In order to illustrate the full potential that a model such as a TVCM has, we provide below a fair revision on extensions of the model that have taken place over recent years. Motivated by applications rather than simply a desire to modify a statistical model, several extensions of the TVCM (1) have been proposed over the years in all sorts of directions, along with more complex structures and weaker assumptions. Some of these extensions typically share characteristics with each other. Here, we list some relevant instances in no particular order. A popular extension emerged naturally in order to efficiently capture both population and individual relationships. In that way, mixed-effects time-varying coefficient models extended TVCMs by dividing the error term into two parts, one part representing the subject-specific deviation from the population mean function, and the other representing the measurement error [7,31,34,40,53,67]; we consider a model of this sort in our first simulation study (see Section 5.1 for details). Another widespread extension took place when non-Gaussian responses where modeled directly, ranging from dichotomous and categorical outcomes to variables with skewed distributions, which provided a unified framework to do so. Thus, generalized time-varying coefficient models extended TVCMs by introducing a known link function to relate the dynamic linear predictor and the response process to each other [3,5,30,39,40,49,50]. Also, more complex types of dynamic functional dependencies have been developed, such as the relationships provided in time-varying additive and nonlinear models [13,45,51,59,64]. In addition, more adaptations were developed to deal with the same issues that non-varying models deal with. For instance, quantile regression [1,57], variable selection and shrinkage estimation [12,33,56,58,61], and even spatial modeling [2,20,31,43,52,55,66].

On the other hand, several smoothers can be used to estimate the dynamic coefficients of TVCM (1). The key idea behind the estimation process relies on rewriting each ${\mathrm{\text{β}}}_r\left(t\right)$ through a linear expansion of parametric functions in order to make possible parametric-like inference as in standard models. In general, each smoother is indexed by a smoothing parameter vector that controls the trade-off between goodness-of-fit and model complexity. Thus, smoothing parameter selection criteria are in order. Some of the most popular smoothers include local polynomial smoothers, regression spline, smoothing spline, and P-splines [65,69]. Different smoothers have different strengths in one aspect or another. For example, smoothing spline may be good for handling sparse data, while local polynomial smoothers may be computationally advantageous for handling dense designs.

The purpose of this paper is twofold. First, in order to estimate the time-varying coefficients ${\mathrm{\text{β}}}_r\left(t\right)$, we propose a linear smoother based on kernel functions, by treating them as if they were radial basis functions [4, see for a complete catacterization of radial basis funcions]. This approach has been used in both the semiparametric regression [48] and statistical learning literature [23,25] but, to the best of our knowledge, it has not been fully exploited yet in the context of longitudinal data analysis, apart from the work of [52] on space-time varying coefficients models. Our proposal applies to both time-invariant and time-dependent covariates as well as regular and irregular placed design times, and also allows for different amounts of smoothing for different coefficients. Second, since a radial kernel expansion resembles very closely an approximation using spline basis functions, we compare these smoothing alternatives with each other in terms of goodness-of-fit and prediction using both Frequentist and Bayesian inference frameworks. To that end, from a Frequentist perspective, we consider weighted least squares and bootstrap methods. From a Bayesian perspective, we consider Markov chain Monte Carlo (MCMC) along with variational methods. The Bayesian approach has become more popular in recent years [3,19,28,31,39,41,55], but variational algorithms have not been explored to ease the computational burden under this framework.

The remainder of the paper is structured as follows: Section 2 introduces the estimation of time-varying coefficients through radial kernel functions and regression spline. Section 3 discusses different approaches to statistical inference. Section 4 discusses the choice of knots and smoothing parameter selection. Section 5 compares the estimation alternatives through an extensive simulation study. Section 6 illustrates our proposal by analyzing AIDS data coming from two clinical studies. Finally, Section 7 presents some concluding remarks and directions for future work.

2. Estimation using radial kernel functions

The main idea behind estimation through radial kernel functions consists in expressing each dynamic coefficient in model (1) as a linear combination of kernel functions by treating them as radial basis functions. A radial smoother can be constructed using the following set of radial basis:

$$1,t,\dots ,t^g,\xi (|t-{\kappa }_1|),\dots ,\xi (|t-{\kappa }_k|),$$ (2)

where $\xi (\cdot )$ is a kernel function, $|\cdot |$ is the Euclidean norm in $\mathbb{R}$, and ${\kappa }_1<\dots <{\kappa }_k$ are k knots covering the time domain. The smoother performance strongly depends on the proper selection of both location and number of knots (see Section 4 for details). The basis degree g is usually less crucial and it is typically taken as 1, 2, or 3, for computational convenience. On the other hand, note that the first g + 1 basis functions of (2) are polynomials of degree up to g, and the others are all kernel functions, which satisfy the property $\xi \left(t\right)=\xi \left(|t|\right)$. Such functions are known as radial functions [4]. Different kinds of kernel functions are commonly used in practice, such as Gaussian or Epanechnikov kernels, among many others [60, see for a review]. This choice is less significant in terms of smoothing.

Using the radial basis (2), we can express each time-varying coefficient ${\mathrm{\text{β}}}_r\left(t\right)$, $r=0,1,\dots ,d$, as

$${\mathrm{\text{β}}}_r\left(t\right)=\sum _{\ell =0}^g{\alpha }_{r,\ell }{t}^{\ell }+\sum _{\ell =1}^{k_r}{\alpha }_{r,g+\ell }\xi (|t-{\kappa }_{\ell }|)={\mathbf{\Xi }}_r(t{)}^{\mathrm{T}}{\mathit{\alpha }}_r,$$ (3)

where ${\mathbf{\Xi }}_r\left(t\right)=(1,t,\dots ,t^g,\xi (|t-{\kappa }_1|),\dots ,\xi (|t-{\kappa }_{k_r}|\left)\right)$ and ${\mathit{\alpha }}_r=({\alpha }_{r,0},\dots ,{\alpha }_{r,k_r+g+1})$ are $p_r\times 1$ column vectors with $p_r=k_r+g+1$, composed of basis functions evaluated at time t and unknown parameters, respectively. Such a representation is able to accommodate a variety of shapes and smoothness for the dynamic coefficients without overfitting the data, since separate number of knots are allowed. This means that, for a fixed degree basis g, the number of knots $k_0,\dots ,k_d$ play the role of smoothing parameters.

In this way, the dynamic vector $\mathrm{\text{β}}\left(t\right)$ in model (1) becomes

$$\mathrm{\text{β}}\left(t\right)=\mathbf{\Xi }(t{)}^{\mathrm{T}}\mathit{\alpha },$$ (4)

where $\mathit{\alpha }=({\mathit{\alpha }}_0^{\mathrm{T}},\dots ,{\mathit{\alpha }}_d^{\mathrm{T}})$ and $\mathbf{\Xi }\left(t\right)=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{\Xi }}_0\right(t),{\mathbf{\Xi }}_1(t),\dots ,{\mathbf{\Xi }}_d(t\left)\right]$. Note that $\mathit{\alpha }$ is a column vector of size $p\times 1$, $p=\sum _{r=0}^d{p}_r$, whereas $\mathbf{\Xi }\left(t\right)$ is a rectangular matrix of size $p\times (d+1)$. Now, substituting $\mathrm{\text{β}}\left(t\right)$ in model (1) for its equivalent expression given in (4), it follows that the TVCM (1) can be approximately written as

$$y_{i,j}={\mathit{z}}_{i,j}^{\mathrm{T}}\mathit{\alpha }+{ϵ}_{i,j},j=1,\dots ,n_i,i=1,\dots ,n,$$ (5)

where $y_{i,j}\equiv y_i\left(t_{i,j}\right)$, ${ϵ}_{i,j}\equiv {ϵ}_i\left(t_{i,j}\right)$, and ${\mathit{z}}_{i,j}=({\mathit{z}}_{0,i,j}^{\mathrm{T}},\dots ,{\mathit{z}}_{d,i,j}^{\mathrm{T}})$, with

$${\mathit{z}}_{r,i,j}=x_{r,i}\left(t_{i,j}\right){\mathbf{\Xi }}_r\left(t_{i,j}\right)r=0,1,\dots ,d.$$ (6)

Similar to $\mathit{\alpha }$, each ${\mathit{z}}_{i,j}$ is a $p\times 1$ column vector of covariate values times radial basis functions. For the i-th subject, $i=1,\dots ,n$, we denote the response vector, the random error vector, and the design matrix as

$${\mathit{y}}_i=(y_{i,1},\dots ,y_{i,n_i}),{\mathit{ϵ}}_i=({ϵ}_{i,1},\dots ,{ϵ}_{i,n_i}),{\mathit{Z}}_i=[{\mathit{z}}_{i,1},\dots ,{\mathit{z}}_{i,n_i}{]}^{\mathrm{T}}.$$

Consistently, we denote the response vector, the random error vector, and the design matrix for the whole dataset as

$$\mathit{y}=({\mathit{y}}_1^{\mathrm{T}},\dots ,{\mathit{y}}_n^{\mathrm{T}}),\mathit{ϵ}=({\mathit{ϵ}}_1^{\mathrm{T}},\dots ,{\mathit{ϵ}}_n^{\mathrm{T}}),\mathit{Z}=[{\mathit{Z}}_1^{\mathrm{T}},\dots ,{\mathit{Z}}_n^{\mathrm{T}}{]}^{\mathrm{T}},$$

which allow us to express model (5) in matrix form as a standard linear model:

$$\mathit{y}=\mathit{Z}\mathit{\alpha }+\mathit{ϵ}.$$ (7)

Once an estimate for $\mathit{\alpha }$, $\hat{\mathit{\alpha }}=({\hat{\mathit{\alpha }}}_0^{\mathrm{T}},\dots ,{\hat{\mathit{\alpha }}}_d^{\mathrm{T}})$, is obtained; it is straightforward to get an estimate for $\mathrm{\text{β}}\left(t\right)$, $\widehat{\mathrm{\text{β}}}\left(t\right)=\left({\widehat{\mathrm{\text{β}}}}_0\right(t),\dots ,{\widehat{\mathrm{\text{β}}}}_d(t\left)\right)$, by simply letting $\widehat{\mathrm{\text{β}}}\left(t\right)=\mathbf{\Xi }(t{)}^{\mathrm{T}}\hat{\mathit{\alpha }}$. Therefore, our task reduces to estimate $\mathit{\alpha }$ in model (7) using an appropriate method.

Similarly, the key concept working with spline functions is to represent each dynamic coefficient through a regression spline basis, such as the truncated power basis, B-spline basis, or wavelet basis, among others [46, see for a review]. The B-spline basis is a powerful choice due to its simplicity and capability to capture local features of dynamic relationships. In this way, emulating the methodology described above, in order to estimate the time-varying coefficient ${\mathrm{\text{β}}}_r\left(t\right)$, we consider the following expansion based on truncated power functions:

$${\mathrm{\text{β}}}_r\left(t\right)=\sum _{\ell =0}^g{\alpha }_{r,\ell }{t}^{\ell }+\sum _{\ell =1}^{k_r}{\alpha }_{r,g+\ell }(t-{\kappa }_{\ell }{)}_{+}^g={\mathbf{\Phi }}_r(t{)}^{\mathrm{T}}{\mathit{\alpha }}_r,$$ (8)

where $x_{+}^g$ denotes the g-th power of the positive part of x, $x_{+}=max(0,x)$, and ${\kappa }_1,\dots ,{\kappa }_{k_r}$ are $k_r$ knots (in an increasing order) scattered in the range of interest. As before, note that ${\mathbf{\Phi }}_r\left(t\right)=(1,t,\dots ,t^g,(t-{\kappa }_1{)}_{+}^g,\dots ,(t-{\kappa }_{k_r}{)}_{+}^g)$ and ${\mathit{\alpha }}_r=({\alpha }_{r,0},\dots ,{\alpha }_{r,k_r+g+1})$ are $p_r\times 1$ column vectors, $p_r=k_r+g+1$, composed of basis functions evaluated at time t and unknown parameters, respectively. Then, it is simple to obtain an estimate for $\mathrm{\text{β}}\left(t\right)$ as $\widehat{\mathrm{\text{β}}}\left(t\right)=\mathbf{\Phi }(t{)}^{\mathrm{T}}\hat{\mathit{\alpha }}$, where $\mathbf{\Phi }\left(t\right)=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathbf{\Phi }}_0\right(t),{\mathbf{\Phi }}_1(t),\dots ,{\mathbf{\Phi }}_d(t\left)\right]$ and $\hat{\mathit{\alpha }}$ is an estimate of $\mathit{\alpha }$ in model (7), whose design matrix $\mathit{Z}$ is constructed from (6) using ${\mathbf{\Phi }}_r\left(t_{i,j}\right)$ instead of ${\mathbf{\Xi }}_r\left(t_{i,j}\right)$.

Finally, the reader should note that our proposal given in equation (3) is a direct reformulation of the expansion based on truncated power functions (which have been extensively investigated in the literature as in Wu and Zhang [69], for example; and therefore, we use them as a baseline), obtained by using other kind of basis functions. We argue that this is a sensible thing to do since radial functions (kernels in particular) have very desirable properties (e.g. the semiparametric regression expansion considered here based on radial functions are kernel machines within the reproducing kernel Hilbert space framework; see Harezlak et al., [23], for example) to representing (smoothing) all sorts of functional behaviors. That is why we believe that our approach constitutes a reasonable choice to represent dynamic coefficients in any TVCM. As a final comment, we note that, within a given inference paradigm, the ‘computational complexity’ of either expansion is equivalent, because each basis is composed of 1 + g + K real-valued functions.

3. Inference methods

According to the previous section, TVCM (1) is locally equivalent to standard linear model (7) in which it is required to estimate the parameter vector $\mathit{\alpha }$. In what follows, we consider both Frequentist and Bayesian approaches to carry out statistical inference on $\mathit{\alpha }$, and as a consequence, on $\mathrm{\text{β}}\left(t\right)$. First, we motivate a very popular sampling distribution as well as widely known classical bootstrap methods for performing statistical inference. Then, we consider information external to the dataset by means of a conjugate prior distribution, along with our proposal for quantifying uncertainty based on simulation and variational methods. We discuss in detail implications, challenges, and algorithms for each protocol, but focus our attention on our estimation approach embedded in the Bayesian paradigm.

3.1. Frequentist inference

From a likelihood point of view, in its simplest form, we can consider the sampling distribution

$$\mathit{y}\mid \mathit{Z},\mathit{W},\mathit{\alpha },{\sigma }^2\sim \mathsf{N}(\mathit{Z}\mathit{\alpha },{\sigma }^2{\mathit{W}}^{-1}),$$ (9)

where $\mathit{W}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[{\mathit{W}}_1,\dots ,{\mathit{W}}_n\right]$ and ${\mathit{W}}_i=w_i{\mathit{I}}_{n_i}$ is the weight matrix for the i-th experimental unit, $i=1,\dots ,n$, which is equivalent to assuming $\mathit{ϵ}\mid \mathit{W},{\sigma }^2\sim \mathsf{N}(\mathbf{0},{\sigma }^2{\mathit{W}}^{-1})$ in model (7), in a way that $ϵ\left(t\right)\sim \mathrm{G}\mathrm{P}(\mu ,\gamma )$ is a Gaussian process with $\mu \left(t\right)=0$ and $\gamma (s,t)={\sigma }^2{1}_{\{s=t\}}$ in model (1). The weights $w_1,\dots ,w_n$ are known positive constants such that $\sum _{i=1}^n{n}_i{w}_i=1$, which quantify the relative importance of experimental units. In our experiments, we follow [65] and consider the ‘subject uniform weight’, $w_i=1/\left(n{n}_i\right)$, where each subject is inversely weighted by its number of repeated measurements $n_i$, so that the subjects with fewer repeated measurements receive more weight than the subjects with more repeated measurements. The above independence assumption is convenient mathematically and works well when longitudinal data tend to be sparse. However, in our experience, and also, considering empirical evidence from both Wu and Zhang [69] and Wu and Tian [65], such an assumption can be robust to some deviations from data sparsity. Therefore, the sampling distribution (9) is an appealing choice in practice.

Under this setting, the resulting maximum likelihood estimator of $\mathit{\alpha }$ is given by

$$\hat{\mathit{\alpha }}=({\mathit{Z}}^{\mathrm{T}}\mathit{W}\mathit{Z}{)}^{-1}{\mathit{Z}}^{\mathrm{T}}\mathit{W}\mathit{y},$$ (10)

which is equivalent to the estimator obtained as a result of minimizing the weighted least squares (WLS) criterion

$$WLS\left(\mathit{\alpha }\right)=\sum _{i=1}^n\sum _{j=1}^{n_i}{w}_i(y_{i,j}-{\mathit{z}}_{i,j}^{\mathrm{T}}\mathit{\alpha }{)}^2.$$ (11)

Note that according to the Gauss-Markov theorem, the estimator provided in (10) is the best linear unbiased estimator (BLUE) of $\mathit{\alpha }$. Furthermore, it can be shown that an unbiased estimator of ${\sigma }^2$ is

$${\hat{\sigma }}^2=\frac{1}{N-p}(\mathit{y}-\mathit{Z}\hat{\mathit{\alpha }}{)}^{\mathrm{T}}\mathit{W}(\mathit{y}-\mathit{Z}\hat{\mathit{\alpha }}),$$ (12)

where $N=\sum _{i=1}^n{n}_i$ is the total number of observations, $p=\sum _{r=0}^d{p}_r$ is the expansion dimension, and $\hat{\mathit{\alpha }}$ is the estimator of $\mathit{\alpha }$ given in (10).

Under the Frequentist paradigm, confidence intervals can be computed based on either asymptotic distributions or bootstrap methods [10]. However, given the compound structure of longitudinal data, inferences that are based on asymptotic distributions are typically difficult to justify in practice, since they heavily rely on assumptions that are difficult to meet. Thus, we consider bootstrap methods, which can always be implemented based on the available data regardless of sample sizes and sampling distributions. See Appendix A.1 for a detailed description of the bootstrap algorithm. The main advantage of using a bootstrap procedure is that it does not rely on asymptotic distributions and can be used to construct confidence intervals. For instance, at a given time t, the $100(1-\alpha )\mathrm{\%}$ percentile-based confidence interval for ${\mathrm{\text{β}}}_r\left(t\right)$, $r=0,1,\dots ,d$, is given by

$$\left({\mathrm{\text{β}}}_{r,\alpha /2}\left(t\right),{\mathrm{\text{β}}}_{r,1-\alpha /2}\left(t\right)\right),$$ (13)

where ${\mathrm{\text{β}}}_{r,\alpha /2}\left(t\right)$ and ${\mathrm{\text{β}}}_{r,1-\alpha /2}\left(t\right)$ are the $\alpha /2$ and $1-\alpha /2$ quantiles of the bootstrap samples ${\mathrm{\text{β}}}_r(t{)}^{\left(1\right)},\dots ,{\mathrm{\text{β}}}_r(t{)}^{\left(B\right)}$, which are computed based on ${\mathit{\alpha }}^{\left(1\right)},\dots ,{\mathit{\alpha }}^{\left(B\right)}$ and a given set of basis functions. Other types of confidence intervals are available [e.g. normal-based confidence intervals; see 10 for a review]. We highlight that the confidence intervals given above correspond to pointwise confidence sets that only work for ${\mathrm{\text{β}}}_r\left(t\right)$ at a given time t. In most practical situations, such pointwise inferences are sufficient. However, in some studies, we might require a confidence band that simultaneously includes the true dynamic coefficient ${\mathrm{\text{β}}}_r\left(t\right)$ for a range (typically large) of time values. In such situations, we need to construct a simultaneous confidence band for ${\mathrm{\text{β}}}_r\left(t\right)$ for t within a given time interval. We refer the reader to Wu and Tian [65] for details about this matter.

Even though theoretical properties of bootstrap procedures in this setting have not been systematically investigated, we are quite confident about the coverage rates in this case, given previous simulations studies about this matter as in Hoover et al. [27] and Wu and Chiang [62] [65, see also and refereces therein for a comprehensive review and also more empirical evidence in this regard].

3.2. Bayesian inference

Under a Bayesian framework, in order to obtain an estimate for $\mathit{\alpha }$ under the sampling distribution (9), it suffices to consider the standard normal regression model

$$\mathit{y}\mid \mathit{Z},\mathit{\alpha },{\sigma }^2\sim \mathsf{N}(\mathit{Z}\mathit{\alpha },{\sigma }^2{\mathit{I}}_N),$$ (14)

since it can be easily obtained from (9) by means of a linear transformation on $\mathit{y}$ based on a Choleski factorization of ${\mathit{W}}^{-1}$ [17, see for details]. In what follows, we consider the sampling distribution (14), having in mind that a preprocessing step is required before fitting the model. In order to complete the model we choose the so-called independent Zellner's $g$-prior as a simple semiconjugate prior distribution on $\mathit{\alpha }$ and ${\sigma }^2$ to be used when there is little prior information available. Under this invariant g-prior, we let

$$\mathit{\alpha }\mid {\sigma }^2\sim \mathsf{N}(\mathbf{0},N{\sigma }^2{\mathit{I}}_N)\mathrm{a}\mathrm{n}\mathrm{d}{\sigma }^2\sim \mathsf{I}\mathsf{G}\mathsf{a}\mathsf{m}(a_{\sigma },b_{\sigma }),$$

where $a_{\sigma }$ and $b_{\sigma }$ are known hyperparameters.

Regarding the hyperparameter elicitation, we need a prior distribution to be as minimally informative as possible in the absence of real external information. We recommend setting $g=N$, $a_{\sigma }=2$, and $b_{\sigma }={\hat{\sigma }}^2$ as in (12). This choice of $g$ makes $\frac{g}{g+1}$ very close to 1, and therefore, we are practically centering $\mathit{\alpha }$ around $\hat{\mathit{\alpha }}$ a priori. Similarly, the prior distribution of ${\sigma }^2$ is also weakly centered around ${\hat{\sigma }}^2$, since $a_{\sigma }=2$ implies an infinite variance on ${\sigma }^2$ a priori. Such a distribution cannot be strictly considered as a real prior distribution, as it requires knowledge of $\mathit{y}$ to be constructed. However, it only uses a small amount of the information in $\mathit{y}$, and can be loosely thought of as the prior distribution of a researcher with unbiased but weak prior information [26, see for a discussion]. Refer to Appendix A.2 for details about the Gibbs sampler.

Even though the MCMC algorithm is straightforward in this case, inference may become impractical as the number of experimental units and the number of covariates grow. For this reason, we also implement a variational Bayes alternative that can potentially alleviate the computational burden in big data scenarios. See Appendix A.3 for details regarding the variational algorithm.

4. Location and number of knots

The smoothers' quality strongly depends on both knot locations and the number of knots. The degree of the expansion g is usually less crucial and it is often taken as 1, 2, or 3. In terms of knot location, we distinguish two widely used alternatives. The first method locates equally spaced points in the range of interest, independently of the design time points. It is usually employed when the design time points are uniformly scattered in the range of interest. The second method locates equally spaced quantiles of the design time points as knots. It locates more knots where more design time points are scattered. These methods are equivalent when the design time points are uniformly scattered. However, in our experience, the equally spaced method to locate knots is very convenient due to both its simplicity and proclivity to work well even in all sort of situations.

Another essential feature that we need to handle in practice is how to choose the smoothing parameter vector $\mathit{p}=(p_0,\dots ,p_d)$. A popular method to do so is the so called leave-one-point-out cross-validation [11, PCV,]. This approach aims to select a good smoothing parameter vector via trading-off the goodness-of-fit and the model complexity. The idea behind this criteria consists in choosing the smoothing parameter vector $\mathit{p}$ that minimizes the expression

$$PCV=\sum _{i=1}^n\sum _{j=1}^{n_i}{w}_i{\left[y_{i,j}-{\mathit{x}}_i\left(t_{i,j}{)}^{\mathrm{T}}{\widehat{\mathrm{\text{β}}}}^{(-i,j)}\right(t_{i,j})\right]}^2,$$ (15)

where ${\widehat{\mathrm{\text{β}}}}^{(-i,j)}\left(t_{i,j}\right)$ is an estimate of $\mathrm{\text{β}}\left(t_{i,j}\right)$ using the entire dataset except the j-th measurement of the i-th experimental unit. It can be shown that expression (15) is equivalent to

$$PCV=\frac{(\mathit{y}-\mathbf{A}\mathit{y}{)}^{\mathrm{T}}\mathit{W}(\mathit{y}-\mathbf{A}\mathit{y})}{{\left(1-\mathrm{t}\mathrm{r}\left(\mathbf{A}\right)/N\right)}^2},$$ (16)

where $\mathrm{t}\mathrm{r}\left(\mathbf{A}\right)$ is the trace of the smoothing matrix $\mathbf{A}$, which is a square matrix such that $\hat{\mathit{y}}=\mathbf{A}\mathit{y}$. Even though the PCV criteria does not account for the within-subject correlation effectively, it is a suitable method since the computational performance is substantially better than that provided by other alternatives [e.g. leave-one-subject-out cross-validation; see 65, for details]. As discussed in Section 7, other alternatives relying on model-based knot introduction or deletion are available.

5. Simulation study

In this section, we present two benchmark simulation scenarios to evaluate the performance of our proposed methodology and compare the different inferential methods. The first simulation scenario is inspired on an experiment originally proposed by [67,69]. The second experiment follows very closely a simulation study performed by Wu and Chiang [62], Huang et al. [29], and Wu and Tian [65].

5.1. Simulation scenario 1

In order to test our methodology with challenging real-life like datasets, and also, evaluating the robustness of the model to typical deviations from the true data generating process, we consider in this experiment a mixed-effects time-varying coefficient model with no covariate information. Such a model takes a (fixed-effects) TVCM with d = 0 and $x_0\left(t\right)\equiv 1$ for all t, and decomposes the error term into two random parts: the first one, which is subject-specific, describes the characteristics of each individual that deviate from the mean population behavior; whereas the second one, which handles directly pure random error, encompasses all those factors out of reach by the modeler (such as error measurement). Thus, we generate synthetic datasets as follows:

$$y_i\left(t\right)={\mathrm{\text{β}}}_0\left(t\right)+{\upsilon }_i\left(t\right)+{ϵ}_i\left(t\right),i=1,\dots ,n,$$ (17)

where ${\mathrm{\text{β}}}_0\left(t\right)$ is a known time-varying coefficient, ${\upsilon }_i\left(t\right)=a_{i,0}+a_{i,1}\cos \left(2\pi t\right)+a_{i,2}\sin \left(2\pi t\right)$ is a subject-specific random effect, with $(a_{i,0},a_{i,1},a_{i,2})\mathsf{N}(\mathbf{0},\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}[{\sigma }_0^2,{\sigma }_1^2,{\sigma }_2^2\left]\right)$, and ${ϵ}_i\left(t\right)$ is a zero-mean Gaussian process such that ${ϵ}_i\left(t\right)\mathsf{N}(0,{\sigma }_{ϵ}^2[1-\mathrm{e}\mathrm{x}\mathrm{p}(-0.5t-i/n){]}^2)$. We assume that ${\sigma }_1^2={\sigma }_2^2={\sigma }_{ϵ}^2={\sigma }^2$, and therefore, the correlation between repeated measurements ρ within each experimental unit is bounded by $({\sigma }_0^2+2{\sigma }^2{)}^{-1}({\sigma }_0^2-{\sigma }^2)$ and $({\sigma }_0^2+2{\sigma }^2{)}^{-1}({\sigma }_0^2+{\sigma }^2)$. We consider three cases in order to simulate different correlation levels, namely, weak within-subject correlation, ${\sigma }^2=0.01$ and ${\sigma }_0^2=0.01$, which corresponds to $0.00\le \rho \le 0.67$; medium within-subject correlation, ${\sigma }^2=0.01$ and ${\sigma }_0^2=0.04$, which corresponds to $0.50\le \rho \le 0.83$; and strong within-subject correlation, ${\sigma }^2=0.01$ and ${\sigma }_0^2=0.09$, which corresponds to $0.73\le \rho \le 0.91$.

Additionally, design times are simulated as $t_{i,j}=j/(m+1)$, $i=1,\dots ,n$, $j=1,\dots ,m$, where m is a positive integer. In order to simulate unbalanced datasets for each subject, repeated measures are randomly removed with a rate r = 0.5; thus, we expect $m(1-r)$ repeated measurements per experimental unit and $nm(1-r)$ measurements in total. In addition, number an location of knots are chosen according to the PCV criteria and the equally spaced method described in Section 4, respectively. We generated 250 datasets with two dynamic parameters, ${\mathrm{\text{β}}}_0\left(t\right)=2e^t$ and ${\mathrm{\text{β}}}_0\left(t\right)=1+\cos \left(2\pi t\right)+\sin \left(2\pi t\right)$, and also, three sample sizes, n = 25, n = 50, and n = 100. Each time, once the number and location of knots are fixed, we fitted model (1) using both radial kernel and regression spline functions setting g = 2 as a degree, with Frequentist, Bayesian, and variational methods. Setting the prior distribution as discussed in Section 3.2, Bayesian and variational estimates are based on 2, 000 samples from the posterior distribution. For Bayesian inference, we use a burn-in period of 500 samples; whereas for variational inference, we use a negligible increased in ELBO of 1e-06. Such a setting showed no evidence of lack of convergence in any case.

In this case, the performance of an estimate is measured by means of the average mean square error (AMSE), which is defined as

$$AMSE\left({\mathrm{\text{β}}}_r\right)=\sum _{i=1}^n\sum _{j=1}^{n_i}\frac{1}{n{n}_i}{\left({\mathrm{\text{β}}}_r\left(t_{i,j}\right)-{\widehat{\mathrm{\text{β}}}}_r\left(t_{i,j}\right)\right)}^2,r=0,\dots ,d.$$

Figures 1 and 2 show the AMSE distribution of ${\mathrm{\text{β}}}_0\left(t\right)=2e^t$ and ${\mathrm{\text{β}}}_0\left(t\right)=1+\cos \left(2\pi t\right)+\sin \left(2\pi t\right)$, respectively, corresponding to 250 synthetic datasets generated according to TVCM (17), in each of nine scenarios delimited by correlation level (weak, medium, and high within-subject correlation) and sample size (n = 25, n = 50, and n = 100). In general, the AMSE distribution is quite consistent across inference paradigms, which is particularly evident in the first case. Such a behavior was somewhat predictable because the number of measurements, even for the smallest datasets, is big enough (about 375 observations) to allow the likelihood to overcome the prior distribution. Even though AMSEs are also very similar across basis functions, error rates are slightly smaller in the second case when Bayesian methods along with radial functions are employed.

Figure 1.

AMSE distribution of ${\mathrm{\text{β}}}_0=2e^t$ corresponding to 250 synthetic datasets generated according to TVCM (17). Scenarios are delimited by correlation level in rows (weak, medium, and high within-subject correlation) and sample size in columns (n = 25, n = 50, and n = 100). The model is fitted each time using both radial kernel (K) and regression spline (S) functions, with Frequentist (blue), Bayesian (black), and variational (green) methods.

Figure 2.

AMSE distribution of ${\mathrm{\text{β}}}_0\left(t\right)=1+\cos \left(2\pi t\right)+\sin \left(2\pi t\right)$ corresponding to 250 synthetic datasets generated according to TVCM (17). Scenarios are delimited by correlation level in rows (weak, medium, and high within-subject correlation) and sample size in columns (n = 25, n = 50, and n = 100). The model is fitted each time using both radial kernel (K) and regression spline (S) functions, with Frequentist (blue), Bayesian (black), and variational (green) methods.

Furthermore, Frequentist inferences are equivalent regardless of the smoothing approach. Also, we observe that the variational approximation to the posterior distribution under the Bayesian paradigm is very precise. This is the case because the mean field assumption is breaking negligible correlations in the posterior distribution. Moreover, as expected, estimates are more consistent as the sample size increases since the variability of the AMSE distribution decreases and its center remains stable. On the contrary, such variability increases as the within-subject correlation becomes higher, which strongly suggests that despite our approach not taking into account within-subject correlation directly, estimates are robust enough to produce accurate results.

5.2. Simulation scenario 2

In this experiment, we generate synthetic datasets as follows:

$$y_i\left(t\right)={\mathrm{\text{β}}}_0\left(t\right)+{\mathrm{\text{β}}}_1\left(t\right)x_{1,i}+{\mathrm{\text{β}}}_1\left(t\right)x_{1,i}+{ϵ}_i\left(t\right),i=1,\dots ,n,$$ (18)

where

$$\begin{array}{rl}{\mathrm{\text{β}}}_0\left(t\right)& =3.5+6.5\sin \left(t\pi /60\right),\\ {\mathrm{\text{β}}}_1\left(t\right)& =-0.2-1.6\cos \left(\right(t-30\left)\pi /60\right),\mathrm{a}\mathrm{n}\mathrm{d}\\ {\mathrm{\text{β}}}_3\left(t\right)& =0.25-0.0074\left(\right(30-t)/10{)}^3\end{array}$$

are three nonlinear time-varying coefficients, associated with time-invariant independent covariates $x_{1,i}$ and $x_{2,i}$ such that $x_{1,i}\mathsf{B}\mathsf{e}\mathsf{r}\left(0.5\right)$ and $x_{2,i}\mathsf{N}(0,4^2)$, and finally, ${ϵ}_i\left(t\right)$ is Gaussian process with zero mean and covariance function $\gamma (s,t)=0.0625\exp (-|s-t|)$. Also, we assume that subjects are independent of each other and scheduled to be observed at m = 31 equally spaced time design points $0,1,\dots ,m$. However, at each given time point, a subject has 50% probability to be randomly missing. As before, we generated 250 datasets with three sample sizes, n = 25, n = 50, and n = 100, and once again, we fitted the model (1) using exactly the same settings as in Section 6.1.

In order to measure the performance of an estimate fairly, we define the mean absolute deviation of errors (MADE) as

$$MADE=\sum _{r=0}^d\sum _{i=1}^n\sum _{j=1}^{n_i}\frac{1}{n{n}_i}\frac{|{\mathrm{\text{β}}}_r\left(t_{i,j}\right)-{\widehat{\mathrm{\text{β}}}}_r\left(t_{i,j}\right)|}{\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\left({\mathrm{\text{β}}}_r\right)},r=0,\dots ,d.$$

Figure 3 shows the MADE distribution along with the dynamic parameter Bayesian estimates of the coefficients, corresponding to 250 synthetic datasets generated according to TVCM (17), in each of three scenarios delimited by sample size (n = 25, n = 50, and n = 100). Again, it is quite obvious the effect of the sample size on the error rates. This behavior is evident from both the decreasing variability of the MADE distribution and the consistent estimates of the coefficients around their true value. Clearly, these simulation results demonstrate that both estimation approaches independently of the inference paradigm, provide reasonably good estimators, at least for interior time points.

Figure 3.

MADE distribution and dynamic parameter Bayesian estimates of ${\mathrm{\text{β}}}_0\left(t\right)$, ${\mathrm{\text{β}}}_1\left(t\right)$, and ${\mathrm{\text{β}}}_2\left(t\right)$ (bold lines are the true coefficient functions and gray lines correspond to ten randomly selected estimates), corresponding to 250 synthetic datasets generated according to TVCM (18). Sample sizes are displayed in columns (n = 25, n = 50, and n = 100). The model is fitted each time using both radial kernel (K) and regression spline (S) functions, with Frequentist (blue), Bayesian (black), and variational (green) methods.

5.3. Execution time

Following a suggestion given by one of the referees, here, we provide a comparison in terms of execution time between our two competing approaches to carry out Bayesian inference, namely, MCMC methods and variational methods, i.e. simulation-based methods and optimization-based methods.

In this spirit, Table 1 contains mean running times (in milliseconds) using a single core of an AMD A12-9730P processor, when generating 2,000 samples of the posterior distribution for the model based on radial kernel functions (our proposal) using both MCMC and variational methods, for each synthetic dataset under all simulation settings considered in Section 5. We see that the variational approach clearly greatly outperforms its simulation-based counterpart in terms of execution time. Such an effect is particularly clearer for bigger samples sizes, where variational methods can even be 45 faster than MCMC methods. Lastly, note that MCMC execution times increase notoriously as the sample size grow, whereas variational execution times remain roughly constant.

Table 1.

Mean running times (in milliseconds) using a single core of an AMD A12-9730P processor, when generating 2,000 samples of the posterior distribution for the model based on radial functions (our proposal) using both MCMC (MC) and variational (V) methods, for each synthetic dataset under all simulation settings considered in Section 5.

	n = 25		n = 50		n = 100
Setting	MC	V	MC	V	MC	V
Scenario 1a	128.60	7.93	160.17	6.07	347.17	7.13
Scenario 1b	151.30	10.33	204.10	7.10	458.77	9.87
Scenario 2	280.43	12.03	381.67	12.37	581.30	12.77

6. Illustrations

6.1. Case study 1

Our first illustration is based on an AIDS clinical trial developed by the AIDS Clinical Trials Group¹ (ACTG). In this group, [18] evaluated two different 4-drug regimens containing indinavir with either efavirenz or nelfinavir for the treatment of 517 patients with advanced HIV disease (i.e.patients with high HIV-1 RNA levels and low CD4 cell counts). This study was a randomized, open-label study and initially planned to last 72 weeks but later increased to 120 weeks beyond the enrollment of the last subject. The randomization was carried out by using a permuted block design and stratified according to CD4 cell count and HIV-1 RNA level at screening, as well as previous antiretroviral experience. In addition, clinical assessments, HIV-1 RNA measurements, CD4 cell counts, and routine laboratory tests were performed before study entry, at the time of study entry, at weeks 4 and 8, and every 8 weeks thereafter. More details about design, subjects, treatments and outcome measurements of this study are given in [18].

Here, we model the CD4 cell count, which is an essential marker for assessing immunologic response of an antiviral regimen, in one of the two treatment arms. This group includes 166 patients treated with highly active antiretroviral therapy for 120 weeks, during which CD4 cell counts were monitored along with other important markers. Patients might not exactly follow the designed schedule, and missing clinical visits for CD4 cell measurements frequently occurred, which makes this dataset¹ (named ACTG 388) a typical longitudinal dataset. The main goal in this study is to model the mean CD4 cell count trajectories over the treatment period for the entire treatment arm.

In this specific group of patients, the number of CD4 cell count measurements per patient varies from 1 to 18 observations, and the CD4 cell count ranges from 0 to 1,364. Figure 4 shows CD4 cell counts (in logarithmic scale) for each one of the n = 166 patients during the 120 weeks of treatment. Even though individual cell counts are quite noisy and there is evidence of some atypical trajectories associated with low counts, this plot suggests that cell counts tend to stabilize around the middle of the treatment. Thus, it is not possible to ensure that the antiviral treatment was quite effective since there are no apparent reasons to believe that CD4 cell counts profiles are either increasing continuously or at least remaining stable.

Figure 4.

ACTG 388 data: CD4 cell counts (logarithmic scale).

In order to estimate the mean trajectory of CD4 cell counts over the treatment period, we fit the TVCM

$$y_{i,j}={\mathrm{\text{β}}}_0\left(t_{i,j}\right)+{ϵ}_{i,j},j=1,\dots ,n_i,i=1,\dots ,n,$$

under a Bayesian approach with the prior distribution given in Section 3.2, employing both Gaussian kernel and regression spline functions with g = 2, where $y_{i,j}$ is the CD4 cell count (in logarithmic scale) of the j-th measurement of the i-th patient, and ${\mathrm{\text{β}}}_0\left(t\right)$ is a unknown time-varying parameter describing the mean dynamic trend of CD4 cell counts over time. Again, the number and location of knots are chosen according to the PCV criteria and the equally spaced method described in Section 4, respectively. According to this criteria, the optimal number of knots are $k_0^K=4$ and $k_0^S=8$, respectively. Once the number and location of knots are fixed, Bayesian estimates are based on 2, 000 samples from the posterior distribution after a burn-in period of 500 iterations. Convergence was monitored by tracking the variability of the joint distribution of data and parameters using the multi-chain procedure discussed in Gelman and Rubin [22].

Estimates of ${\mathrm{\text{β}}}_0\left(t\right)$ (in logarithmic and natural scale) along with their corresponding 95% credible intervals are shown in Figure 5. Both estimates are very similar, except for the small jump at the beginning of the treatment exhibited by the regression spline-based estimate. Such trajectories, which are very precise since the credible intervals are quite narrow, reveal that the mean CD4 cell counts increase quite sharply during the first 40 weeks of treatment, and continue to increase at a slower rate until about week 100, and then dropped towards the end of the study. This makes evident that under this antiviral regimen, the overall CD4 counts increased dramatically during the first 40 weeks, but the effect of the drug therapy fades over time and completely disappeared after about week 100, when cell counts begin to drop. Almost identical results were obtained by Wu and Zhang [69]; the only difference is that they concluded that the inflection point after which the CD4 cell count started to drop was on week 110. A residual analysis (not shown here) indicates that the model fits the data adequately because there are no signs of particular shapes, patterns or significant deviations.

Figure 5.

${\mathrm{\text{β}}}_0\left(t\right)$ estimates (solid lines) along with 95% credible intervals (dotted lines) obtained through radial kernel functions (blue) and regression spline functions (black). Individual trajectories are displayed in gray.

6.2. Case study 2

We consider another AIDS clinical study carried out by the ACTG. In this case, [32] evaluated a highly active antiretroviral therapy containing zidovudine, lamivudine, and ritonavir, for the treatment of patients with moderately advanced HIV-1 infection. This study was designed to ascertain if administration of highly active antiretroviral therapy to patients with moderately advanced HIV-1 infection was associated with evidence of immunologic restoration. More details about design, subjects, treatments and outcome measurements of this study are given in Lederman et al. [32].

The viral load (plasma HIV RNA level) and immunologic response (CD4 cell counts) are negatively correlated and their relationship is approximately linear during antiviral treatments. However, their relationship may not be a constant during the whole period of treatment [34]. Thus, the main goal in this study is to model the dynamic relationship between the viral load and the immunologic response over the treatment period, which plays an essential role in evaluating the antiviral therapy. Fifty-three patients were enrolled in the trial, out of which n = 46 received the treatment for at least 9 of the first 12 weeks and were therefore eligible for analysis. Intolerance of the treatment regimen was responsible for almost all treatment discontinuations. Patients might not exactly follow the designed schedule, and missing clinical visits occurred frequently, which makes this dataset³ (named ACTG 315) quite unbalanced. Additional analyses of this and other trajectories, as well as more scientific findings of the study, can be found in [9,32,34,67].

After starting treatment, the plasma HIV RNA level and the CD4 cell count were measured simultaneously (both of them were reported in logarithmic scale) at days 0, 2, 7, 10, 14, 28, 56, and 86. Figure 6 shows the corresponding cell counts. The number of repeated measurements per subject varies from 4 to 8, and the total number of observations is 328. Simple linear regressions of cell counts against plasma HIV RNA level at each visit (not shown here) evidence that the slope associated with the viral load changes over time because some days the slope is significantly different from zero. This simple observation motivates fitting a TVCM in order to characterize and quantify such relationship.

Figure 6.

ACTG 315 data: CD4 cell counts (logarithmic scale).

Once again, under a Bayesian setting, employing both Gaussian kernel and regression spline functions with g = 2, we fit the TVCM given by

$$y_{i,j}={\mathrm{\text{β}}}_0\left(t_{i,j}\right)+{\mathrm{\text{β}}}_1\left(t_{i,j}\right)x_{1,i}\left(t_{i,j}\right)+{ϵ}_{i,j},j=1,\dots ,n_i,i=1,\dots ,n,$$

where $y_{i,j}$ and $x_{1,i}\left(t_{ij}\right)$ are the viral load (in logarithmic scale) and the CD4 cell count (also in logarithmic scale) associated with the j-th measurement of the i-th patient, respectively. The time-varying coefficient ${\mathrm{\text{β}}}_1\left(t\right)$ characterizes the dynamic relationship between the viral load and the immunologic response over the treatment period. Interestingly, according to PCV criteria, the optimal number of knots in both cases are $k_0^K=k_0^S=k_1^K=k_1^S=1$. Following exactly the same setting as in the case study #1, the results we report are based on 2,000 samples obtained after a burn-in period of 500 iterations.

Figure 7 shows estimates of the dynamic slope. Both estimated trajectories are quite smooth. Here, we see a significant negative correlation between viral load and immunologic response at the beginning of the treatment. Then, the relationship consistently attenuates until reaching zero about the fourth week. At that point, the negative correlation gradually strengthens again, and continuously to do so towards the end of the treatment period considered in this analysis. Almost identical results were obtained by [67].

Figure 7.

${\mathrm{\text{β}}}_1\left(t\right)$ estimates (solid lines) along with 95% credible intervals (dotted lines) obtained through radial kernel functions (blue) and regression spline functions (black).

6.3. Goodness-of-fit and predictive performance

The modeling literature has largely focused on both Akaike Information Criteria (AIC) and Bayesian Information Criteria (BIC) as a tool for model selection (e.g. see [69]). However, under a Bayesian setting, the BIC is inappropriate for hierarchical models since it underestimates the complexity of the model. An alternative to BIC that addresses this issue is the Deviance Information Criterion (DIC), $DIC=-2\log p(\mathit{y}\mid \widehat{\mathbf{{\rm Y}}})+2p_{DIC}$, where $\widehat{\mathbf{{\rm Y}}}$ is the posterior mean of model parameters and $p_{DIC}=2\log p(\mathit{y}\mid \widehat{\mathbf{{\rm Y}}})-2E\left[\log p(\mathit{y}\mid \mathbf{{\rm Y}})\right]$ is the model complexity [see 21, for a discussion]. Table 2 presents the DIC for the TVCMs fitted above. It is clear the the TVCM based on radial kernel functions is preferred, specially for the ACTG 388 dataset.

Table 2.

DIC and AMSE to assess goodness-of-fit and out-sample predictive performance, respectively, of both radial kernel and regression spline functions.

	Kernel		Spline
Dataset	DIC	AMSE	DIC	AMSE
ACTG 388	64,257.1	0.761	65,306.7	0.759
ACTG 315	2,960.9	0.758	2,955.8	0.811

On the other hand, in order to compare the ability of each alternative to predict missing observations, we evaluate their out-of-sample predictive performance by means of a cross-validation (CV) experiment. Thus, for each combination of dataset and model, we performed an L-fold CV in which L randomly selected subsets of roughly equal size in the dataset are treated as missing and then predicted using the rest of the data. We summarize our findings in Table 2, where we report the average mean square error (AMSE) corresponding to the prediction of missing measurements in the datasets. In this context, the AMSE is a measure of how well a given model is capable of predicting missing observations. We can see from this table that both alternatives have comparable predictive capabilities.

7. Discussion

In this paper, we review two simple but powerful alternatives based on linear expansions to estimate time-varying coefficients: a new approach using radial kernel functions and a more standard method using regression spline functions. We framed the estimation procedure under both Frequentist and Bayesian inference paradigms using bootstrap techniques, Gibbs sampling and a variational Bayes method. From an empirical perspective, we provide two simulations studies. These experiments strongly suggest that either combination of basis representation and inference approach are comparable and mostly equivalent. From a practical perspective, the first case study shows that the overall CD4 counts increased dramatically during the first 40 weeks under a specific anti-viral treatment, but the effect of the drug therapy faded over time and completely disappeared after about week 100. On the other hand, the second case study evidences a strong negative correlation between the viral load and the immunologic response at the beginning of an anti-viral treatment, and then shows evidence of a weak correction about the fifth week, when gradually strengthened again and reached the largest value at the end of the treatment period.

As part of the revision process, one of the referees suggested that, in order to avoid inconsistencies in terms of exposition, the model needed to be presented in a general fashion as in a mixed-effects time-varying coefficient model (see Section 5.1). Even though we agree on the convenience of working with a more general model, we consider that our approach should be treated in terms of a ‘standard’ TVCM as in Equation (1) since it makes exposition straightforward, given that our main contribution rely on an estimation protocol based on radial functions, along with inference strategies according to the Bayesian paradigm. Nonetheless, we exhort the reader to pursue such an extension employing the ideas discussed in this manuscript.

The estimation protocol presented here is susceptible of many extensions. For instance, to avoid the curse of dimensionality, the model can be extended to account for longitudinal inhomogeneity of varying coefficients via Bayesian basis selection or adaptive knot selection which is an integral part of the data generating mechanism. Another interesting extension involves the incorporation of specific working correlation matrices in the probabilistic structure of the random error using more convoluted covariance functions. Extensions to more complex situations, including multivariate or spatial data are also possible, and are the subject of future work.

Appendix A. Algortihms for Inference

A.1. Bootstrap algorithm

Since all subjects are assumed to be independent, a natural sampling scheme consists in resampling the entire repeated measurements of each subject with replacement from the original dataset. The bootstrap samples ${\mathbf{{\rm Y}}}^{\left(1\right)},\dots ,{\mathbf{{\rm Y}}}^{\left(B\right)}$, where ${\mathbf{{\rm Y}}}^{\left(b\right)}=({\mathit{\alpha }}^{\left(b\right)},{\sigma }^{\left(b\right)})$, $b=1,\dots ,B$, can be generated using the following bootstrap algorithm:

1. Randomly select n bootstrap subjects with replacement from the original dataset, and put together the bootstrap dataset as ${\mathcal{D}}^{\left(b\right)}=\left\{\right(y_{i,j}^{\ast },{\mathit{x}}_{i,j}^{\ast },t_{i,j}^{\ast },):j=1,\dots ,n_i,i=1,\dots ,n\}.$ Note that the entire set of repeated measurements for some subjects may appear multiple times in the bootstrap dataset.

2. Compute $N^{\left(b\right)}$, ${\mathit{y}}^{\left(b\right)}$, ${\mathit{Z}}^{\left(b\right)}$, and ${\mathit{W}}^{\left(b\right)}$ based on ${\mathcal{D}}^{\left(b\right)}$.

3. Compute ${\mathit{\alpha }}^{\left(b\right)}$ as in (10) based on ${\mathit{y}}^{\left(b\right)}$, ${\mathit{Z}}^{\left(b\right)}$, ${\mathit{W}}^{\left(b\right)}$.

4. Compute ${\sigma }^{\left(b\right)}$  as in (12) based on ${\mathit{y}}^{\left(b\right)}$, ${\mathit{Z}}^{\left(b\right)}$, ${\mathit{W}}^{\left(b\right)}$, ${\mathit{\alpha }}^{\left(b\right)}$.

5. Repeat the previous steps B times.

A.2. MCMC algorithm

The posterior distribution of the parameters can be explored using MCMC algorithms in which the posterior distribution is approximated using dependent but approximately identically distributed samples ${\mathbf{{\rm Y}}}^{\left(1\right)},\dots ,{\mathbf{{\rm Y}}}^{\left(B\right)}$, where ${\mathbf{{\rm Y}}}^{\left(b\right)}=({\mathit{\alpha }}^{\left(b\right)},{\sigma }^{\left(b\right)})$, $b=1,\dots ,B$. In this case, the joint posterior distribution is given by:

$$\begin{array}{rl}p(\mathit{\alpha },{\sigma }^2\mid \mathit{Z},\mathit{y})& \propto ({\sigma }^2{)}^{-N/2}\mathrm{e}\mathrm{x}\mathrm{p}\left\{-\frac{1}{2{\sigma }^2}(\mathit{y}-\mathit{Z}\mathit{\alpha }{)}^{\mathrm{T}}(\mathit{y}-\mathit{Z}\mathit{\alpha })\right\}\\ & \times ({\sigma }^2{)}^{-p/2}\mathrm{e}\mathrm{x}\mathrm{p}\left\{-\frac{1}{2N{\sigma }^2}{\mathit{\alpha }}^{\mathrm{T}}\mathit{\alpha }\right\}\times ({\sigma }^2{)}^{-(a_{\sigma }+1)}\mathrm{e}\mathrm{x}\mathrm{p}\left\{-\frac{b_{\sigma }}{{\sigma }^2}\right\},\end{array}$$ (A1)

where $p=\sum _{r=0}^d{p}_r$ is the expansion dimension. The MCMC algorithm iterates over the full conditional distributions of the model parameters $\mathit{\alpha }$ and ${\sigma }^2$ by generating a new state ${\mathbf{{\rm Y}}}^{(b+1)}$ from a current state ${\mathbf{{\rm Y}}}^{\left(b\right)}$ as follows:

1. Choose an initial value for $\mathit{\alpha }$, say ${\mathit{\alpha }}^{\left(0\right)}$.

2. Update $({\sigma }^2{)}^{(b+1)}$ and ${\mathit{\alpha }}^{(b+1)}$ until convergence:

   • i.
     Sample $({\sigma }^2{)}^{(b+1)}$ from

     $$\mathsf{I}\mathsf{G}\mathsf{a}\mathsf{m}\left(a_{\sigma }+\frac{N}{2}+\frac{p}{2},b_{\sigma }+\frac{1}{2}\parallel \mathit{y}-\mathit{Z}{\mathit{\alpha }}^{\left(b\right)}{\parallel }^2+\frac{1}{2N}\parallel {\mathit{\alpha }}^{\left(b\right)}{\parallel }^2\right).$$
   • ii.
     Sample ${\mathit{\alpha }}^{(b+1)}$ from

     $$\mathsf{N}\left({\left({\mathit{Z}}^{\mathrm{T}}\mathit{Z}+\frac{1}{N}{\mathit{I}}_p\right)}^{-1}{\mathit{Z}}^{\mathrm{T}}\mathit{y},({\sigma }^2{)}^{(b+1)}{\left({\mathit{Z}}^{\mathrm{T}}\mathit{Z}+\frac{1}{N}{\mathit{I}}_p\right)}^{-1}\right).$$

Posterior summaries along with point and interval estimates can be approximated based on the Monte Carlo samples. As before, for a given t, the $100(1-\alpha )\mathrm{\%}$ percentile-based credible interval for ${\mathrm{\text{β}}}_r\left(t\right)$, $r=0,1,\dots ,d$, can be computed as in (13).

A.3. Variational Bayes algorithm

The Markov chain defined in the previous section is guaranteed to converge eventually to the posterior distribution $p(\mathit{\alpha },{\sigma }^2\mid \mathit{Z},\mathit{y})$ given in (A1). Here, we consider the problem of finding a function $q(\cdot )$ in a family of functions closest to the posterior distribution $p(\cdot )$, according to a given dissimilarity measure. This idea is known as variational Bayes [see 44, for a review].

Briefly, the main idea can be summarized as follows. Let $\mathit{\theta }=({\theta }_1,\dots ,{\theta }_k)$ be a set of parameters and $p(\mathit{\theta }\mid \mathit{X})$ its posterior distribution after data $\mathit{X}$ have been observed. The purpose consists in finding a function $q\left(\mathit{\theta }\right)$ that minimizes the Kullback–Leibler divergence respect to $p(\mathit{\theta }\mid \mathit{X})$:

$$\mathrm{K}\mathrm{L}\left(p||q\right)=\int q\left(\mathit{\theta }\right)\log \frac{q\left(\mathit{\theta }\right)}{p(\mathit{\theta }\mid \mathit{X})}\mathrm{d}\mathit{\theta }=\log p\left(\mathit{X}\right)-\int q\left(\mathit{\theta }\right)\log \frac{p(\mathit{\theta },\mathit{X})}{q\left(\mathit{\theta }\right)}\mathrm{d}\mathit{\theta }.$$

Thus, minimizing $\mathrm{K}\mathrm{L}\left(p||q\right)$ is equivalent to maximizing

$${\mathrm{E}}_{q\left(\mathit{\theta }\right)}\left[\log \frac{p(\mathit{\theta },\mathit{X})}{q\left(\mathit{\theta }\right)}\right]=\int q\left(\mathit{\theta }\right)\log \frac{p(\mathit{\theta },\mathit{X})}{q\left(\mathit{\theta }\right)}\mathrm{d}\mathit{\theta },$$

which is known as evidence lower bound (ELBO). Furthermore, if $q\left(\mathit{\theta }\right)$ is assumed to satisfy the mean field assumption

$$q\left(\mathit{\theta }\right)=\prod _{i=1}^k{q}_i\left({\theta }_i\right),$$

where the $q_i\left({\theta }_i\right)$ are marginal variational densities, the solution $q^{\ast }\left(\mathit{\theta }\right)$ satisfies

$$\log q_i^{\ast }\left({\theta }_i\right)\propto {\mathrm{E}}_{q\left({\mathit{\theta }}_{-i}\right)}\left[\log p({\theta }_i\mid \mathit{X},{\mathit{\theta }}_{-i})\right],$$

with ${\mathit{\theta }}_{-i}=({\theta }_1,\dots ,{\theta }_{i-1},{\theta }_{i+1},\dots ,{\theta }_k)$, which leads to a coordinate optimization algorithm.

In this case, the product density approximation to $p(\mathit{\alpha },{\sigma }^2\mid \mathit{Z},\mathit{y})$ is $q(\mathit{\alpha },{\sigma }^2)=q\left(\mathit{\alpha }\right)q\left({\sigma }^2\right).$ The optimal densities take the form

$$\log q\left(\mathit{\alpha }\right)\propto E_{q\left({\sigma }^2\right)}\left[\log p(\mathit{\alpha }\mid \mathit{Z},\mathit{y},{\sigma }^2)\right]\mathrm{a}\mathrm{n}\mathrm{d}\log q\left({\sigma }^2\right)\propto E_{q\left(\mathit{\alpha }\right)}\left[\log p({\sigma }^2\mid \mathit{Z},\mathit{y},\mathit{\alpha })\right].$$

Then,

$$q\left(\mathit{\alpha }\right)\propto \mathrm{e}\mathrm{x}\mathrm{p}\left\{-\frac{1}{2}\left[{\mathit{\alpha }}^{\mathrm{T}}{E}_{q\left({\sigma }^2\right)}\left[{\sigma }^{-2}\right]\left({\mathit{Z}}^{\mathrm{T}}\mathit{Z}+\frac{1}{N}{\mathit{I}}_p\right)\mathit{\alpha }-2{\mathit{\alpha }}^{\mathrm{T}}{E}_{q\left({\sigma }^2\right)}\left[{\sigma }^{-2}\right]{\mathit{Z}}^{\mathrm{T}}\mathit{y}\right]\right\}$$

which we can recognize as a member of the normal family $\mathsf{N}({\mathit{m}}^{\ast },{\mathit{V}}^{\ast })$ where

$${\mathit{V}}^{\ast }={\left(E_{q\left({\sigma }^2\right)}{\sigma }^{-2}\right)}^{-1}{\left({\mathit{Z}}^{\mathrm{T}}\mathit{Z}+\frac{1}{N}{\mathit{I}}_p\right)}^{-1}\mathrm{a}\mathrm{n}\mathrm{d}{\mathit{m}}^{\ast }={\mathit{V}}^{\ast }\left(E_{q\left({\sigma }^2\right)}{\sigma }^{-2}\right){\mathit{Z}}^{\mathrm{T}}\mathit{y}.$$

Similar arguments lead to

$$q\left({\sigma }^2\right)\propto ({\sigma }^2{)}^{-(a_{\sigma }+N/2+p/2+1)}\mathrm{e}\mathrm{x}\mathrm{p}\left\{-\frac{1}{{\sigma }^2}\left(b_{\sigma }+\frac{1}{2}{E}_{q\left(\mathit{\alpha }\right)}\parallel \mathit{y}-\mathit{Z}\mathit{\alpha }{\parallel }^2+\frac{1}{2N}{E}_{q\left(\mathit{\alpha }\right)}\parallel \mathit{\alpha }{\parallel }^2\right)\right\},$$

which we can recognize as a member of the inverse-gamma family $\mathsf{I}\mathsf{G}\mathsf{a}\mathsf{m}(a^{\ast },b^{\ast })$ where

$$\begin{array}{rl}{a}^{\ast }& =a_{\sigma }+N/2+p/2\mathrm{a}\mathrm{n}\mathrm{d}\\ b^{\ast }& =b_{\sigma }+\frac{1}{2}\left(\parallel \mathit{y}{\parallel }^2-2{\mathit{y}}^{\mathrm{T}}\mathit{Z}{E}_{q\left(\mathit{\alpha }\right)}\mathit{\alpha }+E_{q\left(\mathit{\alpha }\right)}{\mathit{\alpha }}^{\mathrm{T}}\left({\mathit{Z}}^{\mathrm{T}}\mathit{Z}+\frac{1}{N}{\mathit{I}}_p\right){\mathbb{E}}_{\mathit{\alpha }}\mathit{\alpha }\right.\\ & +\left.\mathrm{t}\mathrm{r}\left[\left({\mathit{Z}}^{\mathrm{T}}\mathit{Z}+\frac{1}{N}{\mathit{I}}_p\right){Var}_{q\left(\mathit{\alpha }\right)}\mathit{\alpha }\right]\right).\end{array}$$

Hence, the algorithm for obtaining the parameters in $q\left(\mathit{\alpha }\right)$ and $q\left({\sigma }^2\right)$ is the following:

1. Initialize $b^{\ast }>0$.

2. Repeat until the increase in ELBO is negligible:

   • i.
     ${\mathit{V}}^{\ast }\leftarrow \frac{b^{\ast }}{a^{\ast }}({\mathit{Z}}^{\mathrm{T}}\mathit{Z}+\frac{1}{N}{\mathit{I}}_p{)}^{-1}$
   • ii.
     ${\mathit{m}}^{\ast }\leftarrow \frac{a^{\ast }}{b^{\ast }}{\mathit{V}}^{\ast }{\mathit{Z}}^{\mathrm{T}}\mathit{y}$
   • iii.
     $b^{\ast }\leftarrow b_{\sigma }+\frac{1}{2}(\parallel \mathit{y}{\parallel }^2-2{\mathit{y}}^{\mathrm{T}}\mathit{Z}{\mathit{m}}^{\ast }+{\mathit{m}}^{\ast \mathrm{T}}({\mathit{Z}}^{\mathrm{T}}\mathit{Z}+\frac{1}{N}{\mathit{I}}_p){\mathit{m}}^{\ast }+\mathrm{t}\mathrm{r}[({\mathit{Z}}^{\mathrm{T}}\mathit{Z}+\frac{1}{N}{\mathit{I}}_p){\mathit{V}}^{\ast }\left]\right)$

The lower bound is given by

$$ELBO\left(q\right)=E_q\log p(\mathit{y}\mid \mathit{Z},\mathit{\alpha },{\sigma }^2)+E_q\log p(\mathit{\alpha },{\sigma }^2)-E_q\log q\left(\mathit{\alpha }\right)-E_q\log q\left({\sigma }^2\right).$$

Using standard results, the evaluation of each term is straightforward and results in:

$$\begin{array}{rl}ELBO\left(q\right)& =-\frac{1}{2}(N\log 2\pi +p\log N-p)+a_{\sigma }\log b_{\sigma }-\log \mathrm{\Gamma }\left(a_{\sigma }\right)\\ & +a^{\ast }(1+\log b^{\ast }-2\psi (a^{\ast }\left)\right)+\log \mathrm{\Gamma }\left(a^{\ast }\right)+2(\log b^{\ast }-\psi (a^{\ast }\left)\right)+\frac{1}{2}\log |{\mathit{V}}^{\ast }|\\ & -\frac{a^{\ast }}{b^{\ast }}\left[b_{\sigma }+\frac{1}{2}\left(\parallel \mathit{y}{\parallel }^2-2{\mathit{y}}^{\mathrm{T}}\mathit{Z}{\mathit{m}}^{\ast }+{\mathit{m}}^{\ast \mathrm{T}}\left({\mathit{Z}}^{\mathrm{T}}\mathit{Z}+\frac{1}{N}{\mathit{I}}_p\right){\mathit{m}}^{\ast }\right.\right.\\ & +\left.\left.\mathrm{t}\mathrm{r}\left[\left({\mathit{Z}}^{\mathrm{T}}\mathit{Z}+\frac{1}{N}{\mathit{I}}_p\right){\mathit{V}}^{\ast }\right]\right)\right],\end{array}$$

where $\psi (\cdot )$ is the digamma function.

Posterior summaries can be obtained via Monte Carlo simulation using standard random number generation routines.

Appendix B. Notation

Matrices and vectors with entries consisting of subscripted variables are denoted by a boldfaced version of the letter for that variable. For example, $\mathit{x}=(x_1,\dots ,x_n)$ denotes an $n\times 1$ column vector with entries $x_1,\dots ,x_n$. We use $\mathbf{0}$ and $\mathbf{1}$ to denote the column vector with all entries equal to 0 and 1, respectively, and $\mathit{I}$ to denote the identity matrix. A subindex in this context refers to the corresponding dimension; for instance, ${\mathit{I}}_n$ denotes the $n\times n$ identity matrix. The transpose of a vector $\mathit{x}$ is denoted by ${\mathit{x}}^{\mathrm{\top }}$; analogously for matrices. Moreover, if $\mathit{X}$ is a square matrix, we use $\mathrm{t}\mathrm{r}\left(\mathit{X}\right)$ to denote its trace and ${\mathit{X}}^{-1}$ to denote its inverse. The norm of $\mathit{x}$, given by $\sqrt{{\mathit{x}}^{\mathrm{\top }}\mathit{x}}$, is denoted by $\parallel \mathit{x}\parallel$.

Disclosure statement

No potential conflict of interest was reported by the author(s).