A Bayesian regression model for multivariate functional data

Abstract

In this paper we present a model for the analysis of multivariate functional data with unequally spaced observation times that may differ among subjects. Our method is formulated as a Bayesian mixed-effects model in which the fixed part corresponds to the mean functions, and the random part corresponds to individual deviations from these mean functions. Covariates can be incorporated into both the fixed and the random effects. The random error term of the model is assumed to follow a multivariate Ornstein–Uhlenbeck process. For each of the response variables, both the mean and the subject-specific deviations are estimated via low-rank cubic splines using radial basis functions. Inference is performed via Markov chain Monte Carlo methods.

1. Introduction

The term functional data analysis describes non-parametric analyses of longitudinal data which focus on the curves themselves as the basic unit of data. Some of the goals of functional data analysis include exploring individual variation of curves from an overall mean function, and modeling the dependence of the curves on covariates. The mean function, as well as the subject-specific functions are estimated non-parametrically. In this paper, we propose a method for analyzing multivariate functional data with unequally spaced observation times that may differ among subjects. It is assumed that all variables are observed at the same time points. Fitting a regression model with a multivariate response may be done by either fitting a separate regression for each of the response variables or by fitting a single regression with all response variables simultaneously. The latter may be advantageous if the error terms corresponding to each variable are correlated. Thus, fewer observations may be required to obtain reliable non-parametric function estimates compared to fitting each regression separately and ignoring the correlation. This has been shown to be the case in seemingly unrelated regression (see for example Smith and Kohn, 2000).

Our method is formulated as a Bayesian mixed-effects model in which the fixed part corresponds to the mean functions, and the random part corresponds to individual deviations from these mean functions. Covariates can be incorporated into both the fixed and the random effects. The random error term of the model is assumed to follow a first order continuous-time multivariate autoregression, also known as a multivariate Ornstein–Uhlenbeck process. For each of the response variables, both the mean and the subject-specific deviations are estimated via low-rank cubic splines using radial basis functions. Inference is performed via Markov chain Monte Carlo methods.

Our model is closest in spirit to the functional mixed effects model of Guo (2002), where the fixed and random effects are modeled by cubic smoothing splines. However, Guo’s model accommodates only a univariate response variable, and does not allow correlated error terms. It can be fit either via standard mixed effects software or by Kalman filtering. Inference and model selection are based on a generalized maximum likelihood ratio test. Baladandayuthapani et al. (2008) have proposed a Bayesian model for spatially correlated functional data analysis. The smoothing technique they use is similar to ours, but their emphasis is on spatial correlation rather than on temporal correlation. Smith and Kohn (2000) consider multivariate non-parametric regression using the seemingly unrelated regression approach. They show that if the error terms of the regression equations are correlated, better non-parametric estimates of the regression functions are obtained by accounting for this correlation compared to fitting separate regressions ignoring the correlation. It is noted that Smith and Kohn (2000) consider multivariate non-parametric regression, not functional data analysis. In functional data analysis, each individual subject has its own function which needs to be estimated for each variable. Smith and Kohn (2000) only estimate a single function for each variable.

The Ornstein–Uhlenbeck process has been used before in various contexts. Unlike most diffusion processes, its transition density is available in closed form, which results in a closed-form expression for the likelihood function. Jones (1993), Chapter 8, uses a state-space approach to parameter estimation. Sy et al. (1997) present a model for multivariate repeated measures which allows unequally spaced observations by using the multivariate integrated Ornstein–Uhlenbeck process. The fixed and random effects in their model have parametric forms. Markov chain Monte Carlo methods for inference on the Ornstein–Uhlenbeck process (univariate or multivariate) parameters have also been proposed. A recent review of estimation for discretely observed diffusion processes is given in Beskos et al. (2006). Golightly and Wilkinson (2006) discuss Bayesian inference for non-linear multivariate diffusions. A number of authors have assumed a common spacing between the observed times. Blackwell (2003) takes this common spacing to be the most frequently occurring interval between observations. De la Cruz-Mesía and Marshall (2003, 2006) discuss the univariate Ornstein–Uhlenbeck process and take the common spacing to be the average time difference between two consecutive observations.

The example for application of our methodology is taken from a recent psychiatric study comparing psychotherapy to pharmacotherapy carried out at the University of Pittsburgh and the University of Pisa, Italy (Frank et al., 2008). This study sought differential baseline predictors of response to these two forms of treatment of major depression. Here, we examine the interaction effect of treatment group with Lifetime Depressive Spectrum symptoms (LDS; (Cassano et al., 1997)) in 252 patients entering the study in an acutely depressive episode. Levels of depression are determined by the clinician-administered Hamilton Rating Scale for Depression (HRSD) and the Quick Inventory for Depression Self-report (QIDS). These two scales were given to patients at baseline and again roughly weekly over the course of each subject’s acutely depressive episode.

Our main contribution in this paper is accommodating multivariate functional data including covariates and accounting for correlation across variables and time using smoothing techniques in combination with modeling the error term via the multivariate Ornstein–Uhlenbeck process.

The rest of the paper is organized as follows. In Section 2, we describe our model, the prior distributions and the sampling scheme. Section 3 provides results of a simulation study. Section 4 discusses an application, and Section 5 ends with a brief discussion.

2. The model, priors and sampling scheme

2.1. The model

Suppose y_i(t_ij) is a p × 1 vector of response variables on subject i at time t_ij, i = 1, …, n, j = 1, …, m_i, and consider the model

$${\mathit{y}}_i(t_{\mathit{ij}})=X_{\mathit{ij}}\mathit{\mu }(t_{\mathit{ij}})+Z_{\mathit{ij}}{\mathit{g}}_i(t_{\mathit{ij}})+{\mathit{\delta }}_i(t_{\mathit{ij}}).$$ (1)

In (1), $\mathit{\mu }(t)={({\mathit{\mu }}_1^{\prime }(t),\dots ,{\mathit{\mu }}_p^{\prime }(t))}^{\prime }$ and ${\mathit{g}}_i(t)={({\mathit{g}}_{i1}^{\prime }(t),\dots ,{\mathit{g}}_{\mathit{ip}}^{\prime }(t))}^{\prime }$, where μ_k(t) = (μ_k1(t), …, μ_kr (t))′ and g_ik(t) = (g_ik1(t), …, g_iks(t))′ are an r × 1 vector of fixed functions, and an s × 1 vector of random functions, respectively, for k = 1, …, p. Associated with μ(t_ij) is an r × 1 covariate vector x_ij, and with g_i(t_ij) – an s × 1 covariate vector z_ij, such that $X_{\mathit{ij}}=I_p\otimes {\mathit{x}}_{\mathit{ij}}^{\prime }$ and $Z_{\mathit{ij}}=I_p\otimes z_{\mathit{ij}}^{\prime }$, where I_p is a p × p identity matrix, and ⊗ denotes the Kronecker product. We have assumed here that the p response variables share the same covariates.

Before proceeding to specify the p × 1 vector of random errors, δ_i(t_ij), we give an example which is a special case of model (1). Suppose p = 2, r = 2 and s = 1, with x_ij taking values in {(1 0)′, (1 1)′}, and z_ij = 1 with corresponding functions μ₁(t) = (μ₁₁(t), μ₁₂(t))′, μ₂(t) = μ₂₁(t), μ₂₂(t))′, g_i1(t) and g_i2(t). In this case, model (1) reduces to

$$\begin{array}{l}{y}_{i1}(t_{\mathit{ij}})={\mu }_{11}(t_{\mathit{ij}})+x_{\mathit{ij}2}·{\mu }_{12}(t_{\mathit{ij}})+g_{i1}(t_{\mathit{ij}})+{\delta }_{i1}(t_{\mathit{ij}})\\ y_{i2}(t_{\mathit{ij}})={\mu }_{21}(t_{\mathit{ij}})+x_{\mathit{ij}2}·{\mu }_{22}(t_{\mathit{ij}})+g_{i2}(t_{\mathit{ij}})+{\delta }_{i2}(t_{\mathit{ij}}),\end{array}$$

where x_ij2, the second entry of x_ij, can take the values 0 or 1. In this example, there are two groups of subjects – control (x_ij2 = 0) and treatment (x_ij2 = 1). Each group has its own mean curve for each of the two variables, and individual deviations from these curves are accommodated by the random functions g_i1(t) and g_i2(t), i = 1, …, n.

The error term in model (1) is assumed to follow a multivariate Ornstein–Uhlenbeck process. More specifically, δ_i(t) satisfies the stochastic differential equation

$$d{\mathit{\delta }}_i(t)=-A{\mathit{\delta }}_i(t)+\mathit{Bd}{\mathit{W}}_i(t),$$

where A and B are p × p matrices of full rank common to all i = 1, …, n, and W_i(t) is the p-dimensional Wiener process. Three properties of the Ornstein–Uhlenbeck process (Gardiner, 1983, pp. 110–111) which will be useful in what follows are

1. The Ornstein–Uhlenbeck process will be stationary provided the eigenvalues of A have positive real parts.

2. The solution Σ to the matrix equation AΣ + ΣA′ = BB′ is the stationary variance–covariance matrix of the process.

3. In the stationary state, the covariance of δ_i(t) and δ_i(s), for s < t, is

$$\mathrm{Cov}({\mathit{\delta }}_i(t),{\mathit{\delta }}_i(s))=\exp \{-A(t-s)\}\sum .$$ (2)

Let Δt_ij = t_ij − t_i,j−1 for j = 1, …, m_i, where t_i0 = 0. The transition density of the Ornstein–Uhlenbeck process is given by

$$p({\mathit{\delta }}_i(t_{\mathit{ij}})\mid {\mathit{\delta }}_i(t_{i,j-1}),\Delta t_{\mathit{ij}})\propto {|{\Omega }_{\Delta t_{\mathit{ij}}}|}^{-1/2}\exp \{-\frac{1}{2}{\mathit{\gamma }}_{t_{\mathit{ij}}}^{\prime }{\Omega }_{\Delta t_{\mathit{ij}}}^{-1}{\mathit{\gamma }}_{t_{\mathit{ij}}}\},$$ (3)

where γ_tij = δ_i(t_ij) − exp(−AΔt_ij)δ_i(t_i,j−1) and Ω_Δtij = Σ − exp(−AΔt_ij) Σ exp(−A′ Δt_ij).

The functions μ_kl(t) and g_ikm (t), k = 1, …, p, l = 1, …, r, m = 1, …, s, i = 1, …, n, are modeled as cubic splines using low-rank radial basis functions (French et al., 2001; Ruppert et al., 2003). In Section 2.2 we review briefly non-parametric function estimation.

2.2. Non-parametric function estimation

The functions μ_kl(t) and g_ikm(t) are estimated non-parametrically. In this section we explain the basis function approach which is used in turn in Section 2.3 to estimate these functions. For simplicity, we focus on scatterplot smoothing, with observations (x_i, y_i), i = 1, …, n. The description in this section is based on French et al. (2001), Ruppert et al. (2003) and Crainiceanu et al. (2005). Consider the model

$$y_i=f(x_i)+{\epsilon }_i,$$

where E (ε_i) = 0, i = 1, …, n, and f is an unknown smooth function. A linear spline basis function can be expressed as (x − κ)₊ = max (0, x − κ), where κ is a knot. Any linear combination of linear spline basis functions 1, x, (x − κ₁)₊, …, (x − κ_K)₊ is a piecewise linear function with knots at κ₁, …, κ_K. The function f may thus be expressed as

$$f(x)={\beta }_0+{\beta }_{1}x+\sum _{k=1}^K{u}_k{(x-{\kappa }_k)}_{+},$$ (4)

where the u_ks are the coefficients of the basis functions. We comment later in this section on the value of K.

To understand how the spline model (4) can be used for fitting a non-parametric curve to data, consider Fig. 1, which displays in the top panel whip-shaped data similar to the example in Ruppert et al. (2003). The left half of the data exhibits linear behavior while curvature is apparent on the right-hand side. The bottom panel presents the basis functions used in the spline model. In particular, the equally-spaced knots are 0.50, 0.55, 0.60, …, 0.95. Comparing the two panels of Fig. 1, it is quite easy to see that a linear combination of the basis functions in the bottom panel should be able to capture the data structure in the top panel.

Fig. 1.

Top: data with fitted curve. Bottom: the basis functions.

In general, any structure can be accommodated by placing basis functions at additional knots. To automate the process, two main approaches are commonly taken. One approach is automatic knot selection which can be carried out via Bayesian variable selection. Specifically, a large number of knots are placed at either equally spaced locations or at specific percentiles of the covariate, and an indicator variable is attached to each knot (see for example Thompson and Rosen, 2008). The indicator value is 1 if a knot is to be retained at a given location or 0 if the knot should be removed from that location. In a Bayesian MCMC procedure, the indicator variables are sampled from at each iteration. The other approach is to retain all the knots but to constrain their influence. This can be accomplished by penalized spline regression or equivalently by using a linear mixed effects model formulation. In this paper, we use the latter approach in a Bayesian framework. In both approaches, the value of K is not crucial, as long as it is not too small. Typically, 30–40 knots are sufficient for medium-sized datasets. Instead of the linear spline representation (4), we use in this paper the low rank thin-plate spline representation

$$f(x)={\beta }_0+{\beta }_{1}x+\sum _{k=1}^K{u}_k{|x-{\kappa }_k|}^3.$$ (5)

Using cubic radial basis functions tends to result in a more aesthetically appealing fit, compared to that of the truncated-line basis (Fig. 2), and may lead to faster convergence of the MCMC algorithm. The penalized spline approach prevents overfitting by adding a roughness penalty. Specifically, the minimization criterion is

$$\sum _{i=1}^n{(y_i-f(x_i))}^2+\frac{1}{\mathrm{\lambda }}{\mathit{\theta }}^{\prime }D\mathit{\theta },$$ (6)

where θ = (β₀, β₁, u₁, …, u_K)′, λ is the smoothing parameter and D is a known positive-definite penalty matrix. For thinplate splines, the matrix D is given by

$$D=\left(\begin{array}{cc}{0}_{2\times 2}& 0_{2\times K}\\ 0_{K\times 2}& {\Omega }_K\end{array}\right),$$

where the (k, l)th element of Ω_K is |κ_k − κ_l|³. From the structure of the matrix D it is clear that only the u_ks are penalized. Let y = (y₁, …, y_n)′, X = [1 x_i]_1≤i≤n, Z_K = [|x_i − κ₁|³ … |x_i − κ_K|³]_1≤i≤n. Dividing (6) by ${\sigma }_{\epsilon }^2$ and expressing the penalty term explicitly as a function of Ω_K results in

$$\frac{1}{{\sigma }_{\epsilon }^2}{\Vert \mathit{y}-X\mathit{\beta }-Z_K\mathit{u}\Vert }^2+\frac{1}{\lambda {\sigma }_{\epsilon }^2}{\mathit{u}}^{\prime }{\Omega }_K\mathit{u},$$ (7)

where β = (β₀, β₁)′ and u = (u₁, …, u_K)′ are considered fixed and random parameters, respectively. The solution to (7) is equal to the best linear unbiased predictor (BLUP) in the linear mixed model

$$\mathit{y}=X\mathit{\beta }+Z_K\mathit{u}+\mathit{\epsilon },\mathrm{Cov}(\mathit{u})={\sigma }_u^2({\Omega }_K^{-1/2}){({\Omega }_K^{-1/2})}^{\prime },$$ (8)

where ${\Omega }_K^{-1/2}$ is based on the singular value decomposition. Note that Ω_K is not a positive definite matrix so it is not a proper covariance matrix; however, French et al. (2001) show that the smooth fit is not affected by this fact. Let $\mathit{b}={\Omega }_K^{1/2}\mathit{u}$ and $Z=Z_K{\Omega }_K^{-1/2}$, then the mixed model (8) is equivalent to

$$\mathit{y}=X\mathit{\beta }+Z\mathit{b}+\mathit{\epsilon },\mathrm{Cov}\left(\begin{array}{c}\mathit{b}\\ \mathit{\epsilon }\end{array}\right)=\left(\begin{array}{cc}{\sigma }_b^2{I}_K& \mathbf{0}\\ \mathbf{0}& {\sigma }_{\epsilon }^2{I}_n\end{array}\right).$$ (9)

Fig. 2.

The data with a fitted curve. Left: truncated lines basis. Right: Cubic radial basis functions.

In a Bayesian framework, prior distributions need to be placed on all the model parameters.

2.3. Estimating μ_kl(t) and g_ikm(t)

To estimate the functions μ_kl(t) and g_ikm(t), k = 1, …, p, l = 1, …, r, m = 1, …, s, i = 1, …, n, we use the basis function approach described in Section 2.2. In particular, let κ₁, …, κ_K be K knots obtained as sample quantiles of t_ij, i = 1, …, n, j = 1, …, m_i, and let Λ_K = [|κ_k − κ_k′|³]_{1≤k,k′≤K} be a K × K matrix. Let ${\mathit{\varphi }}_{\mathit{ij}}^{\prime }=(\begin{array}{cc}1& t_{\mathit{ij}}\end{array}),{\mathit{\xi }}_{\mathit{ij}}^{\prime }=({|t_{\mathit{ij}}-{\kappa }_1|}^3,\dots ,{|t_{\mathit{ij}}-{\kappa }_K|}^3)$, and ${\mathit{\psi }}_{\mathit{ij}}^{\prime }={\mathit{\xi }}_{\mathit{ij}}^{\prime }{\Lambda }_K^{-1/2}$, where ${\Lambda }_K^{1/2}$ is obtained via the singular value decomposition. The vectors ϕ_ij and ψ_ij are basis functions evaluated at t_ij and are used to model the linear part and the non-linear part, respectively, of the fixed and random functions. In particular, μ_kl(t) and g_ikm(t) can be evaluated at t_ij by

$${\mu }_{\mathit{kl}}(t_{\mathit{ij}})={\mathit{\varphi }}_{\mathit{ij}}^{\prime }{\mathit{\beta }}_{\mathit{kl}}+{\mathit{\psi }}_{\mathit{ij}}^{\prime }{\mathit{\nu }}_{\mathit{kl}}\mathrm{and}{g}_{\mathit{ikm}}(t_{\mathit{ij}})={\mathit{\varphi }}_{\mathit{ij}}^{\prime }{\mathit{w}}_{\mathit{ikm}}+{\mathit{\psi }}_{\mathit{ij}}^{\prime }{\mathit{u}}_{\mathit{ikm}}$$ (10)

for k = 1, …, p, l = 1, …, r, m = 1, …, s and i = 1, …, n. In (10), β_kl, ν_kl,w_ikm and u_ikm are unknown parameter vectors.

2.4. Priors on the basis function coefficients and the variance components

We place the following prior distributions on β_kl, ν_kl, w_ikm and u_ikm, k = 1, …, p, l = 1, …, r, m = 1, …, s and i = 1, …, n.

1. ${\mathit{\beta }}_{\mathit{kl}}\stackrel{\mathit{ind}}{\sim }N(\mathbf{0},{\sigma }_{{\beta }_{\mathit{kl}}}^2{I}_2)$, where I₂ is a 2 × 2 identity matrix, and ${\sigma }_{{\beta }_{\mathit{kl}}}^2$ is a large known value.

2. ${\mathit{\nu }}_{\mathit{kl}}\stackrel{\mathit{ind}}{\sim }N(\mathbf{0},{\sigma }_{{\upsilon }_{\mathit{kl}}}^2{I}_K)$, where I_K is a K × K identity matrix, and K is the number of knots.

3. ${\mathit{w}}_{\mathit{ikm}}\stackrel{\mathit{ind}}{\sim }N(\mathbf{0},\mathrm{diag}({\sigma }_{w_{\mathit{km}0}}^2,{\sigma }_{w_{\mathit{km}1}}^2))$.

4. ${\mathit{u}}_{\mathit{ikm}}\stackrel{\mathit{ind}}{\sim }N(\mathbf{0},{\sigma }_{u_{\mathit{km}}}^2{I}_K)$.

Similar prior distributions on the coefficients of the basis functions were used by Durbán et al. (2005). Note that the variances of the elements of w_ikm are different while those of the elements of u_ikm are all the same. This is merely for computational convenience to avoid an additional (K − 1) parameters. The priors on the variance components ${\sigma }_{{\upsilon }_{\mathit{kl}}}^2$, k = 1, …, p, l = 1, …, r, are independent inverse gamma distributions with densities

$$p({\sigma }_{{\upsilon }_{\mathit{kl}}}^2)\propto {({\sigma }_{{\upsilon }_{\mathit{kl}}}^2)}^{-({\alpha }_1+1)}\exp (-b_1/{\sigma }_{{\upsilon }_{\mathit{kl}}}^2),$$

where a₁ and b₁ are known small values reflecting vague knowledge on ${\sigma }_{\mathit{\upsilon kl}}^2$. The priors on ${\sigma }_{w_{\mathit{km}0}}^2$, ${\sigma }_{w_{\mathit{km}1}}^2$ and ${\sigma }_{u_{\mathit{km}}}^2$ are similar inverse gamma distributions. Recently, a number of authors (see for example Gelman, 2006) have proposed alternative prior distributions for variance components which may exhibit superior behavior to that of inverse gamma distributions. However, Zhao et al. (2006) reported good performance of inverse gamma priors in the case of non-parametric regression, provided the hyperparameters are not too small. Specifically, hyperparameter values of 0.01 worked well, whereas values of 0.001 behaved erratically.

2.5. Priors on the Ornstein–Uhlenbeck parameters

The Ornstein–Uhlenbeck process parameters are the matrix A and the matrix C = BB′. Both matrices consist of parameters which are constrained to satisfy certain conditions. In particular, as mentioned in Section 2, the stationarity condition requires the real parts of the eigenvalues of A to be positive. Also, the matrix C is required to be symmetric and positive definite. Imposing the constraints directly on the elements of these matrices would be difficult. Instead, we first express each of these matrices in an appropriate decomposition and then place prior distributions on the parameters of the decomposition factors. This is a much easier task, as the factor parameters are either unconstrained or constrained to be non-negative. To place a prior on A, we express it as A = SΨS⁻¹, where S is a matrix of linearly independent eigenvectors, and Ψ is a diagonal matrix of real positive eigenvalues. This parameterization, used also by Sy et al. (1997) for the bivariate Ornstein–Uhlenbeck process, satisfies the stationarity condition mentioned above for the Ornstein–Uhlenbeck process. Aït-Sahalia (2008) discusses identifiability related to A and expresses it as a lower triangular matrix with positive diagonal elements. Kessler and Rahbek (2004) discuss identifiability issues in the case of equidistant observation times. The matrix S is parameterized as S = (s_ij), i, j = 1, …, p, with unit diagonal elements. Independent N (0, ${\sigma }_a^2$) priors are placed on the off-diagonal elements of S, and on the logarithms of the diagonal elements of Ψ.

The matrix C is symmetric and positive definite. To place priors on its elements which satisfy the symmetry and positive definiteness, we first express the matrix C as a modified Cholesky factorization, C = LDL′, where L is unit lower triangular, and D is diagonal. This approach was used for example in Smith and Kohn (2002) and in Rosen and Stoffer (2007). The emphasis of Rosen and Stoffer (2007) is on estimation in the frequency domain for multivariate time series observed at equally spaced time points. The priors on the off-diagonal elements of L are taken to be independent N(0, ${\sigma }_L^2$) with a fixed large value of ${\sigma }_L^2$. The priors placed on log(D_i), where D_i is the ith diagonal element of D, are independent N(0, ${\sigma }_D^2$) with a fixed large value of ${\sigma }_D^2$.

2.6. The sampling scheme

Let ${\mathit{\theta }}_k={\mathit{(}{\beta }_{k1}^{\prime },{\mathit{\nu }}_{k1}^{\prime },\dots ,{\mathit{\beta }}_{\mathit{kr}}^{\prime },{\mathit{\nu }}_{\mathit{kr}}^{\prime })}^{\prime }$, for k = 1, …, p, and let $\mathit{\theta }={({\mathit{\theta }}_1^{\prime },\dots ,{\mathit{\theta }}_p^{\prime })}^{\prime }$. Similarly, let ${\mathit{\eta }}_{\mathit{ik}}={({\mathit{w}}_{\mathit{ik}1}^{\prime },{\mathit{u}}_{\mathit{ik}1}^{\prime },\dots ,{\mathit{w}}_{\mathit{iks}}^{\prime },{\mathit{u}}_{\mathit{iks}}^{\prime })}^{\prime }$, for k = 1, …, p and ${\mathit{\eta }}_i={({\mathit{\eta }}_{i1}^{\prime },\dots ,{\mathit{\eta }}_{\mathit{ip}}^{\prime })}^{\prime }$ for i = 1, …, n. The sampling scheme consists of the following stages. More details are given in the Appendix.

1. Initialize θ, η_i, i = 1, …, n, and the variance components by fitting p mixed effects models, for k = 1, …, p. Initialize A and C by maximizing numerically the log conditional joint posterior distribution of A and C.

2. Generate θ from its full conditional posterior distribution, which is multivariate normal.

3. For each i, i = 1, …, n, generate η_i from its full conditional posterior distribution, which is multivariate normal.

4. For k = 1, …, p, l = 1, …, r, m = 1, …, s, generate the variance components ${\sigma }_{{\upsilon }_{\mathit{kl}}}^2$, ${\sigma }_{w_{\mathit{km}0}}^2$, ${\sigma }_{w_{\mathit{km}1}}^2$ and ${\sigma }_{u_{\mathit{km}}}^2$ from their full conditional posterior distributions, which are inverse gamma.

5. Generate A from its full conditional posterior distribution. Since this distribution is not standard, we use a Metropolis step with a multivariate normal proposal density centered at the current value of A. The variance–covariance matrix of this normal proposal is based on the inverse of the estimated negative hessian of the log conditional posterior distribution.

6. Generate C from its full conditional posterior distribution using a Metropolis step.

3. Simulations

In this section, we explore by simulation the potential improvement in curve fitting when modeling the correlation structure of multivariate functional data rather than ignoring it. Specifically, we examine improvements in mean squared error for the individual subject-level functions. For this purpose we generated 100 datasets, with each dataset consisting of observations, without covariates, on n = 50 subjects. The number of observations per subject is m_i = 2 + w_i, where w_i is a Poisson random variable with expectation 5, giving an average of 7 observation times per subject. The observation times themselves were independently generated from a uniform distribution on the interval [0, m_i]. For each subject, there are p = 3 response variables with overall subject mean functions chosen to represent a variety of possible relationships. The first true mean function is μ₁(t) = 7 sin(−.5t), which exhibits low frequency variation on the range of t. Note that the second subscript on μ₁(t) was dropped, since there are no covariates in our simulation setting. The second true mean function is μ₂(t) = 10ϕ(t; 1.5, .3) + 6ϕ(t; 4, .6), where ϕ(t; a, b) is a univariate normal density with mean a and standard deviation b. The third true mean function is μ₃(t) = 2 sin(−t), which has higher frequency oscillations on the range of t. Let f_ik(t) = μ_k(t) = g_ik(t), k = 1, 2, 3, be the individual subject functions, where we have again dropped the covariate subscript. In particular,

$$\begin{array}{l}{f}_{i1}(t)=a_{i1}\sin (-.5t)+a_{i2}\\ f_{i2}(t)=b_{i1}\varphi (t;1.5,.3)+b_{i2}\varphi (t;4,.6)\\ f_{i3}(t)=c_{i1}\sin (-t)+c_{i2},\end{array}$$ (11)

where a_i1 ~ N (7, .5), a_i2 ~ N (0, .2), b_i1 ~ N(10, .25), b_i2 ~ N (6, .25), c_i1 ~ N (2, .5) and c_i2 ~ N (0, .2). Here, N (a, b) indicates the univariate normal distribution with mean a and standard deviation b. The observations y_i(t_ij) were obtained by drawing the random coefficients a_i1, a_i2, b_i1, b_i2, c_i1, c_i2, evaluating the equations in (11) at time t_ij and adding δ_i(t_ij), which was in turn generated according to a multivariate Ornstein−Uhlenbeck error process with parameter values

$$A=\left(\begin{array}{rrr}2& -0.6& -0.6\\ -0.6& 2& -0.6\\ 0& 0& 2\end{array}\right)\mathrm{and}C=\left(\begin{array}{rrr}15& 0& 0\\ 0& 15& 0\\ 0& 0& 15\end{array}\right).$$

These settings result in fairly noisy data with cross-correlations ranging from .25 to .47 among the three variables when evaluated at $\overline{\Delta }{t}_i=\frac{1}{m_i}{\sum }_{j=1}^{m_i}(t_{\mathit{ij}}-t_{i,j-1})\approx 1$. The cross-correlation matrix is obtained from the cross-covariance (2), evaluated at Δ̄t_i. Plots of these mean functions along with one randomly generated dataset can be seen in Fig. 3.

Fig. 3.

Mean functions (heavy lines) and data from one randomly generated dataset (light lines) with 50 individual subjects.

Our model was fitted four times for each dataset, once for each univariate outcome separately (thereby ignoring across-variable correlation) and then to all three outcomes simultaneously. The sampling scheme was run for 10,000 iterations per dataset, with a burn-in period of 5000. Median estimates  and Ĉ of the A and C matrices across all 100 multivariate fits were given by

$$\widehat{A}=\left(\begin{array}{rrr}2.91& -0.72& -1.31\\ -0.38& 2.29& -0.63\\ 0.02& -0.07& 3.03\end{array}\right)\mathrm{and}\widehat{C}=\left(\begin{array}{rrr}18.82& -0.81& -1.67\\ -0.81& 16.81& -0.19\\ -1.67& -0.19& 19.59\end{array}\right).$$

To assess the quality of the resulting estimates of the three mean functions, we calculated the average squared difference between the function estimates and the true mean functions at the unique observation times, t₁ <, …, < t_M. For the kth function this was computed by

$$\mathit{MS}{E}_k^{(1)}=\frac{1}{M}{\sum _{m=1}^M({\widehat{\mu }}_k(t_m)-{\mu }_k(t_m))}^2,$$

where μ̂_k(·) is the fitted mean function for the kth response variable. This was done for all three univariate fits, as well as for the joint trivariate fit. Boxplots of the resulting $\mathit{MS}{E}_k^{(1)},k=1,2,3$, are displayed in Fig. 4. These boxplots show that the separate univariate fittings and the joint multivariate fitting resulted in little difference in the mean squared error for the first variable but lower mean squared error for the second two. For the univariate fits, the median estimates for $\mathit{MS}{E}_k^{(1)}$ were .244, 1.19, and .206 for k = 1, 2, 3, respectively. For the multivariate fits, the corresponding median estimates of $\mathit{MS}{E}_k^{(1)}$ were .225, 1.03, and .188, for k = 1, 2, 3, respectively. Paired t-tests of the log $\mathit{MS}{E}_k^{(1)}$ showed no significant difference of log $\mathit{MS}{E}_1^{(1)}$ corresponding to the univariate and multivariate fits (t = 1.0, p = 0.16) but that log $\mathit{MS}{E}_2^{(1)}$ and log $\mathit{MS}{E}_3^{(1)}$ were significantly lower for the multivariate fits (t = 2.3, p = 0.01 and t = 3.4, p < .0005, respectively). To assess the quality of the fitted individual subject functional estimates, we calculated the average squared difference between the true individual subject functions and their estimates from the model at the measured observation times. For the kth variable, this mean squared error for the individual subject functions was computed by

$$\mathit{MS}{E}_k^{(2)}=\frac{1}{m}\sum _{i=1}^n{\sum _{j=1}^{m_i}({\widehat{f}}_{\mathit{ik}}(t_{\mathit{ij}})-f_{\mathit{ik}}(t_{\mathit{ij}}))}^2,$$

where $m=\sum _{i=1}^n{m}_i$ and f̂_ik(·) is the fitted function for the ith subject’s kth response. Boxplots of the resulting mean squared errors for the individual subject functions are displayed in Fig. 5. The mean squared errors for the subject functions show a similar pattern as for the overall mean. The median mean squared errors for each of the three outcome variables for the univariate fits were .702, 3.02, and .445, respectively. The median values for the corresponding multivariate fits were .676, 2.584, and .351. Thus, there was a 15%–20% reduction in mean squared error for the individual functions for the last two variables when accounting for the multivariate covariance among them. Again, there was no significant difference of log $\mathit{MS}{E}_1^{(2)}$ between the multivariate and univariate fits (t = 0.33, p = 0.39) but the multivariate fits had significantly lower log $\mathit{MS}{E}_1^{(2)}$ for k = 2, 3 (t = 4.6, p < 0.0005, t = 9.2, p < 0.0005, respectively). One possible reason why the first variable exhibits no improvement in mean squared errors is that the low-frequency variation of the corresponding mean function renders it easier to fit and hence it is less important to borrow information across variables.

Fig. 4.

Boxplots of $\mathit{MS}{E}_k^{(1)}$ for the posterior means μ̂_k(·), k = 1, 2, 3, based on 100 simulated samples. Univariate and multivariate fits are denoted by U and M, respectively.

Fig. 5.

Boxplots of $\mathit{MS}{E}_k^{(2)}$ for the posterior means f̂_ik(·), k = 1, 2, 3, i = 1, …, 25, based on 100 simulated samples. Univariate and multivariate fits are denoted by U and M, respectively.

4. Application

As described in Section 1, we apply our methodology to the results of a randomized clinical trial conducted at the University of Pittsburgh and the University of Pisa, Italy (Frank et al., 2008). Despite decades of clinical trial experience in major depression, there is only limited understanding of which patients with major depressive disorder respond better to psychotherapy or to pharmacotherapy. This clinical trial compares the effects of psychotherapy (129 subjects) vs. pharmacotherapy (123 subjects). For clarity, Fig. 6 shows the trajectories corresponding to 25 subjects only. We limit the current analysis to the first 12 weeks after baseline, at which about 95% of the subjects were still on study. Our methodology, which allows for non-linear estimation of time courses, can accommodate the subject trajectories which are clearly non-linear. In addition, our methodology accounts for the possibility of non-linear effects of baseline covariates over time. Of particular interest is the identification of baseline subject characteristics which differentially predict treatment response in the two groups. The treatment response was change over time in two depression scales, the clinician-administered Hamilton Rating Scale for Depression (HRSD) and the Quick Inventory of Depression - Self Report (QIDS). These measures were collected more than once per week on average, though there was variation both within and between patients in the actual timing and number of measurements, with a mean of 11.2 measurement times per subject over the course of treatment. The HRSD scores ranged from 0–31 with a median of 10, and the QIDS scores ranged from 0–26 with a median of 6. In both measures, higher values indicate more depressive symptoms. Both measures were log transformed and standardized before running the analyses. A Lifetime Depression Spectrum (LDS) score was assessed on each patient at baseline; this gives an omnibus measure of depressive symptomatology over a patient’s lifetime (Cassano et al., 1997). In this example, we considered the LDS score to be a pre-treatment covariate with potentially differential effects on treatment outcomes for the two treatment groups. To explore this possibility, treatment group, LDS score, and their interaction were entered as time-varying fixed effects into our model with responses HRSD and QIDS entered as bivariate dependent variables. A time-varying random intercept was also included in the model. In the notation of Section 2, x_ij = (x_ij1, x_ij2, x_ij3, x_ij4)′ and z_ij = 1, where x_ij1 = 1, x_ij2 is a group indicator (equal to 1 if subject i received psychotherapy and zero otherwise), x_ij3 is the ith subject’s LDS score, and x_ij4 = x_ij2x_ij3. The sampling scheme described in Section 2.6 was run for 10,000 iterations with a burn-in period of 5000 iterations. The estimated parameters of the Ornstein–Uhlenbeck process are

$$\widehat{A}=\left(\begin{array}{ll}5.75& -3.88\\ -4.40& 7.04\end{array}\right)\mathrm{and}\widehat{C}=\left(\begin{array}{cc}3.03& 0.05\\ 0.05& 3.50\end{array}\right).$$

Fig. 6.

HRSD subject trajectories (left panel) and QIDS subject trajectories (right panel) for acutely depressed subjects. For clarity, only 25 subject trajectories are displayed. Scores are standardized to have zero mean and unit variance. Trajectories were truncated at 12 weeks.

The estimated time-varying functional coefficients μ̂_k(t) = (μ̂_k1(t), (μ̂_k2(t), (μ̂_k3(t), (μ̂_k4(t))′, = k 1, 2, for the HRSD and QIDS responses are plotted in Figs. 7 and 8, respectively. Solid lines correspond to the multivarite fits; for comparison, the univariate fits appear in dashed lines. As can be seen in these plots, the multivariate fits show little evidence for a treatment group effect on HRSD, but evidence for a slight difference in the QIDS at around 3 weeks. However, there is a significant effect of LDS score on both outcomes, so that higher lifetime depression spectrum predicts worse outcomes over the first eight weeks or so. The interaction term is insignificant for HRSD and marginally significant in the 2–8 week time period for the QIDS responses. The effect of the interaction on responses is that LDS score is less predictive of poor QIDS response in the psychotherapy group than in the pharmacotherapy group. In general, the pointwise 95% credible intervals are wider for the univariate fits. While the functional coefficient estimates were substantially similar, the interaction coefficient for both models were not significant, i.e., the pointwise 95% credible intervals contained zero for the entire time course.

Fig. 7.

Time-varying functional coefficients for HRSD responses. The solid lines are μ̂_1l(t), 1 ≤ l ≤ 4, and their corresponding pointwise 95% credible intervals. The dashed lines are the analogous estimates and credible intervals corresponding to the univariate fits. Upper left panel: μ̂₁₁(t). Upper right panel: μ̂₁₂(t). Lower left panel: μ̂₁₃(t). Lower right panel: μ̂₁₄(t).

Fig. 8.

Time-varying functional coefficients for QIDS responses. The solid lines are μ̂_2l(t), 1 ≤ 1 ≤ 4, and their corresponding 95% credible intervals. The dashed lines are the analogous estimates and credible intervals corresponding to the univariate fits. Upper left panel: μ̂₂₁(t). Upper right panel: μ̂₂₂(t). Lower left panel: μ̂₂₃(t). Lower right panel: μ̂₂₄(t).

5. Discussion

In this paper we have devised a regression model apropriate for multivariate functional responses with unequallyspaced observation times. Efficiency may be gained by fitting a single regression with all response variables simultaneously, as opposed to fitting regression models for each functional response separately. This is especially true if the error terms corresponding to each variable are correlated. In our formulation, the random error terms of the model were assumed to follow a multivariate Ornstein–Uhlenbeck process. Through this formulation we were able to extend the seemingly unrelated regression framework to the unequally-spaced multivariate functional data context.

The model we proposed uses a Bayesian mixed-effects approach, where the fixed part corresponds to the mean functions, and the random part corresponds to individual deviations from these mean functions. Covariates were allowed as either fixed or random effects. For each of the response variables, both the mean and the subject-specific deviations were estimated via low-rank cubic splines using radial basis functions. Thus mean and subject-specific deviation from the mean were allowed to vary smoothly as a function of time. Inference was performed via Markov chain Monte Carlo methods.

We demonstrated the improvement in efficiency that is possible by using this model in simulations which show mean squared-error is lower for the full multivariate algorithm compared to fitting each of the functional responses unvariately, thereby ignoring the across-variable correlation. This seems especially important when the mean functions are wiggly, so that borrowing information across multiple responses becomes more important.

Finally, the utility of this methodology was demonstrated by application to a real-life psychiatric dataset looking at the relationship of multiple depression measures over time in a clinical trial. Here, using a multivariate approach resulted in narrower posterior confidence bands.

We plan future research to extend the multivariate functional model to mixed discrete and continuous functional outcome data. We also plan to develop methods for the joint analysis of mulitvariate functional data and time-to-event data.

Appendix

Starting values for θ, η_is and σ²

Let $D_{{\varphi }_{\mathit{ij}}}^r=I_r\otimes {\mathit{\varphi }}_{\mathit{ij}}^{\prime }$, where ⊗ denotes the Kronecker product, and I_r is an r × r identity matrix. Define $D_{{\psi }_{\mathit{ij}}}^r$, $D_{{\varphi }_{\mathit{ij}}}^s$ and $D_{{\psi }_{\mathit{ij}}}^s$ similarly.

Let

$$X_0=\left(\begin{array}{c}{X}_1\\ ⋮\\ X_n\end{array}\right)\mathrm{and}{Z}_0=\left(\begin{array}{c}{Z}_{\upsilon 1}\\ Z_{\upsilon 2}\\ ⋮\\ Z_{\upsilon n}\end{array}\left|\begin{array}{c}\\ \underset{i=1,\dots ,n}{\mathrm{blockdiag}(Z_{\mathit{wi}})}\\ \\ \end{array}\right|\begin{array}{c}\\ \underset{i=1,\dots ,n}{\mathrm{blockdiag}(Z_{\mathit{ui}})}\\ \\ \end{array}\right),$$

where

$$X_i=\left(\begin{array}{c}{\mathit{x}}_{i1}^{\prime }{D}_{{\varphi }_{i1}}^r\\ ⋮\\ {\mathit{x}}_{i{m}_i}^{\prime }{D}_{{\varphi }_{i{m}_i}}^r\end{array}\right),Z_{\mathit{\upsilon i}}=\left(\begin{array}{c}{\mathit{x}}_{i1}^{\prime }{D}_{{\psi }_{i1}}^r\\ ⋮\\ {\mathit{x}}_{i{m}_i}^{\prime }{D}_{{\psi }_{i{m}_i}}^r\end{array}\right),Z_{wi}=\left(\begin{array}{c}{\mathit{z}}_{i1}^{\prime }{D}_{{\varphi }_{i1}}^s\\ ⋮\\ {\mathit{z}}_{i{m}_i}^{\prime }{D}_{{\varphi }_{i{m}_i}}^s\end{array}\right)\mathrm{and}{Z}_{\mathit{ui}}=\left(\begin{array}{c}{\mathit{z}}_{i1}^{\prime }{D}_{{\psi }_{i1}}^s\\ ⋮\\ {\mathit{z}}_{i{m}_i}^{\prime }{D}_{{\psi }_{i{m}_i}}^s\end{array}\right).$$

To obtain starting values for θ, {η_i}_i=1,…,n and σ², we fit the mixed effects model

$$\mathit{y}=X_0{\mathit{\beta }}_{k0}+Z_0{\mathit{\nu }}_{k0}+{\mathit{\epsilon }}_k,$$

for k = 1, …, p, where ${\mathit{\beta }}_{k0}={({\mathit{\beta }}_{k1}^{\prime },\dots ,{\mathit{\beta }}_{\mathit{kr}}^{\prime })}^{\prime },{\mathit{\nu }}_{k0}={({\mathit{\nu }}_{k1}^{\prime },\dots ,{\mathit{\nu }}_{\mathit{kr}}^{\prime },{\{{\mathit{w}}_{\mathit{ik}1}^{\prime },\dots ,{\mathit{w}}_{\mathit{iks}}^{\prime }\}}_{i=1,\dots ,n},{\{{\mathit{u}}_{\mathit{ik}1}^{\prime },\dots ,{\mathit{u}}_{\mathit{iks}}^{\prime }\}}_{i=1,\dots ,n})}^{\prime }$ and

$$\mathrm{cov}({\mathit{\nu }}_{k0},{\mathit{\epsilon }}_k)=\left(\begin{array}{llll}{\sigma }_{\mathit{k\upsilon }0}^2{I}_{\mathit{Kr}}& 0_{\mathit{Kr}\times 2\mathit{sn}}& 0_{\mathit{Kr}\times \mathit{Ksn}}& 0_{\mathit{Kr}\times n}\\ 0_{2\mathit{sn}\times \mathit{Kr}}& {\sigma }_{\mathit{kw}0}^2{I}_{2\mathit{sn}}& 0_{2\mathit{sn}\times K\mathit{sn}}& 0_{2\mathit{sn}\times n}\\ 0_{K\mathit{sn}\times \mathit{kr}}& 0_{K\mathit{sn}\times 2\mathit{sn}}& {\sigma }_{\mathit{ku}0}^2{I}_{\mathit{Ksn}}& 0_{\mathit{Ksn}\times n}\\ 0_{n\times \mathit{kr}}& 0_{n\times 2\mathit{sn}}& 0_{n\times \mathit{ksn}}& {\sigma }_{\mathit{k\epsilon }}^2{I}_n\end{array}\right).$$

Generating θ

Let ${\Gamma }_{\mathit{rij}}=I_r\otimes (\begin{array}{cc}{\mathit{\varphi }}_{\mathit{ij}}^{\prime }& {\mathit{\psi }}_{\mathit{ij}}^{\prime }\end{array}),{\Gamma }_{\mathit{sij}}=I_s\otimes (\begin{array}{cc}{\mathit{\varphi }}_{\mathit{ij}}^{\prime }& {\mathit{\psi }}_{\mathit{ij}}^{\prime }\end{array}),{\chi }_{\mathit{ij}}=I_p\otimes ({\mathit{x}}_{\mathit{ij}}^{\prime }{\Gamma }_{\mathit{rij}})$ and $E_{\mathit{ij}}=I_p\otimes ({\mathit{z}}_{\mathit{ij}}^{\prime }{\Gamma }_{\mathit{sij}})$. The error term in model (1) can be expressed as

$${\mathit{\delta }}_i(t_{\mathit{ij}})={\mathit{y}}_i(t_{\mathit{ij}})-{\chi }_{\mathit{ij}}\mathit{\theta }-E_{\mathit{ij}}{\mathit{\eta }}_i,$$ (A.1)

where the vectors θ and η_i are as defined at the beginning of Section 2.6. Plugging (A.1) into γ_tij = δ_i(t_ij) − exp(−AΔt_ij) δ_i(t_i,j−1) gives

$${\mathit{\gamma }}_{t_{\mathit{ij}}}={\mathit{\zeta }}_i(t_{\mathit{ij}},t_{i,j-1})-{\chi }_i(t_{\mathit{ij}},t_{i,j-1})\mathit{\theta },$$

where

$${\mathit{\zeta }}_i(t_{\mathit{ij}},t_{i,j-1})={\mathit{y}}_i(t_{\mathit{ij}})-E_{\mathit{ij}}{\mathit{\eta }}_i-\exp (-\mathit{A\Delta }{t}_{\mathit{ij}})[{\mathit{y}}_i(t_{i,j-1})-E_{i,j-1}{\mathit{\eta }}_i]$$

and

$${\chi }_i(t_{\mathit{ij}},t_{i,j-1})={\chi }_{\mathit{ij}}-\exp (-\mathit{A\Delta }{t}_{\mathit{ij}}){\chi }_{i,j-1.}$$

Let $G=\mathrm{blockdiag}({\sigma }_{{\beta }_{11}}^{-2}{I}_2,{\sigma }_{{\upsilon }_{11}}^{-2}{I}_k,\dots ,{\sigma }_{{\beta }_{1r}}^{-2}{I}_2,{\sigma }_{{\upsilon }_{1r}}^{-2}{I}_K,\dots ,{\sigma }_{{\beta }_{p1}}^{-2}{I}_2,{\sigma }_{{\upsilon }_{p1}}^{-2}{I}_K,\dots ,{\sigma }_{{\beta }_{pr}}^{-2}{I}_2,{\sigma }_{{\upsilon }_{pr}}^{-2}{I}_k)$. Then,

$$[\mathit{\theta }\mid \mathit{\eta },A,C,{\mathit{\sigma }}^2,\mathit{y}]~N({\mathit{\mu }}_{\theta },{\sum }_{\theta }),$$

where ${\sigma }^2={({\sigma }_{{\beta }_{\mathit{kl}}}^2,{\sigma }_{{\upsilon }_{\mathit{kl}}}^2,{\sigma }_{w_{\mathit{km}0}}^2,{\sigma }_{w_{\mathit{km}1}}^2,{\sigma }_{u_{\mathit{km}}}^2)}^{\prime }$ for k = 1, …, p, l = 1, …, r, m = 1, …, s,

$${\sum }_{\theta }={[G+\sum _{i=1}^n\sum _{j=1}^{m_i}{\chi }_i^{\prime }(t_{\mathit{ij}},t_{i,j-1}){\Omega }_{\Delta t_{\mathit{ij}}}^{-1}{\chi }_i(t_{\mathit{ij}},t_{i,j-1})]}^{-1}$$

and

$${\mathit{\mu }}_{\theta }^{\prime }=\left[\sum _{i=1}^n\sum _{j=1}^{m_i}{\mathit{\zeta }}_i^{\prime }(t_{\mathit{ij}},t_{i,j-1}){\Omega }_{\Delta t_{\mathit{ij}}}^{-1}{\chi }_i(t_{\mathit{ij}},t_{i,j-1})\right]{\sum }_{\theta }.$$

In the expression for Ω_Δtij, when p = 2, the stationary variance of the Ornstein–Uhlenbeck process is given by

$$\sum =\frac{\det (A)C+{[A-\mathrm{tr}(A)I]C[A-\mathrm{tr}(A)I]}^{\prime }}{2\mathrm{tr}(A)\det (A)},$$

see Gardiner (1983). For p = 1, this reduces to Σ = C/(2A). For p > 2, Σ can be obtained numerically by Matlab’s lyap function, for example.

Generating η_i

Let

$${\mathit{q}}_i(t_{\mathit{ij}},t_{i,j-1})={\mathit{y}}_i(t_{\mathit{ij}})-{\chi }_{\mathit{ij}}\mathit{\theta }-\exp (-A\Delta t_{\mathit{ij}})[{\mathit{y}}_i(t_{i,j-1})-{\chi }_{i,j-1}\mathit{\theta }]$$

and

$$G_{\mathit{wu}}=\mathrm{blockdiag}({\sum }_{w_{11}}^{-1},{\sigma }_{u11}^{-2}{I}_K,\dots ,{\sum }_{w_{1s}}^{-1},{\sigma }_{u_{1s}}^{-2}{I}_K,\dots ,{\sum }_{w_{p1}}^{-1},{\sigma }_{u_{p1}}^{-2}{I}_K,\dots ,{\sum }_{w_{\mathit{ps}}}^{-1},{\sigma }_{u_{\mathit{ps}}}^{-2}{I}_K),$$

where ${\mathit{\sum }}_{w_{\mathit{km}}}=\mathrm{diag}({\sigma }_{w_{\mathit{km}0}}^2,{\sigma }_{w_{\mathit{km}1}}^2)$. Then

$$[{\mathit{\eta }}_i|\mathit{\theta },A,C,{\sigma }^2,\mathit{y}]~N({\mathit{\mu }}_{{\eta }_i},{\sum }_{{\eta }_i}),$$

where

$${\sum }_{{\eta }_i}={\left[\sum _{j=1}^{m_i}{E}_i^{\prime }(t_{\mathit{ij}},t_{i,j-1}){\Omega }_{\Delta t_{\mathit{ij}}}^{-1}{E}_i(t_{\mathit{ij}},t_{i,j-1})+G_{\mathit{wu}}\right]}^{-1}$$

and

$${\mathit{\mu }}_{{\eta }_i}^{\prime }=[\sum _{j=1}^{m_i}{\mathit{q}}_i^{\prime }(t_{\mathit{ij}},t_{i,j-1}){\Omega }_{\Delta t_{\mathit{ij}}}^{-1}{E}_i(t_{\mathit{ij}},t_{i,j-1})]{\sum }_{{\eta }_i}.$$

Generating σ²

$${\sigma }_{{\upsilon }_{kl}}^2\mid {\upsilon }_{\mathit{kl}}~\mathit{IG}(K/2+a_1,b_1+\frac{1}{2}{\mathit{\nu }}_{\mathit{kl}}^{\prime }{\mathit{\nu }}_{\mathit{kl}})$$

for k = 1, …, p, l = 1, …, r.

$$\begin{array}{l}{\sigma }_{w_{\mathit{km}0}}^2\mid \{\underset{i=1,\dots ,n}{{\mathit{w}}_{\mathit{ikm}0}}\}~\mathit{IG}(n/2+a_2,b_2+\frac{1}{2}\sum _{i=1}^n{w}_{\mathit{ikm}0}^2)\\ {\sigma }_{w_{\mathit{km}1}}^2\mid \{\underset{i=1,\dots ,n}{{\mathit{w}}_{\mathit{ikm}1}}\}~\mathit{IG}(n/2+a_3,b_3+\frac{1}{2}\sum _{i=1}^n{w}_{\mathit{ikm}1}^2)\\ {\sigma }_{u_{\mathit{km}}}^2\mid \{\underset{i=1,\dots ,n}{{\mathit{u}}_{\mathit{ikm}}}\}~\mathit{IG}(\frac{\mathit{nK}}{2}+a_4,b_4+\frac{1}{2}\sum _{i=1}^n{\mathit{u}}_{\mathit{ikm}}^{\prime }{\mathit{u}}_{\mathit{ikm}})\end{array}$$

for k = 1, …, p, m = 1, …, s.

Starting values for A and C

Starting values for A and C are obtained by numerically maximizing the conditional posterior

$$p(A,C\mid {\mathit{\theta }}_0,\underset{i=1,\dots ,n}{\left\{{\mathit{\eta }}_{i0}\right\}},\mathit{y})\propto \prod _{i=1}^n\prod _{j=1}^{m_i}{|{\Omega }_{\Delta t_{\mathit{ij}}}|}^{1/2}\exp \{-\frac{1}{2}{\mathit{\gamma }}_{t_{\mathit{ij}}}^{\prime }{\Omega }_{\Delta t_{ij}}^{-1}{\mathit{\gamma }}_{t_{\mathit{ij}}}\}\times p(A)\times p(C),$$

where θ₀ and η_i0, i = 1, …, n, are the starting values for the basis function coefficients. Note that γ_tij depends on θ₀ and η_i0, i = 1, …, n, through δ_i(t_ij) (expression (A.1)).

Generating the Ornstein–Uhlenbeck process parameters

To generate A, note that

$$p(A\mid C,\mathit{\theta },\underset{i=1,\dots ,n}{\left\{{\mathit{\eta }}_i\right\}},\mathit{y})\propto \prod _{i=1}^n\prod _{j=1}^{m_i}{|{\Omega }_{\Delta t_{\mathit{ij}}}|}^{-1/2}\exp \{-\frac{1}{2}{{\mathit{\gamma }}^{\prime }}_{t_{\mathit{ij}}}{\Omega }_{\Delta t_{\mathit{ij}}}^{-1}{\mathit{\gamma }}_{t_{\mathit{ij}}}\}\times p(A).$$ (A.2)

Since (A.2) is not a standard distribution, we use a Metropolis step to generate A. The proposal distribution is multivariate normal centered at the current value of A with a variance–covariance matrix equal to the inverse of the negative Hessian of the log of (A.2) evaluated numerically at the mode. This variance–covariance matrix is computed once, conditional on the starting values for the other parameters, and is then fixed throughout the sampling scheme. To increase the acceptance rate, this variance–covariance matrix is multiplied by 5.76/p, as proposed in Gelman et al. (2004), page 306. More generally, when using a normal proposal distribution centered at the current point, Gelman et al. (2004) suggest using c²Σ as the covariance matrix of that proposal distribution. Among this class of proposal densities, the most efficient one has scale $c\approx 2.4/\sqrt{p}$. The acceptance probability is

$$\min \{1,\frac{p(A^p\mid C,\mathit{\theta },{\left\{{\mathit{\eta }}_i\right\}}_{i=1,\dots n},\mathit{y})}{p(A^c\mid C,\mathit{\theta },{\left\{{\mathit{\eta }}_i\right\}}_{i=1,\dots n},\mathit{y})}\}.$$

The matrix C is generated by first generating the matrices L and D via a Metropolis step based on

$$p(L,D\mid A,\mathit{\theta },{\left\{{\mathit{\eta }}_i\right\}}_{i=1,\dots ,n},\mathit{y})\propto \prod _{i=1}^n\prod _{j=1}^{m_i}{|{\Omega }_{\Delta t_{\mathit{ij}}}|}^{-1/2}\exp \{-\frac{1}{2}{{\mathit{\gamma }}^{\prime }}_{t_{\mathit{ij}}}{\Omega }_{\Delta t_{\mathit{ij}}}^{-1}{\mathit{\gamma }}_{t_{\mathit{ij}}}\}\times p(L)\times p(D).$$

An iterate for C is then given by C = LDL′.