Flexible Bayesian Dynamic Modeling of Correlation and Covariance Matrices

Abstract

Modeling correlation (and covariance) matrices can be challenging due to the positive-definiteness constraint and potential high-dimensionality. Our approach is to decompose the covariance matrix into the correlation and variance matrices and propose a novel Bayesian framework based on modeling the correlations as products of unit vectors. By specifying a wide range of distributions on a sphere (e.g. the squared-Dirichlet distribution), the proposed approach induces flexible prior distributions for covariance matrices (that go beyond the commonly used inverse-Wishart prior). For modeling real-life spatio-temporal processes with complex dependence structures, we extend our method to dynamic cases and introduce unit-vector Gaussian process priors in order to capture the evolution of correlation among components of a multivariate time series. To handle the intractability of the resulting posterior, we introduce the adaptive Δ-Spherical Hamiltonian Monte Carlo. We demonstrate the validity and flexibility of our proposed framework in a simulation study of periodic processes and an analysis of rat’s local field potential activity in a complex sequence memory task.

1. Introduction

Modeling covariance matrices—or more broadly, positive definite (PD) matrices—is one of the most fundamental problems in statistics. In general, the task is difficult because the number of parameters grows quadratically with the dimension of the matrices. The complexity of the challenge increases substantially if we allow dependencies to vary over time (or space) in order to account for the dynamic (non-stationary) nature of the underlying probability model. In this paper, we propose a novel solution to the problem by developing a flexible and yet computationally efficient Bayesian inferential framework for both static and dynamic covariance matrices.

This work is motivated by modeling the dynamic brain connectivity (i.e., associations between brain activity at different regions). In light of recent technical advances that allow the collection of large, multidimensional neural activity datasets, brain connectivity analyses are emerging as critical tools in neuroscience research. Specifically, the development of such analytical tools will help elucidate fundamental mechanisms underlying cognitive processes such as learning and memory, and identify potential biomarkers for early detection of neurological disorders. There are a number of new methods that have been developed (Cribben et al., 2012; Fiecas and Ombao, 2016; Lindquist et al., 2014; Ting et al., 2015; Prado, 2013) but the main limitation of these methods (especially the ones that have a frequentist approach) is a lack of natural framework for inference. Moreover, parametric approaches (e.g. vector auto-regressive models) need to be tested for adequacy for modeling complex brain processes and often have high dimensional parameter spaces (especially with a large number of channels and high lag order). This work provides both a nonparametric Bayesian model and an efficient inferential method for modeling the complex dynamic dependence among multiple stochastic processes that is common in the study of brain connectivity.

Within the Bayesian framework, it is common to use an inverse-Wishart prior on the covariance matrix for computational convenience (Mardia et al., 1980; Anderson, 2003). This choice of prior however is very restrictive (e.g. common degrees of freedom for all components of variance) (Barnard et al., 2000; Tokuda et al., 2011). Daniels (1999); Daniels and Kass (2001) propose uniform shrinkage priors. Daniels and Kass (1999) discuss three hierarchical priors to generalize the inverse-Wishart prior. Alternatively, one may use decomposition strategies for more flexible modeling choices (see Barnard et al. (2000) for more details). For instance, Banfield and Raftery (1993), Yang and Berger (1994), Celeux and Govaert (1995), Leonard and Hsu (1992), Chiu et al. (1996), and Bensmail et al. (1997) propose methods based on the spectral decomposition of the covariance matrix. Another strategy is to use the Cholesky decomposition of the covariance matrix or its inverse, e.g., Pourahmadi (1999, 2000); Liu (1993); Pinheiro and Bates (1996). There are other approaches directly related to correlation, including the constrained model based on truncated distributions (Liechty, 2004), the Cholesky decomposition of correlation matrix using an angular parametrization (Pourahmadi and Wang, 2015), and methods based on partial autocorrelation and parametrizations using angles (Rapisarda et al., 2007). In general, these methods fail to yield full flexibility and generality; and often sacrifice statistical interpretability.

While our proposed method in this paper is also based on the separation strategy (Barnard et al., 2000) and the Cholesky decomposition, the main distinction from the existing methods is that it represents each entry of the correlation matrix as a product of unit vectors. This in turn provides a flexible framework for modeling covariance matrices without sacrificing interpretability. Additionally, this framework can be easily extended to dynamic settings in order to model real-life spatio-temporal processes with complex dependence structures that evolve over the course of the experiment.

To address the constraint for correlation processes (positive definite matrix at each time having unit diagonals and off-diagonal entries with magnitudes no greater than 1), we introduce unit-vector Gaussian process priors. There are other related works, e.g. generalized Wishart process (Wilson and Ghahramani, 2011), and latent factor process (Fox and Dunson, 2015), that explore the product of vector Gaussian processes. In general they do not grant full flexibility in simultaneously modeling the mean, variance and correlation processes. For example, latent factor based models link the mean and covariance processes through a loading matrix, which is restrictive and undesirable if the linear link is not appropriate, and thus are outperformed by our proposed flexible framework (See more details in Section 4.2). Other approaches to model non-stationary processes use a representation in terms of a basis such as wavelets (Nason et al., 2000; Park et al., 2014; Cho and Fryzlewicz, 2015) and the smooth localized complex exponentials (SLEX) (Ombao et al., 2005), which are actually inspired by the Fourier representations in the Dahlhaus locally stationary processes Dahlhaus (2000); Priestley (1965). These approaches are frequentist and do not easily provide a framework for inference (e.g., obtaining confidence intervals). The class of time-domain parametric models allows for the autoregressive-moving-average (ARMA) parameters to evolve over time (see, e.g. Rao, 1970) or via parametric latent signals (West et al., 1999; Prado et al., 2001). A restriction for this class of parametric models is that some processes might not be adequately modeled by them.

This main contributions of this paper are: (a.) a sphere-product representation of correlation/covariance matrix is introduced to induce flexible priors for correlation/covariance matrices and processes; (b.) a general and flexible framework is proposed for modeling mean, variance, and correlation processes separately; (c.) an efficient algorithm is introduced to infer correlation matrices and processes; (d.) the posterior contraction of modeling covariance (correlation) functions with Gaussian process prior is studied for the first time.

The rest of the paper is organized as follows. In the next section, we present a geometric view of covariance matrices and extend this view to allow covariance matrices to change over time. In Section 3, we use this geometrical perspective to develop an effective and computationally efficient inferential method for modeling static and dynamic covariance matrices. Using simulated data, we will evaluate our method in Section 4. In Section 5, we apply our proposed method to local field potential (LFP) activity data recorded from the hippocampus of rats performing a complex sequence memory task (Allen et al., 2014, 2016; Ng et al., 2017). In the final section, we conclude with discussions on the limitations of the current work and future extensions.

2. Structured Bayesian Modeling of the Covariance (Correlation) Matrices

To derive flexible models for covariance and correlation matrices, we start with the Cholesky decomposition, form a sphere-product representation, and finally obtain the separation decomposition in Barnard et al. (2000) with correlations represented as products of unit vectors. The sphere-product representation is amenable for the inferential algorithm to handle the resulting intractability, and hence lays the foundation for full flexibility in choosing priors.

Any covariance matrix Σ = [σ_ij] > 0 is symmetric positive definite, and hence has a unique Cholesky decomposition Σ = LL^⊤ where the Cholesky factor L = [l_ij] is a lower triangular matrix such that ${\sigma }_{ij}={\sum }_{k=1}^{\text{min}\{i,j\}}{l}_{ik}{l}_{jk}$. We denote the variance vector as ${\sigma }^2:={\left[{\sigma }_1^2,\cdots ,{\sigma }_D^2\right]}^{\top }$, then each variance component, ${\sigma }_i^2:={\sigma }_{ii}$, can be written in terms of the corresponding row l_i of L as follows:

$${\sigma }_i^2=\sum _{k=1}^i{l}_{ik}^2={\Vert l_i\Vert }^2,l_i:=\left[l_{i1},l_{i2},\cdots ,l_{ii}\right].$$ (2.1)

For Σ to be positive definite, it is equivalent to require all the leading principal minors {M_i} to be positive,

$$M_i=\prod _{k=1}^i{l}_{kk}^2>0,i=1,\cdots ,D\iff l_{ii}\ne 0,i=1,\cdots ,D.$$ (2.2)

Based on (2.1) and (2.2), for i ∈ {1,⋯,D}, l_i can be viewed as a point on a sphere with radius σ_i excluding the equator, denoted as ${\mathcal{S}}_0^{i-1}\left({\sigma }_i\right):=\left\{l\in {ℝ}^i|\Vert l{\Vert }_2={\sigma }_i,l_{ii}\ne 0\right\}$. Therefore the space of the Cholesky factor in terms of its rows can be written as a product of spheres and we require

$$\left(l_1,l_2,\cdots ,l_D\right)\in {\mathcal{S}}_0^0\left({\sigma }_1\right)\times {\mathcal{S}}_0^1\left({\sigma }_2\right)\cdots \times {\mathcal{S}}_0^{D-1}\left({\sigma }_D\right).$$ (2.3)

Note that (2.3) is the sufficient and necessary condition for the matrix Σ = LL^⊤ to be a covariance matrix.

We present probabilistic models involving covariance matrices in the following generic form:

$$\begin{gathered} y|\Sigma (\sigma ,L)~\mathcal{l}(y;\Sigma (\sigma ,L)),\Sigma (\sigma ,L)=L{L}^{\top }, \\ \sigma ~p(\sigma ), \\ L|\sigma ~p(L;\sigma ),{\mathrm{vech}}^{\top }(L)\in \prod _{i=1}^D{\mathcal{S}}_0^{i-1}\left({\sigma }_i\right), \end{gathered}$$ (2.4)

where σ := [σ₁,⋯,σ_D]^⊤, and the half-vectorization in row order, vech^⊤, transforms the lower triangular matrix L into a vector (l₁, l₂,⋯,l_D). The total dimension of (σ,L) is $\frac{D(D+1)}{2}$.¹

Alternatively, if we separate variances from covariance, then we have a unique Cholesky decomposition for the correlation matrix P = [ρ_ij] = L*(L*)^⊤, where the Cholesky factor L* = diag(σ⁻¹)L can be obtained by normalizing each row of L. The magnitude requirements for correlations are immediately satisfied by the Cauchy-Schwarz inequality: $\left|{\rho }_{ij}\right|=\frac{\left|{\sigma }_{ij}\right|}{{\sigma }_i{\sigma }_j}=\frac{\left|\langle l_i,l_j\rangle \right|}{{\Vert l_i\Vert }_2{\Vert l_j\Vert }_2}\le 1$. Thus we require

$$\left(l_1^{*},l_2^{*},\cdots ,l_D^{*}\right)\in {\mathcal{S}}_0^0\times {\mathcal{S}}_0^1\cdots \times {\mathcal{S}}_0^{D-1},$$ (2.5)

where ${\mathcal{S}}_0^{i-1}:={\mathcal{S}}_0^{i-1}(1)$. Similarly, (2.5) is the sufficient and necessary condition for P = L*(L*)^⊤ to be a correlation matrix. Then we have the following alternatively structured model for covariance Σ that involves correlation P explicitly

$$\begin{gathered} y|\Sigma \left(\sigma ,L^{*}\right)~\mathcal{l}\left(y;\Sigma \left(\sigma ,L^{*}\right)\right),\Sigma \left(\sigma ,L^{*}\right)=\mathrm{diag}(\sigma )P\mathrm{diag}(\sigma ),P=L^{*}{\left(L^{*}\right)}^{\top }, \\ \sigma ~p(\sigma ), \\ {L}^{*}~p\left(L^{*}\right),{\mathrm{vech}}^{\top }\left(L^{*}\right)\in \prod _{i=1}^D{\mathcal{S}}_0^{i-1}. \end{gathered}$$ (2.6)

Note, this direct decomposition Σ = diag(σ)P diag(σ) as a separation strategy is motivated by statistical thinking in terms of standard deviations and correlations (Barnard et al., 2000). This setting is especially relevant if the statistical quantity of interest is correlation matrix P itself, and we can then skip inference of the standard deviation σ by fixing it to a data-derived point estimate.

In what follows, we will show that the above framework includes the inverse-Wishart prior as a special case, but it can be easily generalized to a broader range of priors for additional flexibility. Such flexibility enables us to better express prior knowledge, control the model complexity and speed up computation in modeling real-life phenomena. This is crucial in modeling spatio-temporal processes with complex structures.

2.1. Connection to the Inverse-Wishart Prior

There are some interesting connections between the spherical product representations (2.3) (2.5) and the early development of the Wishart distribution (Wishart, 1928). The original Wishart distribution was derived by orthogonalizing multivariate Gaussian random variables leading to a lower triangular matrix whose elements $\left\{t_{ij}^{*}|i\ge j\right\}$ (analogous to l_ij or $l_{ij}^{*}$) were called rectangular coordinates. This way, the probability density has a geometric interpretation as a product of volumes and approximate densities on a series of spherical shells with radius $\left\{t_{ii}^{*}\right\}$ (See more details in Sverdrup, 1947; Anderson, 2003). Now we demonstrate that the proposed schemes (2.4) (2.6) include the commonly used inverse-Wishart prior as a special case in modeling covariances.

Suppose Σ is a random sample from the inverse-Wishart distribution ${\mathcal{W}}_D^{-1}(\Psi ,\nu )$ with the scale matrix Ψ > 0 and the degree of freedom ν ≥ D. Therefore, ${\mathcal{W}}_D\left({\Psi }^{-1},\nu \right)$. Denote C as the Cholesky factor of Ψ⁻¹, i.e. Ψ⁻¹ = CC^⊤. Then Σ⁻¹ has the following Bartlett decomposition (Anderson, 2003; Smith and Hocking, 1972)

$${\Sigma }^{-1}=T{T}^{\top },T:=C{T}^{*},t_{ij}^{*}~\{\begin{array}{cc}{\chi }_{D-i+1},& i=j,\\ \mathcal{N}(0,1),& i>j,\\ {\delta }_0,& i<j,\end{array}$$ (2.7)

where the lower triangular matrix T, named Bartlett factor, has the following density (Theorem 7.2.1 of Anderson, 2003)

$$p(T)=\frac{|\Psi {|}^{\nu /2}}{2^{D(\nu -2)/2}{\mathrm{\Gamma }}_D(\nu /2)}\prod _{i=1}^D{\left|t_{ii}\right|}^{\nu -i}\text{exp}\left(-\frac{1}{2}\mathrm{tr}\left(\Psi T{T}^{\top }\right)\right)$$

with multivariate gamma function defined as ${\mathrm{\Gamma }}_D(x):={\pi }^{D(D-1)/4}{\prod }_{i=1}^D\mathrm{\Gamma }[x+(1-i)/2]$.

Now taking the inverse of the first equation in (2.7) yields the following reversed Cholesky decomposition²

$$\Sigma =U{U}^{\top },{\sigma }_{ij}=\sum _{k=\text{max}\{i,j\}}^D{u}_{ik}{u}_{jk},\mathrm{vech}\left(U^{\top }\right)\in \prod _{i=1}^D{\mathcal{S}}_0^{D-i}\left({\sigma }_i\right),$$

where U := T^−⊤ is an upper triangular matrix. The following proposition describes the density of the reversed Cholesky factor U of Σ, which enables us to treat the inverse-Wishart distribution as a special instance of strategy (2.4) or (2.6).

Proposition 2.1. Assume $\Sigma ~{\mathcal{W}}_D^{-1}(\Psi ,\nu )$. Then its reversed Cholesky factor U has the following density

$$p(U)=\frac{|\Psi {|}^{\nu /2}}{2^{D(\nu -2)/2}{\mathrm{\Gamma }}_D(\nu /2)}|U{|}^{-(\nu +D+1)}\prod _{i=1}^D{u}_{ii}^i\text{exp}\left(-\frac{1}{2}\mathrm{tr}\left(\Psi U^{-\text{T}}{U}^{-1}\right)\right).$$

Proof. See Section A in the supplementary file (Lan et al., 2019). □

If we normalize each row of U and write

$$U=\mathrm{diag}(\sigma )U^{*},{\sigma }_i=\sqrt{{\sigma }_{ii}}=\Vert u_i\Vert ,u_{ij}^{*}=u_{ij}/{\sigma }_i,$$

then the following joint prior of (σ,U*) is inseparable in general:

$$p\left(\sigma ,U^{*}\right)\propto \prod _{i=1}^D{\left|{\sigma }_i{u}_{ii}^{*}\right|}^{i-(\nu +D+1)}\text{exp}\left\{-\frac{1}{2}\mathrm{tr}\left(\Psi \mathrm{diag}\left({\sigma }^{-1}\right){\left(U^{*}\right)}^{-\top }{\left(U^{*}\right)}^{-1}\mathrm{diag}\left({\sigma }^{-1}\right)\right)\right\}.$$ (2.8)

With this result, we can conditionally model variance and correlation factor as p(σ|U*) and p(U*|σ) respectively, similarly as in our proposed scheme (2.4) or (2.6). It is also used to verify the validity of our proposed method (2.6) (see more details in Section 4.1). A similar result exists for the Wishart prior distribution regarding the Cholesky factor. This representation facilitates the construction of a broader class of more flexible prior distributions for covariance matrix detailed below.

2.2. More Flexible Priors

Within the above framework, the only constraint on U or L is that it resides on the product of spheres with increasing dimensions. Using this fact, we can develop a broader class of priors on covariance matrices and thus be able to model processes with more complicated dependence in covariance structures. Since σ and L* have independent priors in (2.6), in what follows we focus on the scheme (2.6), and for simplicity, we denote the normalized Cholesky factor as L. Also, following Barnard et al. (2000), we assume a log-Normal prior on σ:

$$\text{log}(\sigma )~\mathcal{N}(\xi \text{,}\Lambda ).$$

We now discuss priors on L that properly reflect the prior knowledge regarding the covariance structure among variables. If two variables, y_i and y_j (assuming i < j) are known to be uncorrelated a priori, i.e. 0 = ρ_ij = 〈l_i, l_j〉, then we can choose a prior that encourages l_i ⊥ l_j, e.g. l_jk ≈ 0 for k ≤ i. In contrast, if we believe a priori that there is a strong correlation between the two variables, we can specify that l_i and l_j be linearly dependent, e.g., by setting [l_jk]_k≤i ≈ ±l_i. When there is no prior information, we might assume that components are uncorrelated and consider the following Jeffreys prior for l_i that concentrates on the (two) poles of ${\mathcal{S}}_0^{i-1}$,

$$p\left(l_i\right)\propto \sqrt{\text{det}G\left(l_i\right)}={\left|l_{ii}\right|}^{-1},i=2,\cdots ,D,$$ (2.9)

where G(l_i) is the canonical metric on spheres (Lan et al., 2014). Putting more prior probability on the diagonal elements of L renders fewer non-zero off-diagonal elements, which in turn leads to a larger number of perpendicular variables; that is, such a prior favors zeros in the correlation matrix P. More generally, one can map a probability distribution defined on the simplex onto the sphere and consider the following squared-Dirichlet distribution.

Definition 1 (Squared-Dirichlet distribution). A random vector $l_i\in {\mathcal{S}}^{i-1}$ is said to have a squared-Dirichlet distribution with parameter α_i := (α_i1,α_i2,⋯,α_ii) if

$$l_i^2:=\left(l_{i1}^2,l_{i2}^2,\cdots ,l_{ii}^2\right)~\mathrm{Dir}\left({\alpha }_i\right).$$

Denote l_i ~ Dir²(α_i). Then l_i has the following density

$$p\left(l_i\right)=p\left(l_i^2\right)\left|2l_i\right|\propto {\left(l_i^2\right)}^{{\alpha }_i-1}\left|l_i\right|={\left|I_i\right|}^{2{\alpha }_i-1}:=\prod _{k=1}^i{\left|l_{ik}\right|}^{2{\alpha }_{ik}-1}.$$ (2.10)

Remark 1. This definition includes a large class of flexible prior distributions on the unit sphere that specify different concentrations of probability density through the parameter α_i. For example, the Jeffreys prior (2.9) corresponds to.α_i = (1/2,⋯,1/2, 0).

To induce a prior distribution for the correlation matrix P = LL^⊤, one can specify priors on row vectors of L, l_i ~ Dir²(α_i) for i = 2,⋯,D. Each pair for (l_i, l_j) in turn defines a prior distribution for the correlation ρ_ij = 〈l_i, l_j〉, whose sign is determined by the angle between l_i and l_j. To encourage small correlation, we choose the concentration parameter α_i so that the probability density concentrates around the (two) poles of ${\mathcal{S}}_0^{i-1}$, e.g. 0 < α_ik ≪ α_ik for k < i. Figure 1 illustrates the density heat maps of some symmetric squared-Dirichlet distributions Dir²(α1) on ${\mathcal{S}}^2$. It is interesting that the squared-Dirichlet distribution induces two important uniform prior distributions over correlation matrices from Barnard et al. (2000) in an effort to provide flexible priors for covariance matrices, as stated in the following theorem.

Figure 1:

Symmetric squared-Dirichlet distributions Dir²(α) defined on the 2-sphere with different settings for concentration parameter α = α1. The uniform distribution on the simplex, Dir(1), becomes non-uniform on the sphere due to the stretch of geometry (left); the symmetric Dirichlet distribution Dir(1/21) becomes uniform on the sphere (middle); with α closer to 0, the induced distribution becomes more concentrated on the polar points (right).

Theorem 2.1 (Uniform distributions). Let P = LL^⊤. Suppose l_i ~ Dir²(α_i), for i = 2,⋯,D, are independent, where l_i is the i-th row of L. We have the following

1. If ${\alpha }_i=\left(\frac{1}{2}{1}_{i-1}^{\top },{\alpha }_{ii}\right)$, ${\alpha }_{ii}=\frac{(i-2)D-1}{2}$, then P follows a marginally uniform distribution, that is, ρ_ij ~ Unif(−1, 1), i ≠ j.

2. If ${\alpha }_i=\left(\frac{1}{2}{1}_{i-1}^{\top },{\alpha }_{ii}\right)$, ${\alpha }_{ii}=\frac{D-i}{2}+1$, then P follows a jointly uniform distribution, that is, p(P) ∝ 1.

Proof. See Section A in the supplementary file (Lan et al., 2019). □

Another natural spherical prior can be obtained by constraining a multivariate Gaussian random vector to have unit norm. This is later generalized to a vector Gaussian process constrained to a sphere that serves as a suitable prior for modeling correlation processes. Now we consider the following unit-vector Gaussian distribution:

Definition 2 (Unit-vector Gaussian distribution). A random vector $l_i\in {\mathcal{S}}^{i-1}$ is said to have a unit-vector Gaussian distribution with mean μ and covariance Σ if

$$l_i~{\mathcal{N}}_i(\mu ,\Sigma ),\text{with }{\Vert l_i\Vert }_2=1.$$

Then we denote $l_i~{\mathcal{N}}_i^{\mathcal{S}}(\mu ,\Sigma )$ and l_i has the following (conditional) density

$$p\left(l_i|{\Vert l_i\Vert }_2=1\right)=\frac{1}{{(2\pi )}^{\frac{i}{2}}|\Sigma {|}^{\frac{1}{2}}}\text{exp}\left\{-\frac{1}{2}{\left(l_i-\mu \right)}^{\top }{\Sigma }^{-1}\left(l_i-\mu \right)\right\},{\Vert l_i\Vert }_2=1.$$

Remark 2. This conditional density essentially defines the following Fisher-Bingham distribution (a.k.a. generalized Kent distribution, Kent, 1982; Mardia and Jupp, 2009). If Σ = I, then the above distribution reduces to the von Mises-Fisher distribution (Fisher, 1953; Mardia and Jupp, 2009) as a special case. If in addition μ = 0, then the above density becomes a constant; that is, the corresponding distribution is uniform on the sphere $S_0^{i-1}$. See more details in Section E.1 of the supplementary file (Lan et al., 2019).

2.3. Dynamically Modeling the Covariance Matrices

We can generalize the proposed framework for modeling covariance/correlation matrices to the dynamic setting by adding subscript t to variables in the model (2.4) and the model (2.6), thus called dynamic covariance and dynamic correlation models respectively. We focus the latter in this section. One can model the components of σ_t as independent dynamic processes using, e.g. ARMA, generalized autoregressive conditional heteroskedasticity (GARCH), or log-Gaussian process. For L_t, we use vector processes. Since each row of L_t has to be on a sphere of certain dimension, we require the unit norm constraint for the dynamic process over time. We refer to any multivariate process l_i(x) satisfying ‖l_i(x)‖ ≡ 1, $\forall x\in \mathcal{X}$ as unit-vector process (uvP). A unit-vector process can be obtained by constraining an existing multivariate process, e.g. the vector Gaussian process (vGP), as defined below.

Definition 3 (Vector Gaussian process). A D-dimensional vector Gaussian process Z(x) := (Z₁(x),⋯,Z_D(x)), with vector mean function μ(x) = (μ₁(x),⋯,μ_D(x)), covariance function C and (D-dimensional) cross covariance V_D×D,

$$Z(x)~\mathcal{G}{\mathcal{P}}_D\left(\mu ,\mathcal{C},V_{D\times D}\right)$$

is a collection of D-dimensional random vectors, indexed by $x\in \mathcal{X}$, such that for any finite set of indices {x₁,⋯,x_N}, the random matrix ${\tilde{Z}}_{N\times D}:={\left(Z\left(x_1\right),\cdots ,Z\left(x_N\right)\right)}^{\top }$ has the following matrix normal distribution

$${\tilde{Z}}_{N\times D}~\mathcal{M}{\mathcal{N}}_{N\times D}\left(M_{N\times D},K_{N\times N},V_{D\times D}\right),$$

where M_N×D := (m₁,⋯,m_D), and m_k = (μ_k(x₁),⋯,μ_k(x_N))^⊤, and K is the kernel matrix with elements K_ij = C(x_i, x_j).

Remark 3. Note for each k = 1,⋯D, we have the following marginal GP

$$Z_k(x)~\mathcal{G}\mathcal{P}\left({\mu }_k,\mathcal{C}\right).$$

In the above definition, we require a common kernel C for all the marginal GPs, whose dependence is characterized by the cross covariance V_D×D. On the other hand, for any fixed $x^{*}\in \mathcal{X}$, we have

$$Z\left(x^{*}\right)~{\mathcal{N}}_D\left(\mu \left(x^{*}\right),V_{D\times D}\right).$$

For simplicity, we often consider μ ≡ 0 and V_D×D = I_D. That is, $Z_k(x)\stackrel{iid}{~}\mathcal{G}\mathcal{P}(0,\mathcal{C})$ for k = 1,⋯,D.

Restricting vGP Z(·) to sphere yields a unit-vector Gaussian process (uvGP) Z*(·) := Z(·)| {‖Z(·)‖₂ ≡ 1}, denoted as $Z^{*}(\cdot )~\mathcal{G}{\mathcal{P}}_D^{\mathcal{S}}(\mu ,\mathcal{C},V)$. Note for any fixed $x^{*}\in \mathcal{X}$, $Z^{*}\left(x^{*}\right)~{\mathcal{N}}_D^S(\mu ,V)$. Setting μ ≡ 0, V = I, and conditioned on the length ℓ_n of each row of $\tilde{Z}$, we have

$$p\left({\tilde{Z}}^{*}|\left\{\Vert z_{n\cdot }\Vert ={\mathcal{l}}_n\right\}\right)=\frac{\prod _{n=1}^N{\mathcal{l}}_n^D}{{(2\pi )}^{\frac{ND}{2}}|K{|}^{\frac{D}{2}}}\text{exp}\left\{-\frac{1}{2}\mathrm{tr}\left[{\left({\tilde{Z}}^{*}\right)}^{\top }\mathrm{diag}\left(\left\{{\mathcal{l}}_n\right\}\right)K^{-1}\mathrm{diag}\left(\left\{{\mathcal{l}}_n\right\}\right){\tilde{Z}}^{*}\right]\right\}.$$

This conditional density is preserved by the inference algorithm in Section 3 and used for defining priors for correlations with all ℓ_n = 1. For each marginal GP, we select the following powered exponential function as the common kernel

$$\mathcal{C}\left(x,x^{\prime }\right)=\gamma \text{exp}\left(-0.5{\Vert x-x^{\prime }\Vert }^s/{\rho }^s\right),$$

where s controls the smoothness, the scale parameter γ is given an inverse-Gamma prior, and the correlation length parameter ρ is given a log-normal prior. Figure 2 shows a realization of vector GP Z_t, unit-vector GP (forming rows of) L_t and the induced correlation process P_t respectively.

Figure 2:

A realization of vector GP Z_t (left), unit-vector GP (forming rows of) L_t (middle) and the induced correlation process P_t (right).

In what follows, we focus on multivariate time series; therefore, we use the one dimensional time index $t\in \mathcal{X}={ℝ}^{+}$. The overall dynamic correlation model can be summarized as follows:

$$\begin{gathered} y_t~\mathcal{N}\left({\mu }_t,{\Sigma }_t\right),{\Sigma }_t=\mathrm{diag}\left({\sigma }_t\right)L_t{L}_t^{\top }\mathrm{diag}\left({\sigma }_t\right), \\ {\mu }_t~\mathcal{G}{\mathcal{P}}_D\left(0,{\mathcal{C}}_{\mu },I\right),{\mathcal{C}}_{\mu }\left(t,t^{\prime }\right)={\gamma }_{\mu }\text{exp}\left(-0.5{\Vert t-t^{\prime }\Vert }^s/{\rho }_{\mu }^s\right), \\ \text{log}{\sigma }_t~\mathcal{G}{\mathcal{P}}_D\left(0,{\mathcal{C}}_{\sigma },I\right),{\mathcal{C}}_{\sigma }\left(t,t^{\prime }\right)={\gamma }_{\sigma }\text{exp}\left(-0.5{\Vert t-t^{\prime }\Vert }^s/{\rho }_{\sigma }^s\right), \\ {l}_i(t)~\mathcal{G}{\mathcal{P}}_i^S\left(n_i,{\mathcal{C}}_L,I\right),{\mathcal{C}}_L\left(t,t^{\prime }\right)={\gamma }_L\text{exp}\left(-0.5{\Vert t-t^{\prime }\Vert }^s/{\rho }_L^s\right), \\ {\gamma }_{*}~{\mathrm{\Gamma }}^{-1}\left(a_{*},b_{*}\right),\log {\rho }_{*}~\mathcal{N}\left(m_{*},V_{*}\right),*=\mu ,\sigma ,\text{ or }L, \end{gathered}$$ (2.11)

where a constant mean function n_i = (0,⋯,0,1) is used in the uvGP prior for l_i(t), with mean matrix $M=1_N\otimes n_i^{\top }$ for the realization ${\tilde{l}}_i$. This model (2.11) captures the spatial dependence in the matrix Σ_t, which evolves along the time; while the temporal correlation is characterized by various GPs. The induced covariance process Σ_t is not a generalized Wishart process (Wilson and Ghahramani, 2011), which only models Cholesky factor of covariance using GP. Though with GP, dynamic covariance model may work similarly as the dynamic correlation model (2.11), yet the latter provides extra flexibility in modeling the evolution of variances and correlations separately. In general such flexibility could be useful in handling constraints for processes, e.g. modeling the dynamic probability for binary time series.

With this structured model (2.11), one can naturally model the evolution of variances and correlations separately in order to obtain more flexibility. If the focus is on modeling the correlation among multiple time series, then one can substitute σ_t with a point estimate $\widehat{\sigma }$ from one trial and assume a steady variance vector. Alternatively, if sufficient trials are present, one can obtain an empirical estimate ${\widehat{\sigma }}_t$, from multiple trials at each time point. In the following, we study the posterior contraction of GP modeling in this setting.

2.4. Posterior Contraction Theorem

We now provide a theorem on the posterior contraction of the dynamic covariance model before we conclude this section. Because the posterior contraction for mean regression using Gaussian process has been vastly investigated in the literature (van der Vaart and van Zanten, 2008a, 2009, 2011; Yang and Dunson, 2016), we only investigate the posterior contraction for the covariance regression and set μ_t ≡ 0. We leave the posterior contraction of the dynamic correlation model (2.11) for future work. Note, the Cholesky decomposition of covariance matrix Σ = LL^⊤ is unique if all the diagonal entries of L are positive. Therefore in the remaining of this section, we identify Cholesky factors up to a column-wise sign, i.e. L ~ Ldiag(−∑_j∈J e_j) for J ⊂{1,⋯,D}where e_j is the j-th column of identity matrix I_D.

In most cases, Gaussian process L_t can be viewed as a tight Borel measurable map in a Banach space, e.g. a space of continuous functions or L_p space. It is well known that the support of a centered GP is equal to the closure of the reproducible kernel Hilbert space (RKHS) $ℍ$ associated to this process (Lemma 5.1 of van der Vaart and van Zanten, 2008b). Because the posterior distribution necessarily puts all its mass on the support of the prior, the posterior consistency requires the true parameter L₀ governing the distribution of the data to fall in this support (van der Vaart and van Zanten, 2008a). Following van der Vaart and van Zanten (2008a,b, 2011), we express the rate of the posterior contraction in terms of the concentration function

$${\varphi }_{L_0}(\epsilon )=\underset{h\in ℍ:\Vert h-L_0\Vert <\epsilon }{\text{inf}}\Vert h{\Vert }_{ℍ}^2-\text{log}\mathrm{\Pi }(L:\Vert L\Vert <\epsilon ),$$ (2.12)

where ‖·‖ is the norm of the Banach space where the GP L takes value, Π is the GP prior and $ℍ$ is the associated RKHS with norm $\Vert \cdot {\Vert }_{ℍ}$. Under certain regularity conditions, the posterior contracts with increasing data expressed in n at the rate ε_n → 0 satisfying

$${\varphi }_{L_0}\left({\epsilon }_n\right)\le n{\epsilon }_n^2.$$ (2.13)

Let $\Vert L{\Vert }_{\infty }:={\text{max}}_{1\le i,j\le D}{\text{sup}}_{t\in \mathcal{X}}\left|l_{ij}(t)\right|$. Consider Banach space $L^{\infty }{(\mathcal{X})}^{D(D+1)/2}:=\left\{L:\Vert L{\Vert }_{\infty }<+\infty \right\}$. Let p be a (centered) Gaussian model, which is uniquely determined by the covariance matrix Σ = LL^⊤. Therefore the model density is parametrized by L, hence denoted as p_L. Denote $P_L^{(n)}:={\otimes }_{i=1}^n{P}_{L,i}$ as the product measure on ${\otimes }_{i=1}^n\left({\mathcal{X}}_i,{\mathcal{B}}_i,{\mu }_i\right)$. Each P_L,i has a density p_Li with respect to the σ-finite measure μ_i. Define the average Hellinger distance as $d_n^2\left(L,L^{\prime }\right)=\frac{1}{n}{\sum }_{i=1}^n\int {(\sqrt{p_{L,i}}-\sqrt{p_{L^{\prime },i}})}^{2}d{\mu }_i$. Denote the observations $Y^{(n)}={\left\{Y_i\right\}}_{i=1}^n$ with Y_i = y(t_i). Note they are independent but not identically distributed (inid). Now we state the main theorem of posterior contraction.

Theorem 2.2 (Posterior contraction). Let L − I be a Borel measurable, zero-mean tight Gaussian random element in $L^{\infty }{(\mathcal{X})}^{D(D+1)/2}$ and $P_L^{(n)}={\otimes }_{i=1}^n{P}_{L,i}$ be the product measure of Y ⁽ⁿ⁾ parametrized by L. Let ϕ_L0 be the function in (2.12) with the uniform norm ‖·‖_∞. If L₀ is contained in the support of L and ϕ_L0 satisfies (2.13) with ε_n ≥ n^−1/2, then Π_n(L : d_n(L,L₀) > M_nε_n|Y⁽ⁿ⁾) → 0 in $P_{L_0}^{(n)}$-probability for every M_n → ∞.

Proof. See Section B in the supplementary file (Lan et al., 2019). □

Remark 4. In principle, the smoothness of GP should match the regularity of the true parameter to achieve the optimal rate of contraction (van der Vaart and van Zanten, 2008a, 2011). One can scale GP, e.g. using an inverse-Gamma bandwidth, to get optimal contraction rate for every regularity level so that the resulting estimator is rate adaptive (van der Vaart and van Zanten, 2009, 2011). One can refer to Section 3.2 of (van der Vaart and van Zanten, 2011) for posterior contraction rates using squared exponential kernel for GP. We leave further investigation on contraction rates in the setting of covariance regression to future work.

Remark 5. Here the GP prior L defines a (mostly finite) probability measure on the space of bounded functions. The true parameter function L₀ is required to be contained in the support of the prior, the RKHS of L. The contraction rate depends on the position of L₀ relative to the RKHS and the small-ball probability Π(‖L‖ < ε)

3. Posterior Inference

Now we obtain the posterior probability of mean μ_t, variance σ_t, Cholesky factor of correlation L_t, hyper-parameters γ := (γ_μ, γ_σ, γ_L) and ρ := (ρ_μ, ρ_σ, ρ_L) in the model (2.11) Denote the realization of processes μ_t, σ_t, L_t at discrete time points ${\left\{t_n\right\}}_{n=1}^N$ as ${\tilde{\mu }}_{N\times D}$, ${\tilde{\sigma }}_{N\times D}$, ${\tilde{L}}_{N\times D\times D}$ respectively. Transform the parameters $\tilde{\tau }:=\text{log}(\tilde{\sigma })$, η :=log(ρ) for calculation convenience. Denote ${\tilde{Y}}_{M\times N\times D}:=\left\{Y_1,\cdots ,Y_M\right\}$ for M trials, (Y_m)_N×D := [y_m1,⋯,y_mN]^⊤ and $y_{mn}^{*}:=\left(y_{mn}-{\mu }_n\right)°e^{-{\tau }_n}$ where ○ is the Hadamard product (a.k.a. Schur product), i.e. the entry-wise product. Let K_*(γ_*, η_*) = γ_*K_0*(η_*) and ${\tilde{l}}_i^{*}:={\tilde{\text{I}}}_i-1_N\otimes n_i^{\top }$.

3.1. Metropolis-within-Gibbs

We use a Metropolis-within-Gibbs algorithm and alternate updating the model parameters $\tilde{\mu }$, $\tilde{\tau }$, $\tilde{L}$, γ, η, We now list the parameters and their respective updates one by one. (γ) Note the prior for γ is conditionally conjugate given * = μ, τ, or L,

$${\gamma }_{*}|\widetilde{*},{\eta }_{*}~{\mathrm{\Gamma }}^{-1}\left(a_{*}^{\prime },b_{*}^{\prime }\right),a_{*}^{\prime }=a_{*}+\frac{ND}{2}{\left(\frac{D+1}{2}-\frac{1}{D}\right)}^{[*=L]},b_{*}^{\prime }=b_{*}+\frac{1}{2}\mathrm{tr}\left({\widetilde{*}}^{\top }{K}_{0*}{\left({\eta }_{*}\right)}^{-1}\widetilde{*}\right),$$

where [condition] is 1 with the condition satisfied and 0 otherwise.

(η) Given * = μ, τ, or L, we could sample η_* using the slice sampler (Neal, 2003), which only requires log-posterior density and works well for scalar parameters,

$$\text{log}p\left({\eta }_{*}|\widetilde{*},{\gamma }_{*}\right)=-\frac{D{\left(\frac{D+1}{2}-\frac{1}{D}\right)}^{[*=L]}}{2}\text{log}\left|K_{0*}\left({\eta }_{*}\right)\right|-\frac{\mathrm{tr}\left({\widetilde{*}}^{\top }{K}_{0*}{\left({\eta }_{*}\right)}^{-1}\widetilde{*}\right)}{2{\gamma }_{*}}-\frac{{\left({\eta }_{*}-m_{*}\right)}^2}{2V_{*}}.$$

$(\tilde{\mu })$ By the definition of vGP, we have $\tilde{\mu }|{\gamma }_{\mu },{\eta }_{\mu }~\mathcal{M}{\mathcal{N}}_{N\times D}\left(0,K_{\mu },I_D\right)$; therefore, $\mathrm{Vec}(\tilde{\mu })|{\gamma }_{\mu },{\eta }_{\mu }~{\mathcal{N}}_{ND}\left(0,I_D\otimes K_{\mu }\right)$. On the other hand, one can write

$$\begin{gathered} \sum _{m=1}^M\sum _{n=1}^N{y}_{mn}^{*}{}^{\top }{P}_n^{-1}{y}_{mn}^{*}=\sum _{m=1}^M\mathrm{vec}{\left({\left(Y_m-\tilde{\mu }\right)}^{\top }\right)}^{\top }\mathrm{diag}\left(\left\{{\tilde{\Sigma }}_n^{-1}\right\}\right)\mathrm{vec}\left({\left(Y_m-\tilde{\mu }\right)}^{\top }\right) \\ =\sum _{m=1}^M{\left(\mathrm{vec}\left(Y_m\right)-\mathrm{vec}(\tilde{\mu })\right)}^{\top }{\tilde{\Sigma }}_K^{-1}\left(\mathrm{vec}\left(Y_m\right)-\mathrm{vec}(\tilde{\mu })\right), \end{gathered}$$

where ${\tilde{\Sigma }}_K^{-1}:=K_{(D,N)}\mathrm{diag}{\left(\left\{{\tilde{\Sigma }}_n\right\}\right)}^{-1}{K}_{(N,D)}$, and K_(N,D) is the commutation matrix of size ND × ND such that for any N × D matrix A, K_(N,D) vec(A) = vec(A^⊤) (Tracy and Dwyer, 1969; Magnus and Neudecker, 1979). Therefore, the prior on $\mathrm{vec}(\tilde{\mu })$ is conditionally conjugate, and we have

$$\begin{gathered} \mathrm{vec}(\tilde{\mu })|\tilde{Y},\tilde{\Sigma },{\gamma }_{\mu },{\eta }_{\mu }~{\mathcal{N}}_{ND}\left({\mu }^{\prime },{\Sigma }^{\prime }\right) \\ {\mu }^{\prime }={\Sigma }^{\prime }{\tilde{\Sigma }}_K^{-1}\sum _{m=1}^M\mathrm{vec}\left(Y_m\right),{\mathit{\Sigma }}^{\prime }={\left(I_D\otimes K_{\mu }^{-1}+M{\tilde{\Sigma }}_K^{-1}\right)}^{-1}. \end{gathered}$$

$(\tilde{\tau })$ Using a similar argument by matrix Normal prior for $\tilde{\tau }$, we have $\mathrm{vec}(\tilde{\tau })|{\gamma }_{\tau },{\eta }_{\tau }~{\mathcal{N}}_{ND}\left(0,I_D\otimes K_{\tau }\right)$. Therefore, we could use the elliptic slice sampler (ESS, Murray et al., 2010), which only requires the log-likelihood

$$\text{log}p(\tilde{\mathit{\tau }};\tilde{Y},\tilde{\mu })=-M{1}_{ND}^{\top }\mathrm{Vec}(\tilde{\mathit{\tau }})-\sum _{m=1}^M\frac{1}{2}\mathrm{Vec}{\left(Y_m^{*}\right)}^{\top }{\tilde{P}}_K^{-1}\mathrm{Vec}\left(Y_m^{*}\right),$$

where ${\tilde{P}}_K^{-1}:=K_{(D,N)}\mathrm{diag}{\left(\left\{{\tilde{P}}_n\right\}\right)}^{-1}{K}_{(N,D)}$ and $Y_m^{*}:=\left(Y_m-\tilde{\mu }\right)°\text{exp}(-\tilde{\tau })$.

$(\tilde{L})$ For each n ∈ {1,⋯N}, we have ${\mathrm{vech}}^{\top }\left(L_n\right)\in {\prod }_{i=1}^D{\mathcal{S}}_0^{i-1}$. We could sample from its posterior distribution using the Δ-Spherical Hamiltonian Monte Carlo (Δ-SphHMC) described below. The log-posterior density of $\tilde{L}$ is

$$\begin{gathered} \text{log}p\left(\tilde{L}|\tilde{Y},\tilde{\mu },\tilde{\tau },{\gamma }_L,{\eta }_L\right)=-\sum _{n=1}^N\left[M\text{log}\left|L_n\right|+\sum _{m=1}^M\frac{1}{2}{y}_{mn}^{*}{}^{\top }{P}_n^{-1}{y}_{mn}^{*}\right] \\ -\frac{1}{2}\sum _{i=2}^D\mathrm{tr}\left({\widetilde{\mathbf{\text{l}}}}_i^{*}{}^{\top }{K}_L^{-1}{\widetilde{\mathbf{\text{l}}}}_i^{*}\right). \end{gathered}$$

The derivative of log-likelihood with respect to L_n and the derivative of log-prior with respect to ${\tilde{l}}_i$ can be calculated as

$$\begin{gathered} \frac{\partial \text{log}p(\tilde{L};\tilde{Y},\tilde{\mu },\tilde{\tau })}{\partial L_n}=-M\frac{I_D}{L_n}+\sum _{m=1}^M\mathrm{tril}\left(P_n^{-1}{y}_{mn}^{*}{y}_{mn}^{*}{}^{\top }{L}_n^{-\top }\right), \\ \frac{\partial \text{log}p\left(\tilde{L}|{\gamma }_L,{\eta }_L\right)}{\partial {\tilde{l}}_i}=-K_L^{-1}{\tilde{l}}_i^{*}. \end{gathered}$$

3.2. Spherical HMC

We need an efficient algorithm to handle the intractability in the posterior distribution of $\tilde{L}$ introduced by various flexible priors. Spherical Hamiltonian Monte Carlo (SphHMC, Lan et al., 2014; Lan and Shahbaba, 2016) is a Hamiltonian Monte Carlo (HMC, Duane et al., 1987; Neal, 2011) algorithm on spheres that can be viewed as a special case of geodesic Monte Carlo (Byrne and Girolami, 2013), or manifold Monte Carlo methods (Girolami and Calderhead, 2011; Lan et al., 2015). The algorithm was originally proposed to handle norm constraints in sampling so it is natural to use it to sample each row of the Cholesky factor of a correlation matrix with unit 2-norm constraint. The general notation q is instantiated as l_i in this section.

Assume a probability distribution with density function f(q) is defined on a (D–1) dimensional sphere with radius r, ${\mathcal{S}}^{D-1}(r)$. Due to the norm constraint, there are (D–1) free parameters q_−D := (q₁,⋯q_D–1), which can be viewed as the Cartesian coordinates for the manifold ${\mathcal{S}}_{+}^{D-1}(r)$. To induce Hamiltonian dynamics on the sphere, we define the potential energy for position q as U(q) := −log f(q). Endowing the canonical spherical metric $G(q-D)=I_{D-1}+\frac{q_{-}{}_D{q}_{-D}^{\top }}{q_D^2}$ on the Riemannian manifold ${\mathcal{S}}^{D-1}(r)$, we introduce the auxiliary velocity vector $v|q~\mathcal{N}\left(0,G{(q)}^{-1}\right)$ and define the associated kinetic energy as $K(v;q):=-\text{log}{f}_{\mathcal{N}}(v|q)=-\frac{1}{2}\text{log}|G(q_{-D})|+\frac{1}{2}{v}_{-D}^{\top }G(q_{-D})V_{-D}$ (Girolami and Calderhead, 2011). Therefore the total energy is defined as

$$E(q,v):=U(q)+K(v;q)=\tilde{U}(q)+K_0(v;q),$$ (3.1)

where we denote $\tilde{U}(q):=U(q)-\frac{1}{2}\text{log}|G(q_{-D})|=-\text{log}f(q)+\text{log}\left|q_D\right|$, and $K_0(v;q):=\frac{1}{2}{v}_{-D}^{\top }G(q_{-D})v_{-D}=\frac{1}{2}{v}^{\top }v$. (Lan and Shahbaba, 2016). Therefore the Lagrangian dynamics with above total energy (3.1) is (Lan et al., 2015)

$$\begin{gathered} {\dot{q}}_{-D}=v_{-D}, \\ {\dot{v}}_{-D}=-v_{-D}^{\top }\mathrm{\Gamma }\left(q_{-D}\right)v_{-D}-G{\left(q_{-D}\right)}^{-1}{\nabla }_{q-D}\tilde{U}(q), \end{gathered}$$ (3.2)

where Γ(q_–D) = r⁻²G(q_−D) ⊗ q_−D is the Christoffel symbols of second kind (see details in Lan and Shahbaba, 2016, for r = 1). A splitting technique is used to yield a geometric integrator called leapfrog that mainly moves along geodesics (great circles) on the sphere with random perturbation in the direction. It is then applied with discrete step size h for T times to numerically solve (3.2) in order to obtain a proposed state (q_T, v_T). This proposal can be accepted with certain probability expressed in the total energy (3.1). Interested readers can refer to Lan et al. (2014), Lan and Shahbaba (2016), or Section C in the supplementary file (Lan et al., 2019). In addition to the original work in Lan et al. (2014) and Lan and Shahbaba (2016), we prove the following result on energy conservation of the algorithm (Beskos et al., 2011).

Theorem 3.1. Let h → 0 we have the following energy conservation

$$E(q(T),v(T))-E(q(0),v(0))=\tilde{U}(q(T))-\tilde{U}(q(0))-{\int }_0^T\langle v(t),g(q(t))\rangle dt=0.$$

Proof. See Section C in the supplementary file (Lan et al., 2019). □

3.3. Adaptive Spherical HMC

There are two tuning parameters in HMC and its variants: the step size h and the number of integration (leapfrog) steps T. Hand tuning heavily relies on domain expertise and could be inefficient. Here, we adopt the ‘No-U-Turn’ idea from Hoffman and Gelman (2014) and introduce a novel adaptive algorithm beyond Lan et al. (2014); Lan and Shahbaba (2016) that obviates manual tuning of these parameters.

First, for any given step size h, we adopt a rule for setting the number of leapfrog steps T based on the same philosophy as ‘No-U-Turn’ (Hoffman and Gelman, 2014). The idea is to avoid waste of computation occurred (e.g. when the sampler backtracks on its trajectory) without breaking the detailed balance condition for the Markov chain Monte Carlo (MCMC) transition kernel. $S^{D-1}(r)$ is a compact manifold where any two points q(0), $q(t)\in {\mathcal{S}}^{D-1}(r)$ have bounded geodesic distance πr. We adopt the stopping rule for the leapfrog when the sampler exits the orthant of the initial state, that is, the trajectory measured in geodesic distance is at least $\frac{\pi }{2}r$, which is equivalent to 〈q(0), q(t)〉 < 0. On the other hand, this condition may not be satisfied within reasonable number of iterations because the geometric integrator does not exactly follow a geodesic in general (only the middle part does), therefore we set some threshold T_max for the number of tests, and adopt the following ‘Two-Orthants’ (as the starting and end points occupy two orthants) rule for the number of leapfrogs:

$$T_{2\mathit{\text{orth}}}=\underset{\tau \in \left\{0,\cdots ,T_{\text{max}}\right\}}{\text{min}}\left\{\tau :\langle q_0,q_{\tau }\rangle <0\right\}.$$ (3.3)

Alternatively, one can stop the leapfrog steps in a stochastic way based on the geodesic distance travelled:

$$T_{\mathit{\text{stoch}}}=\underset{\tau }{\text{min}}\left\{\tau :Z_{\tau }=0\right\},Z_{\tau }~\mathrm{Bern}\left(p_{\tau }\right),p_{\tau }=\frac{r^{-2}\langle q_0,q_{\tau }\rangle +1}{2}.$$ (3.4)

These stopping criteria are already time reversible, so the recursive binary tree as in ‘No-U-Turn’ algorithm (Hoffman and Gelman, 2014) is no longer needed.

Lastly, we adopt the dual averaging scheme (Nesterov, 2009) for the adaptation of step size h. See Hoffman and Gelman (2014) for more details. We summarize our Adaptive Spherical Hamiltonian Monte Carlo (adp-SphHMC) in the supplementary file (Lan et al., 2019).

To sample L (or L_t), we could update each row vector $l_i\in {\mathcal{S}}_0^{i-1}$ in parallel, and accept/reject vech^⊤(L) (or vech^⊤(L_t)) simultaneously in terms of the sum of total energy of all components. We refer to the resulting algorithm as Δ-Spherical HMC (Δ-SphHMC).

The computational complexity involving GP prior is $\mathcal{O}\left(N^3\right)$, and that of the likelihood evaluation is $\mathcal{O}\left(M{D}^2\right)$. MCMC updates of ${\tilde{\mu }}_{N\times D}$, ${\tilde{\sigma }}_{N\times D}$, ${\tilde{L}}_{N\times D\times D}$ have complexity $\mathcal{O}(ND)$, $\mathcal{O}(ND)$ and $\mathcal{O}\left(N{D}^2\right)$ respectively. To scale up applications to larger dimension D, one could preliminarily classify data into groups, and arrange the corresponding blocks of their covariance/correlation matrix in some ‘band’ along the main diagonal assuming no correlation among groups. More specifically, we can assume L_t is w-band lower triangular matrix for each time t, i.e. l_ij = 0 for i < j or i − j ≥ w, then the resulting covariance/correlation matrix will be (2w − 1)-banded. In this way the complexity of likelihood evaluation and updating $\tilde{L}$ will be reduced to $\mathcal{O}(MwD)$ and $\mathcal{O}(NwD)$ resepctively. Therefore the total computational cost would scale linearly with the dimension D. This technique will be investigated in Section 4.2.

4. Simulation Studies

In this section, we use simulated examples to illustrate the advantage of our structured models for covariance. First, we consider the normal-inverse-Wishart problem. Since there is conjugacy and we know the true posterior, we use this to verify our method and investigate flexible priors in Section 2.2. Then we test our dynamical modeling method in Section 2.3 on a periodic process model. Our model manifests full flexibility compared to a state-of-the-art nonparametric covariance regression model based on latent factor process (Fox and Dunson, 2015).

4.1. Normal-inverse-Wishart Problem

Consider the following example involving inverse-Wishart prior

$$\begin{gathered} y_n|\Sigma ~\mathcal{N}\left({\mu }_0,\Sigma \right),n=1,\cdots ,N, \\ \Sigma ~{\mathcal{W}}_D^{-1}(\Psi ,\nu ). \end{gathered}$$ (4.1)

It is known that the posterior of Σ|Y is still inverse-Wishart distribution:

$$\Sigma |Y~{\mathcal{W}}_D^{-1}\left(\Psi +\left(Y-{\mu }_0\right){\left(Y-{\mu }_0\right)}^{\top },\nu +N\right),Y={\left[y_1,\cdots ,y_N\right]}^{\top }.$$ (4.2)

We consider dimension D = 3 and generate data Y with μ₀ = 0, $\Sigma ={\Sigma }_0=\frac{1}{11}\left(I+11^{\top }\right)$ for N = 20 data points so that the prior is not overwhelmed by data.

Verification of Validity

Specifying conditional priors based on (2.8) in the structured model (2.6), we want to check the validity of our proposed method by comparing the posterior estimates using Δ-SphHMC agains the truth (4.2).

We sample τ := log(σ) using standard HMC and U* using Δ-SphHMC. They are updated in Metropolis-Within-Gibbs scheme. 10⁶ samples are collected after burning the first 10% and subsampling every 1 of 10. For each sample of τ and vech(U*), we calculate Σ = diag(e^τ)U*(U*)^⊤ diag(e^τ). Marginal densities of entries in Σ are estimated with these samples and plotted against the results by direct sampling in Figure 3. Despite of sampling variance, these estimates closely match the results by direct sampling, indicating the validity of our proposed method.

Figure 3:

Marginal posterior densities of σ_ij in the normal-inverse-Wishart problem. Solid blue lines are estimates by Δ-SphHMC and dashed red lines are estimates by direct sampling. All densities are estimated with 10⁶ samples.

Examining Flexibility of Priors

We have studied several spherical priors for the Cholesky factor of correlation matrix proposed in Section 2.2. Now we examine the flexibility of these priors in providing prior information for correlation with various parameter settings.

With the same data generated according to (4.1), we now consider the squared-Dirichlet prior (2.10) for L in the structured model (2.6) with the following setting

$$\begin{gathered} {\tau }_i=\text{log}\left({\sigma }_i\right)~\mathcal{N}\left(0,{0.1}^2\right),i=1,\cdots ,D, \\ {l}_i~{\mathrm{Dir}}^2\left({\alpha }_i\right),{\alpha }_i=\left(\alpha 1_{i-1},{\alpha }_0\right),i=2,\cdots ,D, \end{gathered}$$ (4.3)

where we consider three cases i) α = 1, α₀ = 1; ii) α = 0.1, α₀ = 1; iii) α = 0.1, α₀ = 10.

We generate 10⁶ prior samples (according to (4.3)) and posterior samples (by Δ-SphHMC) for L respectively and covert them to P = LL^⊤. For each entry of ρ_ij, we estimate the marginal posterior (prior) density based on these posterior (prior) samples. The posteriors, priors and maximal likelihood estimates (MLEs) of correlations ρ_ij are plotted in Figure 4 for different α’s respectively. In general, the posteriors are compromise between priors and the likelihoods (MLEs). With more and more weight (through α) put around the poles (last component) of each factor sphere, the priors become increasingly dominant that the posteriors (red dash lines) almost fall on priors (blue solid lines) when α = (0.1, 0.1, 10). In this extreme case, the squared-Dirichlet distributions induce priors in favor of trivial (zero) correlations. We have similar conclusion on Bingham prior and von Mises-Fisher prior but results are reported in Section E.1 of the supplementary file (Lan et al., 2019).

Figure 4:

Marginal posterior, prior (induced from squared-Dirichlet distribution) densities of correlations and MLEs with different settings for concentration parameter α, estimated with 10⁶ samples.

4.2. Simulated Periodic Processes

In this section, we investigate the performance of our dynamic model (2.11) on the following periodic process example

$$\begin{gathered} y(t)~{\mathcal{N}}_D(\mu (t),\Sigma (t)),\Sigma (t)=L(t)L{(t)}^{\top }°S,t\in [0,2], \\ {\mu }_i(t)=\text{sin}(it\pi /D),L_{ij}(t)={\left(-1\right)}^i\text{sin}(it\pi /D){\left(-1\right)}^j\text{cos}(jt\pi /D),j\le i=1,\cdots ,D, \\ {S}_{ij}={\left(\left|i-j\right|+1\right)}^{-1},i,j=1,\cdots ,D. \end{gathered}$$ (4.4)

Based on the model (4.4), we generate M trials (process realizations) of data y at N evenly spaced points for t in [0, 2], and therefore the whole data set {y(t)} is an M × N × D array. We first consider D = 2 to investigate the posterior contraction phenomena and the model flexibility; then we consider D = 100 over a shorter period [0, 1] to show the scalability using the ‘w-band’ structure.

Posterior Contraction

Posterior contraction describes the phenomenon that the posterior concentrates on smaller and smaller neighborhood of the true parameter (function) given more and more data (van der Vaart and van Zanten, 2008a). We investigate such phenomena in both mean functions and covariance functions in our model (2.11) using the following settings i) M = 10,N = 20; ii) M = 100,N = 20; iii) M = 10,N = 200; iv) M = 100,N = 200.

To fit the data using the model (2.11), we set s = 2, a = (1, 1, 1), b = (0.1, 10⁻³, 0.2), m = (0, 0, 0) for all settings, V = (1,0.5,1) for N = 20 and V = (1,1,0.3) for N = 200. We also add an additional nugget of 10⁻⁵I_n to all the covariance kernel of GPs to ensure non-degeneracy. Following the procedure in Section 3.1, we run MCMC for 1.5 × 10⁵ iterations, burn in the first 5 × 10⁴ and subsample 1 for every 10. Based on the resulting 10⁴ posterior samples, we estimate the underlying mean functions and covariance functions and plot the estimates in Figure 5.

Figure 5:

Estimation of the underlying mean functions μ_t (left in each of 4 subpannels) and covariance functions Σ_t (right in each of 4 subpannels) of 2-dimensional periodic processes. M is the number of trials, and N is the number of discretization points. Dashed lines are true values, solid lines are estimates and shaded regions are 95% credible bands.

Note in Figure 5, both M and N have effect on the amount of data information thereafter on the posterior contraction but the contraction rate may depend on them differently. Both mean and covariance functions have narrower credible bands for more discretization points N (comparing N = 20 in the first row with N = 200 for the second row). On the other hand, both posteriors contract further with more trials M (comparing M = 10 in the first column agains M = 100 for the second column). In general the posterior of mean function contracts to the truth faster than the posterior of covariance function. With M = 100 trials and N = 200 discretization points, both mean and covariance functions are almost recovered by the model (2.11).

Full Flexibility

Our method (2.11) grants full flexibility because it models mean, variance and correlation processes separately. This is particularly useful if they behave differently. It contrasts with latent factor based models that tie mean and covariance processes together. One of the state-of-the-art models of this type is Bayesian nonparametric covariance regression (Fox and Dunson, 2015):

$$y(x)~{\mathcal{N}}_D(\mu (x),\Sigma (x)),\mu (x)=\mathrm{\Lambda }(x)\psi (x),\Sigma (x)=\mathrm{\Lambda }(x)\mathrm{\Lambda }{(x)}^{\top }+{\Sigma }_0.$$ (4.5)

We tweak the simulated example (4.4) for D = 2 to let mean and correlation processes have higher frequency than variance processes, as shown in the dashed lines in Figure 6. We generate M = 10 trials of data over N = 200 evenly spaced points. In this case, the true mean processes μ(x) and true covariance processes Σ(x) behave differently but are modeled with a common loading matrix Λ(x) in model (4.5). This imposes difficulty on (4.5) to have a latent factor process ψ(x) that could properly accommodate the heterogeneity in mean and covariance processes. Figure 6 shows that due to this reason, latent factor based model (4.5) (upper row) fails to generate satisfactory fit for all of the mean, variance and correlation processes. Our fully flexible model (2.11) (bottom row), on the contrary, successfully produces more accurate characterization for all of them. Note that this artificial example is used to demonstrate the flexibility of our dynamic model (2.11). For cases that are not as extreme, (4.5) may performance equally well. See more discussion in Section 6 and more details in Section E.2 in the supplementary file (Lan et al., 2019).

Figure 6:

Estimation of the underlying mean functions μ_t (left column), variance functions σ_t (middle column) and correlation function ρ_t (right column) of 2-dimensional periodic processes, using latent factor process model (upper row) and our flexible model (lower row), based on M = 10 trials of data over N = 200 evenly spaced points. Dashed lines are true values, solid lines are estimates and shaded regions are 95% credible bands.

Scalability

Now we use the same simulation model (4.4) for D = 100 dimensions to test the scalability of our dynamic model (2.11). However instead of the full covariance, we consider a covariance matrix with 110 non-zero off-diagonal entries ρ_1,2, ρ_3,4,⋯,ρ_99,100 and a few outside the ‘2-band’ of the diagonal. The sparsity structure of the covariance is shown in the left panel of Figure 7 where the red lines indicate the ‘2-band’ structure. We focus on the correlation process in this example, thus set μ_t ≡ 0 and σ_t ≡ 1, for t ∈ [0, 1]. More specifically when generating the data {y_t}, we only use L_ij and S_ij in (4.4) for non-zero entries.

Figure 7:

Underlying structure of the correlation matrix (left), selective posterior estimates of the correlation functions P_t (middle), and its Frobenius-norm distance to the truth (right) based on 100-dimensional periodic processes with 2-band structure, using M = 100 trials over N = 100 discretization points. Dashed lines are true values, solid lines are estimates and shaded regions are the 95% credible bands.

To apply our dynamical model (2.11) in this setting, we let L_t have ‘w-band’ structure with w = 2 at each time t. Setting s = 2, a = 1, b = 0.1, m = 0 and V = 10⁻³, N = 100 and M = 100, we repeat the MCMC runs for 7.5 × 10⁴ iterations, burn in the first 2.5 × 10⁴ and subsample 1 for every 10 to obtain 5 × 10³ posterior samples in the end. Based on those samples, we estimate the underlying correlation functions and only plot selective correlations ρ_1,2, ρ_3,12, ρ_50,51 and ρ_99,100 in Figure 7. With the ‘w-band’ structure, we have less entries in the covariance matrix and focus on the ‘in-group’ correlation. Our dynamical model (2.11) is sensitive enough to discern the informative non-zero components from the non-informative ones in these correlation functions. Unit-vector GP priors provide flexibility for the model to capture the changing pattern of informative correlations. The middle panel of Figure 7 shows that except ρ_3,12 which falls out of the ‘2-band’ as indicated by red circle in the left panel of Figure 7, the model (2.11) correctly identify the non-zero components ρ_1,2 and ρ_99,100 and characterize their evolution. The right panel shows that the relative Frobenius-norm distance between the estimated and true correlation matrices, $\Vert \widehat{P}(t)-P(t){\Vert }_F/\Vert P(t){\Vert }_F$ is small, indicating that our dynamic model (2.11) performs well with higher dimension in estimating complex dependence structure among multiple stochastic processes.

5. Analysis of Local Field Potential Activity

Now we use the proposed model (2.11) to analyze a local field potential (LFP) activity dataset. The goal of this analysis is to elucidate how memory encoding, retrieval and decision-making arise from functional interactions among brain regions, by modeling how their dynamic connectivity varies during performance of complex memory tasks. Here we focus on LFP activity data recorded from 24 electrodes spanning the dorsal CA1 subregion of the hippocampus as rats performed a sequence memory task (Allen et al., 2014, 2016; Ng et al., 2017; Holbrook et al., 2017). The task involves repeated presentations of a sequence of odors (e.g., ABCDE) at a single port and requires rats to correctly determine whether each odor is presented ‘in sequence’ (InSeq; e.g., ABCDE; by holding their nosepoke response until the signal at 1.2s) or ‘out of sequence’ (OutSeq; e.g., ABDDE; by withdrawing their nose before the signal). In previous work using the same dataset, Holbrook et al. (2016) used a direct MCMC algorithm to study the spectral density matrix of LFP from 4 selected channels. However, they did not examine how their correlations varied across time and recording site. These limitations are addressed in this paper.

We focus our analyses on the time window from 0ms to 750ms (with 0 corresponding to when the rat’s nose enters the odor port). Critically, this includes a time period during which the behavior of the animal is held constant (0–500ms) so differences in LFP reflect the cognitive processes associated with task performance, and, to serve as a comparison, a time period near 750ms during which the behavioral state of the animal is known to be different (i.e., by 750ms the animal has already withdrawn from the port on the majority of OutSeq trials, but is still in the port on InSeq trials). We also focus our analyses on two sets of adjacent electrodes (electrodes 20 and 22, and electrodes 8 and 9), which allows for comparisons between probes that are near each other (<1mm; i.e., 20:22 and 8:9) or more distant from each other (>2mm; i.e., 20:8, 20:9, 22:8, and 22:9). Figure 8 shows M = 20 trials of these LFP signals from D = 4 channels under both InSeq and OutSeq conditions. Our main objective is to quantify how correlations among these LFP channels varied across trial types (InSeq vs OutSeq) and over time (within the first 750ms of trials). To do so, we discretize the time window of 0.75 seconds into N = 300 equally-spaced small intervals. Under each experiment condition (InSeq or OutSeq), we treat all the signals as a 4 dimensional time series and fit them using our proposed dynamic correlation model (2.11) in order to discover the evolution of their relationship. Note that we model the mean, variance, and correlation processes separately but only report findings about the evolution of correlation among those brain signals.

Figure 8:

LFP signals on “in sequence” and “out of sequence” trials. It is difficult to identify differences between the two conditions based on a mere visual inspection of the LFPs.

We set s = 2, a = (1, 1, 1), b = (1, 0.1, 0.2), m = (0, 0, 0), V = (1, 1.2, 2); and the general results are not very sensitive to the choice of these fine-tuning parameters. We also scale the discretized time points into (0, 1] and add an additional nugget of 10⁻⁵I_n to the covariance kernel of GPs. We follow the same procedure in Section 3.1 to collect 7.5 × 10⁴ samples, burn in the first 2.5 × 10⁴ and subsample 1 for every 10. The resulting 10⁴ samples yield estimates of correlation processes as shown in Figure 9 for beta-filtered traces (20–40Hz) but similar patterns were also observed for theta-filtered traces (4–12Hz; see the supplement). The bottom panel of Figure 9 shows the dissimilarity between correlation processes under different conditions measured by the Frobenius norm of their difference.

Figure 9:

Estimated correlation processes of LFPs (beta) under in-sequence condition (top), out-of-sequence condition (middle) and the (Frobenius) distance between two correlation matrices (bottom).

Our approach revealed many important patterns in the data. First, it showed that electrodes near each other (20:22 and 8:9) displayed remarkably high correlations in their LFP activity on InSeq and OutSeq trials, whereas correlations were considerably lower among more distant electrodes (20:8, 20:9, 22:8, and 22:9). Second, it revealed that the correlations between InSeq and OutSeq matrices evolved during the presentation of individual trials. These results are consistent with other analyses on learning (see, e.g., Fiecas and Ombao, 2016). As expected, InSeq and OutSeq activity was very similar at the beginning of the time window (e.g., before 350ms), which is before the animal has any information about the InSeq or OutSeq status of the presented odor, but maximally different at the end of the time window, which is after it has made its response on OutSeq trials. Most important, however, is the discovery of InSeq vs OutSeq differences before 500ms, which reveal changes in neural activity associated with the complex cognitive process of identifying if events occurred in their expected order. These findings highlight the sensitivity of our novel approach, as such differences have not been detected with traditional analyses. Interested readers can find more results about all the 12 channels in Section E.3 of the supplementary file (Lan et al., 2019).

6. Conclusion

In this paper, we propose a novel Bayesian framework that grants full flexibility in modeling covariance and correlation matrices. It extends the separation strategy proposed by Barnard et al. (2000) and uses the Cholesky decomposition to maintain the positive definiteness of the correlation matrix. By defining distributions on spheres, a large class of flexible priors can be induced for covariance matrix that go beyond the commonly used but restrictive inverse-Wishart distribution. Furthermore, the structured models we propose maintain the interpretability of covariance in terms of variance and correlation. Adaptive Δ-Spherical HMC is introduced to handle the intractability of the resulting posterior. Furthermore, we extend this structured scheme to dynamical models to capture complex dependence among multiple stochastic processes, and demonstrate the effectiveness and efficiency in Bayesian modeling covariance and correlation matrices using a normal-inverse-Wishart problem, a simulated periodic process, and an analysis of LFP data. In addition, we provide both theoretic characterization and empirical investigation of posterior contraction for dynamically covariance modeling, which to our best knowledge, is a first attempt.

In this work, we consider the marginal (pairwise) dependence among multiple stochastic processes. The priors for correlation matrix specified through the sphere-product representation are in general dependent among component variables. For example, the method we use to induce uncorrelated prior between y_i and y_j (i < j) by setting l_jk ≈ 0 for k ≤ i has a direct consequence that Cor(y_i′, y_j) ≈ 0 for i′ ≤ i. In another word, more informative priors (part of the components are correlated) may require careful ordering in {y_i}. To avoid this issue, one might consider the inverse of covariance (precision) matrices instead. This leads to modeling the conditional dependence, or Markov network (Dempster, 1972; Friedman et al., 2008). Our proposed methodology applies directly to (dynamic) precision matrices/processes, which will be our future direction.

To further scale our method to problems of greater dimensionality in future, one could explore the low-rank structure of covariance and correlation matrices, e.g. by adopting the similar factorization as in (Fox and Dunson, 2015) and assuming ${\mathrm{vech}}^{\top }\left(L_t\right)\in {\left({\mathcal{S}}^k\right)}^D$ for some k ≪ D, or impose some sparse structure on the precision matrices.

We have proved that the posterior of covariance function contracts at a rate given by the general form of concentration function (van der Vaart and van Zanten, 2008a). Empirical evidence (Section 4.2) shows that the posterior of covariance contracts slower than that of mean. More theoretical works is needed to compare their contraction rates. Also, future research could involve investigating posterior contraction in covariance regression with respect to the optimal rates under different GP priors.

While our research has generated interesting new findings regarding brain signals during memory tasks, one limitation of our current analysis on LFP data is that it is conducted on a single rat. The proposed model can be generalized to account for variation among rats. In the future, we will apply this sensitive approach to other datasets, including simultaneous LFP recordings from multiple brain regions in rats as well as BOLD functional magnetic resonance imaging (fMRI) data collected from human subjects performing the same task.