Efficient Bayesian functional principal component analysis of irregularly-observed multivariate curves

Abstract

The analysis of multivariate functional curves has the potential to yield important scientific discoveries in domains such as healthcare, medicine, economics and social sciences. However, it is common for real-world settings to present longitudinal data that are both irregularly and sparsely observed, which introduces important challenges for the current functional data methodology. A Bayesian hierarchical framework for multivariate functional principal component analysis is proposed, which accommodates the intricacies of such irregular observation settings by flexibly pooling information across subjects and correlated curves. The model represents common latent dynamics via shared functional principal component scores, thereby effectively borrowing strength across curves while circumventing the computationally challenging task of estimating covariance matrices. These scores also provide a parsimonious representation of the major modes of joint variation of the curves and constitute interpretable scalar summaries that can be employed in follow-up analyses. Estimation is conducted using variational inference, ensuring that accurate posterior approximation and robust uncertainty quantification are achieved. The algorithm also introduces a novel variational message passing fragment for multivariate functional principal component Gaussian likelihood that enables modularity and reuse across models. Detailed simulations assess the effectiveness of the approach in sharing information from sparse and irregularly sampled multivariate curves. The methodology is also exploited to estimate the molecular disease courses of individual patients with SARS-CoV-2 infection and characterise patient heterogeneity in recovery outcomes; this study reveals key coordinated dynamics across the immune, inflammatory and metabolic systems, which are associated with long-COVID symptoms up to one year post disease onset. The approach is implemented in the R package bayesFPCA.

1. Introduction

The availability of longitudinal datasets is rising sharply, benefitting from the progress of technologies and monitoring tools in domains where cross-sectional studies were previously the norm. This paradigm shift calls for principled statistical approaches that can flexibly model curves and uncover shared dynamics across them to enhance statistical power and interpretation.

Functional data analysis (FDA) has enjoyed increasing applicability in various areas, including neuroimaging (Wang et al., 2019) and wearable technology (Goldsmith et al., 2015). Functional principal components analysis (FPCA) is a key technique in this field that is used for dimensionality reduction on the inherently infinite-dimensional functional data. The resulting principal components and scores can then be used for further analysis, such as an orthogonal basis and uncorrelated covariates in functional regression models. Such techniques, however, are classically restricted to univariate datasets, but more realistic problems in biomedical research involve numerous functional observations for each subject. In fact, the current article was inspired by longitudinal measurements on several molecular markers to characterise systemic recovery from SARS-CoV-2 infection (Bergamaschi et al., 2021; Ruffieux et al., 2023).

The multivariate setting for FPCA, which we call multivariate FPCA (mFPCA), was investigated by Ramsay and Silverman (2005, Chapter 8.5) for dense functional data, where they proposed concatenating the multivariate functional data and applying multivariate PCA to the resulting sample of long vectors. However, real-world multivariate functional data generally involves complicated observation settings, with sparse and irregular data on correlated variables, possibly also irregularly sampled across different subjects. For irregular multivariate functional data, extensions of covariance-based FPCA methods (Yao et al., 2005) have become popular, permitting adaptations to multivariate longitudinal datasets. Happ and Greven (2018) estimate covariances and cross-covariances via the scores estimated from univariate FPCA, however accurate covariance estimation may be difficult for sparse functional data because scores from univariate FPCA are shrunk towards zero. Li et al. (2020) estimate the covariance and cross-covariance function via B-spline smoothing through a tensor product formulation, which provides fast and accurate estimation in the presence of sparse functional data. Once an estimate of the covariance function has been attained, eigenfunctions and scores can be gathered via an eigendecomposition.

Although covariance-based mFPCA methods have provided a means of analysing sparse functional data, the bivariate smoothing operation can be a computationally challenging task for large datasets. In the case of univariate FPCA, Bayesian implementations allow for direct inference on the eigenfunctions and scores without requiring an estimate of the covariance function as all quantities are considered unknown and estimated jointly. Such approaches have built on or are similar to the probabilistic PCA framework that was introduced by Tipping and Bishop (1999) and Bishop (1999). James et al. (2000) used an expectation maximisation algorithm for estimation and inference in the context of sparsely observed curves. Variational inference for FPCA was introduced by van der Linde (2008) via a generative model with a factorised approximation of the full posterior density function. Goldsmith et al. (2015) introduced a fully Bayes framework for multilevel function-on-scalar regression models with FPCA applied to two levels of residuals. Nolan et al. (2023) introduced a variational message passing framework for Bayesian inference on the model parameters. The parameters in the latter approach are updated by messages passed between computational units known as fragments (Wand, 2017), facilitating efficient extensions of univariate Bayesian FPCA to more elaborate models, which Nolan et al. (2023) achieved for the multilevel setting. Finally, selecting the numbers of functional principal components and basis functions for representing the latent components is also an important problem. Although such choices are typically not the object of extensive sensitivity analysis in the FPCA literature, various approaches have been described, based on cross-validation (e.g., Huang et al., 2008), information criteria (e.g., AIC or BIC; Yao et al., 2005), estimation of the variance explained by the components (e.g., Greven et al., 2011), or Bayesian model selection by placing prior distributions on these numbers (e.g., Suarez and Ghosal, 2017). In some of these approaches, inference is performed with an upper bound on the number of components, relying on regularisation to adaptively discard the superfluous components.

While Bayesian mFPCA remains underdeveloped, Bayesian methodologies for various univariate and multivariate FDA models have been established. In particular, in the context of Bayesian functional latent factor models, a substantial focus has been placed on achieving ordered shrinkage and rank selection. Notable approaches include the use of multiplicative gamma process shrinkage priors, as proposed by Bhattacharya and Dunson (2011); Montagna et al. (2012), and cumulative shrinkage process priors, as proposed by Legramanti et al. (2020); Kowal and Canale (2023). These approaches employ prior distributions on factors or loading coefficients to encourage increasing shrinkage on higher-index factors, effectively removing irrelevant components and encoding ordering constraints.

In this article, we present a variational mFPCA framework based on a Bayesian hierarchical model that allows borrowing information across related variables, from sparsely or irregularly sampled functional curves. We employ a generalisation of the Karhunen–Loève theorem for multivariate Hilbert spaces (Happ and Greven, 2018) to implement a direct representation of the curves as multivariate Karhunen–Loève expansions, with joint estimation of all model parameters. Specifically, we formulate a hierarchical functional factor model for multivariate curves, which enforces the necessary regularisation on variable-specific factor loading curves. Rather than encoding identifiability constraints into the model itself, such as via the above-mentioned dedicated prior formulations proposed in the context of FDA, we estimate the components up to a rotation of the parameter space and subsequently restore orthonormalisation and ordered contributions to the data variability, required by the mFPCA decomposition. We develop variational message passing (VMP) and mean-field variational Bayes (MFVB) algorithms for our proposed model. Variational inference constitutes a scalable alternative to MCMC inference whose computational burden can be prohibitive for the large multivariate functional settings we are interested in. Crucially, it also permits approximating the posterior distribution of all parameters, unlike other approximate Bayesian inference approaches, such as the expectation-maximisation algorithm, which only provide point estimates. We evaluate the statistical and computational performance of our approach in simulations emulating real-data settings, benchmarking it against MCMC inference on the same model, as well as against the frequentist mFPCA approach of Happ and Greven (2018) and separate applications of Bayesian univariate FPCA (Nolan et al., 2023). Doing so, we illustrate the effectiveness of our joint framework in pulling information across sparsely and irregularly sampled curves to improve estimation of subject-level trajectories, principal component scores and latent functions, and we assess the impact of model misspecification. We then exploit our approach to clarify the latent dynamics driving patient-to-patient variability in recovery from COVID-19 using detailed longitudinal data covering one year post infection.

This article is organised as follows. Section 2 presents the COVID-19 study, and motivates the need for new methodology to tackle biomedical research questions based on complex longitudinal measurement settings. Section 3 recalls the multivariate Karhunen–Loève decomposition and presents our joint hierarchical model for mFPCA. Section 4 details our variational inference framework and describes a post-variational procedure to obtain orthonormal eigenfunctions and uncorrelated scores. Section 5 presents the results of a series of simulation studies. Section 6 applies our approach to the COVID-19 study and discusses the possible biomedical implications of our findings, focusing on how the immune, metabolic and inflammatory systems jointly coordinate organismal recovery. Section 7 summarises our work and suggests further methodological developments. We provide a software implementation for our approach as an R package called bayesFPCA.

2. Data and motivating example

COVID-19 is a systemic disease, causing widespread dysregulation across the immune, metabolic and inflammatory systems (Lucas et al., 2020; Masuda et al., 2021). While biological alterations resolve for most individuals soon after the acute phase, they can also be associated with short- and long-term complications, such as ICU admission, prolonged symptoms (“long COVID”) or death. Evolution of clinical and molecular parameters over time has also been shown to be very heterogeneous between patients (Holmes et al., 2021; Peluso et al., 2021), but the mechanisms underlying the different disease dynamics, and the coordination of these dynamics across biological systems, remain largely unclear. Such an understanding could guide the development therapeutic strategies to anticipate and prevent serious disease trajectories, as well as help formulate individualised recommendations for patients with long COVID. It could also provide a basis for studying other infectious diseases, such as caused by the Epstein–Barr virus (EBV) for example.

The numerous studies conducted over the past years have achieved varying degrees of success. These research efforts have however underscored the necessity of coupling access to detailed data, obtained from patient samples assayed repeatedly over months post infection, and use of principled statistical approaches tailored to longitudinal multi-parameter settings. Such approaches should be equipped to (i) jointly model several blood markers across different systems, in order to quantify coordinated alterations in organismal functions; (ii) disentangle the inter- and intra-patient temporal covariation of these alterations over the course of illness; (iii) reconstruct the marker trajectories at the patient level; (iv) uncover latent dynamics driving incomplete recovery in order to relate them to biological pathways influencing the risk of death and of long COVID.

Here we propose to develop new methodology based on these desiderata, and apply it to the analysis of data comprising longitudinal measurements of polar metabolites, serum cytokines and C-reactive protein levels from a cohort of 215 infected subjects with different clinical severities (Bergamaschi et al., 2021; Ruffieux et al., 2023). Specifically, we will model the disease trajectories of symptomatic SARS-CoV-2 PCR-positive patients admitted to Cambridge Hospitals (CITIID-NIHR BioResource COVID-19 Collaboration), and compare them with measurements from uninfected individuals as controls.

This problem requires analysis tools adapted to sparse multivariate functional settings, since multiple markers are quantified for each patient over time, and some are scarcely observed (the number of observations per patient and marker may be as low as two). Moreover, the observation grid is irregular, as it differs both across markers for a same patient, and across patients for a same marker. Finally, the dataset is relatively large: it comprises longitudinal measurements across multiple cellular and molecular markers, for tens of subjects. As explained before, these characteristics (large-data setting with sparse observations on an irregular grid) are poorly handled by covariance-based multivariate frequentist FPCA approaches, as they induce unwanted shrinkage and computational intractabilities. This, and the need to flexibly borrow strength between markers from different biological systems, motivates the development of a Bayesian joint FPCA approach that can effectively pull information to estimate shared latent dynamics from related, yet sparsely observed curves.

We will detail our methodology in the next sections, and employ it on the COVID-19 data in Section 6 to estimate individual disease trajectories as well as patient-level scores summarising the disease dynamics of each patient. This will hopefully allow us to identify latent kinetics that are common to multiple markers and that drive patient heterogeneity, thereby clarifying how the different biological systems coordinate the organismal response to infection. We will also exploit patient questionnaires on long-COVID symptoms to illustrate the added value of our joint approach compared to separate univariate FPCA analyses of individual markers.

3. Multivariate functional principal components analysis

3.1. Karhunen–Loève representation of multivariate functional data

Univariate FPCA is concerned with dimensionality reduction of independent realisations of a random function x ∈ L²([0, 1]) into a finite eigenbasis that is constructed from the leading eigenfunctions of the covariance operator of x. In the multivariate setting, random functions are replaced by row vectors of random functions x(t) = [x⁽¹⁾(t₁) ⋯ x^(p)(t_p)] for t = (t₁, …, t_p)^⊺ ∈ [0, 1]^p. The vector random functions are elements of the Hilbert space ℋ ≡ L²([0, 1])^p, which is equipped with the norm defined by the inner product:

$${\langle f,g\rangle }_{\mathscr{H}}\equiv \sum _{j=1}^p\underset{0}{\overset{1}{\int }}{f}^{(j)}(t)g^{(j)}(t)dt,f,g\in \mathscr{H}.$$ (1)

Henceforth, the elements of ℋ will be referred to simply as random functions.

For x ∈ ℋ, the mean function is defined as μ(t) ≡ [𝔼{x⁽¹⁾(t₁)} ⋯ 𝔼{x^(p)(t_p)}]. Next, define the matrix of covariances C(s, t) for s, t ∈ [0, 1]^p, with (j, j′)-entry

$$C_{j{j}^{\prime }}\left(s_j,t_{j^{\prime }}\right)=ℂ\text{ov}\left\{x^{(j)}\left(s_j\right),x^{\left(j^{\prime }\right)}\left(t_{j^{\prime }}\right)\right\},s_j,t_{j^{\prime }}\in [0,1].$$ (2)

Finally, we define the covariance operator Σ : ℋ → ℋ with j′th element of Σf, f ∈ ℋ given by

$${\left(\text{Σ}f\right)}^{\left(j^{\prime }\right)}\left(t_{j^{\prime }}\right)\equiv \sum _{j=1}^p\underset{0}{\overset{1}{\int }}{C}_{j{j}^{\prime }}\left(s_j,t_{j^{\prime }}\right)f^{(j)}\left(s_j\right)d{s}_j,t_{j^{\prime }}\in [0,1].$$

According to the technical details of Proposition 2 of Happ and Greven (2018), there exists a complete orthonormal basis of eigenfunctions ψ_l ∈ ℋ, for l ∈ ℕ, of Σ such that Σψ_l = λ_lψ_l with λ_l → 0 as l → ∞.

These are the main ingredients for establishing the multivariate version of the FPCA decomposition. First, we have the multivariate version of Mercer’s Theorem, which states that $C_{jj}\left(s_j,t_j\right)={{\displaystyle \sum }}_{l=1}^{\infty }{\lambda }_l{\psi }_l^{(j)}\left(s_j\right){\psi }_l^{(j)}\left(t_j\right),$ for j = 1, …, p and s_j, t_j ∈ [0, 1]. Next, for a set of independent realisations x_i(t), i = 1, …, n, the multivariate Karhunen–Loève decomposition is the basis for the mFPCA expansion (Yao et al., 2005):

$$x_i(\mathit{\text{t}})=\mu (\mathit{\text{t}})+\sum _{l=1}^{\infty }{\zeta }_{il}{\psi }_l(\mathit{\text{t}}),i=1,\dots ,n,\mathit{\text{t}}\in {[0,1]}^p,$$ (3)

where ζ_il = 〈{x_i − μ}, ψ_l〉_ℋ are the principal component scores. The ζ_il are independent across i and uncorrelated across l, with 𝔼(ζ_il) = 0 and 𝕍 ar(ζ_il) = λ_l. The decay of the eigenvalues with increasing l ensures that we can truncate the sum in (3) for large enough L, such that

$${\widehat{x}}_i(\mathit{\text{t}})=\mu (\mathit{\text{t}})+\sum _{l=1}^L{\zeta }_{il}{\psi }_l(\mathit{\text{t}}),i=1,\dots ,n,\mathit{\text{t}}\in {\left[0,1\right]}^p.$$ (4)

Each observation x_i can then be represented by its vector of scores ζ_i = (ζ_i1, …, ζ_iL)^⊺ and used for further analysis, such as regression (Müller and Stadtmüller, 2005). The key difference between mFPCA and applying univariate FPCA to each of the p variables is that there is only one score ζ_il for the lth multivariate eigenfunction in the former approach, whereas there would be a separate score for each element of the lth eigenfunction in the latter approach. In this way, mFPCA inherently accounts for correlations between the variables, thereby borrowing information across them and producing a common score for each component l = 1, …, L.

As in univariate FPCA (Nolan et al., 2023, Section 2), expansions similar to (4) are also possible, where

$${\widehat{x}}_i(\mathit{\text{t}})\equiv \mu (\mathit{\text{t}})+\sum _{l=1}^L{z}_{il}{h}_l(\mathit{\text{t}}),i=1,\dots ,n,\mathit{\text{t}}\in {\left[0,1\right]}^p,$$ (5)

where z_il are correlated across l, but remain independent across i, and the h_l are not orthonormal. Theorem 3.1 is a generalisation of Theorem 2.1 of Nolan et al. (2023), and it shows that an orthogonal decomposition of the resulting basis functions and weights is sufficient for establishing the appropriate estimates (4) from (5).

Theorem 3.1

Given the decomposition in (5), there exists a unique set of orthonormal eigenfunctions ψ₁, …, ψ_L and a set of uncorrelated scores ζ_i1, …, ζ_iL, i = 1, …, n, such that ${\widehat{x}}_i(\mathit{\text{t}})=\mu (\mathit{\text{t}})+{{\displaystyle \sum }}_{l=1}^L{\zeta }_{il}{\psi }_l(\mathit{\text{t}}),\mathit{\text{t}}\in {\left[0,1\right]}^p.$

Remark

The form of the proof of Theorem 3.1 is identical to that of Theorem 2.1 of Nolan et al. (2023), with inner products and norms in L² replaced by those in ℋ. Therefore, we refer the reader to the proof of Theorem 2.1 of Nolan et al. (2023) in Section A of their online supplementary material.

Theorem 3.1 permits direct estimation of the scores and eigenfunctions in the multivariate Karhunen–Loève decomposition (3), without initially estimating a covariance function as in Happ and Greven (2018) and Li et al. (2020). According to Theorem 3.1, we can simply orthogonalise the basis functions and decorrelate the weights to gather estimate of the orthonormal eigenfunctions and uncorrelated scores. There are several advantages in this method in that it does not require estimation or smoothing of a large covariance and can more directly handle sparse or irregular functional data.

3.2. A Bayesian hierarchical model for mFPCA

In practice, the functional data are collected as a set of noisy observations over discrete points in time. Let the set of design points for the ith subject’s measurements on the jth variable be summarised by the vector ${\mathit{\text{t}}}_i^{(j)}\equiv (t_{i1}^{(j)},\dots ,t_{i{n}_i^{(j)}}^{(j)}{)}^{\top }$. Then, the corresponding vector of observations is given by ${\mathit{\text{x}}}_i^{(j)}\equiv x_i^{(j)}\left({\mathit{\text{t}}}_i^{(j)}\right)+{\mathit{\text{ε}}}_i^{(j)}$, where ${\mathit{\text{ε}}}_i^{(j)}\stackrel{\text{ind}.}{~}\text{N}\left({\mathbf{0}}_{n_i^{(j)}},{\sigma }_{ϵ}^{(j)2}{\mathit{\text{I}}}_{n_i^{(j)}}\right)$. In addition, we set ${\mathit{\text{μ}}}_i^{(j)}\equiv {\mu }^{(j)}\left({\mathit{\text{t}}}_i^{(j)}\right)$ and ${\mathit{\text{ψ}}}_{il}^{(j)}\equiv {\psi }_l^{(j)}\left({\mathit{\text{t}}}_i^{(j)}\right)$. Then, the multivariate Karhunen–Loève decomposition in (3) takes the form:

$${\mathit{\text{x}}}_i^{(j)}={\mathit{\text{μ}}}_i^{(j)}+\sum _{l=1}^L{\zeta }_{il}{\mathit{\text{ψ}}}_{il}^{(j)}+{\mathit{\text{ε}}}_i^{(j)},i=1,\dots ,n,j=1,\dots ,p.$$ (6)

As already alluded to in Section 3.1, a key feature of this decomposition is the shared score parametrisation, ζ_i = (ζ_i1, …, ζ_iL), which enables borrowing strength across the p variables and provides a parsimonious subject-level representation of the variation in the data. This, combined with the flexible accommodation of variable- and subject-specific time grids, tailors it to irregular and sparse functional data settings, where estimation from scarcely observed curves particularly benefits from pulling information across all curves at different time points. Moreover, while the scores are common to all p variables, the latent functions ${\psi }_l^{(j)}(t)$ are variable-specific which enables flexible modelling tailored each variable’s dynamics, whose complexity may differ across dimensions of a same component l.

Another interesting modelling treatment of multivariate curves, in the broader functional data analysis field, is to allow for variable-specific scores, ${\zeta }_{il}^{(j)}$, but enforce common latent functions across the p variables, ψ_l(t), as for instance considered by Kowal et al. (2017). Under this perspective, the score for the jth variable determines how much the latent function contributes to its variability. Hence, such a setting can be useful for problems where one expects a common process for the p variables and can provide nuanced insights into how each variable uniquely relates to this process. An example in biology would be the modelling of expression levels of genes from a given molecular pathway (for instance related to inflammation), measured over time. It then may be reasonable to expect that the genes all “feed into” a single latent dynamics reflecting the same biological process or its disruption. Such a framework could retain interpretability through the inspection of gene-specific scores reflecting their relative contributions to this dynamics, possibly offering insights into the biological mechanisms of disease.

Finally, assuming variable-specific scores and latent functions is most flexible, and can be advised in cases where the underlying latent dynamics are not expected to share much across the p variables (as we will illustrate in our simulations of Section 5.6). However, if the dynamics are related, such a specification will fail to exploit shared structures. Kowal et al. (2017) also assume variable-specific functions and scores, but borrow information at a higher level in the model hierarchy, which may be seen as an intermediate approach, between a multivariate and a fully univariate treatment.

Hence, these model specifications each involve different assumptions about the nature of the latent processes underlying the multivariate curves; here, because the focus is on proposing a novel Bayesian treatment of multivariate FPCA, we consider the assumptions of shared scores and variable-specific latent functions in line with the multivariate Karhunen–Loève expansion (Equation (6)).

We represent continuous curves from discrete observations via semiparametric regression (Ruppert et al., 2003, 2009), with the mixed model-based penalised spline basis function representation, as in Durbán et al. (2005). The representations for the jth elements of the mean function and latent functions are:

$${\mu }^{(j)}(t)\approx {\beta }_{\mu ,0}^{(j)}+{\beta }_{\mu ,1}^{(j)}t+\sum _{k=1}^K{u}_{\mu ,k}^{(j)}{z}_k(t)\text{and}{\psi }_l^{(j)}(t)\approx {\beta }_{{\psi }_l,0}^{(j)}+{\beta }_{{\psi }_l,1}^{(j)}t+\sum _{k=1}^K{u}_{{\psi }_l,k}^{(j)}{z}_k(t),$$ (7)

for j = 1, …, p and l = 1, …, L, where {z_k(⋅)}_1≤k≤K is a suitable set of basis functions. Splines and wavelet families are the most common choices for the z_k; in our simulations, we use O’Sullivan penalised splines, which are described in Section 4 of Wand and Ormerod (2008). Next, set ${\mathit{\text{v}}}_{\mu }^{(j)}\equiv {\left({\beta }_{\mu ,0}^{(j)},{\beta }_{\mu ,1}^{(j)},u_{\mu ,1}^{(j)},\dots ,u_{\mu ,K}^{(j)}\right)}^{\top },{\mathit{\text{v}}}_{{\psi }_l}^{(j)}\equiv {\left({\beta }_{{\psi }_l,0}^{(j)},{\beta }_{{\psi }_l,1}^{(j)},u_{{\psi }_l,1}^{(j)},\dots ,u_{{\psi }_l,K}^{(j)}\right)}^{\top }$ and ${\mathit{\text{C}}}_i^{(j)}\equiv \left[\begin{array}{ccccc}{\mathbf{1}}_{n_i^{(j)}}& {\mathit{\text{t}}}_i^{(j)}& z_1\left({\mathit{\text{t}}}_i^{(j)}\right)& \cdots & z_K\left({\mathit{\text{t}}}_i^{(j)}\right)\end{array}\right]$. For notational convenience, the dependence that any matrix or vector has on the vector of observations times ${\mathit{\text{t}}}_i^{(j)}$ will be understood, rather than shown explicitly. For example, we use ${\mathit{\text{C}}}_i^{(j)}$, as opposed to ${\mathit{\text{C}}}_i^{(j)}\left({\mathit{\text{t}}}_i^{(j)}\right)$.

With these notational definitions at hand, we have ${\mathit{\text{μ}}}_i^{(j)}\approx {\mathit{\text{C}}}_i^{(j)}{\mathit{\text{v}}}_{\mu }^{(j)}$ and ${\mathit{\text{ψ}}}_{i,l}^{(j)}\approx {\mathit{\text{C}}}_i^{(j)}{\mathit{\text{v}}}_{{\psi }_l}^{(j)}$. Then simple derivations that stem from (6) show that the vector of observations on each response curve satisfies the representation ${\mathit{\text{x}}}_i^{(j)}={\mathit{\text{C}}}_i^{(j)}\left({\mathit{\text{v}}}_{\mu }^{(j)}+{{\displaystyle \sum }}_{l=1}^L{\zeta }_{il}{\mathit{\text{v}}}_{{\psi }_l}^{(j)}\right)+{\mathit{\text{ε}}}_i^{(j)}$ In the following, we set ${\mathit{\text{v}}}^{(j)}\equiv {\left({\mathit{\text{v}}}_{\mu }^{(j)\top },{\mathit{\text{v}}}_{{\psi }_1}^{(j)\top },\dots ,{\mathit{\text{v}}}_{{\psi }_L}^{(j)\top }\right)}^{\top }$ for j = 1, …, p. We are now in a position to introduce the Bayesian model:

$$\begin{array}{l}\begin{array}{ccc}{\mathit{\text{x}}}_i^{(j)}\mid {\mathit{\text{v}}}^{(j)},{\mathit{\text{ζ}}}_i,{\sigma }_{ϵ}^{(j)2}\stackrel{\text{ind}.}{~}\text{N}\left\{{\mathit{\text{C}}}_i^{(j)}\left({\mathit{\text{v}}}_{\mu }^{(j)}+\sum _{l=1}^L{\zeta }_{il}{\mathit{\text{v}}}_{{\psi }_l}^{(j)}\right),{\sigma }_{ϵ}^{(j)2}{\mathit{\text{I}}}_{n_i^{(j)}}\right\},& i=1,\dots ,n,& j=1,\dots ,p,\end{array}\\ \begin{array}{ccc}\left[\begin{array}{l}{\mathit{\text{v}}}_{\mu }^{(j)}\hfill \\ {\mathit{\text{v}}}_{{\psi }_l}^{(j)}\hfill \end{array}\right]|{\sigma }_{\mu }^{(j)2},{\sigma }_{{\psi }_l}^{(j)2}\stackrel{\text{ind}.}{~}\text{N}\left(\left[\begin{array}{c}{\mathbf{0}}_{K+2}\\ {\mathbf{0}}_{K+2}\end{array}\right],\left[\begin{array}{cc}{\mathbf{\text{Σ}}}_{\mu }^{(j)}& \mathit{\text{O}}\\ \mathit{\text{O}}& {\mathbf{\text{Σ}}}_{{\psi }_l}^{(j)}\end{array}\right]\right),& {\mathit{\text{ζ}}}_i\stackrel{\text{ind}.}{~}\text{N}\left({\mathbf{0}}_L,{\mathit{\text{I}}}_L\right),& l=1,\dots ,L,\end{array}\\ \begin{array}{c}\begin{array}{cc}{\sigma }_{\mu }^{(j)2}\mid a_{\mu }^{(j)}\stackrel{\text{ind}.}{~}\text{Inverse}-{\chi }^2\left(1,1/a_{\mu }^{(j)}\right),& a_{\mu }^{(j)}\stackrel{\text{ind}.}{\text{~}}\text{Inverse}-{\chi }^2\left(1,1/A^2\right),\end{array}\\ \begin{array}{cc}{\sigma }_{{\psi }_l}^{(j)2}\mid a_{{\psi }_l}^{(j)}\stackrel{\text{ind}.}{~}\text{Inverse}-{\chi }^2\left(1,1/a_{{\psi }_l}^{(j)}\right),& a_{{\psi }_l}^{(j)}\stackrel{\text{ind}.}{\text{~}}\text{Inverse}-{\chi }^2\left(1,1/A^2\right),\end{array}\\ \begin{array}{cc}{\sigma }_{ϵ}^{(j)2}\mid a_{ϵ}^{(j)}\stackrel{\text{ind}.}{~}\text{Inverse}-{\chi }^2\left(1,1/a_{ϵ}^{(j)}\right),& a_{ϵ}^{(j)}\stackrel{\text{ind}.}{\text{~}}\text{Inverse}-{\chi }^2\left(1,1/A^2\right),\end{array}\end{array}\end{array}$$ (8)

with ζ_i ≡ (ζ_i,1, …, ζ_i,L)^⊺, ${\mathbf{\text{Σ}}}_{\mu }^{(j)}\equiv \text{blockdiag}\left({\sigma }_{\beta }^2{\mathit{\text{I}}}_2,{\sigma }_{\mu }^{(j)2}{\mathit{\text{I}}}_K\right)$ and ${\mathbf{\text{Σ}}}_{{\psi }_l}^{(j)}\equiv \text{blockdiag}\left({\sigma }_{\beta }^2{\mathit{\text{I}}}_2,{\sigma }_{{\psi }_{\mathit{\text{l}}}}^{(j)2}{\mathit{\text{I}}}_K\right)$, where blockdiag_i=1,…,d (M_i) is the block diagonal matrix with matrices M_i, i = 1, …, d, arranged on the diagonal. Furthermore, ${\sigma }_{\beta }^2$ and A are user-specified hyperparameters; following Wand and Ormerod (2008), we recommend setting them to large values resulting in diffuse prior specifications. For instance, choosing σ_β = A = 10⁵ yields reliable estimates in our numerical experiments (Section 5). Note that the iterated inverse-χ² prior specification on each ${\sigma }_{\mu }^{(j)2}$, which involves an inverse-χ² prior on the auxiliary variable $a_{\mu }^{(j)}$, is equivalent to ${\sigma }_{\mu }^{(j)}$ ~ Half-Cauchy(A). This hierarchical construction based on auxiliary variables facilitates arbitrarily non-informative priors on standard deviation parameters (Gelman, 2006). The same comments apply to the iterated inverse-χ² specifications for each ${\sigma }_{{\psi }_1}^{(j)2},\dots ,{\sigma }_{{\psi }_L}^{(j)2},$ and ${\sigma }_{ϵ}^{(j)2}$.

In model (8), the prior variance of the scores is set to unity, which helps prevent undesired compensation effects between the scores and the latent functions’ spline coefficients, given that the variances of the latter are inferred. While estimating the score variances instead may provide a more direct account of the relevance of each component in explaining the variation in the functional data, this information can easily be regained in a post-hoc manner as we will explain in Section 4.4. Moreover, estimating the variable- and component-specific spline coefficient variances with our proposed specification enables flexible representation of the latent functions, adapting to variations in complexity across different components and variables. Such a specification also ensures that sufficient regularisation is induced, shrinking the coefficients of latent functions irrelevant to specific components and variables to zero. To check that the proposed formulation with fixed score variance is sufficient to flexibly account for the variation explained by the eigenfunctions and their level of smoothness, we compared it with an implementation where both sets of spline coefficient variances and score variances are inferred (Appendix D.1 of the Supplementary material). We observed no difference in accuracy, however estimating the score variances substantially increased the number of iterations until convergence, and hence the runtime, likely because of a more complicated parameter space.

The latent functions in model (8) are only identifiable up to rotation. Moreover, a unique and interpretable ordering these functions is also not enforced. Recall that, to obtain a multivariate FPCA representation, we require orthonormal multivariate eigenfunctions with respect to the ℋ inner product and uncorrelated scores with non-increasing variances. These orthonormality constraints imply that the latent functions can be seen as an orthonormal basis for the functional observations ${\mathit{\text{x}}}_i^{(j)}$, thereby preventing information redundancy among components. In Section 4.4, we will detail a post-processing treatment of the unconstrained posterior estimates to restore these orthogonality constraints along with the ordering of the components in capturing the variability in the functional data.

4. Variational Bayes inference

4.1. Background

For notational convenience, we set ${\mathit{\text{σ}}}_{ϵ}^2\equiv {\left\{{\sigma }_{ϵ}^{(j)2}\right\}}_{j=1,\dots ,p}$ and similarly for ${\mathit{\text{σ}}}_{\mu }^2,{\mathit{\text{σ}}}_{{\psi }_1}^2,\dots ,{\mathit{\text{σ}}}_{{\psi }_L}^2,{\mathit{\text{a}}}_{ϵ},{\mathit{\text{a}}}_{\mu }$, and ${\mathit{\text{a}}}_{{\psi }_1},\dots ,{\mathit{\text{a}}}_{{\psi }_L}$. We also write ${\mathit{\text{v}}}^{(j)}\equiv {\left({\mathit{\text{v}}}_{\mu }^{(j)\top },{\mathit{\text{v}}}_{{\psi }_1}^{(j)\top },\dots ,{\mathit{\text{v}}}_{{\psi }_L}^{(j)\top }\right)}^{\top }$, v ≡ (v⁽¹⁾, …, v^(p))^⊺, and ${\mathit{\text{x}}}_i\equiv {\left({\mathit{\text{x}}}_i^{(1)\top },\dots ,{\mathit{\text{x}}}_i^{(p)\top }\right)}^{\top },\mathit{\text{x}}\equiv {\left({\mathit{\text{x}}}_1^{\top },\dots ,{\mathit{\text{x}}}_n^{\top }\right)}^{\top }$. The full Bayesian inference on the model parameters requires estimating the posterior density $p\left(\mathit{\text{ν}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\mathit{\text{σ}}}_{ϵ}^2,{\mathit{\text{a}}}_{ϵ},{\mathit{\text{σ}}}_{\mu }^2,{\mathit{\text{a}}}_{\mu },{\mathit{\text{σ}}}_{{\psi }_1}^2,\dots ,{\mathit{\text{σ}}}_{{\psi }_L}^2,{\mathit{\text{a}}}_{{\psi }_1},\dots ,{\mathit{\text{a}}}_{{\psi }_L}\mid \mathit{\text{x}}\right)$. As will be shown in Section 5, classical inference via Markov chain Monte Carlo (MCMC) methods for model (8) can be very slow, even for moderate values of v. Variational inference is a fast alternative to MCMC methods (Ormerod and Wand, 2010; Blei et al., 2017). In this article, the intractability of the full posterior density function is handled by using the following product density approximation, also called mean-field approximation:

$$\begin{array}{l}p\left(\mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\mathit{\text{σ}}}_{ϵ}^2,{\mathit{\text{a}}}_{ϵ},{\mathit{\text{σ}}}_{\mu }^2,{\mathit{\text{a}}}_{\mu },{\mathit{\text{σ}}}_{{\psi }_1}^2,\dots ,{\mathit{\text{σ}}}_{{\psi }_L}^2,{\mathit{\text{a}}}_{{\psi }_1},\dots ,{\mathit{\text{a}}}_{{\psi }_L}\mid \mathit{\text{x}}\right)\approx \\ q(\mathit{\text{v}}){\displaystyle \prod _{i=1}^{n}q\left({\zeta }_i\right)}{\displaystyle \prod _{j=1}^p}\left[q\left({\sigma }_{ϵ}^{(j)2}\right)q\left(a_{ϵ}^{(j)}\right)q\left({\sigma }_{\mu }^{(j)2}\right)q\left(a_{\mu }^{(j)}\right){\displaystyle \prod _{l=1}^L}\left\{q\left({\sigma }_{{\psi }_l}^{(j)2}\right)q\left(a_{{\psi }_l}^{(j)}\right)\right\}\right],\end{array}$$ (9)

where each q represents an approximate density function that is specified by its argument. In variational inference, the q-density functions are chosen to minimise the Kullback–Leibler divergence of the left-hand side of (9) from its right-hand side (“reverse” Kullback–Leibler divergence). Classical variational algorithms rely on the observation that minimising the reverse Kullback–Leibler divergence amounts to maximising a lower bound on the marginal log-likelihood, called the ELBO, for evidence lower bound (Blei et al., 2017). Because the expression of the ELBO does not involve the marginal likelihood, it can conveniently be used as objective function. In (9), we have assumed posterior independence of the mean and latent functions, the global parameters, from the scores, the subject-specific parameters. The posterior independence of the variance parameters and their associated hyperparameters is a consequence of incorporating asymptotic independence between regression coefficients and variance parameters (Menictas and Wand, 2013) and induced factorisations based on graph theoretic results (Bishop, 2006, Section 10.2.5). Proposition 4.1 permits further factorisations for the complete vector of spline coefficients v. Its proof is provided in Appendix A of the Supplementary material.

Proposition 4.1

The approximate q-density function for the full vector of spline coefficients v in (9) factorises according to $q(\mathit{\text{v}})=\prod _{j=1}^{p}q\left({\mathit{\text{ν}}}^{(j)}\right)$.

The parameters for each of the q-density functions in (9) are interrelated, but can be determined via a coordinate ascent algorithm (Ormerod and Wand, 2010, Algorithm 1). This corresponds to the classical mean-field variational Bayes (MFVB) approach. Classical MFVB requires the derivation of all approximate posterior density functions, and does not take advantage of the fragment-based variational message passing (VMP) set-up of the Bayesian model for univariate FPCA in Nolan et al. (2023), where the authors presented convenient model extensions based on a factor graph approach.

4.2. Variational message passing

We next provide a brief overview of VMP tailored towards mFPCA, before detailing our new multivariate functional principal component Gaussian likelihood fragment. For a deeper exposition to VMP, we refer the reader to Minka (2005) and Wand (2017).

VMP for arbitrary Bayesian models relies on identifying fragments within the model. Each fragment consists of one probabilistic specification and all model parameters within that specification. For instance, the fragment for the likelihood specification in model (8) is presented in blue in Fig. 1. The central blue square node, the factor, represents the likelihood specification, while the circular nodes represent the parameters {v^(j)} _j=1,…,p, {ζ_i}_i=1,…,n and ${\left\{{\sigma }_{ϵ}^{(j)2}\right\}}_{j=1,\dots ,p}$ that are arguments for the likelihood. Notice that the parameters are separated according to the product density restriction in (9) and that Proposition 4.1 permits further parameter decomposition of v.

Fig. 1.

The factor graph representation of the Bayesian model for mFPCA (8). The multivariate functional principal component Gaussian likelihood fragment is represented in blue. It consists of the likelihood specification as the blue square node and all parameters contributing to that specification as blue circles connected to the square node. (For interpretation of the colours in the figure(s), the reader is referred to the web version of this article.)

Throughout the VMP iterations, approximate posterior densities are updated according to messages passed between factors and stochastic nodes. The messages have the general form m_f→θ(θ), where f represents an arbitrary factor and θ represents an arbitrary parameter vector. The arrow in the subscript represents the direction of the message, while the message itself is a function of the stochastic node that participates in the update. In order to infer the multivariate latent functions and scores, we have to determine the q-density functions for v⁽¹⁾, …, v^(j) and ζ ₁, …, ζ _n. These can be expressed as

$$\begin{array}{l}q\left({\mathit{\text{v}}}^{(j)}\right)\propto m_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\mathit{\text{v}}}^{(j)}}\left({\mathit{\text{v}}}^{(j)}\right)m_{p\left({\mathit{\text{v}}}^{(j)}\mid {\sigma }_{\mu }^{(j)2},{\sigma }_{{\psi }_1}^{(j)2},\dots ,{\sigma }_{{\psi }_L}^{(j)2}\right)\to {\mathit{\text{v}}}^{(j)}}\left({\mathit{\text{v}}}^{(j)}\right),j=1,\dots ,p,\\ q\left({\mathit{\text{ζ}}}_i\right)\propto m_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\mathit{\text{ζ}}}_i}\left({\mathit{\text{ζ}}}_i\right)m_{p\left({\mathit{\text{ζ}}}_i\right)\to {\mathit{\text{ζ}}}_i}\left({\zeta }_i\right),i=1,\dots ,n.\end{array}$$ (10)

A key step in developing the message passing framework is to express density functions in exponential family form: p(x) ∝ exp{T (x)^⊺η}, where T (x) is a vector of sufficient statistics that identify the distributional family, and η is the natural parameter vector; the messages in (10) are typically in the exponential family of density functions. Under these circumstances, the associated natural parameter vector updates for the approximate posterior densities in (10) take the form:

$$\begin{array}{l}{\mathit{\text{η}}}_{q\left({\mathit{\text{v}}}^{(j)}\right)}={\mathit{\text{η}}}_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\mathit{\text{v}}}^{(j)}}+{\mathit{\text{η}}}_{p\left({\mathit{\text{v}}}^{(j)}\mid {\sigma }_{\mu }^{(j)2},{\sigma }_{{\psi }_1}^{(j)2},\dots ,{\sigma }_{{\psi }_L}^{(j)2}\right)\to {\mathit{\text{v}}}^{(j)}},j=1,\dots ,p,\\ {\mathit{\text{η}}}_{q\left({\mathit{\text{ζ}}}_i\right)}={\eta }_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\mathit{\text{ζ}}}_i}+{\mathit{\text{η}}}_{p\left({\mathit{\text{ζ}}}_i\right)\to {\mathit{\text{ζ}}}_i},i=1,\dots ,n\end{array}$$ (11)

We outline the exponential family forms of the normal and inverse chi-squared density functions in Appendix B of the Supplementary material.

The fragment-based representation of the Bayesian model is the key ingredient in taking advantage of previous algebraic derivations and computer coding. Although the likelihood fragment (blue in Fig. 1) is specific for Bayesian mFPCA and requires derivation, all other fragments in model (8) have been identified and derived in previous publications; this is a major advantage in obtaining VMP updates for the mFPCA model, compared to MFVB updates (Ormerod and Wand, 2010, Algorithm 1). The Gaussian prior specifications for the each p(ζ _i), i = 1, …, n, are examples of Gaussian prior fragments (Wand, 2017, Section 4.1.1); the inverse chi-squared prior specifications on all the subscripted a^(j) parameters, j = 1, …, p, are univariate inverse G-Wishart prior fragments (Maestrini and Wand, 2021, Algorithm 1); similarly, the subscripted variance parameter specifications of the form σ^(j)2 ∣ a^(j) ~ Inverse–X²(1, 1/a^(j)), j = 1, …, p, are instances of the univariate iterated inverse G-Wishart fragment (Maestrini and Wand, 2021, Algorithm 2). Finally, each of the p fragments representing the penalised specifications on a given v^(j) is an example of the multiple Gaussian penalisation fragment derived in Algorithm 2 of Nolan et al. (2023) (red in Fig. 1) and which have proven to be central computational units in VMP for Bayesian FPCA.

We name the new fragment for the likelihood specification in (8) the multivariate functional principal component Gaussian likelihood fragment and outline its updates in the next section.

4.3. Multivariate functional principal component Gaussian likelihood fragment

For each j = 1, …, p, the message from $p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)$ to each v^(j) can be shown to be proportional to a multivariate normal density function, with natural parameter vector

$${\mathit{\text{η}}}_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\mathit{\text{v}}}^{(j)}}\leftarrow \left[\begin{array}{c}{\mathbb{E}}_q\left(1/{\sigma }_{ϵ}^{(j)2}\right)\sum _{i=1}^n{\left\{{\mathbb{E}}_q{\left({\widetilde{\mathit{\text{ζ}}}}_i\right)}^{\top }\otimes {\mathit{\text{C}}}_i^{(j)}\right\}}^{\top }{\mathit{\text{x}}}_i^{(j)}\\ -\frac{1}{2}{\mathbb{E}}_q\left(1/{\sigma }_{ϵ}^{(j)2}\right)\sum _{i=1}^n\text{vec}\left\{{\mathbb{E}}_q\left({\widetilde{\mathit{\text{ζ}}}}_i{\widetilde{\mathit{\text{ζ}}}}_i^{\top }\right)\otimes \left({\mathit{\text{C}}}_i^{(j)\top }{\mathit{\text{C}}}_i^{(j)}\right)\right\}\end{array}\right],$$ (12)

where ⊗ is the Kronecker product, ${\widetilde{\mathit{\text{ζ}}}}_i\equiv {\left(1,{\mathit{\text{ζ}}}_i^{\top }\right)}^{\top }$, i = 1, …, n, and, for a d₁ × d₂ matrix A, vec(A) concatenates the columns of A from left to right.

For each i = 1, …, n, the message from $p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\zeta }_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)$ to ζ_i is proportional to a multivariate normal density function, with natural parameter vector

$${\mathit{\text{η}}}_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\mathit{\text{ζ}}}_i}\leftarrow \left[\begin{array}{c}{{\displaystyle \sum }}_{j=1}^p{\mathbb{E}}_q\left(1/{\sigma }_{ϵ}^{(j)2}\right)\left\{{\mathbb{E}}_q{\left({\mathit{\text{v}}}_{\psi }^{(j)}\right)}^{\top }{\mathit{\text{C}}}_i^{(j)\top }{\mathit{\text{x}}}_i^{(j)}-{\mathbb{E}}_q\left({\mathit{\text{h}}}_{\mu \psi ,i}^{(j)}\right)\right\}\\ -\frac{1}{2}{{\displaystyle \sum }}_{j=1}^p{\mathbb{E}}_q\left(1/{\sigma }_{ϵ}^{(j)2}\right){\mathit{\text{D}}}_L^{\top }\text{vec}\left\{{\mathbb{E}}_q\left({\mathit{\text{h}}}_{\psi ,i}^{(j)}\right)\right\}\end{array}\right],$$ (13)

where ${\mathit{\text{v}}}_{\psi }^{(j)}\equiv \left[\begin{array}{c}\begin{array}{ccc}{\mathit{\text{v}}}_{{\psi }_1}^{(j)}& \dots & {\mathit{\text{v}}}_{{\psi }_L}^{(j)}\end{array}\end{array}\right],{\mathit{\text{h}}}_{\mu \psi ,i}^{(j)}\equiv {\mathit{\text{v}}}_{\psi }^{(j)\top }{\mathit{\text{C}}}_i^{(j)\top }{\mathit{\text{C}}}_i^{(j)}{\mathit{\text{v}}}_{\mu }^{(j)}\text{and}{\mathit{\text{H}}}_{\psi ,i}^{(j)}\equiv {\mathit{\text{V}}}_{\psi }^{(j)\top }{\mathit{\text{C}}}_i^{(j)\top }{\mathit{\text{C}}}_i^{(j)}{\mathit{\text{V}}}_{\psi }^{(j)}$.

For each j = 1, …, p, the message from $p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)$ to ${\sigma }_{ϵ}^{(j)2}$ is an inverse-X² density function, with natural parameter vector

$${\mathit{\text{η}}}_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\sigma }_{ϵ}^{(j)2}}\leftarrow \left[\begin{array}{c}-\frac{1}{2}\sum _{i=1}^n{n}_i^{(j)}\\ -\frac{1}{2}\sum _{i=1}^n{\mathbb{E}}_q\left\{{\left({\mathit{\text{x}}}_i^{(j)}-{\mathit{\text{C}}}_i^{(j)}{\mathit{\text{V}}}^{(j)}{\widetilde{\mathit{\text{ζ}}}}_i\right)}^{\top }\left({\mathit{\text{x}}}_i^{(j)}-{\mathit{\text{C}}}_i^{(j)}{\mathit{\text{V}}}^{(j)}{\widetilde{\mathit{\text{ζ}}}}_i\right)\right\}\end{array}\right],$$ (14)

where ${\mathit{\text{V}}}^{(j)}\equiv \left[\begin{array}{c}\begin{array}{cccc}{\mathit{\text{v}}}_{\mu }^{(j)}& {\mathit{\text{v}}}_{{\psi }_1}^{(j)}& \dots & {\mathit{\text{v}}}_{{\psi }_L}^{(j)}\end{array}\end{array}\right]$.

Pseudocode for the multivariate functional principal component Gaussian likelihood fragment is presented in Algorithm 1. A derivation of all the relevant expectations and natural parameter vector updates is provided in Appendix C of the Supplementary material.

Algorithm 1. Pseudocode for the multivariate functional principal component Gaussian likelihood fragment.

Inputs: $\left\{{\mathit{\text{η}}}_{q\left({\mathit{\text{v}}}^{(j)}\right)}:j=1,\dots ,p\right\},\left\{{\mathit{\text{η}}}_{q\left({\zeta }_i\right)}:i=1,\dots ,n\right\},\left\{{\mathit{\text{η}}}_{q\left({\sigma }_{\epsilon }^{(j)2}\right)}:j=1,\dots ,p\right\}$

Updates:

  1:   Update posterior expectations.                                    ⊳ see Appendix C

  2:   for j=1, …, p do

  3:       Update ${\mathit{\text{η}}}_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\mathit{\text{v}}}^{(j)}}$                        ⊳ Equation (12)

  4:       Update ${\mathit{\text{η}}}_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\sigma }_{ϵ}^{(j)2}}$                      ⊳ Equation (12)

  5:   for i =1, …, n do

  6:       Update ${\mathit{\text{η}}}_{p(\mathit{\text{y}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{0)2}\to {\mathit{\text{ζ}}}_i}$                                      ⊳ Equation (12)

Outputs: $\left\{{\mathit{\text{η}}}_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\mathit{\text{v}}}^{(j)}}:j=1,\dots ,p\right\},$

                $\left\{{\mathit{\text{η}}}_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\mathit{\text{ζ}}}_i}:i=1,\dots ,n\right\}$

                $\left\{{\mathit{\text{η}}}_{p\left(\mathit{\text{x}}\mid \mathit{\text{v}},{\mathit{\text{ζ}}}_1,\dots ,{\mathit{\text{ζ}}}_n,{\sigma }_{ϵ}^{(1)2},\dots ,{\sigma }_{ϵ}^{(p)2}\right)\to {\sigma }_{ϵ}^{(j)2}}:j=1,\dots ,p\right\}$

4.4. Post-inference orthogonalisation

Although the hierarchical model (8) induces regularisation through component- and variable-specific prior variance formulations on the spline coefficients, the multivariate FPCA restrictions of orthonormal multivariate eigenfunctions and uncorrelated scores with non-increasing variances are not enforced. As a consequence, the variational inference-based decomposition must be altered to satisfy these conditions. Theorem 3.1 guarantees that we can recover the orthogonal decomposition through an appropriate sequence of orthogonalisations.

In order to achieve an orthogonal mFPCA decomposition, we will generalise the post-processing steps of Nolan et al. (2023, Section 5). The key posterior densities that we require for orthogonal Bayesian mFPCA are q(v^(j)), j = 1, …, p, and q(ζ _i), i = 1, …, n, all of which are normal density functions. Firstly, the natural parameter vectors for each q(v^(j)) and q(ζ _i) can be determined via (11). Next, the common parameters 𝔼_q(v^(j)) and ℂov_q(v^(j)) for q(v^(j)), and the common parameters 𝔼_q(ζ _i) and ℂov_q(ζ _i) for q(ζ _i) can be computed from (C7), respectively (C10) of Appendix C of the Supplementary material. In the remainder, we partition 𝔼_q(v^(j)) as ${\mathbb{E}}_q\left({\mathit{\text{v}}}^{(j)}\right)\equiv {\{{\mathbb{E}}_q\left({\mathit{\text{v}}}_{\mu }^{(j)\text{⊤}}\right),{\mathbb{E}}_q\left({\mathit{\text{v}}}_{{\psi }_1}^{(j)\top }\right),\dots ,{\mathbb{E}}_q\left({\mathit{\text{v}}}_{{\psi }_L}^{(j)\top }\right)\}}^{\top }.$

First construct the n_g -length vector ${\mathit{\text{t}}}_g\equiv {\left(t_1,\dots ,t_{n_g}\right)}^{\top }$ of equidistant time points, where t₁ =0 and $t_{n_g}=1$, and establish the spline design matrix ${\mathit{\text{C}}}_g\equiv \left[\begin{array}{c}\begin{array}{ccccc}{\mathbf{1}}_{n_g}& {\mathit{\text{t}}}_g& z_1\left({\mathit{\text{t}}}_g\right)& \dots & z_K\left({\mathit{\text{t}}}_g\right)\end{array}\end{array}\right]$, where ${\mathbf{1}}_{n_g}$ is an n_g -length vector of ones. Then, the posterior estimate for each of the mean functions is $\widehat{\mu }\left({\mathit{\text{t}}}_g\right)\equiv {\mathit{\text{C}}}_g{\mathbb{E}}_q\left({\mathit{\text{ν}}}_{\mu }^{(j)}\right)$, for j = 1, …, p. Likewise, the variational Bayesian estimates of the latent functions are ${\mathbb{E}}_q\left\{{\psi }_l^{(j)}\left({\mathit{\text{t}}}_g\right)\right\}={\mathit{\text{C}}}_g{\mathbb{E}}_q\left({\mathit{\text{ν}}}_{{\psi }_l}^{(j)}\right),$ for l = 1, …, L and j=1, …, p.

Next define the vectors ${\mathit{\text{ψ}}}_l\equiv {({\mathbb{E}}_q{\{{\psi }_l^{(1)}({\mathit{\text{t}}}_g)\}}^{\top },\dots ,{\mathbb{E}}_q\{{\psi }_l^{(p)}({\mathit{\text{t}}}_g)\})}^{\top }$, for l = 1, …, L. Then bind these vectors column wise to obtain the matrix Ψ ≡ [ψ ₁ ⋯ ψ_L] and establish the singular value decomposition $\mathbf{\text{Ψ}}={\mathit{\text{U}}}_{\psi }{\mathit{\text{D}}}_{\psi }{\mathit{\text{v}}}_{\psi }^{\top }$, with U _ψ and V _ψ unitary matrices and D_ψ a diagonal matrix whose elements are the singular values of Ψ.

Now, establish the matrix $\mathit{\text{Ξ}}\equiv {\left[{\mathbb{E}}_q\left({\mathit{\text{ζ}}}_1\right)\cdots {\mathbb{E}}_q\left({\mathit{\text{ζ}}}_n\right)\right]}^{\top }$. Then define C_ζ to be the L × L covariance matrix of the columns of ΞV _ψ D_ψ and establish its spectral decomposition C_ζ = QΛQ^⊺, where Λ is a diagonal matrix whose elements are the eigenvalues of C_ζ in descending order, and Q is an orthogonal matrix containing the corresponding eigenvectors.

Finally, define the matrices $\dot{\mathbf{\text{Ψ}}}\equiv {\mathit{\text{U}}}_{\psi }\mathit{\text{Q}}{\mathbf{\text{Λ}}}^{1/2}$ and $\dot{\mathbf{\text{Ξ}}}\equiv \mathbf{\text{Ξ}}\mathit{\text{R}}$, where R ≡ V _ψ D_ψ QΛ^−1/2. It can be seen that the columns of $\dot{\mathbf{\text{Ψ}}}$ are orthogonal and the columns of $\dot{\mathbf{\text{Ξ}}}$ are uncorrelated. Next, set the lth column of $\dot{\mathbf{\text{Ψ}}}$ as ${\dot{\psi }}_l\left({\mathit{\text{t}}}_g\right)$ and the ith row of $\dot{\mathbf{\text{Ξ}}}$ as ${\stackrel{\cdot }{\mathit{\text{ζ}}}}_i$. In addition, partition ${\dot{\psi }}_l\left({\mathit{\text{t}}}_g\right)$ according to ${\dot{\psi }}_l\left({\mathit{\text{t}}}_g\right)\equiv {\left\{{\dot{\psi }}_l^{(1)}{\left({\mathit{\text{t}}}_g\right)}^{\text{⊤}},\dots ,{\dot{\psi }}_l^{(p)}{\left({\mathit{\text{t}}}_g\right)}^{\text{⊤}}\right\}}^{\text{⊤}}$, where each ${\dot{\psi }}_l^{(j)}\left({\mathit{\text{t}}}_g\right)$ is an n_g -length vector.

While the columns of $\dot{\mathbf{\text{Ψ}}}$ are orthogonal vectors, we require orthonormal multivariate functions in ℋ. We can approximate $\Vert {\dot{\psi }}_l{\Vert }_{\mathscr{H}}$, the ℋ-norm of ${\dot{\psi }}_l$, using numerical integration on (1). This permits a posterior estimate on the lth multivariate eigenfunction over the vector t_g as

$${\widehat{\psi }}_l\left({\mathit{\text{t}}}_g\right)=\frac{{\dot{\psi }}_l\left({\mathit{\text{t}}}_g\right)}{\Vert {\dot{\psi }}_l{\Vert }_{\mathscr{H}}},l=1,\dots ,L.$$

This unit-norm constraint preserves identifiability with respect to scaling (up to a change of sign).

We may partition this vector as ${\widehat{\psi }}_l\left({\mathit{\text{t}}}_g\right)={\left\{{\widehat{\psi }}_l^{(1)}{\left({\mathit{\text{t}}}_g\right)}^{\text{⊤}},\dots ,{\widehat{\psi }}_l^{(p)}{\left({\mathit{\text{t}}}_g\right)}^{\text{⊤}}\right\}}^{\text{⊤}}$, where each ${\widehat{\psi }}_l^{(j)}\left({\mathit{\text{t}}}_g\right)$ is an n_g -length vector that is the posterior estimate of the jth element of the lth multivariate eigenfunction. In addition, the posterior estimate for each of the scores is given by ${\widehat{\zeta }}_{il}=\Vert {\dot{\psi }}_l{\Vert }_{\mathscr{H}}{\dot{\zeta }}_{il}$, and it now accounts for the contribution to the variance in the functional data, in descending order across l = 1, …, L, thanks to the ordered diagonal elements of Λ from the spectral decomposition of C_ζ.

The truncated Karhunen–Loève expansion is left unchanged by this transformation, since

$$\dot{\mathbf{\text{Ψ}}}{\dot{\mathbf{\text{Ξ}}}}^T={\mathit{\text{U}}}_{\psi }\mathit{\text{Q}}{\mathbf{\text{Λ}}}^{1/2}{\mathbf{\text{Λ}}}^{-1/2}{\mathit{\text{Q}}}^T{\mathit{\text{D}}}_{\psi }{\mathit{\text{V}}}_{\psi }^T\mathbf{\text{Ξ}}T=\mathbf{\text{Ψ}}{\mathbf{\text{Ξ}}}^T.$$

In particular, setting ${\widehat{x}}_i\left({\mathit{\text{t}}}_g\right)\equiv {\left\{{\widehat{x}}_i^{(1)}{\left({\mathit{\text{t}}}_g\right)}^{\text{T}},\dots ,{\widehat{x}}_i^{(p)}{\left({\mathit{\text{t}}}_g\right)}^{\text{⊤}}\right\}}^{\text{⊤}}$, the posterior trajectories from the variational algorithm satisfy

$$\begin{array}{l}{\widehat{x}}_i\left({\mathit{\text{t}}}_g\right)=\widehat{\mu }\left({\mathit{\text{t}}}_g\right)+\mathbf{\text{Ψ}}{\mathbb{E}}_q\left({\mathit{\text{ζ}}}_i\right)=\widehat{\mu }\left({\mathit{\text{t}}}_g\right)+{\mathit{\text{U}}}_{\psi }{\mathit{\text{D}}}_{\psi }{\mathit{\text{V}}}_{\psi }^{\top }{\mathbb{E}}_q\left({\mathit{\text{ζ}}}_i\right)\\ =\widehat{\mu }\left(t_g\right)+{\mathit{\text{U}}}_{\psi }\mathit{\text{Q}}{\mathbf{\text{Λ}}}^{1/2}{\mathbf{\text{Λ}}}^{-1/2}{\mathit{\text{Q}}}^{\top }{\mathit{\text{D}}}_{\psi }{\mathit{\text{V}}}_{\psi }^{\top }{\mathbb{E}}_q\left({\mathit{\text{ζ}}}_i\right)\\ =\widehat{\mu }\left({\mathit{\text{t}}}_g\right)+\dot{\mathbf{\text{Ψ}}}{\dot{\mathit{\text{ζ}}}}_i.\end{array}$$

Setting $\dot{\mathit{\text{D}}}\equiv \text{diag}\left(\Vert {\dot{\psi }}_1{\Vert }_{\mathscr{H}},\dots ,\Vert {\dot{\psi }}_L{\Vert }_{\mathscr{H}}\right),$ we obtain ${\widehat{x}}_i\left({\mathit{\text{t}}}_g\right)=\widehat{\mu }\left({\mathit{\text{t}}}_g\right)+\mathbf{\text{Ψ}}{\dot{\mathit{\text{D}}}}^{-1}\dot{\mathit{\text{D}}}{\dot{\mathit{\text{ζ}}}}_i$, which is equivalent to

$${\widehat{x}}_i\left({\mathit{\text{t}}}_g\right)=\widehat{\mu }\left({\mathit{\text{t}}}_g\right)+\sum _{l=1}^L{\widehat{\zeta }}_{il}{\widehat{\psi }}_l\left({\mathit{\text{t}}}_g\right),i=1,\dots ,n.$$

Uncertainty quantification for the transformed estimates is achieved as follows. The covariance for the new scores ${\widehat{\mathit{\text{ζ}}}}_i$ is obtained by applying the above rotation and normalisation transformations to the original variational covariance parameter, ℂov_q(ζ _i), for q(ζ _i), namely,

$${\mathbf{\text{Σ}}}_{{\widehat{\mathit{\text{ζ}}}}_i}=\dot{\mathit{\text{D}}}{R}^{\top }ℂ{\text{ov}}_q\left({\mathit{\text{ζ}}}_i\right)R\dot{\mathit{\text{D}}}.$$

This covariance is then used to construct credible contours for the scores, as well as pointwise posterior credible bands for the response curves. For the latter, we consider the estimated mean function and eigenfunctions as fixed curves derived from the posterior mean of the spline coefficients, 𝔼_q(v^(j)). As a result, the pointwise posterior variance in the estimated response curves is solely attributed to the variance in the principal component scores. This approach aligns with standard FPCA methodologies, where randomness is introduced through the scores (see, e.g., Yao et al., 2005; Benko et al., 2009). For the jth response, define ${\widehat{\mathbf{\text{Ψ}}}}_j$ as the n_g × L matrix formed of the columns ${\widehat{\psi }}_l^{(j)}\left({\mathit{\text{t}}}_g\right)$, l = 1, …, L. The covariance corresponding to the jth FPCA expansion for subject i is then

$${\mathbf{\text{Σ}}}_{{\widehat{x}}_i^{(j)}\left({\mathit{\text{t}}}_g\right)}={\widehat{\mathbf{\text{Ψ}}}}_j{\mathbf{\text{Σ}}}_{{\widehat{\mathit{\text{ζ}}}}_i}{\widehat{\mathbf{\text{Ψ}}}}_j^{\top }.$$ (15)

The variances used to construct pointwise credible and prediction bands for response j and subject i are extracted from the diagonal elements of (15).

4.5. Adaptive selection of the number of spline functions and latent components

Model (8) involves specifying (i) the dimension K of the spline basis used to describe the mean and latent functions, and (ii) the number L of latent components necessary to describe the variation in the functional data. In this section, we outline two possible approaches to select each of these quantities adaptively.

The choice of K can be guided by empirical considerations aimed at balancing enough flexibility to fit the functional data and avoiding overfitting, especially when the number of observations is small. With this in mind, we propose adapting the rule of thumb proposed by Ruppert (2002) to our multivariate irregular-grid setting. Specifically, we set variable-specific numbers of splines as

$${\mathit{\text{K}}}_j=\max \left\{\min (\lfloor n_{obs,j}/4\rfloor ,40),7\right\},j=1,\dots ,p,$$

where n_obs,j = median{n^(j); i = 1, …, n}. The lower bound ensures that the spline basis is sufficiently complex to capture essential data features, while the upper bound is based on noting that increasing the number of spline coefficients yields diminishing returns in terms of model accuracy and fit, but adds to the computational burden.

A second approach is to learn K from the data, using a model-choice approach, namely, treating K as a model parameter and estimating its posterior distribution through approximations of the marginal likelihood for each model conditional on K. Here, we propose using the ELBO as a proxy of the marginal log-likelihood (see, e.g., Blei et al., 2017) and placing a discrete uniform prior on the number of spline coefficients:

$$p(\mathit{\text{K}}\mid \mathit{\text{x}})\propto \exp \{\log \mathit{\text{p}}(\mathit{\text{x}}\mid K)\}p(\mathit{\text{K}})\approx \exp \left\{{\text{ELBO}}_K\right\}p(\mathit{\text{K}}),$$

where p(K) = Unif(K_min, …, K_max), and K_min and K_max are user-specified hyperparameters. In our simulations of Section 5, we use K_min = 5 and K_max = 20, which allows running the procedure in parallel on a 16-core machine; the posterior mass is well-within this range, however the support of the prior on K can be extended as needed. The use of the ELBO provides a tractable means of model selection and posterior approximation, bypassing direct computation of the marginal likelihood. A similar approach has been employed by Suarez and Ghosal (2017) for choosing the number of components L in the context of univariate FPCA, but they approximate the marginal likelihood by running MCMC chains for each possible model.

The rule-of-thumb approach has several advantages: it is simple, based on well-established considerations, requires no extra computational resources and is tailored to the multivariate and irregular-grid settings, making it versatile for complex data structures. The model-based approach is more principled and learns the K from the data. Both approaches are implemented in R package bayesFPCA, and the simulations presented in Section 5.3 compare the two approaches. Note that we use O’Sullivan’s penalised splines, which prevent overfitting and make inference relatively insensitive to reasonable choices of K.

A similar model-choice approach can be taken for setting the number of components L, whereby L is assigned a prior distribution; our R package implementation permits placing a truncated Poisson on L with support {1, …, L_max}, for some user-specified upper bound L_max. An alternative and elegant approach to choosing L is via prior distributions (placed on scores or spline coefficients) that encode ordering constraints by increasing the degree of shrinkage with the component index l. Multiplicative gamma process shrinkage priors (Bhattacharya and Dunson, 2011; Montagna et al., 2012), cumulative shrinkage process priors (Legramanti et al., 2020; Kowal and Canale, 2023) and variants thereof (Kowal et al., 2017; Shamshoian et al., 2022) have been proposed based on this idea. These priors offer an effective means to induce structured, ordered shrinkage via the prior specification, a topic that remains at the forefront of current research with recent discussions and contributions, e.g., by Schiavon et al. (2022), Frühwirth-Schnatter (2023) and Frühwirth-Schnatter et al. (2024). In our case, thanks to the component-specific variances on the spline coefficients in (8), the variances of irrelevant latent functions are effectively shrunk, therefore enforcing the required regularisation. However, instead of encouraging ordering of the components through the prior, we exploit the orthonormalisation procedure described in Section 4.4 and learn this ordering from the estimation of the proportion of variance explained by each FPC component. Specifically, exploiting the fact that 𝕍 ar(ζ_il) = λ_l, with λ₁ ≥ ⋯ ≥ λ_L in the multivariate Karhunen–Loève expansion, we form estimates ${\widehat{\lambda }}_l$ by taking the empirical variance of the scores post-orthonormalisation and obtain the estimated proportion of variance explained (PVE) by the lth component as ${\widehat{\lambda }}_l/{{\displaystyle \sum }}_{l^{\prime }=1}^{L_{\max }}{\widehat{\lambda }}_{l^{\prime }}$, for L_max sufficiently large. For a computationally faster alternative to placing a prior on L, we also implement a PVE-based selection approach for L. Specifically, in our numerical experiments, we programmatically estimate of the number of components by performing a single run the algorithm with an upper bound L_max = 10 (also likely to be an overestimate in real settings), and retain the leading components until their cumulated PVE reaches 95%. This approach, or alike, is routinely employed in FPCA work such as Ramsay and Silverman (2005); Greven et al. (2011); Happ and Greven (2018); Li et al. (2020) and Li and Xiao (2023), where 95% is a common default threshold. Our simulations of Section 5.3 compare this simple approach and the model-choice approach for L, and assess their performance in recovering the correct number of simulated components.

5. Simulations

5.1. Data generation, hyperparameter settings and performance metrics

We illustrate our approach in a series of simulation studies aimed at assessing its statistical and computational performance. We place particular emphasis on evaluating the benefits of pulling information from sparse and irregularly observed curves. Unless stated otherwise, for each numerical experiment, we generate synthetic multivariate response curves from expansion (4), also adding a centred error term with variance unity. We use different numbers of observations $n_i^{(j)}$, for i = 1, …,n, j = 1, …, p, with time sampled uniformly over the interval [0, 1] (subject- and variable-specific grids). We evaluate performance for two different types of simulated mean function and eigenfunctions (orthonormal in ℋ≡ L²([0, 1])^p), namely, periodic functions:

$$\begin{array}{l}{\mu }^{(j)}(t)={(-1)}^{j}2\sin \{(2\pi +j)t\},j=1,\dots ,p,\hfill \\ {\psi }_{2l^{\prime }-1}^{(j)}(t)={(-1)}^j\sqrt{2/p}\cos (2l^{\prime }\pi t),{\psi }_{2l^{\prime }}^{(j)}(t)={(-1)}^j\sqrt{2/p}\sin (2l^{\prime }\pi t),l^{\prime }=1,\dots ,L/2,\hfill \end{array}$$

for an even number of eigenfunctions L, or B-splines orthonormalised using the Gram–Schmidt method. Finally, we simulate the scores ζ_il independently from a centred Gaussian distribution with standard deviation l^−1/α, l = 1, … L, for α ∈ {1, 2, 8} to cover different relative contributions to the variation in the data across components. We also vary the total number of components L ∈ {1, …, 8}.

We run our approach from its R package implementation (bayesFPCA); the package also includes functions to simulate data based on the above data generation procedure. We set the model hyperparameters to ensure uninformative specifications for the standard deviations, namely, σ_β = A = 10⁵. In all experiments, we perform inference agnostically of the number of simulated components L, using the PVE-based procedure with L_max = 10, and learn K from the data using the model-choice or rule-of-thumb procedure, as described in Section 4.5. Moreover, Section 5.3 is dedicated to assessing both the sensitivity of inference to these choices of K and L, and the performance of our procedure for estimating L. Finally, we use a convergence tolerance of τ = 10⁻⁵ on the relative changes in the ELBO (variational objective function) as stopping criterion.

We evaluate estimation accuracy by calculating the root mean square error (RMSE) for the scores and the integrated squared error, $\text{ISE}(f,\widehat{f})={{\displaystyle \int }}_0^1|f(x)-\widehat{f}(x){|}^2\text{d}x$, for the mean function and eigenfunctions, where f (⋅) is the function used for data generation and $\widehat{f}(\cdot )$ is its corresponding posterior estimate. The latent function estimates are based on the variational posterior means of the spline coefficients.

5.2. Accuracy of variational inference

We start by evaluating the accuracy of the MFVB and VMP algorithms through direct comparisons with MCMC for the same model. While variational procedures use a prescribed tolerance, MCMC sampling requires evaluating the chain’s ability to explore the model space, which can be difficult for large problems. Since the two types of algorithms have different stopping rules and convergence diagnostics, comparing their accuracy and runtime can be challenging. To alleviate the risk of unfair comparisons, we conducted MCMC inference with the popular probabilistic programming language Stan (Carpenter et al., 2017), using the default no-U-turn sampler (NUTS) with 2 000 iterations of which 1 000 were discarded as burn-in. We acknowledge that custom MCMC implementations for our model could be more efficient than Stan’s general-purpose engine; however our focus is on providing a practical comparison using an off-the-shelf tool, while sidestepping the need to develop a tailored MCMC algorithm.

Our approach to obtaining identifiable and orthonormalised estimates from the MCMC posterior summaries mirrors the technique employed for variational estimates. Rather than enforcing orthonormality during the sampling phase, we initially sample the parameters without constraints. Once convergence is reached, we apply the orthonomalisation steps described in Section 4.4 individually to each MCMC sample, after discarding the burn-in samples. Hence, after addressing potential sign changes in the basis functions, these transformed scores and eigenfunctions are directly comparable to the post-processed variational estimates.

We simulate problems with p = 3 variables, n = 200 subjects and a number of observations drawn uniformly from {10, …, 20} for each subject and each variable, and with L = 2 simulated latent components; this number of components is successfully retrieved by all methods, using PVE estimation with L_max = 10. Fig. 2 shows the reconstructed trajectories and scores for randomly selected subjects, as well as the estimated latent functions, indicating a very good agreement between the VMP and MCMC estimates. The same holds for comparisons between MFVB and MCMC estimates (Appendix D.2 of the Supplementary material). Variational inference is known to be prone to posterior variance underestimation, especially when poor mean-field factorisations are employed (see, e.g., Blei et al., 2017); here however, the excellent agreement of the MCMC and variational posterior intervals for the estimated trajectories and scores provides empirical evidence that this issue is not encountered under the factorisation we employ. As expected the runtime, covering the inference and post-processing steps, is largely in favour of the variational procedure, with 20 seconds for the VMP algorithm and 18 minutes 28 seconds for the MCMC algorithm on an Intel Xeon CPU, 2.60 GHz machine.

Fig. 2.

MCMC and VMP estimates for a problem with p = 3 variables observed at an average of 15 time points for n = 200 subjects; L is learnt from the data by estimating the PVE with L_max = 10, and K is set using the rule of thumb described in Section 4.5. Top left: estimated trajectories for a random subset of 3 subjects, with posterior means (solid lines) and 95% pointwise prediction bands (dashed). The lines corresponding to MCMC (red) and VMP (blue) inference overlap. Bottom left: mean and latent functions simulated (black) and estimated by MCMC inference (red) with 95% credible bands (dashed) and by VMP inference (blue) for which estimates from 100 replicates are overlaid. Right: scores simulated (black dots) and estimated by MCMC (posterior mean, red dots) and VMP (posterior mean, blue dots) inference, with 95% credible contours, for a random subset of 16 subjects.

A comprehensive comparison of the errors on the estimated scores and latent functions for a grid of problems with n ∈ {50, 100, 200, 300, 400, 500} further shows strong agreement between the VMP, MFVB and MCMC, as well as a greater accuracy as n increases (Figs. D3 & D4 of the Supplementary material). As our MFVB implementation tends to be faster than its VMP counterpart for a virtually indistinguishable statistical performance, our subsequent numerical experiments employ the MFVB algorithm; the computational advantages of variational inference over MCMC inference are discussed further in Section 5.7.

5.3. Selection of K and L

In Section 4.5, we have presented approaches for selecting (i) the dimension K of the spline basis used to represent the mean and latent functions, and (ii) the number of components L explaining the variation in the functional data. In this section, we assess these approaches empirically, by inspecting the resulting errors on the scores and latent functions, as well as the ability to successfully retrieve the number of components simulated.

We simulate a series of problems with numbers of eigenfunctions ranging from 1 to 8, corresponding to orthonormalised B-splines. In each case, we generate 100 datasets with p = 3 variables, n = 100 subjects and 20 observations per subject on average.

We start by evaluating the model-choice approach for choosing both K and L. Recall that the marginal log-likelihood p(x ∣ K, L) is approximated by the ELBO_K,L, for K = K_min, …, K_max and L = L_min, …, L_max (here K_min = 5, K_max = 20, L_min = 1, L_max = 10), and model posterior probabilities are obtained assuming a discrete uniform prior and a truncated Poisson prior with λ = 1 for K and L, respectively, with the above supports. Fig. 3 (left) shows the posterior probabilities p(K, L ∣ x) averaged across the 100 data replicates, for each problem with number of simulated components L ∈ {1, …, 8}. This figure indicates that the marginal probabilities p(L ∣ x) concentrate on models with the correct L, while the marginal probabilities p(K ∣ x) are more spread out, covering values from K = 12 to 18.

Fig. 3.

Adaptive selection of the number of latent functions, L, and spline basis functions, K, for problems with L = 1, …, 8 simulated latent functions (100 data replicates per problem). Left: Posterior probabilities p(K, L ∣ x) estimated from the model-choice approach for learning L and K, using truncated Poisson (λ = 1) and discrete uniform prior distributions with support {1, …, 10} and {5, …, 20}, respectively. The red contours indicate the number of simulated components L for each problem. Right: Estimated PVE obtained when inferring model parameters for L_max = 10 components (upper bound). The dotted horizontal lines indicate the 95%-PVE threshold and the dashed red vertical lines indicate the number of simulated components L for each problem.

We next evaluate the PVE-based approach for choosing L, coupled with the model-choice approach for K, on the same simulated datasets: we infer the model parameters with L_max = 10 components and select L as the minimum number of components whose estimated cumulated PVE exceeds 95% (see Section 4.5). Fig. 3 (right) shows that the correct L is again selected in the vast majority of runs. Inspecting the variational estimates before orthonormalisation reveals that the appropriate regularisation is enforced by the model, since the spline coefficient variances corresponding to superfluous components are effectively shrunk to small values. The 95%-threshold rule is of course arbitrary; its relevance may be questioned when the contribution to the variance by the last components is small (see, e.g., panels with L = 7 and L = 8 simulated components). In general, there is no reason why it would recover the correct number of components when the portion of the variance explained by the last components is < 5%. For this reason, in applications, we recommend choosing L by visually inspecting scree plots of the cumulated PVE: as exemplified in Fig. 3, in our numerical experiments, the estimated PVE effectively plateaus after the correct L is reached, with the subsequent components accounting for a negligible amount of the variation in the data.

We also compare the errors (score RMSE and latent functions ISE) achieved under the PVE-based and model-choice approach for learning L, as well as under the rule of thumb adapted from Ruppert (2002) and model-choice approach for learning K (Section 4.5). These results, reported in Appendix D.3 of the Supplementary material, indicate comparable errors, with no sign of overfitting. These results also suggest that the model choice and PVE-based approaches perform similarly well for recovering L. Therefore, in practice, we recommend using the PVE-based approach which doesn’t require running the algorithm for every possible model. Note that in order to obtain reliable estimates of the PVE, L_max should be an upper bound on L, which should ensure that the estimated eigenvalues corresponding to the last components are close to zero.

5.4. Comparison with frequentist multivariate FPCA

We next compare our Bayesian approach with the covariance-based frequentist multivariate approach proposed by Happ and Greven (2018). Here, we consider problems n = 100 subjects, for whom p = 3 variables are measured longitudinally, with an average number of observations per subject and variable ranging from 20 to 260; we generate 200 data replicates for each setting. For this simulation study, and all those presented in the remainder of the paper, we run our approach in parallel using model choice for setting K, and we estimate L using the PVE-based procedure (Section 4.5). Here, this procedure successfully recovers the number of simulated components for all replicates and for all settings, namely L = 2. In contrast, Happ and Greven (2018)’s implementation does not allow learning L, so we run it assuming the correct number of components as known, giving their method an advantage.

Table 1 reports the ISE on the latent functions and the RMSE on the scores. Estimation accuracy is higher with our approach for the two eigenfunctions and comparable for the mean function, with a tendency for Happ and Greven (2018)’s method to improve as the average number of observations increases. Similar observations hold for the estimation of the two sets of scores. This aligns with the fact that our model-based Bayesian approach better handles sparse observation settings, as it eliminates the need for estimating and smoothing covariances, unlike conventional decomposition frequentist procedures, such as used in Happ and Greven (2018)’s approach. Moreover, while statistical performance becomes comparable for the two methods as the number of observations increases, the covariance-estimation requirement induces computational intractability for large covariance matrices, which prevents applications of Happ and Greven (2018)’s method on problems where the average number of observations per variable and subject exceeds 80. We provide a more comprehensive discussion of this computational limitation in Section 5.7.

Table 1.

Estimation errors obtained using our approach (with selection of K and L by model choice and PVE estimation, respectively) and Happ’s approach. The integrated mean squared errors (ISE) ×100 for the latent functions and root mean square errors (RMSE) for the scores are shown for a problem with p = 3 variables, n = 100 subjects, and average number of observations per variable and subject ranging from 20 to 260 (rows). The median and interquartile range (IQR, parentheses) obtained from 200 data replicates are shown, and the smallest median error of each row is highlighted in bold. For μ(t), Ψ₁(t) and Ψ₂(t), and for each of the data replicates, the per-variable ISE are computed and averaged across the p variables. A “-” indicates scenarios where simulations for Happ’s method fail to complete within 36 hours (Intel Xeon CPU, 2.60 GHz).

Average		μ(t)			Ψ1(t)			Ψ2(t)			ζl			ζ2
$n_i^{(j)}$		mFPCA	Happ		mFPCA	Happ		mFPCA	Happ		mFPCA	Happ		mFPCA	Happ
20		0.81 (0.63)	0.88 (0.61)		0.42 (0.32)	0.80 (0.43)		1.37 (0.77)	4.11 (2.97)		0.24 (0.04)	0.28 (0.04)		0.22 (0.02)	0.26 (0.04)
40		0.74 (0.87)	0.62 (0.50)		0.27 (0.26)	0.46 (0.32)		0.73 (0.43)	1.66 (0.71)		0.19 (0.06)	0.19 (0.03)		0.17 (0.03)	0.19 (0.03)
60		0.80 (0.98)	0.53 (0.48)		0.21 (0.23)	0.33 (0.27)		0.55 (0.35)	1.10 (0.48)		0.18 (0.08)	0.16 (0.04)		0.15 (0.03)	0.16 (0.03)
80		0.75 (0.93)	0.39 (0.40)		0.17 (0.23)	0.28 (0.22)		0.43 (0.31)	0.85 (0.42)		0.16 (0.07)	0.14 (0.04)		0.13 (0.03)	0.14 (0.03)
100		0.78 (0.98)	-		0.15 (0.17)	-		0.35 (0.19)	-		0.16 (0.08)	-		0.12 (0.03)	-
120		0.84 (1.18)	-		0.14 (0.22)	-		0.32 (0.25)	-		0.16 (0.10)	-		0.12 (0.05)	-
140		0.78 (1.06)	-		0.14 (0.21)	-		0.30 (0.20)	-		0.15 (0.10)	-		0.11 (0.04)	-
160		0.58 (0.95)	-		0.14 (0.19)	-		0.28 (0.21)	-		0.13 (0.09)	-		0.11 (0.04)	-
180		0.54 (0.77)	-		0.11 (0.18)	-		0.23 (0.17)	-		0.12 (0.08)	-		0.11 (0.04)	-
200		0.51 (0.73)	-		0.13 (0.17)	-		0.23 (0.20)	-		0.12 (0.07)	-		0.11 (0.04)	-
220		0.62 (0.80)	-		0.11 (0.17)	-		0.21 (0.18)	-		0.13 (0.09)	-		0.10 (0.04)	-
240		0.54 (0.80)	-		0.089 (0.12)	-		0.18 (0.13)	-		0.12 (0.09)	-		0.10 (0.04)	-
260		0.54 (0.88)	-		0.095 (0.21)	-		0.18 (0.22)	-		0.12 (0.09)	-		0.10 (0.05)	-

For the setting where Happ and Greven (2018)’s method has the highest accuracy – namely, with 80 observations per subject and variable, on average – the estimates obtained by the two approaches are visually very close (Appendix D.4 of the Supplementary material). An advantage of our approach is that posterior credible bands are readily obtained from the inferred variational posterior distributions. Such uncertainty estimates are unavailable using Happ and Greven (2018)’s method. Pointwise bootstrap confidence bands for the eigenvalues and eigenfunctions can be obtained, but the computational cost associated with the resampling becomes prohibitive for our problem sizes.

5.5. Borrowing strength in sparse settings

We next examine the benefits of our hierarchical framework for handling estimation in complicated sparse-data settings, where borrowing information across variables and subjects is essential. We simulate a problem with n = 200 subjects and p = 6 variables, of which the first is very scarcely observed (number of observations uniformly drawn from {5, …, 10}), while the remaining five variables are observed more frequently (number of observations uniformly drawn from {50, …, 75}). We then apply our mFPCA approach jointly on the six variables, and compare the posterior estimates of the scores with those of separate univariate FPCA runs. Specifically, we conduct a series of six variational inference runs, with p = 1 in (8), and rescale their posterior mean and standard deviation by a factor $\sqrt{p}$ to make them orthonormal in the multivariate Hilbert space ℋ ≡ L²([0, 1])^p.

In the previous section we have seen that the variational algorithm advantageously provides uncertainty quantification for the estimated scores. Here we further examine the effect of pulling information with joint modelling on the credible bands and posterior means of the scores, placing special emphasis on the first variable which is infrequently observed.

Fig. 4 shows the estimated scores for the first and second principal components along with the 95% credible intervals, for a random subset of 8 subjects. The small number of observations collected for the first variable leads to very wide credible intervals for its corresponding scores (FPC 1 & 2) estimated with univariate FPCA; these intervals cover zero. This suggests that x⁽¹⁾ is too infrequently observed in order for univariate FPCA to provide any useful results. The remaining five curves (x^(j), j = 2, …, 6) are more frequently observed, and therefore the credible intervals of their corresponding univariate scores are narrower, yet in a few instances the true score is not covered (for j = 5 subject 63 FPC 1, for j = 2, 3, 4, 6 subjects 16 & 57 FPC 2, and for j = 2 subject 21 FPC 2). The mFPCA intervals all contain the true simulated scores, and they are substantially narrower than their univariate counterparts for FPC 1.

Fig. 4.

Posterior means with 95% credible intervals for the scores for the first two components, obtained from univariate and multivariate Bayesian FPCA for a problem with n = 200 subjects and p = 6 variables, of which the first, x⁽¹⁾, is infrequently observed (500 data replicates). Left: Estimates shown for a random subset of 8 subjects (first data replicate). The mFPCA estimates are in light red, the true simulated values are indicated by black triangles and the remaining colours correspond to estimates obtained from univariate FPCA runs. Right: Empirical coverage (top) and interval length (bottom). The dashed horizontal line indicates the 95% threshold.

Fig. 4 also shows the empirical coverage and length of these credible intervals based on 500 data replicates. The interval lengths (averaged across subjects) for both FPC 1 and 2 obtained by mFPCA are generally smaller than those obtained by univariate FPCA for the densely-observed variables j = 2, …, 6. For the univariate FPCA of variable j = 1, interval lengths are substantially larger for FPC 1, but are highly variable for FPC 2: for some runs, the FPC 2 scores are shrunk to zero, due to an insufficient number of observations on that variable and a lower proportion of the variance explained by the second component. The coverage obtained by mFPCA is close to 95%, suggesting that the independence assumptions of our variational factorisation are reasonable, causing no noticeable underestimation of posterior variances. It is comparable to the univariate runs for variables j = 2, …, 6: it tends to be slightly worse for FPC 1 (mean 93.5%, sd 2.3%), but better for FPC 2 (mean 94.0%, sd 2.3%). The coverage achieved with univariate FPCA for the sparsely observed variable j =1 is clearly insufficient in most runs (FPC 1: mean 78.5%, sd 19.2%; FPC 2: mean 53.3%, sd 31.2%). These results suggest that the pulling information across variables and subjects observed at irregular temporal grids is particularly beneficial in settings where some variables entail very few measurements. Another notable advantage of mFPCA is that it produces a single set of posterior estimates, under the form of scalar scores, for all variables (rather than six separate sets of scores here), thereby offering a very parsimonious summary of the temporal covariation in the multivariate data.

5.6. Performance under model misspecification

Borrowing information using joint approaches is sensible when the modelled variables are expected to covary over time. However assessing this can be difficult in practice. It is therefore important to evaluate the potential consequences on inference when this assumption is violated. In this section, we examine the robustness of our approach when the joint model is misspecified. Specifically, we generate p variables from distinct univariate models, whose scores are simulated independently (ρ = 0), or with varying degrees of correlation (ρ ∈ {0.2, 0.4, 0.6, 0.8}) up to complete correlation (ρ = 1). This last case corresponds to no misspecification, since the data share the same set of scores and therefore can be thought as generated from the joint model (8). For each scenario, we compare the accuracy of our joint approach with its univariate counterpart, applied independently to each variable as in the previous section. We consider 100 data replicates of a problem with p = 6 variables, L = 3 simulated latent components, n = 50 subjects, and with sparse observations drawn uniformly from {5, …, 10} for each curve.

As anticipated, Fig. 5 indicates a deterioration on the error of the scores using mFPCA as the correlation of the scores weakens, to the point where the univariate model outperforms the joint model, for unrelated or weakly related variables where ρ ∈ {0, 0.2, 0.4}. The estimation of the latent functions tends to be more robust to the misspecification, because, unlike the scores, the mean function and eigenfunctions are variable specific, which confers great flexibility, even under different latent dynamics for the p variables. The fact that the ISE for the first two eigenfunctions is smaller under the joint model than under the univariate model when the correlation is weak to moderate (ρ ∈ {0.4, 0.6}) can be attributed to the “virtually larger sample sizes” obtained by accounting jointly for the observations on all p variables. When the scores are more strongly correlated (ρ ∈ {0.8, 1}), there is a substantial reduction in the RMSE of the scores, which is particularly striking for the first component. This again demonstrates the benefits of borrowing strength across related variables.

Fig. 5.

Comparison of the errors obtained using univariate and multivariate Bayesian FPCA for a problem with p = 6 variables, L = 3 simulated latent components, n = 50 subjects and numbers of observations drawn uniformly from {5, …, 10} for each variable and each subject. The data were simulated from univariate FPCA models with varying degrees of score correlation ρ ∈ {0, 0.2, 0.4, 0.6, 0.8, 1} (x-axis). The median (dots) and first and third quartiles (grey) of the errors from 100 data replicates are shown. Left: RMSE for the scores. For each data replicate, the univariate FPCA per-variable RMSE are computed and the average across all p variables is shown. Right: ISE for the mean function and eigenfunctions. For each data replicate, and both univariate and multivariate FPCA, per-variable ISE are computed and the average across all p variables is shown.

Altogether, these results suggest that the multivariate model is reasonably robust to misspecifications caused by a lack of common latent dynamics across the variables. Moreover, as shown in Section 5.5, when similar dynamics exist, joint modelling also tends to produce narrower credible intervals for the scores, which contain the true value.

5.7. Computational performance

Our variational algorithm has important computational advantages compared to other estimation approaches. These advantages concern both runtime and memory (RAM) usages, and can be attributed to the combination of two features: (1) a direct, simultaneous inference of the Karhunen–Loève expansion, treating all parameters as unknown and estimating them jointly – this bypasses the need to model the covariance function, unlike with frequentist approaches which typically use a sequence of time- and memory-greedy smoothing and eigendecomposition steps to estimate large covariance functions; (2) a fast, deterministic inference approach, which scales to large numbers of subjects and time points, unlike more conventional Bayesian inference approaches based on MCMC inference.

Fig. 6 shows the runtime profiles as a function of (i) the number of subjects n, (ii) the average number of observations per subject n_i and (iii) the number of variables p, presented on the logarithmic scale. Additionally, Appendix D.5 of the Supplementary material presents the runtime as a function of the upper bound on the number of functional principal components used for inference, L_max. Profiling studies (i) and (ii) correspond to the simulations presented in Sections 5.2 and 5.4. All results for our model measure the total time needed for the inference procedure and the subsequent post-processing steps applied for orthonormalisation.

Fig. 6.

Runtime profiling (on the logarithmic scale) as a function of (i) the number of subjects n, (ii) the average number of observations per subject n_i, and (iii) the number of variables p, obtained on an Intel Xeon CPU, 2.60 GHz machine. Left: study (i); comparison of the MFVB (green), VMP (blue) and MCMC (red) algorithms on the problems of Section 5.2 (100 replicates). Middle: study (ii); comparison of Happ’s (grey) and our (blue) approaches on the problems of Section 5.4 (200 replicates). The results for Happ’s method correspond to the four highest boxplots in grey in bins 20 to 80; the method does not complete within 36 hours for problems with more than 80 observations per variable and subject, hence no boxplots are displayed. Right: study (iii); comparison of the MFVB (green) and VMP (blue) algorithms on problems with n = 100 subjects, average number of observations per subject n_i = 20 and varying numbers of variables p (100 replicates). All mFPCA methods are run with PVE-based selection of L, using L_max = 10, and, for the choice of K, either the rule of thumb (studies (i) and (iii)) or a 16-core implementation of the model-choice approach (study (ii)); see Section 4.5. The total time needed for both the inference procedure and subsequent orthogonalisation is shown.

Study (i) indicates that the variational algorithms are 1 to 3 orders of magnitude faster than a Stan’s implementation of NUTS for the same model. As already noted, while Stan is a powerful tool, it may not be the most efficient choice for the specific class of models we consider. Here, however, our intention is to provide a practical comparison using an off-the-shelf tool, rather than to develop and benchmark a highly optimised MCMC algorithm. Our VMP implementation also tends to be slower than the MFVB implementation, due to a higher per-iteration cost. This might be explained by the fact that it involves the additional overhead of passing messages between nodes in the graphical model, while the MFVB updates are simpler; we also do not rule out the possibility that our code may benefit from optimisation.

Study (ii) indicates a large computational advantage of our MFVB algorithm compared the frequentist multivariate approach of Happ and Greven (2018). In particular, the latter approach could not complete within 36 hours for problems with an average number of observations per variable and subject exceeding 80. Moreover, for Happ and Greven (2018)’s approach, memory is also an important limiting factor, as the runs with ≤ 80 observations required compute nodes with tens of GBs of RAM in order to store all matrix objects pertaining to the estimation of the covariance function.

Finally, study (iii) involves large problems, totalling up to 75 × 200 × 20 = 300 000 observations; the variational algorithms completed in less than 1.5 hour (MFVB: average 51 minutes; VMP: average 1 hour 28 minutes). The sole purpose of this profiling for large p is to evaluate the scalability of our algorithms as, in practice, the shared-score assumption implied by the multivariate Karhunen–Loève expansion is unlikely to hold for a large number of variable p; it implies that all these variables share the same underlying dynamics which can be unrealistic.

Our variational inference implementation demonstrates excellent computational efficiency, not only when compared to Stan’s generic implementation, but also against custom MCMC algorithms. Inspecting the runtimes reported in various Bayesian functional latent factor analysis articles, we found that the method of Kowal and Canale (2023) ran in ≈ 5 minutes or less on problems with a total number of observations of ≈ 2 000 − 3 000, Montagna et al. (2012) required several hours for a 40 000-observation problem, and Kowal et al. (2017) and Goldsmith et al. (2015) took up to 6 hours and 10 days, respectively, for problems with 400 000 − 450 000 observations. When matched with problems of similar numbers of observations, our algorithm is 1 to 2 orders of magnitude faster than the above methods, except for Kowal et al. (2017), where it is only twice faster. The runtimes in Kowal et al. (2017) are very good for an MCMC algorithm; it should also be noted that, in their experiments, the number of components L is fixed to low values while we learn this number using L_max = 10. Of course, these comparisons are only indicative as the runtimes reported in this literature concern implementations for models different to ours (most of which are models for a single functional variable). In summary, the efficiency of our algorithm, combined with its accuracy and uncertainty quantification properties, positions it as highly competitive for large and complex datasets.

6. The latent underpinnings of the immune response to SARS-CoV-2 infection

We return to the SARS-CoV-2 study presented in Section 2. As motivated there, since COVID-19 is a multi-system, heterogeneous disease, we will undertake to disentangle the patient variability of short- and long-term disease trajectories by applying mFPCA on molecular markers across several biological systems. Such an analysis will hopefully help us interpret the shared latent dynamics driving disease severity and incomplete organismal recovery.

Patient were categorised according to five clinical severity classes, based on their hospitalisation status and oxygen needs, namely, A, asymptomatic (n = 18); B, mild symptomatic (n = 40); C, hospitalised without supplemental oxygen (n = 50); D, hospitalised with supplemental oxygen (n = 38), and E, hospitalised with assisted ventilation (n = 69). Time was measured from the first positive swab for patients of class A, and from symptom onset for patients of the remaining classes; to prevent biased inferences due to temporal shifts resulting from these different definitions, we focus our analysis on symptomatic patients, i.e., patients from classes B to E. We analyse data collected during the first 7-weeks from symptom onset (acute and post-acute phase), examining five important molecular markers, namely, C-reactive protein (CRP) levels, interleukin 10 (IL-10) cytokine levels, glycoprotein B (glyc-B) levels and levels of two metabolites from the kynurenine pathway, quinolinic acid and tryptophan. We also exclude patients that had fewer than two measurements collected within the 7-week time window for any given marker, leaving a total of n = 82 patients for analysis; the sparse observations recorded at different time points across subjects and variables calls for flexible hierarchical modelling with effective pooling of information. Finally, we use one-time measurements from 45 SARS-CoV-2 negative healthy controls (HC), and use the IQR of the HC measurements for the five parameters as a reference for normal parameter levels.

We perform inference using model choice for selecting the dimension of the spline basis K and PVE estimation for learning the number of eigenfunctions L, with L_max = 10 (Section 4.5). The posterior probability for K is maximal for K = 5, and the first two components are sufficient to capture > 95% of the variation (78.7% and 16.7% respectively); we therefore focus on interpreting these two components. Fig. 8 shows that these eigenfunctions, estimated for the five markers, have clear interpretations. Specifically, the first eigenfunction is above zero for CRP, IL-10, glyc-B and quinolinic acid, while it is below zero for tryptophan, over the entire 7-week disease course. This means that patients with a positive score for the first eigenfunction tend to have positive deviations from the mean for the first four markers, and a negative deviation for tryptophan. This suggests that the scores corresponding to the first eigenfunction can serve as proxies for disease severity since it has been established that patients with severe COVID-19 infection tend to have increased levels of CRP, IL-10, glyc-B and quinolinic acid, and depleted tryptophan levels; in fact, these five markers have well-established roles in inflammation and immune regulation (Bergamaschi et al., 2021; Masuda et al., 2021). The second eigenfunction reflects parameter recovery over time, since it decreases for the first four markers and increases for tryptophan, meaning that patients with a positive score for the second eigenfunction tended to see their parameter disruption resolve over time, i.e., a decrease for the levels of the first four markers and an increase for tryptophan, toward normal levels.

Fig. 8.

Eigenfunctions and patient trajectories from the mFPCA analysis of the COVID-19 data. Top row: first (blue) and second (light blue) eigenfunctions estimated for each marker and displayed over the first 7 weeks post symptom onset. Bottom rows: estimated trajectories (posterior mean with 95% prediction bands) for the four patients P1, P2, P3, P4 with most extreme scores as shown in Fig. 7. The scores of each patient for the first and second eigenfunctions are indicated in the legend. The horizontal grey band corresponds to the healthy control (HC) IQR, reflecting normal levels of the corresponding markers.

Fig. 7 displays the estimated patient scores corresponding to the two eigenfunctions. It suggests that the first set of scores reflects the clinical severity classes B to E, which corroborates their interpretation as proxies for disease severity, and this association is significant (anova p < 0.001). Moreover, the group of six patients that did not survive had significantly higher scores for the first eigenfunction and lower scores for the second eigenfunction on average, compared to random patient groups of same size (p = 0.0038 and p = 0.0085, respectively) and four of these six patients had negative scores for the second eigenfunction (poor recovery). We thus hereafter refer to the scores corresponding to the first and second eigenfunctions as “severity” and “recovery” scores, respectively.

Fig. 7.

mFPCA analysis of the COVID-19 data. Top right: cumulative percentage of variance explained by the first L_max = 10 components. The first two components capture 78.7% and 16.7% of the total variance, respectively. Bottom right: FPC 1 scores (“severity scores”) as a function of the clinical severity classes B to E. Stars: one vs. all two-sided t-tests (**** : p < 0.0001, *** : p < 0.001, ** : p < 0.01 and * : p < 0.05). Right: estimates of the FPC 1 scores (x-axis, “severity scores”) and FPC 2 scores (y-axis, “recovery scores”). The red labels P1, P2, P3 and P4 indicate the patients with the most extreme severity or recovery scores, and the grey levels indicate the clinical severity classes B (light grey), C (grey), D (dark grey) or E (black) to which each patient belongs; the gradient of grey points along the x-axis again reflects the association of the FPC 1 scores with the severity classes. The six points with a black cross indicate patients who died from COVID-19.

Fig. 8 shows the reconstructed parameter trajectories for four patients with extreme scores (referred to as “P1”, “P2”, “P3” and “P4” in Fig. 7). They also corroborate the interpretation on the scores, since the trajectories of patient P1, with the most favourable severity and recovery scores, largely overlap the normal HC range over the entire disease course. The trajectories of patient P2, with a high severity score, tend to stay well above (or below, for tryptophan) the normal range. Finally, the trajectories of patient P3 (P4), with unfavourable (favourable) recovery scores, tend to deteriorate (resolve) over time, as expected.

To further illustrate the interpretation of the scores, we use questionnaires collected up to one year after infection to ask whether the latent dynamics underlying the disease courses over the first 7 weeks post symptom onset are associated with specific long-COVID symptoms. The six patients who did not survive were all under assisted ventilation (class E) and died at 19, 20, 44, 109, 153 and 158 days post symptom onset, respectively. Table 2 shows associations of the severity and recovery scores with specific long-COVID symptoms that were evaluated for a subset of 67 patients on a scale from 0 (no symptom) to 5 (extreme manifestation of the symptom). In particular, using Kendall’s rank correlation tests with Benjamini–Hochberg multiplicity correction, severity scores are associated with six long-term symptoms, namely, dyspnoea, cough and four neurological symptoms (FDR < 5%), while there is weaker evidence of associations of recovery scores with fatigue, pain and cognition defects (FDR < 15%). Moreover, both sets of severity and recovery scores are associated with an “Overall physical and mental recovery” category recorded in all patient questionnaires.

Table 2.

Association of the scores with long-COVID symptoms. Kendall’s rank correlation and significance after Benjamini–Hochberg multiplicity adjustment (FDR: ****: < 0.1%, ***: < 1%, **: < 5%, *: < 10%, : < 15%) for the association of the severity and recovery mFPC scores with long-COVID symptoms ranked from 0 (no symptom) to 5 (extreme manifestation of the symptom). Kendall’s rank correlation and significance with over-all physical and mental recovery is also assessed for scores estimated by separate univariate FPCA analysis (p: ***: < 0.001, **: < 0.01, *: < 0.05); the sample size for the tests are indicated and correspond to the intersection of the number of patients used in each FPCA/mFPCA analysis and the number patients for which long-COVID questionnaires were collected.

mFPCA	Severity scores		Recovery scores
Dyspnoea	0.35	***	0.21
Cough	0.31	**	0.23
Chest Pain on exertion, palpitations or swollen ankles	0.11		0.01
New leg swelling in one leg or shortness of breath with chest pain	0.11		0.08
New skin rashes or sores	0.24	.	0.26	.
Voice alteration	0.22	.	0.19
Difficulties eating, drinking or swallowing	0.19		-0.02
Constant noisy breathing or throat whistling	0.04		0.17
Anosmia or dysgeusia	0.09		0.11
Difficulty to gain or maintain weight, loss of appetite	0.03		0.06
New neurology in one or more limbs	0.33	**	0.07
New pain in one or more parts of the body	0.5	****	0.24	.
General muscle weakness, balance or range of movement of joints	0.43	***	0.16
Fatigue	0.37	***	0.28	.
Cognition: memory, concentration and thinking skills	0.11		0.25	.
Overall physical & mental recovery:
mFPCA (n = 40)	0.41	***	0.25	*
FPCA CRP (n = 51)	0.31	**	0.09
FPCAIL-10 (n = 40)	0.19		0.23
FPCA glyc-B (n = 57)	0.21	*	0.01
FPCA quinolinic acid (n = 57)	0.27	**	0.08
FPCA tryptophan (n = 57)	0.14		0.11

Finally, we explore the extent to which the analysis benefits from borrowing information across the five markers using our mFPCA approach, by inspecting whether additional biological insights are obtained compared to separate univariate FPCA analyses. To this end, we perform separate applications of the univariate version of our Bayesian approach (p = 1) for each of the five markers. Except for the IL-10 analysis, the number of patients analysed is larger than for the multivariate analysis as it is not limited by the threshold ≥ 2 on the number of observations across all five markers (i.e., we have CRP: n = 96, IL-10: n = 82, glyc-B: n = 106, quinolinic acid: n = 106 and tryptophan: n = 106). Interestingly, in each univariate analysis, the scores corresponding to the first two eigenfunctions have the same interpretation as in the multivariate analysis, that is, they are proxies of severity and recovery, respectively. However, as shown in Table 2, inspecting their association with long-term recovery indicates that, while three sets of univariate severity scores, for CRP, glyc-B and quinolinic acid, are associated with the “Overall physical and mental recovery” category, none of the univariate recovery scores are associated with it. This may suggest that the joint analysis better captures the latent dynamics underlying recovery patterns, since associations with the mFPCA severity and recovery scores are both highly significant, as discussed above. Finally, as emphasised in the different simulation studies, the fact that mFPCA provides single set of scalar scores, FPC 1 and FPC 2, common to all five markers, allows effectively summarising their temporal covariation, unlike with univariate FPCA which yields separate sets of scores for the different markers, making it challenging to achieve a synthetic picture.

7. Discussion

We have presented a hierarchical modelling framework for multivariate functional principal component analysis (mFPCA) using variational message passing (VMP) and mean-field variational Bayes (MFVB) inference, which addresses important challenges posed in complex real-world observational settings, by flexibly pooling information from limited data and infrequently observed curves. This is, to the best of our knowledge, the first Bayesian approach to mFPCA.

We model the temporal covariation of multivariate curves via the model hierarchy, namely, via shared scores which allow borrowing strength across related longitudinal measurements. In addition to enhancing statistical power, this model-based approach circumvents the estimation of large covariance and cross-covariance matrices, and thus enables mFPCA in (i) sparse and irregular sampling settings and (ii) sizable real-data settings which are typically beyond the scope of existing frequentist mFPCA approaches. We acknowledge that the development of a custom MCMC algorithm could make inference feasible for a range of problem sizes, which is an interesting prospect in its own right. Instead, our focus in this work was to introduce and validate a scalable variational inference framework, which provides reliable uncertainty quantification and opportunities for direct modelling extensions, for instance tailored to large-p settings. Due to the model’s conjugacy properties, all mean-field variational inference updates are obtained in closed-form. Additionally, our variational message passing implementation, based on the same posterior factorisation, permits an elegant modularisation of the algorithm algebra under the form of fragments. Thanks to this principle, we could seamlessly repurpose fragments obtained in our previous work (Nolan et al., 2023) and combine them with a newly-derived multivariate functional principal component Gaussian likelihood fragment. This new fragment, on its own, constitutes a novel contribution, with utility extending beyond the present work context: its algebraic derivation and computer code are now readily available to statisticians willing to employ VMP inference for any models involving multivariate Gaussian likelihood components.

Our detailed simulation studies indicate that our variational implementation (whether based on MFVB and VMP) is both accurate and computationally efficient: it produces comparable or lower estimation errors on the scores and latent functions compared to MCMC inference on the same model and to the frequentist approach of Happ and Greven (2018), while being orders of magnitude faster. Our experiments also suggest that the variational lower bound (ELBO) can conveniently serve as proxy for the marginal log-likelihood in model-choice procedures to learn the numbers of spline coefficients and eigenfunctions. Crucially, we have also seen how our hierarchical model enables estimation from data with very sparsely observed curves (5-10 observations), exploiting information about other related curves with sampling grids that differ across variables and subjects.

In addition to its accuracy, flexibility and computational convenience, our Bayesian mFPCA framework possesses important advantages in terms of interpretability and uncertainty quantification. The “shared score” parametrisation offers a parsimonious representation of the major modes of joint variation of the curves, and our variational procedure approximates the full posterior distribution of the scores, making it straightforward to construct credible boundaries. These scores therefore constitute a principled, subject-level scalar summary of the multivariate curves that can be used for further analysis tasks, such as regression or clustering. We have illustrated such a use case in the COVID-19 study where we inspected the association of patient-specific “severity” (FPC 1) and “recovery” (FPC 2) scores estimated from related molecular markers, with long-term symptoms. Specifically, our mFPCA frame-work highlighted complex coordinated dynamics across the inflammatory, immune and metabolic systems, and suggested that the patients’ molecular status during the acute and post-acute phase is interlinked with incomplete clinical recovery up to one year post disease onset. This should prompt further research on organismal recovery from COVID-19, to pinpoint the specific inflammatory and immune mechanisms mobilised early in the disease course, understand their possible long-term consequences, and help formulate therapeutic recommendations applicable soon after infection to prevent adverse outcomes.

Thanks to its versatility and scalability, our framework is readily applicable to any study that involves multiple parameters measured longitudinally, possibly with scarce and irregular observations across subjects and functional curves. We anticipate that frameworks like ours will gain relevance in the near future for acquiring early personalised insights into disease risk, development and monitoring, especially when applied to routinely collected blood test data (complete blood count or CBC) available from electronic health records.

There are several avenues for methodological development. One of them concerns extensions to high-dimensional curves, a setting which is expected to gain prominence; for instance, in genomics, there is growing interest in collecting longitudinal observations on a large number of genes (up to 20 000 within the human genome) to understand how dynamic gene expression programs drive disease formation. In this context, coupling sparse prior formulations with shared scores or shared latent function assumptions could prove relevant. Another interesting question, motivated by the application to the COVID-19 data, is the formulation of a Bayesian FPCA-based joint survival model, to account for survival information in the estimation of the FPC scores, while propagating the uncertainty associated with their estimation into the survival analysis. One natural specification would be to link the longitudinal mFPCA model and the survival model through the FPC scores although this would require integrating over scores in the survival model.

Software

The R package bayesFPCA is available at https://github.com/hruffieux/bayesFPCA. The source code accompanying this article is available at https://github.com/hruffieux/VB-mFPCA-paper-code.

Supplementary Material

Supplementary material related to this article can be found online at https://doi.org/10.1016/j.csda.2024.108094.

Data availability

Data from the NIHR CITIID COVID-19 Cohort is available at https://www.covid19cellatlas.org/patient/citiid/.