Unsupervised Bayesian classification for models with scalar and functional covariates

Abstract

We consider unsupervised classification by means of a latent multinomial variable which categorizes a scalar response into one of the L components of a mixture model which incorporates scalar and functional covariates. This process can be thought as a hierarchical model with the first level modelling a scalar response according to a mixture of parametric distributions and the second level modelling the mixture probabilities by means of a generalized linear model with functional and scalar covariates. The traditional approach of treating functional covariates as vectors not only suffers from the curse of dimensionality, since functional covariates can be measured at very small intervals leading to a highly parametrized model, but also does not take into account the nature of the data. We use basis expansions to reduce the dimensionality and a Bayesian approach for estimating the parameters while providing predictions of the latent classification vector. The method is motivated by two data examples that are not easily handled by existing methods. The first example concerns identifying placebo responders on a clinical trial (normal mixture model) and the other predicting illness for milking cows (zero-inflated mixture of the Poisson model).

1. Introduction

Mixture models are popular statistical tools for classification purposes in a broad range of applied fields. Although the Gaussian mixture model is by far the most used approach for model-based cluster analysis, classification problems based on mixture models often require non-Gaussian mixture distributions. Many times we have extra information on explanatory variables and many modern applications routinely have more complex covariates in the form of vectors, matrices, functions, or images. Even though the prevailing approaches in these cases use either a parametric or a nonparametric approach and model the mean of the distribution as a function of the covariates (Cardot et al., 1999; Ferraty & Vieu, 2009; James, 2002; McLean et al., 2014, among others), there are other applications where the interest is to study the effect of the covariates on the entire distribution of the response, for example, quantile regression (e.g. Park et al., 2019, and references therein). On the other hand, some studies show no interest in modelling the mean of the distribution and instead they want to carry out classification purposes, as for example studies related to regression trees, as proposed by Breiman (2017). In this case, modelling the probability of classification through covariates is more important than the mean.

Our approach classifies an outcome as accurately as possible using only the information on the scalar and functional covariates as explanatory variables on the modelling of mixture probabilities, providing practitioners with direct interpretable cluster classification, that is, having the interpretation of the coefficients of the regression equation directly on the probability of belonging to a class. A latent variable is used to model the mixture components and divides the sample of observations into subgroups according to some similarity measure. Many authors have studied the classification problem, see, for example, Titterington et al. (1985) and McLachlan and Peel (2004) and references therein, and there are R packages (R Core Team, 2021) that perform inference for mixture models such as mixtools (Benaglia et al., 2009) but these tools cannot implement one or more functional covariates. When dealing with functional covariates, it is necessary to reduce the dimensionality of the data. One way of dealing with it is to consider a variation of principal components analysis built upon a low rank Candecomp/Parafac (CP) decomposition, a popular dimensionality-reduction method for multiway data (Kolda & Bader, 2009). In particular, a CP decomposition imposes a special low rank structure on the target regression coefficient matrix that explicitly captures the bilinear row and column effects of the matrix covariate (Jiang et al., 2017). Although this method is powerful, it requires all functional covariates to be observed at the same points across observations. Also, viewing the functional covariates as rows of a matrix and applying functional data analysis type procedures do not take advantage of the intrinsic ordering of observations (e.g. in time or space) given by the functional structure of the covariate.

We propose a more flexible classification procedure that can incorporate functional covariates on the regression of the mixture probability. Although the dimension reduction of the problem shares some characteristics of the approach of Ciarleglio et al. (2018), we focus on the regression of the mixture probability instead of the mean. We consider functional covariates as acting through linear or nonlinear effects, taking advantage of their functional nature. Additionally, the method does not restrict the functions to be observed at common points neither to have the same support, as required in Jiang et al. (2017). We employ Bayesian methods for parameter estimation by considering the classical Monte Carlo Markov chain (MCMC) approach as well as variational Bayes (VB) inference, which is an alternative method to the MCMC algorithm. This method facilitates approximation of the posterior distribution in complex models using massive data sets (Blei et al., 2017; Hoffman et al., 2013; Ormerod & Wand, 2012).

We now present two data examples.

1.1. Data set 1: identification of early responders using electroencephalography data

Placebo responders are those patients whose response is termed ‘non-specific’, e.g. in a drug trial, an improvement in symptoms that is not due to the effect of the active chemicals in the drug. There is an intense debate about how to identify placebo responders in clinical trials of medications, in particular, for major depressive disorder (MDD; Walsh et al., 2002) since there could be placebo responders among either the control or the treatment group. Furthermore, it is known that there is a high rate of placebo responders among patients in MDD treatment trials and in some experiments with selective serotonin reuptake inhibitors it was found that some patients can have a better response using placebo (Gueorguieva et al., 2011). Identifying such patients using covariates would be an important tool in clinical research. Scalar covariates such as sex and disease severity are typically included in the modelling. On the other hand, there are several studies relating differences in neural processing between placebo and active treatments (see, for example, Ciarleglio et al., 2018; Leuchter et al., 2002; Wager & Atlas, 2015; Watson et al., 2007; Zhang & Luo, 2009, and references therein).

Jiang et al. (2017) analysed data from a randomized placebo controlled depression clinical trial of sertraline in order to identify early responders to treatment (which is indicative of a placebo response since it is believed that response to the active treatment is not immediate). The data set consists of 96 MDD patients, randomized to either a drug or placebo treatment. For each subject, several scalar and categorical covariates are available, as well as their resting state electroencephalography (EEG) under a closed eyes condition. These EEG data contain the current source density amplitude spectrum values (${\mathrm{V}/\mathrm{m}}^2$) (Nunez & Srinivasan, 2006) at a total of 14 electrodes ($P_9$, $P_{10}$, $P_7$, $P_8$, $P_5$, $P_6$, $P{O}_7$, $P{O}_8$, $P{O}_3$, $P{O}_4$, $O_1$, $O_2$, $P{O}_Z$, and $O_Z$) located in occipital and parietal brain regions. Each electrode is measured at 45 frequencies at a 0.25 Hz resolution within the theta (4–7 Hz) and alpha (7–15 Hz) frequency bands. The response variable for each subject is the Hamilton Depression Rating Scale (HAM-D), measured before the treatment (baseline) and after one week into the study. It is believed that the active drug treatment can only have an effect on symptoms after two weeks. Therefore, any improvement observed after one week is likely to be due to placebo effect (or spontaneous improvement). For more details, we refer to Jiang et al. (2017) and references therein. For EEG location maps, see Figure 7 in Rupasov et al. (2012).

In order to have a baseline to compare our results, we used the hierarchical model proposed by Jiang et al. (2017) with the same scalar covariates, sex and chronicity, and functional covariates given by data taken from 14 EEG electrodes. The unobserved binary subgroup indicators are modelled via a hierarchical probit model as a function of the baseline EEG measurements and other scalar covariates of interest. In their work, instead of the regression component used in our model (cf. (17)), they propose to use the EEG data in the form of a ($14\times 45$) matrix-valued covariate. Moreover, they assume a low-dimensional structure through CP decomposition (Kolda & Bader, 2009), reducing the matrix dimension to $9\times 4$. One disadvantage of this approach is that it does not take advantage of the functional nature of the data and it lacks direct interpretability for the estimated parameters. Also, the analysis uses a tensor product and so the results will likely depend on how the electrodes are ordered in the matrix.

The pattern of EEG data varies from one subject to the other. Figure 1 shows the EEG plots for nine subjects. There may exist correlations among EEG signals. However, our research question at this point is more concerned with finding solutions to solve computational complexity, and thus we focus on comparing the MCMC and VB methods while improving our Bayesian approach by considering the method proposed by Gelman et al. (2008) by incorporating a weakly informative default prior distribution. Additionally, by simplifying our analysis to treat the EEG curves as independent, we gain more straightforward insights into individual curve characteristics and trends. In our forthcoming research, we intend to address correlation among curves by introducing a shared random curve for each EEG signal recorded from the same subject. Nonetheless, this augmentation will greatly increase the complexity of the model and it is beyond the scope of the current study.

Figure 1.

EEG patterns for nine subjects. EEG = electroencephalography.

The left panel of Figure 3 shows histograms of the change in HAM-D (baseline—week 1) showing the amount of improvement in depression symptoms after 1 week; a positive change indicates improvement in symptoms. Notice that there is a strong indication of a mixture of two distributions. Also, the choice of two classes are natural in this context since the practioners want to identify early responders. To explore the data set, we fit a parametric model using the expectation-maximization (EM) algorithm for a mixture of Gaussian distributions (different means and different variances) with no covariates, and the two fitted Gaussian curves are shown in the left panel of Figure 3, with more than 40% of the subjects classified as early responders (rightmost curve of left panel). Notice that the EM fit has low power to discriminate the subjects into two classes.

Figure 3.

Histogram of the change in HAM-D (baseline—week 1) showing the amount of improvement in depression symptoms after 1 week for both drug- and placebo-treated patients. The left panel presents the normal mixture with two components given by the EM algorithm without covariates. The right panel shows the fit using mixture of two normal distributions $0.84\times \mathcal{N}(0.82,17.08)+0.16\times \mathcal{N}(10.72,17.08)$ chosen by the linear model with normal prior and logit link. Gaussian curves are the mixture component densities. HAM-D = Hamilton Depression Rating Scale; EM = expectation-maximization.

1.2. Data set 2: predicting illness for milking cows based on functional covariates measured 30 days before lactation

Understanding factors that affect productivity of dairy cattle is essential for optimizing profitability and sustainability of dairy farms. The transition from late pregnancy to early lactation in cows is one of the most important stages of the lactation cycle of these animals as cows are at greatest risks of experiencing health disorders during this period. The occurrence of health disorders affects the productivity of cows, and thus is a determining factor of the profitability of dairy herds.

A potential strategy to identify cows with health disorders in early lactation for treatments and other interventions is through the use of automated monitoring of cow behavioural, physiological, and performance parameters with automated health monitoring systems based on sensor data. This is because it has been demonstrated that multiple sensor parameters, such as, for example, rumination time, physical activity, resting time, body temperature, milk volume, and component yield, are useful for monitoring cow health as they are dramatically altered during episodes of health disorders (Stangaferro et al., 2016a, 2016b, 2016c) and thus can be used to predict the health status of cows. Moreover, data from these sensors systems can also be combined with non-sensor data to increase the accuracy of alerts used to identify cows with health disorders. Our data set for this application consists of data collected in order to train, validate, and test machine-learning algorithms models created through a combination of sensor data from Automated Health Monitoring System and non-sensor data available at commercial dairy farms. This project funded by the United States Department of Agriculture-National Institute of Food and Agriculture was conducted by the Dairy Cattle Biology and Management Laboratory at Cornell University. We have data on daily clinical examination of cows during the first 40 days after calving (days in milk) from 258 cows. Information on whether a cow had a health disorder (1) or not (0) was collected by the research team on a daily basis. The data consist of 41 rows (0–40) per cow with sensor and non-sensor data from the previous lactation (i.e. before calving) and after calving. We focused on four sensor parameters related to physical activity for the 30 days prior to calving. The selected parameters were: activity (total number of steps in a given day divided by 24), number of resting bouts, average rest time, and total rest time per day, see Figure 2. Moreover, in this analysis, in addition to the four functional covariates as described above, we considered four scalar variables related to previous lactation period known to be associated with health outcomes after calving: $Z_1=$ age of first calving, $Z_2=$ previous lactation days in milk, $Z_3=$ previous lactation health event (0 or 1), and $Z_4=$ previous lactation number of previous health events.

Figure 2.

Functional covariates for 10 cows.

The objective of this application example was to model the number of days a cow was sick in the first 40 days after calving. Our main goal was to classify cows into three classes of status by using the scalar and functional covariates related to activity and resting times. There was high variability in the data set as seen in Table 1. During the observation period, 175 cows were not diagnosed with a health disorder whereas 83 cows with at least one health disorder event with a mean value of 1.60 sick days and a variance of 10.30. If we just consider the values not equal to zero, we get a mean value of 4.96 sick days and variance 15.95. As previously noted, we have 68% of zeros in the sample (175/258) and probably a zero-inflated distribution to accommodate overdispersion caused by zeros is appropriate. On the other hand, if we consider only the non-zero observations, we still have a large variance as compared with the mean. Therefore, we fitted a zero-inflated mixture of two Poisson distributions to accomodate the large number of zeros as well as overdispersion beyond that.

Table 1.

Frequency of sick days for lactating cows (training $+$ testing set)

Sick days	0	1	2	3	4	5	6	7	8	9	10	11	12	13	18	Total
Freq	175	22	10	7	7	4	5	4	7	6	2	3	3	2	1	258

In this case, we fitted the model to 208 cows randomly selected as a training set and used the remaining 50 cows to evaluate the prediction using the model.

The remainder of this paper is organized as follows. Section 2 introduces the hierarchical mixture model with latent variable and regression of mixture probabilities as function of functional covariates and a Bayesian approach is presented in Section 3. Sections 4 and 5 present the results for the two data examples. An extensive simulation study is given in online supplementary material, with the primary goal of examining the performance of the normal mixture model and the zero-inflated mixture Poisson (ZIMP) model by considering aspects of sample size and ability of the functional curves to predict the latent variables correctly. The secondary goal of the simulation study is to compare the estimation ability of the MCMC and VB methods, as well their performance with respect to computational time-consuming.

2. General model

For each subject $i=1,\dots ,n$, we observe: $y_i$ the scalar response, ${\mathbf{z}}_i^{\mathrm{\top }}$ a vector of scalar covariates, and ${\mathbf{X}}_i=\{(t_{ijs},X_{ij}(t_{ijs})),1\le s\le S_{ij},1\le j\le J\}$, J functional covariates observed at discrete points $t_{ijs},1\le s\le S_{ij}$ (which do need to be the same along the subjects), in closed domains ${\tau }_1,\dots ,{\tau }_J$. In general, these sets are closed intervals on the real line, and although we interpret $t_{ijs}$ as time in this study, our proposed model works for functional covariates observed at points in space or time-space. Notice that neither the domains in which we observe the functional covariates nor the observation points need to be the same for all covariates. We model the distribution of $y_i$ hierarchically by means of a latent class model, postulating a mixture distribution for the observed response to classify subjects into L classes. We will assume an L-mixture latent class model with unobserved multinomial random variables indicating class membership $\{{\mathit{\gamma }}_i=({\gamma }_{i1},\dots ,{\gamma }_{iL}{)}^{\mathrm{\top }},i=1,\dots ,n\}$ where $\sum _{l=1}^L{\gamma }_{il}=1$ and $p_{il}=P({\gamma }_{il}=1)$. Let $\mathbf{y}=(y_1,\dots ,y_n{)}^{\mathrm{\top }}$ be the vector of responses with dimension $n\times 1$ and $\mathbf{X}=\{{\mathbf{X}}_{ij}^{\mathrm{\top }},i=1,\dots ,n,j=1,\dots ,J\}$ a vector with components ${\mathbf{X}}_{ij}^{\mathrm{\top }}$ of dimension $1\times n_{ij}$, where $n_{ij}$ is the number of observed points of curve j of subject i. Additionally, let $f_l(y_i;{\mathit{\lambda }}_l)$, $i=1,\dots ,n$, be the pdf of the scalar response $y_i$ in class l, such as ${\mathit{\lambda }}_l$ is the vector of parameters in that class. The probability of the random variable ${\gamma }_{il}$ is modelled as a function of the scalar and functional covariates by

$$g(p_{il})={\mathbf{z}}_i^{\mathrm{\top }}{\mathit{\theta }}_l+{\displaystyle \sum _{j=1}^J}{\int }_{{\tau }_j}{F}_{jl}(X_{ij}(t),t,{\mathit{\varphi }}_{jl})dt$$ (1)

where g is a known link function (e.g. probit or logit), ${\mathbf{z}}_i$ is a vector of dimension $p\times 1$ representing scalar covariates, and ${\mathit{\theta }}_l$ is a vector of parameters of dimension $p\times 1$ that captures the linear additive effect of those covariates in class l. Finally, $F_{jl}(\cdot ,\cdot )$ is a bivariate smooth function related to the jth functional covariate in component l which depends on the vector of parameters ${\mathit{\varphi }}_{jl}$ . Notice that $F_{jl}$ can be very general and it includes as a special case, $F_{jl}(X_{ij}(t),t)=w(t)X_{ij}(t)$ (general regression model). This is termed the functional generalized additive model by McLean et al. (2014). Then the complete-data likelihood can be written as

$$f(\mathbf{y},\gamma \mid {\mathit{\lambda }}_1,\dots ,{\mathit{\lambda }}_L,\mathit{\theta },\mathit{\varphi },\mathbf{z},\mathbf{X})={\displaystyle \prod _{i=1}^n}{\displaystyle \prod _{l=1}^L}[f_l(y_i;{\mathit{\lambda }}_l)p_{il}{]}^{{\gamma }_{il}}$$ (2)

where the relationship between $p_{il}$ with $\mathit{\theta },\mathit{\varphi },{\mathbf{z}}_1,\dots ,{\mathbf{z}}_n$ and $\mathbf{X}$ is given by (1). Here, $\mathit{\theta }=({\mathit{\theta }}_1^{\mathrm{\top }},\dots ,{\mathit{\theta }}_L^{\mathrm{\top }}{)}^{\mathrm{\top }}$ and $\mathit{\varphi }=\{{\mathit{\varphi }}_{jl},j=1,\dots ,J,l=1,\dots ,L\}$ is a vector with components ${\mathit{\varphi }}_{jl}$ of length $p_{jl}$, where $p_{jl}$ is the number of parameters in ${\mathit{\varphi }}_{jl}$.

In general, functional covariates are considered to model the mean or any other parameter in ${\mathit{\lambda }}_l$ that indexes $f_l(\cdot ;\cdot )$ but not the mixing probabilities $p_{il}$. However, in our applications, the interest lies in using functional covariates solely to classify subjects into L classes.

2.1. Functional linear model

Model (1) can be restricted to be linear by specifying $F_{jl}(x(t),t)=w_{jl}(t)x(t)$ , yielding the more common generalized linear functional model

$$g(p_{il})={\mathbf{z}}_i^{\mathrm{\top }}{\mathit{\theta }}_l+{\displaystyle \sum _{j=1}^J}{\int }_{{\tau }_j}{w}_{jl}(t)X_{ij}(t)dt,l=1,\dots ,L.$$ (3)

To fit model (3), we consider each weight function $w_{jl}(.)$ as a smooth function belonging to the finite-dimensional space spanned by B-splines basis functions. This is not the only possibility, as other bases could be chosen such as Fourier expansion, wavelets, natural splines, etc. (Silverman, 2018). Also, we are going to choose the number of knots and knots placement in an ad hoc manner. Expressing the weight function as a linear combination of basis functions transforms the challenge of selecting $w_j$ from an infinite-dimensional class of functions into a more manageable finite class. Then, two critical issues must be addressed: determining the number of basis functions and optimizing knot placement. The impact of a small number of knots is known to cause oversmoothing, while a large number can yield overly rough estimates. It is noteworthy that, after establishing the number of basis functions, poorly placed knots can lead to undesired multimodality or a failure to capture distinct features. However, it is important to clarify that addressing knot placement is beyond the scope of this work and will not be discussed here. Interested readers can refer to Wahba (1982) and Eilers and Marx (1996) for in-depth information. To address both challenges simultaneously, penalized methods such as the Bayesian Lasso recommended by Park and Casella (2008) and Hans (2009) can be employed. However, they are very expensive to implement along with MCMC methods. In our study, we leverage commonly used procedures in nonparametric regression, substantially reducing computational costs. In the applications, the number of knots were based on visual inspection. In the early responder example, we adopt a heuristic approach to distribute knots based on the suggestion by Wegman and Wright (1983). Knots are strategically placed at the endpoints (frequencies 16 and 60) and around inflection points, specifically at the more frequent peaks with frequencies 18, 20, 22, 25, 30, 35, 40, 45, 50, and 55. Conversely, in the investigation of cow illnesses, we choose to space knots equal distantly.

For a positive integer $K_j$ and a vector of $(K_j-4)$ interior knots ${\mathrm{{\rm Y}}}_j\subset {\tau }_j$, we express the weight function as

$$w_{jl}(t)={\displaystyle \sum _{k=1}^{K_j}}{\varphi }_{jlk}{B}_k^{(j)}(t)$$ (4)

where $\{B_1^{(j)},\dots ,B_{K_j}^{(j)}\}$ are cubic B-spline basis functions determined by ${\mathrm{{\rm Y}}}_j$.

Substituting (4) into (3) yields

$$g(p_{il})={\mathbf{z}}_i^{\mathrm{\top }}{\mathit{\theta }}_l+{\displaystyle \sum _{j=1}^J}{\mathbf{R}}_{ij}^{\mathrm{\top }}{\mathit{\varphi }}_{jl}$$ (5)

where, for each pair $(j,l)$, ${\mathit{\varphi }}_{jl}=({\varphi }_{jl1},\dots ,{\varphi }_{jl{K}_j}{)}^{\mathrm{\top }}$ is a vector of parameters of dimension $K_j\times 1$ and ${\mathbf{R}}_{ij}=(R_{ij1},\dots ,R_{ij{K}_j}{)}^{\mathrm{\top }}$ is $K_j\times 1$ vector, with $R_{ijk}={\int }_{{\tau }_j}{B}_k^{(j)}(t)X_{ij}(t)dt$.

The linear case has the advantage of easy interpretability of the weight functions. If the weight $w_{jl}(t)$ is positive (negative) over the interval $(t_a,t_b)$, this means that the higher the value of $X_{ij}(t)$ in this interval, the higher (lower) the probability of ${\gamma }_{il}=1$, considering all other explanatory variables fixed. In this work, we used functional boxplots to select the functional covariates. An interesting question to be addressed in a future work is to use the posterior sample and Bayes factor to derive a test of $w_{jl}(t)=0$ over an interval.

2.2. Functional nonlinear model

For the more general model (1), we consider each function $F_{jl}(\cdot ,\cdot )$ to be a smooth surface generated by a family of tensor products of cubic B-splines (see, for example, Kim et al., 2018). That is, for each function $F_{jl}$, there exist positive integers $K_{1j}$ and $K_{2j}$ and vectors of $(K_{1j}-4)$ interior knots ${\mathrm{{\rm Y}}}_{1j}\subset {\chi }_j$ (the image of $X_{ij}$) and $(K_{2j}-4)$ interior knots ${\mathrm{{\rm Y}}}_{2j}\subset {\tau }_j$, such that

$$F_{jl}(s,t)={\displaystyle \sum _{k_1=1}^{K_{1j}}}{\displaystyle \sum _{k_2=1}^{K_{2j}}}{\varphi }_{jl{k}_1{k}_2}{B}_{k_1}^{({\mathrm{{\rm Y}}}_{1j})}(t)B_{k_2}^{({\mathrm{{\rm Y}}}_{2j})}(s)$$ (6)

where $\{B_1^{({\mathrm{{\rm Y}}}_{1j})},\dots ,B_{K_{1j}}^{({\mathrm{{\rm Y}}}_{1j})}\}$ and $\{B_1^{({\mathrm{{\rm Y}}}_{2j})},\dots ,B_{K_{2j}}^{({\mathrm{{\rm Y}}}_{1j})}\}$ are B-spline basis determined by ${\mathrm{{\rm Y}}}_{1j}$ and ${\mathrm{{\rm Y}}}_{2j}$, respectively.

Substituting (6) into (1) yields

$$g(p_{il})={\mathbf{z}}_i^{\mathrm{\top }}{\mathit{\theta }}_l+{\displaystyle \sum _{j=1}^J}{\mathbf{R}}_{ij}^{\mathrm{\top }}{\mathit{\varphi }}_{jl}$$ (7)

where ${\mathbf{R}}_{ij}$ are vectors of dimension $(K_1{K}_2)\times 1$ with ${\mathit{\varphi }}_{jl}(k_1,k_2)={\varphi }_{jl{k}_1{k}_2}$ and ${\mathbf{R}}_{ij}(k_1,k_2)={\int }_{{\chi }_j}{\int }_{{\tau }_j}{B}_{k_1}^{(X,j)}(X_{ij}(s))B_{k_2}^{(T,j)}(t)dtds$ properly stacked.

As we can see, from (5) and (7), using basis expansions for both the linear and nonlinear cases, we end up with the same linear structure in terms of parameters $\mathit{\theta }$ and $\mathit{\varphi }$. Notice that the functional covariates are used to compute the ${\mathbf{R}}_{ij}$ and they are not expanded using basis functions.

3. A Bayesian approach to the mixture model regression with functional covariates

Model (2) specifies the distribution of the response $y_i$ depending on which mixture component subject i belongs to. The mixture component densities are parameterized by the vector $\mathit{\lambda }=({\mathit{\lambda }}_1^{\mathrm{\top }},\dots ,{\mathit{\lambda }}_L^{\mathrm{\top }}{)}^{\mathrm{\top }}$ whose components are row vectors related to the densities $f_1,\dots ,f_L$, respectively, considering category L as the baseline. Therefore, we will denote by $\mathrm{\Theta }=({\mathit{\lambda }}^{\mathrm{\top }},{\mathit{\beta }}^{\mathrm{\top }}{)}^{\mathrm{\top }}$ the vector of unknown parameters in the mixture model, where $\mathit{\beta }=({\mathit{\beta }}_1^{\mathrm{\top }},\dots ,{\mathit{\beta }}_{L-1}^{\mathrm{\top }}{)}^{\mathrm{\top }}$ with components ${\mathit{\beta }}_l=({\mathit{\theta }}_l^{\mathrm{\top }},{\mathit{\varphi }}_{1l}^{\mathrm{\top }},\dots ,{\mathit{\varphi }}_{Jl}^{\mathrm{\top }}{)}^{\mathrm{\top }}$ is a column vector of parameters for the regression coefficients under component l with ${\mathbf{x}}_i:=({\mathbf{z}}_i^{\mathrm{\top }},{\mathbf{R}}_{i1}^{\mathrm{\top }},\dots ,{\mathbf{R}}_{iJ}^{\mathrm{\top }}{)}^{\mathrm{\top }}$ being the row vector of covariates for subject i. Without loss of generality, we consider the same ${\mathbf{x}}_i$ across the L mixture components in the development of the methodology. However, the methodology still applies when the covariates differ across the mixture components.

3.1. Hierarchical structure specification and prior specification

A formal Bayesian analysis of a mixture model usually leads to intractable calculations. Data augmentation is an efficient procedure for mixture models that leads to feasible computations using Gibbs sampling (Diebolt & Robert, 1994). The joint augmented posterior distribution is the product of (2) and the prior distributions, which has no closed form. There is no standard solution for the general case. However, in both applications, using suitable prior distributions, all the conditional distributions are available and the Gibbs sampling algorithm is suitable to sample from the posterior distribution of $\mathit{\gamma }$, $\mathit{\lambda }$, and $\mathit{\beta }$.

The nature of the application under study dictates the form of $f_l(y_i;{\mathit{\lambda }}_l)$ which in turn provides knowledge about the nature of parameters in ${\mathit{\lambda }}_l$. There is a very rich family of distributions $f_l(y_i;{\mathit{\lambda }}_l)$ that may characterize the mixture distribution of $y_i$. Instead of focusing on a specific distribution for mixture components, we focus on a general solution for posterior sampling of the latent variables $\mathit{\gamma }$ and parameters in $\mathbf{\Theta }$, which are developed with a general $f_l(y_i;{\mathit{\lambda }}_l)$ without loss of generality. Therefore, prior distributions for $\mathit{\lambda }$ are problem specific. The full conditional posterior distribution of ${\mathit{\beta }}_l$ cannot be computed explicitly except for when we are using the probit link function and thus we can apply the simple latent variable method of Albert and Chib (1993). Other methods for calculating the full conditional have been proposed using data-augmentation or multiple layers of latent variables, see for example Held and Holmes (2006), Frühwirth-Schnatter and Frühwirth (2010), Gramacy and Polson (2012), and Polson et al. (2013). In our approach, for each of the components of $\mathit{\beta }$, we will assume a Student-t prior distribution with mean 0, degrees-of-freedom parameter df, and scale $\varsigma$, with df and $\varsigma$ providing minimal prior information to constrain the coefficients to lie in a reasonable range (see Section 2 of Gelman et al., 2008). An advantage of the t family is that fat-tailed distributions allow for flexible inference, since it includes both the normal ($df=\mathrm{\infty }$) and the Cauchy ($df=1$) distributions.

3.2. Posterior computation of parameters

We sample from the posterior distribution using a Gibbs sampling scheme, and most of the full conditional posterior distribution of the latent variables $\mathit{\gamma }$ and parameters in $\mathbf{\Theta }$ are given by standard methods. The only non-standard computation is the conditional posterior distributions for $\mathit{\beta }$.

3.2.1. Full conditional posterior distribution of ${\mathit{\beta }}_l$

We follow Gelman et al. (2008), which incorporates a prior distribution into classical logistic regression computations. As in the classical iteratively weighted least-squares algorithm, at each iteration the algorithm determines the pseudo-data ${\psi }_{il}$ given by

$${\psi }_{il}=g(p_{il})+({\gamma }_{il}-p_{il})g^{\prime }(p_{il}),i=1,\dots ,n$$ (8)

and weights ${\sigma }_{\psi ,il}^2$ satisfying the equation

$$[{\sigma }_{\psi ,il}^2{]}^{-1}={[g^{\prime }(p_{il})]}^2{V}_{il}^{-1}$$ (9)

where $V_{il}$ is the variance function. In the classical approach of McCullagh and Nelder (1989), at each iteration of the algorithm, equations (8) and (9) are evaluated at ${\hat{p}}_{il}$, given in equation (7). Under Gelman et al. (2008) approach, ${\psi }_{il}$ is approximated by a normal distribution with mean $g(p_{il})={\mathbf{x}}_i^{\mathrm{\top }}\mathit{\beta }$ and variance ${\sigma }_{\psi ,il}^2$. Considering a normal prior distribution for $\mathit{\beta }$, the full conditional of $\mathit{\beta }$ conjugates to a normal distribution. On the other hand, if independent Student-t distributions are considered for each ${\beta }_{ld}$ in ${\mathit{\beta }}_l$, $d=1,\dots ,D$, being expressed as a mixture of a normal distribution with mean ${\mu }_{ld,\beta }$ and unknown ${\sigma }_{ld}^2$ with an inverse scale chi-square distribution with scale parameter ${\varsigma }_{ld}$ and degrees of freedom ${\nu }_{ld}$, the approximate logarithm of the full conditional posterior density of ${\mathit{\beta }}_l$ and ${\mathit{\sigma }}_l=({\sigma }_{l1},\dots ,{\sigma }_{lD}{)}^{\mathrm{\top }}$ is given by

$$\begin{array}{rl}\log p({\mathit{\beta }}_l,{\mathit{\sigma }}_l\mid \mathit{\psi })& \propto -\frac{1}{2}{\displaystyle \sum _{i=1}^n}\frac{1}{{\sigma }_{\psi ,il}^2}({\psi }_{il}-{\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_l{)}^2\\ & -\frac{1}{2}{\displaystyle \sum _{d=1}^D}[\frac{1}{{\sigma }_{ld}^2}({\beta }_{ld}-{\mu }_{ld,\beta }{)}^2+\log ({\sigma }_{ld}^2)\\ & +\log p({\sigma }_{ld}\mid s_{ld},{\nu }_{ld})].\end{array}$$ (10)

Notice (10) has no closed form. Gelman et al. (2008) use an approximate EM algorithm (Dempster et al., 1977) and their algorithm proceeds taking the average over ${\mathit{\beta }}_l$ at each step, treating them as missing data and performing the EM algorithm to estimate $\mathit{\sigma }$. The algorithm starts at some value of $\mathit{\sigma }$ and proceeds by alternating one step of iteratively weighted least squares to update $\mathit{\beta }$ and one step of EM to update $\mathit{\sigma }$. Once enough iterations have been performed to reach approximate convergence, we get an sample of ${\mathit{\beta }}_l=({\mathit{\theta }}_l^{\mathrm{\top }},{\mathit{\varphi }}_{1l}^{\mathrm{\top }},\dots ,{\mathit{\varphi }}_{Jl}^{\mathrm{\top }}{)}^{\mathrm{\top }}$. This step is performed inside the Gibbs algorithm. In our case, we follow this scheme to obtain a sample from the full conditional of $\mathit{\beta }$ using the bayesglm function implemented in R by Gelman et al. (2008). To use the function bayesglm, we specify the link function g as either the probit or logit function, and set the degrees of freedom ${\nu }_{ld}$ appropriately to have normal (${\nu }_{ld}=\mathrm{\infty }$) and the Cauchy (${\nu }_{ld}=1$) as prior distributions.

3.3. Variational inference

To scale the problem described in the introduction of this paper for large samples and to include more functional covariates, we must resort to approximate posterior inference. VB inference is a machine-learning technique that facilitates approximation of the posterior distribution in complex models using massive data sets (Blei et al., 2017; Hoffman et al., 2013; Ormerod & Wand, 2012). VB inference provides the main alternative to the Markov chain Monte Carlo (MCMC) algorithm. It starts by introducing a variational family of distributions, indexed by some variational parameters $\mathit{\kappa }$ and a criterion function. Then it searches for the member $q(\cdot \mid \mathit{\kappa })$ of the family that best approximates the predictive distribution. The optimization criterion is derived based on the log-marginal posterior distribution of the observed data, an usual model selection criterion, $\log (p(y\mid \mathbf{M}\mathbf{)}\mathbf{)}$ . Often this quantity evolves to where it requires the solution to an intractable integral. To avoid this tedious calculation, a lower bound quantity, called ELBO (evidence lower bound), is easily evaluated as

$$\begin{array}{rl}\log (p(y\mid \mathbf{M}))& =\log (\int \int \int p(y,\mathit{\lambda },\mathit{\beta },\mathit{\gamma })d\mathit{\gamma }d\mathit{\beta }d\mathit{\lambda })\\ & =\log \left(E_q\left[\frac{p(y,\mathit{\lambda },\mathit{\beta },\mathit{\gamma })}{q(\mathit{\lambda },\mathit{\beta },\mathit{\gamma }\mid \mathit{\kappa })}\right]\right)\ge E_q\left[\log \frac{p(y,\mathit{\lambda },\mathit{\beta },\mathit{\gamma })}{q(\mathit{\lambda },\mathit{\beta },\mathit{\gamma }\mid \mathit{\kappa })}\right]\end{array}$$ (11)

where $\mathit{\gamma }$ and $(\mathit{\lambda },\mathit{\beta })$ represent local and global quantities/parameters, respectively. The inequality in (11) is obtained by Jensen’s inequality. It is natural to use this lower bound as a model selection criterion in place of the predictive distribution, avoiding cumbersome high-dimensional integration. Therefore, the VB inference objective is to maximize ELBO, which is equivalent to minimizing the Kulback–Leibler divergence up to an additive constant (Blei et al., 2017). We propose the following partition of the joint distribution of local and global, denominated mean field family (Parisi, 1988):

$$q(\mathit{\lambda },\mathit{\beta },\mathit{\gamma }\mid \mathit{\kappa })=q(\mathit{\lambda }\mid \mathit{\kappa })q(\mathit{\beta }\mid \mathit{\lambda },\mathit{\kappa }){\displaystyle \prod _{i=1}^n}q({\gamma }_i\mid \mathit{\beta },\mathit{\kappa })$$ (12)

where $\mathit{\kappa }$ comprises all the parameters of the variational family. To avoid a cumbersome notation, we are using the same notation q for the joint variational distribution of (11) and the conditional distributions in (12).

The approximate conditional inference is viewed as an optimization problem. Given the above setup, the mean field family, and the ELBO criterion, one can find the optimal solution via the coordinate ascent variational inference algorithm (Bishop, 2006). Each factor of the mean field variational density is optimized iteratively, while keeping the others fixed, climbing the ELBO to a local optimum.

Letting $\mathit{\nu }=(\mathit{\lambda },\mathit{\beta },\mathit{\gamma })$, we need to compute $q^{*}(\mathit{\nu }\mid \mathit{\kappa })=q^{*}(\mathit{\lambda }\mid \mathit{\kappa })q^{*}(\mathit{\beta }\mid \lambda ,\mathit{\kappa })\prod _{i=1}^n{q}^{*}({\gamma }_i\mid \mathit{\beta },\mathit{\kappa })$ where

$$q^{*}(\mathit{\lambda }\mid \mathit{\kappa })\propto \text{exp}\{E_{(\mathit{\beta },\mathit{\gamma })}\log p(\mathit{\lambda }\mid \mathit{\beta },\mathit{\gamma },\mathbf{y},\mathit{\kappa })\}$$ (13)

$$q^{*}(\mathit{\beta }\mid \mathit{\kappa })\propto \text{exp}\{E_{(\mathit{\lambda },\mathit{\gamma })}\log p(\mathit{\beta }\mid \mathit{\lambda },\mathit{\gamma },\mathbf{y},\mathit{\kappa })\},\text{and}$$ (14)

$$q^{*}({\gamma }_i\mid \mathit{\kappa })\propto \text{exp}\{E_{(\mathit{\gamma }(-{\gamma }_i),\mathit{\lambda },\mathit{\beta })}\log p({\gamma }_i\mid \mathit{\gamma }(-{\gamma }_i),\mathit{\lambda },\mathit{\beta },\mathbf{y})\}$$ (15)

It is worth pointing out that the form of the optimal densities involves the full conditional distributions, revealing a link with Gibbs sampling. However, the VB algorithm does not repeatedly simulate from the full conditional distributions as is done by the Gibbs sampler.

4. Identification of early responders using EEG data

Let $y_i$ denote the change in the HAM-D (baseline—week 1) for subject i, $i=1,\dots ,96$, where a positive change indicates diminished depression symptom severity. In order to compare our results with Jiang et al. (2017), we will focus on the same model, given by (16), with the same scalar covariates, sex and chronicity, and functional covariates given by data taken from 14 EEG electrodes. The unobserved binary subgroup indicators are modelled via a hierarchical probit model as a function of the baseline EEG measurements and other scalar covariates of interest.

For this data set, we deal with the mixture model of two normal distributions with different means but the same variance. However, it is straightforward to generalize the computations for L classes and different variances. Let $Y_1,\dots ,Y_n$ be independent random variables with

$$p(y_i\mid {\mu }_0,{\mu }_1,{\sigma }^2,{\gamma }_i)={\gamma }_i\varphi (y_i;{\mu }_1,{\sigma }^2)+(1-{\gamma }_i)\varphi (y_i;{\mu }_0,{\sigma }^2),{\mu }_1>{\mu }_0$$ (16)

where $\varphi (.;\mu ,{\sigma }^2)$ is the normal density with parameters μ and ${\sigma }^2$, and ${\gamma }_1,\dots ,{\gamma }_n$ are binary latent random variables with $p({\gamma }_i\mid \mathit{\beta })=p_i(\mathit{\beta }{)}^{{\gamma }_i}(1-p_i(\mathit{\beta }){)}^{1-{\gamma }_i},{\gamma }_i=0\text{or}1$, where

$$p_i(\mathit{\beta })=g^{-1}\left({\mathbf{x}}_i^{\mathrm{\top }}\mathit{\beta }\right)$$ (17)

g is a link function, and $\mathit{\beta }=({\mathit{\theta }}^{\mathrm{\top }},{\mathit{\varphi }}_1^{\mathrm{\top }},\dots ,{\mathit{\varphi }}_J^{\mathrm{\top }}{)}^{\mathrm{\top }}$ with $\mathit{\theta }$ being a column vector of parameters associated to scalar effects and ${\mathit{\varphi }}_j^{\mathrm{\top }}$ are column vectors representing the coefficients of a function written from a B-splines expansion given by (5) or (7).

For the parameters in $\mathit{\lambda }=({\mu }_0,{\mu }_1,{\sigma }^2{)}^{\mathrm{\top }}$ in the model, we use diffuse priors, ${\mu }_0\sim N(0,{\tau }_0^2)$ and ${\mu }_1\sim N(0,{\tau }_1^2)$ with ${\tau }_0^2={\tau }_1^2=100$.

For each coefficient in $\mathit{\beta }$, we specify weakly informative t family prior distribution.

The observed data likelihood for the hierarchical model is difficult to optimize directly because the unobserved vector $\mathit{\gamma }=\{{\gamma }_i{\}}_{i=1}^n$. Denoting by $\mathbf{\Theta }=({\mathit{\lambda }}^{\mathrm{\top }},{\mathit{\beta }}^{\mathrm{\top }}{)}^{\mathrm{\top }}$, the complete likelihood is given by

$$f(\mathbf{y},\mathit{\gamma }\mid \mathbf{\Theta })={\displaystyle \prod _{i=1}^n}\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\{-\frac{(y_i-{\mu }_0(1-{\gamma }_i)-{\mu }_1{\gamma }_i{)}^2}{2{\sigma }^2}\}[p_i(\mathit{\beta }){]}^{{\gamma }_i}[1-p_i(\mathit{\beta }){]}^{(1-{\gamma }_i)}$$ (18)

4.1. Full conditional posterior distributions

The joint augmented posterior distribution is proportional to the product of the likelihood given by (18) and prior distributions specified in the previous section and has no closed form. Therefore, we adapt the Gibbs sampling algorithm to sample from the full conditional posterior distribution of $\mathbf{\Theta }$ and the latent variables $\mathit{\gamma }$.

4.1.1. Full conditional posterior distribution of ${\gamma }_i$

Let ${\mathit{\gamma }}_{-i}$ be the vector $\mathit{\gamma }=({\gamma }_1,\dots ,{\gamma }_{i-1},{\gamma }_{i+1},\dots ,{\gamma }_n)$. The full conditional posterior distribution of ${\gamma }_i$ is given by

$$P({\gamma }_i=1\mid \mathbf{\Theta },\mathbf{y},{\mathit{\gamma }}_{-i})=\frac{\varphi \left(\frac{y_i-{\mu }_1}{\sigma }\right)p_i(\mathit{\beta })}{\varphi \left(\frac{y_i-{\mu }_1}{\sigma }\right)p_i(\mathit{\beta })+\varphi \left(\frac{y_i-{\mu }_0}{\sigma }\right)(1-p_i(\mathit{\beta }))}$$

since $P({\gamma }_i=x,\mathbf{\Theta },\mathbf{y},{\mathit{\gamma }}_{-i})=\varphi \left(\frac{y_i-{\mu }_{1}x-{\mu }_0(1-x)}{\sigma }\right)p_i(\mathit{\beta })$ where $p_i(\mathit{\beta })$ depends on the covariates through the regression term and it is given by (17).

4.1.2. Full conditional posterior distribution of ${\mu }_0$ and ${\mu }_1$

We update ${\mu }_0$ using a normal distribution with mean ${(\frac{1}{{\tau }_0^2}+\frac{1}{{\sigma }^2}\sum _{i=1}^n(1-{\gamma }_i))}^{-1}\sum _{i=1}^n{y}_i(1-{\gamma }_i)$ and variance ${(\frac{1}{{\tau }_0^2}+\frac{1}{{\sigma }^2}\sum _{i=1}^n(1-{\gamma }_i))}^{-1},$ while ${\mu }_1$ is updated, conditionally on ${\mu }_0$, with a truncated normal distribution on $({\mu }_0,\mathrm{\infty })$, with mean and variance given by

$${(\frac{1}{{\tau }_1^2}+\frac{1}{{\sigma }^2}{\displaystyle \sum _{i=1}^n}{\gamma }_i)}^{-1}{\displaystyle \sum _{i=1}^n}{y}_i{\gamma }_i\text{and}{(\frac{1}{{\tau }_1^2}+\frac{1}{{\sigma }^2}{\displaystyle \sum _{i=1}^n}{\gamma }_i)}^{-1}$$

respectively.

4.1.3. Full conditional posterior distribution of ${\sigma }^2$

We update ${\sigma }^2$ using an inverse gamma distribution with parameters $a_0+\frac{n}{2}$ and $b_0+\sum _{i=1}^n(y_i{\gamma }_i-{\mu }_1{)}^2+(y_i(1-{\gamma }_i)-{\mu }_0{)}^2$.

4.1.4. Full conditional posterior distribution of ${\mathit{\beta }}_l$

To derive the full conditional posterior distribution of ${\mathit{\beta }}_l$, we proceed as described in Section 3.2.1. The distribution will depend on the prior of ${\mathit{\beta }}_l$, which can be normal, Cauchy, or t.

4.2. VB for the normal mixed model

For the mixture of normal distributions model, the augmented vector of unknown parameters is $\mathit{\nu }=({\mathit{\lambda }}^{\mathrm{\top }},{\mathit{\beta }}^{\mathrm{\top }},{\mathit{\gamma }}^{\mathrm{\top }}{)}^{\mathrm{\top }},$ where $\mathit{\lambda }=({\mu }_0,{\mu }_1,{\sigma }^2{)}^{\mathrm{\top }}$, $\mathit{\beta }=({\mathit{\theta }}^{\mathrm{\top }},{\mathit{\varphi }}^{\mathrm{\top }}{)}^{\mathrm{\top }}$, and $\mathit{\gamma }=({\gamma }_1,\dots ,{\gamma }_n{)}^{\mathrm{\top }}$. For the variational distribution we choose:

• {${\gamma }_i,i=1,\dots ,n\}$: family of independent Bernoulli random variable with variational parameter ${\alpha }_i$;

• ${\mu }_0$ and ${\mu }_1$: independent normal distributions with means $m_0$ and $m_1$ and variances $s_0^2$ and $s_1^2$, respectively;

• ${\sigma }^2$ : inverse gamma with parameters $A_0$ and $B_0$;

• $\mathit{\beta }$ : normal distribution with covariance matrix ${\mathbf{V}}_{\beta }^{*}$ and mean ${\mathit{\mu }}_{\beta }^{*}$.

Denote the parameters of the variational distributions as

$$\mathit{\kappa }=(m_0,s_0^2,m_1,s_1^2,\mathit{\alpha },A_0,B_0,{\mathit{\mu }}_{\beta },{\mathbf{V}}_{\mathit{\beta }})$$

and define $q^{*}(\mathit{\nu }\mid \mathit{\kappa })=q^{*}({\mu }_0\mid \mathit{\kappa })q^{*}({\mu }_1\mid \mathit{\kappa })q^{*}({\sigma }^2\mid \mathit{\kappa })q^{*}(\mathit{\beta }\mid \mathit{\kappa })\prod _{i=1}^n{q}^{*}({\gamma }_i\mid \mathit{\kappa })$. According to equations (13), (14), and (15), we have to calculate the variational distributions of ${\mu }_0,{\mu }_1,{\sigma }^2,\mathit{\beta }$, and ${\gamma }_1,\dots ,{\gamma }_n$.

In the next sections, to simplify the notation, we will omit the dependence on $\mathit{\kappa }$ when writing the variational distributions $q^{*}$. The details of computations can be found in the online supplementary material.

4.2.1. Variational density $q^{*}({\mathit{\gamma }}_i)$

If we consider $q^{*}({\mu }_0)$ and $q^{*}({\mu }_1)$ belonging to the family of independent distributions with means $m_0$ and $m_1$ and variances $s_0^2$ and $s_1^2$, respectively, we get that ${\gamma }_i$ is a Bernoulli random variable with variational parameter ${\alpha }_i={\alpha }_{i1}/({\alpha }_{i0}+{\alpha }_{i1})$ where

$${\alpha }_{il}=\text{exp}\{E_{q^{*}(\mathit{\theta })}\log [1-p_i(\mathit{\beta })]-E_{q^{*}({\sigma }^2)}\left(\frac{1}{2{\sigma }^2}\right)[(y_i-m_l{)}^2+s_l^2]\},l=0,1$$

4.2.2. Variational densities $q^{*}({\mu }_0)$ and $q^{*}({\mu }_1)$

For $k=0,1$, the variational distribution of ${\mu }_k$ is Gaussian with mean $m_k$ and variance $s_k^2$ given by

$$m_0=s_0^2{E}_{q^{*}({\sigma }^2)}[1/{\sigma }^2]{\displaystyle \sum _{i=1}^n}(1-{\alpha }_i)y_i,s_0^2=\frac{1}{E_{q^{*}({\sigma }^2)}[1/{\sigma }^2]{\displaystyle \sum _{i=1}^n}(1-{\alpha }_i)+1/{\tau }_0^2}$$

and

$$m_1=s_1^2{E}_{q^{*}({\sigma }^2)}[1/{\sigma }^2]{\displaystyle \sum _{i=1}^n}{\alpha }_i{y}_i,s_1^2=\frac{1}{E_{{\sigma }^2}[1/{\sigma }^2]{\displaystyle \sum _{i=1}^n}{\alpha }_i+1/{\tau }_1^2}$$

where ${\tau }_0^2$ and ${\tau }_1^2$ are the parameters from the prior distribution.

4.2.3. Variational density $q^{*}({\sigma }^2)$

The variational density of ${\sigma }^2$, considering the likelihood and the prior distribution of ${\sigma }^2\sim IG(a_0,b_0)$, is given by an inverse gamma with parameters $A_0=a_0+n/2$ and

$$B_0=b_0+{\displaystyle \sum _{i=1}^n}\frac{{\alpha }_i}{2}((y_i-m_1{)}^2+s_1^2)+{\displaystyle \sum _{i=1}^n}\frac{(1-{\alpha }_i)}{2}((y_i-m_0{)}^2+s_0^2)$$

4.2.4. Variational density $q^{*}(\mathit{\beta })$

Assuming a t prior distribution to each component of ${\mathit{\beta }}_l$ in $\mathit{\beta }$, the variational density $q^{*}({\mathit{\beta }}_l)$ of ${\mathit{\beta }}_l$ is derived by taking the expectation of equation (10) with respect to ${\gamma }_i$, derived from

$$E_{q*({\gamma }_i)}({\psi }_{il})=g({\hat{p}}_{ilk})+({\alpha }_i-{\hat{p}}_{ilk})g^{\prime }({\hat{p}}_{ilk})$$ (19)

with ${\alpha }_i=E_{q^{*}({\gamma }_i)}({\gamma }_i)$ given in Section 4.2.1. There is no closed form for $q^{*}({\mathit{\beta }}_l)$ and therefore we cannot compute the ELBO.

However, taking the prior distribution of ${\mathit{\beta }}_l$ as a normal distribution with mean ${\mathit{\mu }}_{\beta }$ and covariance matrix ${\mathbf{\Sigma }}_{\beta }$, the variational $q^{*}({\mathit{\beta }}_l)$ is given by a normal distribution with covariance matrix ${\mathbf{V}}_{\beta }^{*}=({\mathbf{x}}^{\mathrm{\top }}{\mathbf{\Sigma }}_{\psi }^{-1}\mathbf{x}+{\mathbf{\Sigma }}_{\beta }^{-1}{)}^{-1}$ and mean ${\mathit{\mu }}_{\beta }^{*}={\mathbf{V}}_{\beta }^{*}({\mathbf{x}}^{\mathrm{\top }}{\mathbf{\Sigma }}_{\psi }^{-1}{E}_{q^{*}(\gamma )}({\mathit{\psi }}_l)+{\mathbf{\Sigma }}_{\beta }^{-1}{\mathit{\mu }}_{\beta })$, with elements in $E_{q^{*}(\gamma )}({\mathit{\psi }}_l)$ given by (19) and ${\mathbf{\Sigma }}_{\psi }$ is a diagonal matrix of size n with elements $[{\sigma }_{\psi ,il}^2{]}^{-1}$ given in equation (9).

4.3. Results

To sample from the posterior distribution, we use the Gibbs sampler scheme considering the posterior calculation described in Section 3.2. We ran two chains for $M=150,000$ steps. We calculated Gelman and Rubin’s convergence diagnostic (Gelman & Rubin, 1992) using the R package coda and the statistics and plots showed convergence for all parameters leading to a burn-in of $M_L=M/2=75,000$ steps and samples were taken every 100 steps. Table 2 presents the posterior estimates, using one of the chains, of the parameters using logit link function and t, Cauchy and normal priors for the parameters in $\mathit{\beta },$ and linear vs. nonlinear functional models. The value $\hat{p}$ in each model represents

Table 2.

Posterior mean (standard deviation) of ${\mu }_0,{\mu }_1,{\sigma }^2$, $\mathit{\theta }$, and p for different prior distributions and linear and nonlinear functional models

	Logit link
	Nonlinear model			Linear model
Parameter	Student-t	Normal	Cauchy	Student-t	Normal	Cauchy
${\mu }_0$	0.88 (0.48)	0.87 (0.47)	0.89 (0.48)	0.86 (0.61)	0.82 (0.61)	0.89 (0.60)
${\mu }_1$	11.52 (1.18)	11.57 (1.18)	11.44 (1.16)	10.81 (1.57)	10.72 (1.52)	10.86 (1.55)
Intercept	$-$ 0.008 (0.005)	$-$ 0.008 (0.005)	$-$ 0.008 (0.005)	$-$ 1.07 (0.34)	$-$ 1.07 (0.34)	$-$ 1.26 (0.79)
Sex	$-$ 0.34 (0.21)	$-$ 0.34 (0.21)	$-$ 0.30 (0.19)	$-$ 1.85 (0.48)	$-$ 1.85 (0.45)	$-$ 1.79 (0.54)
Chronicity	$-$ 0.43 (0.19)	$-$ 0.44 (0.19)	$-$ 0.39 (0.17)	$-$ 1.76 (0.45)	$-$ 1.80 (0.44)	$-$ 1.70 (0.54)
${\sigma }^2$	15.82 (2.30)	15.81 (2.34)	15.95 (2.48)	17.21 (3.00)	17.08 (2.85)	17.32 (2.93)
$\hat{p}$	0.18	0.16	0.18	0.15	0.16	0.15

$$\hat{p}=\frac{1}{96}{\displaystyle \sum _{i=1}^{96}}\mathbf{1}({\hat{p}}_i>1/2).$$

Notice that all results in Table 2 are very similar. To choose the best model, we compute the posterior predictive checks (Bayarri & Berger, 2000; Gelman & Meng, 1996) for each $t=-5,0,5$, and 10 and each model, we sample ${\mathbf{\Theta }}_{1,t},\dots ,{\mathbf{\Theta }}_{M,t}$ from the posterior distribution, generate ${\tilde{y}}_{1rt},\dots ,{\tilde{y}}_{96rt}$ , $r=1,\dots ,M$, from the likelihood and considering M samples after burn-in $M_L$ and thinning of 100, leading to 750 samples. Then we compute

$$h({\tilde{\mathbf{y}}}_t)=\frac{1}{M}{\displaystyle \sum _{r=1}^{750}}(1/96){\displaystyle \sum _{i=1}^{96}}\mathbf{1}({\tilde{y}}_{irt}>t)$$

as shown in Table 3.

Table 3.

Posterior predictive checks as a function of the threshold (t) for different prior distributions and linear and nonlinear functional models with logit link

t	Nonlinear model			Linear model
	Student-t	Normal	Cauchy	Student-t	Normal	Cauchy
$-5$	0.51	0.51	0.53	0.44	0.45	0.46
0	0.61	0.61	0.63	0.62	0.62	0.63
5	0.34	0.33	0.36	0.47	0.48	0.48
10	0.44	0.47	0.47	0.39	0.39	0.40

Gelman (2013) showed that posterior predictive values do not have uniform distribution under the null hypothesis, but rather tend to be concentrated near 0.5. Therefore, the closer to 0.5, the better the model. Although the nonlinear model is slightly better than the linear model for $t=-5$ and $t=10$ as shown in Table 3 independently of the prior distribution, by the parsimonious criterion we chose the linear model fitted with the normal prior distribution and consider this model.

Figure 3 shows histograms of the change in HAM-D (baseline—week 1) showing the amount of improvement in depression symptoms after 1 week, a positive change indicates improvement in symptoms. The right panel presents the fitted distribution ($0.84\times N(0.82,17.08)+0.16\times N(10.72,17.08)$) as chosen by the linear model with normal prior and logit link. For comparison, we present the parametric model using the EM algorithm for a mixture of Gaussian distributions (different means and different variances) with no covariates. Comparing left and right panels of Figure 3, we can see that the fitted model has a much better ability to discriminate between the two groups since the two distributions have less overlap.

In order to classify subjects as early responders, we used the maximum posterior estimate of $p({\gamma }_i\mid \mathbf{y})$, see Figure 4. Specifically, 10 patients exhibit near-certain early responder characteristics (posterior mean probability greater than $0.9$) while an additional 5 patients show a high likelihood of being early responders (posterior mean probability in the range of $[0.5,0.9]$).

Figure 4.

Posterior mean for the probability of being an early responder for the 96 patients.

For comparison between MCMC and VB results, Table 4 shows the estimated expected value estimates for the parameters based on posterior mean for MCMC and the expectation of the variational distribution ($E_{q*}$) for VB for the linear/nonlinear model, logit link. We present only the results for the normal prior as we operate VB with normal prior only (cf. Table 2). Although point estimates of the parameters obtained by MCMC and VB appear to be discrepant, comparing results for linear logit model with normal prior, except by the intercept and female effect, all estimates for the posterior expectation of the parameters under VB approach are inside the MCMC high probability density 95% credible interval.

Table 4.

Estimated posterior expectation of the parameters and the computational time for the normal mixture model using MCMC (posterior mean—PM and high probability density interval—HPDI) and VB ($E_{q^{*}}[\cdot ]$) approaches

	VB		MCMC
	Nonlinear	Linear	Nonlinear	Linear
Parameter	$E_{q*}[\cdot ]$	$E_{q^{*}}[\cdot ]$	PM	PM	HPDI
${\mu }_0$	1.40	1.13	0.87	0.82	($-$0.26; 2.08)
${\mu }_1$	13.34	12.38	11.57	10.72	(7.80; 13.56)
${\sigma }^2$	18.14	16.68	15.81	17.08	(11.78; 22.64)
Intercept (${\theta }_0$)	$-$ 0.01	$-$ 0.30	$-$ 0.008	$-$ 1.07	($-$1.81; $-$0.48)
Female (${\theta }_1$)	$-$ 0.14	$-$ 0.93	$-$ 0.34	$-$ 1.85	($-$2.73; $-$1.04)
Chronicity (${\theta }_2$)	$-$ 0.13	$-$ 1.39	$-$ 0.44	$-$ 1.80	($-$2.68; $-$1.12)
Time		12 min		80 min

Note. MCMC = Monte Carlo Markov chain; VB = variational Bayes.

For the linear model, we estimate the weight functions $w_j$, $j=1,\dots ,14$, which gives the effect to what extent each resting state EEG alpha and theta power in the posterior region of brain under a closed eyes condition could help identify a potential early responders subgroup (which is believed to consist of subjects susceptible to non-specific placebo effects). To understand the variability in the estimates of the weight functions, we use functional boxplots which are the analogue to usual boxplots, and they are a graphical method to display five descriptive statistics: the median, the first and third quartiles, and the non-outlying minimum and maximum observations. The functional boxplot proposed by Sun and Genton (2011) is similar in spirit to the classical ‘box-and-whisker plot’ and displays the functional median, the envelope of the 50% central region and the maximum envelope based on the ranks to functional data induced by the modified band depth proposed by López-Pintado and Romo (2009). Given the ranks, it is possible to order the functional data according to decreasing depth values. This ordering is taken from the centre outwards, therefore the functional observation with the largest depth value has the most central position and it is called the functional median. The sample 50% central region is the set $\{(t,y(t)):y(t)$ is sandwiched between the minimum of half of the ranked observations with larger depths and the maximum of half of the ranked observations with smaller depths $\}$. Notice that the functional median is one of the curves whereas the 50% central region consists of pieces from different observations and it is not one of the original observations in general. The maximum envelope is constructed after removing outliers and determined by inflating the envelope of the 50% central region by 1.5 times its range for each t.

In the literature, depression is associated with impaired right parietal cortex function, see Bruder et al. (1995) and Heller and Nitscke (1997) and references therein. In particular, Stewart et al. (2011) suggest using asymmetry scores to measure parietal brain asymmetry, in particular, comparing $P_5$ (right) with $P_6$ (left) and $P_7$ (right) with $P_8$ (left). Figure 5 displays functional boxplots for the weight functions $w_j(t);j=3,5,\text{and}6$. These weight functions were found to be the only significant ones for predicting early responders using a linear model with the logit link and a normal prior. They are associated with the EEGs located at positions $P_5$, $P_6$, and $P_7$ aligning with the observation mentioned earlier. Notably, higher EEG signals in the frequency range of 20–45Hz for $P_5$, 30–45Hz for $P_6$, and 16–35Hz for $P_7$ correspond to an increased probability of being an early responder.

Figure 5.

Functional boxplots for the significant weight functions using the linear model with logit link function and normal prior. The shaded area is the 50% central region, and red dashed curves are outlier candidates.

5. Predicting illness for milking cows based on functional covariates measured 30 days before lactation

In this section, we will analyse the case where the observed sample is given by $Y_1,Y_2,\dots ,Y_n$, independent non-negative integer-valued random variables, such that

$$P(Y_i=y\mid {\lambda }_1,{\lambda }_2,{\mathit{\gamma }}_i)=[I(y_i=0){]}^{{\gamma }_{i0}}{\left[\frac{1}{y_i!}{e}^{-{\lambda }_1}{\lambda }_1^{y_i}\right]}^{{\gamma }_{i1}}{\left[\frac{1}{y_i!}{e}^{-{\lambda }_2}{\lambda }_2^{y_i}\right]}^{{\gamma }_{i2}}$$ (20)

for $y_i=0,1,\dots$, where ${\mathit{\gamma }}_i=({\gamma }_{i0},{\gamma }_{i1},{\gamma }_{i2})$ are latent multinomial random variables Multinomial $(1,p_{i0},p_{i1},p_{i2})$ with

$$p_{i0}=\frac{1}{1+\text{exp}({\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_1)+\text{exp}({\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_2)},p_{il}=\frac{\text{exp}({\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_l)}{1+\text{exp}({\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_1)+\text{exp}({\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_2)},l=1,2$$

Therefore, the vector of unknowns is $\mathit{\nu }=({\lambda }_1,{\lambda }_2,{\mathit{\beta }}_1^{\mathrm{\top }},{\mathit{\beta }}_2^{\mathrm{\top }},{\mathit{\gamma }}_1^{\mathrm{\top }},{\mathit{\gamma }}_2^{\mathrm{\top }}{)}^{\mathrm{\top }}=({\mathbf{\Theta }}^{\mathrm{\top }},{\mathit{\gamma }}^{\mathrm{\top }}{)}^{\mathrm{\top }}$. Here, ${\mathbf{x}}_i$ is row i of matrix of covariates.

For the model parameters, we propose using the following priors:

• ${\lambda }_1\sim \text{Gamma}(a_1,b_1)$ , ${\lambda }_2\sim \text{Gamma}(a_2,b_2)$.

• Elements in ${\mathit{\beta }}_l=({\mathit{\theta }}_l^{\mathrm{\top }},{\mathit{\varphi }}_{1l}^{\mathrm{\top }},\dots ,{\mathit{\varphi }}_{jl}^{\mathrm{\top }}{)}^{\mathrm{\top }}$ will have independent t prior distributions.

The complete likelihood is given by

$$f(\mathbf{y},\mathit{\gamma }\mid \mathbf{\Theta })={\displaystyle \prod _{i=1}^n}{[p_{i0}I(y_i=0)]}^{{\gamma }_{i0}}{\left[p_{i1}\frac{1}{y_i!}{e}^{-{\lambda }_1}{\lambda }_1^{y_i}\right]}^{{\gamma }_{i1}}{\left[p_{i2}\frac{1}{y_i!}{e}^{{\lambda }_2}{\lambda }_2^{y_i}\right]}^{{\gamma }_{i2}}$$ (21)

The joint augmented posterior distribution is the product of the likelihood and priors specified above and has no closed form. Therefore, we adapt the Gibbs sampling algorithm to sample from the posterior distribution of $\mathbf{\Theta }$ and the latent variables $\mathit{\gamma }$.

5.1. Full conditional posterior distributions

The posterior distribution of parameters is obtained based on a Gibbs sampling scheme. For that, we calculate the full conditional posterior distribution of parameters in $\mathbf{\Theta }$, similarly to the one described in Section 4.1.

5.1.1. Full conditional posterior distribution of ${\gamma }_i$

Let ${\mathit{\gamma }}_{-i0}$, ${\mathit{\gamma }}_{-i1}$, and ${\mathit{\gamma }}_{-i2}$ be the vector ${\mathit{\gamma }}_l=({\mathit{\gamma }}_{1l},\dots ,{\mathit{\gamma }}_{nl}{)}^{\mathrm{\top }}$ without observation ${\mathit{\gamma }}_{il}$, $l=0,1,2$, respectively. The full conditional posterior distribution of ${\mathit{\gamma }}_i$ for $l=1,2$ is given by

$$P({\gamma }_{il}=1\mid \mathbf{\Theta },\mathbf{y},{\mathit{\gamma }}_{-il})=\frac{e^{-{\lambda }_l}{\lambda }_l^{y_i}\text{exp}({\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_l)}{I(y_i=0)+e^{-{\lambda }_1}{\lambda }_1^{y_i}\text{exp}({\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_1)+e^{-{\lambda }_2}{\lambda }_2^{y_i}\text{exp}({\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_2)}$$ (22)

for $l=1,2$ and

$$P({\gamma }_{i0}=1\mid \mathbf{\Theta },\mathbf{y},{\mathit{\gamma }}_{-i0})=\frac{I(y_i=0)}{I(y_i=0)+e^{-{\lambda }_1}{\lambda }_1^{y_i}\text{exp}({\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_1)+e^{-{\lambda }_2}{\lambda }_2^{y_i}\text{exp}({\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_2)}.$$ (23)

5.1.2. Full conditional posterior distribution of ${\lambda }_1$ and ${\lambda }_2$

We update ${\lambda }_l,l=1,2$, using a gamma distribution with parameters

$$a_l+{\displaystyle \sum _{i=1}^n}{y}_i{\gamma }_{il}\text{and}{b}_l+{\displaystyle \sum _{i=1}^n}{\gamma }_{il},l=1,2.$$

5.1.3. Full conditional posterior distribution of ${\mathit{\beta }}_l=({\mathit{\theta }}_l^{\mathrm{\top }},{\mathit{\varphi }}_{1l}^{\mathrm{\top }},\dots ,{\mathit{\varphi }}_{jl}^{\mathrm{\top }}{)}^{\mathrm{\top }},l=1,2$

Computations of the full conditional posterior distribution ${\mathit{\beta }}_l$ follow from Section 3.2.1.

5.2. VB of the ZIMP model

Analogously to the normal mixture case, we define the variational densities as

$$q^{*}(\mathit{\nu }\mid \mathit{\kappa })=q^{*}({\lambda }_1\mid \mathit{\kappa })q^{*}({\lambda }_2\mid \mathit{\kappa })q({\mathit{\beta }}_1\mid \mathit{\kappa })q({\mathit{\beta }}_2\mid \mathit{\kappa })q(\gamma \mid \mathit{\kappa })$$

where $\mathit{\kappa }=({\mathit{\alpha }}^{\mathrm{\top }},{\psi }_1,{\zeta }_1,{\psi }_2,{\zeta }_2,({\mathit{\mu }}_{{\mathit{\beta }}_1}^{*}{)}^{\mathrm{\top }},({\mathit{\mu }}_{{\mathit{\beta }}_2}^{*}{)}^{\mathrm{\top }},vec({\mathbf{V}}_{{\mathit{\beta }}_1}),vec({\mathbf{V}}_{{\mathit{\beta }}_2}){)}^{\mathrm{\top }}$ is the vector of variational parameters. For all the cases, the variational densities $q^{*}({\mathit{\beta }}_1\mid \mathit{\kappa })$ and $q^{*}({\mathit{\beta }}_2\mid \mathit{\kappa })$ will have exactly the same computations as in the mixture of two normal distributions models, see Section 4.2.4. The vector of unknowns is $\mathit{\nu }=({\lambda }_1,{\lambda }_2,{\mathit{\beta }}_1^{\mathrm{\top }},{\mathit{\beta }}_2^{\mathrm{\top }},{\mathit{\gamma }}^{\mathrm{\top }}{)}^{\mathrm{\top }}=({\mathbf{\Theta }}^{\mathrm{\top }},{\mathit{\gamma }}^{\mathrm{\top }}{)}^{\mathrm{\top }},$ where $\mathit{\gamma }=({\mathit{\gamma }}_1,\dots ,{\mathit{\gamma }}_n{)}^{\mathrm{\top }}$.

Again, to simplify the notation, we will omit the dependence on $\mathit{\kappa }$ when writing the variational distributions $q^{*}$. The details of computations can be found in the online supplementary material.

5.2.1. Variational density $q^{*}({\mathit{\gamma }}_i)$

If we consider $q^{*}({\lambda }_1)$ and $q^{*}({\lambda }_2)$ belonging to the gamma family of distributions with parameters $({\psi }_1,{\zeta }_1)$ and $({\psi }_2,{\zeta }_2)$, respectively, we get ${\mathit{\gamma }}_i$ as a multinomial random variable Multinomial $(1,{\alpha }_{i0},{\alpha }_{i1},{\alpha }_{i2})$ with ${\alpha }_{il}={\rho }_{il}/(\sum _{j=0}^2{\rho }_{ij})$, for $l=0,1,2$, where

$$\begin{array}{rl}{\rho }_{i0}& =I(y_i=0)\\ {\rho }_{il}& =\text{exp}(-\frac{{\psi }_l}{{\zeta }_l}+y_i(-\log ({\zeta }_l)+\mathrm{\Psi }({\psi }_l))+E_{q^{*}({\mathit{\beta }}_l)}[{\mathbf{x}}_i^{\mathrm{\top }}{\mathit{\beta }}_l])\text{for}l=1,2\end{array}$$

5.2.2. Variational density $q^{*}({\lambda }_1)$ and $q^{*}({\lambda }_2)$

The variational distribution of ${\lambda }_l,l=1,2$, is gamma density with parameters

$${\psi }_l:=a_l+\left({\displaystyle \sum _{i=1}^n}{\alpha }_{il}{y}_i\right)\text{and}{\zeta }_l:=b_l+{\displaystyle \sum _{i=1}^n}{\alpha }_{il}$$

5.3. Results

For the MCMC, we ran two chains for 150,000 steps. We calculated Gelman and Rubin’s convergence diagnostic (Gelman & Rubin, 1992) using the R package coda and the statistics and plots showed convergence for all parameters leading to a burn-in of 75,000 steps and samples were taken every 100 steps, resulting into 750 samples. After fitting the model, Table 5 shows the classification using the posterior mean for the latent gamma variable (the classification according to the maximum value of the estimated probability is the same using MCMC and VB estimates). From the 138 cows in the training set that did not get sick during lactation, 136 were classified as belonging to the pure-zero class with high probability while the other 2 cows were classified as low incidence. All 43 cows that were sick between 1 and 4 days were classified as low incidence whereas only one of the cows diagnosed with a health disorder for 5 days was classified as low incidence and the other one was classified as high incidence. All cows with 6 or more days with a health disorder were classified as high incidence.

Table 5.

Classification for the training set according to the class presenting the maximum mean value of the posterior distribution and VB estimation of the latent variable γ

Class	Number of sick days
	0	1	2	3	4	5	6	7	8	9	11	12	13	18
Pure zero	136	0	0	0	0	0	0	0	0	0	0	0	0	0
Low incidence	2	20	10	6	5	1	0	0	0	0	0	0	0	0
High incidence	0	0	0	0	0	1	5	4	6	5	2	3	1	1

Note. VB = variational Bayes.

The posterior mean and standard deviation and quantiles of the posterior distribution as well as the mean for the VB mean value for these classes are shown in Table 6 and the left panel of Figure 6. Notice that the estimates of the mean and standard deviation of the posterior distribution for ${\lambda }_1$ and ${\lambda }_2$ are very similar using VB and MCMC, with the VB method presenting smaller standard deviations. However, the quantiles from the MCMC method indicate that the $95\mathrm{\%}$ credible intervals of ${\lambda }_1$ (1.20;2.13) and ${\lambda }_2$ (7.53;9.83) clearly discriminate the classes. On the other hand, from Figure 6, we can see that the VB estimates for the mean of the intercept and scalar coefficients for variables are not so good for the coefficients that are not significant whereas they are similar for the ones which are significant. Direct comparison for the coefficients of the expansion for the weights of the functional covariates are not meaningful and we constructed the functional boxplot for the posterior estimates of the weight functions as shown in Figure 7. To understand these plots, recall that the functional boxplots are similar to the usual boxplots, where the shaded region comprises the 50% central functional observations and can be thought as 50% credible intervals. Notice that apparently, none of the functional covariates are significant for the regression term for low incidence class since the horizontal line at zero is almost completely contained inside the shaded regions. The covariate activity seems to be significant to determine the probabilities of the high incidence latent class; if the cow has high activity between 30 and 20 days prior to labour, it impacts negatively in the probability whereas if the cow has high activity between 20 and 5 days prior to labour, it increases the probability of belonging to the high intensity class.

Table 6.

VB mean, posterior mean, and quantiles for the parameters according to MCMC sample for logit link and Cauchy prior for zero-inflated mixture of two Poisson distributions

			Quantiles—MCMC
Parameter	VB mean (SD)	MCMC mean	2.5%	25%	50%	75%	97.5%
${\lambda }_1$	1.60 (0.18)	1.63 (0.24)	1.20	1.47	1.62	1.79	2.13
${\lambda }_2$	8.45 (0.53)	8.64 (0.59)	7.53	8.24	8.63	9.03	9.83
Time	20.35 s	2.55 h

Note. VB = variational Bayes; MCM = Monte Carlo Markov chain.

Figure 6.

Histogram for posterior distribution of ${\lambda }_1$ and ${\lambda }_2$. Intercepts and coefficients related to the scalar variables (${\theta }_1$, ${\theta }_2$, ${\theta }_3$, and ${\theta }_4$) for the ZIMP model. ZIMP = zero-inflated mixture Poisson.

Figure 7.

Functional boxplots for the weight functions corresponding to covariates ‘activity’, ‘number of rest times’, ‘average rest time’, and ‘total rest time’, respectively, under linear model with logit link function and Cauchy prior. Left panels corresponds to class 1 (low intensity) and right panels to class 2 (high intensity).

Figure 8 shows as a stacked barplot the estimated probability of belonging to the pure zero, low and high incidence class $({\hat{p}}_{i0},{\hat{p}}_{i1},{\hat{p}}_{i2})$ for the 50 cows in the test set. The left panel presents the 37 cows that did not get sick whereas the right panel displays the 13 cows that were sick at least for a day. It is important to note that the current model does not exhibit strong predictive power. There are various prepartum risk factors linked to postpartum diseases. According to studies by Urton et al. (2005) and Huzzey et al. (2007), reduced feeding times and dry matter intake before parturition serve as predictors for metritis. Additionally, Riekerink et al. (2007) report a seasonal effect associated with mastitis. Further research by Pérez-Báez et al. (2019a, 2019b) explores the connection between prepartum dry matter intake, energy balance, and subsequent postpartum health disorders. Consequently, in future investigations, it is advisable to incorporate some of these factors as additional covariates for a more comprehensive understanding.

Figure 8.

Estimated probability of belonging to the pure zero, low and high incidence class $({\hat{p}}_{i0},{\hat{p}}_{i1},{\hat{p}}_{i2}),i=1,\dots 50$, for the cows in the test set using the ZIMP model. The left panel represents the cows that did not get sick. The numbers under the bars in the right panel represent the number of sick days. ZIMP = zero-inflated mixture Poisson.

6. Discussion

In this paper, we consider two applications where the heart of the problem was to classify the individuals based solely on the covariates. We proposed a classification procedure based on a mixture model driven by latent variables under the Bayesian paradigm. We used a semi-parametric regression model incorporating functional covariates as predictors for the latent group membership. Other methods could be used to analyse the data such as regression trees or regression mixture models. However, all methods have to be adapted to deal with functional data. In the case of regression trees, one approach to include the functional covariates would be discretizing the functional predictor by partitioning their domains into several (disjoint) intervals and use the mean values computed from each interval as covariates. However, again we return to the approach of treating functional data as multivariate vectors. Using regression mixture models would almost double the number of parameters to be estimated. On the other hand, Bayesian analysis not only provides information on the whole distribution of the parameters, going beyond a point or an interval estimate, but also it simplifies the task of estimating latent models since we can augment the parameter space. In this work, we used Gibbs sampler due to the conjugacy of the priors. Other choices would be possible, e.g. the Metropolis-Hastings algorithm. Although VB approach leads to faster computational time, we see from simulations that MCMC produces more reliable estimates. Also, computational times are hard to compare since they depend on chain size and precision in MCMC and VB approaches, respectively.

The main features of our methodology are:

1. The nonparametric approach of expanding the unknown functions $w_j$ and $F_j$ into B-splines basis reduces the dimension of the problem.

2. Linear and nonlinear regressions can be implemented.

3. It can be used with any link function g.

4. It incorporates Student-t prior information for the regression coefficients applying an efficient approximate EM algorithm already implemented in R.

5. It can be used with any distributions.

6. The functional covariates do not need to be observed concurrently and they can even be different for each subject.

7. In the case of the linear model, it has the added advantage of interpretability of the weight functions which might naturally incorporate prior information that is available to experts in the field.

The strength of our method is demonstrated by the simulation study as well as the analysis of the two data sets.

Funding

This work was partially supported by NIMH grant 5 R01 MH099003, USDA National Institute of Food and Agriculture Animal Health Program award 2017-67015-26772 to J.O.G., FAPESP grants 2017/15306-9, 2018/06811-4, 2019/10800-0, and 2023/00592-7, and CNPq grants 302598/2014-6, 442012/2014-4, and 304148/2020-2.

Data availability

The code and data files used in the real data applications and simulation studies are available at https://github.com/nancyg-unicamp/Unsupervised.

Supplementary material

Supplementary material is available online at Journal of the Royal Statistical Society: Series C. As a complement to this paper we provide a file with extensive simulation study to investigate the performance of the proposed method as well as acomparation between MCMC and Variational Bayes Inference.