Bayesian Semiparametric Multivariate Density Deconvolution

Abstract

We consider the problem of multivariate density deconvolution when interest lies in estimating the distribution of a vector valued random variable X but precise measurements on X are not available, observations being contaminated by measurement errors U. The existing sparse literature on the problem assumes the density of the measurement errors to be completely known. We propose robust Bayesian semiparametric multivariate deconvolution approaches when the measurement error density of U is not known but replicated proxies are available for at least some individuals. Additionally, we allow the variability of U to depend on the associated unobserved values of X through unknown relationships, which also automatically includes the case of multivariate multiplicative measurement errors. Basic properties of finite mixture models, multivariate normal kernels and exchangeable priors are exploited in novel ways to meet modeling and computational challenges. Theoretical results showing the flexibility of the proposed methods in capturing a wide variety of data generating processes are provided. We illustrate the efficiency of the proposed methods in recovering the density of X through simulation experiments. The methodology is applied to estimate the joint consumption pattern of different dietary components from contaminated 24 hour recalls. Supplementary Material presents substantive additional details.

1 Introduction

Many problems of practical importance require estimation of the density f_X of a vector valued random variable X. Precise measurements on X may not, however, be available, observations being contaminated by measurement errors U. Under the assumption of additive measurement errors, the observations are generated from a convolution of the density f_X of X and the density f_U of the measurement errors U. The problem of estimating the density f_X from available contaminated measurements then becomes a problem of multivariate density deconvolution.

This article proposes novel Bayesian semiparametric density deconvolution approaches based on finite mixtures of latent factor analyzers for robust estimation of the density f_X when the measurement error density f_U is not known, but replicated proxies contaminated with measurement errors U are available for at least some individuals. The proposed deconvolution approaches are highly robust, not having to impose restrictive parametric assumptions on f_X or f_U. Additionally, the variability of U is allowed to depend on the associated unobserved values of X through unknown relationships.

While the focus of the article will primarily be on additive measurement errors, importantly, the methodology for additive conditionally heteroscedastic measurement errors developed here also automatically encompasses the case of multivariate multiplicative measurement errors.

To the best of our knowledge, all existing multivariate deconvolution approaches assume that U is independent of X and that the error density f_U is completely known. Ours is thus the first paper that allows the density of the measurement errors to be unknown and free from parametric laws and additionally also accommodates conditional heteroscedasticity in the measurement errors.

The literature on the problem of univariate density deconvolution, in which context we denote the variable of interest by X and the measurement errors by U, is vast. Most of the early literature considered scenarios when the measurement error density f_U is completely known. Fourier inversion based deconvoluting kernel density estimators have been studied by Carroll and Hall (1988), Liu and Taylor (1989), Devroye (1989), Fan (1991a, 1991b, 1992) and Hesse (1999) among many others. For a review of these methods, the reader may be referred to Section 12.1 in Carroll, et al. (2006) and Section 10.2.3 in Buonaccorsi (2010). In reality f_U is rarely known. The problem of deconvolution when the errors are homoscedastic with an unknown density and replicated proxies are available for each subject has been addressed by Li and Vuong (1998). See also Diggle and Hall (1993), Neumann (1997), Carroll and Hall (2004) and the references therein. The assumptions of homoscedasticity of U and their independence from X are also often unrealistic. Flexible Bayesian density deconvolution approaches that allow U to be conditionally heteroscedastic have recently been developed in Staudenmayer, et al. (2008) and Sarkar, et al. (2014). Staudenmayer, et al. (2008) assumed the measurement errors to be normally distributed and used finite mixtures of B-splines to estimate f_X and a variance function that captured the conditional heteroscedasticity. Sarkar, et al. (2014) further relaxed the assumption of normality of U employing flexible infinite mixtures of normal kernels induced by Dirichlet processes to estimate both f_X and f_U. Sieve based methods developed in Schennach (2004) and Hu and Schennach (2008) can also handle conditional heteroscedasticity.

In sharp contrast to the univariate case, the literature on multivariate density deconvolution is quite sparse. We can only mention Masry (1991), Youndjé and Wells (2008), Comte and Lacour (2013), Hazelton and Turlach (2009, 2010) and Bovy, et al. (2011). The first three considered deconvoluting kernel based approaches assuming the measurement errors U to be distributed independently from X according to a known probability law. Hazelton and Turlach (2009, 2011), working with the same assumptions on U, proposed weighted kernel based methods. Bovy, et al. (2011) modeled the density f_X using flexible mixtures of multivariate normal kernels, but they assumed f_U to be multivariate normal with known covariance matrices, independent from X. As in the case of univariate problems, the assumptions of a fully specified f_U, known covariance matrices, and independence from X are highly restrictive for most practical applications.

The focus of this article is on multivariate density deconvolution when f_U is not known but replicated proxies are available for at least some individuals. The proposed deconvolution approaches can additionally accommodate conditional heteroscedasticity in U. The problem is important, for instance, in nutritional epidemiology, where nutritionists are typically interested not just in the consumption behaviors of individual dietary components but also in their joint consumption patterns. The data are often available in the form of dietary recalls and are contaminated by measurement errors that show strong patterns of conditional heteroscedasticity.

As in Sarkar, et al. (2014), we use mixture models to estimate both f_X and f_U but the multivariate nature of the problem brings in new modeling challenges and computational obstacles that preclude straightforward extension of their univariate deconvolution approaches. Instead of using infinite mixtures induced by Dirichlet processes, we use finite mixtures of multivariate normal kernels with exchangeable Dirichlet priors on the mixture probabilities. The use of finite mixtures and exchangeable priors greatly reduces computational complexity while retaining essentially the same flexibility. Carefully constructed priors also allow automatic model selection and model averaging. To save space, detailed discussions on these important issues are moved to Section S.6 in the Supplementary Material.

We also exploit symmetric Dirichlet priors and properties of multivariate normal distributions and finite mixture models to develop a novel strategy that enables us to enforce a required zero mean restriction on the measurement errors. Our proposed technique, as opposed to the one adopted by Sarkar, et al. (2014), is particularly suitable for high dimensional applications and can be easily generalized to enforce moment restrictions on other types of finite mixture models.

It is well known that inverse Wishart priors, due to their dense parametrization, are not suitable for modeling covariance matrices in high dimensional applications. In deconvolution problems the issue is further complicated since X and U are both latent. This results in numerically unstable estimates even for small and moderate dimensions, particularly when the true covariance matrices are sparse and the likelihood function is of complicated form. To reduce the effective number of parameters required to be estimated, we consider factor-analytic representation of the component specific covariance matrices with sparsity inducing shrinkage priors on the factor loading matrices.

Models for multivariate regression errors that assume normality but allow the covariance matrix to vary flexibly with associated precisely measured and possibly multivariate predictors have recently been developed in the literature (Hoff and Niu, 2012; Fox and Dunson, 2016, etc.). Unlike regression settings, exclusive relationships exist between different components of multivariate measurement errors U and different components of the associated multivariate latent ‘predictor’ X - the ℓ^th component U_ℓ of U contaminates only the ℓ^th component X_ℓ of X but not others. We thus deem covariance regression models that allow cov(U|X) to vary arbitrarily with all components of X to be inappropriate in multivariate measurement error settings. As discussed above, the assumption of multivariate normality is also particularly restrictive in measurement error problems. In this article, we develop a semiparametric approach that appropriately highlights the exclusive associations between U_ℓ and X_ℓ while allowing the distribution of (U|X) to depart from normality. Importantly, the model also arises naturally from multivariate multiplicative measurement error settings, automatically encompassing such cases. Diagnostic tools for checking model adequacy are also discussed.

The likelihood function for the conditional heteroscedastic model poses significant computational challenges. We overcome these obstacles by designing a novel two-stage procedure that exploits the unique properties of conditionally heteroscedastic multivariate measurement errors to our advantage. The procedure first estimates the variance functions characterizing var(U_ℓ|X_ℓ) using reparametrized versions of the corresponding univariate submodels. The estimates obtained in the first stage are then plugged-in to estimate the remaining parameters in the second stage. Having two estimation stages, our deconvolution method for conditionally heteroscedastic measurement errors is not purely Bayesian. But they show good empirical performance and, with no other solution available in the existing literature, they provide at least workable starting points towards more sophisticated methodology.

The article is organized as follows. Section 2 details the models. Model identifiability issues and implementation details, including the choice of hyper-parameters and Markov chain Monte Carlo (MCMC) algorithms to sample from the posterior, are discussed in the Supplementary Material. Section 4 discusses model identifiability issues. Section 5 presents theoretical results showing flexibility of the proposed models. Simulation studies comparing the proposed deconvolution methods to a naive method that ignores measurement errors are presented in Section 6. Section 7 presents an application of the proposed methodology in estimation of the joint consumption pattern of dietary intakes from contaminated 24 hour recalls in a nutritional epidemiologic study. Section 8 includes a discussion. An unnumbered section concludes the article with a description of the Supplementary Material.

2 Deconvolution Models

The goal is to estimate the unknown joint density of a p-dimensional multivariate random variable X. There are i = 1,…, n subjects. Precise measurements of X are not available. Instead, for j = 1,…, m_i, replicated proxies W_ij contaminated with measurement errors U_ij are available for each subject i. The replicates are assumed to be generated by the model

$${\mathbf{W}}_{ij}={\mathbf{X}}_i+{\mathbf{U}}_{ij}.$$ (1)

Given X_i, U_ij are independently distributed with E(U_ij|X_i) = 0. The marginal density of W_ij is denoted by f_W. The implied conditional distributions of W_ij and U_ij, given X_i, are denoted by f_W|X and f_U|X, respectively.

2.1 Modeling the Density f_X

In this article f_X is specified as a mixture of multivariate normal kernels

$$f_{\mathbf{X}}(\mathbf{X})={\sum }_{k=1}^{K_{\mathbf{X}}}{\pi }_{\mathbf{X},k}{\text{MVN}}_p(\mathbf{X}|{\mathit{\mu }}_{\mathbf{X},k},{\mathbf{\sum }}_{\mathbf{X},k}),$$ (2)

where MVN_p(·|μ, Σ) denotes a p-dimensional multivariate normal density with mean μ and covariance matrix Σ. For the rest of this subsection, the subscript X is kept implicit to keep the notation clean.

We assign a finite Dirichlet prior to the mixture probability vector π = (π₁,…, π_K)^T as

$$\mathbf{\pi }~\text{Dir}(\alpha /K,\dots ,\alpha /K).$$ (3)

Here Dir(α₁,…, α_K) denotes a finite dimensional Dirichlet distribution on the K-dimensional unit simplex with concentration parameter (α₁,…, α_K). Given K and the latent cluster membership indices, the prior is conjugate. The symmetry of the assumed Dirichlet prior helps in additional reduction of computational complexity by simplifying MCMC mixing issues. Provided K is sufficiently large, a carefully chosen α can impart the posterior with certain properties that simplify model selection and model averaging issues by influencing the posterior to concentrate in regions that favor empty redundant components, see Section S.1 and Section S.6 of the Supplementary Material. We assign conjugate multivariate normal priors to the component specific mean vectors μ_k, so that

$${\mathit{\mu }}_k~{\text{MVN}}_p({\mathit{\mu }}_0,{\mathbf{\sum }}_0).$$ (4)

The conjugacy again helps in simplifying posterior calculations. Later on, we will employ similar mixture models for the density of the measurement errors, and this conjugacy, along with some basic properties of multivariate normal kernels, will also help us enforce the mean zero restriction on the measurement errors. For the component specific covariance matrices Σ_k, we first consider conjugate inverse Wishart priors

$${\mathbf{\sum }}_k~{\text{IW}}_p({\nu }_0,{\mathbf{\Psi }}_0).$$ (5)

Here IW_p(ν, Ψ) denotes an inverse Wishart density on the space of p × p positive definite matrices with mean Ψ/(ν -p-1). While the conjugacy of the inverse Wishart priors helps in simplifying posterior calculations, in complex high dimensional problems its dense parameterization may result in numerically unstable estimates, particularly when the covariance matrices are sparse. In a deconvolution problem the issue is compounded further by the nonavailability of the true X_i’s. To reduce the effective number of parameters to be estimated, we consider a parsimonious factor-analytic representation of the component specific covariance matrices:

$${\mathbf{\sum }}_k={\mathbf{\Lambda }}_k{\mathbf{\Lambda }}_k^{\mathrm{T}}+\mathbf{\Omega },$$ (6)

where Λ_k are p × q_k factor loading matrices and Ω is a diagonal matrix with non-negative entries. In practical applications q_k will typically be much smaller than p, inducing parsimonious characterizations of the unknown covariance matrices Σ_k. Model (2) can be equivalently represented as

$$\text{Pr}(C_i=k)={\pi }_k,$$ (7)

$$({\mathbf{X}}_i|C_i=k)={\mathit{\mu }}_k+{\mathbf{\Lambda }}_k{\mathbf{\eta }}_i+{\mathbf{\Delta }}_i,$$ (8)

$${\mathbf{\eta }}_i~{\text{MVN}}_p(\mathbf{0},{\mathbf{I}}_p),{\mathbf{\Delta }}_i~{\text{MVN}}_p(\mathbf{0},\mathbf{\Omega }),$$ (9)

where C_i are the mixture labels associated with X_i, η_i are latent factors, and Δ_i are errors with covariance Ω = diag( ${\sigma }_1^2,\dots ,{\sigma }_p^2$).

The above characterization of Σ_k is not unique, since for any semi-orthogonal matrix P the loading matrix ${\mathbf{\Lambda }}_k^1={\mathbf{\Lambda }}_k\mathbf{P}$ also satisfies (6). Since interest lies primarily in estimating the density f_X, identifiability of the latent factors is, however, not required. This also allows the loading matrices to have a-priori a potentially infinite number of columns. Sparsity inducing priors, that favor more shrinkage as the column index increases, can then be used to shrink the redundant columns towards zero. In this article, we do this by adapting the shrinkage priors proposed in Bhattacharya and Dunson (2011) that allow easy posterior computation. Let ${\mathbf{\Lambda }}_k={(({\lambda }_{k,jh}))}_{j=1,h=1}^{p,\infty }$, where j and h denote the row and the column indices, respectively. For h = 1,…, ∞, we assign priors as follows

$${\lambda }_{k,jh}~\text{Normal}(0,{\varphi }_{k,jh}^{-1}{\tau }_{k,h}^{-1}),{\varphi }_{k,jh}~\text{Ga}(\nu /2,\nu /2),$$ (10)

$${\tau }_{k,h}~{\mathrm{\Pi }}_{\ell =1}^h{\delta }_{k,\ell },{\delta }_{k,\ell }~\text{Ga}(a_{\ell },1),{\sigma }_j^2~\text{Inv-Ga}(a_{\sigma },b_{\sigma }).$$ (11)

Here Ga(α,β) denotes a Gamma distribution with shape parameter α and rate parameter β and IG(a, b) denotes an inverse-Gamma distribution with shape parameter a and scale parameter b. In the k^th component factor loading matrix Λ_k, the parameters ${\{{\varphi }_{k,jh}\}}_{j=1}^p$ control the local shrinkage of the elements in the h^th column, whereas τ_k,h controls the global shrinkage. When a_h > 1 for h = 2,…, ∞, the sequence ${\{{\tau }_{k,h}\}}_{h=1}^{\infty }$ becomes stochastically increasing and thus favors more shrinkage as the column index h increases.

In addition to inducing adaptive sparsity and hence numerical stability, by favoring more shrinkage as the column index increases, the shrinkage priors play another important role in making the proposed factor analytic model highly robust to misspecification of the number of latent factors, allowing us to adopt simple strategies to determine the number of latent factors to be included in the model in practice. Details are deferred to Section S.1 in the Supplementary Material.

Throughout the rest of the paper, mixtures with inverse Wishart prior on the covariance matrices will be referred to as MIW models and mixtures of latent factor analyzers will be referred to as MLFA models.

For a review of finite mixture models and mixtures of latent factor analyzers, without moment restrictions or sparsity inducing priors and with applications in measurement error free scenarios, see Fokoué and Titterington (2003), Frühwirth-Schnatter (2006), Mengersen, et al. (2011) and the references therein. For other types of shrinkage priors, see Brown and Griffin (2010), Carvalho, et al. (2010), Bhattacharya, et al. (2014) etc.

2.2 Modeling the Density of the Measurement Errors

2.2.1 Independently Distributed Measurement Errors

In this section, we develop models for the measurement errors U assuming them to be independent from X. That is, we assume f_U|X = f_U for all X. This remains the most extensively researched deconvolution problem for both univariate and multivariate cases. The techniques developed in this section will also provide crucial building blocks for more realistic models in Section 2.2.2. The measurement errors and their density are now denoted by ε_ij and f_ε, respectively, for reasons to become obvious shortly in Section 2.2.2.

As in Section 2.1, a mixture of multivariate normals can be used to model the density f_ε but the model now has to satisfy a mean zero constraint. That is

$$f_{\mathbf{\epsilon }}(\mathbf{\epsilon })={\sum }_{k=1}^{K_{\mathbf{\epsilon }}}{\pi }_{\mathbf{\epsilon },k}{\text{MVN}}_p(\mathbf{\epsilon }|{\mathbf{\mu }}_{\mathbf{\epsilon },k},{\mathrm{\sum }}_{\mathbf{\epsilon },k}),$$ (12)

$$\text{subject to }{\sum }_{k=1}^{K_{\mathbf{\epsilon }}}{\pi }_{\mathbf{\epsilon },k}{\mathbf{\mu }}_{\mathbf{\epsilon },k}=0.$$ (13)

To get numerically stable estimates of the density of the errors, latent factor characterization of the covariance matrices with sparsity inducing shrinkage priors as in Section 2.1 may again be used. Details are curtailed to avoid unnecessary repetition and we only present the mechanism to enforce the zero mean restriction on the model. The subscript ε is again dropped in favor of cleaner notation. In later sections, the subscripts X and ε reappear to distinguish between the parameters associated with f_X and f_ε, when necessary.

Without the mean restriction and under conjugate multivariate normal priors μ_k ~ MVN_p(μ₀, Σ₀), the posterior full conditional of ${\mathit{\mu }}^{K_{p\times 1}}={({\mathit{\mu }}_1^{\mathrm{T}},\dots ,{\mathit{\mu }}_K^{\mathrm{T}})}^{\mathrm{T}}$ is given by

$${\text{MVN}}_{Kp}\left\{\right(\begin{array}{c}\hfill {\mathit{\mu }}_1^0\hfill \\ \hfill {\mathit{\mu }}_2^0\hfill \\ \hfill ⋮\hfill \\ \hfill {\mathit{\mu }}_K^0\hfill \end{array}),(\begin{array}{cccc}\hfill {\sum }_1^0\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill {\sum }_2^0\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \hfill & \hfill ⋮\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill {\sum }_K^0\hfill \end{array}\left)\right\}\equiv {\text{MVN}}_{Kp}({\mathit{\mu }}^0,{\sum }^0),$$ (14)

where ε_ij and other conditioning variables are implicitly understood. Explicit expressions of μ⁰ and Σ⁰ in terms of the conditioning variables can be found in Section S.1 in the Supplementary Material. The posterior full conditional of μ under the mean restriction can then be obtained easily by further conditioning the distribution in (14) by ${\mathit{\mu }}_R={\sum }_{k=1}^K{\pi }_k{\mathit{\mu }}_k=0$ and is given by

$$(\mathit{\mu }|{\mathit{\mu }}_R=\mathbf{0})~{\text{MVN}}_{Kp}\{{\mathit{\mu }}^0-{\sum }_{1,R}^0{({\sum }_{R,R}^0)}^{-1}{\mathit{\mu }}_R^0,{\sum }^0-{\sum }_{1,R}^0{({\sum }_{R,R}^0)}^{-1}{\sum }_{R,1}^0\},$$ (15)

where and ${\mathit{\mu }}_R^0={\mathbf{\sum }}_{k=1}^K{\pi }_k{\mathit{\mu }}_k^0=E({\mathit{\mu }}_R),{\mathbf{\sum }}_{k,K}={\pi }_k{\mathbf{\sum }}_k^0=\text{cov}({\mathit{\mu }}_k,{\mathit{\mu }}_R),{\mathbf{\sum }}_{R,R}^0={\mathbf{\sum }}_{K+1,K+1}={\mathbf{\sum }}_{k=1}^K{\pi }_k^2{\mathbf{\sum }}_k^0=\text{cov}({\mathit{\mu }}_R)$ and ${\mathbf{\sum }}_{R,1}^0=({\mathbf{\sum }}_{1,K+1},{\mathbf{\sum }}_{2,K+1},\dots ,{\mathbf{\sum }}_{K,K+1})$. To sample from this singular density, we can first sample from the non-singular distribution of $\{{({\mathit{\mu }}_1^{\mathrm{T}},{\mathit{\mu }}_2^{\mathrm{T}},\dots ,{\mathit{\mu }}_{K-1}^{\mathrm{T}})}^{\mathrm{T}}|{\mathit{\mu }}_R=\mathbf{0}\},$, which can also be trivially obtained from (15), and then set ${\mathit{\mu }}_K=-{\mathbf{\sum }}_{k=1}^{K-1}{\pi }_k{\mathit{\mu }}_k/{\pi }_K.$.

2.2.2 Conditionally Heteroscedastic Measurement Errors

We now consider the case when the variances of the measurement errors depend on the associated unknown values of X through unknown relationships.

Interpreting the conditioning variables X broadly as predictors, one can loosely connect our problem of modeling conditionally heteroscedastic U to the problem of covariance regression (Hoff and Niu, 2012; Fox and Dunson, 2016, etc.), where the covariance of the multivariate regression errors are allowed to vary flexibly with precisely measured and possibly multivariate predictors. In such problems, the dimension of the regression errors is unrelated to the dimension of the predictors and different components of the regression errors are assumed to be equally influenced by different components of the predictors. In multivariate deconvolution problems, in contrast, the dimension of U_ij is exactly the same as the dimension of X_i, the ℓ^th component U_ijℓ being the measurement error associated exclusively with X_iℓ. See Figure 1. While different components of U_ij may be correlated, this exclusive association between U_ijℓ and X_iℓ implies that the dependence of U_ijℓ on X_i should be explained primarily through X_iℓ. Figure 7, for instance, suggests strong conditional heteroscedasticity patterns and it is plausible to assume that this conditional variability in U_ijℓ can be explained mostly through X_iℓ only. It is interesting to note these contrasts between conditionally varying regression and measurement errors become particularly prominent in the multivariate set up. Additionally, the aforementioned covariance regression approaches all assume multivariate normality of the regression errors. As discussed in the introduction, such strong parametric assumptions on the error distribution are particularly restrictive in measurement error problems. Additional detailed discussions of these important issues and resulting modeling implications can be found in Section S.5 of the Supplementary Material. They preclude direct application of existing covariance regression approaches to multivariate deconvolution problems but warrant models that can highlight the aforementioned unique dependence relationships, accommodate distributional flexibility while enforcing the mean zero restriction, and produce computationally stable estimates even in the absence of precise information on the conditioning variable X.

Figure 1.

Dependency structures in trivariate deconvolution problems with (a) independently distributed and (b) conditionally varying measurement errors. (c) Dependency structure in a trivariate regression problem with response Y, regression errors U and bivariate predictor X. The filled rectangular regions focus on the relationship between the (potentially conditionally varying) errors U and the (corresponding conditioning) variable X. The unfilled and the shaded nodes signify latent and observable variables, respectively. The directed and the undirected edges represent one and two-way relationships, respectively. The solid black and the dashed gray edges in panel (b) signify strong and weak dependencies, respectively.

Figure 7.

Estimated variance functions var(U|X) = s²(X)var(ε) produced by the univariate density deconvolution method for each component of X for the EATS data set with sample size n = 965, m_i = 4 replicates for each subject. See Section 7 for additional details. The figure is in color in the electronic version of this article.

The semiparametric approach that we adopt in this article achieves distributional flexibility, enforces the mean zero restriction, accommodates the exclusive relationships between U_ijℓ and X_iℓ but ignores the weak dependencies of U_ijℓ on {X_im}_m≠ℓ depicted in Figure 1(b). Specifically, we let

$$({\mathbf{U}}_{ij}|{\mathbf{X}}_i)=\mathbf{S}({\mathbf{X}}_i){\mathbf{\epsilon }}_{ij},$$ (16)

where S(X_i) = diag{s₁(X_i1), s₂(X_i2),…, s_p(X_ip)} and ε_ij, henceforth referred to as the ‘scaled errors’, are distributed independently of X_i. Model (16) implies that cov(U_ij|X_i) = S(X_i) cov(ε_ij) S(X_i) and marginally $\text{var}(U_{ij\ell }|{\mathbf{X}}_i)=s_{\ell }^2(X_{i\ell })\text{var}({\epsilon }_{ij\ell })$, a function of X_iℓ only. The techniques developed in Section 2.2.1 can now be employed to model the density of ε_ij, allowing different components of U_ij to be correlated and their joint density to deviate from multivariate normality.

We model the variance functions $S_{\ell }^2$, denoted also by v_ℓ, using positive mixtures of B-spline basis functions with smoothness inducing priors on the coefficients as in Staudenmayer, et al. (2008). For the ℓ^th component, partition an interval [A_ℓ, B_ℓ] of interest into L_ℓ subintervals using knot points A _ℓ= t_ℓ,1 = ⋯ = t_ℓ,q+1 < t_ℓ,q+2 < t_ℓ,q+3 < ⋯ < t_ℓ,q+Lk < t_ℓ,q+Lℓ+1 = ⋯ = t_ℓ,2q+L_ℓ+1 = B_ℓ. A flexible model for the variance functions is given by

$$v_{\ell }(X_{i\ell })=s_{\ell }^2(X_{i\ell })={\mathbf{\sum }}_{j=1}^{J_{\ell }}{b}_{q,j,\ell }(X_{i\ell })\text{exp}({\xi }_{j\ell })={\mathbf{B}}_{q,J_{\ell },\ell }(X_{i\ell })\text{exp}({\mathit{\xi }}_{\ell }),$$ (17)

$$({\mathit{\xi }}_{\ell }|J_{\ell },{\sigma }_{\xi ,\ell }^2)\propto {(2\pi {\sigma }_{\xi ,\ell }^2)}^{-J_{\ell }/2}\text{exp }\{-{\mathit{\xi }}_{\ell }^{\mathrm{T}}{P}_{\ell }{\mathit{\xi }}_{\ell }/(2{\sigma }_{\xi ,\ell }^2)\},{\sigma }_{\xi ,\ell }^2~\text{Inv-Ga}(a_{\xi },b_{\xi }).$$ (18)

Here ${\{b_{q,j,\ell }\}}_{j=1}^{J_{\ell }}$ denote J_ℓ = (q +L_ℓ) B-spline bases of degree q as defined in de Boor (2000), ξ_ℓ = {ξ_1ℓ, ξ_2ℓ,…, ξ_Jℓℓ}^T; exp(ξ_ℓ) = {exp(ξ_1ℓ), exp(ξ_2ℓ),…, exp(ξ_Jℓℓ)}^T; and $P_{\ell }=D_{\ell }^{\mathrm{T}}{D}_{\ell }$, where D_ℓ is a J_ℓ × (J_ℓ + 2) matrix such that D_ℓξ_ℓ computes the second differences in ξ_ℓ. The prior $P_0({\mathit{\xi }}_{\ell }|{\sigma }_{\xi ,\ell }^2)$ induces smoothness in the coefficients because it penalizes ${\sum }_{j=1}^{J_k}{({\mathrm{\Delta }}^2{\xi }_{j\ell })}^2={\mathit{\xi }}_{\ell }^{\mathrm{T}}{P}_{\ell }{\mathit{\xi }}_{\ell }$, the sum of squares of the second order differences in ξ_ℓ (Eilers and Marx, 1996). The parameters ${\sigma }_{\xi ,\ell }^2$ play the role of smoothing parameter - the smaller the value of ${\sigma }_{\xi ,\ell }^2$, the stronger the penalty and the smoother the variance function. The inverse-Gamma hyperpriors on ${\sigma }_{\xi ,\ell }^2$ allow the data to have influence on the posterior smoothness and make the approach data adaptive.

Since $s_{\ell }^2(X_{i\ell })\text{var}({\epsilon }_{ij\ell })=\{s_{\ell }^2(X_{i\ell })c\}\{\text{var}({\epsilon }_{ij\ell })/c\}$ for any c > 0, the variance functions $s_{\ell }^2\prime \mathrm{s}$ can not be uniquely determined without additional restrictions on var(ε_ijℓ). Separate identifiability of S and f_ε is, however, not required for inference on f_X or to assess the conditional variability in U_ijℓ. The latter, for instance, may simply be obtained as $\text{var}(U_{ij\ell }|X_i)=s_{\ell }^2(X_{i\ell })\text{var}({\epsilon }_{ij\ell })$. We thus avoid additional identifiability restrictions that would further compound modeling challenges. Adjustments made to the estimates of $s_{\ell }^2$ and f_ε to enable comparisons with the corresponding true values in simulation experiments are discussed in Section S.3 in the Supplementary Material.

2.2.3 Multiplicative Measurement Errors

In this section we consider the case of multivariate multiplicative measurement errors. The replicates are now assumed to be generated by the model

$${\mathbf{W}}_{ij}={\mathbf{X}}_i○{\stackrel{\sim }{\mathbf{U}}}_{ij},$$ (19)

where ○ denotes element wise product and the errors Ũ_ij are distributed independently of X_i with E(Ũ_ij) = 1. Importantly, model (19) can be reformulated to arrive at model (16) as

$${\mathbf{W}}_{ij}={\mathbf{X}}_i○{\stackrel{\sim }{\mathbf{U}}}_{ij}={\mathbf{X}}_i+{\mathbf{U}}_{ij},\text{ with }{\mathbf{U}}_{ij}={\mathbf{X}}_i○({\stackrel{\sim }{\mathbf{U}}}_{ij}-\mathbf{1})=\mathbf{S}({\mathbf{X}}_i){\mathbf{\epsilon }}_{ij}.$$ (20)

with E(U_ij|X_i) = X_i ○ E(Ũ_ij - 1) = 0, S(X_i) = diag{s₁(X_i1),…, s_p(X_ip)} with s_ℓ(X_iℓ) = X_iℓ and ε_ij = (Ũ_ij - 1) are independent of X_i with E(ε_ij) = 0. This observation precludes the need for separate methodology to be developed for the problem of multivariate density deconvolution in the presence of multiplicative measurement errors and further emphasizes the importance of the additive conditionally heteroscedastic measurement error model (16) developed in Section 2.2.2.

3 Posterior Inference

Inference is based on samples drawn from the posterior using MCMC algorithms. A Gibbs sampler for the independent error case discussed in Section 2.2.1 is presented in Section S.2 of the Supplementary Material. For the conditionally heteroscedastic case discussed in Section 2.2.2, the full conditionals of the parameters characterizing the variance functions do not have closed form expressions. MCMC algorithms where we tried to integrate Metropolis-Hastings (MH) steps within the Gibbs sampler to generate samples from the full posterior were numerically unstable and failed to converge sufficiently quickly. To address this challenge, we designed a novel two-stage procedure. For each k, we first estimate the functions s_ℓ(X_iℓ) by fitting the univariate deconvolution models W_ijℓ = X_iℓ+s_ℓ(X_iℓ)ε_ijℓ. High precision estimates of the variance functions $s_{\ell }^2(X_{i\ell })$ can be obtained using the univariate deconvolution models. See Figure 2 in the main article and Figure S.7 in the Supplementary Material for illustrations. Parameters characterizing other components of the full model are then sampled using a Gibbs sampler keeping the estimates of the variance functions fixed. Additional details are deferred to Sections S.3 and S.4 of the Supplementary Material.

Figure 2.

Results for conditional variability var(U|X) = s²(X)var(ε) produced by the univariate density deconvolution method for each component of X for the conditionally heteroscedastic error distribution $f_{\mathbf{\epsilon }}^{(2)}$ with sample size n = 1000, m_i = 3 replicates for each subject and identity matrix (I) for the component specific covariance matrices. The results correspond to the data set that produced the median of the estimated integrated squared errors (ISE) out of a total of 100 simulated data sets for the MLFA (mixtures of latent factor analyzers) method. For each component of X, the true variance function is s²(X) = (1+X/4)². See Section 2.2.2 and Section S.3 in the Supplementary Material for additional details. In each panel, the true (lighter shaded green lines) and the estimated (darker shaded blue lines) variance functions are superimposed over a plot of subject specific sample means vs subject specific sample variances. The figure is in color in the electronic version of this article.

4 Model Identifiability

This section presents a discussion of model identifiability issues. The density of interest f_X is identifiable under mild technical assumptions. In the case of independently distributed measurement errors considered in Section 2.2.1 of the main paper, appealing to Li and Vuong (1998), the densities f_X and f_ε are identifiable provided m_i ≥ 2 replicates are available for some individuals, and the characteristics functions ϕ_X (t) = E {exp(℩t^TX)} and ϕε(t) = E{exp(℩t^Tε)} are non-vanishing everywhere.

In the case of conditionally heteroscedastic measurement errors considered in Section 2.2.2 of the main paper, appealing to Hu and Schennach (2004), the densities f_X and f_U|X are identifiable provided m_i ≥ 3 replicates are available for some individuals, the joint, conditional and marginal densities of W₁, W₂, W₃, X are all bounded, and the density f_X|W is bounded complete in the sense that the unique solution to ∫ f_X|W (X)g(X)dX = 0 for all W and for all bounded g(X) is g(X) = 0 for all X. The following lemma provides a sufficient condition for the density f_X|W to be bounded complete.

Lemma 1

f_X|W is bounded complete if E {exp(℩t^T X|W)} is non-vanishing everywhere for all W.

Proof

By Theorem 10C of Goldberg (1961), since E{exp(℩t^TX|W)} is non-vanishing everywhere for all W, the closed linear span of f_X|W(·) is L₁(ℝ). By Hahn-Banach Theorem, the dual space of L₁(ℝ) is L_∞(ℝ) and there is an isometric isomorphism from L_∞(ℝ) to L₁(ℝ) given by g ↦ Φ_g where Φ_g(f_X|W) = ∫ f_X|W(X)g(X)dX for all W. Since the closed linear span of f_X|W(·) for all W is L₁(ℝ), ∫ f_X|W(X)g(X)dX = 0 for all W implies that the mapping Φ_g is identically 0. By the isometric isomorphism above, it follows that g should be identically 0.

Different types of completeness of densities are often used as key identifying conditions in measurement error problems. See, for example, d’Haultfoeuille (2011) and Carroll, et al. (2010). Here, we have provided a general sufficient condition for bounded completeness to hold true and a novel proof using functional analysis techniques. Loosely speaking, if the density f_X|W(X) varies with X, its characteristic function does not vanish. Without sufficient variability of the density of X|W, observations on W do not have enough information to recover the density of X.

Model parameters specifying the components f_X, f_ε, s_ℓ etc. are not separately identifiable. For inference on identifiable functional model components, identifiability of individual parameters is, however, not required. Indeed, the mixture models and the associated priors were so chosen that the mixture components remain unidentifiable. This helps simplify MCMC mixing issues. See Section S.6 of the Supplementary Material.

5 Model Flexibility

This section presents a theoretical study of the flexibility of the proposed models. Proofs of the results are presented in the Supplementary Material. We focus on the deconvolution models for conditionally heteroscedastic measurement errors, the case of independently distributed errors following as a special case. First we show that componentwise our models for the density f_X of X, the density f_ε of the scaled errors ε, and the variance functions v_ℓ are all highly flexible. Building on these results, we then show that our proposed deconvolution models can accommodate a large class of data generating processes.

Let the generic notation Π denote a prior on some class of random functions. Also let 𝒯 denote the target class of functions to be modeled by Π. The support of Π throws light on the flexibility of Π. For Π to be a flexible prior, one would expect that 𝒯 or a large subset of 𝒯 would be contained in the support of Π.

For investigating the flexibility of priors for density functions, a relevant concept is that of Kullback-Leibler (KL) support. The KL divergence between two densities f₀ and f, denoted by d_KL (f₀, f), is defined as d_KL (f₀, f) = ∫ f₀(Z) log {f₀(Z)/f(Z)}dZ. Let Π_f denote a prior assigned to a random density f. A density f₀ is said to belong to the KL support of Π_f if Π_f {f : d_KL (f₀, f) < δ}> 0 ∀δ > 0. The class of densities in the KL support of Π_f is denoted by KL(Π_f).

Let ℱ be the class of target densities to be modeled by the prior Π_f. Let 𝒮 denote the support of ℱ and ℱ̃ ⊆ ℱ denote the class of densities that satisfy the following fairly minimal set of regularity conditions. Since ℱ̃ is a large subclass of ℱ, its inclusion in the KL support of Π_f would establish the flexibility of Π_f.

Conditions 1

1. f₀ is continuous on 𝒮 except on a set of measure zero.

2. The second order moments of f₀ are finite.

3. For some r > 0 and for all z ε 𝒮, there exist hypercubes C_r (z) with side length r and z ε C_r(z) such that

   $$\int f_0(\mathbf{z})\mathit{\text{ log }}\left\{\frac{f_0(\mathbf{z})}{{\text{inf}}_{\mathrm{t}\in C_r(\mathbf{z})}{f}_0(\mathbf{t})}\right\}d\mathbf{z}<\infty .$$

Let Π_X be a generic notation for both the MIW and the MLFA prior on f_X defined in Section 2.1. Similarly, let Π_ε be a generic notation for both the MIW and the MLFA prior on f_ε defined in Section 2.2. When the measurement errors are distributed independently of X, the support of f_X, say 𝒳, may be taken to be any subset of ℝ^p. For conditionally heteroscedastic measurement errors, the variance functions $s_{\ell }^2(·)$ that capture the conditional variability are modeled by mixtures of B-splines defined on closed intervals [A_k, B_k]. In this case, the support of f_X is assumed to be the closed hypercube 𝒳 = [A₁, B₁]×⋯ ×[A_p, B_p]. Let ℱ_X denote the set of all densities on 𝒳, the target class of densities to be modeled by Π_X and ℱ̃_X ⊆ ℱ_X denote the class of densities f_0X that satisfy Conditions 1. Similarly, let ℱ_ε denote the set of all densities on ℝ^p that have mean zero and ℱ̃_ε ⊆ ℱ_ε denote the class of densities f_0ε that satisfy Conditions 1. The following Lemma establishes the flexibility of the models for f_X and f_ε.

Lemma 2

1. ℱ̃_X ⊆ KL (Π_X)2. ℱ̃ ⊆ KL(Π_ε).

For investigating the flexibility of models for general classes of functions, a relevant concept is that of sup norm support. The sup norm distance between two functions g₀ and g, denoted by ‖ g₀ - g‖_∞, is defined as ‖g₀ - g‖_∞ = sup_Z|g₀(Z) - g(Z)|. Let Π_g denote a prior assigned to a random function g. A function g₀ is said to belong to the sup norm support of Π_g if Π_g(g: ‖g₀ - g‖_∞ < δ) > 0 ∀ δ > 0. The class of functions in the sup norm support of Π_g is denoted by SN(Π_g).

Let Π_V denote the prior on the variance functions based on mixtures of B-spline basis functions defined in Section 2.2.2. For notational convenience we consider the case of a univariate variance function supported on [A,B]. Extension to the multivariate case with variance functions supported on 𝒳 is technically trivial. Let 𝒞₊[A,B] denote the set of continuous functions from [A,B] to ℝ⁺. Also, for α ≤ (q+1), let ${𝒞}_{+}^{\alpha }[A,B]\subseteq {𝒞}_{+}[A,B]$ denote the set of functions that are α₀ times continuously differentiable, and for all $v_0\in {𝒞}_{+}^{\alpha }[A,B],{\Vert v_0\Vert }_{\alpha }<\infty ,$, where α₀ is largest integer less than or equals to α and the seminorm is defined by ${\Vert v_0\Vert }_{\alpha }={\text{sup}}_{X,X\prime \in [A,B],X\ne X\prime },\{|v_0^{({\alpha }_0)}(X)-v_0^{({\alpha }_0)}(X\prime )|/{|X-X\prime |}^{\alpha -{\alpha }_0}\}.$. The local support properties of B-splines make the models for the variance functions very flexible as is indicated by the following lemma.

Lemma 3

${𝒞}_{+}^{\alpha }[A,B]\subseteq {𝒞}_{+}[A,B]\subseteq SN({\mathrm{\Pi }}_{\mathbf{V}}).$

Although technically the sup norm distance between linear combinations of B-splines and any continuous function can be made arbitrarily small by increasing the number of knots, for obvious reasons the actual bounds for the sup norm distance may not be very sharp if the function to be modeled is wiggly. However, for most applications of practical importance, the true variance function may be assumed to be smooth, that is, to belong to some ${𝒞}_{+}^{\alpha }[A,B]$ with α ≥ 1. Therefore, for practical reasons, it is only important that the smaller Hölder class of functions ${𝒞}_{+}^{\alpha }[A,B]$ belongs to the sup norm support of Π_V. As shown in Section S.7.2 of the Supplementary Material, the bounds for sup norm distance in this case will also be much sharper.

Since the models for the variance functions v_ℓ and the models for the density of the scaled errors f_ε are separately very flexible, under model (16) on the measurement errors, the implied conditional and joint densities are also expected to be very flexible. This is investigated in the next lemma. For a given X, let Π_U|X denote the prior for f_U|X induced by Π_ε and Π_V under model (16). Define ${\stackrel{\sim }{ℱ}}_{\mathbf{U}|\mathbf{X}}=\{f_{0\mathbf{U}|\mathbf{X}}:f_{0\mathbf{U}|\mathbf{X}}(\mathbf{U})={\mathbf{\Pi }}_{k=1}^p{s}_{0k}^{-1}(X_k)f_{0\mathbf{\epsilon }}\{{\mathbf{S}}_0^{-1}(\mathbf{X})\mathbf{U}\},s_{0k}^2\in {𝒞}_{+}[A_k,B_k]$ for k = 1, …,p, f_0ε ε ℱ̃_ε. Also let Π_U|V denote the prior for the unknown conditional density of U induced by Π_ε and Π_V under model (16). Define ℱ̃_U|• = {f_0U|• : for any given X ε 𝒳, f_0U|• = f_0U|X ε ℱ̃_U|X}. Finally, let Π_X,U denote the prior for the joint density of (X, U) induced by Π_X, Π_ε and Π_V under model (16). Define ℱ̃ _X,U = {f_0,X,U : f_0,X,U(X, U) = f_0,X(X)f_0,U|X(U|X), where f_0X ε ℱ̃_X and f_0U|X ε ℱ̃_U|X for all X ε 𝒳}.

Lemma 4

1. ℱ̃_U|X ⊆ K L(Π_U|X) for any given X ε 𝒳.

2. For any f_0U|• ε ℱ̃_U|V, Π_U|V{sup_Xε𝒳 d_KL (f_0U|X, f_U|X) < δ} > 0 for all δ > 0.

3. ℱ̃_X,U ⊆ K L(Π_X,U).

The flexibility of the implied model for the marginal density f_W is the subject of our final result. Since the only observed quantities are W_ij, the support of the induced prior on f_W tells us about the types of likelihood functions the model can approximate.

Let Π_W denote the prior for the density of W induced by Π_X, Π_ε and Π_V under model (16). Also let ℱ̃_W = {f_0W : f_0W(W) = ∫ f_0X(X)f_0U|X(W-X)dX, f_0X ε ℱ̃_X, f_0U|• ε ℱ̃_U|•}, the class of densities f_0W that can be obtained as the convolution of two densities f_0X and f_0U|•, where f_0X ℱ̃_X and f_0U|• ε ℱ̃_U|•.

Since the supports of Π_X and Π_U|X are large, it is expected that the support of Π_W will also be large. However, because convolution is involved, investigation of KL support of Π_W is a difficult problem. A weaker but relevant concept is that of L₁ support. The L₁ distance between two densities f₀ and f, denoted by ‖f₀ - f‖₁, is defined as ‖f₀ - f‖₁ = ∫|f₀(Z) - f(Z)|dZ. A density f₀ is said to belong to the L₁ support of Π_f if Π_f (f : ‖f₀ - f‖₁ < δ) > 0 ∀ > 0. The class of densities in the L₁ support of Π_f is denoted by L₁(Π_f). The following theorem shows that the L₁ support of Π_W is large.

Theorem 1

ℱ̃_W ⊆ L₁(Π_W).

The proofs of these results are deferred to Section S.7 of the Supplementary Material. The proofs require that the number of mixture components K be allowed to vary over ℕ, the set of all positive integers, through priors, denoted by the generic notation P₀(K), that assign positive probability to all K ε ℕ. Posterior computation for such methods will be computationally intensive, specially in a complicated multivariate set up like ours. In our implementation, we thus keep the number of mixture components fixed at finite values.

6 Simulation Experiments

The mean integrated squared error (MISE) of estimation of f_X by f̂_X is defined as MISE = E_fX ∫ {f_X(X) - f̂_X(X)}² dX. Based on B simulated data sets, a Monte Carlo estimate of MISE is given by ${\mathit{\text{MISE}}}_{\mathit{\text{est}}}=B^{-1}{\sum }_{b=1}^B{\sum }_{m=1}^M{\{f_{\mathbf{X}}({\mathbf{X}}_{b,m})-{\widehat{f}}_{\mathbf{X}}^{(b)}({\mathbf{X}}_{b,m})\}}^2/p_0({\mathbf{X}}_{b,m}),$, where ${\{{\mathbf{X}}_{b,m}\}}_{b=1,m=1}^{B,M}$ are random samples from the density p₀. We designed simulation experiments to evaluate the MISE performance of the proposed models for a wide range of possibilities. The MISEs we report here are all based on 100 simulated data sets and M = 10⁶ samples generated from each of the two densities (a) p₀ = f_X, the true density of X, and (b) p₀ that is uniform on the hypercube with edges min_k {μ_X,k - 31_p} and max_k {μ_X,k + 31_p}. With carefully chosen initial values and proposal densities for the MH steps, we were able to achieve quick convergence for the MCMC samplers. The use of exchangeable Dirichlet priors helped simplify mixing issues (Geweke, 2007). See Section S.6.2 in the Supplementary Material for additional discussions. We programmed our methods in R. In each case, we ran 3000 MCMC iterations and discarded the initial 1000 iterations as burn-in. The post burn-in samples were thinned by a thinning interval of length 5. For the univariate samplers, 1000 MCMC iterations with a burn-in of 500 sufficed to produce stable estimates of the variance functions. In our experiments with much larger iteration numbers and burn-ins, the MISE performances remained practically the same. This being the first article that tries to solve the problem of multivariate density deconvolution when the measurement error density is unknown, the proposed MIW and MLFA models have no competitors. We thus compared our models with a naive Bayesian method that ignores measurement errors and treats the subject specific means as precisely measured observations instead, modeling f_X by a finite mixture of multivariate normals as in (2) with inverse Wishart priors on the component specific covariance matrices.

We considered two choices for the sample size n = 500, 1000. For each subject, we simulated m_i = 3 replicates. The true density of X was chosen to be $f_{\mathbf{X}}(\mathbf{X})={\sum }_{k=1}^{K_{\mathbf{X}}}{\pi }_{\mathbf{X},k}{\text{MVN}}_p(\mathbf{X}|{\mathit{\mu }}_{\mathbf{X},k},{\mathbf{\sum }}_{\mathbf{X},k})$ with p = 4, K_X = 3, π_X = (0.25, 0.50, 0.25)^T, μ_X,1 = (0.8, 6, 4, 5)^T, μ_X,2 = (2.5, 4, 5, 6)^T and μ_X,3 = (6, 4, 2, 4)^T. For the density of the measurement errors f_ε we considered two choices, namely

1. $f_{\mathbf{\epsilon }}^{(1)}(\mathbf{\epsilon })={\text{MVN}}_p(\mathbf{\epsilon }|\mathbf{0},{\mathbf{\sum }}_{\mathbf{\epsilon }})$, and

2. $f_{\mathbf{\epsilon }}^{(2)}(\mathbf{\epsilon })={\mathbf{\sum }}_{k=1}^{K_{\mathbf{\epsilon }}}{\pi }_{\mathbf{\epsilon },k}{\text{MVN}}_p(\mathbf{\epsilon }|{\mathit{\mu }}_{\mathbf{\epsilon },k},{\mathbf{\sum }}_{\mathbf{\epsilon },k})$ with K_ε = 3, π_ε = (0.2, 0.6, 0.2)^T, μ_{ε, 1} = (−0.3, 0, 0.3, 0)^T, μ_{ε, 2} = (−0.5, 0.4, 0.5, 0)^T and μ_ε,3 = −(π_{ε, 1} μ_ε,1 + π_ε,2 μ_{ε, 2}/π_ε,3.

For the component specific covariance matrices, we set Σ_X,k = D_X Σ_X,0 D_X for each k, where D_X = diag(0.75^1/2,…, 0.75^1/2). Similarly, Σ_ε,k = D_ε Σ_ε,0 D_ε for each k, where D_ε = diag(0.3^1/2,…, 0.3^1/2). For each pair of f_X and f_ε, we considered four types of covariance structures for ${\mathbf{\sum }}_{\mathbf{X},0}=\{({\sigma }_{ij}^{\mathbf{X},0})\}$, and ${\mathbf{\sum }}_{\mathbf{\epsilon },0}=\{({\sigma }_{ij}^{\mathbf{\epsilon },0})\}$, namely

1. Identity (I): Σ_X,0 = Σ_ε,0 = I_p,

2. Latent Factor (LF): Σ_X,0 = Λ_X Λ_X + Ω_X, with Λ_X = (0.7,…, 0.7)^T and Ω_X = diag(0.51,…, 0.51), and Σ_ε,0 = Λ_ε Λ_ε + Ω_ε, with Λ_ε = (0.5,…, 0.5)^T and Ω_ε = diag(0.75,…, 0.75),

3. Autoregressive (AR): ${\sigma }_{ij}^{\mathbf{X},0}={0.7}^{|i-j|}$ and ${\sigma }_{ij}^{\mathbf{\epsilon },0}={0.5}^{|i-j|}$ for each (i, j), and

4. Exponential (EXP): ${\sigma }_{ij}^{\mathbf{X},0}=\text{exp}(-0.5|i-j|)$ and ${\sigma }_{ij}^{\mathbf{\epsilon },0}=\text{exp}(-0.9|i-j|)$ for each (i, j).

The parameters were chosen to produce a wide variety of one and two dimensional marginal densities, see Figure 4 and also Figure 6. Scale adjustments by multiplication with D_X and D_ε were done so that the simulated values of each component of X fall essentially in the range (−2, 6) and the simulated values of all components of ε fall essentially in the range (−3, 3). For conditionally heteroscedastic measurement errors, we set the true variance functions at $s_{\ell }^2(X)={(1+X/4)}^2$ for each component ℓ. A total of 16 (2 × 1 × 2 × 4) cases were thus considered for both independent and conditionally heteroscedastic measurement errors.

Figure 4.

Results for the f_X produced by the MLFA (mixtures of latent factor analyzers) method for the conditionally heteroscedastic error distribution $f_{\mathbf{\epsilon }}^{(2)}$ with sample size n = 1000, m_i = 3 replicates for each subject and identity matrix (I) for the component specific covariance matrices. The results correspond to the data set that produced the median of the estimated integrated squared errors (ISE) out of a total of 100 simulated data sets. See Section 6 for additional details. The upper triangular panels show the contour plots of the true two dimensional marginal densities. The lower triangular diagonally opposite panels show the corresponding estimates. The numbers i, j at the bottom right corners of the off-diagonal panels show that the marginal densities f_Xi,Xj are plotted in those panels. The diagonal panels show the true (lighter shaded green lines) and the estimated (darker shaded blue lines) one dimensional marginals. The figure is in color in the electronic version of this article.

Figure 6.

Results for the density of the scaled measurement errors f_ε produced by the MLFA (mixtures of latent factor analyzers) method for the conditionally heteroscedastic error distribution $f_{\mathbf{\epsilon }}^{(2)}$ with sample size n = 1000, m_i = 3 replicates for each subject and identity matrix (I) for the component specific covariance matrices. The results correspond to the data set that produced the median of the estimated integrated squared errors (ISE) out of a total of 100 simulated data sets. See Section 6 for additional details. The upper triangular panels show the contour plots of the true two dimensional marginal densities. The lower triangular diagonally opposite panels show the corresponding estimates. The numbers i, j at the bottom right corners of the off-diagonal panels show that the marginal densities f_εi,_εj are plotted in those panels. The diagonal panels show the true (lighter shaded green lines) and the estimated (darker shaded blue lines) one dimensional marginals. The figure is in color in the electronic version of this article.

We first discuss the results of the simulation experiments when the measurement errors U were independent of X. The estimated MISEs are presented in Table 1. When the true f_ε was a single component multivariate normal, the MLFA model produced the lowest MISE when the true covariance matrices were diagonal. In all other cases the MIW model produced the best results. When the true f_ε was a mixture of multivariate normals, the model complexity increases and the performance of the MIW model started to deteriorate. In this case, the MLFA model dominated the MIW model when the true covariance matrices were either diagonal or had a latent factor characterization.

Table 1.

Mean integrated squared error (MISE) performance of MLFA (mixtures of latent factor analyzers) and MIW (mixtures with inverse Wishart priors) density deconvolution models described in Section 2 of this article for homoscedastic errors compared with a naive method that ignores measurement errors for different measurement error distributions. The minimum value in each row is highlighted.

True Error Distribution	Covariance Structure	Sample Size	MISE × 104
			MLFA	MIW	Naive
(a) Multivariate Normal	I	500	1.24	3.05	8.01
		1000	0.59	1.33	6.58
	LF	500	6.88	6.33	33.41
		1000	5.15	3.10	32.42
	AR	500	11.91	5.51	27.17
		1000	9.82	2.78	26.01
	EXP	500	7.15	4.40	17.82
		1000	5.46	2.19	17.40
(b) Mixture of Multivariate Normal	I	500	1.28	3.24	5.97
		1000	0.64	1.37	4.99
	LF	500	7.28	7.51	31.62
		1000	4.17	4.34	31.48
	AR	500	10.43	6.66	30.74
		1000	7.75	4.35	28.90
	EXP	500	7.16	5.18	17.85
		1000	4.87	2.66	17.26

The estimated MISEs for the cases when U were conditionally heteroscedastic are presented in Table 2. Models that accommodate conditional heteroscedasticity are significantly more complex compared to models that assume independence of the measurement errors from X. The numerically more stable MLFA model thus out-performed the MIW model in all 32 cases. The improvements were particularly significant when the true covariance matrices were sparse and the number of subjects was small (n = 500). The true and estimated univariate and bivariate marginals of f_X produced by the MIW and the MLFA methods when the true density of the scaled errors was a mixture of multivariate normals ( $f_{\mathbf{\epsilon }}^{(2)}$) and the component specific covariance matrices were diagonal (I) are summarized in Figure 3 and Figure 4, respectively. The true and estimated univariate and bivariate marginals for the density of the scaled errors f_ε for this case produced by the two methods are summarized in Figure 5 and Figure 6, respectively. The true and the estimated variance functions produced by the univariate submodels are summarized in Figure 2. Comparisons between Figure 3 and Figure 4 illustrate the limitations of the MIW models in capturing high dimensional sparse covariance matrices and the improvements that can be achieved by the MLFA models. The estimates of f_ε produced by the two methods are in better agreement. This may be attributed to the fact that many more residuals are available for estimating f_ε than there are X_i’s to estimate f_X. Figure 2 in the main paper and Figures S.7 and S.16 in the Supplementary Material show that the univariate submodels can recover the true variance functions well. Additional figures when the true covariance matrices had auto-regressive structure (AR) are presented in the Supplementary Material. In this case the true covariance matrices were not sparse. The MLFA method still vastly dominated the MIW method when the sample size was small (n = 500). When the sample size was large (n = 1000) the two methods produced comparable results.

Figure 3.

Results for f_X produced by the MIW (mixtures with inverse Wishart priors) method for the conditionally heteroscedastic error distribution $f_{\mathbf{\epsilon }}^{(2)}$ with sample size n = 1000, m_i = 3 replicates for each subject and identity matrix (I) for the component specific covariance matrices. The results correspond to the data set that produced the median of the estimated integrated squared errors (ISE) out of a total of 100 simulated data sets. See Section 6 for additional details. The upper triangular panels show the contour plots of the true two dimensional marginal densities. The lower triangular diagonally opposite panels show the corresponding estimates. The numbers i, j at the bottom right corners of the off-diagonal panels show that the marginal densities f_Xi, X_j are plotted in those panels. The diagonal panels show the true (lighter shaded green lines) and the estimated (darker shaded blue lines) one dimensional marginals. The figure is in color in the electronic version of this article.

Figure 5.

Results for the density of the scaled measurement errors f_ε produced by the MIW (mixtures with inverse Wishart priors) method for the conditionally heteroscedastic error distribution $f_{\mathbf{\epsilon }}^{(2)}$ with sample size n = 1000, m_i = 3 replicates for each subject and identity matrix (I) for the component specific covariance matrices. The results correspond to the data set that produced the median of the estimated integrated squared errors (ISE) out of a total of 100 simulated data sets. See Section 6 for additional details. The upper triangular panels show the contour plots of the true two dimensional marginal densities. The lower triangular diagonally opposite panels show the corresponding estimates. The numbers i, j at the bottom right corners of the off-diagonal panels show that the marginal densities f_εi, _εj are plotted in those panels. The diagonal panels show the true (lighter shaded green lines) and the estimated (darker shaded blue lines) one dimensional marginals. The figure is in color in the electronic version of this article.

The proposed deconvolution methods, in particular the MLFA method, are highly scalable. In small scale simulations, not reported here, we tried p = 6, 8 and 10 and observed good empirical performance. We have focused here on p = 4 dimensional problems since with p = 4 the numbers of univariate and bivariate marginals, $p=4\text{ and }\left(\begin{array}{c}p\\ 2\end{array}\right)=6$, remain manageable and the results are conveniently graphically summarized.

Additional small scale simulations for a variety of other distributions with similar MISE patterns are presented in the Supplementary Material.

7 Example

Dietary habits are known to be leading causes of many chronic diseases. Accurate estimation of the distributions of dietary intakes is thus important in nutritional epidemiologic surveillance and epidemiology. Nutritionists are typically interested not just in the consumption patterns of individual dietary components but also in their joint consumption patterns. By the very nature of the problem, X, the average long term daily intakes of the dietary components, can never be directly observed. Data are thus typically collected from a representative sample of the population in the form of dietary recalls, the subjects participating in the study remembering and reporting the type and amount of food they had consumed in the past 24 hours. The problem of estimating the joint consumption pattern of the dietary components from the contaminated 24-hour recalls then becomes a problem of multivariate density deconvolution.

A large scale epidemiologic study conducted by the National Cancer Institute, the Eating at America’s Table (EATS) study (Subar, et al. 2001), serves as the motivation for this paper. In this study n = 965 participants were interviewed m_i = 4 times over the course of a year and their 24 hour dietary recalls (W_ij’s) were recorded. The goal is to estimate the joint consumption patterns of the true daily intakes (X_i’s).

To illustrate our methodology, we consider the problem of estimating the joint consumption pattern of four dietary components, namely (a) carbohydrate, (b) fiber, (c) protein and (d) a mineral potassium. Figure 7 shows the plots of subject-specific means versus subject-specific variances for daily intakes of the dietary components with the estimates of the variance functions produced by univariate submodels superimposed over them. As is clearly identifiable from this plot, conditional heteroscedasticity is a very prominent feature of the measurements errors contaminating the 24 hour recalls. The estimated univariate and bivariate marginal densities of average long term daily intakes of the dietary components produced by the MIW method and the MLFA method are summarized in Figure 8. The estimated univariate and bivariate marginal densities for the scaled errors are summarized in Figure 9. The estimated marginals of X produced by the two methods look quite different, while the estimated marginals of ε are in close agreement. The estimated univariate and bivariate marginal densities of the long term intakes of the dietary components produced by the MIW model look irregular and unstable, whereas the estimates produced by the MLFA model look relatively more regular and stable. In experiments not reported here, we observed that the estimates produced by the MIW method were sensitive to the choice of the number of mixture components, but the estimates produced by the MLFA model were quite robust. The trace plots and the frequency distributions of the of the numbers of nonempty mixture components are summarized in Figures S.14 and S.15 in the Supplementary Material and provide some idea about the relative stability of the two methods. These observations are similar to that made in Section 6 for conditionally heteroscedastic measurement errors and sparse covariance matrices.

Figure 8.

Results for the EATS data set for the f_X. The off-diagonal panels show the contour plots of two-dimensional marginals estimated by the MIW method (upper triangular panels) and the MLFA method (lower triangular panels). The numbers i, j at the bottom right corners of the off-diagonal panels show that the marginal densities f_Xi,Xj are plotted in those panels. The diagonal panels show the one dimensional marginal densities estimated by the MIW method (darker shaded blue lines) and the MLFA method (lighter shaded green lines). The figure is in color in the electronic version of this article.

Figure 9.

Results for the EATS data set for the density of the scaled errors f_ε. The off-diagonal panels show the contour plots of two-dimensional marginals estimated by the MIW method (upper triangular panels) and the MLFA method (lower triangular panels). The numbers i, j at the bottom right corners of the off-diagonal panels show that the marginal densities f_εi, _εj are plotted in those panels. The diagonal panels show the one dimensional marginal densities estimated by the MIW method (darker shaded blue lines) and the MLFA method (lighter shaded green lines). The figure is in color in the electronic version of this article.

We next comment only on the estimates produced by the MLFA method assuming them to be closer to the truth. The estimates show that the long term daily intakes of the four dietary components are strongly correlated. The shapes of the bivariate consumption patterns suggest deviations from normality. Similarly, the shapes of the bivariate marginals for the scaled errors suggest that the measurement errors in the reported 24 hour recalls are positively correlated and deviate from normality. People who consume more are expected to do so for most dietary components. Strong correlations between the intakes of the dietary components are thus somewhat expected. The correlations among different components of the measurement errors suggest that people usually have a tendency to either over-report or under-report the daily intakes. These findings illustrate the importance of robust but numerically stable multivariate deconvolution methods in nutritional epidemiologic studies.

Additional discussions on potentially far-reaching impact of our work on nutritional epidemiology studies are deferred to Section S.10 in the Supplementary Material.

8 Discussion

We considered the problem of multivariate density deconvolution when the measurement error density is not known but replicated proxies are available for some individuals. We used flexible finite mixtures of multivariate normal kernels with symmetric Dirichlet priors on the mixture probabilities to model both the density of interest and the density of the measurement errors. We proposed a novel technique to make the model for the density of the errors satisfy a zero mean restriction. We showed that the dense parametrization of inverse Wishart priors are not suitable for modeling covariance matrices in the presence of measurement errors. We proposed a numerically more stable approach based on latent factor characterization of the covariance matrices with sparsity inducing priors on the factor loading matrices. We built models for conditionally heteroscedastic additive measurement errors that also automatically accommodate multivariate multiplicative measurement errors.

The methodological contributions of this article are not limited to deconvolution problems. Mixtures of latent factor analyzers with sparsity inducing priors on the factor loading matrices can be used in other high dimensional applications including ordinary density estimation. The techniques proposed in Section 2.2.1 to enforce the mean zero moment restriction on the measurement errors can be readily used to model multivariate regression errors that are distributed independently of the predictors. The technique can also be adapted to relax the strong assumption of multivariate normality made by Hoff and Niu (2012) and Fox and Dunson (2016) in covariance regression problems.

As explained in Sections 2.2.2 and 2.2.3 in the main paper and also in Section S.5 in the Supplementary Material, the structural separability assumption (16) arises naturally in both additive and multiplicative multivariate measurement error settings. It would still be interesting, in future work, to consider more general covariance models that allow var(U_ijℓ|X) to be explained primarily by X_iℓ, as in the current approach, but would allow the residual variability to be explained by the remaining components {X_im}_m≠ℓ of X. The current MCMC based implementation of the proposed methodology is computationally intensive. We are pursuing the development of faster algorithms for approximate posterior inference as the subject of a separate manuscript.

The question of consistency of Bayesian procedures is intimately related to the flexibility of the priors. For instance, in ordinary density estimation problems inclusion of the true density in the KL support of the prior is a sufficient condition to ensure weak consistency via the Schwartz theorem. In density deconvolution problems such a condition is not sufficient but is still required. The results from Section 5 thus provide crucial first steps in that direction. We have not pursued the question of consistency of the proposed deconvolution methods any further in this article. It remains an important direction for future research.

Table 2.

Mean integrated squared error (MISE) performance of MLFA (mixtures of latent factor analyzers) and MIW (mixtures with inverse Wishart priors) density deconvolution models described in Section 2 of this article for conditionally heteroscedastic errors compared with a naive method that ignores measurement errors for different measurement error distributions. The minimum value in each row is highlighted.

True Error Distribution	Covariance Structure	Sample Size	MISE × 104
			MLFA	MIW	Naive
(a) Multivariate Normal	I	500	2.53	19.08	10.64
		1000	1.15	9.43	9.14
	LF	500	11.46	34.21	21.33
		1000	5.78	15.98	20.75
	AR	500	17.11	30.83	36.44
		1000	10.77	12.46	36.37
	EXP	500	11.63	26.99	24.28
		1000	6.67	10.56	23.36
(b) Mixture of Multivariate Normal	I	500	2.79	22.17	20.16
		1000	1.38	10.55	19.39
	LF	500	13.39	35.67	43.43
		1000	7.50	20.86	43.28
	AR	500	18.27	35.70	75.26
		1000	12.06	16.64	77.55
	EXP	500	12.11	34.50	48.76
		1000	7.59	13.74	50.02