Identifying influential observations in a Bayesian multi-level mediation model

ABSTRACT

Increasingly complex models are being fit to data these days. This is especially the case for Bayesian modelling making use of Markov chain Monte Carlo methods. Tailored model diagnostics are usually lacking behind. This is also the case for Bayesian mediation models. In this paper, we developed a method for the detection of influential observations for a popular mediation model and its extensions in a Bayesian context. Detection of influential observations is based on the case-deletion principle. Importance sampling with weights which take advantage of the dependence structure in hierarchical models is utilized in order to identify the part of the model which is influenced most. We make use of the variance of log importance sampling weights as the measure of influence. It is demonstrated that this approach is useful when interest lies in the impact of individual observations on a subset of model parameters. The method is illustrated on a three-level data set from the field of nursing research, which was previously used to fit a mediation model of patient satisfaction with care. We focused on influential cases on both the second and the third level of the data.

1. Introduction

Increasingly complex models, often estimated in a Bayesian way, are being proposed in the current statistical literature. A lot of effort has been spent to develop general models to describe complex phenomena. However, diagnostics and model checking are lagging behind, although they are important to avoid that wrong or misleading results and conclusions are used. One of such diagnostics checks whether observations have a high impact on the estimated model parameters.

In this paper we focus on mediation models, which constitute a broad class of often complex models that received quite some attention in recent biostatistical literature. More specifically, we wish to develop a general method to detect influential observations in a popular mediation model for ordinal outcomes and its extensions to the multi-level case. In a mediation analysis one aims to explain the causal path from risk factor to outcome via a set of explanatory variables. Most common approaches to mediation analysis assume that the outcome variable Y as well as all explanatory variables are continuous and the analysis proceeds by using suitable linear models (see, e.g. Preacher et al. [11]). To introduce the concept, we now describe a popular but simple mediation model. Let us assume pairs $\{Y_i,M_i\}$ ( $i=1,\dots ,N$) with $Y_i$ a continuous outcome and $M_i$ a continuous mediator both observed on a set of N independent units. We wish to investigate their relationship with a covariate $W_i$ ( $i=1,\dots ,N$). A simple mediation model is specified through two model equations based on a classical linear model:

$$\begin{array}{rl}{Y}_i& ={\beta }_0+{\beta }_W{W}_i+{\beta }_M{M}_i+{\epsilon }_i,\end{array}$$ (1)

$$\begin{array}{rl}{M}_i& ={\gamma }_0+{\gamma }_W{W}_i+{\xi }_i,i=1,\dots ,N,\end{array}$$ (2)

where ${\epsilon }_1,\dots ,{\epsilon }_N$, ${\xi }_1,\dots ,{\xi }_N$ are model error terms, and ${\beta }_0$, ${\beta }_W$, ${\beta }_M$, ${\gamma }_0$, ${\gamma }_W$ are regression coefficients.

In the context of mediation modelling, Equation (1) is referred to as a measurement equation, whereas Equation (2) is called a mediation equation. The coefficient ${\beta }_W$ is referred to as the direct effect, whereas the product ${\beta }_M{\gamma }_W$ is referred to as the indirect (mediated) effect of the variable W on the outcome. Equations (1)–(2) are fitted jointly in order to estimate the indirect effect. For the identification of the indirect effect, the following assumptions need to be satisfied: (1) no unmeasured treatment-outcome confounding, (2) no unmeasured mediator-outcome confounding, (3) no unmeasured treatment-mediator confounding, and (4) no mediator-outcome confounder affected by treatment (VanderWeele [16]). The direct and indirect effects are of primary interest, therefore it is essential that their estimates are stable. The above simple model will further be extended to capture more complex data structures.

The Bayesian approach is often utilized to fit complex models. In general, the Markov chain Monte Carlo (MCMC) methods are employed to generate a sample from the posterior distribution of the parameters. This sample is further used to draw inference on the model parameters. However, no general model diagnostic tool for the identification of influential observations is readily available for our model and its extensions. Therefore, we deem the development of such an instrument as necessary, see also Section 1.2 for our motivation in a practical setting. In this paper, we consider an observation (or a group of observations) as influential if the case-deleted (or group-deleted) posterior distribution of the parameter of interest considerably differs from that obtained with the posterior distribution based on all observations. The metric that defines a considerable difference is defined below.

Case-deletion as a tool for detecting influential observations has been widely used in linear regression models in both frequentist and Bayesian analysis (see, e.g. Cook and Weisberg [4], Weiss [17] and Kass et al. [7]). While in the frequentist approach, analytical results or approximate formulas are available to express the case-deleted parameter estimates as a function of the full sample (FS) estimates, this is not the case anymore in the Bayesian approach. Importance sampling is a common method to calculate the case-deleted posterior parameter estimates based on the FS posterior distribution determined by MCMC methods (e.g. Robert and Casella [12], Section 3.3). In addition, large variability of the importance weights indicates that the case-deleted posterior estimates may be quite different from the FS estimates. For this reason, Thomas et al. [15] proposed to use the variance of log importance sampling weights as a measure of influence.

In this paper, we combine the approach of Bradlow and Zaslavsky [2] with that of Thomas et al. [15] for the detection of influential observations in the mediation model defined in (1, 2) extended to the multi-level setting with ordinal responses and multiple covariates ( $W_i$) and mediators ( $M_i$), first proposed in our earlier work Rusá et al. [13].

The importance weights suggested here take advantage of the dependence structure in models analogous to (1, 2) in order to determine which part of the model is influenced. This may add additional insight to the analysts and can help them to point out what part of the model could be improved. With a hierarchical model structure, the choice of suitable importance weights facilitates the utilization of the leave-one-out diagnostics at any level of the data. We propose to use a diagnostic plot with two marginal influence measures (these reflect the measurement and the mediation part of the model) plotted on the X-axis and the Y-axis, respectively. The overall influence is evaluated as the sum of the marginal influence measures so it is easily detected from the off diagonals of the plot.

We now review the existing literature in Section 1.1. In Section 1.2 we formulate the research questions that motivated our developments and introduce the motivating dataset.

1.1. Review and need for extensions

The assessment of influence in Bayesian normal linear models and hierarchical normal random effects models has been studied in Weiss and Cho [18]. The approach allowed to focus on a subset of parameters thereby treating the remaining ones as nuisance. In Bayesian generalized linear models where the likelihood dominates the prior, the MCMC output can be used to approximate the frequentist case-deleted maximum likelihood estimate and employed as a measure of influence (Jackson et al. [6]). Hodges [5] expressed hierarchical models as ordinary linear models, which facilitates the development of diagnostics for model selection, case influence and residuals. This method is appropriate for the simple mediation model given in (1, 2). Bayarri and Morales [1] suggested to carry out preliminary checks and proposed several measures for outlier detection in Gaussian models. These proposals may be of interest because outliers are those observations that are remote from the bulk of the data and are potentially influential (Zaslavsky and Bradlow [19]).

Model diagnostics have not been restricted to case deletion. Cook [3] proposed general perturbation schemes for assessing local influence of frequentist models. The Bayesian approach to local influence, based on perturbing the value of a suitably chosen hyperparameter away from an initial choice, is discussed in McCulloch [8]. Moreover, it was shown that the posterior variance of the log-likelihood of a given observation is a measure of that observation's local influence in models with conditionally independent observations (Millar and Stewart [9]).

Importance sampling is a popular technique used for the estimation of case-deleted posterior statistics (e.g. posterior means) in Bayesian models. For hierarchical models, the importance weights can be easily computed due to the (assumed) conditional hierarchical independence in the model (Bradlow and Zaslawsky [2]). However, the use of classical importance weights to estimate the case-deleted posterior means may fail because an excessive (even infinite) variance of the importance weights when the case-deleted posterior differs too much from the original (see, e.g. Hodges [5] or Peruggia [10]). While Bradlow and Zaslavsky [2] tried to reduce the variability of the importance weights using a convenient factorization, they have not solved this issue. Recently, Thomas et al. [15] obtained low-dimensional case-influence summaries entirely via the variance-covariance matrix of the log importance sampling weights. They also considered the joint influence for sets of cases. The influence measures are computed from a principal component analysis (PCA), which reveals the most interesting directions of the local curvature of the Kullback-Leibler divergence of the full posterior from a geometrically perturbed quasi-posterior. In mediation models we are specifically interested in knowing what part of the model (measurement and/or mediation part) is grossly influenced by the observation. Combining the approaches of Bradlow and Zaslavsky [2] and Thomas et al. [15] might therefore be useful for mediation models.

Nevertheless, none of previously mentioned methods may be directly applied to detect influential observations in the mediation models considered in Section 2. Diagnostic checks should be an integral part of any model evaluation and it does not make sense to fit complex models without standard model checking, which includes the search for influential observations. Therefore, we deem it necessary to develop computationally tractable methods for the identification of influential observations for complex Bayesian mediation models. To this end, we adopted the approach of Bradlow and Zaslavsky [2] to case influence diagnostics but our methods differ from theirs in three ways: (i) we use an extended factorization scheme, which is advantageous for mediation models, (ii) we propose simple importance weights if case-deletion for parameters from only a single model equation is of interest, and finally, (iii) we suggest how to combine the importance weights in case we are interested in the detection of influence of a group of observations. Consequently, we utilize the same measure of influence as in Thomas et al. [15]. However, for mediation models the natural decomposition of the model (the measurement equation and mediation equations) is preferred over PCA used there.

1.2. Motivating example: RN4CAST study and research questions

The Registered Nurse Forecasting (RN4CAST) study (Sermeus et al. [14]) is a cross-sectional survey of patients and nurses conducted in 12 European countries. This FP7-funded project ran from 2009 to 2010 and collected data on hospitals, nursing units, nurses and patients. In each of the 12 countries (except Sweden) at least 30 hospitals depending upon country size and number of hospitals were involved in the study. Hospitals were selected using different selection mechanisms depending on the country. Nurses were invited to participate as well as patients who were hospitalized in a priori chosen periods. Various hospital characteristics were recorded, such as nurse staffing, nurse education, number of beds, etc. The nurses were interviewed on various nursing care aspects, such as their well-being, satisfaction with their job, their willingness to recommend the hospital and overtime work. The patients provided information on their satisfaction with hospital care and hospital rating.

In Rusá et al. [13], a multi-level mediation model with ordinal outcome is presented to evaluate how the patients' willingness to recommend the hospital relates to the system-level features in the hospital organization or in the nursing care. It was also of interest to know whether the association is mediated by two measures of ‘nursing care left undone’. Here, we aim to discover whether there are hospitals or countries that are influential for estimating the model parameters. In addition, we wish to disentangle their effect into their effect on the measurement model and their effect on the mediation model.

The outline of the paper is as follows. In Section 2 we describe the multi-level moderated mediation model with ordinal outcome and multiple mediators suggested in Rusá et al. [13]. The Bayesian methods for the detection of influential observations are explained in Section 3. In Section 4 we apply our model to the mediation model outlined in Section 2. Concluding remarks are given in Section 5.

2. A general multi-level mediation model

In this section we treat case influence diagnostics for the general multi-level mediation model with ordinal outcome proposed by Rusá et al. [13]. This model generalizes the joint model Equations (1, 2) to a hierarchical model with an ordinal outcome. The model was motivated by research questions raised in the RN4CAST study, which is briefly described below. We also show how the posterior distribution can be factorized for the computation of convenient case influence diagnostics using importance sampling described in Section 3.

2.1. Model formulation

Rusá et al. [13] suggested a multivariate extension of the mediation model given by (1) and (2). We describe the model with the RN4CAST data in mind, but the proposed model is of course more general. Suppose that the data can be structured into three hierarchical levels. Let K be the number of units on the third level (countries in our application); $N_k,k=1,\dots ,K,$ the number of units (hospitals) within the kth unit of the third level (country); and $N_{jk}$, $j=1,\dots ,N_k$, $k=1,\dots ,K$, denotes the number of subjects (patients) within a second-level unit (hospital) j of the third-level unit (country) k. Finally, let $n=\sum _{k=1}^K\sum _{j=1}^{N_k}{N}_{jk}$ be the total number of subjects in the data set.

Further, let $Y_{ijk}\in \{0,\dots ,L\}$ be the ordinal outcome with L + 1 levels measured on the ith patient in hospital j and country k. A natural approach for an ordinal outcome is to assume that it is derived from a continuous latent variable $Y_{ijk}^{\star }$ as follows:

$$Y_{ijk}=l,\text{iff }{\alpha }_l<Y_{ijk}^{\star }\le {\alpha }_{l+1},l=0,\dots ,L,$$

with unknown thresholds $\mathit{\alpha }=({\alpha }_1,\dots ,{\alpha }_L{)}^{\mathrm{\top }}$ such that $-\mathrm{\infty }={\alpha }_0<{\alpha }_1<\dots <{\alpha }_L<{\alpha }_{L+1}=\mathrm{\infty }$.

Motivated by our application, we assume that all main explanatory variables, confounders and mediators are available only at the hospital level. However, the methods presented in Rusá et al. [13] are directly applicable to variables measured on any higher level. Let ${\mathit{W}}_{jk}$, ${\mathit{C}}_{jk}$ and ${\mathit{M}}_{jk}$, respectively, be the vector of the main explanatory variables, confounders, mediators, for hospital j of country k, $j=1,\dots ,N_k$, $k=1,\dots ,K$.

To deal with the multi-level structure of the RN4CAST data, we included the country-level effects ( ${\beta }_k$) and hospital-level effects ( $U_{jk}$) in the model equation. Consequently, the mediation model for the latent outcome can now be summarized as ( $i=1,\dots ,N_{jk}$, $j=1,\dots ,N_k$, $k=1,\dots ,K$)

$$\begin{array}{rl}& Y_{ijk}^{\star }=H_{jk}+{\epsilon }_{ijk},\\ & H_{jk}={\beta }_k+{\mathit{\beta }}_C^{\mathrm{\top }}{\mathit{C}}_{jk}+{\mathit{\beta }}_W^{\mathrm{\top }}{\mathit{W}}_{jk}+{\mathit{\beta }}_M^{\mathrm{\top }}{\mathit{M}}_{jk}+U_{jk},\\ & {\mathit{M}}_{jk}={\mathit{\gamma }}_k+{\mathbf{\Gamma }}_W{\mathit{W}}_{jk}+{\mathit{\xi }}_{jk},\\ & U_{jk}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathcal{N}(0,{\sigma }_{\mathrm{hosp}}^2),{\epsilon }_{ijk}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathcal{N}(0,{\sigma }_{\epsilon }^2),{\mathit{\xi }}_{jk}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }{\mathcal{N}}_T(\mathbf{0},{\mathbf{\Sigma }}_{\xi }),\end{array}$$ (3)

where ${\beta }_k$, $k=1,\dots ,K,$ are the unknown country-specific intercepts, and ${\mathit{\beta }}_C$, ${\mathit{\beta }}_W$, ${\mathit{\beta }}_M$, ${\mathit{\gamma }}_k$, $k=1,\dots ,K$, ${\mathbf{\Gamma }}_W$ are unknown regression parameters, which will be further referred to as $\mathit{\beta }:=({\beta }_1,\dots ,{\beta }_K,{\mathit{\beta }}_C^{\mathrm{\top }},{\mathit{\beta }}_W^{\mathrm{\top }},{\mathit{\beta }}_M^{\mathrm{\top }}{)}^{\mathrm{\top }}$ and $\mathit{\gamma }:=\mathsf{\text{vec}}({\mathit{\gamma }}_1,\dots {\mathit{\gamma }}_K,{\mathbf{\Gamma }}_W)$. We assume that $U_{jk}$, ${\epsilon }_{ijk}$ and ${\mathit{\xi }}_{jk}$, $i=1,\dots ,N_{jk}$, $j=1,\dots ,N_k$, $k=1,\dots ,K$, are mutually independent.

2.2. The factorization of the posterior distribution

In the following, let ${\mathit{Y}}_{jk}=(Y_{1,jk},\dots ,Y_{N_{jk},jk}{)}^{\mathrm{\top }}$ be the vector of the ordinal outcome for all patients in hospital j of country k. We denote the total set of outcomes, mediator measurements and latent hospital-level means by $\mathit{Y}=\mathsf{\text{vec}}({\mathit{Y}}_{11},\dots ,{\mathit{Y}}_{N_{K}K})$, $\mathit{M}=\mathsf{\text{vec}}({\mathit{M}}_{11},\dots ,{\mathit{M}}_{N_{K}K})$, $\mathit{H}=(H_{11},\dots ,H_{N_{K}K}{)}^{\mathrm{\top }}$, respectively.

The vector  $\mathit{\theta }$ represents the unknown model parameters (without latent data $\mathit{H}$), which consists of (i) the thresholds $\mathit{\alpha }$, (ii) the measurement equation regression coefficients $\mathit{\beta }$, (iii) the mediation equations regression coefficients $\mathit{\gamma }$ and (iv) the variance parameters ${\sigma }_{\mathrm{hosp}}^2$, ${\sigma }_{\epsilon }^2$, ${\mathbf{\Sigma }}_{\xi }$. That is, $\mathit{\theta }=\mathsf{\text{vec}}(\mathit{\alpha },\mathit{\beta },{\sigma }_{\mathrm{hosp}}^2,{\sigma }_{\epsilon }^2,\mathit{\gamma },{\mathbf{\Sigma }}_{\xi })$. Furthermore, let ${\mathit{\theta }}_1=\mathsf{\text{vec}}(\mathit{\alpha },\mathit{\beta },{\sigma }_{\mathrm{hosp}}^2,{\sigma }_{\epsilon }^2)$ and ${\mathit{\theta }}_2=\mathsf{\text{vec}}(\mathit{\gamma },{\mathbf{\Sigma }}_{\xi })$, so $\mathit{\theta }=({\mathit{\theta }}_1,{\mathit{\theta }}_2{)}^{\mathrm{\top }}$.

Further, let $p(\cdot )$ and $p(\cdot |\cdot )$ be generic symbols for a density and a conditional density, respectively. As usual, the lower case letters denote the observed values of the corresponding upper case letter random variables. The most important property of our model is that certain subsets of parameters are a posteriori independent (provided they are independent a priori). In particular, in our case, this property holds for parameters ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ – if we assume that the parameters $\mathit{\beta }$, $\mathit{\gamma }$, ${\sigma }_{\epsilon }$, ${\sigma }_{\mathrm{hosp}}$, ${\mathbf{\Sigma }}_{\xi }$ and $\mathit{\alpha }$ are a priori independent then the conditional independence of some parameters and data and the prior independence implies:

$$\begin{array}{rl}p(\mathit{\theta },\mathbf{h}|\mathbf{y},\mathit{m})& \propto p(\mathbf{y}|\mathbf{h},\mathit{\alpha },{\sigma }_{\epsilon }^2)p(\mathbf{h}|\mathit{m},\mathit{\beta },{\sigma }_{\mathrm{hosp}}^2)p(\mathit{m}|\mathit{\gamma },{\mathbf{\Sigma }}_{\xi })\\ & \times p(\mathit{\beta })p(\mathit{\gamma })p({\sigma }_{\epsilon }^2)p({\sigma }_{\mathrm{hosp}}^2)p({\mathbf{\Sigma }}_{\xi })p(\mathit{\alpha })\\ & =[p(\mathbf{y}|\mathbf{h},\mathit{\alpha },{\sigma }_{\epsilon }^2)p(\mathbf{h}|\mathit{m},\mathit{\beta },{\sigma }_{\mathrm{hosp}}^2)p(\mathit{\beta })p({\sigma }_{\epsilon }^2)p({\sigma }_{\mathrm{hosp}}^2)p(\mathit{\alpha })]\\ & \times [p(\mathit{m}|\mathit{\gamma },{\mathbf{\Sigma }}_{\xi })p(\mathit{\gamma })p({\mathbf{\Sigma }}_{\xi })]\\ & \propto p({\mathit{\theta }}_1,\mathbf{h}|\mathbf{y},\mathit{m})p({\mathit{\theta }}_2|\mathit{m}).\end{array}$$ (4)

Hence, ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ are a posteriori independent. This property allows for the derivation of the importance sampling weights suitable for the efficient calculation of case-deleted posterior summary statistics based on a single MCMC run.

3. The detection of influential observations

The Bayesian analysis of model (3) was carried out in Rusá et al. [13]. Apart from the estimation of the parameters it may be of interest to assess the influence of some (sets of) observations on the parameter estimates. In our application, we aim to evaluate the influence on (i) the parameters in the measurement equation $\mathit{\beta }$, (ii) the parameters in the mediating equations $\mathit{\gamma }$, (iii) the indirect effects $\sum _{t=1}^T{\beta }_{M,t}{\mathit{\gamma }}_{t,W}$. To this end, importance sampling has been utilized in the literature, see, e.g. Bradlow and Zaslavsky [2]. The importance weights suggested by them will be employed for the parameters of the measurement and the mediation equations separately. In this paper, we are interested in the posterior distribution of $\mathit{\theta }$ given the data while leaving out a particular second-level observation. Hence, the term ‘case-deleted’ data will implicitly mean the complete data without an observation in the second level. In the following, $\{{\mathit{Y}}_{-jk},{\mathit{M}}_{-jk}\}$ denotes the data excluding the second-level observation jk.

3.1. Importance sampling and its use in the Bayesian model diagnostics

The posterior distribution of any function $f(\mathit{\theta })$ can be obtained from the posterior of $\mathit{\theta }$. The dissimilarity of the posterior distribution of $f(\mathit{\theta })$ computed from the full dataset $p(f(\mathit{\theta })|\mathbf{y},\mathit{m})$ and the posterior distribution based on all data but a second-level observation (here hospital jk) $p(f(\mathit{\theta })|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})$ indicates how influential that observation is. However, most often the posterior distribution cannot be derived analytically and must be estimated using MCMC. This was also done in Rusá et al. [13]. Such a comparison requires the computation of multivariate numerical integration. Bradlow and Zaslavsky [2] defined the difference of the posterior distributions via their posterior means, i.e. $\mathsf{\text{E}}[f(\mathit{\theta })|\mathit{Y},\mathit{M}]-\mathsf{\text{E}}[f(\mathit{\theta })|{\mathit{Y}}_{-jk},{\mathit{M}}_{-jk}]$, but also measures such as quantiles can be used. The case-deleted posterior means and hence the above difference can be obtained from the generated Markov chain to determine the posterior distribution based on all observations. This is done by reweighing the generated values of the parameters in such a way that the case-deleted posterior estimates are consistently estimated. This technique is based on importance sampling. Importance sampling implies that the case-deleted posterior $p(\mathit{\theta }|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})$ can be approximated from the full posterior distribution $p(\mathit{\theta }|\mathbf{y},\mathit{m})$ by resampling the elements of the original chain using the weight function

$$w_{jk}(\mathit{\theta },\mathbf{y},\mathit{m})=\frac{p(\mathit{\theta }|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})}{p(\mathit{\theta }|\mathbf{y},\mathit{m})}.$$

Then, the case-deleted posterior mean of an arbitrary function $f(\mathit{\theta })$ can be rewritten as

$$\begin{array}{rl}\mathsf{\text{E}}[f(\mathit{\theta })|{\mathit{Y}}_{-jk},{\mathit{M}}_{-jk}]& =\int f(\mathit{\theta })\frac{p(\mathit{\theta }|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})}{p(\mathit{\theta }|\mathbf{y},\mathit{m})}p(\mathit{\theta }|\mathbf{y},\mathit{m})\mathrm{d}\mathit{\theta }\\ & =\int f(\mathit{\theta })w_{jk}(\mathit{\theta },\mathbf{y},\mathit{m})p(\mathit{\theta }|\mathbf{y},\mathit{m})\mathrm{d}\mathit{\theta },\end{array}$$

implying that it can be estimated by the weighted sample mean

$$\frac{1}{S}\sum _{s=1}^{S}f({\mathit{\theta }}^{(s)})w_{jk}({\mathit{\theta }}^{(s)},\mathit{Y},\mathit{M}),$$

with ${\mathit{\theta }}^{(1)},\dots ,{\mathit{\theta }}^{(S)}$ the elements from the original chain and $w_{jk}({\mathit{\theta }}^{(s)},\mathit{Y},\mathit{M})$ normalized importance weights for estimating $p(\mathit{\theta }|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})$ at the s-th draw of the chain. From the Law of Large numbers it follows that this estimate is consistent if the original sample converges in distribution to the full data posterior. A similar approach is possible in case we know $p(\mathit{\theta }|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})$ and $p(\mathit{\theta }|\mathbf{y},\mathit{m})$ (or their ratio) up to a constant. Let us define such ${\tilde{w}}_{jk}$ as

$${\tilde{w}}_{jk}(\mathit{\theta },\mathbf{y},\mathit{m})\propto \frac{p(\mathit{\theta }|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})}{p(\mathit{\theta }|\mathbf{y},\mathit{m})},$$

then the so-called self-normalized importance sampling estimate

$$\frac{\sum _{s=1}^{S}f({\mathit{\theta }}^{(s)}){\tilde{w}}_{jk}({\mathit{\theta }}^{(s)},\mathit{Y},\mathit{M})}{\sum _{s=1}^S{\tilde{w}}_{jk}({\mathit{\theta }}^{(s)},\mathit{Y},\mathit{M})},$$ (5)

is a consistent estimate of $\mathsf{\text{E}}[f(\mathit{\theta })|{\mathit{Y}}_{-jk},{\mathit{M}}_{-jk}]$. In the remainder of this paper, we will sometimes use a more concise notation $w_{jk}^{(s)}=w_{jk}({\theta }^{(s)},\mathit{Y},\mathit{M})$ or ${\tilde{w}}_{jk}^{(s)}={\tilde{w}}_{jk}({\theta }^{(s)},\mathit{Y},\mathit{M})$ respectively. Furthermore, we can observe that the estimate (5) corresponds to the estimate with normalized weights with $w_{jk}^{(s)}={\tilde{w}}_{jk}^{(s)}/\sum _{r=1}^N{\tilde{w}}_{jk}^{(r)}$.

Bradlow and Zaslavsky [2] showed that the ratio $p(\mathit{\theta }|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})/p(\mathit{\theta }|\mathbf{y},\mathit{m})$ is proportional to $[p({\mathbf{y}}_{jk},{\mathit{m}}_{jk}|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk},\mathit{\theta }){]}^{-1}$ so we can define the non-normalized importance weights ${\tilde{w}}_{jk}^{(n)}$ to be equal to these weights and utilize the estimate (5). Nevertheless, following Bradlow and Zaslavsky [2], in some models (e.g. in hierachical models) we can suggest more suitable importance weights than these elementary weights. Such weights are often based on some convenient factorization of the posterior distribution and are especially useful when some parameters are considered as nuisance parameters. To be more specific, this approach may be particularly suitable when data augmentation is utilized because it often leads to a more simple and computationally less intensive form of the importance weights. However, we rarely want to infer about the augmented data so it makes sense to ‘integrate them out’ from the importance weights.

3.2. The importance weights for $p(h(\mathit{\eta })|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})$

In hierarchical models, we can often divide the parameters to a group of ‘interesting’ parameters $\mathit{\eta }$ and a group of ‘nuisance’ parameters $\mathit{\psi }$ (e.g. augmented data) and consider a factorization $\mathit{\theta }=({\mathit{\eta }}^{\mathrm{\top }},{\mathit{\psi }}^{\mathrm{\top }}{)}^{\mathrm{\top }}$. In our application, we might utilize three different partitions of $\mathit{\theta }$ dictated by our intention to study the 3 different effects aforementioned in the model: (i) $\mathit{\eta }={\mathit{\theta }}_1$, $\mathit{\psi }={\mathit{\theta }}_2$ in case we want to infer about the parameters in the measurement equation only, (ii) $\mathit{\eta }={\mathit{\theta }}_2$, $\mathit{\psi }={\mathit{\theta }}_1$ in case we want to infer about the parameters in the mediating equations, (iii) $\mathit{\eta }=\mathit{\theta }$, $\mathit{\psi }=\{\}$ in the general case. In general, we are interested in the estimation of the case-deleted posterior distribution of a function of the ‘interesting’ parameters $\mathit{\eta }$, $f(\mathit{\eta })$.

In Equation (4) we showed that the parameters from the measurement equation ${\mathit{\theta }}_1$ and the parameters from the mediation equation ${\mathit{\theta }}_2$ are a posteriori independent. Consequently, the influence diagnostics of the two parts of the model can be carried out separately (forgetting the other part of the model) and then combined later in order to assess which part of the model is influenced more. In the following, we will present the importance weights for (i) the parameters in the measurement equation (Section 3.3), (ii) the mediation equation (Section 3.4) and (iii) the indirect effect or any other function of the parameters in both the measurement and the mediation equations (Section 3.5). Furthermore, in Section 3.6 we show that the higher-level importance weights can be obtained as the product of the relevant lower-level importance weights.

3.3. The importance weights for the parameters in the measurement equation

Let $\mathit{\eta }={\mathit{\theta }}_1$ and $\mathit{\psi }={\mathit{\theta }}_2$, where ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ are the parameters in the measurement equation and mediation equations, respectively. From the following factorization of the posterior distribution of ${\mathit{\theta }}_1$, we can derive the suitable weights suggested by Bradlow and Zaslavsky [2]:

$$p({\mathit{\theta }}_1,\mathbf{h}|\mathbf{y},\mathit{m})\propto p({\mathbf{y}}_{jk}|h_{jk},{\mathit{\theta }}_1)p(h_{jk}|{\mathit{m}}_{jk},{\mathit{\theta }}_1)p({\mathit{\theta }}_1,{\mathbf{h}}_{-jk}|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk}).$$ (6)

In their terminology, we set ${\mathit{\theta }}_{jk}=H_{jk}$, ${\mathit{\theta }}_{-jk}=\mathsf{\text{vec}}({\mathit{\theta }}_1,{\mathit{H}}_{-jk})$. The purpose of this factorization is to rewrite the posterior distribution $p({\mathit{\theta }}_1,\mathbf{h}|\mathbf{y},\mathit{m})$ as the product of the posterior distribution of the parameters ${\mathit{\theta }}_1$ and the augmented data ${\mathbf{h}}_{-jk}$ given the case-deleted data $p({\mathit{\theta }}_1,{\mathbf{h}}_{-jk}|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})$ and some other factors.

The structure of the desired importance weights is such that it ensures the consistency of the importance weighted estimate. In the denominator we put the first two factors of the right side of Equation (6), whereas in the numerator we can utilize an arbitrary positive function of the augmented data $h_{jk}$ and the data $g(h_{jk},\mathbf{y},\mathit{m})$. On the basis of this motivation, we can define the importance weights as

$$w_{jk}^{(1)}({\mathit{\theta }}_1,h_{jk},\mathbf{y},\mathit{m})={\displaystyle \frac{g(h_{jk},\mathbf{y},\mathit{m})}{p({\mathbf{y}}_{jk}|h_{jk},{\mathit{\theta }}_1)p(h_{jk}|{\mathit{m}}_{jk},{\mathit{\theta }}_1)}},$$ (7)

for some $g(h_{jk},\mathbf{y},\mathit{m})>0$. From Equation (7), we see that the estimate based on these importance weights, computed as in Equation (5), converges in probability to

$$\begin{array}{rl}& \frac{\int \int f({\mathit{\theta }}_1)w_{jk}^{(1)}({\mathit{\theta }}_1,h_{jk},\mathbf{y},\mathit{m})p({\mathit{\theta }}_1,\mathbf{h}|\mathbf{y},\mathit{m})\mathrm{d}{\mathit{\theta }}_1\mathrm{d}\mathbf{h}}{\int \int w_{jk}^{(1)}({\mathit{\theta }}_1,h_{jk},\mathbf{y},\mathit{m})p({\mathit{\theta }}_1,\mathbf{h}|\mathbf{y},\mathit{m})\mathrm{d}{\mathit{\theta }}_1\mathrm{d}\mathbf{h}}\\ & =\frac{\int \int f({\mathit{\theta }}_1)g(h_{jk},\mathbf{y},\mathit{m})p({\mathit{\theta }}_1,{\mathbf{h}}_{-jk}|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})\mathrm{d}{\mathit{\theta }}_1\mathrm{d}\mathbf{h}}{\int \int g(h_{jk},\mathbf{y},\mathit{m})p({\mathit{\theta }}_1,{\mathbf{h}}_{-jk}|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})\mathrm{d}{\mathit{\theta }}_1\mathrm{d}\mathbf{h}}\\ & =\frac{\int \int \int [f({\mathit{\theta }}_1)p({\mathit{\theta }}_1,{\mathbf{h}}_{-jk}|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})]g(h_{jk},\mathbf{y},\mathit{m})\mathrm{d}{\mathit{\theta }}_1\mathrm{d}{\mathbf{h}}_{-jk}\mathrm{d}{h}_{jk}}{\int \int \int p({\mathit{\theta }}_1,{\mathbf{h}}_{-jk}|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})g(h_{jk},\mathbf{y},\mathit{m})\mathrm{d}{\mathit{\theta }}_1\mathrm{d}{\mathbf{h}}_{-jk}\mathrm{d}{h}_{jk}}\\ & =\mathsf{\text{E}}[f({\mathit{\theta }}_1)|{\mathit{Y}}_{-jk},{\mathit{M}}_{-jk}],\end{array}$$

as long as $\int g(h_{jk},\mathbf{y},\mathit{m})\mathrm{d}{h}_{jk}<\mathrm{\infty }$.

Generally, it is a good strategy to choose g such that the factors in g and the denominator of the importance weights (7) (almost) cancel out (Bradlow and Zaslavsky [2]). To be more specific, we replace the parameter ${\mathit{\theta }}_1$ in all the terms of the denominator of Equation (7) by a suitable constant (depending on the iteration in chain), such as the posterior mean ${\mathit{\theta }}_1^{\star }$. Also the replacement of the target parameters ${\mathit{\theta }}_1$ by its posterior mean was first suggested by Bradlow and Zaslavsky [2]. Hence, we define

$$g(h_{jk},\mathbf{y},\mathit{m})=p({\mathbf{y}}_{jk}|h_{jk},{\mathit{\theta }}_1^{\star })p(h_{jk}|{\mathit{m}}_{jk},{\mathit{\theta }}_1^{\star }).$$ (8)

Consequently, the importance weights for the parameters in the measurement equation of model (3) can be defined as

$$w_{jk}^{(1)}({\mathit{\theta }}_1,h_{jk},\mathbf{y},\mathit{m})={\displaystyle \frac{p({\mathbf{y}}_{jk}|h_{jk},{\sigma }_{\epsilon }^{2\star },{\mathit{\alpha }}^{\star })p(h_{jk}|{\mathit{\beta }}^{\star },{\sigma }_{\mathrm{hosp}}^{2\star },{\mathit{m}}_{jk})}{p({\mathbf{y}}_{jk}|h_{jk},{\sigma }_{\epsilon }^2,\mathit{\alpha })p(h_{jk}|\mathit{\beta },{\sigma }_{\mathrm{hosp}}^2,{\mathit{m}}_{jk})}}.$$ (9)

We can now see that the importance weights (9) have similar terms in the nominator and denominator and these almost cancel out by which means we eliminate the variability of the importance weights caused by $h_{jk}$.

3.4. The importance weights for the parameters in the mediation equations

Now, we will consider the importance weights for the case-deleted posterior distribution of the parameters in the mediation equations ( $\mathit{\eta }={\mathit{\theta }}_2$ and $\mathit{\psi }={\mathit{\theta }}_1$). In this case, there are no augmented data so we can use the simple importance weights. To be specific,

$$w_{jk}^{(2)}({\mathit{\theta }}_2,\mathit{m})=[p({\mathit{m}}_{jk}|\mathit{\gamma },{\mathbf{\Sigma }}_{\xi }){]}^{-1}.$$ (10)

3.5. The importance weights for an arbitrary function of parameters in the mediation model

In the most general case we are interested in an arbitrary function of $\mathit{\theta }$, $f(\mathit{\theta })$, e.g. the indirect effect $f(\mathit{\theta })={\mathit{\beta }}_M^{\mathrm{\top }}{\mathbf{\Gamma }}_{W,r}$, so in this section, we consider a scenario with $\mathit{\eta }=\mathit{\theta }$ and $\mathit{\psi }=\{\}$. The importance weights can be computed as the product of the importance weights for the measurement equation $w_{jk}^{(1)}({\mathit{\theta }}_1,h_{jk},\mathbf{y},\mathit{m})$ and the importance weights for the mediation equations $w_{jk}^{(2)}({\mathit{\theta }}_2,\mathit{m})$, that is

$$w_{jk}^{(1,2)}(\mathit{\theta },h_{jk},\mathbf{y},\mathit{m})=w_{jk}^{(1)}({\mathit{\theta }}_1,h_{jk},\mathbf{y},\mathit{m})w_{jk}^{(2)}({\mathit{\theta }}_2,\mathit{m}).$$ (11)

This follows from

$$\begin{array}{rl}& \frac{\int \int f(\mathit{\theta })w_{jk}^{(1)}({\mathit{\theta }}_1,h_{jk},\mathbf{y},\mathit{m})w_{jk}^{(2)}({\mathit{\theta }}_2,\mathit{m})p(\mathit{\theta },\mathbf{h}|\mathbf{y},\mathit{m})\mathrm{d}\mathit{\theta }\mathrm{d}\mathbf{h}}{\int \int w_{jk}^{(1)}({\mathit{\theta }}_1,h_{jk},\mathbf{y},\mathit{m})w_{jk}^{(2)}({\mathit{\theta }}_2,\mathit{m})p(\mathit{\theta },\mathbf{h}|\mathbf{y},\mathit{m})\mathrm{d}\mathit{\theta }\mathrm{d}\mathbf{h}}\\ & =\frac{\int \int \int [f(\mathit{\theta })p(\mathit{\theta },{\mathbf{h}}_{-jk}|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})]g(h_{jk},\mathbf{y},\mathit{m})\mathrm{d}\mathit{\theta }\mathrm{d}{\mathbf{h}}_{-jk}\mathrm{d}{h}_{jk}}{\int \int \int p(\mathit{\theta },{\mathbf{h}}_{-jk}|{\mathbf{y}}_{-jk},{\mathit{m}}_{-jk})g(h_{jk},\mathbf{y},\mathit{m})\mathrm{d}\mathit{\theta }\mathrm{d}{\mathbf{h}}_{-jk}\mathrm{d}{h}_{jk}}\\ & =\mathsf{\text{E}}[f(\mathit{\theta })|{\mathit{Y}}_{-jk},{\mathit{M}}_{-jk}].\end{array}$$

Thus, the importance weights for $\mathit{\eta }=\mathit{\theta }$ are given by

$$w_{jk}^{(1,2)}(\mathit{\theta },h_{jk},\mathbf{y},\mathit{m})={\displaystyle \frac{p({\mathbf{y}}_{jk}|h_{jk},{\sigma }_{\epsilon }^{2\star },{\mathit{\alpha }}^{\star })p(h_{jk}|{\mathit{\beta }}^{\star },{\sigma }_{\mathrm{hosp}}^{2\star },{\mathit{m}}_{jk})}{p({\mathbf{y}}_{jk}|h_{jk},{\sigma }_{\epsilon }^2,\mathit{\alpha })p(h_{jk}|\mathit{\beta },{\sigma }_{\mathrm{hosp}}^2,{\mathit{m}}_{jk})p({\mathit{m}}_{jk}|\mathit{\gamma },{\mathbf{\Sigma }}_{\xi })}}.$$ (12)

Summarized, we have proposed suitable importance weights to evaluate the influence of the hospitals on the posterior distribution of parameters of interest. In case (i) ${\mathit{\theta }}_1$ is of interest, we choose the importance weights $w_{jk}^{(1)}({\mathit{\theta }}_1,h_{jk},\mathbf{y},\mathit{m})$ defined in Equation (9), (ii) ${\mathit{\theta }}_2$ is of interest, we choose the importance weights $w_{jk}^{(2)}({\mathit{\theta }}_2,\mathit{m})$ defined in Equation (10), (iii) an arbitrary function $f(\mathit{\theta })$ is of interest, we choose the importance weights $w_{jk}^{(1,2)}(\mathit{\theta },h_{jk},\mathbf{y},\mathit{m})$ defined in Equation (12).

3.6. The importance weights for higher levels of the data

For the multi-level setting, as in our example, the choice of suitable importance weights facilitates the utilization of the case-deletion diagnostics at any level of the data. Having proposed importance weights for deletion of observations at level two, we could now determine the weighting scheme for level three deletion. It is then clear that the same reasoning applies for the higher levels, if any, of the data. It is straightforward to show that the importance weights for level three deletion can be computed from importance weights for level two deletion. Similarly as in the previous section, we can condition by ${\mathit{H}}_k=(H_{1k},\dots ,H_{N_k,k})$ and set $\mathit{\psi }={\mathit{\theta }}_2$ for case-deletion in the measurement equation and $\mathit{\psi }={\mathit{\theta }}_1$ for case-deletion in the mediation equations.

Further, as in the factorization for a specific second-level observation (6), we have

$$p({\mathit{\theta }}_1,\mathbf{h}|\mathbf{y},\mathit{m})\propto p({\mathbf{y}}_k|{\mathbf{h}}_k,{\mathit{\theta }}_1)p({\mathbf{h}}_k|{\mathit{m}}_k,{\mathit{\theta }}_1)p({\mathit{\theta }}_1,{\mathbf{h}}_{-k}|{\mathbf{y}}_{-k},{\mathit{m}}_{-k}),$$

so we can see that the only difference between the level two and level three deletion is that instead of one term $p({\mathbf{y}}_{jk}|h_{jk},{\mathit{\theta }}_1)$ and $p(h_{jk}|{\mathit{m}}_{jk},{\mathit{\theta }}_1)$, respectively, we get the products across the second level units in the third level unit $\prod _{j=1}^{N_k}p({\mathbf{y}}_{jk}|h_{jk},{\mathit{\theta }}_1)$ and $\prod _{j=1}^{N_k}p(h_{jk}|{\mathit{m}}_{jk},{\mathit{\theta }}_1)$, respectively. The same reasoning can be applied to the importance weights for the parameters of mediation equations. Therefore, the third-level deletion importance weights can be defined as

$$\begin{array}{rl}{w}_k^{(1)}({\mathit{\theta }}_1,{\mathbf{h}}_{jk},\mathbf{y},\mathit{m})& =\prod _{j=1}^{N_k}{\displaystyle \frac{p({\mathbf{y}}_{jk}|h_{jk},{\sigma }_{\epsilon }^{2\star },{\mathit{\alpha }}^{\star })p(h_{jk}|{\mathit{\beta }}^{\star },{\sigma }_{\mathrm{hosp}}^{2\star },{\mathit{m}}_{jk})}{p({\mathbf{y}}_{jk}|h_{jk},{\sigma }_{\epsilon }^2,\mathit{\alpha })p(h_{jk}|\mathit{\beta },{\sigma }_{\mathrm{hosp}}^2,{\mathit{m}}_{jk})}}\\ & =\prod _{j=1}^{N_k}{w}_{jk}^{(1)}({\mathit{\theta }}_1,h_{jk},\mathbf{y},\mathit{m}),\\ w_k^{(2)}({\mathit{\theta }}_2,\mathit{m})& =\prod _{j=1}^{N_k}{\displaystyle \frac{1}{p({\mathit{m}}_{jk}|\mathit{\gamma },{\mathbf{\Sigma }}_{\xi })}}\\ & =\prod _{j=1}^{N_k}{w}_{jk}^{(2)}({\mathit{\theta }}_2,\mathit{m}),\\ w_k^{(1,2)}(\mathit{\theta },{\mathbf{h}}_{jk},\mathbf{y},\mathit{m})& =\prod _{j=1}^{N_k}{\displaystyle \frac{p({\mathbf{y}}_{jk}|h_{jk},{\sigma }_{\epsilon }^{\star 2},{\mathit{\alpha }}^{\star })p(h_{jk}|{\mathit{\beta }}^{\star },{\sigma }_{\mathrm{hosp}}^{2\star },{\mathit{m}}_{jk})}{p({\mathbf{y}}_{jk}|h_{jk},{\sigma }_{\epsilon }^2,\mathit{\alpha })p({\mathit{m}}_{jk}|\mathit{\gamma },{\mathbf{\Sigma }}_{\xi })p(h_{jk}|\mathit{\beta },{\sigma }_{\mathrm{hosp}}^2,{\mathit{m}}_{jk})}}\\ & =\prod _{j=1}^{N_k}{w}_{jk}^{(1,2)}(\mathit{\theta },h_{jk},\mathbf{y},\mathit{m}).\end{array}$$

To summarize, we can compute the level three deleted importance weights as the product of the corresponding importance weights of level two units in the level three unit.

3.7. The diagnostic plot for the detection of influential observations in mediation models

Having derived suitable importance weights for different parts of the mediation model, we now aim to find the most influential ones. We will be interested in the observations with the highest variance of the log importance weights with respect to (i) the measurement part of the model; (ii) the mediation part of the model and (iii) the model generally. In the following, the importance weights $w_{jk}^{(1,2)}(\mathit{\theta },h_{jk},\mathbf{y},\mathit{m})$, $w_{jk}^{(1)}({\mathit{\theta }}_1,h_{jk},\mathbf{y},\mathit{m})$ and $w_{jk}^{(2)}({\mathit{\theta }}_2,\mathit{m})$, respectively, will be denoted more concisely as $w_{jk}^{(1,2)}$, $w_{jk}^{(1)}$ and $w_{jk}^{(2)}$, respectively.

We will utilize Equation (11) that states that the joint importance weights $w_{jk}^{(1,2)}$ can be computed as the product of the importance weights for the measurement equation $w_{jk}^{(1)}$ and the importance weights for the mediation equations $w_{jk}^{(2)}$ or equivalently, that $\log w_{jk}^{(1,2)}=\log w_{jk}^{(1)}+\log w_{jk}^{(2)}$. Moreover, the importance weights $w_{jk}^{(1)}$ and $w_{jk}^{(2)}$ are a posteriori independent so

$$\mathsf{\text{var}}(\log w_{jk}^{(1,2)}\mid \mathit{Y},\mathit{M})=\mathsf{\text{var}}(\log w_{jk}^{(1)}|\mathit{Y},\mathit{M})+\mathsf{\text{var}}(\log w_{jk}^{(2)}|\mathit{Y},\mathit{M}).$$

Therefore it is possible to determine which part of the model is influenced by observation (hospital) jk, if any. As a diagnostic tool, we propose to draw a plot with the values of $\mathsf{\text{var}}(\log w_{jk}^{(1)}|\mathit{Y},\mathit{M})$ on the x-axis and $\mathsf{\text{var}}(\log w_{jk}^{(2)}|\mathit{Y},\mathit{M})$ on the y-axis for all hospitals (that is $j=1,\dots ,N_k$, $k=1,\dots ,K$). From this plot it is straightforward to see the influence on the whole model as well as which part of the model is influenced most. From this graphical check one can identify the most influential cases and investigate them further, e.g. run a new MCMC chain based on the data set without that observation.

For the detection of case influence on the higher (country) level, one can use the same method with the corresponding importance weights $w_k^{(1)}$, $w_k^{(2)}$ and $w_k^{(1,2)}$ defined in Section 3.6.

4. Application on RN4CAST data

4.1. Prior specification

The parameters $\mathit{\beta }$, $\mathit{\gamma }$, ${\sigma }_{\epsilon }$, ${\sigma }_{\mathrm{hosp}}$, ${\mathbf{\Sigma }}_{\xi }$ and $\mathit{\alpha }$ were assumed to be a priori independent. Normal priors were specified for $\mathit{\beta }$ and $\mathit{\gamma }$, proper uniform priors on $[0,100]$ were specified for ${\sigma }_{\epsilon }$ and ${\sigma }_{\mathrm{hosp}}$. The covariance matrix ${\mathbf{\Sigma }}_{\xi }$ was parametrized using the variance of ${\xi }_{1,jk}$ denoted by ${\sigma }_{\xi 1}^2$, the variance of ${\xi }_{2,jk}$ denoted by ${\sigma }_{\xi 2}^2$ and their correlation ρ. For the standard deviations ${\sigma }_{\xi 1},{\sigma }_{\xi 2}$, we also specified proper uniform priors on $[0,100]$. A uniform prior $[-1,1]$ was considered for ρ: $\rho \sim U(-1,1)$. To ensure the identification of the thresholds α and the mean and the variance of the latent variable, we fix the extreme thresholds at preassigned values ${\alpha }_1=-1$ and ${\alpha }_L=1$. The following non-informative prior was used for the thresholds:

$$p({\alpha }_2,\dots ,{\alpha }_{L-1})\propto C,\text{for }{\alpha }_1<{\alpha }_2<\dots <{\alpha }_{L-1}<{\alpha }_L.$$

4.2. The influence diagnostics

The model suggested in Rusá et al. [13] aimed to find the effect of nurse staffing, the quality of nurse work environment and other hospital characteristics ${\mathit{W}}_{jk}$ on patient outcome $Y_{ijk}$. In addition, it was of interest to see if this effect was mediated by two measures of care left undone ${\mathit{M}}_{jk}$. In this section, we investigate whether the model estimates are too dependent on some hospitals or countries, i.e. whether they are stable with respect to deleting hospitals or countries.

We ran 2 Markov chain cycles, each with 20,000 samples after 10,000 burn-in samples, so in our case S = 40,000. Then we computed the importance weights $w_{jk}^{(1,2)(s)}$, $w_{jk}^{(1)(s)}$ and $w_{jk}^{(2)(s)}$ for $s=1,\dots ,S$, $j=1,\dots ,N_k$, $k=1,\dots ,K$ by plugging in the posterior means of ${\mathit{\theta }}_1$ in the function g defined in Equation (8). Next, we computed the variance of the log weights $\mathsf{\text{var}}(\log w_{jk}^{(1)}|\mathit{Y},\mathit{M})$, $\mathsf{\text{var}}(\log w_{jk}^{(2)}|\mathit{Y},\mathit{M})$ and $\mathsf{\text{var}}(\log w_{jk}^{(1,2)}|\mathit{Y},\mathit{M})$. The diagnostic plot of $\mathsf{\text{var}}(\log w_{jk}^{(1)}|\mathit{Y},\mathit{M})$ on the x-axis and $\mathsf{\text{var}}(\log w_{jk}^{(2)}|\mathit{Y},\mathit{M})$ on the y-axis suggested in Section 3.7 is shown in Figure 1.

Figure 1.

The hospital-level diagnostic plot with the variance of the log importance weights for the measurement equation $\mathsf{\text{var}}(\log w_{jk}^{(1)}|\mathit{Y},\mathit{M})$ on the x-axis and the variance of the log importance weights for the mediation equation $\mathsf{\text{var}}(\log w_{jk}^{(2)}|\mathit{Y},\mathit{M})$ on the y-axis for all hospitals ( $j=1,\dots ,N_k$, $k=1,\dots ,K$). The overall influence on the model is determined by the sum of these two values and can be visually compared with the diagonal lines.

Several hospitals, namely hospitals with number 147, 151, 152, 156, 158 and 160 stand out in the plot. From these, hospitals 152 and 158 influence mainly the measurement part of the model, while hospitals 147, 151, 156 and 160 influence the mediation part of the model.

With regard to the country level, we can see from Figure 2 that the most influential country is Greece, which seems to influence both parts of the model. This is not surprising since most of the outlying hospitals from Figure 1 are from Greece. Belgium is the second most influential with respect to the mediation part of the model.

Figure 2.

The country-level diagnostic plot with the variance of the log importance weights for the measurement equation $\mathsf{\text{var}}(\log w_k^{(1)}|\mathit{Y},\mathit{M})$ on the x-axis and the variance of the log importance weights for the mediation equation $\mathsf{\text{var}}(\log w_k^{(2)}|\mathit{Y},\mathit{M})$ on the y-axis for all hospitals ( $k=1,\dots ,K$). The overall influence on the model is determined by the sum of these two values and can be visually compared with the diagonal lines.

However, it is desirable to check the usefulness of the diagnostic plots. To this end, we fitted 14 models with case-deleted data (which we would normally want to avoid). To be specific, we ran new MCMC chains while removing hospitals 147, 151, 152, 156, 158 and 160 and all 8 countries from the data one at a time. For each element of $\mathit{\beta }$ and $\mathit{\gamma }$ we computed the difference of the estimates of the posterior mean of the parameter and standardized it by the estimated posterior standard deviation of the parameter based on complete data. The standardized differences are shown in Figure 3, where the parameters are aggregated in two groups given by the part of the model. Elements of $\mathit{\beta }$ from the measurement part of the model are part of a group abbreviated as Meas. in the plot, whereas the parameters from the mediation equations $\mathit{\gamma }$ are grouped together and abbreviated as Med.

Figure 3.

The standardized difference of the posterior means of the elements of $\mathit{\beta }$ and $\mathit{\gamma }$ based on complete data and based on a new MCMC chain without hospital for the hospitals marked as possibly influential (i.e. $jk\in \{147,151,152,156,158,160\}$). On the x-axis, we distinguish between elements of $\mathit{\beta }$ (measurement equation) and $\mathit{\gamma }$ (mediation equations).

In accordance with the diagnostic plot (Figure 1) we can see that for hospitals 147, 151, 156 and 160 the standardized differences are generally higher for the elements of $\mathit{\gamma }$ corresponding to mediation equations (Med.). For hospitals 152 and 158, the standardized differences for the elements of $\mathit{\beta }$ are higher. This also corresponds to the diagnostic plot.

The standardized differences for the countries are depicted in Figure 4. Greece seems to be the most influential country and the influence seems to stem from both parts of the model which corresponds to the diagnostic plot (Figure 2). The standardized differences are relatively high for the parameters in the mediation equations (Med.) for Belgium. Thus, we conclude that our diagnostic plot gives the same information as when removing the hospitals and countries effectively.

Figure 4.

The standardized difference of the posterior means of the elements of $\mathit{\beta }$ and $\mathit{\gamma }$ based on complete data and based on a new MCMC chain without country for all countries (i.e. $k\in \{1,\dots ,8\}$). On the x-axis, we distinguish between elements of $\mathit{\beta }$ (measurement equation) and $\mathit{\gamma }$ (mediation equations).

Next, we compared the conclusions of our method above with the traditional approach of the estimation of case-deleted estimates using importance sampling weights (5). In Section 1.1, we discussed a possible drawback of the traditional approach, namely that the variance of the importance weights can be too large. The importance weighted estimate may therefore be dominated by a few samples. In our case, the maximal importance weights were quite large ( $\underset{s}{max}{w}_{IE}^{(1)(s)}\doteq 0.71$ for Ireland measurement part of the model, $\underset{s}{max}{w}_{FI}^{(2)(s)}\doteq 0.84$ for Finland mediation part). The standardized differences of the country-deleted importance weighted estimates and the posterior means are shown in Figure 5. According to the traditional approach, Finland is marked as influential in the mediation part and Ireland in the measurement part. Looking at Figure 2 we can realize this is not true – the countries were marked as influential due to the high variability of the importance weights.

Figure 5.

The standardized difference of the posterior means of the elements of $\mathit{\beta }$ and $\mathit{\gamma }$ based on complete data and country-deleted importance weighted estimates for all countries (i.e. $k\in \{1,\dots ,8\}$). On the x-axis, we distinguish between elements of $\mathit{\beta }$ (measurement equation) and $\mathit{\gamma }$ (mediation equations).

5. Discussion

In this paper, we combined the approach Bradlow and Zaslavsky [2] with that in Thomas et al. [15] for the detection of influential observations in a complex mediation model. We have proposed a general method for the evaluation of the influence of observations on any level of hierarchy in multi-level data which is applicable, e.g. for complex Bayesian mediation models. Due to our application, we assumed the errors in our model were normally distributed but the method would be applicable if their distribution were non-normal.

The diagnostics are based on a suitable plot of variance of the log importance weights with the measurement part on the x-axis and the mediation part on the y-axis. The overall influence can also be determined from the plot. The software used for the computation of the importance weights, which also produces the diagnostic plots, is briefly introduced in the Appendix and is available as supplementary material.

The method was applied on a data set from nursing research. The methodology makes use of the latent data in our model which can be conditioned by and used to propose easy to compute importance weights. Moreover, we are able to investigate the influence on subsets of parameters, thereby ignoring the rest of the parameters. This is done by making use of conditional independences in hierarchical models, which leads to simple but at the same time effective importance weights. Another advantage of our approach is that it can be directly generalized to assess the influence of a particular group of observations because the joint importance weights can be obtained easily as the product of the individual importance weights.

Our method can be applied to various mediation models, which satisfy the condition that the parameters in the measurement equation and the parameters in the mediation equations are a posteriori independent. As long as this assumption holds, the influence can be assessed at all levels of the data.

We advocate that the model diagnostics, or more specifically influence diagnostics, be carried out routinely also in other complex Bayesian models. The proposed general importance weights are computationally feasible so the applied researchers can compute the relevant importance weights in real time and plot the variance of the log weights of the measurement equation and the mediation part of the model against each other.

Funding Statement

The work on this paper was supported by the Czech Science Foundation grant GAČR 19-00015S and GAUK 1583317.

Disclosure statement

No potential conflict of interest was reported by the author(s).