On the bias of complete‐ and shifting‐case meta‐regressions with missing covariates

Abstract

Missing covariates is a common issue when fitting meta‐regression models. Standard practice for handling missing covariates tends to involve one of two approaches. In a complete‐case analysis, effect sizes for which relevant covariates are missing are omitted from model estimation. Alternatively, researchers have employed the so‐called "shifting units of analysis" wherein complete‐case analyses are conducted on only certain subsets of relevant covariates. In this article, we clarify conditions under which these approaches generate unbiased estimates of regression coefficients. We find that unbiased estimates are possible when the probability of observing a covariate is completely independent of effect sizes. When that does not hold, regression coefficient estimates may be biased. We study the potential magnitude of that bias assuming a log‐linear model of missingness and find that the bias can be substantial, as large as Cohen's d = 0.4–0.8 depending on the missingness mechanism.

Highlights.

Missing covariates are a common problem when conducting meta‐regressions. A common practice for meta‐regression analyses has been to ignore effects for which covariates are missing. However, a vast statistical literature suggests that analyses that ignore missing data can only provide accurate estimates of relevant quantitites under certain conditions. In this article, we examine conditions under which ignoring missing covariates in a meta‐regression can still lead to unbiased estimation of regression coefficients. We also investigate the possible magnitude and sources of bias when those conditions do not hold. Our findings highlight that substantial bias can be induced by ignoring missing data in a meta‐regression.

1. INTRODUCTION

Meta‐regression is a useful tool for studying important sources of variation between effects in a meta‐analysis. ¹ , ² Analyses of these models in the absence of missing data have been studied thoroughly in the literature. ³ , ⁴ , ⁵ , ⁶ , ⁷ However, it is common for meta‐analytic datasets to be missing data. ⁸ In the context of meta‐regression, issues with missing data frequently involve missing covariates. ⁹ , ¹⁰

Precisely how to proceed with a meta‐regression when missing covariates remains something of an open question. Statistical guidance suggests that analyses ought to consider the mechanism that causes covariates to be missing. ⁹ , ¹¹ However, it appears that doing so is less common in practice for meta‐analyses. A recent review found that meta‐regressions with missing data tend to take one of two strategies. ¹⁰ An analyst may conduct a complete‐case analysis (CCA) that excludes any effects for which a relevant covariate is missing (i.e., only analyze complete cases). This is often referred to as “listwise deletion” in data analyses. However, if there are very few such effects, a common approach is to use shifting units of analysis, which we refer to in this article as a shifting‐case analysis (SCA). ¹² In an SCA, analysts fit a series of meta‐regression models on subsets of relevant covariates, so that each model selectively omits certain covariates. This is equivalent to “pairwise deletion” in data analyses.

Both CCA and SCA ignore effects for which a covariate is missing. Ignoring missing data can potentially lead to biased estimates of parameters of interest. ¹³ , ¹⁴ Despite authors pointing out such issues in meta‐analysis, these methods continue to enjoy widespread use. ¹¹ Existing meta‐analysis literature on this discussion has yet to detail precisely how much bias can arise in a complete‐ or shifting‐case analysis, nor is there exhaustive guidance on when these methods produce unbiased estimates. In short, there is an understanding that these methods can induce bias, but less is known about how much and under what conditions.

This article examines the potential bias of complete‐ and shifting‐case analyses. The following section provides a demonstration of these methods on data concerning a meta‐analysis of substance abuse interventions. ¹⁵ We then introduce a statistical framework for studying bias for incomplete data meta‐regressions that incorporates a model for whether or not a covariate is observed. Using this framework, we describe conditions under which CCA and SCA are unbiased. When these conditions are not met, we derive an approximation for the bias of CCA and SCA using standard models for missingness and examine the magnitude of bias. We find that bias is highly dependent on the precise mechanism by which data are missing, and is less reliant on more traditional missingness mechanism classifications (e.g., missing at random vs. not at random).

2. EXAMPLE: SUBSTANCE ABUSE INTERVENTIONS

Tanner‐Smith et al. ¹⁵ conducted a meta‐analysis that examined the effects of substance abuse interventions on future substance use among adolescents. The studies included in this meta‐analysis involved a variety of different treatment types (e.g., cognitive behavioral therapy, family therapy, and pharmacological therapy) and treatment intensities (measured in hours per week), and were carried out in a variety of contexts, including in‐patient and out‐patient centers. Tanner‐Smith et al. used meta‐regression models to study potential moderators of these effects, and their analyses had to contend with a number of effects that were missing covariates. While in practice, models were estimated via the expectation–maximization (EM) algorithm rather than complete‐ or shifting‐case methods, we use a subset of this data in order to illustrate complete‐ and shifting‐case analyses.

Consider a subset of the Tanner‐Smith et al. data comprising 74 effect estimates of substance abuse interventions from 46 studies. These effect estimates involve contrasts between groups in a study that are subjected to different treatment conditions, denoted in the data as Group 1 and Group 2, so that each treatment effect can be thought of as Group 1 minus Group 2. Typically, researchers avoided no‐treatment or placebo conditions in studies over ethical concerns surrounding the failure to treat adolescents with substance abuse disorders. Thus, contrasts within studies (i.e., effect estimates) tended to focus on a specific treatment of interest to the researcher versus some alternate treatment. Effect estimates are reported on the scale of bias‐corrected standardized mean differences.

Suppose the analysis of interest involves the impact of high‐ versus low‐intensity interventions on treatment effects, where a high‐intensity intervention consisted of more than 1.5 h per week of treatment. Then this analysis might use a pair of binary covariates for each effect: one would indicate whether Group 1 received a high‐intensity intervention (i.e., X ₁ = 1 if Group 1 treatment was high‐intensity) and the other would indicate whether Group 2 received a high intensity (i.e., X ₂ = 1 if Group 2 treatment was high‐intensity). The relevant meta‐regression model would regress the effect estimates on these two covariates.

In the data, the treatment intensity is missing for some of the effects, and Table 1 summarizes missingness for these covariates. Table 1 shows that only 37 of the 74 (50%) have a reported treatment intensity for both groups (i.e., X ₁ and X ₂ are both observed), but that 54 (73%) of effects report Group 1's treatment intensity (i.e., X ₁ is observed) and 41 (55%) effects report Group 2's treatment intensity.

TABLE 1.

The total number and percentage of effect sizes that are missing covariates regarding whether Group 1 or Group 2 received high‐intensity interventions in the substance abuse intervention meta‐analysis

Group 1 Hi‐intensity	Group 2 Hi‐intensity	Count	Percent
Observed	Observed	37	0.50
Observed	Missing	17	0.23
Missing	Observed	4	0.05
Missing	Missing	16	0.22

A complete‐case analysis would include only the 37 effects for which both covariates were observed. Using robust variance estimation to account for dependence between effect sizes, a CCA would result in the coefficient estimates and standard errors displayed in the first column of Table 2. Based on these estimates, when Group 1 receives a high‐intensity treatment, we would expect an effect to be larger by d = 0.44 (in standard deviation units) than when Group 1 receives a low‐intensity treatment, which is statistically significant at the α = 0.10 level. Note that the estimated between‐effect variance is ${\widehat{\tau }}^2=0.08$.

TABLE 2.

The meta‐regression results for the model regressing effect sizes on high‐intensity indicator variables when using complete‐ and shifting‐case analyses

Term	Complete‐case	Shifting‐case Group 1	Shifting‐case Group 2
Intercept	0.11 (SE = 0.06, p = 0.11)	0.14 (SE = 0.06, p = 0.02)	0.15 (SE = 0.06, p = 0.03)
Group 1 Hi‐Int.	0.44 (SE = 0.16, p = 0.06)	0.27 (SE = 0.15, p = 0.07)	–
Group 2 Hi‐Int.	−0.21 (SE = 0.26, p = 0.46)	–	0.16 (SE = 0.26, p = 0.54)
Variance comp. τ 2	0.08	0.06	0.09

However, the model above is estimated on only half of the data. Concern over using a small proportion of the data, or a relatively few number of effects often leads meta‐analysts to opt for a shifting‐case analysis. An example of an SCA would use the 54 effects for which Group 1's treatment intensity is observed (i.e., X ₁ is observed), but only including X ₁ in the model. Doing so leads to the estimates in second column of Table 2. Note that the coefficient estimate for Group 1's treatment intensity is still positive, but is roughly 60% the magnitude of the estimate in the complete‐case model.

Finally, an analogous model in an SCA would include the 41 effects for which Group 2's intensity is observed, and include only that covariate in the model. The third column of Table 2 shows that this results in a coefficient estimate for Group 2's treatment intensity (0.16) that is in the opposite direction of the estimate from the CCA (−0.21).

It should be noted that all of these estimates and comparisons between them ought to be interpreted with caution. The complete‐case analysis includes only half of the effect sizes, which comprises a missingness rate well beyond what might be considered negligible. ¹⁶ , ¹⁷ The shifting‐case analyses include more of the data, but because each shifting‐case model omits one of the covariates, these models are not equivalent to the model that includes both covariates. ¹⁸ It could even be argued that the parameters in the model with both covariates are not comparable to parameters in models with only one covariate; coefficients in a model with multiple covariates must be interpreted in relation to other variables in the model. The remainder of this article quantifies the bias induced by omitting effect sizes and/or covariates from meta‐regressions.

3. MODEL AND NOTATION

Suppose a meta‐analysis involves k effects estimated from collection of studies. For the ith effect, let T _i be the estimate of the effect parameter θ _i , and let v _i be the estimation error variance of T _i . Denote a vector of covariates that pertain to effect estimate T _i as X _i  = [1, X _i1, …, X _ip ]. Note that the first element of X _i is a 1, which corresponds to an intercept term in a meta‐regression model, and that X _ij for j = 1, …p corresponds to different covariates. The meta‐regression model can be expressed as:

$$T_i\mid X_i,v_i,\eta =X_i\beta +u_i+e_i.$$ (1)

Here, $\beta \in {\mathrm{ℝ}}^{p+1}$ is the vector of regression coefficients. The estimation errors e _i are typically assumed to be normally distributed with mean zero and variance V [e _i ] = v _i . This assumption is true of some effect size indices and is a very accurate large‐sample approximation for others. ¹⁹ The term u _i represents the random effect such that u _i  ⊥ e _i and V [u _i ] = τ ². This model is equivalent to the standard mixed‐effects meta‐regression model, and it is also consistent with subgroup analysis models. ¹⁹ , ²⁰ The vector η = [β, τ ²] refers to the parameters of model. Under a fixed‐effects model, it is assumed that τ ² = 0, in which case η = β, and u _i  ≡ 0.

A common assumption in random effects meta‐regression is that the random effects u _i are independent and normally distributed with mean zero and variance τ ²: ²⁰ , ²¹ , ²² , ²³

$$u_i\sim N\left(0,{\tau }^2\right).$$

This could correspond to a scenario of k independent effect estimates presumably from k different studies. In that case, the distribution p(T|X, v, η) can be written as

$$p\left(T_i|X_i,v_i,\eta \right)=\frac{1}{\sqrt{2\pi \left({\tau }^2+v_i\right)}}{e}^{-\frac{{\left(T_i-X_i\beta \right)}^2}{2\left({\tau }^2+v_i\right)}}.$$ (2)

Thus, the joint likelihood for all k effects can be written as:

$$p\left(\mathbf{T}\mid \mathbf{X},\mathbf{v},\eta \right)={\left(2\pi \right)}^{-k/2}\left[\prod _{i=1}^k\left({\tau }^2+v_i\right)\right]e^{-\sum _{i=1}^k\frac{{\left(T_i-X_i\beta \right)}^2}{2\left({\tau }^2+v_i\right)}},$$ (3)

where $\mathbf{T}\in {\mathrm{ℝ}}^k$ is the vector of effect estimates, $\mathbf{v}\in {\mathrm{ℝ}}^k$ is the vector of estimation variances, and $\mathbf{X}\in {\mathrm{ℝ}}^{k\times \left(p+1\right)}$ is the matrix of covariates where each row of X is simply the row vector X _i . Note that the functions in both (2) and (3) assume that all of the p covariates are observed. Equation (3) is referred to as the complete‐data likelihood function. ¹³ , ²⁴ We note that a meta‐regression with no missing data will be accurate if the complete‐data model is correctly specified. Thus, to illustrate the properties of incomplete data meta‐regression, we assume that the complete‐data model is correctly specified.

The vector of regression coefficient estimates for the complete‐data model when there is no missing data is typically estimated by

$$\widehat{\beta }={\left({\mathbf{X}}^T\mathbf{WX}\right)}^{-1}{\mathbf{X}}^T\mathbf{WT}.$$ (4)

Here, W = diag[w _i ] is the diagonal matrix of weights such that w _i  = 1/(v _i  + τ ²). The covariance matrix of $\widehat{\beta }$ is given by

$$V\left[\widehat{\beta }\right]={\left({\mathbf{X}}^T\mathbf{WX}\right)}^{-1}$$ (5)

Note that the weights involve the true variance component τ ². In practice, τ ² must be estimated by ${\widehat{\tau }}^2$, and the resulting weights used in analyses can be written ${\widehat{w}}_i=1/\left(v_i+{\widehat{\tau }}^2\right)$. For the sake of simplicity, we use w _i to derive results in this article, and so results do not depend on variance component estimators. Presumably, use of ${\widehat{w}}_i$ would induce additional variation into analyses.

The substance abuse data contains multiple effect estimates per study that are likely correlated. This differs from the model above. However, we can expand this model to account for dependent effect sizes by assuming that $T_i\in {\mathrm{ℝ}}^{k_i}$ is a vector of k _i effects from the same study, e _i is vector of estimation errors, u _i is a vector of random effects, and e _i  + u _i has covariance matrix ∑ _i . In this model, X _i is a matrix of covariates for each effect in T _i . The resulting formulas for the complete‐data likelihood function and coefficient estimators will be more complex (including a variance–covariance weight matrix), but they will have a similar form as the independent effect size model.

Not all relevant variables may be observed in a meta‐analytic dataset. Let R _i be a vector of response indicators that correspond with effect i. This article concerns missing covariates, and we assume that T _i and v _i are observed for every effect of interest in a meta‐analysis. Thus, each element R _ij of R _i corresponds to a covariate X _ij . The R _ij take a value of either 0 or 1: R _ij  = 1 indicates the corresponding X _ij is observed and R _ij  = 0, indicates a that the corresponding X _ij is not observed. Note that $R_i\in \mathrm{ℛ}\equiv {\left\{0,1\right\}}^p$ is a vector of 0 and 1 s of length p. For instance, X _i2 were missing, this would be indicated by R _i2 = 0.

Denote O = {(i, j): R _ij  = 1} as the indices of covariates that are observed and M = {(i, j): R _ij  = 0} be the set of indices for missing covariates. Then, the complete‐data model can be written as

$$p\left(\mathbf{T}\mid \mathbf{X},\mathbf{v},\eta \right)=p\left(\mathbf{T}|{\mathbf{X}}_{\mathrm{O}},{\mathbf{X}}_{\mathrm{M}},\mathbf{v},\eta \right).$$ (6)

Note that the complete‐data model depends on entries of X _M, which are unobserved. It is worth pointing out that the complete‐data model, which refers to the model with no missing data, is distinct from the complete‐case analysis, which is an estimation procedure that conditions only on observed data.

3.1. Complete‐case estimators

A common approach in meta‐regression with missing covariates is to use a complete‐case analysis. ¹⁰ , ¹¹ This approach simply omits rows in the data for which any covariate is missing. Thus, this analysis method only uses effects and covariates for which R _i  = [1, …, 1] = 𝟙.

Let C = {i: R _i  = 𝟙} index all relevant effects i such that R _i  = 𝟙, so that X _C is the matrix of covariates such R _i  = 𝟙, T _C is the corresponding subset of effect estimates, and W _C is the corresponding subset of weights. The CCA estimates the coefficients β with

$${\widehat{\beta }}_{\mathrm{C}}={\left({\mathbf{X}}_{\mathrm{C}}^{\mathrm{T}}{\mathbf{W}}_{\mathrm{C}}{\mathbf{X}}_{\mathrm{C}}\right)}^{-1}{\mathbf{X}}_{\mathrm{C}}^{\mathrm{T}}{\mathbf{W}}_{\mathrm{C}}{\mathbf{T}}_{\mathrm{C}}.$$ (7)

3.2. Shifting‐case estimators

When there are multiple covariates of interest, each of which has some missingness, there may only be a few effects for which all covariates of interest are observed. When that happens, a complete‐case analysis can be unfeasible. A common solution to this in meta‐analysis is to use an available‐case analysis. ¹¹ In practice, an available‐case meta‐regression is often equivalent to a shifting‐case analysis, referred to in the literature as shifting units of analysis. ¹⁰ , ¹²

Shifting‐case analyses involve fitting multiple regression models, each including a subset of the covariates of interest. Sometimes this even takes the form of regressing effect estimates on one covariate at a time. ¹⁰ , ¹¹ In the substance abuse data example, we focused on two covariates of interest X _i1 and X _i2. The SCA first regressed T _i on observed values of X _i1. This regression included observations for which both X _i1 and X _i2 are observed (i.e., R _i  = [1, 1]) and observations for which X _i1 is observed but X _i2 is missing (i.e., R _i  = [1, 0]). We then regressed T _i on X _i2, which included effects for which R _i ∈{[1, 1], [0, 1]}. In sum, the SCA demonstrated in the previous section involved two regressions, each of which conditioned on different sets of missingness patterns.

To formalize SCA estimators, consider a single regression in an SCA, and let S index the component of X _i (i.e., the intercept term and relevant covariates) included in that model S = {j: j = 0 or X _ij in analysis}. Let E be the complement of S so that E indexes the covariates excluded from the regression. Then, the regression is used to estimate and make inferences about coefficients β _S. In the following section, we discuss β _S and its relationship to β, but here assume that the target of inference for an SCA is β and hence β _S comprises a subset of the components of β. For instance, in the first substance abuse SCA regression, T _i was regressed on only X _i1, so that β _S = [β ₀, β ₁].

Denote ${\mathrm{ℛ}}_j$ as the set of missingness patterns such that all included covariates are observed: ${\mathrm{ℛ}}_j=\left\{R\in \mathrm{ℛ}:R_S=\mathbb{1}\right\}$. Note that ${\mathrm{ℛ}}_j$ contains missingness patterns such that all the included covariates are observed, but any excluded covariates may be either observed or unobserved. For instance, in the first substance abuse SCA regression of T _i on X _i1, the analysis included effects such that $R_i\in {\mathrm{ℛ}}_1=\left\{\left[1,1\right],\left[1,0\right]\right\}$. Finally, let U denote the indices i of effects for which X _iS are observed; note that U depends on S, so we may write $U\left(S\right)=\left\{i:R_i\in {\mathrm{ℛ}}_j\right\}$. Then, the shifting‐case estimators for β _S are given by:

$${\widehat{\beta }}_S={\left({\mathbf{X}}_{\mathit{US}}^{\mathrm{T}}{\mathbf{W}}_U{\mathbf{X}}_{\mathit{US}}\right)}^{-1}{\mathbf{X}}_{\mathit{US}}^{\mathrm{T}}{\mathbf{W}}_U{\mathbf{T}}_{\mathrm{U}},$$ (8)

where X _US contains the columns (S) of X that pertain to the covariates that are included in the SCA regression, and the rows (U) for which all of those covariates are observed. The matrix W _U is a square matrix containing the relevant rows and columns of W for which X _iS are observed, while T _U contains the effect sizes in T for which X _iS is observed.

3.3. Omitted variables

A common concern in meta‐regression is that models may not be able to account for all relevant covariates, either due to sample size constraints or because some covariates were not observed. ²⁵ Such concerns pertain to meta‐regressions both with and without missing data. In contrast to primary data analysis, meta‐analysts tend to have little control over the availability of covariates relevant for a meta‐regression. Information regarding covariates must be extracted from primary studies and this process is restricted by the ways in which research is reported. Some studies may not clearly report covariates deemed of interest in a meta‐regression. Even for key conceptually or theoretically important moderators, meta‐regression models must often contend with variation in reporting of such moderators across studies. Thus, the issue of omitted variables in meta‐regression is both prevalent and difficult to overcome (e.g., via additional data collection).

The implication of omitting observed variables in SCA can be understood via the parameter β _S . It has been noted that there are various conditions under which components of β _S are unequal to their counterparts in β. ²⁶ For instance, it can be the case that ${\beta }_S=\left[{\tilde{\beta }}_0,{\tilde{\beta }}_1\right]\ne \left[{\beta }_0,{\beta }_1\right]\subset \beta$. The difference between β _S and components of β is often referred to as omitted variable bias in the statistical and econometric literature. ²⁷ , ²⁸ This conception inherently assumes that β in the full model is of interest to the analyst, which may not necessarily be the case. Indeed, one may assume that β _S is of interest, rather than β, so that the components of β _S comprise parameters distinct from β. In this approach, β _S characterizes the relationship between X _iS and T _i in a more restricted model that does not account for X _iE. This could be consistent with analyses that seek to summarize effects within specific subgroups of studies delineated by covariates.

We refer to the difference between β _S and β as omitted variable bias, in keeping with the literature on linear models. In doing so, we treat SCA as a missing data analytic strategy, wherein the target of inference is β. Subsequent sections present findings on bias induced by omitting observed covariates in an SCA, which reflect the findings of Lipsey (2003), who points out that interpretation of meta‐regression coefficients when covariates are omitted can lead to misleading interpretations about the correlates of effective interventions. However, if the intent of the analysis is to examine restricted models or specific subgroups of effects/studies, the omitted variable bias presented in this article may be less applicable, though Lipsey's caveats for interpreting such models may still apply.

3.4. Missingness mechanisms

Both the complete‐ and shifting‐case estimators are analyses of incomplete data. Analyses of incomplete data require some assumption about why data are missing, which is referred to as the missingness mechanism. The mechanism by which missingness arises is typically modeled through the distribution of R. Let ψ denote the parameter (or vector of parameters) that index the distribution of R so that the probability mass function of R can be written as p(R|T, X, v, ψ). Assumptions about the missingness mechanism are therefore equivalent to assumptions about p(R|T, X, v, ψ).

Rubin ²⁹ defined three types of mechanisms in terms of the distribution of R. Data could be missing completely at random (MCAR), which means that the probability that a given value is missing is independent of all of the observed or unobserved data:

$$p\left(\mathbf{R}|\mathbf{T},{\mathbf{X}}_{\mathrm{O}},{\mathbf{X}}_{\mathrm{M}},\mathbf{v},\psi \right)=p\left(\mathbf{R}|\psi \right).$$

MCAR implies that probability that a given value is missing depends only on the missingness parameter ψ.

Covariates could be missing at random (MAR), which implies the distribution of missingness depends only on observed data and the missingness parameter:

$$p\left(\mathbf{R}|\mathbf{T},{\mathbf{X}}_{\mathrm{O}},{\mathbf{X}}_{\mathrm{M}},\mathbf{v},\psi \right)=p\left(\mathbf{R}|\mathbf{T},{\mathbf{X}}_{\mathrm{O}},\mathbf{v},\psi \right).$$

MAR differs from MCAR in that missingness might be related to observed values. As an example, if studies with larger standard errors are less likely to report the racial composition of their samples, then missingness would depend on the (observed) estimation error variances. Data missing according to this mechanism would violate an assumption of MCAR, since missingness is related to an observed value.

Finally, data are said to be missing not at random (MNAR) if the distribution of R depends on unobserved data in some way. In the context of the meta‐regression data, this would imply that R is related to X _M, so that the probability of a covariate not being observed depends on the value of the covariate itself. For instance, data would be MNAR if studies with larger standard errors and a greater proportion of minorities are less likely to report the racial composition of their samples because the likelihood that racial composition is not reported will depend on the composition itself.

A related concept in missing data is that of ignorability, which means that the missingness pattern does not contribute any additional information. When missing data are ignorable, it is not necessary to know (or estimate) ψ in order to conduct inference on η. ¹³ , ¹⁴ , ²⁴ , ³⁰ In practice, missing data are ignorable if they are MAR and if ψ and η are distinct.

4. CONDITIONAL INCOMPLETE DATA META‐REGRESSION

Because both complete‐ and available‐case analyses depend on the value of R _i , they can be seen as models that condition on missingness. Models that condition on missingness are not necessarily identical to the complete‐data model, which is the model of interest, because the complete‐data model does not condition on R _i . Yet, CCA and SCA proceed as if the complete‐data and conditional models on missingness are equivalent. Doing so ignores the missingness mechanism and its potential impact on the accuracy of analytic results.

The complete‐data model can be related to the conditional models through the distribution of missingness R _i . This approach is referred to as a selection model in the missing data literature. ¹³ , ²⁴ , ³⁰ We can write the selection model for meta‐regression with missing covariates as:

$$p\left(T_i|X_i,v_i,R_i\in {\mathrm{ℛ}}_j,\eta ,\psi \right)=\frac{p\left(R_i\in {\mathrm{ℛ}}_j|T_i,X_i,v_i,\psi \right)p\left(T_i|X_i,v_i,\eta \right)}{p\left(R_i\in {\mathrm{ℛ}}_j|X_i,v_i,\psi ,\eta \right)},$$ (9)

where ψ indexes the distribution of R|T, X, v. Here, ${\mathrm{ℛ}}_j$ refers to the relevant subset of $\mathrm{ℛ}$ on which the analysis conditions; for a CCA, ${\mathrm{ℛ}}_j=\left\{\mathbb{1}\right\}$.

Equation (9) describes the conditional model as a function of the complete‐data model p(T _i |X _i , v _i , η) and a selection model $p\left(R_i\in {\mathrm{ℛ}}_j|T_i,X_i,v_i,\psi \right)$ that gives the probability that a given set of covariates are observed. The denominator on the right hand side of (9) is a normalizing factor that is equivalent to the probability of observing the missingness pattern ${\mathrm{ℛ}}_j$ given the estimation error variance v _i and the observed and unobserved covariates in the vector X _i , and can be written as

$$p\left(R_i\in {\mathrm{ℛ}}_j|X_i,v_i,\psi ,\eta \right)=\int p\left(R_i\in {\mathrm{ℛ}}_j|T_i,X_i,v_i,\psi \right)p\left(T_i|X_i,v_i,\eta \right){\mathit{dT}}_i.$$ (10)

Note that when the complete‐data model in (2) is not equivalent to the conditional model in (9), the resulting coefficient estimators in a meta‐regression can be biased. To see this, we can write:

$$E\left[T_i|X_i,v_i,R_i\in {\mathrm{ℛ}}_j\right]=E\left[T_i|X_i,v_i\right]+{\delta }_{\mathit{ij}}=X_i\beta +{\delta }_{\mathit{ij}}.$$ (11)

Here, we see that the expectation of T _i given X _i and R _i can be written as the complete‐data expectation X _i β (i.e., the regression model) plus a bias term δ _ij . The bias term δ _ij refers to the bias induced in the regression model due to conditioning on missingness pattern ${\mathrm{ℛ}}_j$, which can affect individual components of η. If δ _ij  ≠ 0, it follows that conditioning on R _i induces bias in the distribution of T _i used in an analysis. Because the CCA estimator (7) and SCA estimator in (8) are weighted averages of the T _i , they can be biased if δ _ij  ≠ 0. The precise magnitude of the δ _ij will depend on the selection model in (9) and hence on the missingness mechanism. It is worth noting that the subsequent sections show that bias depends on the precise selection model rather than the class of mechanism (MCAR or MAR).

A standard approach for modeling missingness mechanisms for covariates is to assume R _i follows some log‐linear distribution. ³¹ Various authors have described approaches to modeling R for missing covariates in generalized linear models that include logistic and multinomial logistic models. ³² , ³³ , ³⁴ Thus, one class of models for missingness would involve the logit probability of observing some missingness patterns $R_i\in {\mathrm{ℛ}}_j\subset \mathrm{ℛ}$:

$$\text{logit}\left[p\left(R_i\in {\mathrm{ℛ}}_j|T_i,X_i,v_i\right)\right]=\sum _{m=0}^{m_j}{\psi }_{\mathit{mj}}{f}_{\mathit{mj}}\left(T_i,X_i,v_i\right).$$ (12)

Here, f _mj (T _i , X _i , and v _i ) are assumed to be differentiable basis functions of the data and m _j is the number of terms in the selection model. In theory, m _j could be arbitrarily large, but the model is only estimable if m _j  < k. Finally, we assume f _0j (T _i , X _i , v _i ) = 1, so that ψ _0j would be the intercept term for the logit model for the set of missingness patterns ${\mathrm{ℛ}}_j$.

In general, it is impossible to know whether a selection model is correctly specified, but the formulation in (12) offers a few important advantages. First, it is fairly general: the only assumption made of the basic functions f _mj is that they are differentiable, which means model (12) allows for nonlinear or interaction terms. Second, it expresses the relationships between the probability of the event $\left\{R_i\in {\mathrm{ℛ}}_j\right\}$ and observed variables on the scale of the log odds ratio, a well‐understood scale in meta‐analysis. Third, it allows for closed‐form expressions for the approximate bias of coefficient estimates by virtue of the logit link function. Thus, it comprises a large class of models for selection that can be more clearly interpreted.

4.1. Approximate bias for log‐linear selection models

As argued above, the bias of complete‐case estimators ${\widehat{\beta }}_{\mathrm{C}}$ or shifting‐case estimators ${\widehat{\beta }}_S$ will depend in some way on the bias δ _ij induced in T _i by conditioning on $R_i\in {\mathrm{ℛ}}_j$. The magnitude and direction of δ _ij will in turn depend on the selection model.

It is possible to derive an approximation for δ _ij under certain conditions. If p(T _i |X _i , v _i ) is the standard fixed‐ or random effects meta‐regression model in Equation (2), and $p\left(R_i\in {\mathrm{ℛ}}_j|T_i,X_i,v_i\right)$ follows the log‐linear model in (12), and the f _mj are differentiable with respect to T _i , then

$${\delta }_{\mathit{ij}}\approx H_j\left(X_i\beta ,X_i,v_i\right)\left({\tau }^2+v_i\right)\sum _{m=0}^{m_j}{\psi }_{\mathit{mj}}{f}_{{\mathit{mj}}^{\prime }}\left(X_i\beta ,X_i,v_i\right),$$ (13)

where H _j (X _i β, X _i , v _i ) is equivalent to $p\left(R_i\cancel{\in }{\mathrm{ℛ}}_j|T_i,X_i,v_i\right)$ evaluated at T _i  = X _i β and

$$f_{{\mathit{mj}}^{\prime }}\left(X_i\beta ,X_i,v_i\right)={\left.\frac{\partial f_{\mathit{mj}}}{\partial T_i}\right|}_{T_i=X_i\beta }$$

is the derivative of f _mj with respect to T _i evaluated at T _i  = X _i β. A more detailed proof is presented in Appendix.

While the following sections will examine possible values that δ _ij may take under different selection models, we can gain some insight on bias by examining (13). The expression for δ _ij depends on three main quantities. First, δ _ij is an increasing of H _j (X _i β, X _i , v _i ), which is the probability that $R_i\cancel{\in }{\mathrm{ℛ}}_j$. This implies that the bias will be greater as the probability of omitting an observation increases. Second, δ _ij increases in the sum of variance components τ ² + v _i , which means that the bias will be larger when T _i varies more around the regression line. Finally, δ _ij depends on ψ _mj f _mj ^' (X _i β, X _i , v _i ). Since f _mj ^' is the derivative of f _mj with respect to T, when f _mj does not depend on T, then f _mj ^'  = 0, and hence ψ _mj f _mj ^'  = 0. Thus, δ _ij depends on the components of the selection model that are functions of T _i and how strongly those components are related to the probability of observing X _i via the parameter ψ.

5. BIAS IN COMPLETE‐CASE ANALYSES

Complete‐case analyses only include effects for which all relevant covariates are observed. The complete‐case coefficient estimator ${\widehat{\beta }}_{\mathrm{C}}$ given in Equation (7) conditions on R _i  = 𝟙. As noted above, conditioning on R _i can induce bias, however there are conditions under which the CCA will lead to unbiased coefficient estimates. These conditions largely amount to whether or not R _i is independent of the effect size estimate T _i , the outcome of meta‐regression model. When the distribution of R _i depends on T _i , then complete‐case estimators will be biased.

The general condition under which CCA estimators are unbiased is that R _i  ⊥ T _i , which occurs for different types of selection models. First, if the covariates are MCAR, then R _i  ⊥ (T _i , X _i , v _i ). Alternatively, if the selection model depends only on v _i , but not X _i or T _i , then R _i ⊥ (T _i , X _i )|v _i ; this would constitute a MAR mechanism. Finally, if the selection model depends only on v _i and X _i , but not T _i , then R _i  ⊥ T _i |(X _i , v _i ), which would correspond to an MNAR mechanism. Under each of these assumptions, it can be shown that the model that conditions on complete cases R _i  = 𝟙 is identical to the complete‐data model, and hence CCA estimators will be unbiased:

$$p\left(T_i|X_i,v_i,R_i=\mathbb{1},\eta ,\psi \right)=\frac{p\left(R_i=\mathbb{1}|T_i,X_i,v_i,\psi \right)p\left(T_i|X_i,v_i,\eta \right)}{p\left(R_i=\mathbb{1}|X_i,v_i,\eta ,\psi \right)}=p\left(T_i|X_i,v_i,\eta \right).$$ (14)

This result is consistent with prior work regarding linear regression models with missing covariates. ³⁵ , ³⁶

An important aspect of this result is that whether or not a CCA produces unbiased coefficient estimates depends more on the role of T _i in the selection model rather than traditional mechanism classifications of MCAR, MAR, or MNAR. However, various selection models satisfy the conditions of MAR, and similarly with MNAR, the key factor for bias in CCA estimators is the relationship between R _i and T _i . Should R _i ⊥̸T _i , then CCA estimators can be biased, regardless of whether the mechanism is MAR or MNAR. Similarly, if R _i ⊥ T _i , CCA estimators can be unbiased, regardless of MAR or MNAR.

When R _i is not independent of T _i (given X _i or v _i ), then CCA can be biased. Let ${\mathrm{ℛ}}_1=\left\{\mathbb{1}\right\}$ so that the CCA conditions on $R_i\in {\mathrm{ℛ}}_1$. Based on Equation (11), the bias of ${\widehat{\beta }}_C$ will depend on the δ _i1. If we let Δ = [δ ₁₁, …, δ _k1] be the vector of δ _i1 and let Δ _C be the subset of Δ for which all covariates are observed (i.e., R _i  = 𝟙). Then the bias of the complete‐case analysis can be written as

$$\text{Bias}\left[{\widehat{\beta }}_{\mathrm{C}}\right]={\left({\mathbf{X}}_{\mathrm{C}}^{\mathrm{T}}{\mathbf{W}}_{\mathrm{C}}{\mathbf{X}}_{\mathrm{C}}\right)}^{-1}{\mathbf{X}}_{\mathrm{C}}^{\mathrm{T}}{\mathbf{W}}_{\mathrm{C}}{\mathrm{\Delta }}_{\mathrm{C}}.$$ (15)

The bias in Equation (15) is a weighted average of individual biases δ _i1. Hence, the bias will be larger if the δ _i1 are larger (and in the same direction).

Precisely, how large the bias in (15) is will depend on the distribution of R _i and its relationship to effect estimates T _i and their covariates X _i . When R _i follows the log‐linear model in (12), the approximate bias can be written as

$$\text{Bias}\left[{\widehat{\beta }}_{\mathrm{C}}\right]\approx {\left({\mathbf{X}}_{\mathrm{C}}^{\mathrm{T}}{\mathbf{W}}_{\mathrm{C}}{\mathbf{X}}_{\mathrm{C}}\right)}^{-1}{\mathbf{X}}_{\mathrm{C}}^{\mathrm{T}}{\mathbf{W}}_{\mathrm{C}}{\mathbf{H}}_{1\mathrm{C}}{\mathbf{f}}_{1\mathrm{C}}{\psi }_1,$$ (16)

where

$${\mathbf{H}}_1=\mathrm{diag}\left[H_1\left(X_i\beta ,X_i,v_i\right)\right]$$

is a k × k diagonal matrix where entries refer to the probability that an observation is not a complete case,

$${\mathbf{f}}_1=\left[f_{{01}^{\prime }}\left(X_i^T\beta ,X_i,v_i\right),\dots ,f_{m_1{1}^{\prime }}(X_i^T\beta ,X_i,v_i)\right]$$

is a k × m ₁ matrix of derivatives, and ${\psi }_1={\left[{\psi }_{01},\dots ,{\psi }_{m_{1}1}\right]}^{\mathrm{T}}$ is a vector of parameters that index the selection model. Note that the bias in (16) involves H _1C which contains the rows of H ₁ for which R _i  = 𝟙; similarly for f _1C.

While (16) provides a general expression for the approximate bias of ${\widehat{\beta }}_{\mathrm{C}}$, it can be a little difficult to interpret. Loosely, we can see that the bias depends on the probability that covariates are missing, reflected in H _1C, as well as some function of the components of the log‐linear selection model f _1C ψ ₁. To better intuit this bias, we provide a simple example in the following section.

5.1. Example: complete‐case analysis with a single binary covariate

Suppose the model of interest includes a single binary covariate X _i1 ≡ X _i  ∈ {0, 1}, so that the complete data model is

$$T_i={\beta }_0+{\beta }_1{X}_i+u_i+e_i,$$ (17)

where β ₀ and β ₁ are the regression coefficients of interest. Note that β ₀ is the average effect when X _i  = 0 and β ₁ is the contrast in mean effects for when X _i  = 1 versus when X _i  = 0.

Because X _i is a scalar, so is R _i ; R _i  = 0 indicates that X _i is missing, R _i  = 1 indicates that X _i is observed. A CCA would include only effects i for which X _i is observed (i.e., R _i  = 1). The complete‐case estimator for β ₀ is given by a weighted sum of T _i among the effects for which X _i  = 0 and R _i  = 1:

$${\widehat{\beta }}_{0C}=\frac{\sum _{i:X_i=0,R_i=1}{w}_i{T}_i}{\sum _{i:X_i=0,R_i=1}{w}_i}.$$ (18)

The complete‐case estimator for β ₁ is given by the difference between the (weighted) mean effect for X _i  = 1 versus X _i  = 0:

$${\widehat{\beta }}_{1C}=\frac{\sum _{i:X_i=1,R_i=1}{w}_i{T}_i}{\sum _{i:X_i=1,R_i=1}{w}_i}-{\widehat{\beta }}_{0\mathrm{C}}.$$ (19)

Assume that the selection model is log‐linear, and that for the sake of simplicity the probability of observing X _i depends on the size of the effect T _i and the value of X _i :

$$\text{logit}\left[p\left(R_i=1|T_i,X_i,v_i\right)\right]={\psi }_0+{\psi }_1{T}_i+{\psi }_2{X}_i.$$ (20)

Note that this is an MNAR mechanism, since the probability X _i is observed depends on X _i itself; a MAR mechanism would involve ψ ₂ = 0 in Equation (20). Because (20) depends on T _i , δ _ij  ≠ 0 for this selection model regardless of MAR or MNAR (i.e., regardless of whether ψ ₂ = 0 or not), the CCA estimators may be biased.

Under this model, H ₁(X _i β, X _i , v _i ) depends only on X _i and not v _i , so we can write $H_1\left(X_i\right)=p{\left.\left(R\ne \mathbb{1}|T_i,X_i,v_i\right)\right|}_{T_i=X_i^T\beta }$. As well, f ₁₁(T _i , X _i , v _i ) = T _i and f ₂₁(T _i , X _i , v _i ) = X _i . Given the result in Equation (13), we can write

$${\delta }_{i1}\approx H_1\left(X_i\right)\left(v_i+{\tau }^2\right){\psi }_1.$$ (21)

Given the selection model in (20), the bias of the complete‐case estimator for the intercept, ${\widehat{\beta }}_{0\mathrm{C}}$, is:

$$\text{Bias}\left[{\widehat{\beta }}_{0\mathrm{C}}\right]\approx H_1\left(0\right)\left({\overline{v}}_0+{\tau }^2\right){\psi }_1,$$ (22)

where ${\overline{v}}_0$ is the average estimation error variance v _i among effects for which X _i  = 0 and R _i  = 1. The expression in (22) depends on three key quantities, and is an increasing function of each of those quantities. First, the bias increases in H ₁(0), which is an approximation of the probability that X _i is missing among studies for which X _i  = 0. While under model (20), this probability is a function of T _i and X _i , we can intuit H ₁(0) loosely as a missingness rate in X _i among effects for which X _i  = 0. Second, the bias in (22) is increasing in ${\overline{v}}_0+{\tau }^2$, the average variation of T _i for which X _i  = 0; the greater the variation, the greater the bias. Because the v _i is typically decreasing in sample size, if studies have smaller samples, the bias will be greater. Finally, the bias depends on ψ ₁, which characterizes the relationship between an X _i being observed (i.e., R _i ) and T _i . When ψ ₁ is positive, larger effect estimates T _i are more likely to have observed X _i and the bias will be positive; if ψ ₁ is negative, so that larger effect sizes are more likely to be missing the covariate X _i , then the bias will be negative.

To gain better insight into Equation (22), suppose $v_i\approx v={\overline{v}}_0$ so that each study has roughly the same estimation error variance. If we assume T _i is on the scale of a standardized mean difference, v _i  ≈ 4/n _i where n _i is the total sample size used to compute T _i . Various researchers have described conventions for the magnitude of τ ² that range from τ ² = v/4 to τ ² = v. ³⁷ , ³⁸ , ³⁹ Thus, we can write τ ² + v = 4(1 + r)/n from some constant r that ranges from 0 to 1.

Further, the parameter ψ ₁ is a log‐odds ratio, which reflects the odds of a complete case for T _i versus T _i  − 1. There are various conventions for the size of an odds ratio that depend on base rates P[R = 𝟙|T] that could be interpreted as ranging from 1.5 to as large as 9.0, though various researchers have noted that odds ratios greater than 3.0 or 4.0 could be considered large. ⁴⁰ , ⁴¹ , ⁴² , ⁴³ Thus, we consider a range of odds ratios from about 1.5 to 4.5. However, the actual size of ψ ₁ will depend on the scale of a change in effect size $D_T=\mid T_i-{\tilde{T}}_i\mid$. Since it corresponds to a difference, D _T should be no larger than an individual |T _i |. Based on conventions in the social and medical sciences (some arbitrary, some empirical), meaningful values of D _T might feasibly range from 0.2 to 1.0. ⁴⁰ , ⁴⁴ These conventions for odds ratios and D _T would imply that relevant values of |ψ ₁| might range from 0.4 (large D _T with small odds ratio) to over 7.5 (small D _T with large odds ratio).

Based on these conventions, Figure 1 shows the potential (approximate) bias of ${\widehat{\beta }}_{0\mathrm{C}}$ for this example. Each panel corresponds to a given within‐study variance v = 4/n and residual heterogeneity τ ². Panels plot the bias contributed by a single case δ _i as a function of the probability of missingness H ₁(0) (x‐axis) and ψ ₁ (color). The panels on the bottom few rows and left most columns show that if both ψ ₁ is small and τ ² + v is small, then δ _i will be less than 0.05. However if τ ² + v _i is larger and the probability of a complete case is strongly related to T _i (i.e., ψ ₁ is large), then the bias can be greater than d = 0.2 or even 0.5.

FIGURE 1.

The bias of the intercept estimate ${\widehat{\beta }}_{0\mathrm{C}}$ (y‐axis) of the example. Bias is shown as a function of the average sampling variance v, residual heterogeneity τ ², the probability of missingness when X ₁ = 0, H ₁(0) (x‐axis), and the correlation between missingness and the effect size as measured by ψ ₁ (color). Note that ψ ₁ is a log‐odds ratio for effect sizes on the scale of Cohen's d [Colour figure can be viewed at wileyonlinelibrary.com]

It is worth noting that Figure 1 gives the bias for when T _i is positively correlated with R _i , and hence ψ ₁ > 0. When ψ ₁ < 0, then the bias of ${\widehat{\beta }}_{0\mathrm{C}}$ is negative, and would be a mirror image of those in Figure 1. Larger, more negative values of ψ ₁ would lead to a greater downward bias.

The bias of the slope coefficient, ${\widehat{\beta }}_{1\mathrm{C}}$, under selection model (20) is given by:

$$\text{Bias}\left[{\widehat{\beta }}_{1\mathrm{C}}\right]\approx \left[H_1\left(1\right)\left({\overline{v}}_1+{\tau }^2\right)-H_1\left(0\right)\left({\overline{v}}_0+{\tau }^2\right)\right]{\psi }_1,$$ (23)

where ${\overline{v}}_1$ is the mean v _i among effects for which X _i  = 1 and R _i  = 1. As with ${\widehat{\beta }}_{0\mathrm{C}}$, the bias of ${\widehat{\beta }}_{1\mathrm{C}}$ is an increasing function of ψ ₁. If T _i has a strong positive correlation with R _i , then ψ ₁ will be larger and so will the bias of ${\widehat{\beta }}_{1\mathrm{C}}$.

When all studies have approximately the same estimation error variance so that v _i  ≈ v and ${\overline{v}}_0\approx {\overline{v}}_1$, then the bias of ${\widehat{\beta }}_{1\mathrm{C}}$ is approximately:

$$\text{Bias}\left[{\widehat{\beta }}_{1\mathrm{C}}\right]\approx \left[H_1\left(1\right)-H_1\left(0\right)\right]\left(v+{\tau }^2\right){\psi }_1.$$ (24)

The expression in (24) is similar to (22), and both expressions depend on similar quantities. Like ${\widehat{\beta }}_{0\mathrm{C}}$, the bias of ${\widehat{\beta }}_{1\mathrm{C}}$ is an increasing function of τ ² + v and ψ ₁. The bias of ${\widehat{\beta }}_{1\mathrm{C}}$ also increases as a function of H ₁(1) − H ₁(0), which can be thought of as a difference in missingness rates between cases where X _i  = 1 and X _i  = 0. Note, however, that this does not imply that MAR data necessarily leads to an unbiased slope estimate. Recall that H ₁ is an approximation of the probability X _i is missing given X _i and T _i in (20): P[R _i ≠ 𝟙|T _i , X _i ]. Even if X _i were MAR (i.e., assuming ψ ₁ ≠ 0 but ψ ₂ = 0), the slope estimate would be unbiased only if the slope was zero: β ₁ = 0. This is because when β ₁ ≠ 0, we would expect different rates of missingness among studies for which X _i  = 1 than X _i  = 0 because of the relationship between R _i and T _i , as well as the relationship between T _i and X _i . Viewed this way, the bias of ${\widehat{\beta }}_{1\mathrm{C}}$ will be greatest when there are fewer complete cases, missingness is strongly related the value of the covariate X _i or to the size of effects (assuming that effects are correlated with X _i ).

To gain insight into the magnitude of bias in (24), consider the values of ψ ₁ ∈ [0.4, 7.5] and τ ² + v = 4(1 + r)/n discussed above. Note that the difference H ₁(1) − H ₁(0) = p(R = 0|X = 1, η) − p(R = 0|X = 0, η) is a difference in conditional probabilities. For reference, because both R _i and X _i are binary, then p(R = 0|X = 1) − p(R = 0|X = 0) would be equal to the correlation between R _i and X _i (assuming equal marginals in a 2 × 2 table). Thus, |p(R = 0|X = 1) − p(R = 0|X = 0)| could be as small as 0, but could possibly be as large as 1; arbitrary conventions on the size of correlations suggest that |p(R = 0|X = 1) − p(R = 0|X = 0)| = 0.5 would be a “large” value. ⁴⁰

Figure 2 shows the potential bias of ${\widehat{\beta }}_{1\mathrm{C}}$ for this example assuming the values of τ ² + v, ψ ₁, and H ₁(1) − H ₁(0) discussed above. Each panel corresponds to a given amount of heterogeneity τ ² + v, and within panels the bias is shown as a function of the difference H ₁(1) − H ₁(0) (x‐axis) and ψ ₁ (color). Figure 2 highlights that the relationship between R _i and T _i (ψ ₁) and between R _i and X _i (x‐axes) can affect the magnitude of the bias. If R _i is strongly correlated with both X _i and T _i the bias can be as large as d = 0.3 or 0.4. However, the less R _i depends on T _i or X _i , the lower the bias is.

FIGURE 2.

The bias of ${\widehat{\beta }}_{1\mathrm{C}}$ (y‐axis). Each panel corresponds to a given value of residual heterogeneity τ ² and estimation error variance v. Within panels, the bias of ${\widehat{\beta }}_{1\mathrm{C}}$ is plotted as function of differential missingness rates (p(R = 0|X = 1) − p(R = 0|X = 0)), which is analogous to the correlation between the value of X and whether it is observed. Bias is also shown as a function of ψ ₁, which is the relationship between the probability of observing X, and the effect size T. Bias is shown on the scale of Cohen's d and ψ ₁ is on the scale of a log‐odds ratio [Colour figure can be viewed at wileyonlinelibrary.com]

Recall that the mechanism in these computations is assumed to be MNAR, since ψ ₂ in (20) is nonzero. A MAR mechanism would require ψ ₂ = 0. In that case, the bias for the CCA intercept estimator ${\widehat{\beta }}_{0\mathrm{C}}$ is identical to that given in (22). However, the bias in the slope will be slightly different when ψ ₂ = 0. This is because, as noted in (24), the bias in the slope depends (loosely) on the correlation between R and X. Given the form of H ₁(X) in this example, it is possible for the bias of ${\widehat{\beta }}_{1\mathrm{C}}$ to be greater when ψ ₂ ≠ 0 (MNAR) than when ψ ₂ = 0 (MAR), which can occur if the correlation between R and T, and R and X are in the same direction (i.e., ψ ₁ and ψ ₂ are in the same direction). However, when ψ ₂ ≠ 0 (MNAR), the bias of ${\widehat{\beta }}_{1\mathrm{C}}$ can also decrease in magnitude relative to when ψ ₂ = 0 if ψ ₁ and ψ ₂ are in the opposite directions.

A key implication of this example is that under the relatively simple selection model in (20), CCA intercept estimators can have substantial bias. This bias does not change even if ψ ₂ = 0 and the data are MAR. Thus, inferences for the group of studies for which X _i  = 0 will be biased. Moreover, because inference for the group of studies for which X _i  = 1 will depend on the intercept estimate, those inferences will also be biased even if the slope estimator ${\widehat{\beta }}_{1\mathrm{C}}$ is unbiased.

6. BIAS IN SHIFTING‐CASE ANALYSES

Shifting‐case analyses (SCA) are a common approach in meta‐regression when there are very few complete cases across multiple covariates. These analyses involve fitting multiple regression models, where each model omits some of the covariates of interest. In this sense, SCA can be thought of as a set of regression models. Consider one model from that set, which estimates regression coefficients for some subset S of the relevant covariates using the estimator ${\widehat{\beta }}_{\mathrm{S}}$ in Equation (8). Recall that E refers to the set of covariates omitted from the model, and that the estimator ${\widehat{\beta }}_{\mathrm{S}}$ conditions on a set of missingness patterns $R_i\in {\mathrm{ℛ}}_j$. The set of missingness patterns ${\mathrm{ℛ}}_j$ is such that R _iS  = 1 so that all included covariates are observed.

To understand the conditions under which ${\widehat{\beta }}_{\mathrm{S}}$ is unbiased, we can write a shifting‐case model as:

$$p\left(T_i|X_{i\mathrm{S}},v_i,R_i\in {\mathrm{ℛ}}_j,\eta ,\psi \right)=\frac{p\left(R_i\in {\mathrm{ℛ}}_j|T_i,X_{\mathit{\text{iS}}},v_i,\psi \right)p\left(T_i|X_{\mathit{\text{iS}}},v_i,\eta \right)}{p\left(R_i\in {\mathrm{ℛ}}_j|X_{\mathit{\text{iS}}},v_i,\eta ,\psi \right)}.$$ (25)

The model in (25) is slightly different from the models in the previous sections in that all of the functions depend on the covariates included in a given regression X _iS rather than the complete set of relevant covariates X _i . Thus, the function p(T _i |X _iS, v _i ) can be thought of as a partial‐data model, since it omits some of the relevant covariates. The partial‐data model p(T _i |X _iS , v _i ) need not be equivalent to the complete‐data model p(T _i |X _i , v _i ) because the former conditions only on X _iS and not the full set of covariates X _i . These models would only be equivalent if T _i  ⊥ X _iE|X _iS, v _i . That is, unless the excluded covariates are completely unrelated to effect size (given the covariates included in the SCA model), then ${\widehat{\beta }}_{\mathrm{S}}$ will be biased even if X _iS are completely observed.

The model in (25) suggests a very strict set of conditions for which ${\widehat{\beta }}_{\mathrm{S}}$ is unbiased which concern the missingness mechanism and the relevance of excluded covariates in a given shifting‐case regression. First, missingness must be independent of effect sizes. This arises if R _i  ⊥ T _i |X _iS, v _i or R _i  ⊥ (T _i , X _iS)|v _i , which is a similar assumption as that made for unbiased CCA. In effect, this assumption implies that missingness is independent of effect sizes T _i (and potentially covariates), but could be correlated with estimation error variances v _i .

Second, any excluded covariates must be completely irrelevant to effect sizes given the included covariates: T _i  ⊥ X _iE|X _iS, v _i . This assumption is equivalent to assuming that β _j  = 0 for all j ∈ E, so that any omitted variables in a given shifting‐case regression are assumed to have a coefficient of zero. A related assumption is that (T _i , X _iS) ⊥ X _iE|v _i , which would imply that the complete‐data likelihood involves no interactions between X _iS and X _iE and that X _iS and X _iE are orthogonal. Given the nature of many meta‐analyses wherein included studies and effects are ostensibly “found objects,” correlation among multiple covariates is a common issue in meta‐regression. ²⁵ Note that conditions on omitted covariates and omitted observations must hold in order for ${\widehat{\beta }}_{\mathrm{S}}$ to be unbiased.

When the assumptions about omitted variables and effect sizes are not met, ${\widehat{\beta }}_{\mathrm{S}}$ will be biased. The magnitude of the bias will depend on a number of factors, including the amount of missingness, the missingness mechanism, and the relevance of any excluded covariates. The bias can be expressed as:

$$\text{Bias}\left[{\widehat{\beta }}_S\right]={\left({\mathbf{X}}_{\mathit{US}}^{\mathrm{T}}{\mathbf{W}}_U{\mathbf{X}}_{\mathit{US}}\right)}^{-1}{\mathbf{X}}_{\mathit{US}}^{\mathrm{T}}{\mathbf{W}}_U{\mathbf{X}}_{\mathit{UE}}{\beta }_{\mathrm{E}}+{\left({\mathbf{X}}_{\mathit{US}}^{\mathrm{T}}{\mathbf{W}}_U{\mathbf{X}}_{\mathit{US}}\right)}^{-1}{\mathbf{X}}_{\mathit{US}}^{\mathrm{T}}{\mathbf{W}}_U{\mathrm{\Delta }}_{\mathit{jU}},$$ (26)

where X _UE is the matrix of omitted covariates and β _E comprises the coefficients for the omitted covariates. The term Δ _j is a vector of biases due to missingness Δ _j  = [δ _1j , …, δ _kj ] and Δ _jU is the subset of Δ _j for which $R_i\in {\mathrm{ℛ}}_j$. Note that the δ _ij are the biases due solely to missingness as in Equation (11).

The expression in (26) shows that a shifting‐case analysis suffers from two sources of bias. The first source, captured in the first term in (26), is a function of the coefficients for the excluded covariates β _E , which we refer to as omitted variable bias. Discussion in a previous section argued that the term omitted variable bias assumes that β is the target of inference in an SCA, which may or may not be the case. If β is the target of inference, omitted variable bias arises even if no X _iS are missing, and is related to the issue of multicollinearity in linear models. In fact, if the columns in X _US and X _UE are orthogonal, so that the omitted variables are independent of the included variables, then the omitted variable bias will be zero. When the omitted variables are not orthogonal to the included variables, the bias will be nonzero, and it will depend in large part on the contribution of the omitted variables in the complete‐data model X _UE β _E. The estimator ${\widehat{\beta }}_{\mathrm{S}}$ will have greater bias if the coefficients for the omitted variables β _E are larger and the omitted covariates X _UE are correlated with the included covariates X _US.

The second term in (26) captures the bias due to ignoring observations missing X _iS. This missingness bias is a function of Δ _jU , which is itself a vector of biases for each effect, and it can be understood in terms of its individual components δ _ij . Because the δ _ij are of the same form for the complete‐case and shifting‐case models, the missing data bias for an SCA is governed by similar factors as the CCA, and are quite possibly similar in magnitude. Based on (13), δ _ij will be positive if T _i is strongly correlated with whether $R_i\in {\mathrm{ℛ}}_j$, and δ _ij will be greater in magnitude when that correlation is larger.

Taken together, shifting‐case estimators can be even more biased than complete‐case estimators. This occurs if the omitted variable and the missingness biases are in the same direction (e.g., both are positive). For both biases to be in the same direction, correlation between T _i and the omitted variables X _iE must be in the same direction as the correlation between T _i the probability that X _iS is observed. If, however, the omitted variable and missingness biases are in opposite directions, this can reduce the bias of a shifting‐case estimator. It is worth noting, however, that it will almost always be impossible to confirm the direction of biases, since they depend on potentially unobserved covariates.

6.1. Example: shifting‐cases analysis with two binary covariates

Suppose X _i  = [1, X _i1, X _i2] and X _i1 and X _i2 are binary covariates such that the regression model of interest is

$$T_i={\beta }_0+{\beta }_1{X}_{i1}+{\beta }_2{X}_{i2}+u_i+e_i.$$ (27)

If there is missingness in both X _i1 and X _i2, then R _i  ∈ {0, 1}² so that R _i  = [1, 1] indicates both covariates are observed, and R _i  = [1, 0] indicates only X _i1 is observed. If missingness is such that R _i  = [1, 1] for very few effect estimates, then an SCA might involve regressing T _i on the observed values of X _i1 and then on the observed values of X _i2.

The first regression would take only rows for which X _i1 is observed, so that $R\in {\mathrm{ℛ}}_1=\left\{\left[1,1\right],\left[1,0\right]\right\}$ and the excluded X _i2 could be either 0 or 1. The shifting‐case estimators follow from Equation (8):

$${\widehat{\beta }}_{0\mathrm{S}}=\frac{\sum _{X_1=0}{w}_i{T}_i}{\sum _{X_1=0}{w}_i},{\widehat{\beta }}_{1\mathrm{S}}=\frac{\sum _{X_1=1}{w}_i{T}_i}{\sum _{X_1=1}{w}_i}-{\widehat{\beta }}_{0\mathrm{S}}.$$

Assume that missigness follows the following log‐linear model:

$$\text{logit}\left[p\left(R_i\in {\mathrm{ℛ}}_1|T_i,X_{i1},v_i\right)\right]={\psi }_0+{\psi }_1{T}_i+{\psi }_2{X}_{i1}.$$ (28)

Note that this gives the log‐odds that an effect is included in the model given T _i and X _i1, and that X _i2 is not involved. Further, because the distribution of R _i depends on X _i1, the mechanism is MNAR.

Given the selection model in (28), the bias of the coefficient estimators can be written as:

$$\text{Bias}\left[{\widehat{\beta }}_{0S}\right]={\beta }_2\frac{\sum _{X_1=0,X_2=1}{w}_i}{\sum _{X_i=0}{w}_i}+\frac{\sum _{X_{i1}=0}{w}_i{\tilde{\delta }}_{i1}}{\sum _{X_{i1}=0}{w}_i},$$ (29)

$$\text{Bias}\left[{\widehat{\beta }}_{1S}\right]={\beta }_2\left(\frac{\sum _{X_{i1}=1,X_{i2}=1}{w}_i}{\sum _{X_{i1}=1}{w}_i}-\frac{\sum _{X_{i1}=0,X_{i2}=1}{w}_i}{\sum _{X_{i1}=0}{w}_i}\right)+\left(\frac{\sum _{X_{i1}=1}{w}_i{\tilde{\delta }}_{i1}}{\sum _{X_{i1}=1}{w}_i}-\frac{\sum _{X_{i1}=0}{w}_i{\tilde{\delta }}_{i1}}{\sum _{X_{i1}=0}{w}_i}\right).$$ (30)

Here ${\tilde{\delta }}_{i1}$ are the missingness biases as defined above, and whose approximate values is given in (13). To distinguish from the δ _i1 from the complete‐case example, we use the $\tilde{\delta }$ notation.

Both the bias of ${\widehat{\beta }}_{0S}$ and ${\widehat{\beta }}_{1S}$ depend on two terms. The first term in each expression is the omitted variable bias, and the second term in each expression is the missingness bias. Consider the omitted variable biases, which characterize differences between β _S and β. Note that model (27) inherently specifies β ₀, β ₁, and β ₂ as parameters of interest. Because of this, we consider omitted variable bias as relevant to the appraisal of ${\widehat{\beta }}_{\mathrm{S}}$. When effects are estimated with roughly the same precision, so that w _i  ≈ w, then the omitted variable biases reduce to

$$\text{Omitted}\mathrm{Var}.\text{Bias}\left[{\widehat{\beta }}_{0\mathrm{S}}\right]={\beta }_{2}p\left(X_2=1|X_1=0\right),$$ (31)

$$\begin{array}{l}\text{Omitted}\mathrm{Var}.\\ \text{Bias}\left[{\hat{\beta }}_{1\mathrm{S}}\right]={\beta }_2\left[p\left(X_2=1|X_1=1\right)-p(X_2=1|X_1=0)\right].\end{array}$$ (32)

The omitted variable biases for each coefficient can be seen as depending on two quantities. Both (31) and (32) are increasing in β ₂, which is the contribution of X _i2 to the complete‐data model. The omitted variable bias for ${\widehat{\beta }}_{0\mathrm{S}}$ is also increasing in p(X ₂ = 1|X ₁ = 0). The bias for ${\widehat{\beta }}_{1\mathrm{S}}$ in (32) is increasing in p(X ₂ = 1|X ₁ = 1) − p(X ₂ = 1|X ₁ = 0). Because both X _i1 and X _i2 are binary, this difference is roughly equivalent to their Pearson correlation (assuming equal marginals). If X _i1 ⊥ X _i2, then their correlation is zero, and the omitted variable bias will be zero. However, if X _i1 and X _i2 are correlated, the bias of ${\widehat{\beta }}_1$ will depend on how strongly correlated X _i1 and X _i2 are, and how big β ₂ is.

Figure 3 shows the omitted variable bias of ${\widehat{\beta }}_0$ (left plot) and ${\widehat{\beta }}_1$ (right plot) as a function of β ₂. Both the bias and β ₂ are shown on the scale of Cohen's d. In the left plot π ₀₁ = p(X _i2 = 1|X _i1 = 0) is the proportion of X _i2 = 1 when X _i1 = 0. In the right plot, ρ ₁₂ = p(X _i2 = 1|X _i1 = 1) − p(X _i2 = 1|X _i1 = 0), which is roughly the correlation between X _i1 and X _i2. Note that because ρ ₁₂ can be intuited as (roughly) a Pearson correlation, the values in the figure include 0, 0.1 (i.e., a “small” correlation), 0.3 (medium correlation), and 0.5 (large correlation). ⁴⁰

FIGURE 3.

The omitted variable bias of ${\widehat{\beta }}_{0\mathrm{S}}$ and ${\widehat{\beta }}_{1\mathrm{S}}$ for model (27) as a function of the omitted variable coefficient β ₂. The bias (y‐axis) and β ₂ (x‐axis) are on the scale of Cohen's d. The bias displayed is solely due to omitting X _i2 from model (27). In the left plot, lines are colored according to π ₀₁ = p(X _i2 = 1|X _i1 = 0). In the right plot, lines are colored according to ρ ₁₂ = p(X _i2 = 1|X _i1 = 1) − p(X _i2 = 1|X _i1 = 0) [Colour figure can be viewed at wileyonlinelibrary.com]

The figure shows that if β ₂ = 0 so that X _i2 is independent of T _i given X _i1, that both ${\widehat{\beta }}_{0\mathrm{S}}$ and ${\widehat{\beta }}_{1\mathrm{S}}$ will be unbiased. However, when β ₂ is nonzero, both estimators will be biased. If X _i1 and X _i2 are highly correlated, or if X _i2 = 1 when X _i1 = 0 with high probability, the bias of both estimators will about as large as a “small” effect (i.e., d = 0.2) when β ₂ is larger than 0.2. For ${\widehat{\beta }}_{1\mathrm{S}}$ the bias will be less than about d = 0.05 when |β ₂| ≤ 0.1 or if ρ ₁₂ < 0.5.

Figure 3 does not take into account any bias induced by missingness. However, because the missingness mechanism in (28) is the same as the mechanism for the complete‐case example (20), the missingness bias for ${\widehat{\beta }}_{0\mathrm{S}}$ is the same as that for ${\widehat{\beta }}_{0\mathrm{C}}$ in (22), which is shown in Figure 1. Likewise, the missingness bias for ${\widehat{\beta }}_{1\mathrm{S}}$ is the same as that for ${\widehat{\beta }}_{1\mathrm{C}}$ in (23), which is shown in Figure 2.

Thus, the total bias of ${\widehat{\beta }}_{0\mathrm{S}}$ will be the sum of the omitted variable biases shown in Figure 3 and the missingness biases shown in Figures 1 and 2. If both the omitted and missingness biases are on the higher end, the total bias of ${\widehat{\beta }}_0$ might be as large as d = 0.6 to over 1.0. Likewise, the total bias of ${\widehat{\beta }}_{1\mathrm{S}}$ will be the sum of the omitted variable biases shown in Figure 2 and the missingness biases shown in Figure 3, and can be larger than d = 0.6.

As noted above, the missingness bias and omitted variable bias can be in the different directions. For instance, if β ₂ < 0 but ${\tilde{\delta }}_{\mathit{ij}}>0$, then the omitted variable bias for ${\widehat{\beta }}_{0\mathrm{S}}$ will be negative, but the missingness bias will be positive. In such cases, the bias of the shifting‐case estimators could be smaller than the bias of the complete‐case estimators. However, because the biases depend on unknown (and potentially unobserved) quantities, it will often be impossible to empirically verify the magnitude or direction of the bias.

7. IMPLICATIONS FOR EMPIRICAL EXAMPLE

The theoretical results above suggest that there are conditions under which the coefficient estimates from the CCA and SCA of the substance abuse data in Table 2 are substantially biased. However, it will be difficult, if not impossible, to determine just how biased those estimates are, even given the simplified examples in the previous sections. First, the missingness mechanism is not known for the substance abuse data. Even if we assume that the mechanism follows a log‐linear model like that in (20) or (28), the resulting formulas for the bias depend on quantities, such as ψ and η that are not known, and cannot be estimated in the presence of missing data without further assumptions.

However, one approach to examining bias in the estimates presented in Table 2 would involve stochastically imputing the missing X _ij in the data. In the same vein as multiple imputation (MI), each set of imputed values constitutes a “complete” dataset from which we can compute the parameters relevant to bias. ¹³ , ⁴⁵ Given an imputed dataset, we can compute (a) the difference in the resulting ${\widehat{\beta }}^{\left(i\right)}$ for the ith imputed dataset and ${\widehat{\beta }}_{\mathrm{S}}$, (b) the quantities that govern bias in the formulas above, including ψ, H(X), and τ ². This allows us not only to assess the bias, but also to examine which aspects of the missing data are driving it.

As with MI, the accuracy of the resulting estimated quantities depends on the validity of assumptions regarding missingness and the accuracy of the imputation model. Thus, we would urge interpretation of the following results as potential biases in the CCA and SCA estimators presented earlier in this article, rather than a precise estimate of the bias. We generated m = 1000 imputations using the mice software in the R programming language. ⁴⁶ Estimates of η were computed using metaphor, specifying a Paule‐Mandel estimator for the variance component τ ². ⁴⁷ To estimate the log‐linear model selection parameters ψ in (28), as well as H(X), we used a logistic regression with the missingness indicator R _ij and T _i and X _ij as the predictors.

Here, we focus on results for β ₀ and β ₁. Consider the regression of T _i on X _i1 reported in Table 2. We can view this as a single regression in an SCA that includes only observations for which X _i1 is observed. As noted above, the resulting estimators of the intercept β ₀ and slope β ₁ will exhibit bias due to missingness given in (22) and (23) and bias due to omitting variables as in (31) and (32). Recall that the bias due to missingness in an SCA under this model will be similar to the bias derived for a CCA.

Figure 4 plots the omitted variable bias, missingness bias, and total bias for both ${\widehat{\beta }}_{0\mathrm{S}}$ and ${\widehat{\beta }}_{1\mathrm{S}}$. Results are reported on the scale of Cohen's d. Omitted variable bias for ${\widehat{\beta }}_{0\mathrm{S}}$ ranges from −0.05 to 0.04 with a mean of −0.01; omitted variable bias for ${\widehat{\beta }}_{1\mathrm{S}}$ ranges from −0.32 to 0.12 with a mean of −0.05. Similarly the bias due to missingness could feasibly range from 0.04 to 0.07 with a mean of 0.06 for ${\widehat{\beta }}_{0\mathrm{S}}$, while the missingness bias of ${\widehat{\beta }}_{1\mathrm{S}}$ might range from 0 to 0.12 with a mean of 0.06. Note that while the omitted variable bias and missingness bias are in opposite directions in this example, this need not be the case in general; both biases could feasibly be in the same direction for other data. In sum, this amounts to a total bias of 0.01 to 0.09 for ${\widehat{\beta }}_{0S}$ and from −0.25 to 0.18 for ${\widehat{\beta }}_{1\mathrm{S}}$.

FIGURE 4.

Bias in shifting‐case analysis (SCA) regression of T _i on X _i1. For both the intercept β ₀ and slope β ₁, these boxplots show the total potential bias of the SCA estimators, as well as the omitted variable and missingness bias. Units are shown on the scale of Cohen's d

8. DISCUSSION

This article described a selection model approach to study the bias of two common methods for conducting meta‐regressions with missing covariates: complete‐case and shifting‐case analyses. Under certain assumptions regarding the selection model, we obtained expressions for the approximate bias of coefficient estimators. These expressions were presented in a general form, which was then unpacked by way of examples.

We found that both CCA and SCA will produce biased coefficient estimates unless certain conditions are met. While discussion regarding potential bias of these analyses has largely focused on traditional mechanism taxonomy of MCAR, MAR, and MNAR, we found that bias depends more on the precise model for missingness rather than these broader classifications. Certain mechanisms that are MAR or MNAR can lead to unbiased estimates with CCA and SCA, while other MAR or MNAR mechanisms can induce substantial bias. Complete‐case estimators are unbiased if the probability that all relevant covariates are observed is (conditionally) independent of the effect size estimate. Shifting‐case estimators are unbiased if, in addition to effect sizes being independent of missingness, the covariates omitted from a model have no relationship with the effect size (assuming the full model involving β is of interest). When these conditions are not met, the bias of coefficient estimates can be substantial—as large as d = 0.4 to d = 0.8—depending on the missingness mechanism (i.e., parameters in the selection model), the missingness rate, and the relevance of any omitted covariates.

Results for both CCA and SCA suggest that bias due to missingness will tend to increase in magnitude as a function of the total variation in the data. This means that if studies have small sample sizes (i.e., v _i are large) or there is substantial residual between‐effect heterogeneity τ ², the bias of a CCA or SCA will be greater. Because meta‐regression is used to explain between‐effect variation τ ², models capable of explaining much of that variation will have lower bias in CCA and SCA estimates. However, even very modest amounts of residual variation can still result in substantial bias.

An important aspect of these findings is that bias will depend on unknown parameters and unobserved data. This means that it will be impossible to empirically verify the magnitude or direction of the bias. Even the estimated biases from the substance abuse data, which were on the order of about d = ±0.1 may not be entirely accurate, as so much of that data is missing. Further, it will require strong assumptions regarding the missingness mechanism to correct any bias. These assumptions may be buttressed by theory about scientific reporting, data collection, and data curation.

In addition, it is not immediately clear how commonly the conditions required for unbiased complete‐ and shifting‐case estimators arise. Recent empirical work on examining missingness in meta‐analytic datasets found that effect sizes can be strongly correlated with missingness, though this is not always the case. ⁴⁸ Further, the issues of multicollinearity and confounding in meta‐regression, including those discussed by Lipsey, ²⁵ would suggest that omitting variables in an SCA are likely to induce bias.

Based on these results, our primary recommendation is that analysts attempt to understand the missingness mechanisms and patterns in their data. This can leverage knowledge about standard reporting and coding practices, as well as exploratory analyses. ⁴⁸ If there is very little missingness, or if there is a good reason to assume that missingness is uncorrelated with effect size estimates, a CCA may be a reasonable option. However, we would discourage analysts from continuing to use SCA because it would seem unlikely that omitted variable biases are zero in practice.

We would also suggest analysts investigate the feasibility of alternative estimation methods. Ibrahim ³² describes an EM algorithm for generalized linear models with missing covariates, and Ibrahim, lipsitz, and Chen ³³ extend that algorithm when covariates are MNAR. In addition, full‐information maximum likelihood (FIML) has long been used in linear models, ¹⁴ , ⁴⁹ and has shown some promise for meta‐regression involving continuous covariates. Finally, multiple imputation has become something of a standard approach for handling missing data across a number of fields. ¹³ , ³⁰ , ⁴⁵

However, employing any of these alternative strategies is not necessarily straightforward for meta‐analysts. To our knowledge, the EM algorithm for missing covariates has yet to be implemented in standard meta‐analytic software. Although FIML for meta‐regression model is available in SEM framework, ⁵⁰ the approach has not been empirically validated under various conditions. How best to specify quality imputation models for MI analyses is something of an open question for meta‐regression, as is the potential inaccuracies incurred by using poor imputation models. Research on and clear implementation of these methods for meta‐regression model would seem to be of great use for meta‐analysts.

CONFLICT OF INTEREST

The authors declare that there is no conflict of interest for this article.

AUTHOR CONTRIBUTIONS

Jacob M. Schauer, Jihyun Lee, Karina Diaz, and Therese D. Pigott all contributed to the mathematical results and applied examples in this paper.

APPENDIX A. Approximate bias for log‐linear selection models

A.1.

Proposition: Suppose p(T _i |X _i , v _i ) is the standard fixed‐ or a random effect meta‐regression model in Equation (2), and suppose $p\left(R_i\cancel{\in }{\mathrm{ℛ}}_j|T_i,X_i,v_i\right)=H_j\left(T_i,X_i,v_i\right)$ follows the log‐linear model in (12). Then:

$$E\left[T_i|X_i,v_i,R_i\in {\mathrm{ℛ}}_j\right]\approx X_i\beta +H_j\left(X_i\beta ,X_i,v_i\right)\left({\tau }^2+v_i\right)\sum _{m=0}^{m_j}{\psi }_{\mathit{mj}}{f}_{{\mathit{mj}}^{\prime }}\left(X_i\beta ,X_i,v_i\right),$$ (A1)

where $f_{{\mathit{mj}}^{\prime }}\left(X_i\beta ,X_i,v_i\right)=\frac{\partial }{\partial T_i}{f}_{\mathit{mj}}{\left.\left(T_i,X_i,v_i\right)\right|}_{T_i=X_i^T\beta }$. Therefore, the bias of the conditional expectation is given by:

$${\delta }_{\mathit{ij}}\approx H_j\left(X_i\beta ,X_i,v_i\right)\left({\tau }^2+v_i\right)\sum _{m=0}^{m_j}{\psi }_{\mathit{mj}}{f}_{{\mathit{mj}}^{\prime }}\left(X_i\beta ,X_i,v_i\right).$$ (A2)

Proof:

In this proof, we drop the subscript i for sake of simplicity. Denote

$$\begin{array}{l}{H}_j\left(\mathit{X\beta },X,v\right)\equiv H_j\left(X,v\right)=P{\left.\left[R=\cancel{\in }{\mathrm{ℛ}}_j\mid T,X,v\right]\right|}_{T=\mathit{X\beta }}\\ G_j\left(X,v\right)=1-H_j\left(X,v\right)\\ g_j\left(X,v\right)=P\left[R\in {\mathrm{ℛ}}_j\mid X,v\right].\end{array}$$

Then an approximation for $E\left[T\mid X,v,R\in {\mathrm{ℛ}}_j\right]$ is as follows:

$$\begin{array}{l}\text{omitting �subscripts},�\text{Taylor series for exponents in}P\left[R\in {\mathrm{ℛ}}_j|T,X,v\right]\\ \mathrm{at}T=\mathit{X\beta }\\ \begin{array}{rcl}E\left[T|X,v,R\in {\mathrm{ℛ}}_j\right]& =& \int \frac{T\exp \left\{-\frac{{\left(T-\mathit{X\beta }\right)}^2}{2\left({\tau }^2+v\right)}+\sum _i{\psi }_{\mathit{ij}}{f}_{\mathit{ij}}\left(T,X,v\right)\right\}}{g_j\left(X,v\right)\sqrt{2\pi \left({\tau }^2+v\right)}\left(1+e^{\sum _i{\psi }_{\mathit{ij}}{f}_{\mathit{ij}}\left(T,X,v\right)}\right)}\mathit{dT}\\ & =& \int \frac{T}{g_j\left(X,v\right)\sqrt{2\pi \left({\tau }^2+v\right)}}\exp \{-\frac{{\left(T-\mathit{X\beta }\right)}^2}{2\left({\tau }^2+v\right)}\\ & & +\sum _i{\psi }_{\mathit{ij}}{f}_{\mathit{ij}}\left(\mathit{X\beta },X,v\right)+\sum _i{\psi }_{\mathit{ij}}{f}_{{\mathit{ij}}^{\prime }}\left(\mathit{X\beta },X,v\right)\left(T-\mathit{X\beta }\right)\\ & & -\log \left(1+e^{\sum _i{\psi }_{\mathit{ij}}{f}_{\mathit{ij}}\left(\mathit{X\beta },X,v\right)}\right)\\ & & -G_j\left(X,v\right)\left(\sum _i{\psi }_{\mathit{ij}}{f}_{{\mathit{ij}}^{\prime }}\left(\mathit{X\beta },X,v\right)\right)\left(T-\mathit{X\beta }\right)+O\left(T^2\right)\}\mathit{dT}\\ & \approx & \int \frac{T}{g_j\left(X,v\right)\sqrt{2\pi \left({\tau }^2+v\right)}}\exp \{-\frac{1}{2\left({\tau }^2+v\right)}(T^2-2\mathit{TX\beta }\\ & & -2T\left({\tau }^2+v\right)\sum _i{\psi }_{\mathit{ij}}{f}_{{\mathit{ij}}^{\prime }}\left(\mathit{X\beta },X,v\right)\\ & & +2T\left({\tau }^2+v\right)G_j\left(X,v\right)\left(\sum _{i\in \mathcal{D}}{\psi }_{\mathit{ij}}{f}_{{\mathit{ij}}^{\prime }}\left(\mathit{X\beta },X,v\right)\right)+\dots )\}\mathit{dT}\\ & =& \mathit{X\beta }+H_j\left(X,v\right)\left({\tau }^2+v\right)\sum _i{\psi }_{\mathit{ij}}{f}_{{\mathit{ij}}^{\prime }}\left(\mathit{X\beta },X,v\right)■\end{array}\end{array}$$

Note that this uses a first order Taylor expansion of the log‐linear model at T = Xβ, and thus assumes the f _mj are differentiable. The approximation will be more accurate if τ ² + v _i are small. A more accurate approximation is possible if the f _mj are linear in T _i . In that case, only an approximation of the denominator of the log‐linear model is required.

Schauer JM, Lee J, Diaz K, Pigott TD. On the bias of complete‐ and shifting‐case meta‐regressions with missing covariates. Res Syn Meth. 2022;13(4):489‐507. doi: 10.1002/jrsm.1558

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available from the corresponding author upon reasonable request. Code used in this project is available in a GitHub repository (link: https://github.com/j3schaue/meta_analysis_md_diagnostics/), which contains analysis code (link: https://github.com/j3schaue/meta_analysis_md_diagnostics/blob/master/writeup/cca_paper/properties_of_cca_sca.Rmd). Note that the key findings of this article concern statistical properties of various incomplete data estimators, while the data were used for demonstration purposes only.