A Method of Moments Estimator for Random Effect Multivariate Meta-Analysis

SUMMARY

Meta-analysis is a powerful approach to combine evidence from multiple studies to make inference about one or more parameters of interest, such as regression coefficients. The validity of the fixed effect model meta-analysis depends on the underlying assumption that all studies in the meta-analysis share the same effect size. In the presence of heterogeneity, the fixed effect model incorrectly ignores the between-study variance and may yield false positive results. The random effect model takes into account both within-study and between-study variances. It is more conservative than the fixed effect model and should be favored in the presence of heterogeneity. In this paper, we develop a noniterative method of moments estimator for the between-study covariance matrix in the random effect model multivariate meta-analysis. To our knowledge, it is the first such method of moments estimator in the matrix form. We show that our estimator is a multivariate extension of DerSimonian and Laird’s univariate method of moments estimator, and it is invariant to linear transformations. In the simulation study, our method performs well when compared to existing random effect model multivariate meta-analysis approaches. We also apply our method in the analysis of a real data example.

1. Introduction

Meta-analysis has been widely used to increase precision and power by combining studies (Cohn and Becker, 2003). Assuming that the effect to be estimated is the same in all studies, the fixed effect meta-analysis is often successfully used to combine the studies and obtain the point estimate for the effect size and its standard error. However, the underlying assumption of equal effect sizes of the fixed effect model may be violated (DerSimonian and Laird, 1986). Different studies may come from different populations, use different protocols and have different levels of confounding or effect modifying variables. Thus, the studies may not share a common effect size of interest. In the presence of heterogeneity, the fixed effect model underestimates the standard error of the point estimate by ignoring the between-study variance. False positive findings may be generated when the fixed effect model is inappropriately used.

The random effect model allows both within-study and between-study variances, and is more conservative than the fixed effect model in declaring significance of the effect size of interest. In the presence of heterogeneity, the random effect model provides more appropriate standard error of the point estimate and better confidence interval than the fixed effect model. When meta-analyzing a single parameter of interest, one can estimate the between-study variance by using the EM algorithm (Dempster et al., 1977) or other iterative methods; a noniterative method of moments estimator has also been proposed (DerSimonian and Laird, 1986).

Meta-analysis has also been applied to two or more correlated effect estimates (Raudenbush et al., 1988), such as regression coefficients (Becker and Wu, 2007). Analogous to the univariate case in which the inverse variance is used as the weight, the inverse of the covariance matrix is the weight in the multivariate fixed effect model. Random effect models have also been proposed to incorporate the between-study covariance matrix (Berkey et al., 1998, van Houwelingen et al., 2002, Riley et al., 2007). However, the iterative procedure is often computer intensive and may not reach convergence. Recently, an extended DerSimonian and Laird’s method of moments estimator was proposed to solve the between-study covariance matrix (Jackson et al., 2010). Though this method is a noniterative approach, it requires calculating each element of the matrix separately, and it is not invariant to reparametrization of effect sizes.

In this paper, we proposed a novel method of moments estimator for the between-study covariance matrix in the multivariate meta-analysis. It is also a multivariate extension of DerSimonian and Laird’s univariate method of moments estimator. To our knowledge, this is the first noniterative estimator for the between-study covariance matrix in the matrix form. It is invariant to linear transformations. We perform a simulation study to compare our method with the restricted maximum likelihood (REML) method (Jennrich and Schluchter, 1986) and Jackson’s multivariate DerSimonian and Laird’s method. We also apply the three random effect methods as well as the fixed effect approach to a real data example.

2. Fixed Effect Multivariate Meta-Analysis

Suppose we are interested in meta-analyzing p correlated effects from k studies. To estimate the true effect sizes

$$\mathit{\beta }=\left[\begin{array}{c}{\beta }_1\\ {\beta }_2\\ ⋮\\ {\beta }_p\end{array}\right],$$

we need effect estimates and their covariance matrix from individual studies. For study i (1 ≤ i ≤ k), we denote the effect estimates

$${\mathit{b}}_i=\left[\begin{array}{c}{b}_{i1}\\ b_{i2}\\ ⋮\\ b_{\mathit{ip}}\end{array}\right]$$

and their covariance matrix Σ_i.

To meta-analyze vectors b₁, b₂, …, b_k and get a summary estimate, we need generalized least squares (GLS) methods instead of ordinary least squares (OLS) methods, because the variances of effect estimates from different studies are unequal. We first stack the k vectors to get a long vector with length kp

$$\mathit{b}=\left[\begin{array}{c}{\mathit{b}}_1\\ {\mathit{b}}_2\\ ⋮\\ {\mathit{b}}_k\end{array}\right].$$

Assuming that the k studies are uncorrelated, we make a blockwise diagonal matrix

$$\mathrm{\Sigma }={\left[\begin{array}{cccc}{\mathrm{\Sigma }}_1& 0& \cdots & 0\\ 0& {\mathrm{\Sigma }}_2& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & {\mathrm{\Sigma }}_k\end{array}\right]}_{\mathit{kp}\times \mathit{kp}}.$$

This is the covariance matrix of vector b.

We assume the following model holds:

$${\mathit{b}}_{\mathit{kp}\times 1}={\mathbf{W}}_{\mathit{kp}\times p}{\mathit{\beta }}_{p\times 1}+{\mathit{e}}_{\mathit{kp}\times 1},$$

where W is a stack of k identity matrices of size p × p, and we assume that the error e follows a multivariate normal distribution with means 0 and covariance matrix Σ, which is the covariance matrix of vector b.

The fixed effect model summary estimator (Raudenbush et al., 1988, Becker and Wu, 2007) is

$${\widehat{\mathit{\beta }}}_F={({\mathbf{W}}^{\prime }{\mathrm{\Sigma }}^{-1}\mathbf{W})}^{-1}{\mathbf{W}}^{\prime }{\mathrm{\Sigma }}^{-1}\mathit{b},$$

with covariance estimator

$$\mathit{Cov}({\widehat{\mathit{\beta }}}_F)={({\mathbf{W}}^{\prime }{\mathrm{\Sigma }}^{-1}\mathbf{W})}^{-1}.$$

The null hypothesis of homogeneity

$$E({\mathit{b}}_1)=E({\mathit{b}}_2)=\cdots =E({\mathit{b}}_k)=\mathit{\beta }$$

can be tested using the homogeneity test statistic

$$Q={(\mathit{b}-\mathbf{W}{\widehat{\mathit{\beta }}}_F)}^{\prime }{\mathrm{\Sigma }}^{-1}(\mathit{b}-\mathbf{W}{\widehat{\mathit{\beta }}}_F).$$

Q is a scalar. Under the null hypothesis of no heterogeneity, it asymptotically follows a chi-square distribution with (k − 1)p degrees of freedom (Becker and Wu, 2007).

3. Random Effect Multivariate Meta-Analysis

Similarly to the fixed effect model, we assume

$${\mathit{b}}_{\mathit{kp}\times 1}={\mathbf{W}}_{\mathit{kp}\times p}{\mathit{\beta }}_{p\times 1}+{\mathit{\delta }}_{\mathit{kp}\times 1}+{\mathit{e}}_{\mathit{kp}\times 1}.$$

We assume that the error e follows a multivariate normal distribution with means 0 and covariance matrix Σ. The random effect vector

$$\mathit{\delta }=\left[\begin{array}{c}{\mathit{\delta }}_1\\ {\mathit{\delta }}_2\\ ⋮\\ {\mathit{\delta }}_k\end{array}\right],$$

where δ_i (1 ≤ i ≤ k) follows a multivariate normal distribution with means 0 and covariance matrix T. Thus the covariance matrix of vector b is

$${\mathbf{\Omega }}_{\mathit{kp}\times \mathit{kp}}={\mathrm{\Sigma }}_{\mathit{kp}\times \mathit{kp}}+{\mathbf{I}}_{k\times k}\otimes {\mathbf{T}}_{p\times p}={\left[\begin{array}{cccc}{\mathrm{\Sigma }}_1+\mathbf{T}& 0& \cdots & 0\\ 0& {\mathrm{\Sigma }}_2+\mathbf{T}& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & {\mathrm{\Sigma }}_k+\mathbf{T}\end{array}\right]}_{\mathit{kp}\times \mathit{kp}}.$$

Σ_i (1 ≤ i ≤ k) is the within-study covariance matrix for study i defined above, I_k×k is the k × k identity matrix, T is the between-study covariance matrix. The symbol ⊗ denotes the Kronecker product of two matrices.

The random effect model summary estimator is

$${\widehat{\mathit{\beta }}}_R={({\mathbf{W}}^{\prime }{\mathbf{\Omega }}^{-1}\mathbf{W})}^{-1}{\mathbf{W}}^{\prime }{\mathbf{\Omega }}^{-1}\mathit{b},$$

with covariance estimator

$$\mathit{Cov}({\widehat{\mathit{\beta }}}_R)={({\mathbf{W}}^{\prime }{\mathbf{\Omega }}^{-1}\mathbf{W})}^{-1}.$$

The crucial step for the random effect approach is to estimate the between-study covariance matrix T. In this paper we use the following three methods. In all the methods, the values of the covariance matrices Σ_i (1 ≤ i ≤ k) are fixed at the obtained study-specific estimates.

3.1 Restricted maximum likelihood method

The REML estimation can be performed by maximizing

$$L(\mathbf{T})=-\frac{1}{2}\log |\mathbf{\Omega }|-\frac{1}{2}\log |{\mathbf{W}}^{\prime }{\mathbf{\Omega }}^{-1}\mathbf{W}|-\frac{1}{2}{\mathit{r}}^{\prime }{\mathbf{\Omega }}^{-1}\mathit{r},$$

where

$$\mathit{r}=\mathit{b}-\mathbf{W}{({\mathbf{W}}^{\prime }{\mathbf{\Omega }}^{-1}\mathbf{W})}^{-1}{\mathbf{W}}^{\prime }{\mathbf{\Omega }}^{-1}\mathit{b},$$

under the constraint that T is positive semi-definite (Jackson et al., 2010, Jennrich and Schluchter, 1986).

3.2 Jackson’s multivariate DerSimonian and Laird’s method

Jackson’s multivariate DerSimonian and Laird’s method calculates each entry of T separately (Jackson et al., 2010). Let

$${\stackrel{\sim }{\mathbf{T}}}_{\mathrm{i},\mathrm{j}}=\underset{\mathrm{u}=1}{\overset{\mathrm{k}}{\mathrm{\Sigma }}}\frac{\left({\mathit{b}}_{u(i)}-{\stackrel{\sim }{b}}_{i[j]}\right)\left({\mathit{b}}_{u(j)}-{\stackrel{\sim }{b}}_{j[i]}\right)}{\sqrt{{\mathrm{\Sigma }}_{u(i,i)}{\mathrm{\Sigma }}_{u(j,j)}}},$$

where

$${\stackrel{\sim }{b}}_{i[j]}=\frac{{\mathrm{\Sigma }}_{u=1}^k\frac{{\mathit{b}}_{u(i)}}{\sqrt{{\mathrm{\Sigma }}_{u(i,i)}{\mathrm{\Sigma }}_{u(j,j)}}}}{{\mathrm{\Sigma }}_{u=1}^k\frac{1}{\sqrt{{\mathrm{\Sigma }}_{u(i,i)}{\mathrm{\Sigma }}_{u(j,j)}}}},$$

and b_u(i) is the ith (1 ≤ i ≤ p) element in the vector b_u (1 ≤ u ≤ k), Σ_u(i,j) is the element on row i (1 ≤ i ≤ p) and column j (1 ≤ j ≤ p) of Σ_u (1 ≤ u ≤ k). Let

$$\begin{array}{l}{\mathit{E}}_{i,j}=\underset{\mathrm{u}=1}{\overset{\mathrm{k}}{\mathrm{\Sigma }}}\frac{{\mathrm{\Sigma }}_{u(i,j)}}{\sqrt{{\mathrm{\Sigma }}_{u(i,i)}{\mathrm{\Sigma }}_{u(j,j)}}}-\frac{{\mathrm{\Sigma }}_{\mathrm{u}=1}^{\mathrm{k}}\frac{{\mathrm{\Sigma }}_{u(i,j)}}{{\mathrm{\Sigma }}_{u(i,i)}{\mathrm{\Sigma }}_{u(j,j)}}}{{\mathrm{\Sigma }}_{\mathrm{u}=1}^{\mathrm{k}}\frac{1}{\sqrt{{\mathrm{\Sigma }}_{u(i,i)}{\mathrm{\Sigma }}_{u(j,j)}}}},\\ {\mathit{F}}_{i,j}=\underset{\mathrm{u}=1}{\overset{\mathrm{k}}{\mathrm{\Sigma }}}\frac{1}{\sqrt{{\mathrm{\Sigma }}_{u(i,i)}{\mathrm{\Sigma }}_{u(j,j)}}}-\frac{{\mathrm{\Sigma }}_{\mathrm{u}=1}^{\mathrm{k}}\frac{1}{{\mathrm{\Sigma }}_{u(i,i)}{\mathrm{\Sigma }}_{u(j,j)}}}{{\mathrm{\Sigma }}_{\mathrm{u}=1}^{\mathrm{k}}\frac{1}{\sqrt{{\mathrm{\Sigma }}_{u(i,i)}{\mathrm{\Sigma }}_{u(j,j)}}}},\end{array}$$

then

$${\widehat{\mathbf{T}}}_{\mathrm{i},\mathrm{j}}=\frac{{\stackrel{\sim }{\mathbf{T}}}_{\mathrm{i},\mathrm{j}}-{\mathit{E}}_{i,j}}{{\mathit{F}}_{i,j}}$$

is the element on row i (1 ≤ i ≤ p) and column j (1 ≤ j ≤ p) of T̂.

While the calculation is generally faster than REML, Jackson’s method requires calculating p² weighted means b̃_i[j] as intermediates for a p × p matrix T̂, and it is not invariant to reparametrization of effect sizes.

3.3 A novel multivariate DerSimonian and Laird’s method

Let

$$\mathbf{\Psi }=\mathit{Cov}({\widehat{\mathit{\beta }}}_F)={({\mathbf{W}}^{\prime }{\mathrm{\Sigma }}^{-1}\mathbf{W})}^{-1}={\left(\underset{j=1}{\overset{k}{\mathrm{\Sigma }}}{\mathrm{\Sigma }}_j^{-1}\right)}^{-1},$$

$$\mathbf{\Phi }={\mathbf{\Psi }}^{-1}-\underset{i=1}{\overset{k}{\mathrm{\Sigma }}}{\mathrm{\Sigma }}_i^{-1}\mathbf{\Psi }{\mathrm{\Sigma }}_i^{-1}=\underset{i=1}{\overset{k}{\mathrm{\Sigma }}}\left[{\mathrm{\Sigma }}_i^{-1}-{\mathrm{\Sigma }}_i^{-1}{\left(\underset{j=1}{\overset{k}{\mathrm{\Sigma }}}{\mathrm{\Sigma }}_j^{-1}\right)}^{-1}{\mathrm{\Sigma }}_i^{-1}\right],$$

$$\mathbf{A}=\underset{j=1}{\overset{k}{\mathrm{\Sigma }}}{\mathrm{\Sigma }}_j^{-1}({\mathit{b}}_j-{\widehat{\mathit{\beta }}}_F){({\mathit{b}}_j-{\widehat{\mathit{\beta }}}_F)}^{\prime }-(k-1){\mathbf{I}}_{p\times p},$$

where Σ_j, b_j (1 ≤ j ≤ k) are defined above, β̂_F is the fixed effect estimate defined above. Then

$$E(\mathbf{A})=\mathbf{\Phi T}.$$

A symmetric method of moments estimator for T is

$$\widehat{\mathbf{T}}=\frac{{\mathbf{\Phi }}^{-1}\mathbf{A}+{\mathbf{A}}^{\prime }{\mathbf{\Phi }}^{-1}}{2}.$$

See Web Appendix A for the derivation. A very nice property of our estimator is that it is invariant to linear transformation (Web Appendix B).

When p = 1, this estimator T̂ is scalar and is equal to DerSimonian and Laird’s estimator (Web Appendix C).

In the presence of heterogeneity, the homogeneity test statistic Q no longer follows a chi-square distribution. However, its expectation is a function of the true between-study covariance matrix T (Web Appendix D).

Since T is a covariance matrix, it should be positive semi-definite. While we can do the maximization under the constraint that T is positive semi-definite for the REML method, we generally have no guarantee that Jackson’s or our method of moments estimator would be positive semi-definite, especially when heterogeneity is low. A remedy for this issue is discussed in Web Appendix E, which we adopted in all the analyses for this paper. Jackson et al. used the same strategy in their paper (Jackson et al., 2010).

4. Simulation

4.1 Simulation design

To compare the performance of the REML method, Jackson’s multivariate DerSimonian and Laird’s method (MDLJ) and Chen’s multivariate DerSimonian and Laird’s method (MDLC) developed in this paper, we conducted simulation studies in the context of bivariate meta-analysis. We performed all the calculation and analysis in R-2.9.2 (R Development Core Team, 2009).

We considered 10 studies with different sample sizes. 100 between-study variances were generated from a chi-square distribution with 1 df, and values less than 0.016 or greater than 2.7 were discarded (corresponding approximately to the 10% and 90% quantiles of 1 df chi-square distribution). Then we randomly chose 2 sets of 10 variances out of the remaining values, sorted and paired them. The smallest pair was assigned to the first study as the within-study variances of the two effects and so on until the largest pair was assigned to the last study. The within-study correlation was set to 0.2 or 0.8 for all 10 studies.

We followed the procedure by Higgins and Thompson to calculate the between-study variances (Jackson et al., 2010, Higgins and Thompson, 2002). Since the variances for both outcomes were simulated from the same population, we first calculated the typical within-study variance

$${\sigma }_{\infty }^2=\underset{n\to \infty }{\lim }\frac{(n-1){\mathrm{\Sigma }}_{i=1}^n\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{${\sigma }_i^2$}\right.}{{\left({\mathrm{\Sigma }}_{i=1}^n\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{${\sigma }_i^2$}\right.\right)}^2-{\mathrm{\Sigma }}_{i=1}^n\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{${\sigma }_i^4$}\right.},$$

where ${\sigma }_i^2$ was generated as discussed above. For this parameter setting we have ${\sigma }_{\infty }^2=0.15$. Then we computed the between-study variances using

$$\begin{array}{l}{I}_1^2=\frac{{\mathbf{T}}_{11}}{{\mathbf{T}}_{11}+{\sigma }_{\infty }^2}=\frac{{\mathbf{T}}_{11}}{{\mathbf{T}}_{11}+0.15}\\ I_2^2=\frac{{\mathbf{T}}_{22}}{{\mathbf{T}}_{22}+{\sigma }_{\infty }^2}=\frac{{\mathbf{T}}_{22}}{{\mathbf{T}}_{22}+0.15}\end{array}$$

where $I_1^2$ and $I_2^2$ are the proportions of marginal variation in the first and second effects due to heterogeneity, respectively. We considered 9 scenarios, in which each of $I_1^2$ and $I_2^2$ was set to 0.2, 0.5 or 0.8 to simulate low, moderate and high heterogeneity for each effect. T₁₁ and T₂₂ are the between-study variances for the first and second effects. The between-study correlation was set to 0.2 or 0.8 to calculate the covariance.

For each study i (1 ≤ i ≤ 10), we generated the effect size vector b_i from a bivariate normal distribution with mean 0 and covariance matrix Σ_i + T, where Σ_i is the within-study covariance matrix and T is the between-study covariance matrix.

4.2 Simulation results

We summarized the simulation results for between-study correlation 0.2 and within-study correlation 0.2 in Table 1. We only presented results for the first effect and the covariance, since the second effect shows similar results. Columns 7 – 9 show the bias, mean squared error for the first summary effect estimator β̂₁ and 95% confidence interval coverage for the first effect size. The confidence interval was constructed as β̂₁ ± t_0.025,9SE (β̂₁) (Follmann and Proschan, 1999). Columns 10 – 13 show the bias of the between-study variance estimator for the first effect T̂₁₁ and that of the between-study covariance estimator T̂₁₂, and their corresponding mean squared errors. Column 14 shows the proportion of the between-study covariance matrix estimate lying on the boundary of its parameter space: either at least one of the variance estimates is 0, or the absolute value of the correlation coefficient is 1. Since the between-study covariance matrix after the Web Appendix E correction when necessary is always positive semi-definite, this column also indicates the percentage of the between-study covariance matrix not being positive definite, that is, its minimum eigenvalue is equal to 0. To allow for rounding errors, we defined being less than 10^-8 as being equal to 0.

Table 1.

Simulation results from 1000 replicates for 10 studies with between-study correlation 0.2 and within-study correlation 0.2.

$I_1^2$	$I_2^2$	T11	T12	T12	Method	Bias β̂1	MSE β̂1	Coverage β̂1	Bias T̂11	Bias T̂12	MSE T̂11	MSE T̂12	% boundary
0.2	0.2	0.0375	0.0075	0.0375	REML	−0.0065	0.0331	0.962	0.0491	0.0167	0.0235	0.0078	0.888
					MDLJ	−0.0073	0.0326	0.963	0.0473	0.0086	0.0142	0.0051	0.819
					MDLC	−0.0072	0.0326	0.963	0.0476	0.0085	0.0142	0.0052	0.824
0.2	0.5	0.0375	0.0150	0.1500	REML	−0.0001	0.0311	0.952	0.0452	0.0096	0.0218	0.0118	0.838
					MDLJ	−0.0017	0.0307	0.957	0.0447	0.0011	0.0154	0.0092	0.748
					MDLC	−0.0016	0.0307	0.958	0.0447	0.0009	0.0152	0.0094	0.751
0.2	0.8	0.0375	0.0300	0.6000	REML	0.0048	0.0334	0.962	0.0553	0.0082	0.0270	0.0343	0.708
					MDLJ	0.0054	0.0328	0.969	0.0499	−0.0044	0.0164	0.0262	0.629
					MDLC	0.0058	0.0328	0.967	0.0505	−0.0044	0.0168	0.0262	0.641
0.5	0.2	0.1500	0.0150	0.0375	REML	0.0103	0.0562	0.934	0.0369	0.0185	0.0470	0.0137	0.814
					MDLJ	0.0104	0.0548	0.938	0.0265	0.0091	0.0361	0.0104	0.738
					MDLC	0.0102	0.0549	0.939	0.0267	0.0089	0.0361	0.0105	0.737
0.5	0.5	0.1500	0.0300	0.1500	REML	0.0108	0.0527	0.924	0.0278	0.0051	0.0490	0.0195	0.731
					MDLJ	0.0121	0.0521	0.930	0.0207	−0.0017	0.0317	0.0161	0.621
					MDLC	0.0120	0.0521	0.929	0.0213	−0.0018	0.0315	0.0162	0.622
0.5	0.8	0.1500	0.0600	0.6000	REML	−0.0001	0.0499	0.925	0.0350	0.0034	0.0530	0.0500	0.552
					MDLJ	−0.0012	0.0492	0.937	0.0265	−0.0079	0.0383	0.0454	0.454
					MDLC	−0.0009	0.0492	0.935	0.0268	−0.0071	0.0380	0.0458	0.451
0.8	0.2	0.6000	0.0300	0.0375	REML	0.0099	0.1132	0.929	0.0296	0.0125	0.2405	0.0347	0.739
					MDLJ	0.0112	0.1130	0.933	0.0296	−0.0009	0.2763	0.0280	0.619
					MDLC	0.0113	0.1131	0.933	0.0299	−0.0006	0.2764	0.0279	0.612
0.8	0.5	0.6000	0.0600	0.1500	REML	0.0055	0.1147	0.925	0.0769	0.0149	0.3099	0.0567	0.545
					MDLJ	0.0046	0.1136	0.934	0.0504	0.0007	0.2876	0.0501	0.443
					MDLC	0.0045	0.1136	0.932	0.0507	0.0013	0.2879	0.0500	0.439
0.8	0.8	0.6000	0.1200	0.6000	REML	0.0256	0.1003	0.925	0.0065	0.0034	0.2153	0.1108	0.288
					MDLJ	0.0264	0.0998	0.928	0.0033	0.0048	0.2551	0.1358	0.210
					MDLC	0.0262	0.0998	0.930	0.0042	0.0055	0.2559	0.1350	0.214

The true values of β₁ and β₂ were both set to 0. $I_1^2$ and $I_2^2$ denote the proportions of marginal variation in the first and second effects due to heterogeneity. T₁₁ and T₂₂ denote the between-study variances for the first and second effects, and T₁₂ denotes the between-study covariance. REML, MDLJ and MDLC denote restricted maximum likelihood method, Jackson’s multivariate DerSimonian and Laird’s method and Chen’s multivariate DerSimonian and Laird’s method in this paper. The Web Appendix E correction was applied when the between-study covariance matrix estimate T̂ from MDLJ or MDLC was not positive semi-definite. Coverage denotes the coverage of 95% confidence interval. % boundary denotes the proportion of the between-study covariance matrix estimate lying on the boundary of its parameter space.

We can see from Table 1 that all three methods give very similar results, and our new approach is closer to Jackson’s method rather than the REML. This is not surprising because REML is a likelihood-based iterative method, while the other two are method of moments. In all scenarios REML gives more boundary-valued between-study covariance matrix estimates than the other non-iterative methods.

Bias T̂₁₁ is generally greater than 0, because we pull back negative eigenvalues to 0 when fixing covariance matrix estimates that are not positive semi-definite, we somehow bias the diagonal elements upwards. We can see that as the heterogeneity increases, this bias generally decreases, and the proportion of boundary-valued between-study covariance matrix estimates also decreases. This is consistent with our prior knowledge that fixed effect models are usually preferred when heterogeneity is low and random effect models are more appropriate when heterogeneity is high. Simulation results for other correlation coefficient settings are summarized in Web Tables 1 – 3.

4.3 Additional simulation study

We performed 6 additional simulation studies to investigate the effect of the number of studies on the results. We set $I_1^2$ and $I_2^2$ to be 0.5, between-study and within-study correlation coefficients to be 0.2, and let the number of studies change from 10 to 5, 20, 50, 100, 200 and 500. Since all three methods give similar results, only results using our new method are shown in Figure 1 and Web Table 4. As the number of studies increases, mean squared errors of β̂₁, the summary effect estimate for the first effect, and T̂₁₁, the between-study variance estimate for the first effect decrease, which is reasonable because the large-sample properties of estimators in a meta-analysis depend on the number of studies. As the number of studies goes to infinity, mean squared errors converge to 0.

Figure 1.

Simulation results from 1000 replicates for various numbers of studies with $I_1^2=I_2^2=0.5$, between-study correlation 0.2 and within-study correlation 0.2. (A) Mean squared errors of β̂₁, the first summary effect estimate, and T̂₁₁, the between-study variance estimate for the first effect. Proportion of the between-study covariance matrix estimate lying on the boundary of its parameter space. (B) Coverage of 95% confidence interval for β₁, the first effect size.

We can also see that as the number of studies increases, the proportion of the between-study covariance matrix estimate lying on the boundary of its parameter space (matrix with minimum eigenvalue 0) decreases dramatically, even though the heterogeneity is only moderate. As we fix fewer and fewer covariance matrices that are not positive semi-definite, bias of T̂₁₁ also decreases quickly to near 0. There is no clear relationship between the coverage of 95% confidence interval for β₁ and the number of studies, although only the largest sample of 500 studies has the correct coverage.

5. Application

In this application we use data from the base year of the High School Longitudinal Study of 2009 (HSLS:09) (Ingels et al., 2011). HSLS:09 is a nationally representative, longitudinal study of more than 21,000 9th graders in 944 schools who will be followed through their secondary and postsecondary years. We are interested in testing whether sex, socio-economic status and sex by socio-economic status interaction are predictive of the mathematics standardized theta score. We estimate the regression coefficients in each of the 8 race groups and perform multivariate meta-analyses on the regression coefficients to obtain the summary effect estimates.

Within each race group i, our model is

$$Y_{\mathit{ij}}={\beta }_{i0}+{\beta }_{i1}{X}_{\mathit{ij}1}+{\beta }_{i2}{X}_{\mathit{ij}2}+{\beta }_{i3}{X}_{\mathit{ij}1}{X}_{\mathit{ij}2}+{\epsilon }_{\mathit{ij}}$$

where Y_ij is the mathematics standardized theta score, X_ij1 is the sex, X_ij2 is the socio-economic status score for student j. ε_ij is the normally distributed error. The regression results are summarized in Table 2.

Table 2.

Regression results for 8 race groups.

Race i	Ni	bi1	bi2	bi3	Var(bi1)	Var(bi2)	Var(bi3)	Cov(bi1, bi2)	Cov(bi1, bi3)	Cov(bi2, bi3)
1	163	0.3161	7.4015	0.4278	2.3568	9.7029	4.4114	−1.2105	0.8524	−6.1753
2	1672	−0.3201	6.9426	−0.9816	0.2529	0.7016	0.2743	0.1498	−0.1019	−0.4167
3	2218	0.6983	4.6680	−0.2415	0.1444	0.6481	0.2608	−0.0652	0.0433	−0.3899
4	204	3.2736	4.3080	0.2052	3.8428	10.3517	4.8268	−4.5587	3.2892	−6.6684
5	3311	−0.1599	5.6398	−0.6782	0.1161	0.4363	0.1733	−0.0992	0.0645	−0.2610
6	1912	−0.6989	6.3158	−0.7918	0.1603	0.7697	0.3180	0.0242	−0.0129	−0.4686
7	110	−3.6094	9.3429	−2.8711	3.2054	17.8889	7.2101	−1.1984	0.8437	−10.7697
8	11854	0.2172	6.4078	−0.6093	0.0278	0.1184	0.0482	0.0136	−0.0091	−0.0716

N_i is the sample size in race group i. b_i1 is the regression coefficient for sex, b_i2 is the regression coefficient for socio-economic status score, b_i3 is the regression coefficient for sex by socio-economic status score interaction in race group i.

We use both the fixed effect model and random effect models to meta-analyze the regression coefficients from the 8 race groups. Table 3 shows the meta-analysis results. For this data, the homogeneity test statistic Q is 54.6, which asymptotically follows a chi-square distribution with 21 degrees of freedom under the null hypothesis of no heterogeneity. The p-value of the homogeneity test is 8.1×10^-5. Thus, the assumption of homogeneous effect sizes for the fixed effect model is violated. Since the fixed effect model does not take between-study variance into consideration, it greatly underestimates the covariance matrix for the summary effect estimates, resulting in an inflated Wald test statistic for testing the hypotheses H₀: β₁ = β₂ = β₃ = 0 versus H₁: at least one of β₁, β₂, β₃ is not 0, where β₁, β₂, β₃ are the summary effect sizes for sex, socio-economic status score and sex by socio-economic status score interaction, respectively.

Table 3.

Meta-analysis results for 8 race groups.

Method	β̂1 (SE)	β̂2 (SE)	β̂3 (SE)	T̂11	T̂22	T̂33	T̂12	T̂13	T̂23	Wald statistic
FEMA	0.0788 (0.1208)	6.2031 (0.2448)	−0.6590 (0.1550)	0	0	0	0	0	0	4141
REML	−0.0244 (0.2427)	6.1674 (0.4843)	−0.6679 (0.1865)	0.2090	1.0028	0.0657	−0.2946	0.1172	−0.2567	431
MDLJ	−0.0612 (0.2599)	6.1873 (0.2973)	−0.7038 (0.1888)	0.2558	0.1279	0.0501	−0.1221	0.0097	0.0542	567
MDLC	−0.0604 (0.2684)	6.1821 (0.2887)	−0.7009 (0.1894)	0.2805	0.1024	0.0532	−0.0948	0.0030	0.0602	571

FEMA denotes the fixed effect meta-analysis. REML, MDLJ and MDLC denote restricted maximum likelihood method, Jackson’s multivariate DerSimonian and Laird’s method and Chen’s multivariate DerSimonian and Laird’s method in this paper. β̂₁, β̂₂ and β̂₃ denote the summary effect size estimates for sex, socio-economic status score and sex by socio-economic status score interaction, respectively. T̂₁₁, T̂₂₂, T̂₃₃, T̂₁₂, T̂₁₃, T̂₂₃ are corresponding elements of the matrix T̂, the between-study covariance matrix estimate. Wald statistic is for testing the hypotheses H₀: β₁ = β₂ = β₃=0 versus H₁: at least one of β₁, β₂, β₃ of is not 0.

Jackson’s multivariate DerSimonian and Laird’s method of moments and our method give close results, while the restricted maximum likelihood method yields a different between-study covariance matrix estimate. However, all three random effect methods give very close summary effect size estimates, and the Wald test statistic reduces dramatically, compared to that from the fixed effect model.

6. Discussion

We propose a new method of moments estimator for the between-study covariance matrix in the random effect multivariate meta-analysis. We have shown in our simulation that our method gives similar results as existing random effect model multivariate meta-analysis methods. Furthermore, our method and Jackson’s multivariate DerSimonian and Laird’s method give very close results in both simulation studies and real data analysis.

Our estimator is the first matrix form method of moments estimator for the between-study covariance matrix in the random effect model multivariate meta-analysis. It is invariant to linear transformations. As a non-iterative estimator, it is very easy to calculate.

Despite its long history in combining published analysis results, meta-analysis is also very useful in multi-center or multi-ethnic studies, when different cohorts can share results from the same analysis but it is often not feasible to share original data. Generally, the fixed effect model is the first choice in a meta-analysis as it is easier to calculate and interpret, and it is more powerful than random effect models. However, in the presence of heterogeneity, results from the fixed effect model are not valid, then random effect models are preferred. When performing the random effect meta-analysis for p effects, Jackson’s method requires calculating p² weighted means b̃_i[j] (1 ≤ i ≤ p, 1 ≤ j ≤ p) in intermediate steps, which are not related to corresponding fixed effect summary estimates. Specifically, the weighted means b̃_i[i] (1 ≤ i ≤ p) are the fixed effect summary estimates in univariate meta-analyses, instead of the multivariate meta-analysis we desire. In the contrast, our method directly uses the fixed effect summary estimates vector to calculate the between-study covariance matrix estimate. It does not require performing p² additional univariate meta-analyses to calculate the intermediates.

Recently, a U-statistic based random effect model multivariate meta-analysis approach was proposed (Ma and Mazumdar, 2011). This approach does not require the normality assumption for the effect sizes as it is nonparametric. However, similar with Jackson’s method, the U-statistic method also requires estimating each element of the between-study covariance matrix separately. Future work involves the development of a matrix form estimator for the U-statistic approach.