Meta-analysis of proportions using generalized linear mixed models

Abstract

Epidemiological research often involves meta-analyses of proportions. Conventional two-step methods first transform each study’s proportion and subsequently perform a meta-analysis on the transformed scale. They suffer from several important limitations: the log and logit transformations impractically treat within-study variances as fixed, known values and require ad hoc corrections for zero counts; the results from arcsine-based transformations may lack interpretability. Generalized linear mixed models (GLMMs) have been advocated in meta-analyses as a one-step approach to fully accounting for within-study uncertainties. However, they are seldom used in current practice to synthesize proportions. This article summarizes various methods for meta-analyses of proportions, illustrates their implementations, and explores their performance using real and simulated datasets. In general, GLMMs led to smaller biases and mean squared errors, and higher coverage probabilities than two-step methods. Many software programs are readily available to implement these methods.

Introduction

Epidemiological research often involves meta-analyses of studies that report certain proportions (e.g., disease prevalence).¹ The current practice usually uses two-step meta-analysis methods. Specifically, as proportions are restricted within 0%–100%, a specific transformation is applied to study-specific proportions for approximating them with the normal distribution or stabilizing their variances. Conventional meta-analysis methods are subsequently performed on the transformed scale, and the synthesized result is then transformed back to the original proportion scale. Meta-analyses of proportions focus on estimating the overall (median or population-averaged) proportion, regardless of the transformation used; in this sense, they differ from meta-analyses of treatment comparisons, which may aim at estimating different relative effects (e.g., odds ratio, risk ratio or risk difference), depending on how event rates are transformed.^2,3

The popular transformations of proportions include the log, logit, arcsine, and Freeman–Tukey double-arcsine transformations.^4–7 They may suffer from several important limitations. The log and logit transformations require ad hoc corrections for zero counts, and typically treat within-study variances as fixed, known values, impacting the accuracy of subsequent statistical inference.⁸ On the other hand, the arcsine-based transformations may reduce the results’ interpretability and potentially lead to misleading results.^9,10

With the rapid development of computational techniques, generalized linear mixed models (GLMMs) have been increasingly advocated for meta-analyses,^11–17 mostly in the context of treatment comparisons but infrequently used for synthesizing proportions. Unlike two-step methods, they offer a one-step procedure for combining proportions; they fully account for within-study uncertainties, which are especially critical for small sample sizes and rare events, and do not require corrections for zero counts.

Although GLMMs are implemented in many software programs, they are seldom applied in practice for synthesizing proportions so far, possibly because they have not been sufficiently introduced to applied scientists. This article reviews GLMMs and two-step methods with various link functions and transformations, including the estimation of the commonly-reported overall median proportion as well as the population-averaged (marginalized) proportion. We illustrate these methods’ implementations and explore their performance using real and simulated datasets.

Methods

Two-step method

Consider a meta-analysis containing N studies, each reporting an event count e_i with sample size n_i (i = 1, …, N). Study i’s event proportion is estimated as ${\widehat{p}}_i=e_i/n_i$. The log, logit, arcsine, and double-arcsine transformations are: $g\left({\widehat{p}}_i\right)=\text{log}{\widehat{p}}_i$, with approximated variance v_i = 1/e_i − 1/n_i; $g\left({\widehat{p}}_i\right)=\text{log}\left[{\widehat{p}}_i/\left(1-{\widehat{p}}_i\right)\right]$ with v_i = 1/e_i + 1/(n_i − e_i); $g\left({\widehat{p}}_i\right)=\text{arcsin}\sqrt{{\widehat{p}}_i}$ with v_i = 1/(4n_i); and $g\left({\widehat{p}}_i\right)=\text{arcsin}\sqrt{e_i/\left(n_i+1\right)}+\text{arcsin}\sqrt{\left(e_i+1\right)/\left(n_i+1\right)}$ with v_i = 1/(n_i + 0.5), respectively.¹⁸ The log and logit transformations cannot directly handle zero counts, which are usually corrected by adding 0.5. The arcsine-based transformations do not need this correction; they are also advantageous for stabilizing variances, because their variances depend only on sample sizes, not on event counts.

Conventional meta-analysis methods are performed using the transformed proportions $y_i=g\left({\widehat{p}}_i\right)$ and their variances v_i. The synthesized estimate is $\widehat{\mu }={\sum }_{i=1}^N{w}_i{y}_i/{\sum }_{i=1}^N{w}_i$ with variance ${\left({\sum }_{i=1}^N{w}_i\right)}^{-1}$, which yields its confidence interval (CI). Here, $w_i=1/\left(v_i+{\widehat{\tau }}^2\right)$ is study i’s weight, and ${\widehat{\tau }}^2$ is an estimate of the between-study variance. The common-effect model sets ${\widehat{\tau }}^2$ to 0; however, meta-analyses of proportions are often highly heterogeneous and thus are often performed under the random-effects model. Finally, $\widehat{\mu }$ and its CI are back-transformed to the original proportion scale. The back-transformations of the log, logit, and arcsine transformations are obvious: $g^{-1}\left(\widehat{\mu }\right)=e^{\widehat{\mu }}$, $g^{-1}\left(\widehat{\mu }\right)=e^{\widehat{\mu }}/\left(1+e^{\widehat{\mu }}\right)$, and $g^{-1}\left(\widehat{\mu }\right)={\text{sin}}^2\widehat{\mu }$, respectively. However, that of the double-arcsine transformation is rather complicated:¹⁹

$$g^{-1}\left(\widehat{\mu };n\right)=0.5\left\{1-\text{sgn}\left(\text{cos}\widehat{\mu }\right){\left[1-{\left(\text{sin}\widehat{\mu }+\frac{\text{sin}\widehat{\mu }-{\text{sin}}^{-1}\widehat{\mu }}{n}\right)}^2\right]}^{0.5}\right\},$$

where sgn(·) denotes the sign function. It depends on a specific sample size n, which is defined explicitly for each study, but not for the combined result. Arguably, n can be the harmonic, geometric, or arithmetic mean of the study-specific sample sizes,¹⁹ or approximated by the inverse of $\widehat{\mu }$’s variance.¹ In addition, the above back-transformation is valid only for $\text{arcsin}\sqrt{1/\left(n+1\right)}\le \widehat{\mu }\le \text{arcsin}\sqrt{n/\left(n+1\right)}+\pi /2$; if $\widehat{\mu }$ is not within the above range, a simpler but less accurate back-transformation $g^{-1}\left(\widehat{\mu }\right)={\text{sin}}^2\left(\widehat{\mu }/2\right)$ can be used.¹

Generalized linear mixed model

GLMMs do not need any data transformations at the study level; instead, they directly model event counts with binomial likelihoods, and use a specific link function to transform latent true proportions to a linear scale. The GLMM specifies:

• e_i ~ bin(n_i, p_i); (likelihood)

• g(p_i) = μ + θ_i; (link function)

• θ_i ~ N(0, τ²), (random effects)

where p_i represents study i’s latent true proportion, μ is the overall proportion on the transformed scale via the link g(·), and τ² is the between-study variance.

GLMMs can be implemented using the function glmer() in the R package “lme4,”²⁰ which is primarily used in this article. Alternatively, one can also use R packages “metafor” and “meta”^21,22 and SAS; eAppendices A and B illustrate their usage.

Various links are widely used for GLMMs, including the log, logit, probit, cauchit, and complementary log-log (cloglog) links. Conceptually, a binary event (e.g., obesity) may be dichotomized based on a continuous latent variable (e.g., body mass index); most links are associated with specific distributions assumed for the latent variable.²³ For example, a normally distributed latent variable yields the probit link; if the latent variable’s distribution likely has heavy tails, one may use the cauchit or cloglog link. Moreover, GLMMs with the log and logit links correspond to two-step methods with the log and logit transformations, respectively, but they directly model event counts without data transformations within studies.

In addition, the commonly-used overall proportion $\widehat{p}=g^{-1}\left(\widehat{\mu }\right)$ may be interpreted as the median proportion from multiple studies. Alternatively, the population-averaged proportion may be also of interest in epidemiological research.^14,24–29 It is defined as the marginal expectation of study-specific proportions, which is estimated as

$$\widehat{p}=\widehat{E}\left[p_i\right]=\widehat{E}\left[g^{-1}\left(\mu +\tau Z\right)\right]=\int g^{-1}\left(\widehat{\mu }+\widehat{\tau }z\right)\varphi \left(z\right)dz.$$

Here, Z is a standard normal random variable and ϕ(·) is its density function. For the probit link, $\widehat{p}=\Phi \left(\widehat{\mu }/\sqrt{1+{\widehat{\tau }}^2}\right)$, where Φ(·) is the standard normal cumulative distribution function.²⁴ However, it may not have closed forms for other links; we can numerically approximate it as $\widehat{p}\approx M^{-1}{\sum }_{m=1}^M{g}^{-1}\left(\widehat{\mu }+\widehat{\tau }{z}_m\right)$ based on a draw of many standard normal samples z_m (say, M=10,000). Its CI can be derived using the bootstrap resampling.³⁰ The population-averaged proportion can be similarly calculated for two-step methods.

Data analyses

We applied GLMMs and two-step methods to two real datasets with different sizes and proportion magnitudes. The first was from Woodd et al.,⁷ containing 21 studies on chorioamnionitis. The second was reported by Rotenstein et al.,⁵ which investigated the prevalence of depression or depressive symptoms among medical students. eAppendix C gives details of these datasets, including their forest plots.

All parameters were estimated via the maximum-likelihood (ML) approach. Bootstrap CIs were produced by 1000 resampling iterations. The ML estimation, instead of the restricted maximum-likelihood (REML), was used because GLMMs are usually implemented via the ML; computational difficulties may occur in the REML estimation for GLMMs. Nevertheless, the REML may be superior for two-step methods,³¹ so we also implemented two-step methods via the REML; see eAppendix D.

In addition, we conducted simulations in various settings, differing by the number of studies, overall proportions, heterogeneity extents, and study-specific sample sizes. Meta-analyses were simulated based on the GLMM with the logit link, because it is conventional to assume the logit proportions to follow the normal distribution. eAppendix E gives the detailed description.

Results

Table 1 presents the results of real data examples. GLMMs with various links produced similar results, while the cauchit link led to noticeably different results. Two-step methods with the arcsine-based transformations also produced similar results, but they were generally larger than those by other methods. GLMMs with the log and logit links led to slightly smaller estimates than two-step methods with the corresponding transformations. In addition, the population-averaged proportions were higher than the commonly-reported overall median proportions, because they incorporated the between-study variance.

Table 1.

Overall median proportion estimates and population-averaged proportion estimates with 95% confidence intervals in the two meta-analyses on chorioamnionitis and on depression or depressive symptoms.

Method	Overall median proportion, % [95% bootstrap CI] (95% CI)	Population-averaged proportion, % [95% bootstrap CI]
Meta-analysis on chorioamnionitis
Generalized linear mixed model:
Log link	3.27 [2.12, 4.97] (2.11, 5.05)	5.32 [3.06, 8.31]
Logit link	3.33 [2.14, 5.11] (2.13, 5.17)	5.17 [3.05, 7.77]
Probit link	3.54 [2.27, 5.53] (2.21, 5.48)	5.11 [3.07, 7.62]
Cauchit link	2.41 [1.69, 3.55] (1.80, 3.64)	11.34 [5.96, 16.10]
Complementary log-log link	3.30 [2.13, 5.05] (2.12, 5.11)	5.22 [3.06, 7.96]
Two-step method:
Log transformation	3.39 [2.20, 5.21] (2.21, 5.21)	5.41 [3.16, 8.38]
Logit transformation	3.45 [2.22, 5.31] (2.22, 5.32)	5.25 [3.14, 7.82]
Arcsine transformation	4.13 [2.56, 6.28] (2.49, 6.16)	5.16 [3.14, 7.55]
Double-arcsine transformation at
harmonic mean of sample sizes = 664	4.14 [2.58, 6.30] (2.50, 6.17)	5.15 [3.15, 7.54]
geometric mean of sample sizes = 17,358	4.21 [2.65, 6.36] (2.57, 6.24)	5.22 [3.21, 7.60]
arithmetic mean of sample sizes = 1,207,998	4.21 [2.66, 6.36] (2.57, 6.24)	5.22 [3.21, 7.60]
inverse of variance = 459	4.11 [2.55, 6.27] (2.47, 6.14)	5.13 [3.12, 7.51]
Meta-analysis on depression or depressive symptoms
Generalized linear mixed model:
Log link	20.12 [12.59, 35.06] (11.93, 33.94)	24.61 [13.04, 40.35]
Logit link	21.53 [12.65, 37.01] (11.71, 36.20)	24.71 [13.04, 38.42]
Probit link	22.15 [12.72, 37.34] (11.67, 36.60)	24.82 [13.03, 38.38]
Cauchit link	18.05 [11.93, 32.21] (11.75, 33.82)	23.95 [12.68, 37.33]
Complementary log-log link	20.82 [12.62, 36.17] (11.83, 35.12)	24.50 [13.04, 38.57]
Two-step method:
Log transformation	20.39 [12.76, 35.35] (12.14, 34.22)	24.84 [13.22, 40.61]
Logit transformation	21.70 [12.78, 37.14] (11.87, 36.30)	24.83 [13.18, 38.50]
Arcsine transformation	23.10 [12.82, 37.77] (11.58, 37.16)	25.02 [13.04, 38.51]
Double-arcsine transformation at
harmonic mean of sample sizes = 231	23.12 [12.85, 37.78] (11.59, 37.15)	25.01 [13.07, 38.51]
geometric mean of sample sizes = 266	23.14 [12.87, 37.79] (11.61, 37.16)	25.03 [13.09, 38.52]
arithmetic mean of sample sizes = 311	23.15 [12.89, 37.80] (11.63, 37.17)	25.04 [13.11, 38.52]
inverse of variance = 41	22.60 [12.13, 37.54] (10.85, 36.90)	24.55 [12.36, 38.29]

Table 2 gives the results of simulated meta-analyses with 30 studies. For relatively small sample sizes and rare events, GLMMs dramatically performed better than two-step methods, because within-study variances had large uncertainties in these cases, and GLMMs fully accounted for such uncertainties while two-step methods did not. As the heterogeneity extent increased, two-step methods may lead to substantial biases. GLMMs with various links had similar performance for relatively rare events; their differences became larger when events were common, especially for the cauchit link. When the number of studies was decreased to 5, eAppendix E gives the results with similar patterns, while the differences between GLMMs and two-step methods tended to be smaller.

Table 2.

Biases, root mean squared errors (RMSEs), and 95% confidence interval coverage probabilities (CPs, in percentage, %) of the estimated overall median proportions (in percentage, %) based on 1000 simulated meta-analyses with N=30 studies under various simulation settings.

	p=1%								p=20%
Method	τ=0.1				τ=0.5				τ=0.1				τ=0.5
	Bias	RMSE	CP		Bias	RMSE	CP		Bias	RMSE	CP		Bias	RMSE	CP
Sample sizes ranging in 100–500
Generalized linear mixed model:
Log	−0.01	0.11	96		0.01	0.16	93		0.01	0.53	94		−0.26	1.56	92
Logit	−0.01	0.11	96		0.01	0.16	93		0.01	0.53	94		0.07	1.59	92
Probit	−0.01	0.11	96		0.01	0.16	93		0.02	0.53	94		0.26	1.61	93
Cauchit	−0.01	0.11	96		0.02	0.15	90		−0.01	0.53	94		−0.99	1.86	87
Complementary log-log	−0.01	0.11	96		0.01	0.16	93		0.01	0.53	94		−0.09	1.56	92
Two-step method:
Log	0.31	0.33	22		0.43	0.46	15		0.19	0.55	93		0.01	1.52	93
Logit	0.30	0.32	24		0.43	0.46	15		0.10	0.52	93		0.23	1.57	93
Arcsine	0.09	0.14	90		0.17	0.23	81		−0.05	0.52	94		0.55	1.67	92
Double-arcsine (harmonic)	0.04	0.11	96		0.13	0.20	87		−0.07	0.52	93		0.55	1.67	92
Double-arcsine (geometric)	0.07	0.13	93		0.16	0.23	83		−0.05	0.52	93		0.57	1.67	92
Double-arcsine (arithmetic)	0.09	0.14	91		0.19	0.24	78		−0.04	0.52	94		0.59	1.68	92
Double-arcsine (IV)	0.24	0.27	41		0.34	0.37	32		0.06	0.52	94		0.64	1.69	92
Sample sizes ranging in 1000–5000
Generalized linear mixed model:
Log	0.00	0.04	92		0.01	0.09	96		0.00	0.34	93		−0.38	1.48	93
Logit	0.00	0.04	92		0.01	0.09	96		0.01	0.34	93		0.00	1.48	93
Probit	0.00	0.04	92		0.02	0.10	95		0.02	0.34	93		0.22	1.50	94
Cauchit	0.00	0.04	92		−0.06	0.11	89		−0.04	0.34	93		−1.42	2.06	85
Complementary log-log	0.00	0.04	92		0.01	0.09	96		0.00	0.34	93		−0.18	1.47	93
Two-step method:
Log	0.02	0.05	89		0.03	0.10	93		0.02	0.33	91		−0.34	1.48	93
Logit	0.02	0.05	89		0.03	0.10	93		0.02	0.33	91		0.02	1.49	94
Arcsine	0.00	0.04	92		0.05	0.11	93		0.03	0.33	92		0.55	1.60	93
Double-arcsine (harmonic)	−0.01	0.04	92		0.05	0.11	93		0.02	0.33	91		0.56	1.60	93
Double-arcsine (geometric)	−0.01	0.04	92		0.06	0.12	92		0.03	0.33	92		0.56	1.60	92
Double-arcsine (arithmetic)	0.00	0.04	92		0.06	0.12	92		0.03	0.33	92		0.56	1.60	92
Double-arcsine (IV)	0.01	0.04	91		0.07	0.12	90		0.04	0.33	91		0.53	1.59	93

Note: p denotes the true overall median proportion, and τ denotes the between-study standard deviation (of proportions on the logit scale). For the double-arcsine transformation used by the two-step method, the “harmonic,” “geometric,” and “arithmetic” denote the harmonic, geometric, and arithmetic means of sample sizes, and the “IV” denotes the inverse of variance.

Discussion

This article has illustrated that GLMMs for meta-analyses of proportions make less assumptions and generally have better performance than conventional two-step methods. They can be feasibly implemented using many software programs. We call for more applications of GLMMs for synthesizing proportions in epidemiological research.

GLMMs may have limitations from computational perspectives. First, they are more computationally demanding than two-step methods, especially when the bootstrap resampling is used. Nevertheless, as meta-analysis datasets are generally not large, their implementation is fast in most cases. Second, convergence issues may occur when estimating GLMM parameters. For example, during bootstrap resampling iterations for the first real data example, 2.9% and 23.6% of resampled datasets yielded convergence issues when implementing GLMMs with the probit and cauchit links, respectively.

We have focused on the univariate meta-analysis of proportions; GLMMs can be similarly applied to meta-regression, multivariate meta-analysis, etc.^32–34 For example, multivariate approaches are particularly important for synthesizing sensitivities and specificities of diagnostic tests, which are likely correlated. More applied work using GLMMs is also encouraged in these settings.