Estimating the sample mean and standard deviation from commonly reported quantiles in meta-analysis

Abstract

Researchers increasingly use meta-analysis to synthesize the results of several studies in order to estimate a common effect. When the outcome variable is continuous, standard meta-analytic approaches assume that the primary studies report the sample mean and standard deviation of the outcome. However, when the outcome is skewed, authors sometimes summarize the data by reporting the sample median and one or both of (i) the minimum and maximum values and (ii) the first and third quartiles, but do not report the mean or standard deviation. To include these studies in meta-analysis, several methods have been developed to estimate the sample mean and standard deviation from the reported summary data. A major limitation of these widely used methods is that they assume that the outcome distribution is normal, which is unlikely to be tenable for studies reporting medians. We propose two novel approaches to estimate the sample mean and standard deviation when data are suspected to be non-normal. Our simulation results and empirical assessments show that the proposed methods often perform better than the existing methods when applied to non-normal data.

Introduction

Meta-analysis is a statistical approach for pooling data from related studies that is widely used to provide evidence for medical research. To pool studies in an aggregate data meta-analysis, each study must contribute an effect measure (e.g., the sample mean for one-group studies, the sample means for two-group studies) of the outcome and its variance. However, primary studies may differ in the effect measures reported. Although the sample mean is the usual effect measure reported for continuous outcomes, authors often report the sample median when data are skewed and may not report the mean.¹ This occurs commonly for time-based outcomes, such as time delays in the diagnosis and treatment of tuberculosis^{2, 3} or colorectal cancer⁴ or length of hospital stay^5–7. Other examples in medical research include muscle strength and mass⁸, molecular concentration levels⁹, tumor sizes¹⁰, motor impairment scores¹¹, and intraoperative blood loss¹². When primary studies report the sample median of an outcome, they typically report the sample size and one or both of (i) the sample minimum and maximum values and (ii) the first and third quartiles.

The same effect measure must be obtained from all primary studies in an aggregate data meta-analysis. In order to meta-analyze a collection of studies in which some report the sample mean and others report the sample median, Hozo et al.¹³, Bland¹⁴, Wan et al.¹⁵, Kwon and Reis¹⁶, and Luo et al.¹⁷ have recently published methods to estimate the sample mean and standard deviation from studies that report medians. These methods have been widely used to meta-analyze the means for one-group studies and the raw or standardized difference of means for two-group studies. Reflecting how commonly these methods are used, Google Scholar listed 3,315 articles citing Hozo et al.¹³ and 866 articles citing Wan et al.¹⁵ as of October 23, 2019.

Commonly used methods that have been proposed to estimate the sample mean and standard deviation in this context can be divided into formula-based methods and simulation-based methods. The methods developed by Luo et al.¹⁷ and Wan et al.¹⁵ are the best-performing formula-based methods for estimating the sample mean and standard deviation, respectively. A major limitation of these methods is that they assume the outcome variable is normally distributed, which may be unlikely because otherwise the authors would have reported the mean. Consequently, Kwon and Reis¹⁶ recently proposed a simulation-based method which is based on different parametric assumptions of the outcome variable. Although the Kwon and Reis¹⁶ sample mean estimator has not been compared to the formula-based method of Luo et al.¹⁷, their proposed standard deviation estimator performed better than the formula-based method of Wan et al.¹⁵ for skewed data when the assumed parametric family is correct. Limitations of this simulation-based method are that (i) it is computationally expensive, (ii) requires users to write their own distribution-specific code, and (iii) its performance can be highly sensitive to several conceptual and computational decisions that one must make when implementing the method (see Discussion).

We propose two novel methods to estimate the sample mean and standard deviation for skewed data when the underlying distribution is unknown. The proposed methods overcome several limitations of the existing methods, and we demonstrate that the proposed approaches often perform better than the existing methods when applied to skewed data.

The objectives of this paper are to describe the existing and proposed methods for estimating the sample mean and standard deviation, systematically evaluate their performance in a simulation study, and empirically evaluate their performance on real-life data sets.

In the following section, we describe the existing and proposed methods. In ‘Results’, we report the results of a simulation investigating the performance of the methods. We illustrate these methods on an example data set and evaluate their accuracy in ‘Example’. In ‘Discussion’, we summarize our findings and provide recommendations for data analysts.

Methods

Throughout this paper, we use the following notation for sample summary statistics: minimum value (Q_min), first quartile (Q₁), median (Q₂), third quartile (Q₃), maximum value (Q_max), mean ($\overline{x}$), standard deviation (s), and sample size (n). Let $\widehat{x}$ and $\widehat{s}$ denote estimates of the sample mean and standard deviation, respectively. As investigated in previous studies^13–17, we consider the following sets of summary statistics that may be reported by a study, denoted by Scenario 1 (S₁), Scenario 2 (S₂), and Scenario 3 (S₃):

$$\begin{gathered} S_1=\left\{Q_{\text{min}},Q_2,Q_{\text{max}},n\right\} \\ {S}_2=\left\{Q_1,Q_2,Q_3,n\right\} \\ {S}_3=\left\{Q_{\text{min}},Q_1,Q_2,Q_3,Q_{\text{max}},n\right\}. \end{gathered}$$

Comparator Methods

The sample mean estimator of Luo et al.¹⁷ and the sample standard deviation estimator of Wan et al.¹⁵ are formula-based methods that are derived from the assumption that the outcome variable is normally distributed.

Luo et al. developed the following sample mean estimators in scenarios S₁, S₂, and S₃:

$$\widehat{x}=\left(\frac{4}{4+n^{0.75}}\right)\frac{Q_{\text{min}}+Q_{\text{max}}}{2}+\left(\frac{n^{0.75}}{4+n^{0.75}}\right)Q_2\text{in}{S}_1$$

$$\widehat{x}=\left(0.7+\frac{0.39}{n}\right)\frac{Q_1+Q_3}{2}+\left(0.3-\frac{0.39}{n}\right)Q_2\text{in}{S}_2$$

$$\widehat{x}=\left(\frac{2.2}{2.2+n^{0.75}}\right)\frac{Q_{\text{min}}+Q_{\text{max}}}{2}+\left(0.7-\frac{0.72}{n^{0.55}}\right)\frac{Q_1+Q_3}{2}+\left(0.3+\frac{0.72}{n^{0.55}}-\frac{2.2}{2.2+n^{0.75}}\right)Q_2\text{in}{S}_3$$

Building on the sample mean estimators of Hozo et al.¹³, Wan et al.¹⁵, and Bland¹⁴ in S₁, S₂, and S₃, respectively, this method optimally weights the median (in S₁, S₂, and S₃), the average of the minimum and maximum values (in S₁ and S₃), and the average of the first and third quartiles (in S₂ and S₃). The weights are set to minimize the mean squared error of the estimator. Numerical simulations have demonstrated that the method of Luo et al. has considerably lower relative mean squared error (RMSE) compared to the method of Bland in S₃ and has comparable RMSE to the method Wan et al. in S₂ under normal and skewed distributions.

Wan et al. proposed the following sample standard deviation estimators in scenarios S₁, S₂, and S₃:

$$\widehat{s}=\frac{Q_{\text{max}}-Q_{\text{min}}}{2{\Phi }^{-1}\left(\frac{n-0.375}{n+0.25}\right)}\text{in}{S}_1$$

$$\widehat{s}=\frac{Q_3-Q_1}{2{\Phi }^{-1}\left(\frac{0.75n-0.125}{n+0.25}\right)}\text{in}{S}_2$$

$$\widehat{s}=\frac{Q_{\text{max}}-Q_{\text{min}}}{4{\Phi }^{-1}\left(\frac{n-0.375}{n+0.25}\right)}+\frac{Q_3-Q_1}{4{\Phi }^{-1}\left(\frac{0.75n-0.125}{n+0.25}\right)}\text{in}{S}_3$$

The standard deviation estimators of Wan et al. are derived using relationships between the distribution standard deviation and the expected values of order statistics for normally distributed data. The expected values of the minimum and maximum values and first and third quartiles are estimated by the respective sample values. The expected value of other order statistics are estimated using Blom’s method¹⁸.

Wan et al. were the first to propose a standard deviation estimator in S₂. Wan et al. showed that their estimator in S₁ and S₃ outperformed the previously developed sample standard deviation estimators of Hozo et al.¹³ and Bland¹⁴, respectively, in regards to average relative error.

For the purpose of the analyses presented herein, we refer to the approach which uses the method of Luo et al. to estimate the sample mean and the method of Wan et al. to estimate the sample standard deviation as the Luo/Wan method.

Proposed Methods

The following two subsections describe the proposed methods for estimating the sample mean and standard deviation from S₁, S₂, andS₃ summary measures. The R package ‘estmeansd’ available on CRAN implements both of the proposed methods.¹⁹ Additionally, the webpage https://smcgrath.shinyapps.io/estmeansd/ provides a graphical user interface for using these methods. Although the first method we introduce was adapted from previous work in McGrath et al.²⁰, no approaches in McGrath et al.²¹ could be adapted to estimate the sample mean or standard deviation in this context.

Quantile Estimation (QE) Method

The QE method was originally introduced in McGrath et al.²⁰ for estimating the variance of the median when summary measures of S₁, S₂, or S₃ are provided. Here, we describe how the QE method can be applied to estimate the sample mean and standard deviation in these contexts.

We pre-specify several candidate parametric families of distributions for the outcome variable, namely the normal, log-normal, gamma, beta, and Weibull. The parameters of each candidate distribution are estimated by minimizing the distance between the observed and distribution quantiles. Let $F_{\theta }^{-1}$ denote the quantile function of a given candidate distribution parameterized by θ. Then, the objective function corresponding to the distribution, denoted by S(θ), is given by

$$S\left(\theta \right)={\left(F_{\theta }^{-1}\left(1/n\right)-Q_{\text{min}}\right)}^2+{\left(F_{\theta }^{-1}\left(0.5\right)-Q_2\right)}^2+{\left(F_{\theta }^{-1}\left(1-1/n\right)-Q_{\text{max}}\right)}^2\text{in}{S}_1$$

$$S\left(\theta \right)={\left(F_{\theta }^{-1}\left(0.25\right)-Q_1\right)}^2+{\left(F_{\theta }^{-1}\left(0.5\right)-Q_2\right)}^2+{\left(F_{\theta }^{-1}\left(0.75\right)-Q_3\right)}^2\text{in}{S}_2$$

$$S\left(\theta \right)={\left(F_{\theta }^{-1}\left(1/n\right)-Q_{\text{min}}\right)}^2+{\left(F_{\theta }^{-1}\left(0.25\right)-Q_1\right)}^2+{\left(F_{\theta }^{-1}\left(0.5\right)-Q_2\right)}^2+{\left(F_{\theta }^{-1}\left(0.75\right)-Q_3\right)}^2+{\left(F_{\theta }^{-1}\left(1-1/n\right)-Q_{\text{max}}\right)}^2\text{in}{S}_3$$

Details concerning the implementation of the optimization algorithm for minimizing S(θ) are provided in Appendix A.

The distribution with the best fit (i.e., yielding the smallest value of $S\left(\widehat{\theta }\right)$ where $\widehat{\theta }$ denotes the estimated parameters of the given distribution) is assumed to be the underlying distribution of the sample. The sample mean and standard deviation are estimated by the mean and standard deviation of the selected distribution.

Box-Cox (BC) Method

Luo et al.¹⁷ and Wan et al.¹⁵ assumed that a sample x of interest follows a normal distribution. To make this assumption more tenable for skewed data, we incorporate Box-Cox transformations into the methods of Luo et al. and Wan et al. The proposed method, which we denote by BC, applies Box-Cox transformations to the quantiles of x and assumes that the underlying distribution of the transformed data is normal.

In brief, the BC method consists of the following four steps. First, an optimization algorithm, such as the algorithm of Brent²², optimizes the power parameter λ such that distribution of the transformed data is most likely to be normal. Letting f_λ denote the Box-Cox transformation, the quantiles of x are transformed into the quantiles of f_λ(x). Afterwards, the methods of Luo et al. and Wan et al. are applied to estimate the mean and standard deviation of f_λ(x), respectively. Finally, the mean and standard deviation of f_λ(x) are inverse-transformed into the mean and standard deviation of x.

Box-Cox transformations f_λ are defined as follows:

$$f_{\lambda }\left(x_i\right)=y_i=\{\begin{array}{c}\frac{x_i^{\lambda }-1}{\lambda }\text{if}\lambda \ne 0\\ \text{ln}\left(x_i\right)\text{if}\lambda =0\end{array}$$

Equivalently, inverse Box-Cox transformations $f_{\lambda }^{-1}$ are defined as follows:

$$f_{\lambda }^{-1}\left(y_i\right)=x_i=\{\begin{array}{c}{\left(\lambda \cdot y_i+1\right)}^{1/\lambda }\text{if}\lambda \ne 0\\ \text{exp}\left(y_i\right)\text{if}\lambda =0\end{array}$$

Box and Cox²³ argued that Box-Cox transformations can transform a dataset into a more normally-distributed dataset. Moreover, for every value of λ, f_λ is monotonically increasing. Therefore, any i^th order statistic of an untransformed dataset, after transformation, is still the i^th order statistic of the corresponding transformed dataset, and vice versa.

The optimization step for finding λ can be described as follows. In S₁ and S₂, λ is chosen so that the transformed minimum and maximum values (in S₁) or first and third quartiles (in S₂) are equidistant from the median, making the transformed data to be most likely symmetric and therefore most normally distributed. Specifically, the BC method finds a finite value of λ such that

$$f_{\lambda }\left(Q_{\text{max}}\right)-f_{\lambda }\left(Q_2\right)=f_{\lambda }\left(Q_2\right)-f_{\lambda }\left(Q_{\text{min}}\right)$$

in S₁ and

$$f_{\lambda }\left(Q_3\right)-f_{\lambda }\left(Q_2\right)=f_{\lambda }\left(Q_2\right)-f_{\lambda }\left(Q_1\right)$$

in S₂. In S₃, a value of λ cannot necessarily be found such that both the first and third quartiles as well as the minimum and maximum values are equidistant from the median. Therefore, λ is found by

$$\underset{\lambda }{\text{argmin}}\left[{\left(f_{\lambda }\left(Q_3\right)-f_{\lambda }\left(Q_2\right)-\left(f_{\lambda }\left(Q_2\right)-f_{\lambda }\left(Q_1\right)\right)\right)}^2+{\left(f_{\lambda }\left(Q_{\text{max}}\right)-f_{\lambda }\left(Q_2\right)-\left(f_{\lambda }\left(Q_2\right)-f_{\lambda }\left(Q_{\text{min}}\right)\right)\right)}^2\right]$$

Appendix B describes the implementation of the optimization algorithm used to find λ.

Then, the BC method applies the Box-Cox transformations with this value of λ on the quantiles of x. That is, the BC method transforms {Q_min,Q₂,Q_max} into {f_λ(Q_min),f_λ(Q₂),f_λ(Q_max)} in S₁, {Q₁,Q₂,Q₃}into {f_λ(Q₁),f_λ(Q₂),f_λ(Q₃)} in S₂, and {Q_min,Q₁,Q₂,Q₃,Q_max} into {f_λ(Q_min),f_λ(Q₁),f_λ(Q₂),f_λ(Q₃),f_λ(Q_max)}in S₃.

Let N′(μ,σ²) ~ N(μ,σ²) conditional on N′(μ,σ²) ∈ [f(0),2μ − f(0)]. Equivalently, N′(μ,σ²) is the symmetrically truncated N(μ,σ²) bounded within the support [f(0),2μ − f(0)]. Then, the BC method assumes that f_λ(x) ~N′(μ,σ²) for some μ and σ and uses the methods of Luo et al. and Wan et al. to calculate μ and σ, respectively. Finally, the assumption made by the BC method implies that $x~f_{\lambda }^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)$. Therefore, the mean and standard deviation of $f_{\lambda }^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)$ are approximately $\overline{x}$ and s.

The mean and standard deviation of $f_{\lambda }^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)$ are found as follows. Let ϕ and Φ be the probability density function and cumulative distribution function of the standard normal distribution, respectively. The following two equations describe the mean and variance of $f_{\lambda }^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)$, respectively:

$$\text{E}\left[f_{\lambda }^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)\right]=\underset{x=f_{\lambda }\left(0\right)}{\overset{x=2\mu -f_{\lambda }\left(0\right)}{\int }}\varphi \left(\frac{x-\mu }{\sigma }\right)\frac{f_{\lambda }^{-1}\left(x\right)}{\sigma \left(\Phi \left(\mu \right)-\Phi \left(-\mu \right)\right)}\partial x$$ (1)

$$\text{Var}\left[f_{\lambda }^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)\right]=\underset{x=f_{\lambda }\left(0\right)}{\overset{x=2\mu -f_{\lambda }\left(0\right)}{\int }}\varphi \left(\frac{x-\mu }{\sigma }\right)\frac{{\left(f_{\lambda }^{-1}\left(x\right)-\text{E}\left[f_{\lambda }^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)\right]\right)}^2}{\sigma \left(\Phi \left(\mu \right)-\Phi \left(-\mu \right)\right)}\partial x$$ (2)

Numerical integration can solve the two above equations. Moreover, the following Monte-Carlo simulation can compute the mean and standard deviation of $f_{\lambda }^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)$: first, generate an independent and identically distributed random sample R from N(μ,σ²); next, let the new R be {r ∈ R: r ∈ [f(0),2μ − f(0)]}, or equivalently, remove any value in R that is not within the range[f(0),2μ − f(0)]; then, calculate the sample mean and sample standard deviation of R; finally, the sample mean and sample standard deviation are estimated as the mean and standard deviation of $f_{\lambda }^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)$. The application of the BC method in this work uses Monte-Carlo simulation to compute the mean and standard deviation of $f_{\lambda }^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)$.

Recall that N′(μ,σ²) is the symmetrically truncated N(μ,σ²) with support [f(0),2μ − f(0)]. In fact, $N^{\prime }\left(\mu ,{\sigma }^2\right)~f_{\lambda =1}^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)$, and $LN\left(\mu ,{\sigma }^2\right)~f_{\lambda =0}^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)$. Therefore, both the normal distribution truncated within the support [f(0),2μ − f(0)] and log-normal distribution are special cases of $f_{\lambda }^{-1}\left(N^{\prime }\left(\mu ,{\sigma }^2\right)\right)$.

Design of Simulation Study

We conducted a simulation study to systematically compare the performance of the existing and proposed approaches when the truth is known.

To be consistent with the work already conducted in this area, we generated data from the same distributions considered in previous studies^13–17. As used by Bland¹⁴, we used the normal distribution with μ = 5 and σ = 1, the log-normal distribution with μ = 5 and σ = 0.25, the log-normal distribution with μ = 5 and σ = 0.5, and the log-normal distribution μ = 5 and σ = 1 in our primary analyses to investigate the effect of skewness on the performance of the sample mean and standard deviation estimators. In sensitivity analyses, we considered the following distributions used in several other studies^{13, 15–17}: the normal distribution with μ = 50 and σ = 17, the log-normal distribution with μ = 4 and σ = 0.3, the exponential distribution with λ = 10, the beta distribution with α = 9 and β = 4, and the Weibull distribution with λ = 2 and k = 35.

For each distribution, a sample of size n was drawn to simulate data from a primary study. Then, the appropriate summary statistics (i.e., S₁, S₂, or S₃) were calculated from this sample. The Luo/Wan, QE, and BC methods were each applied to the summary data in order to estimate the sample mean and standard deviation.

We used the following sample sizes in our simulations: 25, 50, 75, 100, 150, 200, 250, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, 950, 1 000. A total of 1 000 repetitions were performed for each combination of data generation parameters under scenarios S₁, S₂, and S₃.

As used in previous studies^{13, 15, 16}, the average relative error (ARE) was used as a performance measure. For repetition i(i = 1, …, 1 000), let ${\widehat{x}}_i$ and ${\widehat{s}}_i$ denote estimates of the sample mean and standard deviation, respectively, and let ${\overline{x}}_i$ and s_i denote the true sample mean and standard deviation, respectively. The ARE of the sample mean and standard deviation estimators is defined by

$$\text{ARE}\left(\widehat{x}\right)=\frac{1}{1000}{\sum }_{i=1}^{1000}\frac{{\widehat{x}}_i-{\overline{x}}_i}{{\overline{x}}_i},\text{\hspace{1em}\hspace{1em}ARE}\left(\widehat{s}\right)=\frac{1}{1000}{\sum }_{i=1}^{1000}\frac{{\widehat{s}}_i-s_i}{s_i}.$$

As used in Luo et al.¹⁷, we also used the relative mean squared error (RMSE) to evaluate the performance of all methods. Letting μ denote the true distribution mean and σ denote the true distribution standard deviation, the RMSE of the sample mean and standard deviation estimators is given by

$$\text{RMSE}\left(\widehat{x}\right)=\frac{\frac{1}{1000}{\sum }_{i=1}^{1000}{\left({\widehat{x}}_i-\mu \right)}^2}{\frac{1}{1000}{\sum }_{i=1}^{1000}{\left({\overline{x}}_i-\mu \right)}^2},\text{\hspace{1em}\hspace{1em}RMSE}\left(\widehat{s}\right)=\frac{\frac{1}{1000}{\sum }_{i=1}^{1000}{\left({\widehat{s}}_i-\sigma \right)}^2}{\frac{1}{1000}{\sum }_{i=1}^{1000}{\left(s_i-\sigma \right)}^2}.$$

Results of Simulation Study

In the following subsections, we present the results of the simulation study using the set of outcome distributions considered by Bland¹⁴, as these distributions were selected to investigate the effect of skewness on the estimators. The results of the sensitivity analyses where we used the set of outcome distribution used by other authors^{13, 15–17} is given in Section 1 of Supplementary Material.

Because the simulation results in scenarios S₁ and S₃ were similar, the S₃ simulation results are presented in Section 2 of Supplementary Material for parsimony. Additionally, as the focus of this paper is on the analysis of non-normal data, all simulation results where data were generated from a normal distribution are presented in Section 3 of Supplementary Material. We placed the simulation results when using RMSE as the performance measure in Section 4 of the Supplementary Material, as similar trends were observed when using ARE.

Comparison of Methods Under Scenario S₁

Figure 1 displays the ARE of all sample mean and standard deviation estimators under scenario S₁. As the skewness (i.e., the σ parameter) of the log-normal distribution increased, the magnitude of the AREs generally increased for the sample mean and standard deviation estimators, but was inconsequential for the BC method. Moreover, all methods had considerably larger AREs for estimating the sample standard deviation compared to estimating the sample mean.

Figure 1:

ARE of the Luo/Wan (red line, hollow circle), QE (blue line, solid triangle), and BC (green line, solid circle) methods in scenario S₁. The panels in the left and right columns present the ARE of the sample mean estimators and sample standard deviation estimators, respectively.

For estimating the sample mean, the BC method performed best under each distribution and nearly all sample sizes (n) considered in Figure 1; the BC method was nearly unbiased, yielding AREs of magnitude less than 0.004, 0.008, and 0.020 in the Log-Normal(5,0.25), Log-Normal(5,0.5), and Log-Normal(5,1), cases, respectively. Contrary to the Luo et al. sample mean estimator which became more biased as n increased (e.g., ARE = −0.22 for Luo et al. when n = 1 000 in Log-Normal(5,1)), the performance of the QE sample mean estimator improved as n increased. The QE sample mean estimator became preferred over the Luo et al. sample mean estimator when n ≥ 300. However, the QE method always performed worse than the BC method in regards to ARE in Figure 1.

The BC method performed best for estimating the sample standard deviation, achieving AREs of magnitude less than 0.03 in nearly all scenarios investigated in Figure 1. Although the QE standard deviation estimator performed better as n increased, this method typically resulted in larger AREs compared to the BC method. Additionally, the QE standard deviation estimator yielded large ARE values when sample sizes were small (i.e., n ≤ 50), especially for skewed outcomes.

Model selection for the QE method generally performed well. When the outcome distribution was Log-Normal(5,0.25), the QE method selected the log-normal distribution between 58.1% (when n = 25) to 82.3% (when n = 1 000) of repetitions. Moreover, the QE method had comparable performance in the repetitions where it did not select the log-normal distribution (e.g., AREs ranging between −0.01 and 0.01 for estimating the sample mean and between 0.07 and 0.11 for estimating the standard deviation in these repetitions). Model selection improved for the QE method as n and the skewness of the log-normal distribution increased. For example, in the Log-Normal(5,1) case, the QE method selected the log-normal distribution in at least 99% of the repetitions for all n ≥ 50.

Comparison of Methods Under Scenario S₂

Figure 2 gives the ARE of all methods under scenario S₂. As in scenario S₁, we found that (i) the skewness of the underlying distribution strongly affected the performance of the sample mean and standard deviation estimators, and (ii) the sample mean estimators typically had AREs with smaller magnitude.

Figure 2:

ARE of the Luo/Wan (red line, hollow circle), QE (blue line, solid triangle), and BC (green line, solid circle) methods in scenario S₂. The panels in the left and right columns present the ARE of the sample mean estimators and sample standard deviation estimators, respectively.

The BC and QE sample mean estimators performed substantially better than the Luo et al. sample mean estimator in all scenarios investigated in Figure 2. As the skewness of the log-normal distribution increased, the gap in performance between the Luo et al. sample mean estimator and the BC and QE sample mean estimators increased. For instance, when the outcome distribution was Log-Normal(5,1), the ARE of the Luo et al. sample mean estimator was approximately −0.29 for most values of n whereas the QE sample mean estimator had AREs of magnitude less than 0.005 for most n. Although the QE and BC methods performed comparably for the Log-Normal(5,0.25) distribution, the QE sample mean estimator became preferred over the BC method as the skewness increased.

Similar trends held for the corresponding sample standard deviation estimators. The QE and BC methods performed considerably better than the Wan et al. sample standard deviation estimator in nearly all scenarios in Figure 2. There were no clear trends concerning the relative performance between the QE and BC standard deviation estimators.

Lastly, model selection performance was similar to that observed in S₁. In the Log-Normal(5,0.25) case, the QE method selected the log-normal distribution in the majority of repetitions under all values of n. The performance of the QE method slightly worsened in repetitions where the log-normal solution was not selected (e.g., AREs ranging between −0.02 to −0.01 for estimating the sample mean and between −0.08 and −0.03 for estimating the sample standard deviation in these repetitions) As n and the skewness of the underlying log-normal distribution increased, the log-normal distribution was increasingly selected by the QE method. For instance, in the Log-Normal(5,1) case, the QE method selected the log-normal distribution in at least 90% of the repetitions for all n ≥ 250.

Example

In this section, we illustrate the use of the existing and proposed methods when applied to a real-life meta-analysis of a continuous, skewed outcome. Specifically, we used data collected for an individual participant data (IPD) meta-analysis of the diagnostic accuracy of the Patient Health Questionnaire-9 (PHQ-9) depression screening tool.^{24, 25} We chose to use data from an IPD meta-analysis because 1) S₁, S₂, and S₃ summary data can be obtained from each study and 2) the true study-specific sample means and standard deviations are available.

Our analysis focused on the patient scores of the PHQ-9, which is a self-administered screening tool for depression. PHQ-9 scores are measured on a scale from 0 to 27, where higher scores are indicative of higher depressive symptoms. Previous studies have found that the distribution of PHQ-9 scores in the general population is right-skewed^26–28.

For each of the 58 primary studies, we calculated the sample median, minimum and maximum values, and first and third quartiles of the PHQ-9 scores of all patients in order to mimic the scenarios where an aggregate data meta-analysis extracts S₁, S₂, or S₃ summary data. Then, we applied the existing and proposed methods to this summary data to estimate study-specific sample means and standard deviations – we refer to these as the “derived estimated sample means and standard deviations”. Section 5 of Supplementary Material presents the study-specific S₃ summary data.

Some primary studies used weighted sampling. When extracting S₁, S₂, and S₃ summary data from these studies, weighted sample quantiles were used.²⁹ Additionally, weighted sample means and standard deviations were used as the true values for the sample mean and standard deviation, respectively, for studies with weighted sampling.

As PHQ-9 scores are integer-valued, PHQ-9 scores of 0 were observed in most of the primary studies. However, a minimum value and/or first quartile value of 0 result in complications for the QE method when estimating the parameters of the log-normal distribution, as the parameter constraints for the QE method implicitly assume that the extracted summary data are strictly positive. Therefore, when applying all methods, a value of 0.5 was added to the extracted summary data. After estimating the sample mean and standard deviation from the shifted summary data, 0.5 was subtracted from the estimated sample mean.

We compared the derived estimated sample means and standard deviations to the true sample means and standard deviations (Table 1). The QE and BC methods were considerably less biased than the Luo et al. method for estimating the sample mean under S₁, S₂, and S₃. The QE sample mean estimator performed best under S₁ and the BC sample mean estimator performed best under S₂ and S₃. Trends were less conclusive for estimating the standard deviation. The QE method standard deviation estimator was the least biased under S₁ and S₃ and the standard deviation estimator of Wan et al. was the least biased under S₂.

Table 1:

ARE of the methods when applied to estimate the sample means and standard deviations of the 58 primary studies. In each column, the ARE value closest to zero is in bold. The presented ARE values were rounded to two decimal places.

	ARE for $\widehat{x}$			ARE for $\widehat{s}$
	S 1	S 2	S 3	S 1	S 2	S 3
Luo/Wan	−0.14	−0.15	−0.10	−0.15	−0.01	−0.08
QE	−0.05	0.06	0.00	−0.15	0.34	−0.08
BC	−0.08	0.00	0.00	−0.25	0.06	0.11

We meta-analyzed the PHQ-9 scores using the true study-specific sample means and standard deviations (Figure 3) and compared this to a meta-analysis using the derived estimated study-specific sample means and standard deviations (Table 2). The restricted maximum likelihood method was used to estimate heterogeneity in all meta-analyses.³⁰ The QE and BC methods were less biased for estimating the pooled mean compared to the existing methods in S₁, S₂, and S₃. The QE method had relative error closest to zero for estimating the pooled mean in S₁ and S₃ and the BC method had relative error closest to zero in S₂. As one may expect, QE and BC methods performed best in S₃ for estimating the pooled mean, yielding relative errors of −0.0054 and 0.0074, respectively.

Figure 3:

Forest plot from the meta-analysis of mean PHQ-9 scores. The study-specific estimates represent the true sample means and their 95% CIs. The pooled estimate shown was obtained using the true-study-specific sample means and standard deviations. In the “Mean PHQ-9” column, the true study-specific sample means and their 95% CIs as well as the pooled mean and its 95% CI are given.

Table 2:

Estimates of the pooled mean PHQ-9 score and their 95% CIs when using the study-specific derived estimated sample means and standard deviations. For the pooled estimates under the “S₁”, “S₂”, and “S₃” columns, all methods were applied assuming S₁, S₂, and S₃ summary data, respectively, were extracted from all 58 primary studies, and the derived estimated study-specific sample means were meta-analyzed. When using the true study-specific sample means and standard deviations, the pooled estimate was 6.53 [95% CI: 5.97, 7.09]. In each column, the pooled estimate closest to the true value (i.e., 6.53) is in bold.

	S 1	S 2	S 3
Luo/Wan	5.76 [5.15, 6.37]	5.68 [5.06, 6.29]	5.97 [5.36, 6.58]
QE	6.26 [5.67, 6.85]	6.88 [6.22, 7.53]	6.49 [5.92, 7.07]
BC	6.09 [5.48, 6.69]	6.59 [5.91, 7.28]	6.58 [6.01, 7.14]

The primary studies were highly heterogeneous. When using the true study-specific sample means and standard deviations, the I² = 98.15%.³¹ The Luo/Wan, QE, and BC methods yielded similar estimates of I²; using 98.15% as the true value of I², all three methods had relative errors between −0.02 and 0.02 for estimating I² in S₁, S₂, and S₃.

Lastly, we investigated the skewness of the PHQ-9 scores. To mimic how data analysts may evaluate skewness based on available summary data, we used Bowley’s coefficient to quantify skewness, as it only depends on S₂ summary data.³² Bowley’s coefficient values range from −1 to 1, where positive values indicate right skew and negative values indicate left skew. The average value of Bowley’s coefficient taken over all 58 primary studies was 0.18, indicating moderate right skewness. Moreover, the QE method suggested non-normality in many of the primary studies. When given S₂ data, the QE method selected the normal distribution for 21% of studies, the log-normal for 22% of studies, the gamma for 26% of studies, and the Weibull for 31% of studies.

We performed additional analyses to explore the sensitivity of the addition of 0.5 to all summary data. When adding 0.1 or 0.01 to all summary data, all methods obtained similar results.

Discussion

We proposed two methods to estimate the sample mean and standard deviation from commonly reported quantiles in meta-analysis. Because studies typically report the sample median and other sample quantiles when data are skewed, our analyses focused on the application of the proposed QE and BC methods to skewed data. We compared the QE and BC methods to the widely used methods of Wan et al.¹⁵ and Luo et al.¹⁷ in a simulation study and in a real-life meta-analysis.

We found that the QE and BC sample mean estimators performed well, typically yielding average relative error values approaching zero as the sample size increased. In the simulation study and the empirical evaluation, the QE and BC sample mean estimators performed better than the methods of Luo et al. in nearly all scenarios.

Although the BC sample standard deviation estimator performed best or comparably to the best performing method in the primary analyses of the simulation study, the sensitivity analyses and empirical evaluations did not clearly indicate a best performing approach for estimating the sample standard deviation. For all methods, the magnitude of the relative errors for estimating the sample standard deviation was typically higher than for estimating the sample mean.

In practice, the existing and proposed methods enable data analysts to incorporate studies that report medians in meta-analysis. Therefore, we compared the performance of the methods at the meta-analysis level using data from a real-life individual patient data meta-analysis. In this analysis, the methods that performed best for estimating the sample mean often resulted in the most accurate pooled mean estimates as well. As the QE and BC methods performed best for estimating the sample mean, these methods also performed best at the meta-analysis level.

In our empirical assessments, we assumed that all primary studies reported S₁, S₂, or S₃ summary data. Often in aggregate data meta-analyses, however, only a fraction of primary studies report S₁, S₂, or S₃ summary data and the other primary studies report sample means and standard deviations. Therefore, the results of our analyses at the meta-analysis level reflect the extremes in performance between the existing and proposed sample mean and standard deviation estimators. In practice, in meta-analyses where all or nearly all primary studies report medians, directly meta-analyzing medians may be better suited.^{20, 21}

Repeated applications of the BC method to the same summary data will result in slightly different estimates of the sample mean and standard deviation. This is because the BC method uses Monte-Carlo simulation to perform the inverse transformation (i.e., to solve equations (1) and (2)). We considered using deterministic numerical integration methods to perform the inverse transformation. However, we found that they often failed to converge when the transformation parameter λ was close to zero or negative (i.e., λ ≤ 0.01). Therefore, we opted for Monte-Carlo simulation for this step.

Our analyses focused on skewed data. As expected, when data were generated from a normal distribution, the Luo et al. sample mean estimators and the Wan et al. sample standard deviation estimators performed best (see Section 3 of Supplementary Material). However, most methods performed reasonably well in the normal case and the differences in performance amongst the methods were often inconsequential (e.g., AREs of magnitude less than 0.01 for the Luo et al., QE, and BC sample mean estimators in the Normal(5,1) case). When making the same assumption of normality when applying the QE method (i.e., by only fitting the normal distribution), the performance of the method improved but were still not superior to the Luo et al. and Wan et al. methods (data not shown).

Kwon and Reis^{16, 33} proposed methods for estimating the sample mean and standard deviation from the same sets of summary data considered in this work that are based on applying approximate Bayesian computation (ABC). Unlike the methods of Luo et al. and Wan et al. which assume that the outcome variable is normally distributed, the ABC methods can be applied under different parametric assumptions of the underlying distribution (i.e., normal and skewed distributions). We considered including the ABC methods in this paper. However, we found that several implementation decisions strongly affected the performance of the method in the simulation study and empirical assessments. As investigating how to best implement the ABC methods would be beyond the scope of this paper, we decided not to include these methods in this paper and intend to study this in greater detail in future work.

This work has several limitations. Although the settings in our simulation study were based on those used in previous studies^13–17 to make a fair comparison between methods, these settings are not exhaustive and results may vary in other settings. Additionally, our simulation study focused solely on the performance of the methods for estimating the sample mean and standard deviation. In future work, we intend to conduct a simulation study investigating the performance of the methods at the meta-analysis level (e.g., for estimating the pooled effect measure and heterogeneity).

Strengths of this work include (i) including a greater number of outcome distributions and performance measures compared to the simulation studies conducted by previous authors^{13–15, 17}, and (ii) empirically evaluating the accuracy of the methods using real-life data.

In summary, we recommend the QE and BC methods for estimating the sample mean and standard deviation when data are suspected to be non-normal, as they often outperformed the existing methods in the analyses presented herein. To make these methods widely accessible, we developed the R package ‘estmeansd’ (available on CRAN)¹⁹ which implements these methods and launched a webpage (available at https://smcgrath.shinyapps.io/estmeansd/) that provides a graphical user interface for using these methods. We also encourage researchers performing meta-analysis to explore the sensitivity of their conclusions to the choice of method for estimating sample means and standard deviations.

Appendix A

In the QE method, the parameters of a candidate distribution are estimated by minimizing the objective function, S(θ). This section describes the implementation of minimization algorithm.

We set the initial values for the parameters in the optimization algorithm as follows. First, we apply the methods of Luo et al.¹⁷ and Wan et al.¹⁵ to estimate the sample mean and standard deviation, respectively, from S₁, S₂, or S₃. Then, we apply the method of moments estimator of the candidate distribution using the estimated sample mean and standard deviation. The method of moments estimates of the parameters are used as the initial values of the parameters.

To minimize S(θ), we apply the limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm with box constraints (L-BFGS-B), which is implemented in the built-in ‘optim’ function in the statistical programming language R. Reasonable constraints for the parameters are imposed to improve the convergence of the algorithm (e.g., enforcing μ ∈ [Q_min,Q_max] for the Normal(μ,σ²) distribution in S₁). The particular constraints are given in Table A1. These parameter constraints are based on the uniform prior bounds in the ABC method of Kwon and Reis¹⁶. In the simulation study, we found that the solution to the minimization problem was insensitive to perturbations of the parameter constraint values, provided the algorithm converged.

The algorithm is considered to converge when the objective function is reduced by a factor of less than 10⁷ of machine tolerance. In each application of the QE method in the simulation study, the algorithm converged for at least three distributions. If the algorithm failed to converge for a given candidate distribution, that candidate distribution was excluded from the model selection procedure.

Table A1:

Parameter constraints for the L-BFGS-B algorithm.

Scenario	Candidate Distribution	θ 1	θ 2
S 1	Normal	μ ∈ (Qmin, Qmax)	σ ∈ (10−3, 50)
	Log-Normal	μ ∈ (log(Qmin), log(Qmax))	σ ∈ (10−3, 50)
	Gamma	α ∈ (10−3, 100)	β ∈ (10−3, 100)
	Beta	α ∈ (10−3, 40)	β ∈ (10−3, 40)
	Weibull	λ ∈ (10−3, 100)	k ∈ (10−3, 100)
S2 & S3	Normal	μ ∈ (Q1, Q3)	σ ∈ (10−3, 50)
	Log-Normal	μ ∈ (log(Q1), log(Q3))	σ ∈ (10−3, 50)
	Gamma	α ∈ (10−3, 100)	β ∈ (10−3, 100)
	Beta	α ∈ (10−3, 40)	β ∈ (10−3, 40)
	Weibull	λ ∈ (10−3, 100)	k ∈ (10−3, 100)

Appendix B

To estimate sample mean and standard deviation using the BC method, the use of Box-Cox transformations requires the solutions to the following problems.

The first problem is defined as follows. In S₁, given Q_min, Q₂, and Q_max such that Q_min < Q₂ < Q_max, find the finite power λ of transformation such that

$$f_{\lambda }\left(Q_{\text{max}}\right)-f_{\lambda }\left(Q_2\right)=f_{\lambda }\left(Q_2\right)-f_{\lambda }\left(Q_{\text{min}}\right)$$

Equivalently, this problem can be restated as finding λ such that

$${\left(\frac{f_{\lambda }\left(Q_{\text{max}}\right)-f_{\lambda }\left(Q_2\right)}{f_{\lambda }\left(Q_2\right)-f_{\lambda }\left(Q_{\text{min}}\right)}-1\right)}^2$$

is minimized to zero. Similarly, given Q₁, Q₂, and Q₃ such that Q₁ < Q₂ < Q₃, the corresponding minimization problem in S₂ is finding λ such that

$${\left(\frac{f_{\lambda }\left(Q_3\right)-f_{\lambda }\left(Q_2\right)}{f_{\lambda }\left(Q_2\right)-f_{\lambda }\left(Q_1\right)}-1\right)}^2$$

is minimized to zero. Given Q_min, Q₁, Q₂, Q₃, and Q_max such that Q_min < Q₂ <Q_max and Q₁ < Q₂ < Q₃, the corresponding minimization problem in S₃ is finding λ such that the following expression is minimized,

$${\left(\frac{f_{\lambda }\left(Q_3\right)-f_{\lambda }\left(Q_2\right)}{f_{\lambda }\left(Q_2\right)-f_{\lambda }\left(Q_1\right)}-1\right)}^2+{\left(\frac{f_{\lambda }\left(Q_{\text{max}}\right)-f_{\lambda }\left(Q_2\right)}{f_{\lambda }\left(Q_2\right)-f_{\lambda }\left(Q_{\text{min}}\right)}-1\right)}^2.$$

To find λ, we use the built-in function ‘optimize’ in R. This function uses a combination of golden section search and successive parabolic interpolation for one-dimensional optimization.

The second problem arises when λ < 0 because in this case the mean and/or standard deviation are likely to be infinite. For example, λ = −1 results in a Cauchy distribution which has undefined mean and standard deviation. Therefore, we let λ = 0 in this case so that λ is non-negative. By doing so, we implicitly assumed that the underlying distribution cannot be more heavy-tailed than a log-normal distribution. If this assumption does not hold, then estimating the mean and standard deviation of the underlying distribution may not be appropriate.