Comparisons of the mean differences and standardized mean differences for continuous outcome measures on the same scale

Abstract

When conducting systematic reviews and meta-analyses of continuous outcomes, the mean differences (MDs) and standardized mean differences (SMDs) are 2 commonly used choices for effect measures. The SMDs are motivated by scenarios where studies collected in a systematic review do not report the continuous measures on the same scale. The standardization process transfers the MDs to be unit-free measures that can be synthesized across studies. As such, some evidence synthesis researchers tend to prefer the SMD over the MD. However, other researchers have concerns about the interpretability of the SMD. The standardization process could also yield additional heterogeneity between studies. In this paper, we use simulation studies to illustrate that, in a scenario where the continuous measures are on the same scale, the SMD could have considerably poorer performance compared with the MD in some cases. The simulations compare the MD and SMD in various settings, including cases where the normality assumption of continuous measures does not hold. We conclude that although the SMD remains useful for evidence synthesis of continuous measures on different scales, the SMD could have substantially greater biases, greater mean squared errors, and lower coverage probabilities of CIs than the MD. The MD is generally more robust to the violation of the normality assumption for continuous measures. In scenarios where continuous measures are inherently comparable or can be transformed to a common scale, the MD is the preferred choice for an effect measure.

Introduction

A systematic review is a comprehensive synthesis of research literature conducted according to predefined criteria and following internationally recognized standards or methods. There are different types of systematic reviews: those focusing on qualitative studies, those focusing on quantitative studies, and mixed methods reviews, which incorporate both.

Meta-analysis is commonly used to integrate evidence from multiple studies in a systematic review.^1–3 Meta-analysis is specifically used in systematic reviews of quantitative studies and in the quantitative synthesis part of mixed methods reviews. The findings from individual quantitative studies are typically quantified by a specific effect measure, which is a basic element in meta-analysis. It statistically combines the results of multiple studies that are assessing similar outcomes. Various types of effect measures exist for continuous outcomes, which are commonly used in clinical, educational, and psychological studies.

For example, obesity is usually determined by body mass index (BMI), which is a continuous measure based on height and weight. The mean difference (MD) and standardized mean difference (SMD) are popular choices to analyze the difference between the experimental group and the control group. The MD quantifies the difference in means between 2 groups, and it generally has a natural unit corresponding to the continuous measure scale. For instance, the unit of change in BMI is kg/m². In a meta-analysis, different studies may use different scales to measure the continuous outcomes (eg, some studies on obesity may report changes in BMI in the unit of kg/m², whereas others may report changes in weight in the unit of kg. The MDs are not directly comparable across studies. To solve this problem, researchers often opt for the SMD as the effect measure.^4–6 The SMD is calculated by dividing the MD between the experimental and control groups by a common standard deviation (SD). Consequently, the outcome measures represented by SMDs have uniform units across studies, and they can be synthesized in a meta-analysis.

The evidence-based medicine community has debated over the choice of MD or SMD for decades.^7–10 In general, when all studies report on the same scale, both the MD and SMD can be used.¹¹ Some researchers recommend the MD instead of the SMD, as the MD has a meaningful unit,¹² such as kg in weight loss or gain. Because the SMD includes the SD in the calculation and is measured in statistical units, it may be difficult to interpret from a clinical perspective.¹³ However, when the unit is unfamiliar, the SMD is widely used to facilitate comparisons across studies because the SMD is unit-free. The magnitude of the SMD is frequently evaluated based on Cohen’s criterion, where the SMDs of 0.2, 0.5, and 0.8 are used as the cutoff points to classify negligible, small, medium, and large effect sizes.^1,6,14

Notably, Cohen’s thresholds are based on conventions that were originally proposed in the context of psychological research. As such, they may not be universally applicable or appropriate across all fields of study and may only be used as rules of thumb.¹⁵ Furthermore, it is critical to understand the nuances between different estimators of the SMD.¹⁶ Cohen’s $d$ is calculated using a pooled SD, while Glass’ $\mathrm{\Delta }$ uses the control group’s SD, and Hedges’ $g$ is an adjusted form of Cohen’s $d$ for small sample sizes. Each of these estimators has specific contexts where they are most appropriately applied.⁶ Readers seeking more in-depth information may refer to the review articles by Andrade,⁶ Lin and Aloe,¹⁶ and Chapter 6.5 of the Cochrane Handbook,¹¹ which provide detailed explorations and examples about various SMD estimators.

Although the calculation and properties of SMD have been extensively studied in the statistical literature, researchers frequently encounter misconceptions and may make mistakes when applying it in practice. Empirical studies have highlighted the prevalence of disagreements and errors in meta-analyses that utilize SMDs, with discrepancies often exceeding the effect size of commonly used treatments.^17,18 For instance, one study found that in 37% of meta-analyses that reported a result as an SMD, the result could not be replicated within a margin of 0.1 for at least one of the selected trials.¹⁷ This discrepancy often stemmed from errors in data such as patient numbers, means, SDs, and the direction of the effect estimate. Moreover, another study on inter-observer variation in data extraction for meta-analyses using SMDs revealed a 53% agreement at the trial level and only 31% at the meta-analysis level.¹⁸ This study highlighted key reasons for disagreement, including differences in the selection of time points, scales, control groups, and calculation types. It also emphasized that observer errors could lead to different conclusions than those published in the original reviews.

Additionally, our recent work has revealed inconsistencies in the current literature, including books and software packages, regarding the methods for handling SMDs, which can potentially hinder research reproducibility.¹⁶ It is also important to note that SMDs can be calculated from various types of MDs and SDs, such as baseline measurements, changes during the study period, or end-of-study measurements. However, many reports tend to oversimplify interpretations and provide insufficient details in this regard.¹⁹ In addition, the commonly used SMD is criticized because of dividing estimates by sample SDs, which are themselves subject to sampling errors.²⁰ More importantly, the SMD exposes the findings to various types of distortion, as studies can differ significantly in patient population inclusion criteria, leading to concerns about generalizability. For instance, for investigating the effects of antidepressive drugs, studies with narrow inclusion criteria, such as “severe depression,” are expected to yield smaller SDs and consequently larger SMDs compared with studies involving patients with a broader range of depression severity, such as “severe to moderate depression.”

In this article, we focus on comparing the performance of the MD and SMD when the continuous measures are on the same scale. The performance is assessed in various scenarios of the distributional assumptions for the continuous measures. In most applications of evidence syntheses, the continuous measures are often assumed to be normally distributed, but this assumption is rarely assessed.²¹ For example, due to the existence of potential outliers or extreme values, continuous outcome measures may also follow a heavy-tailed distribution. Some medical research may want to include only patients with severe depression scores, and the values below the cutoff of the depression scale are removed; thus, their distributions might be skewed.²² We design simulation studies to examine how the MD and SMD perform when the continuous outcomes follow the various types of distributions.

Methods

Mean difference and standardized mean difference in a meta-analysis

Suppose a meta-analysis includes $N$ studies indexed by $i=1,\dots ,N$. The group index $j$ is 0 and 1, representing the control and experimental groups, respectively. Let the sample size be $n_i$ for each study. Suppose $y_{ijk}$ is the continuous measure of subject $k$ in group $j$ of study $i$ for $k=1,\dots ,n_i$. It is generally not reported in published articles unless individual participant data are available. For all $N$ studies, we assume $y_{ijk}$ to follow a distribution with mean ${\mu }_{ij}$ and variance ${\sigma }_i^2$. Here, ${\mu }_{ij}$ represents the true mean for group $j$ in study $i;{\sigma }_i^2$ is the common variance for both groups.

The true MD in study $i$ is defined as ${\mathrm{\Delta }}_i={\mu }_{i1}-{\mu }_{i0}$, and the true SMD is ${\theta }_i=\frac{{\mathrm{\Delta }}_i}{{\sigma }_i}$. Published articles commonly report sample means and sample variances for continuous outcomes, denoted by ${\stackrel{-}{y}}_{ij}$ and $s_{ij}^2$, respectively, for group $j$ in study $i$. With these summary data, the MD can be estimated as ${\widehat{\mathrm{\Delta }}}_i={\stackrel{-}{y}}_{i1}-{\stackrel{-}{y}}_{i0}$, and its variance is estimated as $\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}\left({\widehat{\mathrm{\Delta }}}_i\right)=\left(\frac{1}{n_{i0}}+\frac{1}{n_{i1}}\right)s_{i,\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}^2$, where $s_{i,\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}^2$ is the pooled sample variance $s_{i,\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}^2=\frac{\left(n_{i0}-1\right)s_{i0}^2+\left(n_{i1}-1\right)s_{i1}^2}{n_{i0}+n_{i1}-2}$. The SMD can be estimated as ${\widehat{\theta }}_i=\frac{{\stackrel{-}{y}}_{i1}-{\stackrel{-}{y}}_{i0}}{s_{i,\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}}$; its variance is approximated as $\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}\left({\widehat{\theta }}_i\right)\approx \frac{1}{n_{i0}}+\frac{1}{n_{i1}}+\frac{{\widehat{\theta }}_i^2}{2\left(n_{i0}+n_{i1}-2\right)}$. This estimator is known as Cohen’s $d$.¹⁴

Other estimators are available for the SMD,¹⁶ such as Hedges’ $g$.²³ Although Hedges’ $g$ can remove bias caused by small sample sizes for an individual study, this property of bias correction is not warranted for the synthesized SMD estimates due to the intrinsic correlation between the SMD estimates and their standard errors.²⁴ To elaborate, Hedges’ $g$ is unbiased within a single study context. A common but overly simplistic argument regarding the unbiasedness of the overall effect estimate $\widehat{\theta }=\sum w_i{g}_i/\sum w_i$ in a meta-analysis is that $E\left[\widehat{\theta }\right]=\sum w_{i}E\left[g_i\right]/\sum w_i=\theta$. However, this overlooks the fact that the weights $w_i$ (inverse of variances) are not fixed values but estimates subject to sampling error. The association between Hedges’ $g$ and the corresponding variances may be strong, particularly when the sample sizes are small, complicating the calculation of the meta-estimate’s expectation. Simulations have shown that Hedges’ $g$ does not necessarily produce less biased meta-estimates compared with Cohen’s $d$.¹⁶ Given these considerations, this article opts for Cohen’s $d$ as the estimator for the SMD.

Simulation designs

We perform simulation studies to compare the results of MD and SMD when the latent continuous measures are from different distributions. To this end, we need to define an overall MD and SMD, denoted by $\mathrm{\Delta }$ and $\theta$, respectively. The random-effects model is often employed to account for between-study heterogeneity, and it assumes that study-specific true effect measures, ${\mathrm{\Delta }}_i$ and ${\theta }_i$, following normal distributions with means $\mathrm{\Delta }$ and $\theta$, respectively, across studies. Of note, such distributions differ from the within-study distributions for the individual participant data $y_{ijk}$. To enable direct comparisons between MD and SMD within the same meta-analyses, it is necessary to assume uniform variances, ${\sigma }_i^2$, across studies, which we denote as ${\sigma }^2$. Absent this assumption, generating meta-analyses with study-specific true MDs, ${\mathrm{\Delta }}_i$, distributed as $N\left(\mathrm{\Delta },{\tau }^2\right)$, would result in true SMDs, ${\theta }_i$ follow $N\left(\frac{\mathrm{\Delta }}{{\sigma }_i},\frac{{\tau }^2}{{\sigma }_i^2}\right)$, which are not identically distributed. This violates the assumptions of the random-effects model based on SMDs and could lead to unfair comparisons between MDs and SMDs.

Our simulations are particularly interested in the performance of the MD and SMD under various distributional assumptions for the latent continuous outcome measure $y_{ijk}$. We consider normal distribution, $t$ distribution, skewed distribution, and distribution with extreme outliers. The location–scale transformation is used for the continuous outcome measures: $y_{ijk}={\mu }_{ij}+\sigma \cdot a\cdot u_{ijk}$. The constant $a$ can vary across distributions, and it is used to adjust the distributions’ variances. The continuous random variable $u_{ijk}$ is assumed to follow different distributions. To make fair comparisons, we design the various distributions to have the same mean ${\mu }_{ij}$ and variance ${\sigma }^2$; that is, $\mathrm{E}\left(u_{ijk}\right)=0$ and $\mathrm{V}\mathrm{a}\mathrm{r}\left(u_{ijk}\right)=1/a^2$.

The true overall MD $\mathrm{\Delta }$ was set to 0 and 1. The common SD $\sigma$ within studies was set to 1 and 2. Consequently, the true overall standardized MD $\theta$ was $\mathrm{\Delta }/\sigma$. The number of studies $N$ in a meta-analysis was set to 5 and 20, representing relatively small and large numbers of studies, respectively. For the total sample sizes $n_i$ within studies, we considered 2 sets of values: $\left\{10,20,30,40,50\right\}$ (relatively small sample sizes) and $\left\{100,150,200,250,300\right\}$ (relatively large sample sizes). For $N=20$, these sets of values were repeated 4 times. The ratio of sample sizes in the experimental and control groups was 1:1, so each group had $n_i/2$ samples. To control reasonable extents of heterogeneity based on the $I^2$ statistic, the between-study SD $\tau$ for the MDs was set to 0.2 for the set of relatively large sample sizes and 0.4 for the set of relatively small sample sizes. Without loss of generality, the mean of subjects’ latent continuous measures in the control group ${\mu }_{i0}$ was set to 0; thus, the mean in the treatment group was ${\mu }_{i1}={\mathrm{\Delta }}_i+{\mu }_{i0}$.

Recall that we used the location–scale transformation for the continuous outcome measures of individual participants: $y_{ijk}={\mu }_{ij}+\sigma \cdot a\cdot u_{ijk}$. Four simulation settings were considered for the distribution of the random variable $u_{ijk}$; the constant $a$ was specific for each setting because $a=\sqrt{1/\mathrm{V}\mathrm{a}\mathrm{r}\left(u_{ijk}\right)}$, so all 4 distributions share the same mean and the same variance. The 4 distributions are as follows.

• Setting 1 (normal distribution): $u_{ijk}~N\left(0,1\right)$ with $a=1$.

• Setting 2 ($t$ distribution): $u_{ijk}~t_{n_{ij}-1}$ with $a=\sqrt{\left(n_{ij}-3\right)/\left(n_{ij}-1\right)}$.

• Setting 3 (skewed distribution): $u_{ijk}~0.8N\left(-0.5,1\right)+0.2N\left(2,0.5\right)$ with $a\approx 0.8$.

• Setting 4 (extreme outliers): $u_{ijk}~0.85N\left(0,0.5\right)+0.15N\left(0,20\right)$ with $a\approx 0.5$.

In Setting 1, the random variable $u_{ijk}$ has a standard normal distribution. Setting 2 uses the $t$ distribution to present some heavy-tailed errors. In Setting 3, the mixture component $N\left(2,0.5\right)$ ensures that $u_{ijk}$ has long right tails. In Setting 4, most of $u_{ijk}$’s were close to 0 due to the component $N\left(0,0.5\right)$, but the component $N\left(0,20\right)$ ensured that a small number of extreme outliers existed in both directions. Figure 1 visualizes these 4 distributions.

Figure 1:

Probability distribution functions of continuous measures $y_{ijk}$ generated from the 4 distributions: Setting 1 (A), Setting 2 (B), Setting 3 (C), and Setting 4 (D).

We used a factorial design, yielding 64 simulation settings in total (2 choices of $\mathrm{\Delta }\times 2$ choices of $\sigma \times 2$ choices of $N\times 2$ choices of $n_i\times 4$ choices of distributions). For each setting, we generated 1000 replicates of meta-analyses. For $N$ studies in each meta-analysis, we generated the study-specific MD ${\mathrm{\Delta }}_i$ from $N\left(\mathrm{\Delta },{\tau }^2\right)$; thus, the study-specific SMD ${\theta }_i$ was ${\theta }_i={\mathrm{\Delta }}_i/\sigma$. We generated the random variable $u_{ijk}$ from Settings 1–4 of distributions, and each individual’s continuous outcome measure was obtained from $y_{ijk}={\mu }_{ij}+\sigma \cdot a\cdot u_{ijk}$ for group $j=0,1$. We calculated the sample means ${\stackrel{-}{y}}_{ij}$ and sample variances $s_{ij}^2$. The estimated MD ${\widehat{\mathrm{\Delta }}}_i$, SMD ${\widehat{\theta }}_i$, and their corresponding sample variances were calculated based on ${\stackrel{-}{y}}_{ij},s_{ij}^2$, and $n_{ij}$.

We used the random-effects model to separately synthesize the observed MDs ${\widehat{\mathrm{\Delta }}}_i$ and observed SMDs ${\widehat{\theta }}_i$ from the $N$ studies. This method was implemented with the function rma() in the R metaphor package (R Foundation for Statistical Computing, Vienna, Austria) with the restricted maximum likelihood estimation for the heterogeneity variance.²⁵ We used the Hartung-Knapp-Sidik-Jonkman method for deriving 95% CIs.^26–28 This method has been shown to be generally more appropriate than the conventional DerSimonian-Laird method.^28,29 We obtained the estimated overall MDs, estimated overall SMDs, and their corresponding 95% CIs.

The bias, root mean squared error (RMSE), and CI coverage probability were calculated to compare the performance of the MD and SMD. We obtained their Monte Carlo standard errors to quantify their uncertainties. In each simulation setting, the bias was computed as $B^{-1}\sum _{b=1}^B{\widehat{d}}_b-d$, where $B=1000$ was the number of replicates, ${\widehat{d}}_b$ was the overall estimate of the MD or SMD in the $b^{\mathrm{t}\mathrm{h}}$ simulation replicate, and $d$ was the true MD or SMD. The RMSE was ${\left[B^{-1}\sum _{b=1}^B{\left({\widehat{d}}_b-d\right)}^2\right]}^{1/2}$, and the coverage probability was $B^{-1}\sum _{b=1}^{B}I\left({\widehat{d}}_{b,\mathrm{l}\mathrm{o}\mathrm{w}}\le d\le {\widehat{d}}_{b,\mathrm{u}\mathrm{p}\mathrm{p}}\right)$, where ${\widehat{d}}_{b,\mathrm{l}\mathrm{o}\mathrm{w}}$ and ${\widehat{d}}_{b,\mathrm{u}\mathrm{p}\mathrm{p}}$ were the lower and upper bounds of the CI, respectively. The indicator function $I\left(\cdot \right)$ returned 1 if the CI covered the true MD $\mathrm{\Delta }$ or SMD $\theta$ and 0 if not. We also converted the bias and RMSE of SMD to the scale of MD by multiplying the common SD $\sigma$. The coverage probability of SMD on the scale of MD was the same as the original coverage probability of SMD.

Results

Across all settings, the Monte Carlo standard errors of biases in MD and SMD were less than 0.014, those of RMSEs were less than 0.012, and those of coverage probabilities were less than 1.5%. The biases, RMSEs, and coverage probabilities of SMDs were converted to the scale of MD for fair comparisons. In the following interpretations, the biases were interpreted in absolute magnitude.

Table 1 summarizes the results when the continuous measures $y_{ijk}$ were generated from the normal distribution in Setting 1. Most biases in SMDs were similar to those in MD. Two scenarios yielded SMDs with noticeably greater biases than MDs. When the true overall MD $\mathrm{\Delta }$ was 1, the number of studies $N$ was 20, and the sample sizes were within 10–50, then the bias in MDs was 0.001 and the bias in SMDs was 0.014 for the within-study SD $\sigma =1$ and true overall SMD $\theta =1$; the bias in MDs was 0.003 and the bias in SMDs was 0.013 for the within-study SD $\sigma =2$ and true overall SMD $\theta =0.5$. The RMSEs of SMDs and MDs were ≤ 0.39. In most scenarios, SMDs had smaller RMSEs than MDs for $\mathrm{\Delta }=0$, while MDs produced smaller RMSEs for $\mathrm{\Delta }=1$. Their differences were tiny by up to 0.01. The CIs of MDs and SMDs also had similar coverage probabilities in most scenarios, which were close to 95%.

Table 1:

Bias, root mean squared errors (RMSEs), and coverage probabilities with their Monte Carlo standard errors (in parentheses) of the mean difference (MD) and standardized mean difference (SMD) in the MD scale when the continuous measures $y_{ijk}$ are generated from the normal distribution in Setting 1.

$\mathbf{\Delta }$	$\mathit{\theta }$	$\mathit{\sigma }$	$\mathit{N}$	${\mathit{n}}_{\mathit{i}}$	$\mathit{\tau }$	Bias		RMSE		Coverage probability
						MD	SMD on MD scale	MD	SMD on MD scale	MD	SMD on MD scale
0	0	1	5	10–50	0.4	0.011 (0.008)	0.010 (0.008)	0.250 (0.005)	0.249 (0.006)	94.7% (0.7%)	94.8% (0.7%)
				100–300	0.2	0.007 (0.003)	0.007 (0.003)	0.110 (0.002)	0.109 (0.002)	94.8% (0.7%)	95.0% (0.7%)
			20	10–50	0.4	−0.001 (0.004)	0.000 (0.004)	0.129 (0.003)	0.127 (0.003)	95.3% (0.7%)	96.2% (0.6%)
				100–300	0.2	−0.003 (0.002)	−0.003 (0.002)	0.057 (0.001)	0.057 (0.001)	95.0% (0.7%)	95.3% (0.7%)
	0	2	5	10–50	0.4	0.009 (0.012)	0.008 (0.012)	0.384 (0.009)	0.383 (0.008)	94.6% (0.7%)	94.7% (0.7%)
				100–300	0.2	0.011 (0.005)	0.011 (0.004)	0.156 (0.003)	0.156 (0.004)	94.1% (0.7%)	94.8% (0.7%)
			20	10–50	0.4	−0.003 (0.006)	−0.002 (0.006)	0.199 (0.004)	0.192 (0.004)	94.3% (0.7%)	95.6% (0.6%)
				100–300	0.2	−0.002 (0.003)	−0.002 (0.002)	0.081 (0.002)	0.080 (0.002)	94.9% (0.7%)	95.0% (0.7%)
1	1	1	5	10–50	0.4	0.011 (0.008)	0.009 (0.008)	0.250 (0.005)	0.260 (0.006)	94.7% (0.7%)	95.0% (0.7%)
				100–300	0.2	0.007 (0.003)	0.005 (0.004)	0.110 (0.002)	0.113 (0.002)	94.8% (0.7%)	93.9% (0.8%)
			20	10–50	0.4	−0.001 (0.004)	−0.014 (0.004)	0.129 (0.003)	0.132 (0.003)	95.3% (0.7%)	94.4% (0.7%)
				100–300	0.2	−0.003 (0.002)	−0.004 (0.002)	0.057 (0.001)	0.059 (0.001)	95.0% (0.7%)	94.7% (0.7%)
	0.5	2	5	10–50	0.4	0.009 (0.012)	0.010 (0.012)	0.384 (0.009)	0.390 (0.010)	94.6% (0.7%)	94.9% (0.7%)
				100–300	0.2	0.011 (0.005)	0.010 (0.006)	0.156 (0.003)	0.159 (0.004)	94.1% (0.7%)	94.0% (0.8%)
			20	10–50	0.4	−0.003 (0.006)	−0.013 (0.006)	0.199 (0.004)	0.196 (0.004)	94.3% (0.7%)	94.8% (0.7%)
				100–300	0.2	−0.002 (0.003)	−0.004 (0.002)	0.081 (0.002)	0.082 (0.002)	94.9% (0.7%)	94.9% (0.7%)

$\mathrm{\Delta }$, true overall mean difference; $\theta$, true overall standardized mean difference; $\sigma$, within-study standard deviation; $N$, number of studies; $n_i$, sample size within a study; $\tau$, between-study standard deviation.

Table 2 shows the results when $y_{ijk}$ were generated from the $t$ distribution in Setting 2. The SMD and MD had very similar biases when the true overall MD $\mathrm{\Delta }$ was 0; their differences were up to 0.001. However, when $\mathrm{\Delta }$ was 1, the biases in SMDs were greater than those in MDs in most scenarios. When $\mathrm{\Delta }=1,N=5$, and sample sizes were within 10–50, then the bias in SMDs was 0.021, greater than that in MDs for $\sigma =1$, and their difference was 0.014 for $\sigma =2$. Most RMSEs of SMDs were slightly greater than those of MDs by up to 0.019. The RMSEs of SMDs ranged from 0.058 to 0.400, and those of MDs ranged from 0.058 to 0.388. SMDs generally had CIs with slightly higher coverage probabilities by up to 1.5% than MDs.

Table 2:

Bias, root mean squared errors (RMSEs), and coverage probabilities with their Monte Carlo standard errors (in parentheses) of the mean difference (MD) and standardized mean difference (SMD) in the MD scale when the continuous measures $y_{ijk}$ are generated from the $t$ distribution in Setting 2.

$\mathbf{\Delta }$	$\mathit{\theta }$	$\mathit{\sigma }$	$\mathit{N}$	${\mathit{n}}_{\mathit{i}}$	$\mathit{\tau }$	Bias		RMSE		Coverage probability
						MD	SMD on MD scale	MD	SMD on MD scale	MD	SMD on MD scale
0	0	1	5	10–50	0.4	0.009 (0.008)	0.009 (0.008)	0.254 (0.006)	0.259 (0.006)	94.8% (0.7%)	95.4% (0.7%)
				100–300	0.2	−0.005 (0.004)	−0.005 (0.004)	0.115 (0.002)	0.115 (0.003)	93.1% (0.8%)	93.3% (0.8%)
			20	10–50	0.4	0.008 (0.004)	0.008 (0.004)	0.125 (0.003)	0.126 (0.003)	93.3% (0.8%)	94.8% (0.7%)
				100–300	0.2	0.002 (0.002)	0.002 (0.002)	0.058 (0.001)	0.058 (0.001)	94.1% (0.7%)	94.1% (0.7%)
	0	2	5	10–50	0.4	0.013 (0.012)	0.014 (0.012)	0.388 (0.009)	0.391 (0.008)	95.2% (0.7%)	96.2% (0.6%)
				100–300	0.2	−0.005 (0.005)	−0.005 (0.006)	0.161 (0.003)	0.161 (0.004)	93.6% (0.8%)	93.9% (0.8%)
			20	10–50	0.4	0.010 (0.006)	0.011 (0.006)	0.192 (0.004)	0.191 (0.004)	93.6% (0.8%)	94.2% (0.7%)
				100–300	0.2	0.002 (0.003)	0.002 (0.002)	0.081 (0.002)	0.081 (0.002)	94.4% (0.7%)	94.5% (0.7%)
1	1	1	5	10–50	0.4	0.009 (0.008)	0.021 (0.009)	0.254 (0.006)	0.273 (0.006)	94.8% (0.7%)	95.3% (0.7%)
				100–300	0.2	−0.005 (0.004)	−0.006 (0.004)	0.115 (0.002)	0.117 (0.002)	93.1% (0.8%)	93.5% (0.8%)
			20	10–50	0.4	0.008 (0.004)	0.010 (0.004)	0.125 (0.003)	0.133 (0.003)	93.3% (0.8%)	94.7% (0.7%)
				100–300	0.2	0.002 (0.002)	0.000 (0.002)	0.058 (0.001)	0.059 (0.001)	94.1% (0.7%)	94.7% (0.7%)
	0.5	2	5	10–50	0.4	0.013 (0.012)	0.027 (0.012)	0.388 (0.009)	0.400 (0.010)	95.2% (0.7%)	96.2% (0.6%)
				100–300	0.2	−0.005 (0.005)	−0.005 (0.006)	0.161 (0.003)	0.163 (0.004)	93.6% (0.8%)	94.3% (0.7%)
			20	10–50	0.4	0.010 (0.006)	0.013 (0.006)	0.192 (0.004)	0.195 (0.004)	93.6% (0.8%)	94.9% (0.7%)
				100–300	0.2	0.002 (0.003)	0.000 (0.002)	0.081 (0.002)	0.082 (0.002)	94.4% (0.7%)	94.6% (0.7%)

$\mathrm{\Delta }$, true overall mean difference; $\theta$, true overall standardized mean difference; $\sigma$, within-study standard deviation; $N$, number of studies; $n_i$, sample size within a study; $\tau$, between-study standard deviation.

When $y_{ijk}$ were generated from the skewed distribution in Setting 3, the results for SMDs and MDs had large differences, as shown in Table 3. The biases in MDs were generally small, ranging from 0.002 to 0.011. The SMD had small biases up to 0.011 when $\mathrm{\Delta }=0$, while they sharply increased when $\mathrm{\Delta }=1$, ranging from 0.084 to 0.095. The RMSEs of SMDs and MDs were up to 0.425. In most cases, SMDs performed better with smaller RMSEs than MDs. The coverage probabilities of MDs’ CIs ranged from 93.2% to 96.6%. The CIs of SMDs had slightly higher coverage probabilities when the true overall MD $\mathrm{\Delta }$ was 0, but they had considerably poor performance when $\mathrm{\Delta }$ was 1. The coverage probabilities of SMDs’ CIs could drop to 65.1% when $\mathrm{\Delta }=1,\theta =1,\sigma =1,N=20$, and sample sizes were within 100–300.

Table 3:

Bias, root mean squared errors (RMSEs), and coverage probabilities with their Monte Carlo standard errors (in parentheses) of the mean difference (MD) and standardized mean difference (SMD) in the MD scale when the continuous measures $y_{ijk}$ are generated from the skewed distribution in Setting 3.

$\mathbf{\Delta }$	$\mathit{\theta }$	$\mathit{\sigma }$	$\mathit{N}$	${\mathit{n}}_{\mathit{i}}$	$\mathit{\tau }$	Bias		RMSE		Coverage probability
						MD	SMD on MD scale	MD	SMD on MD scale	MD	SMD on MD scale
0	0	1	5	10–50	0.4	0.008 (0.008)	0.008 (0.008)	0.266 (0.006)	0.241 (0.006)	94.7% (0.7%)	95.8% (0.6%)
				100–300	0.2	−0.006 (0.004)	−0.005 (0.004)	0.124 (0.003)	0.113 (0.003)	93.4% (0.8%)	93.6% (0.8%)
			20	10–50	0.4	0.007 (0.004)	0.007 (0.004)	0.140 (0.003)	0.126 (0.003)	93.2% (0.8%)	93.9% (0.8%)
				100–300	0.2	0.002 (0.002)	0.002 (0.002)	0.057 (0.001)	0.052 (0.001)	96.6% (0.6%)	96.6% (0.6%)
	0	2	5	10–50	0.4	0.009 (0.013)	0.010 (0.012)	0.425 (0.009)	0.379 (0.008)	95.1% (0.7%)	96.3% (0.6%)
				100–300	0.2	−0.006 (0.006)	−0.006 (0.006)	0.182 (0.004)	0.165 (0.004)	93.7% (0.8%)	94.1% (0.7%)
			20	10–50	0.4	0.011 (0.007)	0.011 (0.006)	0.220 (0.005)	0.195 (0.004)	93.8% (0.8%)	94.4% (0.7%)
				100–300	0.2	0.004 (0.003)	0.004 (0.002)	0.083 (0.002)	0.076 (0.002)	96.1% (0.6%)	96.5% (0.6%)
1	1	1	5	10–50	0.4	0.008 (0.008)	−0.088 (0.008)	0.266 (0.006)	0.259 (0.006)	94.7% (0.7%)	94.2% (0.7%)
				100–300	0.2	−0.006 (0.004)	−0.095 (0.004)	0.124 (0.003)	0.148 (0.003)	93.4% (0.8%)	86.4% (1.1%)
			20	10–50	0.4	0.007 (0.004)	−0.094 (0.004)	0.140 (0.003)	0.159 (0.003)	93.2% (0.8%)	86.0% (1.1%)
				100–300	0.2	0.002 (0.002)	−0.088 (0.002)	0.057 (0.001)	0.103 (0.002)	96.6% (0.6%)	65.1% (1.5%)
	0.5	2	5	10–50	0.4	0.009 (0.013)	−0.084 (0.012)	0.425 (0.009)	0.388 (0.008)	95.1% (0.7%)	95.5% (0.7%)
				100–300	0.2	−0.006 (0.006)	−0.095 (0.006)	0.182 (0.004)	0.191 (0.004)	93.7% (0.8%)	90.6% (0.9%)
			20	10–50	0.4	0.011 (0.007)	−0.088 (0.006)	0.220 (0.005)	0.215 (0.006)	93.8% (0.8%)	91.9% (0.9%)
				100–300	0.2	0.004 (0.003)	−0.086 (0.002)	0.083 (0.002)	0.115 (0.002)	96.1% (0.6%)	81.5% (1.2%)

$\mathrm{\Delta }$, true overall mean difference; $\theta$, true overall standardized mean difference; $\sigma$, within-study standard deviation; $N$, number of studies; $n_i$, sample size within a study; $\tau$, between-study standard deviation.

Table 4 summarizes the results when the generation distribution of $y_{ijk}$ had extreme outliers in Setting 4. MDs generally had small biases, ranging from 0.002 to 0.008. When $\mathrm{\Delta }=0$, SMDs had small biases, which were close to the results of MDs. For $\mathrm{\Delta }=1$, the biases of SMDs were greater than those of MDs; their differences were up to 0.025 when sample sizes were within 100–300. SMDs had relatively large biases ranging from 0.177 to 0.205 when sample sizes were within 10–50. In all scenarios, the RMSEs of SMDs and MDs were ≤ 0.51, and SMDs had larger RMSEs than MDs. Their differences were up to 0.022 when sample sizes were within 100–300, while their differences were large by up to 0.181 when sample sizes were small. The coverage probabilities of CIs of MDs ranged from 92.3% to 95.7%. Most coverage probabilities of SMDs’ CIs were slightly higher than those of MDs. For $\mathrm{\Delta }=1$ and $N=20$, SMDs had considerably lower coverage probabilities than MDs, and they could drop to 86.4% when $\sigma =1$ and sample sizes were within 10–50.

Table 4:

Bias, root mean squared errors (RMSEs), and coverage probabilities with their Monte Carlo standard errors (in parentheses) of the mean difference (MD) and standardized mean difference (SMD) in the MD scale when the continuous measures $y_{ijk}$ are generated from the distribution with extreme outliers in Setting 4.

$\mathbf{\Delta }$	$\mathit{\theta }$	$\mathit{\sigma }$	$\mathit{N}$	${\mathit{n}}_{\mathit{i}}$	$\mathit{\tau }$	Bias		RMSE		Coverage probability
						MD	SMD on MD scale	MD	SMD on MD scale	MD	SMD on MD scale
0	0	1	5	10–50	0.4	0.006 (0.008)	0.009 (0.009)	0.241 (0.006)	0.288 (0.007)	93.0% (0.8%)	95.3% (0.7%)
				100–300	0.2	−0.003 (0.004)	−0.003 (0.004)	0.115 (0.003)	0.118 (0.003)	93.8% (0.8%)	94.5% (0.7%)
			20	10–50	0.4	0.004 (0.004)	0.004 (0.005)	0.127 (0.003)	0.151 (0.004)	93.2% (0.8%)	94.4% (0.7%)
				100–300	0.2	0.003 (0.002)	0.003 (0.002)	0.055 (0.001)	0.057 (0.001)	95.7% (0.6%)	95.8% (0.6%)
	0	2	5	10–50	0.4	0.008 (0.011)	0.008 (0.014)	0.350 (0.008)	0.414 (0.010)	93.2% (0.8%)	94.5% (0.7%)
				100–300	0.2	−0.002 (0.005)	−0.002 (0.006)	0.161 (0.004)	0.164 (0.004)	94.2% (0.7%)	94.7% (0.7%)
			20	10–50	0.4	0.002 (0.006)	0.004 (0.006)	0.180 (0.004)	0.212 (0.004)	92.3% (0.8%)	93.9% (0.8%)
				100–300	0.2	0.004 (0.002)	0.004 (0.002)	0.077 (0.002)	0.079 (0.002)	95.0% (0.7%)	95.0% (0.7%)
1	1	1	5	10–50	0.4	0.006 (0.008)	0.205 (0.012)	0.241 (0.006)	0.422 (0.010)	93.0% (0.8%)	94.0% (0.8%)
				100–300	0.2	−0.003 (0.004)	0.025 (0.004)	0.115 (0.003)	0.137 (0.004)	93.8% (0.8%)	94.2% (0.7%)
			20	10–50	0.4	0.004 (0.004)	0.177 (0.006)	0.127 (0.003)	0.257 (0.006)	93.2% (0.8%)	86.4% (1.1%)
				100–300	0.2	0.003 (0.002)	0.028 (0.002)	0.055 (0.001)	0.073 (0.002)	95.7% (0.6%)	93.8% (0.8%)
	0.5	2	5	10–50	0.4	0.008 (0.011)	0.203 (0.014)	0.350 (0.008)	0.510 (0.012)	93.2% (0.8%)	94.5% (0.7%)
				100–300	0.2	−0.002 (0.005)	0.026 (0.006)	0.161 (0.004)	0.177 (0.004)	94.2% (0.7%)	95.1% (0.7%)
			20	10–50	0.4	0.002 (0.006)	0.177 (0.008)	0.180 (0.004)	0.295 (0.006)	92.3% (0.8%)	89.5% (1.0%)
				100–300	0.2	0.004 (0.002)	0.029 (0.002)	0.077 (0.002)	0.091 (0.002)	95.0% (0.7%)	93.8% (0.8%)

$\mathrm{\Delta }$, true overall mean difference; $\theta$, true overall standardized mean difference; $\sigma$, within-study standard deviation; $N$, number of studies; $n_i$, sample size within a study; $\tau$, between-study standard deviation.

Discussion

This article explores the performance of SMD in various cases of distributions for individual subjects’ continuous outcome measures, including the normal distribution, $t$ distribution, skewed distribution, and distribution with extreme outliers. We conducted simulation studies to compare the performance of the MD and SMD. When the true overall effect size was zero, the SMD generally had slightly better or similar performance compared with the MD in terms of bias, RMSE, and CI coverage probability. Most differences were small and likely due to Monte Carlo standard errors. When the overall effect size was large, the SMD preformation was mostly satisfactory in cases of the normal and $t$ distribution, whereas the SMD could have considerably greater biases and RMSEs, and its CI could have much lower coverage probabilities than the MD in some cases of the skewed distribution and distribution with extreme outliers.

In real-world scenarios, different studies reporting continuous outcomes often employ various rating instruments or present indicators in different units, complicating the direct comparison of results across studies. In such instances, the SMD serves as a useful tool for harmonizing these differences, enabling the synthesis of results. This is particularly relevant in fields such as psychology or education, where diverse scales and measurements are common. However, the inherent variability and subjectivity of these instruments should be acknowledged, as they can introduce heterogeneity into the meta-analysis and challenge the rationale of synthesizing the SMD. Researchers need to carefully consider the context and nature of the instruments used in the original studies when interpreting the results of meta-analyses that utilize SMD.

Our exploratory work may have some limitations. In the simulation studies, a common SD was considered for all studies in the same meta-analysis. As mentioned in our simulation designs, this assumption is needed for fair comparisons between the MD and SMD. Without this assumption, our simulations based on the MD could not yield a well-defined value of the overall SMD, and thus the bias, RMSE, and CI coverage probabilities could not be calculated for the SMD. In practice, this assumption is certainly unrealistic. However, we believe our simulations were sufficient to illustrate the potentially poor performance of the SMD in some cases. For handling real-world meta-analyses with continuous outcomes, researchers should evaluate whether it is sensible to assume the exchangeability of MDs or SMDs across studies. If the exchangeability assumption holds for one effect measure, then it may not be valid for another effect measure, except in some rare cases (eg, approximately common SDs for all studies). In addition, this article only assesses the impact of within-study distributional assumptions, and it assumes normality between studies. However, the between-study normality cannot be taken for granted,^21,30 and must be carefully assessed on a case-by-case basis.

Some extension work based on the continuous effect measures can be considered in future research. This article focuses on only 2 effect measures, the MD and SMD, for continuous outcomes. Other methods can be implemented for evidence syntheses with continuous outcomes, such as the minimally important differences or the ratio of means.⁴

Conclusions

While the SMD can be beneficial for synthesizing evidence from continuous measures that vary in scale, researchers must be aware of its potential limitations, particularly its inferior performance compared with the MD when dealing with measures on identical scales. The MD demonstrates superior robustness when the normality assumption for continuous measures is challenged. In scenarios where there is inherent comparability among continuous measures or the possibility of transforming them to a common scale, the MD emerges as the more suitable effect measure. Therefore, we recommend that researchers make a contextually informed choice between MD and SMD based on the scale and distribution of outcome measures in their studies. Understanding the limitations and sensitivity to distributional assumptions of each effect measure is key for accurate interpretation. This requires careful evaluation of study designs and outcome measures to determine the most appropriate effect measure.