Estimating the Reference Range from a Meta-Analysis

Abstract

Often clinicians are interested in determining whether a subject’s measurement falls within a normal range, defined as a range of values of a continuous outcome which contains some proportion (e.g. 95%) of measurements from a healthy population. Several studies in the biomedical field have estimated reference ranges based on a meta-analysis of multiple studies with healthy individuals. However, the literature currently gives no guidance about how to estimate the reference range of a new subject in such settings. Instead, meta-analyses of such normative range studies typically report the pooled mean as a reference value, which does not incorporate natural variation across healthy individuals in different studies. We present three approaches to calculating the normal reference range of a subject from a meta-analysis of normally or lognormally distributed outcomes: a frequentist random effects model, a Bayesian random effects model, and an empirical approach. We present the results of a simulation study demonstrating that the methods perform well under a variety of scenarios, though users should be cautious when the number of studies is small and between-study heterogeneity is large. Finally, we apply these methods to two examples: pediatric time spent awake after sleep onset and frontal subjective postural vertical measurements.

1. Introduction

The number of published meta-analyses has increased sharply over the past several decades^1,2. While most meta-analyses aim to provide a more precise estimate of the effect of a treatment or a risk factor’s association with a disease¹, the literature has many examples of meta-analyses of normative data^3–13. These studies generally aim to establish “typical” or “normal” values for a measurement or outcome using “healthy” populations from multiple studies, to serve as a reference. However, most often these meta-analysis studies report the pooled mean as the “reference value,” which has limited interpretability when determining whether a measurement is “normal”. Although Bohannon⁵ noted that measurements lying outside the confidence interval for the pooled mean could be considered above or below “average”, a reference range would be more useful in determining whether an observed measurement was within the range of values measured on healthy individuals. Horn et al.¹⁴ define a reference range or interval as “a set of values within which some percentage, 95% for example, of the values of a particular analyte in a healthy population would fall.” In a meta-analysis, this requires accounting for the natural variability in the healthy population as reflected by variation both within and between studies.

Several medical systematic reviews have estimated and reported reference ranges using a meta-analysis of healthy individuals from multiple studies^{3,7,10,12,15–18}. However, some of these studies have used the confidence interval for the pooled mean as the “reference range” ^3,16,17, which reflects uncertainty in the estimated mean, not natural variation in the population. Venner et al.¹⁸ used the measurement ranges reported in each study when available to construct reference ranges based on the overall minimum and maximum values across studies. While this better reflects natural variation across healthy individuals than the confidence interval for the pooled mean, only four out of the twelve studies included in the meta-analysis reported ranges, and this method requires setting the desired percentage of individuals captured in the reference range to 100%.

However, several studies estimate reference ranges containing a specified proportion of measurements from a healthy target population based on the observed mean, standard deviation, and sample size from each study^7,10,12,15. Conceição et al.⁷ use a method similar to the empirical approach proposed later in this paper in order to estimate normal ranges for how accurately healthy participants perceive whether they are oriented vertically in space. Wyman et al.¹² use the fixed effects model by Laird and Mosteller¹⁹ in order to establish normative ranges for non-invasive bladder function measurements in healthy women. Nemeth et al.¹⁰ estimate a reference range for normal concentrations of asymmetric dimethylarginine in the plasma of healthy individuals, though their method for estimating the marginal reference range across all studies is not clear. Finally, Khoshdel et al.¹⁵ simulate individual patient data based on the summary statistics from each study, then use fractional polynomials to estimate age-specific reference ranges for pulse wave velocity.

It is unknown what proportion of measurements from the “true” overall populations these reference ranges capture. Currently, the literature gives no guidance on how to approach the question of estimating reference ranges based on meta-analyses. Several authors have recently advocated reporting prediction intervals for a new study^20–23, but they have not addressed prediction for an individual. To the best of our knowledge, the present paper is the first to propose methods for estimating reference ranges based on meta-analyses.

The first two proposed methods build on the commonly used random effects model. Section 2 motivates this choice and introduces notation. Section 3 proposes three approaches for estimating reference ranges. The first uses results from a frequentist method such as restricted maximum likelihood estimation (REML), while the second is a Bayesian approach using a posterior predictive interval. The final method is an empirical approach similar to that used by Conceição et al.⁷. We call these the frequentist, Bayesian, and empirical approaches. All three of these approaches use the means, standard deviations, and sample sizes reported in each study and do not require individual patient data. Section 4 presents simulation studies illustrating the performance of the three approaches, which we then apply to two examples in Section 5. Finally, Section 6 discusses some of the key distributional assumptions required by these approaches and potential areas of future work.

2. Random effects model

2.1. Choice of model

Three models are commonly used in meta-analysis: the common effect, random effects, and fixed effects models. The common effect model assumes the underlying true mean or effect is the same in each study and that variation between studies in estimated mean arises purely from sampling variation^24,25. This is often called a fixed effect model, which is easily confused with the fixed effects model of Laird and Mosteller¹⁹. We follow Bender et al.²⁴ by using the term “common effect model”. This model imposes a strong assumption that each study population has the same underlying true mean, which may not be appropriate for situations in which the observed means from the studies differ for reasons other than sampling variability²⁶. When there is considerable heterogeneity between studies, as is often the case in meta-analyses of continuous outcomes²⁷, it may instead be desirable to assume that each study has a different true mean and that these means are drawn from separate distributions, as in the fixed effects model of Laird and Mosteller¹⁹. However, because this model makes no assumptions about how the effects in the different studies are related, it may not be used to draw any conclusions about a new study measuring the same outcome, much less about a new individual. Therefore, we will instead focus on the random effects model. This model allows the true means to differ between study populations but assumes they follow some underlying distribution, which is most often a normal distribution^24,25. It is common to interpret this assumption as meaning that each study in the meta-analysis was randomly sampled from a population of theoretically possible studies, including the population studied and methods of measurement. However, Higgins et al.²⁸ point out that this stronger assumption is often violated, as later studies are often designed based on the results of previous studies. However, Higgins et al.²⁸ focus on random effects models comparing treatment effects in two groups, while we focus on estimating normal ranges from a group of healthy subjects. Thus, this assumption may be reasonable.

2.2. Notation

Let y_i denote the observed mean for study i = {1,…,k}, θ_i be study i’s true mean, μ_RE be the overall mean of the distribution of study means, and ${\sigma }_i^2$ be study i’s within-study variance. Also, let τ² be the variance of the θ_i across studies. Then, we have

$${\overline{y}}_i\sim N({\theta }_i,{\sigma }_i^2/n_i),{\theta }_i\sim N({\mu }_{RE},{\tau }^2)$$ (1)

In the frequentist framework, the overall mean μ_RE is traditionally estimated as a weighted average of the study-specific means^24,29

$$\begin{array}{ccc}{\widehat{\mu }}_{RE}=\frac{{\sum }_{i=1}^k{y}_i{\omega }_{i,RE}}{{\sum }_{i=1}^k{\omega }_{i,RE}},& \text{for}& {\omega }_{i,RE}=\frac{1}{s_i^2/n_i+{\widehat{\tau }}^2},\end{array}$$ (2)

where $s_i^2$ is study i’s within-study sample variance though there has been some debate about how to estimate τ². Here, we use the restricted maximum likelihood (REML) estimate, as implemented in the meta package³⁰. The commonly-used estimate originally proposed by DerSimonian and Laird²⁹ has been shown to underestimate the true between-study variance, particularly when the number of studies k is small^31–33. The overall variance in ${\widehat{\mu }}_{RE}$ can be estimated by $V_{RE}=\frac{1}{{\sum }_{i=1}^k{\omega }_{i,RE}}.$ The following is commonly used as an α-level confidence interval for ${\widehat{\mu }}_{RE}:{\widehat{\mu }}_{RE}\pm z_{\alpha /2}\times \sqrt{V_{RE}},$ where z_α/2 is the standard normal critical value for the chosen significance level (α).

Alternatively, one can take a Bayesian approach and place prior distributions on μ_RE and τ as described in Section 3.2^28,34. Because we consider the fixed mean assumption in the common effect model inappropriate in most situations, we use a random effects model for the two model-based approaches presented below. However, one could easily alter these methods to reflect a common effect assumption.

3. Methods for estimating the reference range from a meta-analysis

In estimating a 95% normal reference range, we aim to find an interval that contains approximately 95% of individuals in the target population^35,36. Because the models described in Section 2 only allow inference on the pooled mean, we need additional methods and assumptions to estimate the 95% normal reference range for an individual. First, we must make an assumption about the distribution of the data within each study, assuming we do not have access to the study’s individual patient data (IPD). In this paper, we assume the individual-patient data in each study were generated from either a normal or log-normal distribution, with the family of distribution being consistent across all studies. We present three approaches to estimating the normal reference range: a frequentist approach, a fully Bayesian approach, and an empirical approach. The frequentist and Bayesian methods assume the within-study distributions have the same variance in all studies, while the empirical approach does not. We present each of our three proposed methods under normality, and then show how to apply them under a log-normality assumption.

3.1. A frequentist approach

Under the random effects model, if we assume observations within each study are normally distributed, the within-study variances are the same in all studies, and the study-specific means follow a normal distribution, then we have $y_{ij}\sim N({\mu }_{RE},{\sigma }_T^2),$ where ${\sigma }_T^2={\tau }^2+{\sigma }^2$ and σ² is the common within-study variance. We can estimate ${\widehat{\mu }}_{RE}$ and $\widehat{\tau }$ as described in Section 2, e.g., using REML, and estimate σ² as the unbiased pooled sample variance:

$${\widehat{\sigma }}^2=\frac{{\sum }_{i=1}^k\left(n_i-1\right)s_i^2}{{\sum }_{i=1}^k\left(n_i-1\right)}.$$ (3)

Substituting ${\widehat{\mu }}_{RE}$, ${\widehat{\tau }}^2$, and ${\widehat{\sigma }}^2$ into the marginal distribution of y_ij, the marginal distribution of individuals, marginal to studies, can be estimated as $N({\widehat{\mu }}_{RE},{\widehat{\sigma }}_T^2)$, where ${\widehat{\sigma }}_T^2={\widehat{\tau }}^2+{\widehat{\sigma }}^2$. The α/2 and 1 − α/2 percentiles of this distribution can then be taken as the bounds of the α-level normal reference range: ${\widehat{\mu }}_{RE}\pm z_{1-\alpha /2}\sqrt{{\widehat{\sigma }}^2+{\widehat{\tau }}^2}.$ Because ${\widehat{\sigma }}^2$ and ${\widehat{\tau }}^2$ were estimated, this suggests that a t-distribution may be appropriate instead of a normal. However, most meta-analyses will have a large enough total sample size across studies that the appropriate f-distribution will be closely approximated by a normal distribution. Alternatively, the fully Bayesian method presented in Section 3.2 accounts for the uncertainty in ${\widehat{\sigma }}^2$ and ${\widehat{\tau }}^2$.

3.2. A Bayesian approach

A fully Bayesian approach places prior distributions on μ_RE and τ. As in the frequentist approach, we assume that the true variances are the same in all studies, and now we use the normal-theory sampling distribution of the sample variance to capture uncertainty about the within-study variance σ², according to this model:

$$\begin{array}{r}{\overline{y}}_i\sim N\left({\theta }_i,{\sigma }^2/n_i\right)\\ {\theta }_i\sim N\left({\mu }_{RE},{\tau }^2\right)\\ \left(n_i-1\right)s_i^2\sim gamma\left(\frac{n_i-1}{2},\frac{1}{2{\sigma }^2}\right).\end{array}$$ (4)

We place a N(0,1000) prior on μ_RE and Unif(0,100) priors on τ and σ, then sample from the posterior predictive distribution for a new individual to incorporate into the normal reference range uncertainty about each of the parameter estimates:

$$y_{ne\omega }\sim N\left({\mu }_{RE},{\sigma }^2+{\tau }^2\right),$$ (5)

where the predictive density of y_new given the data {y_ij} is given by:

$$f\left(y_{ne\omega }|\left\{y_{ij}\right\}\right)=\int \int \int f\left(y_{ne\omega }|{\mu }_{RE},{\sigma }^2,{\tau }^2\right)f\left({\mu }_{RE},{\sigma }^2,{\tau }^2|\left\{y_{ij}\right\}\right)d{\mu }_{RE}d{\sigma }^{2}d{\tau }^2$$ (6)

The limits of the α-level normal reference range can then be estimated by the α/2 and 1 − α/2 percentiles of y_new’s predictive distribution.

3.3. An empirical approach

The third approach is a simple empirical approach that does not assume the studies all have the same within-study variances and does not specify the distribution of y_ij within each study. However, like the frequentist approach in Section 3.1, it does not account for estimation uncertainty and assumes the population captured across all studies follows a normal distribution. First, estimate the overall mean across all studies, weighted by study sample size:

$${\widehat{\mu }}_{emp}=\frac{{\sum }_{i=1}^N{n}_i{\overline{y}}_i}{{\sum }_{i=1}^N{n}_i}.$$ (7)

This is equivalent to the pooled mean in Laird and Mosteller’s¹⁹’s fixed effects model, weighted by sample size. Then estimate the marginal variance across studies using the conditional variance formula Var(Y) = E[Var(Y_ij|S = i)] + Var[E(Y_ij|S = i)]:

$${\widehat{\sigma }}_{T,emp}^2=\frac{{\sum }_{i=1}^N\left(n_i-1\right)s_i^2}{{\sum }_{i=1}^N\left(n_i-1\right)}+\frac{{\sum }_{i=1}^N\left(n_i-1\right){\left({\overline{y}}_i-\widehat{\mu }\right)}^2}{{\sum }_{i=1}^N\left(n_i-1\right)}$$ (8)

The limits of the α-level normal reference range are then given by the α/2 and (1 − α/2) percentiles of a $N({\widehat{\mu }}_{emp},{\widehat{\sigma }}_{T,emp}^2)$ distribution: ${\widehat{\mu }}_{emp}\pm z_{1-\alpha /2}\times {\widehat{\sigma }}_{T,emp}^2$. Conceição et al.⁷ used this method but weighted by n rather than n − 1 in the variance calculation. We prefer the unbiased estimate of the variance, but weighting by n will generally give similar results.

3.4. Lognormal distribution for y_ij

Each of the above methods can also be applied when each study’s observations are assumed to be drawn from a $\mathit{lognormal}\left({\theta }_i,{\sigma }_i^2\right)$ distribution, so that $\mathit{log}\left(y_{ij}\right)\sim N\left({\theta }_i,{\sigma }_i^2\right).$ In this case, first transform the observed study means and sample variances to the log scale using Equation (9) before estimating the reference range:

$$\begin{array}{r}{\overline{y}}_i^{*}=\log \left(\frac{{\overline{y}}_i}{\sqrt{1+s_i^2/{\overline{y}}_i^2}}\right)\\ s_i^{2*}=\log \left(1+\frac{s_i^2}{{\overline{y}}_i^2}\right)\end{array}.$$ (9)

This transformation uses the method of moments estimators for the location and scale parameters of the lognormal distribution. For more details, see the Appendix. The normal reference range can then be estimated as before, substituting ${\overline{y}}_i^{*}$ and $s_i^{*}$ for the observed study-level means and standard deviations. Finally, exponentiate the limits of the resulting range to give the normal reference range: $\left(e^{\widehat{\mu }-z_{1-\alpha /2}{\widehat{\sigma }}_T},e^{\widehat{\mu }+z_{1-\alpha /2}{\widehat{\sigma }}_T}\right).$ This method requires that the ${\overline{y}}_i^{*}$’s be normally distributed, an assumption that should be checked using a method such as a Q-Q plot. Depending on the distribution of ${\overline{y}}_i$, the distribution of the ${\overline{y}}_i^{*}$’s can be quite skewed.

4. Simulations

4.1. Methods of simulation

To assess how well each of the three methods captures a true 95% normal reference range, we conducted simulations under a variety of different conditions. In all conditions, we assumed the true distributions within studies were normal and that the true study-specific means varied according to the random effects model in Section 2. For each condition, we then considered different values of true between-study variation (τ²) as a proportion of total variability (τ² + σ²). In all conditions, each study had 50 subjects, with the total number of studies (N) being 5, 10, 20, or 30. The overall pooled mean (μ) was set to 8 and the true total variance (σ² + τ²) was 1.25 for all conditions. We conducted 1000 simulations for each condition. We considered scenarios where the true study-level variances were equal, as well as cases where they were not equal. For the frequentist approach, we used the R package metagen³⁷ to fit the random effects model using REML. For the Bayesian approach, we used JAGS version 4.3.0 with the packages rjags³⁸ and coda³⁹, in R version 3.6.0⁴⁰. We ran two chains each with 10,000 samples and after discarding 1,000 samples for burn-in.

4.1.1. Equal variances

For the equal variance scenario, we first generated the true study-level means (θ_i’s) according to a N(μ, τ²) distribution. For each study i, we then generated the individual-level data according to a N(θ_i, σ²) distribution, where σ² was constant across studies. We summarized the means and standard deviations for each study to give the observed summary data, fit each of the three models, frequentist, Bayesian, and empirical, then found the area under the probability density function of a N(μ, σ² + τ²) distribution between the upper and lower limits of the estimated 95% normal reference ranges.

4.1.2. Unequal variances

Data were simulated in the unequal variance scenario as in the equal variance scenario except that we generated σ_i, the true within-study standard deviation, from a doubly-truncated normal distribution, with both the left truncation point and mean equal to X and the right truncation point equal to X + 1, for X ranging from 0 to 0.64, with increments of 0.02. For each X, we estimated $E[{\sigma }_i^2]$ by simulating from the doubly-truncated normal distribution. These estimates ranged from 0.291 to 1.246. We let ${\tau }^2=1.25-\widehat{E}[{\sigma }_i^2]$ so as X increased, $\widehat{E}[{\sigma }_i^2]$ increased as a proportion of the total variance. Because we truncated the normal distribution, the variance of σ_i remained constant throughout all conditions. We approximated the true reference distribution for y_ij by simulating from the full conditional distributions: σ_i|X, θ_i|μ, σ_i, and y_ij|θ_i, σ_i, τ.

4.2. Simulation results

We first generated the data under the equal within-study variance scenario and measured the fraction of the true population distribution captured by each of the three reference range methods, which we call the “coverage” (Figure 1). For example, when the between study variance comprised 25% of the overall variance, the frequentist reference ranges based on 1000 simulated meta-analyses containing 30 studies captured a median of 94.9% of a N(μ, σ² + τ²) distribution, the true distribution of individual measurements. This can also be interpreted as the frequentist reference ranges excluding a median of 5.1% of extreme values. The observed median and variability of coverage depended on the true ratio of between-study variance (τ²) to total variance (τ² + σ²) and the number of studies N included in the meta-analysis. For the frequentist and empirical methods, the median coverage decreased as τ² increased as a fraction of the total variance; this decrease was most pronounced when the number of studies included was small (N = 5 or 10). In these conditions, the empirical method’s coverage decreased more quickly than the frequentist method’s. Also, variation in coverage increased as τ² increased and decreased as N increased. While the variation in coverage also increased with τ² for the Bayesian posterior predictive interval, this effect was less dramatic. In contrast with the frequentist and empirical methods, the Bayesian method’s median coverage increased with τ². This increase began for smaller τ² and was more extreme for small N. This increase in variation with τ² appears to reflect the additional estimation uncertainty when τ² is large, particularly when N is small. Unlike the frequentist and empirical methods, the Bayesian method accounts for posterior uncertainty about each parameter and thus appears more conservative. The results for the unequal within-study variances case were qualitatively similar to the equal variance case (Figure 2).

Figure 1: Simulation Results, Equal Variances.

Median, 2.5th percentile, and 97.5th percentile of the proportion of the true population distribution captured by the estimated 95% reference range, for different numbers N of studies. The horizontal axis is τ² as a proportion of the total variance.

Figure 2: Simulation Results, Unequal Variances.

Median, 2.5th percentile, and 97.5th percentile of the proportion of the true population distribution captured by the estimated 95% reference range, for different numbers N of studies. The horizontal axis is τ² as a proportion of the total variance.

5. Examples

5.1. Example 1: Pediatric Nighttime Sleep

Galland et al.⁹ sought to establish “reference values” for pediatric nighttime sleep outcomes measured by actigraphy, based on a systematic review and subsequent meta-analysis of 79 studies. We focus on the outcome wake after sleep onset (WASO) time in hours. The authors found 24 studies reporting WASO, with most participants belonging to the same age group (9–11 years) so they focused on the pooled mean across all age groups. In our review of these studies, one study included in this meta-analysis⁴¹ did not appear to actually report WASO but rather reported the average length of wake bouts. We excluded this study from our analysis, which therefore contained 23 studies. Figure 3 shows the pooled mean and corresponding standard errors. In this case, only one of the 95% confidence intervals for the study means overlapped with the point estimate of the pooled mean. Galland et al.⁹ explain that this variability reflects inconsistency across studies in how waking bouts were defined as well as a low specificity when using actigraphy to identify wakefulness. The authors also used meta-regression to investigate regional differences in sleep as a source of variation but did not observe a difference across study regions. This large variation in estimated WASO time across studies provides further evidence that the pooled mean may not provide a full picture of what constitutes a “normal” WASO time and that a reference range may be more useful. To better visualize the heterogeneity in WASO time within and across studies, we also present frequentist 95% prediction intervals based on a t-distribution for each study in the same figure^20,21. Because Galland et al.⁹ only reported study means and standard errors, we obtained the standard deviations directly from each study’s paper. When the paper did not report the standard deviation, we estimated the standard deviation using the standard error reported by Galland et al.⁹ and a normal approximation. Therefore, our results should be interpreted as merely an illustration of the proposed methods.

Figure 3: WASO.

Mean (95% CI) and 95% predictive interval for a new individual for each study, overall estimate of pooled mean (95% CI) based on REML , 95% predictive interval for a new study mean, and 95% reference ranges based on Bayesian, empirical, and frequentist methods.

We checked whether the study means deviated from normality using a QQ-plot (see Supplementary Materials); no apparent departure from normality is observed except a few points at the end of both tails. As in the simulations, we used the R package metagen³⁷ to fit the random effects model using REML. We also used this to estimate the pooled mean across studies and to obtain the prediction interval for a new study²⁸. For the Bayesian approach, we again used JAGS version 4.3.0 with the packages rjags³⁸ and coda³⁹, in R version 3.6.0⁴⁰. We ran two chains each with 50,000 samples and after discarding 5,000 samples for burn-in. Convergence was assessed using MCMC standard error and visual inspection of trace plots.

The estimated 95% normal reference ranges were (−0.47, 2.24), (−0.54, 2.32), and (−0.33, 2.34) for the frequentist, Bayesian, and empirical methods, respectively. We truncated these at zero because negative WASO values are meaningless, giving (0, 2.25), (0, 2.32), and (0, 2.34). Based on the frequentist result, we would expect about 95% of healthy children to have WASO time between 0 and 2.25 hours based on actigraphy. This reflects the large amount of variability between individuals included in the meta-analysis. Before truncation, the Bayesian reference range was widest, followed by the frequentist reference range; the empirical method gave the narrowest interval. This is consistent with simulation results and is likely due to the Bayesian method accounting for uncertainty about the parameters σ, τ, and μ. The code and results for both case study examples are included in the Supplementary Materials.

5.2. Example 2: Frontal SPV

Accurate perception of verticality is an important part of everyday functioning and can be altered in individuals such as “aged people, patients with vestibular disorders, Parkinson’s disease, idiopathic scoliosis, and stroke patients” ⁷. Accurate perception of verticality has also been associated with better functioning in patients following a stroke⁴². A person’s subjective postural vertical (SPV) can be measured by placing them in a tilting chair while blindfolded and asking them to tell an examiner how to adjust the chair so they perceive that they are in an upright position. Frontal and sagittal SPV refer to deviation (in degrees) of the specified position from true verticality in the frontal and sagittal planes. Because SPV measurements can be used to assess neurological functioning, it is important to establish a reference range in healthy persons.

In their meta-analysis, Conceigao et al.⁷ sought to establish reference ranges for frontal and sagittal SPV from 15 studies measuring frontal SPV and 5 studies measuring sagittal SPV. They estimated the reference range using the empirical approach except that they weighted by n rather than n − 1 in the variance calculation. We re-analyze the data for frontal SPV using REML to estimate the pooled mean across all studies and the 95% predictive interval for a new study. We then used the same methods as in Section 5.1 to estimate the three reference ranges.

We again checked for non-normality of the study-means using a Q-Q plot (see Supplementary Materials); no apparent departure from normality is observed except a few points at the upper tail. Figure 4 presents the reference range results as well as the estimated pooled mean and predictive interval for a new study. Conceicao et al.⁷ estimated the frontal SPV reference range as (−2.87°, 3.11°). The frequentist, Bayesian, and empirical methods gave estimated reference ranges (−2.92°, 3.15°), (−3.07°, 3.20°), and (−2.89°, 3.13°), respectively. As expected, the empirical method’s results are quite similar to those reported by Conceigao et al.⁷, while the frequentist and Bayesian methods give slightly wider intervals.

Figure 4: Frontal SPV.

Mean (95% CI) and 95% predictive interval for a new individual for each study, overall estimate of pooled mean (95% CI) based on REML , 95% predictive interval for a new study mean, and 95% reference ranges based on Bayesian, empirical, and methods.

6. Discussion

This paper proposes three methods of estimating reference ranges for an individual from a meta-analysis. The methods are simple to implement and can serve as a starting point for future development. Based on the simulations, all three methods tended to perform best when the number of studies was large and between-study variability was relatively small. However, while the frequentist and empirical methods tended to underestimate the width of the reference range as between-study heterogeneity increased (particularly for small N), the Bayesian posterior predictive interval did not. This is likely because the Bayesian method accounts for estimation uncertainty about the parameters, while the other methods do not. Instead, the posterior predictive interval more often overestimated the width of the interval. Depending on how the reference range is used, one might consider this behavior conservative. We recommend using caution when the number of studies is small, such as 5 or 10. If the number of studies is very small and the estimated between-study variation makes up more than 50% of total estimated variation, it may be more useful to report reference ranges specific to each study, rather than a pooled range.

As for which method might be most appropriate in which circumstance, the simulation results suggest that when the number of studies is large (at least 20), and the normality assumptions hold, the three methods will likely perform similarly. Conceição et al.⁷ used the empirical approach but weighted by n instead of n − 1. This suggests that the calculations required are intuitive and could easily be implemented by clinicians. The frequentist method, by contrast, requires familiarity with random effects models, although investigators are often interested in the pooled mean and thus likely to use this approach anyway. Finally, the Bayesian predictive interval requires familiarity with Bayesian methods and a software such as JAGS, though it is still a simple model to implement and can account for estimation uncertainty.

Each method makes distributional assumptions beyond those needed when estimating the pooled mean. Besides the usual assumption made when using likelihood methods to analyze random effects models — that the study-means are normally distributed — the frequentist and Bayesian approaches also assume a normal distribution for individuals within a study. While this may appear problematic, prediction intervals for a new observation based on a single study regularly impose this assumption⁴³. Unfortunately, if only study means and standard deviations are available, this assumption cannot be validated. Section 3.4 extended our approaches to allow individuals within studies to be lognormally distributed, but we caution that the transformed means on the log scale must be approximately normally distributed. This paper focuses on meta-analyses in which individual participant data (IPD) are not available; when IPD are available, non-parametric approaches using order statistics may be possible, as they are currently used in non-parametric estimates of reference ranges based on single studies^14,44.

Another key assumption of the frequentist and Bayesian methods is that the true within-study variances are the same in all studies and that any observed differences are due to sampling variability. Differences between studies in sampling methods or measurement techniques could render this assumption invalid. However, Section 4.2’s simulation results suggest that these methods may be robust to deviations from this assumption when the true study-specific standard deviations vary between studies according to a truncated normal distribution. Further work is needed to assess the models’ performance under other deviations from this assumption.

Finally, we reiterate that random effects models for estimating the pooled mean, on which we built the frequentist and Bayesian methods, require the study-specific means to be normally distributed. This is true of most random effects methods, except for the method of moments estimator developed by DerSimonian and Laird²⁹, which is known to underestimate between-study variability and therefore give results with inappropriately high precision^31–33. It is a common misconception that normality of the study means is guaranteed by the central limit theorem (CLT)²³. At best, the CLT only ensures that the sampling distribution of an observed average for a single study has an approximate normal distribution, not that the true means of the collected studies follow a normal distribution. One way of assessing departures from this assumption is the use of a normal Q-Q plot.

Although our methods make specific distributional assumptions, they do provide a starting point for additional development. Future work should generalize these methods to address instances where the assumptions used here are likely not met. This could involve cases with or without IPD. Section 4.2’s simulation results show that our approaches work well in cases with many studies and relatively low between-study variability. However, future studies should compare these methods with new methods incorporating IPD. Future methods should also improve performance when the number of studies is small or the between-study variation is large. The proposed methods may also be extended to cases where the data from each study are assumed to follow truncated normal distributions. Finally, these methods could be extended to a meta-regression setting to include characteristics such as age or sex, at either the individual or the study level.

9. Highlights.

What is already known?

• Reference ranges allow clinicians to determine whether an individual’s measurement falls within a normal range.

• Meta-analyses of normative data typically report the estimate of the pooled mean as a reference value. This only provides information on whether an individual might be above or below average.

• A reference range reflecting variability among individuals would be more useful in clinical practice, but the previous literature gives no guidance on how to estimate this in meta-analysis.

What is new?

• We present three methods for estimating the reference range for an individual from a meta-analysis. The first uses results from a frequentist method, such as REML, while the second is a Bayesian posterior predictive interval. The third is an empirical approach.

• Our simulation results suggest that these methods perform best when the number of studies is large and the between-study variability is relatively small. As the between-study heterogeneity increased, the frequentist and empirical methods tended to underestimate the width of the reference range, while the Bayesian posterior predictive interval did not.

• All three methods assume a normal distribution for individuals within a study. We also extended this approach to allow individuals within a study to follow a log-normal distribution.

• The frequentist and Bayesian methods both assume equal within-study variances.

• Our proposed methods provide a starting point for additional development. Future work can generalize these methods to cases where the distributional assumptions are likely not met.

Appendix A

A.1. Method of moments estimators for lognormal distribution

In (9), we use the method of moments estimators for the location and scale parameters of the lognormal distribution in order to transform the observed mean and variance to the log scale, where the observations would be normally distributed. Suppose Y = {y₁, …, y_n} ~ Lognormal(μ, σ²). Then the first two moments of the lognormal distribution are given by⁴⁵:

$$\begin{array}{c}E\left[Y\right]=e^{\mu +\frac{1}{2}{\sigma }^2}\\ E\left[Y^2\right]=e^{2\mu +2{\sigma }^2}.\end{array}$$ (A.10)

We can then set:

$$\begin{array}{r}{e}^{\mu +\frac{1}{2}{\sigma }^2}=\frac{1}{n}\sum _{j=1}^n{y}_i=\overline{y}\\ e^{2\mu +2{\sigma }^2}=\frac{1}{n}\sum _{j=1}^n{y}_j^2.\end{array}$$ (A.11)

Solving for μ and σ², we have:

$$\begin{array}{r}{\widehat{\mu }}_{MM}=\mathit{log}\left(\frac{{\overline{y}}^2}{\sqrt{{\overline{y}}^2+s^2}}\right)=\log \left(\frac{\overline{y}}{\sqrt{1+s^2/{\overline{y}}^2}}\right)\\ {\widehat{\sigma }}_{MM}^2=\mathit{log}\left(\frac{{\overline{y}}^2+s^2}{{\overline{y}}^2}\right)=\mathit{log}\left(1+\frac{s^2}{{\overline{y}}^2}\right),\end{array}$$ (A.12)

where $s^2=\frac{1}{n}{\sum }_{i=1}^n{\left(y_i-\overline{y}\right)}^2.$

Therefore for each study i in a meta-analyses, we can let:

$$\begin{array}{r}{\overline{y}}_i^{*}=\log \left(\frac{{\overline{y}}_j}{\sqrt{1+s_i^2/{\overline{y}}_i^2}}\right)\\ s_i^{2*}=\mathit{log}\left(1+\frac{s_i^2}{{\overline{y}}_i^2}\right)\end{array}.$$ (A.13)

We can then treat ${\overline{y}}_i^{*}$ and $s_i^{2*}$ as approximations of the sample mean and sample variance of the study on the log scale.