Preparing for a meta-analysis of rates: extracting effect sizes and standard errors from studies of count outcomes with person-time denominators

Abstract

Background

Formulas for the extraction of continuous and binary effect sizes that are entered into a meta-analysis are readily available. Only some formulas for the extraction of count outcomes have been presented previously. The purpose of this methodological article is to present formulas for extracting effect sizes and their standard errors for studies of count outcomes with person-time denominators.

Methods

Formulas for the calculation of the number of events in a study and the corresponding person time in which these events occurred are presented. These formulas are then used to estimate the relevant effect sizes and standard errors of interest. These effect sizes are rates, rate ratios and rate differences for a two-group comparison and rate ratios and rate differences for a difference-in-difference design.

Results

Two studies from the field of suicide prevention are used to demonstrate the extraction of the information required to estimate effect sizes and standard errors. In the first example, the rate ratio for a two-group comparison was 0.957 (standard error of the log rate ratio, 0.035), and the rate difference was −0.56 per 100,000 person years (standard error 0.44). In the second example, the rate ratio for a difference-in-difference analysis was 0.975 (standard error of the log rate ratio 0.036) and the rate difference was −0.30 per 100,000 person years (standard error 0.42).

Conclusions

The application of these formulas enables the calculation of effect sizes that may not have been presented in the original study. This reduces the need to exclude otherwise eligible studies from a meta-analysis, potentially reducing one source of bias.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Formulas are widely available to extract effect sizes for continuous and binary outcomes.

WHAT THIS STUDY ADDS

• This study shows how effect sizes for count outcomes (rates, rate ratios and rate differences) and their standard errors can be calculated by estimating the number of events and the person time. Once this information is available for one or more groups, then different effect sizes can be calculated.

HOW MIGHT THIS STUDY AFFECT RESEARCH, POLICY OR PRACTICE

• A potential source of bias with meta-analyses is the exclusion of eligible studies because the primary effect size of interest is not presented. The study shows how to extract information that enables the estimation of a variety of effect sizes, thus reducing a potential source of bias.

Introduction

Meta-analysis is commonly used to summarise the evidence on a topic. A key step in preparation for a meta-analysis is the extraction of effect sizes and their standard errors from eligible studies. These values are then entered into meta-analysis software either on their original scale or on a transformed scale to induce normality.¹

Formulas for the extraction of effect sizes and standard errors are readily available for studies of continuous outcomes (eg, mean differences, standardised mean differences) and binary outcomes (eg, odds ratios, risk ratios, risk differences).^{2 3} However, information on how to extract rates from studies of the number of events with person time denominators is scattered. These designs are common in studies of injury prevention (eg, studies of suicides, transport crashes and falls) where effect sizes are often presented as rates (eg, rate ratios, rate differences).

The purpose of this methodological article is to present in one place the main formulas for extracting rates, rate ratios and rate differences with their standard errors for two-group studies (eg, exposed and unexposed individuals) and difference-in-difference studies (eg, intervention and control arms). The article draws on two studies from the suicide prevention literature to demonstrate the use of these formulas.

Methods

Rate for single group studies

For a study of a single group, let $A$ be the count of the events of interest and $T$ be the person time. The estimated rate, ${\displaystyle \hat{R}}$, is defined as

$${\displaystyle \hat{R}=\frac{A}{T}.}$$ (1)

The natural log of the estimated rate, ${\displaystyle \log (\hat{R})}$, is assumed to be asymptotically normally distributed and is typically used for subsequent calculations, including meta-analysis. The log rate has a standard error that is defined as

$${\displaystyle \text{SE}[\log (\hat{R})]=\frac{1}{A^{1/2}}.}$$ (2)

The 95% confidence interval of the rate is therefore

$${\displaystyle \underset{\_}{R},\overline{R}=\exp (\log (\hat{R})\pm 1.96\times \mathrm{S}\mathrm{E}[\log (\hat{R})])}$$ (3)

where ${\displaystyle \underset{\_}{R}}$ and ${\displaystyle \overline{R}}$ are lower and upper limits. Note that the exponential transformation in equation 3 is used to return these limits to the rate scale. If a study presents the number of events and the rate, $A$ and ${\displaystyle \hat{R}}$, but not the person time, $T$, then the person time can be calculated using

$${\displaystyle T=\frac{A}{\hat{R}}.}$$ (4)

Similarly, if a study reports the 95% confidence interval for the rate but not its standard error, then ${\displaystyle \text{SE}[\log (\hat{R})]}$ can be calculated approximately using the rate and the lower limit of the rate:

$${\displaystyle \text{SE}[\log (\hat{R})]=\frac{\log (\hat{R})-\log (\underset{\_}{R})}{1.96}.}$$ (5)

Finally, ${\displaystyle \text{SE}[\log (\hat{R})]}$ can be used to approximate the number of events if $A$ is not reported as

$${\displaystyle A=(\frac{1}{\text{SE}[\log (\hat{R})]}{)}^2.}$$ (6)

Rate ratios for two-group comparison studies

Rate ratios are commonly reported and show the relative difference between the exposed and unexposed groups. The estimated rate ratio, ${\displaystyle \widehat{RR}}$, is defined as

$${\displaystyle \widehat{RR}=\frac{{\hat{R}}_1}{{\hat{R}}_0}=\frac{A_1/T_1}{A_0/T_0}}$$ (7)

where the subscripts refer to the exposed and unexposed groups (with 1 denoting the exposed group, 0 the unexposed). Like the rate, rate ratios and other ratio measures such as relative risks and odds ratios are typically analysed on the log scale as they are assumed to be asymptotically normally distributed. The log rate ratio, ${\displaystyle \log (\widehat{RR})}$, has standard error

$${\displaystyle \text{SE}[\log (\widehat{RR})]=(\frac{1}{A_1}+\frac{1}{A_0}{)}^{1/2}.}$$ (8)

The 95% confidence interval is then

$${\displaystyle \underset{\_}{RR},\overline{RR}=\exp (\log (\widehat{RR})\pm 1.96\times \text{SE}[\log (\widehat{RR})]).}$$ (9)

The exponential transformation in equation 9 returns the limits to the rate ratio scale. It is common for studies to present the rate ratio and its 95% confidence interval but not the standard error of the rate ratio. The standard error of the log rate ratio can be calculated approximately by

$${\displaystyle \text{SE}[\log (\widehat{RR})]=\frac{\log (\widehat{RR})-\log (\underset{\_}{RR})}{1.96}.}$$ (10)

Rate differences for two-group comparison studies

An alternative to the rate ratio is the rate difference.^{4 5} The rate difference, sometimes referred to as the risk difference, the excess risk and the attributable risk, is the difference in rates between two groups.⁴ This is estimated by

$${\displaystyle \widehat{RD}={\hat{R}}_1-{\hat{R}}_0=\frac{A_1}{T_1}-\frac{A_0}{T_0}.}$$ (11)

The rate difference has standard error

$${\displaystyle \text{SE}(\widehat{RD})=(\frac{A_1}{T_1^2}+\frac{A_0}{T_0^2}{)}^{1/2}.}$$ (12)

Note that, unlike the standard errors of rates and rate ratios, the standard error of the rate difference is not on the log scale. This is because ${\displaystyle \widehat{RD}}$ is assumed to be asymptotically normally distributed; therefore, no transformation is required for the calculation of confidence intervals or other related statistics prior to meta-analysis. The 95% confidence interval of the rate difference is

$${\displaystyle \underset{\_}{RD},\overline{RD}=\widehat{RD}\pm 1.96\times \text{SE}(\widehat{RD}).}$$ (13)

If the standard error is not presented, but the rate difference and its confidence interval are, then the standard error can be approximated by

$${\displaystyle \text{SE}(\widehat{RD})=\frac{\widehat{RD}-\underset{\_}{RD}}{1.96}.}$$ (14)

Rate ratios for difference-in-difference studies

The two-group comparison described above considers the difference in rates between an exposed and unexposed group. In injury prevention, an example of such a study is a comparison of the number of suicides before and after the installation of a safety barrier on a bridge. In the context of the difference-in-difference design, we refer to this comparison as the intervention arm. The difference-in-difference design extends this design by including an additional arm that acts as a control group, for example, the rates during the same before and after periods at a nearby bridge.⁶

The rate ratio for a difference-in-difference design, ${\displaystyle {\widehat{RR}}_{DID}}$, is defined as

$${\displaystyle {\widehat{RR}}_{DID}=\frac{{\widehat{RR}}_1}{{\widehat{RR}}_0}=\frac{A_{11}/T_{11}}{A_{10}/T_{10}}/\frac{A_{01}/T_{01}}{A_{00}/T_{00}}}$$ (15)

where ${\displaystyle {\widehat{RR}}_1}$ is the estimated rate ratio in the intervention arm and ${\displaystyle {\widehat{RR}}_0}$ is the estimated rate ratio in the control arm. The subscripts ${\displaystyle i=1,0}$ refer to the arm (1=intervention, 0=control) and where the subscript $j$ is used, $j=1,0$ refers to the level of exposure (1=exposed, 0=unexposed). Again, calculations are typically performed on the log scale. The log rate ratio, ${\displaystyle \log ({\widehat{RR}}_{DID})}$, has standard error

$${\displaystyle \text{SE}[\log ({\widehat{RR}}_{DID})]=(\frac{1}{A_{11}}+\frac{1}{A_{10}}+\frac{1}{A_{01}}+\frac{1}{A_{00}}{)}^{1/2}}$$ (16)

which leads to the following formula for the 95%

$${\displaystyle {\underset{\_}{RR}}_{DID},{\overline{RR}}_{DID}=\exp (\log ({\widehat{RR}}_{DID})\pm 1.96\times \text{SE}[\log ({\widehat{RR}}_{DID})]).}$$ (17)

If only ${\displaystyle {\widehat{RR}}_{DID}}$ and its 95% confidence interval are presented, then ${\displaystyle \text{SE}[\log ({\widehat{RR}}_{DID})]}$ can be approximated using the formula

$${\displaystyle \text{SE}[\log ({\widehat{RR}}_{DID})]=\frac{\log ({\widehat{RR}}_{DID})-\log ({\underset{\_}{RR}}_{DID})}{1.96}.}$$ (18)

Rate differences for difference-in-difference studies

Rate differences can also be estimated for a difference-in-difference design. Let ${\displaystyle {\widehat{RD}}_{DID}}$ be defined as

$${\displaystyle {\widehat{RD}}_{DID}={\widehat{RD}}_1-{\widehat{RD}}_0=(\frac{A_{11}}{T_{11}}-\frac{A_{10}}{T_{10}})-(\frac{A_{01}}{T_{01}}-\frac{A_{00}}{T_{00}})}$$ (19)

where the subscripts are defined as before. That is, the $i$ subscripts refer to the intervention and control groups (coded 1 and 0, respectively), and when the $j$ subscript is used, this refers to the exposure level (exposed and unexposed, coded 1 and 0). ${\displaystyle {\widehat{RD}}_{DID}}$ has standard error

$${\displaystyle \text{SE}({\widehat{RD}}_{DID})=(\frac{A_{11}}{T_{11}^2}+\frac{A_{10}}{T_{10}^2}+\frac{A_{01}}{T_{01}^2}+\frac{A_{00}}{T_{00}^2}{)}^{1/2}}$$ (20)

which leads to the lower and upper limits of the 95% confidence interval

$${\displaystyle {\underset{\_}{RD}}_{DID},{\overline{RD}}_{DID}={\widehat{RD}}_{DID}\pm 1.96\times \text{SE}({\widehat{RD}}_{DID}).}$$ (21)

Finally, the standard error can be approximated if only ${\displaystyle {\widehat{RD}}_{DID}}$ and its confidence interval are presented

$${\displaystyle \text{SE}({\widehat{RD}}_{DID})=\frac{{\widehat{RD}}_{DID}-{\underset{\_}{RD}}_{DID}}{1.96}.}$$ (22)

Results

This section illustrates the use of these formulas to extract the required information for a meta-analysis using the results from two studies. The first study, by Baran and Kropiwnicki,⁷ examines suicide rates in Sweden before and after the implementation of a national suicide prevention strategy in 2008. It is an example of a two-group comparison study. In the 6 years before the introduction of the strategy, ${\displaystyle {\hat{R}}_0}$ was 12.90 per 100,000 (95% CI 12.39 to 13.41). In the 6 years after, ${\displaystyle {\hat{R}}_1}$ was 12.34 per 100,000 (95% CI 11.69 to 12.99). The goal is to calculate the rate ratio and the log standard error, neither of which are presented in the original study. To do this, we require $A_1$ and $A_0$ as well as $T_1$ and $T_0$. This is done by first estimating the standard error of the log rates. In the period after the strategy was introduced, the standard error of the log rate (from equation 5) is

$${\displaystyle \text{SE}[\log ({\hat{R}}_1)]=\frac{\log (12.34)-\log (11.69)}{1.96}=0.028.}$$

The log standard error can then be used to estimate the number of suicides (equation 6).

$${\displaystyle A_1={\left(\frac{1}{0.028}\right)}^2=1276.}$$

That is, there were 1276 suicides in the period after the strategy was introduced. Now the person time can be estimated. From equation 4,

$${\displaystyle T_1=\frac{1276}{12.34}=103.4.}$$

$T_1$ is the person time divided by 100,000 because the rates are per 100,000 person-years. In its original metric, the data are 10,340,000 person-years. For the period before the strategy was introduced, the corresponding values are ${\displaystyle \text{SE}[\log ({\hat{R}}_0)]=0.021}$, $A_0=2268$ and $T_0=175.8$ (or 17,580,000 person-years). With these values calculated, it is now possible to estimate the rate ratio and the standard error of the log rate ratio using equations 7 and 8.

$${\displaystyle \widehat{RR}=\frac{1276/103.4}{2268/175.8}=\frac{12.34}{12.90}=0.957,}$$

$${\displaystyle \text{SE}[\log (\widehat{RR})]=(\frac{1}{1276}+\frac{1}{2268}{)}^{1/2}=0.035.}$$

The final step is to then transform the rate ratio onto the log scale. This is because the commonly used meta-analysis methods assume the study-specific estimates are normally distributed. The rate ratio is not normally distributed, but its log is approximately normally distributed. Thus, ${\displaystyle \log (\widehat{RR})=\log (0.957)=-0.044}$. The standard error is already on the log scale, so no further transformation is needed.

If the goal of the meta-analysis is instead to analyse rate differences, then these can easily be calculated from the derived values. Using equations 11 and 12,

$${\displaystyle \widehat{RD}=\frac{1276}{103.4}-\frac{2268}{175.8}=-0.56,}$$

$${\displaystyle \text{SE}(\widehat{RD})=(\frac{1276}{{103.4}^2}+\frac{2268}{{175.8}^2}{)}^{1/2}=\mathrm{0.44.}}$$

The rate difference is approximately normally distributed, and its standard error is on the same scale. Therefore, no further transformation of these values is needed to conduct a meta-analysis.

The second example uses data from a difference-in-difference study. Page and colleagues⁸ report on a national suicide prevention trial in Australia. The trial was implemented in 12 areas, and they compared population suicide rates before and after implementation. Comparable data were also available for control areas. The results of this study are summarised in table 1. Although the study already presents estimates of the rate ratio for a difference-in-difference analysis, the calculations are repeated here as a demonstration.

Table 1. Number of suicides and suicide rates from a difference-in-difference study by Page et al.

	Intervention		Control
	Pre-implementation	Post-implementation	Pre-implementation	Post-implementation
Number	$A_{10}=3363$	${\displaystyle A_{11}=1727}$	$A_{00}=7141$	$A_{01}=3821$
Rate	$R_{10}=12.1$	$R_{11}=11.9$	$R_{00}=10.8$	$R_{01}=10.9$

The study reports the number of suicides in the four groups defined by the arm (intervention and control) and exposure period (pre-implementation and post-implementation). The study also reports the corresponding rates in these groups$.$ The first step is to calculate the person time in each group, which is found using equation 4, as above. This gives $T_{11}=145.1$, $T_{10}=277.9$, $T_{01}=350.6$ and $T_{00}=661.2$ . It is now possible to calculate the rate ratio for the difference-in-difference, ${\displaystyle {\widehat{RR}}_{DID}}$, and the standard error of the log rate ratio, ${\displaystyle \text{SE}[\log ({\widehat{RR}}_{DID})]}$, using equations 15 and 16.

$${\displaystyle {\widehat{RR}}_{DID}=\frac{1727/145.1}{3363/277.9}/\frac{3821/350.6}{7141/661.2}=0.975,}$$

$${\displaystyle \text{SE}[\log ({\widehat{RR}}_{DID})]={\left(\frac{1}{1727}+\frac{1}{3363}+\frac{1}{3821}+\frac{1}{7141}\right)}^{1/2}=\mathrm{0.036.}}$$

${\displaystyle {\widehat{RR}}_{DID}=0.975}$ is approximately the same as the value reported in the original study, ${\displaystyle {\widehat{RR}}_{DID}=0.97}$. The standard error is similar (from equation 18).

$${\displaystyle \text{SE}[\log ({\widehat{RR}}_{DID})]=(\frac{\log (0.97)-\log (0.90)}{1.96})=0.038}$$

The difference between the two values is likely due to rounding error. As ${\displaystyle {\widehat{RR}}_{DID}}$ is not normally distributed, a log transformation is required prior to meta-analysis (ie, ${\displaystyle \log (0.975)=-0.025}$). No transformation is required for its standard error as it is already on the log scale. Finally, the rate difference for the difference-in-difference can also be calculated using equations 19 and 20.

$${\displaystyle {\widehat{RD}}_{DID}=(\frac{1727}{145.1}-\frac{3363}{277.9})-(\frac{3821}{350.6}-\frac{7141}{661.2})=-0.30,}$$

$${\displaystyle \text{SE}({\widehat{RD}}_{DID})=(\frac{1727}{{145.1}^2}+\frac{3363}{{277.9}^2}+\frac{3821}{{350.6}^2}+\frac{7141}{{661.2}^2}{)}^{1/2}=\mathrm{0.42.}}$$

The ${\displaystyle {\widehat{RD}}_{DID}}$ is assumed to be approximately normally distributed, so no transformation is needed to undertake a meta-analysis of this effect size.

Discussion

One of the challenging parts of undertaking a meta-analysis is extracting consistent effect sizes and their standard errors from eligible studies. In a field such as injury prevention, a set of eligible studies might include a mixture of rate ratios, rate differences and difference-in-difference rate ratios and rate differences. It is essential to be able to convert these different effect sizes onto the same metric for meta-analysis. While the conversion to a single metric is usually straightforward, the calculation of the appropriate standard error can be challenging because these are often not published in the original studies. This is an important problem. When researchers are unable to extract the required information, a study is often excluded from the meta-analysis. Frequent exclusion of otherwise eligible studies from a meta-analysis creates a potential risk of bias because not all the evidence is considered.

In this methodological article, the formulas for extracting key values for estimating commonly used effect sizes are presented. The basis for estimating the different effect sizes discussed here is the calculation of a rate for a single group. Since rates are calculated from the number of events, $A$, and the person time, $T$, one way to solve the problem is to find these values directly in the study or find ways of calculating these values. Equations 4, 5 and 6 are key to calculating $A$ and $T$ if this information is not presented. Once you have the $A$s and $T$s for each group, the calculation of the appropriate effect size and its standard error easily follows. For rate ratios for a two-group comparison, the key formulae are equations 7 and 8; for rate differences for a two-group comparison, see equations 11 and 12; for rate ratios for a difference-in-difference study, see equations 15 and 16; and for rate differences for a difference-in-difference study, see equations 19 and 20. Importantly, one of the best ways of avoiding errors when calculating estimates is to be as transparent as possible. Documenting the calculations and presenting these in a supplementary appendix to the meta-analysis is an excellent way of picking up any errors and showing the reader precisely how the calculations have been done (along with any assumptions made).

All the formulas presented here for standard errors assume the effect size is normally distributed, either on the log scale (for rates and rate ratios) or on the original scale (for rate differences). This assumption should hold when there are sufficient observations in each group but may not for small samples of rare events. Most formulas presented here will not work when there are counts of zero in one or more study groups because the log of zero is undefined. The solution to this problem is to enter the counts of events and the person time directly into a meta-analysis, modelling the rate ratios using a random effects Poisson regression model or a binomial-normal model. These approaches are covered elsewhere.^{1 9 10} Finally, all formulas are for unadjusted effect sizes. Depending on the study question, if the adjusted effect size and its confidence interval are presented, then a standard error can be calculated using equations 5, 10, 18 and 22. However, this relies on the effect sizes having been adjusted for similar covariates. If that is not the case, it may result in an apples-with-oranges comparison in the meta-analysis. In that vein, it is not good practice to undertake a meta-analysis of different effect sizes, for example, mixing rate ratios with odds ratios. These effect sizes are not the same, and odds ratios only approximate rate ratios when the outcome is a rare event.¹¹

In summary, the formulas reported here enable effect sizes for rates to be calculated when they may not have been included in the original manuscript. These formulas will be most useful when some eligible studies present information on one metric—for instance, rate ratios—and other studies on another—for instance, rate differences. Being able to convert them all to the same metric will increase the number of available studies in a meta-analysis and potentially reduce the risk of bias due to excluding otherwise eligible studies.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information.