Empirical comparisons of heterogeneity magnitudes of the risk difference, relative risk, and odds ratio

Introduction

In epidemiology and medical research, the choices of effect measures for binary outcomes have been long debated. Common choices include the risk difference (RD), relative risk (RR), and odds ratio (OR). The RD is often considered more heterogeneous than the ratio measures, RR and OR [1, 2]. Nevertheless, the arguments supporting this claim have been challenged [3]. For example, more rejections of homogeneity in hypothesis testing of RDs are expected than those of ORs. This article empirically compares the heterogeneity magnitudes between the RD, RR, and OR.

Methods

We applied heterogeneity measures to a large Cochrane database of meta-analyses [4]. The Cochrane Library publishes systematic reviews on a wide range of healthcare-related topics. We searched for all Cochrane reviews available online from issue 1 in 2003 to issue 1 in 2020. The search strategy for an older version of the Cochrane database was used in our earlier work [5–7]. In the Cochrane Library, each issue was published monthly, and it included systematic reviews on new topics with formal meta-analyses as well as protocols without formal analyses. An issue may also publish notices to withdraw outdated or flawed reviews and protocols. In this study, we iteratively included all published reviews that reported statistical data in each issue and excluded all withdrawn reviews. In total, we identified 64,929 meta-analyses.

In addition, a Cochrane review could investigate multiple disease outcomes and/or multiple intervention comparisons. Therefore, the meta-analyses within the review may not be independent due to the correlations between outcomes or intervention comparisons. For removing the impact of such potential correlations on heterogeneity, we also conducted sensitivity analyses, which were restricted to the meta-analyses with the largest number of studies from each Cochrane review. A total of 3125 meta-analyses were included in the sensitivity analyses.

We focused on the heterogeneity measure I ² and also considered the CV_B statistic as a supplemental measure. We reanalyzed each Cochrane meta-analysis and obtained the heterogeneity measures using each effect measure. The RR and OR were analyzed on the logarithmic scale. The I ² is widely used and is interpreted as a percentage of total variation due to heterogeneity rather than sampling error [4]. The CV_B is the between-study coefficient of variation used for providing further insight into heterogeneity magnitudes; it is calculated as the ratio of the between-study standard deviation τ over the absolute value of the overall effect size [8]. In this article, we estimated the between-study variance τ ² using both the DerSimonian–Laird (DL) and restricted maximum likelihood (REML) methods; the former is the most popular while the latter is recommended with better statistical performance.

Results

Figure 1 and Fig. S1 present the histograms of $\widehat{\tau }$ on a logarithmic scale for the RD, RR, and OR based on the REML and DL estimation methods. Because τ that truly equals 0 may not be exactly estimated as 0, depending on the tolerance of the REML algorithm’s convergence, the histograms in Fig. 1 shows small peaks at very small $\widehat{\tau }$ values. As the RD, RR, and OR are on different scales, the magnitudes of their corresponding $\widehat{\tau }$ may not be directly comparable. In general, the RR and OR led to $\widehat{\tau }$ < 0.01 in more meta-analyses than the RD (Table S1).

Fig. 1.

Histograms of between-study standard deviations on a logarithmic scale based on the restricted maximum likelihood method for the RD, RR, and OR. The histograms are restricted to the range from −10 to 2 for $log\widehat{\tau }$

Among the 64,929 Cochrane meta-analyses, 48.09% of RDs had I ² = 0% based on the DL method, while about 56% of RRs and ORs had I ² = 0%. The REML algorithm failed to converge in a few meta-analyses (≤ 0.22%) and I ² was not calculable; for the remainder, 43.56% of RDs had I ² = 0%, while about 50% of RRs and ORs had I ² = 0%. About 6% of RDs, RRs, and ORs had 0% <I ² ≤ 1%; their REML estimates of τ were very close, but not exactly equal, to 0. Fewer DL estimates (≤ 0.40%) led to 0% < I ² ≤ 1%, while the DL and REML methods produced similar numbers of meta-analyses with 0% ≤ I ² ≤ 1% (Table S2). In about 40% of meta-analyses, the RDs’ I ² were larger than the RRs’ or ORs’ by over 1%, while in about 10 to 15% of meta-analyses, the RDs’ I ² were smaller than the RRs’ and ORs’ by over 1% (Table S3). Based on the Q test, there were more meta-analyses (about 10%) with significant heterogeneity for RDs and non-significant heterogeneity for RRs or ORs than meta-analyses (about 1%) with non-significant for RDs and significant heterogeneity for RRs or ORs (Table S4). The RDs’ histogram was right-skewed, with a peak around I ² = 70%; the RRs’ and ORs’ histograms were less skewed, with peaks around I ² = 50% (Figs. 2A and S2). Table S5 presents the mean and quantiles of I ²; they were based on 23,966 meta-analyses with I ² > 0% for all three measures and both the DL and REML methods to avoid the impact of many I ² = 0%. The RDs’ descriptive statistics of I ² were noticeably larger than the RRs’ and ORs’.

Fig. 2.

Histograms of I ² (A) and CV_B on a logarithmic scale (B) based on the restricted maximum likelihood method for the RD, RR, and OR. A Restricted to I ² > 1%. B Restricted to the range from −10 to 10 for better visualizations

Categorized by the number of studies, the average study size, and the total number of events in a meta-analysis, RDs continued to have larger I ² than RRs and ORs in each category (Fig. S3). The I ² slightly decreased as the number of studies increased, consistent with previous findings [9]. It remained nearly unchanged as the average study size increased and noticeably increased as the total number of events increased.

Similar to the trends of I ², the histograms in Fig. 2B and S4 indicate that RDs generally had greater CV_B values than RRs and ORs. The conclusions regarding CV_B by categories of number of studies, average study size, and the total number of events in a meta-analysis were also consistent with those regarding I ² (Figure S5).

In sensitivity analyses using the 3125 meta-analyses with the largest number of studies from each review, the histograms’ overall trends were similar to those based on the complete datasets (Figs. S6 and S7).

Discussion

Our findings consistently supported that the RD seems more heterogeneous than the RR and OR. Yet, large uncertainties in I ² may confound these findings. The accuracy of I ² may also be questionable in meta-analyses with few studies and/or rare events [10]. In addition, I ² has several limitations; for example, it increases as sample sizes increase for the same τ ². The CV_B overcomes this drawback, while it is also subject to some disadvantages, as it increases rapidly for the overall effect size approaching 0. Nevertheless, they are arguably the appropriate tools with intuitive interpretations available in the current research synthesis literature to compare heterogeneity of measures across different scales. We intend our findings as supporting evidence rather than an assertion about heterogeneity magnitudes.

Authors’ contributions

YZ: methodology, software, formal analysis, writing - original draft, writing - review & editing, visualization. EHS: writing - review & editing. CX: writing - review & editing. HC: conceptualization, writing - review & editing, funding acquisition. LL: conceptualization, methodology, data curation, writing - original draft, writing - review & editing, project administration, funding acquisition. All authors read and approved the final manuscript.

Funding

This research was supported in part by the US National Institutes of Health/National Library of Medicine grant R01 LM012982 (LL and HC) and the National Institutes of Health/National Institute of Mental Health grant R03 MH128727 (LL). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Availability of data and materials

The datasets and code for this study are available upon reasonable request from the corresponding author.

Declarations

Competing interests

The authors declare that they have no competing interests.