Variability of relative treatment effect among populations with low, moderate and high control group event rates: a meta-epidemiological study

Abstract

Background

The current practice in guideline development is to use the control group event rate (CR) as a surrogate of baseline risk and to assume portability of the relative treatment effect across populations with low, moderate and high baseline risk. We sought to emulate this practice in a very large sample of meta-analyses.

Methods

We retrieved data from all meta-analyses published in the Cochrane Database of Systematic Reviews (2003–2020) that evaluated a binary outcome, reported 2 × 2 data for each individual study and included at least 4 studies. We excluded studies with no events. We conducted meta-analyses with odds ratios and relative risks and performed subgroup analyses based on tertiles of CR. In sensitivity analyses, we evaluated the use of total event rate (TR) instead of CR and using quartiles instead of tertiles.

Results

The analysis included 2,531 systematic reviews (27,692 meta-analyses, 226,975 studies, 25,669,783 patients).The percentages of meta-analyses with statistically significant interaction (P < 0.05) based on CR tertile or quartile ranged 12–18% across various sensitivity analyses. This percentage increased as the number of studies or range of CR per meta-analysis increased, reflecting increased power of the subgroup test. The percentages of meta-analyses with statistically significant interaction (P < 0.05) with TR quantiles were lower than those with CR but remained higher than expected by chance.

Conclusion

This analysis suggests that when CR or TR are used as surrogates for baseline risk, relative treatment effects may not be portable across populations with varying baseline risks in many meta-analyses. Categroization of the continuous CR variable and not addressing measurement error limit inferences from such analyses and imply that CR is an undesirable source for baseline risk. Guideline developers and decision-makers should be provided with relative and absolute treatment effects that are conditioned on the baseline risk or derived from studies with similar baseline risk to their target populations.

Introduction

Practicing evidence-based medicine (EBM) requires applying the best available evidence to patients with unique individual risks. This paradigm has been applied following the principle that the relative treatment effect is portable (or constant) across populations with different baseline risks. Thus, if statins reduce the risk of myocardial infarction by 25% (i.e., RR 0.75), a group of patients with a baseline risk of 1% will end up after treatment with a risk of 0.75%; and another group with a baseline risk of 10% will end up with a risk of 7.5%. In both groups, the risks were reduced by 25%. This approach was advocated by Sacket [1] and reinforced in EBM curricula and evidence synthesis communities. Based on this, the Cochrane Collaboration and the Grading of Recommendations, Assessment and Development (GRADE) Working Group recommend pooling trials in meta-analysis to generate a relative effect (odds ratio [OR] or relative risk [RR]) that is subsequently applied across patients or populations at a specific baseline risk [2–5]. Some empirical work has supported this approach. Furukawa et al. concluded from 55 meta-analyses that OR and RR were reasonably constant across different baseline risks [6]. They did identify, however, examples in which this portability of the relative effect did not hold. In fact, many other investigations debated the portability of OR and RR [7–10].

In practice, the control group event rate (CR) has been used as a surrogate for the baseline risk despite many known limitations of CR, particularly its susceptibility to measurement error [11–14]. When developing a clinical practice guideline, the default option in guideline development software is to use CR in place of baseline risk [15], although external sources of baseline risk were encouraged [3]. Baseline risk can be inputted in guideline development software as 3 categories (low, moderate and high risk) so that each risk category ends up with its own absolute risk difference. Considering the current practice in which portability of relative effects across multiple groups with different baseline risks is assumed and the frequent use of CR to generate absolute treatment effects, we conducted this meta-epidemiological study. Our aim was to evaluate the heterogeneity of relative treatment effect across quantiles of CR, a proxy for baseline risk. We aimed to emulate the current practice in evidence synthesis and guideline development in which baseline risk is derived from CR, portability of the relative effect across CR is assumed, and the baseline risk is categorized as low, medium and high.

Materials and methods

Data source, acquisition and eligibility criteria

This study follows the reporting guideline for meta-epidemiological methodology research [16]. We identified all systematic reviews published in the Cochrane Database of Systematic Reviews between January 2003 and January 2020 and used the R package “RCurl” to repeatedly download data from the.rm5 files containing quantitative data of these reviews. We included meta-analyses that evaluate a binary outcome, reported 2 × 2 data for each individual study and included at least 4 studies. We excluded studies with no events (i.e., double zero studies).

Statistical analysis

We calculated CR from each individual study. Studies within each meta-analysis were categorized according to a CR quantile. Subgroup analysis was performed on each meta-analysis of relative effects using the quantile of CR as a univariate factor. The outcome of interest was the test for between-subgroup interaction based on heterogeneity statistic Q [17]. All analyses were done using R statistical software [18]. Meta-analyses were performed applying a restricted maximum likelihood tau squared estimator and using the R package “meta” and its subgroup analysis function using the “subgroup” argument. We calculated the percentages of meta-analyses that were statistically significant using a P value < 0.05.

Sensitivity analyses included: 1) using different association measures (OR and RR), 2) using different quantiles of CR (tertiles and quartiles), 3) analyzing only studies with control arms explicitly labeled as receiving a placebo, 4) restricting the analysis to one unique meta-analysis per systematic review, choosing the one with the largest number of included studies (if more than one had the largest number of studies, we chose the one reported first in the review), 5) assuming a P value < 0.10 (because most subgroup analyses are usually underpowered [19]) and also P value < 0.01, and 6) using the total event rate in a trial (TR) instead of CR. TR is the sum of events in both trial arms divided by the total sample size.

Results

The analysis included 2,531 systematic reviews (27,692 meta-analyses, 226,975 individual studies, mean of 8 studies per meta-analysis). The number of patients after deduplicating studies across meta-analyses was 25,669,783.

Subgroup analyses were done in each meta-analysis based on tertiles and quartiles of CR. The results of the main and sensitivity analyses testing various assumptions are summarized in Table 1. The percentages of meta-analyses with a statistically significant interaction based on CR quantiles at a P value < 0.05 was approximately 12%-18% with various sensitivity analyses. When using TR as the subgroup variable, the percentages of meta-analyses with a statistically significant interaction based on TR quantile at a P value < 0.05 was approximately 8%-13% across various sensitivity analyses. There were minimal variations in these percentages between OR and RR, and minimal variation between stratifying CR in tertiles or quartiles.

Table 1.

Proportions of meta-analyses with significant interaction with control group event rate quantiles

Effect measure		All studiesa			Placebo-controlled trialsb			Unique meta-analysis per systematic reviewc
	P-Value	< 0.10	< 0.05	< 0.01	< 0.10	< 0.05	< 0.01	< 0.10	< 0.05	< 0.01
Odds ratios	Tertile CR	19.46%	12.75%	5.33%	18.39%	11.82%	4.61%	24.19%	16.47%	7.45%
	Quartile CR	21.05%	14.24%	6.83%	19.99%	13.44%	6.06%	25.12%	18.00%	9.62%
	Tertile TR	12.84%	7.96%	2.94%	12.87%	7.89%	3.41%	13.99%	9.12%	3.01%
	Quartile TR	16.09%	10.63%	4.81%	16.50%	10.26%	4.65%	17.56%	11.54%	4.90%
Risk ratios	Tertile CR	19.48%	12.84%	5.74%	19.60%	12.64%	5.86%	24.11%	17.67%	8.17%
	Quartile CR	20.93%	14.18%	6.95%	20.77%	14.01%	6.92%	24.84%	17.95%	9.58%
	Tertile TR	14.06%	9.28%	4.03%	14.42%	9.40%	4.47%	16.64%	11.54%	4.94%
	Quartile TR	17.06%	11.55%	5.67%	17.42%	11.63%	5.79%	18.85%	13.18%	6.23%

CR Control group event rate, TR Total event rate of both groups in a trial

^a27,692 meta-analyses (226,975 individual studies), of which 27,388 meta-analyses had model convergence and provided estimates of effect

^b5,035 meta-analyses (34,891 individual studies), of which 4,991 meta-analyses had model convergence and provided estimates of effect

^c2,531 meta-analyses (25,173 individual studies), of which 2,498 meta-analyses had model convergence and provided estimates of effect

The proportion of significant interactions per meta-analysis increased as the number of studies per meta-analysis increased (Fig. 1) and as the width of the range of CR per meta-analysis increased (Fig. 2).

Fig. 1.

The proportion of significant interactions (P < 0.05) per meta-analysis using odds ratios are plotted vs. the number of studies per meta-analysis. Panel A analysis is based on tertiles of the control group event rate. Panel B analysis is based on quartiles of the control group event rate

Fig. 2.

The proportion of significant interactions (P < 0.05) per meta-analysis using odds ratios are plotted vs. the width of range of control group rate per meta-analysis. Panel A analysis is based on tertiles of the control group event rate. Panel B analysis is based on quartiles of the control group event rate

Discussion

This meta-epidemiological study demonstrates that in a substantial number of meta-analyses the relative association measures OR and RR significantly varied based on the tertile or quartile of CR. This was despite a subgroup analysis test that is known to be underpowered [19]. This finding has two possible interpretations. The first one implies lack of portability of the relative effect across various baseline risks in many meta-analyses. The other possibility is that this lack of portability occurs when CR or TR are used as surrogates for the true baseline risk. CR and TR suffer from measurement errors that can lead to regression dilution bias, and CR is structurally correlated with the relative effect, which can lead to mathematical coupling and a spurious association [11, 12]. These limitations can be mitigated by using complex hierarchical models that address measurement error. In a previous study, using such models on the same cohort of meta-analyses included in the current study showed that approximately 28% of meta-analyses demonstrate a significant association between the treatment effect and CR [14]. Thus, lack of portability of the relative effect across various baseline risks cannot be fully explained away by the limitations of CR and TR.

As expected, more significant interactions were noted with more studies included in a meta-analysis, and when the width of the CR range per meta-analysis increased, both observations reflected increased power of the subgroup analysis test. Using TR as the subgroup variable, as opposed to CR, was associated with smaller percentages of significant interactions. This is expected because TR is not structurally correlated with the relative effect. However, the percentages for both TR and CR analyses were higher than what would be expected by chance at all 3 significance levels used in this study.

Implications

Clinical decision-making depends on the absolute effects of treatment, such as the risk difference. The findings of this study suggest that we should not routinely derive the risk difference from OR and RR, which is the standard approach at the present time [20]. Rather, we should consider computing conditional treatment effects based on baseline risks using methods such as bivariate random-effects model [20–23]. A strong rationale for such a model particularly exists when portability of relative association measures is questioned and a sufficient number of studies (at least 6) is available for meta-analysis [5]. Guideline developers and decision-makers should be provided with relative and absolute treatment effects that are conditioned on the true baseline risk or derived from studies with similar baseline risk to their target populations. The correlation between the relative treatment effect and baseline risk is likely specific to certain interventions, diseases or contexts. For example, Furukawa et al. identified lack of portability with anti-arrhythmic drugs after myocardial infarction, carotid endarterectomy, and human immunodeficiency virus infection [6]. Ideally, individual participant meta-analysis should be used to evaluate effect modification caused by specific risk factors. In a study-level meta-analysis, conducting subgroup analyses based on context-specific prognostic risk factors is clearly favored over depending on CR as the subgroup variable [2].

In this study, we emulated the current practice in systematic reviews and guidelines in which [14] CR is used as a surrogate for BR. We also categorized BR into quantiles, which is also a common approach to guideline development in which patient groups are classified as low, medium and high risk. Categorization of CR into quantiles, as opposed to a continuous predictor in regression, may reduce the problem of measurement error that leads to these biases but can also lead to misclassification bias and loss of statistical power. The loss of statistical power due to categorizing a continuous variable suggests that our estimates are likely conservative and that interactions with CR may be more common than we found.

There are other inherent limitations to meta-epidemiological research that deals with a very large number of studies, such as the inability to evaluate more granular details and factors. We were limited to information that can be reliably extracted from.rm5 files used to store quantitative data in Cochrane systematic reviews. Ecological bias is another limitation of analyses in which inferences are made about individuals using population-level variables, such as the CR.

Conclusion

Analysis of a very large sample of studies suggests that when CR is used as a proxy for baseline risk, relative association measures are not portable across populations with varying baseline risk in many meta-analyses. Categroization of the continuous CR variable and not addressing measurement error limit inferences from such analyses and imply that CR is an undesirable source for baseline risk. Computing treatment effects that are conditional on the baseline risk can provide more accurate estimates for shared decision-making that can be tailored to populations with specific risks.

Authors’ contributions

M. Hassan Murad and Lifeng Lin conceived this study. M. Hassan Murad conducted the analysis. All authors (M. Hassan Murad, Zhen Wang, Mengli Xiao, Haitao Chu, Lifeng Lin) provided methodology and analytical interpretation. M. Hassan Murad and Lifeng Lin wrote the first draft. All authors (M. Hassan Murad, Zhen Wang, Mengli Xiao, Haitao Chu, Lifeng Lin) critically revised the manuscript and approved the final version.

Funding

No funding was used for this study.

Data availability

Data are freely available from the Cochrane Collaboration but can also be provided by the corresponding author upon reasonable request.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.