The most critical question when reading a meta-analysis report: Is it comparing apples with apples or apples with oranges?

Abstract

Objective:

While the number of meta-analyses published has increased recently, most of them have problems in the design, analysis, and/or presentation. An example of meta-analyses with a study selection bias is a meta-analysis of over 160,000 patients in 20 clinical trials, published in Eur Heart J in 2012 by van Vark, which concluded that the significant effect of renin-angiotensin-aldosterone system (RAAS) inhibition on all-cause mortality was limited to the class of angiotensin-converting enzyme inhibitors (ACEIs), whereas no mortality reduction could be demonstrated with angiotensin receptor blockers (ARBs). Here, we aimed to discuss how to select studies for a meta-analysis and to present our results of a re-analysis of the van Vark data.

Methods:

The data were re-analyzed in three steps: firstly, only ACEI/ARB-based studies (4 ACEI and 12 ARB studies) were included; secondly, placebo-controlled studies were excluded, and 10 studies left were analyzed; and thirdly, 2 studies that were retracted after the manuscript of van Vark had been published were excluded. The final analysis included 8 studies with ~65,000 patients (3 ACEI and 5 ARB studies).

Results:

The hazard ratios for all-cause mortality and cardiovascular mortality were 0.992 (95% CI 0.899-1.095; p=0.875) and 1.017 (0.932-1.110; p=0.703) for the ACEI versus control group and 1.007 (0.958-1.059; p=0.778) and 0.967 (0.911-1.025; p=0.258) for the ARB versus control group in the first step. The results were similar in the second and third steps.

Conclusion:

The studies to be included in meta-analyses, particularly comparing ACEIs and ARBs, should be chosen carefully.

Introduction

The number of meta-analyses published has increased rapidly in recent years. However, when these meta-analyses are reviewed critically, many of them have flaws in the design, analysis, and/or presentation (1-3). An example of a meta-analysis with a study selection bias is a meta-analysis by van Vark et al. (4).

Vark et al. (4) reported that the significant effect of renin-angiotensin-aldosterone system (RAAS) inhibition on all-cause mortality was limited to the class of angiotensin-converting enzyme inhibitors (ACEIs), whereas no mortality reduction could be demonstrated with angiotensin receptor blocker (ARB) treatment. This conclusion was based on a meta-analysis of data from 160,000 patients in 20 clinical trials, in which patients had been randomized to treatment with a RAAS inhibitor or control. Initially, the conclusions reached by the authors seemed correct, and the data were impressive. However, when the trials included in the meta-analysis were reviewed more closely, particularly the medications used in the experimental arms, it became clear that the trials included in the analysis were not all “apples” but were a mixture of “apples,” “oranges,” and “pears.”

This problem was originally recognized by Donzelli et al. (5), who wrote an open letter to Eur Heart J outlining their objections on the basis that selection bias had yielded mistakenly optimistic results for patients treated with ACEIs. In his letter, Donzelli claimed, correctly, that the positive effects of ACEIs on mortality could not be attributed to only ACEIs. Donzelli’s opinions were based on the fact that the patients in the ACEI arms of the studies that contributed most to the overall effects of ACEIs had not been treated with only ACEIs but were treated with combination therapies of ACEI plus diuretics or amlodipine (6-8).

Incorrectly designed meta-analyses cause misleading conclusions not only because of their original invalid results but also because they form the basis for further studies or papers. If we take the example above, although the validity of the meta-analysis by van Vark et al. (4) was questionable and Donzelli discussed the issues, the results of this meta-analysis were the backbone of a recent review on ACE inhibition and cardiovascular outcomes by Ferrari et al. (9). The main conclusion of this review-that ACEIs have beneficial effects on all-cause mortality and cardiovascular mortality but that ARBs do not have any effect-is, therefore, not legitimate, because it is based on an invalid analysis.

The van Vark meta-analysis is not unique in being open to criticism but is just another example of errors in design due to study selection bias. Therefore, we aimed to discuss the fundamental issue of how to select studies for a meta-analysis and to present our results of a re-analysis of the van Vark data (4).

Methods

This is a re-analysis of a previous meta-analysis based on the data of studies included in the meta-analysis (4). Since this is not an animal-or human-based study, there is no requirement for Ethics Committee approval.

The main concept of our approach was to increase the similarity and comparability of the ACEI studies and ARB studies included in the analysis with regards to the treatment administered in the ACEI/ARB arm and control arm. We re-analyzed the van Vark data in three steps (Fig. 1). In the first step, we intended to make the studies comparable with regards to the ACEI/ARB arms. In the second step, we intended to make the studies that were selected in the first step comparable with regards to the control arms. In the third step, we excluded two studies (KYOTO-HEART and JIKEI-HEART) that were retracted due to some concerns about the data to make the results more valid and updated.

Figure 1.

Scheme of study selection

ACEI - angiotensin-converting enzyme inhibitor; ARB - angiotensin receptor blocker

In the first step, we excluded studies in which an ACEI or ARB was administered in combination with other antihypertensive drugs (3 ACEI studies and 1 ARB study). Therefore, we included only ACEI/ARB-based studies (with treatment arms with “ACEI only” or “ARB only”) (4 ACEI and 12 ARB studies). The control arms in these studies, selected in the first step, were not comparable with regards to the proportion of patients who were administered placebo or active treatment. Therefore, in order to make the ACEI and ARB studies more comparable, we excluded the placebo-controlled studies (1 ACEI study and 5 ARB studies), leaving 10 studies with ~73,000 patients (3 ACEI and 7 ARB studies). Furthermore, we excluded 2 clinical studies that had been included in the van Vark meta-analysis (KYOTO-HEART and JIKEI-HEART) that were retracted due to some concerns about the data during the period between the publication of the meta-analysis by van Vark et al. (4) and the publication of the review by Ferrari et al. (9). The final analysis included 8 studies with ~65,000 patients (3 ACEI and 5 ARB studies). See Table 1 to examine the differences with regards to the proportion of hypertensive patients, the proportion of males, and the background mortality incidence rate between studies included in the final analysis and excluded due to the various reasons reported above.

Table 1.

Basic characteristics of studies included in the final analysis and excluded during the analysis steps

Study	n	RASB	Active drug	Control	HT%	Male%	Mortality incidence rate in control group (per 1000 patient-years)
Studies included to the final analysis (RASB-based studies – RASB only vs. Active Rx)
ALLHAT	33.357	ACEI	Lisinopril	Chlortalidone or amlodipine	100.0%	53.3%	28.5
ANBP-2	6.083	ACEI	Enalapril	HCTZ	100.0%	49.0%	17.1
JMIC-B	1.650	ACEI	ACEI	Nifedipine	100.0%	68.8%	6.2
CASE-J	4.703	ARB	Candesartan	Amlodipine	100.0%	55.2%	11.1
HIJ-CREATE	2.049	ARB	Candesartan	Non-ARB	100.0%	80.2%	14.3
IDNT*	1.146	ARB	Irbesartan	Amlodipine	100.0%	66.5%	54.0
MOSES	1.352	ARB	Eprosartan	Nitrendipine	100.0%	54.2%	31.0
VALUE	15.245	ARB	Valsartan	Amlodipine	100.0%	57.6%	24.8
Studies excluded from final analysis
Studies retracted
JIKEI HEART	3.081	ARB	Valsartan	Non-ARB	87.6%	66.3%	6.2
KYOTO HEART	3.031	ARB	Valsartan	Non-ARB	100.0%	57.0%	7.2
Studies with control arms with mixed or no treatment
Pilot HYVET	1.283	ACEI	Lisinopril	Diuretics or no treatment	100.0%	36.6%	55.4
IDNT	1.715	ARB	Irbesartan	Amlodipine or placebo	100.0%	66.5%	54.0
NAVIGATOR	9.306	ARB	Valsartan	Placebo	77.5%	49.4%	11.5
PRoFESS	20.332	ARB	Telmisartan	Placebo	74.0%	64.0%	29.1
RENAAL	1.513	ARB	Losartan	Placebo	96.5%	63.2%	66.0
SCOPE	4.937	ARB	Candesartan	Placebo	100.0%	35.5%	29.0
TRANSCEND	5.926	ARB	Telmisartan	Placebo	76.4%	57.0%	25.2
Not RASB-based studies (RASB was administered in combination with other drugs)
ADVANCE	11.140	ACEI	Perindopril with indapamide	Placebo	68.7%	57.5%	19.8
ASCOT-BPLA	19.257	ACEI	Amlodipine w/wo perindopril	Atenolol w/wo diuretics	100.0%	76.6%	15.5
HYVET	3.845	ACEI	Indapamide w/wo perindopril	Placebo	89.9%	39.5%	59.3
LIFE	9.193	ARB	Losartan w/wo HCTZ	Atenolol w/wo HCTZ	100.0%	46.0%	19.5

We based our analysis on the hazard ratios (HR) and 95% confidence intervals (CI) that were included in the analysis by van Vark. We also used a random-effects model to compute an overall pooled HR, as van Vark did. Statistical significance was defined as p values less than 0.05 (two-sided). We used Comprehensive Meta Analysis (CMA) v2.2 for Windows for data analysis.

Results

The studies that were analyzed by van Vark et al. (4) were not head-to-head comparisons of ACEI versus ARB studies, and the authors did not use the network meta-analysis method for indirect comparisons. When all 20 studies were included in the analysis, as van Vark did, the HRs for all RAAS inhibitors for all-cause mortality and cardiovascular mortality were 0.95 (95% CI 0.91-1.00; p=0.032) and 0.93 (95% CI 0.88- 0.99; p=0.018), respectively. Separate analyses repeated for ACEI and ARB studies showed that the apparent overall effect of RAAS inhibitors on all-cause mortality and cardiovascular mortality originated from only ACEIs, and ARBs did not have any effect on all-cause mortality and cardiovascular mortality.

In 3 out of 7 ACEI studies included in the van Vark analysis, ACEIs had been used in combination with another drug (Table 2). These studies (ADVANCE, ASCOT-BPLA, HYVET) were large-scale studies with a total of 34,242 patients (6-8). This constituted almost half of the total sample size of the 7 ACEI studies analyzed. In contrast, combination treatment with ARBs had been used in the ARB arm in only 1 of 13 ARB studies (LIFE) (10). The number of patients in this study was 9193 and so constituted only 11% of the total subjects. Since the clinical outcomes of the combination therapy studies could not be attributed solely to RAAS inhibitors, we excluded these 4 studies from the re-analysis.

Table 2.

Study design characteristics, problems, and actions regarding studies included in the meta-analysis

Study	Experimental arm	Control arm	Problem	Action
ACEI studies
ACEI-based studies
ALLHAT	Lisinopril	Chlorthalidone or amlodipine	-	-
ANBP-2	Enalapril	HCTZ	-	-
JMIC-B	ACEI	Nifedipine	-	-
Pilot HYVET	Lisinopril	Diuretics or no treatment	There are two control arms: diuretic and no-treatment arms	Excluded in the second step, since ACEI vs. diuretic data not reported
Studies with arms in which ACEI was administered in combination with other drugs
ADVANCE	Perindopril with indapamide	Placebo	Not an ACEI-based study	Excluded from the analysis
ASCOT-BPLA	Amlodipine w/wo perindopril	Atenolol w/wo diuretics	Not an ACEI-based study	Excluded from the analysis
HYVET	Indapamide w/wo perindopril	Placebo	Not an ACEI-based study	Excluded from the analysis
ARB studies
ARB-based studies
CASE-J	Candesartan	Amlodipine	-	-
HIJ-CREATE	Candesartan	Non-ARB	-	-
JIKEI HEART	Valsartan	Non-ARB		Excluded in the third step, since the publication was retracted due to concerns about data
KYOTO HEART	Valsartan	Non-ARB	-	Excluded in the third step, since the publication was retracted due to concerns about data
IDNT	Irbesartan	Amlodipine or placebo	There are two control arms: amlodipine and placebo arms	Only ARB vs amlodipine data included in the second step
MOSES	Eprosartan	Nitrendipine	-	-
NAVIGATOR	Valsartan	Placebo	-	Excluded in the second step
PRoFESS	Telmisartan	Placebo	-	Excluded in the second step
RENAAL	Losartan	Placebo	-	Excluded in the second step
SCOPE	Candesartan	Placebo	-	Excluded in the second step
TRANSCEND	Telmisartan	Placebo	-	Excluded in the second step
VALUE	Valsartan	Amlodipine	-	-
Study with arm in which ARB was administered in combination with other drugs
LIFE	Losartan w/wo HCTZ	Atenolol w/wo HCTZ	Not an ARB-based study	Excluded from the analysis

ACEIs - angiotensin-converting enzyme inhibitors; ARB - angiotensin receptor blocker; HCTZ - hydrochlorothiazide

In our re-analysis at this step, the HR for all RAAS inhibitors for all-cause mortality and cardiovascular mortality were 1.004 (95% CI 0.960-1.050; p=0.857) and 0.982 (95% CI 0.935-1.031; p=0.467), respectively. The separate HR corresponding to ACEIs and ARBs were also close to 1.00 (Table 3). When we further reviewed the studies accounting for the similarity of study design, it became apparent that the control arms in the ACEI and ARB studies were different. Patients in the control arms of 1 of the 4 ACEI studies (3% of the patients) were not administered antihypertensive therapy, whereas the control arms in 5 of 12 ARB studies (58% of the patient population) were treated with placebo.

Table 3.

The results of the original meta-analysis performed by van Vark et al. (4) compared with the repeated meta-analysis performed by us

Source	All-cause mortality				Cardiovascular mortality
	Number of studies	Number of patients	HR (95% CI)	P	Number of studies	Number of patients	HR (95% CI)	P
Overall
Original analysis (1)	20	158.998	0.95 (0.91-1.00)	0.032	16	149.715	0.93 (0.88-0.99)	0.018
First stepa	16	115.563	1.004 (0.960-1.050)	0.857	12	106.280	0.982 (0.935-1.031)	0.467
Second stepb	10	71.697	1.015 (0.953-1.081)	0.648	7	64.496	1.015 (0.940-1.095)	0.708
Third stepc	8	65.585	1.017 (0.954-1.085)	0.597	5	58.384	1.018 (0.943-1.099)	0.645
ACEI
Original analysis	7	76.615	0.90 (0.84-0.97)	0.004	7	76.615	0.88 (0.77-1.00)	0.051
First step	4	42.373	0.992 (0.899-1.095)	0.875	4	42.373	1.017 (0.932-1.110)	0.703
Second step	3	41.090	0.992 (0.897-1.098)	0.880	3	41.090	1.018 (0.931-1.112)	0.699
Third step	3	41.090	0.992 (0.897-1.098)	0.880	3	41.090	1.018 (0.931-1.112)	0.699
ARB
Original analysis	13	82.383	0.99 (0.94-1.04)	0.683	9	73.100	0.96 (0.90-1.01)	0.143
First step	12	73.190	1.007 (0.958-1.059)	0.778	8	63.907	0.967 (0.911-1.025)	0.258
Second step	7	30.607	1.030 (0.949-1.117)	0.480	4	23.406	1.006 (0.868-1.166)	0.934
Third step	5	24.495	1.035 (0.952-1.124)	0.419	2	17.294	1.020 (0.876-1.187)	0.802

^aACEI/ARB-based studies: ALLHAT, ANBP-2, CASE-J, HIJ-CREATE, IDNT, JIKEI HEART, JMIC-B, KYOTO HEART, MOSES, NAVIGATOR, pilot HYVET, PRoFESS, RENAAL, SCOPE, TRANSCEND, VALUE

^bACEI/ARB-based studies with control arms with active treatments: ALLHAT, ANBP-2, CASE-J, HIJ-CREATE, IDNT, JIKEI HEART, JMIC-B, KYOTO HEART, MOSES, VALUE

^cACEI/ARB-based studies with control arms with active treatments (JIKEI HEART and KYOTO HEART studies excluded): ALLHAT, ANBP-2, CASE-J, HIJ-CREATE, IDNT, JMIC-B, MOSES, VALUE

ACEI - angiotensin-converting enzyme inhibitors; ARB - angiotensin receptor blocker; CI - confidence interval; HR - hazard ratio

In the second step, HR and p values were similar to those calculated in the first step. In addition, separate HR corresponding to ACEIs and ARBs were very close to 1.00 (Table 3).

In addition, within the period between the publication of the meta-analysis by van Vark et al. (4) and the review by Ferrari and Boersma (9), 2 clinical studies that had been included in the van Vark meta-analysis (KYOTO-HEART and JIKEI-HEART) were retracted due to concerns over the data.

When we re-analyzed the data excluding these 2 studies, the calculated HR and p values showed ignorable changes in this step.

Discussion

The approach to selecting the appropriate studies for a meta-analysis is critical. The effects of a treatment on a specific clinical outcome can not be proven easily with a single randomized clinical trial (RCT). This is because of a low statistical power of analysis for non-primary parameters due to a sample size that is too small. A meta-analysis is a useful way to overcome this problem, because when the data from many RCTs are pooled into a single population, the sample size and, hence, statistical power increase (11). The pooled samples, however, should be as homogeneous as possible in order to make valid inferences. The main approach to avoid this very common problem is to build an objective and fair strategy to select studies with comparable study designs and populations. However, comparison of incomparable studies in a meta-analysis that leads to invalid results is a common problem in the literature. Many critics, even only in the field of cardiology, on these types of biased meta-analyses have been published (12, 13).

There are several methods that can be used to compare treatment A with treatment B in a meta-analysis. In order of decreasing quality, examples of these are head-to-head comparison, network meta-analysis, and inclusion of studies with similar study designs and populations (Table 4).

Table 4.

Methods used to compare treatment A with treatment B in meta-analysis

Comparison method	Specific clinical outcome of treatment arms in RCTs	Meta-analysis of RCTs
Head-to-head comparison	RCT1: Apple A is somewhat more delicious than apple B	Apple A is significantly more delicious than apple B
	RCT2: Apple A is somewhat more delicious than apple B
	RCT3: Apple A is somewhat more delicious than apple B
Network meta-analysis	RCT1: Apple A is more delicious than orange	Apple A is more delicious than apple B
	RCT2: Orange is more delicious than plum
	RCT3: Apple B is less delicious than plum
Including studies with similar study designs and populations	RCT1: Apple A is less delicious than orange	Apple A is less delicious than toffee apple B (not than apple B)
	RCT2: Toffee apple B is more delicious than orange

RCT - randomized controlled trial

The studies that were analyzed by van Vark et al. (4) were not head-to-head comparisons of ACEI versus ARB studies. This was the choice of the authors, although there have been many head-to-head comparisons of these two drugs (14-19). They also chose not to use the network meta-analysis method to make indirect comparisons. At this point, they should have eliminated studies from their analysis that had an inappropriate composition of the study arms, but they did not.

The results of our re-analysis indicate that RAAS inhibitors as a whole class do not have any significant effect on all-cause and cardiovascular mortality. In the first step of the analysis, the effects of ACEIs versus other antihypertensive treatments were compared with the effects of ARBs versus a mixed population of patients, half treated with placebo and half treated with antihypertensive drugs. These ACEI and ARB studies are, therefore, not comparable due to significantly different control arms. The results of the second step of analysis in which the placebo-controlled studies were excluded indicate that RAAS inhibitors as a whole class or individual ACEIs and ARBs do not have any significant effect on all-cause and cardiovascular mortality. Major randomized trials have controversial results on the cardiovascular protective effects of ACEIs and ARBs. In an extensive review of these trials by Ong (20), it was shown that the RAAS antagonists did not have special cardiovascular protective properties for hypertensive patients. In our meta-analysis, we included either active-controlled studies with a secondary analysis of ACEIs or placebo-controlled studies that mostly did not evaluate mortality as an end-point. In other words, the studies that were included in the meta-analysis were hypertension studies in which all-cause mortality and cardiovascular mortality were not primary end-points. Therefore, the studies included in the meta-analysis are actually not appropriate to assess the impact of RAAS blockage on cardiovascular events. Furthermore, the purpose of our re-analysis was not to show the impact of RAAS blockage on cardiovascular events but to evaluate whether there is a difference between ACEIs and ARBs on the basis of the data of previous studies on ACEIs and ARBs.

In this paper, we focused on the drawbacks that originated from the bias related to study selection and suggested that the conclusions drawn by van Vark et al. (4) would be quite different if they had included head-to-head comparisons of ACEI versus ARB studies in their studies. Indeed, in the present re-analysis of the van Vark data, we increased the similarity and comparability of ACEI studies and ARB studies included in the analysis and reached a different conclusion from the van Vark study. However, it should be noted that our re-analysis is also a metaanalysis, which has the common advantages and limitations of all meta-analyses.

We should also mention another erroneous approach that van Vark et al. (4) had performed. They claim that the effects of ACEIs on all-cause mortality were significantly better than ARBs based on their findings in which the p value corresponding to the HR value against the control group was less than 0.05 for ACEIs but higher than 0.05 for ARBs. Their conclusion might be simplified as such: “ACEI is better than the control group and ARB is not better than the control group; then, ACEIs are better than ARBs.” This conclusion, which naively seems to be correct, is not supported by the basic concepts of statistics. The 95% confidence limits of HR were 0.840 and 0.970 for ACEIs and 0.940 and 1.040 for ARBs. As seen, the 95% confidence intervals of ACEI and ARB intersect; therefore, it is not possible to claim that ACEIs are better than ARBs, whatever the p values are.

Another issue that should be taken into account-when to compare the drugs that did not entered to the market simultaneously, as in the example of ACEIs and ARBs-is the hidden dissimilarities between study populations. ARBs had been launched several years after ACEIs; therefore, patient populations in ARB trials had been probably treated quite better (more frequent statin use, more widespread use of innovative stents, etc.) than their counterparts in the ACEI trials. Better healthcare might probably decrease the incidence rates of clinical outcomes, even in placebo groups, in ARB trials. Thus, it might not be so easy to prove that the incidence rate in the ARB group is lower than the control group, in which the incidence rate is already quite low.

A faulty meta-analysis is very dangerous, since it spreads incorrect information that misleads other studies and reviews. For example, two recent reviews published in widely respected journals (9, 21) reached serious conclusions on the differentiating roles of ACEI and ARBs in reducing cardiovascular mortality based on the invalid meta-analysis by van Vark et al. (4).

Study limitations

The present study has the limitations that apply to all metaanalyses. Although a meta-analysis is the best way of summarizing vast amounts of RCT in the literature to produce a single estimate of the effect of a treatment, the disadvantages of metaanalyses should always be considered. The main limitation of meta-analyses is that the studies being combined are different-i.e., heterogeneity of studies. Other limitations common to all meta-analyses are publication bias (analysis of only published data) and lack of patient-based data. Although in the present study we aimed to prove the impact of the heterogeneity of the studies of a meta-analyses on the outcome, our analysis was also a meta-analysis itself, having all of the pitfalls of this type of analysis. On the other hand, this study draws attention itself to an important limitation of meta-analyses, which is the heterogeneity of the studies included in the analysis.

Conclusion

In conclusion, because the study selection strategy was incorrect and because the conclusion drawn about the difference between ACEIs and ARBs was not based on confidence intervals, as it should be, the results of the van Vark analysis are invalid, and it can not be concluded that ACEIs are more effective than ARBs. The studies to be included in meta-analyses comparing ACEIs and ARBs should be chosen critically, allowing for the fact that there are several head-to-head comparisons of ACEIs and ARBs and many ACEI and ARB studies with similar designs.