Controversies in Cardiovascular Research: Controversies in meta-analyses results on cardiac cell-based regenerative studies

Abstract

In contrast to multiple publication-based meta-analyses involving clinical cardiac regeneration therapy in patients with recent myocardial infarction, a recently published meta-analysis based on individual patient data (IPD) reported no effect of cell therapy on left ventricular function or clinical outcome. A comprehensive review of the data collection, statistics, and the overall principles of meta-analyses provides further clarification and explanation for this controversy. The advantages and pitfalls of different types of meta-analyses are reviewed here. Each meta-analysis approach has a place when pivotal clinical trials are lacking and sheds light on the magnitude of the treatment in a complex health care field.

The Meta-Analysis of Cell-based CaRdiac stUdiEs (ACCRUE) consortium recently published the results of the first prospective individual patient data (IPD)-based meta-analysis that included 12 randomized clinical trials investigating 1-year outcomes in patients with recent ST-segment elevation myocardial infarction (STEMI) who received intracoronary autologous stem and progenitor cells.¹ This IPD meta-analysis showed no effect of intracoronary autologous cell-based therapy on left ventricular function and clinical outcome in patients with recent STEMI. This contradicts some data from several small or medium-sized single and multicenter studies and multiple publication-based meta-analyses involving clinical cardiac regeneration in STEMI. Although de Jong et al. recently reported no difference between the cell-therapy and control groups if the left ventricular parameters were measured by cardiac magnetic resonance imaging (cMRI) and the clinical end points (adverse events) were not significantly different in most of the meta-analyses,² the major question is, why publication-based meta- analyses were in agreement, reporting beneficial effects of autologous reparative cell therapy in recent STEMI, but the IPD-based meta-analysis reported a negative outcome.¹ A thorough review of the data collection, statistics, and overall principles of meta-analyses may provide some answers and useful insights to interpret current findings.

Summary of aggregate and IPD meta-analyses of studies including patients with recent STEMI and intracoronary cell therapy

A summary of the meta-analyses of randomized trials in patients with recent STEMI receiving intracoronary cell-based therapy is provided in Table 1, showing the results of the efficacy outcome parameters.^1–21 The meta-analyses of clinical studies involving patients with chronic ischemic heart disease who received percutaneous intracoronary or intramyocardial or direct (during coronary artery bypass surgery) intramyocardial cell-based therapy are listed in Online Table I (Online Data Supplement).^S1–S5 The main results of the meta-analyses, including all cardiac studies with cell-based therapy irrespective of randomization, cell type, or delivery mode, are provided in Online Table II (Online Data Supplement).^{19–21, S6–S9} The first analysis of the ACCRUE database reported the outcome of patients with recent STEMI randomized to intracoronary cell-based vs. placebo therapy; therefore, this review concentrates on the discordance between the actual ACCRUE data and the data from the meta-analyses listed in Table 1.

Table 1.

List of meta-analyses of studies including patients with recent acute myocardial infarction and intracoronary cell-based regenerative therapy; efficacy in terms of left ventricular function and remodeling.

Meta-analyses on cardiac cell-based therapies	Year	Type of meta- analysis	Number of studies	Sample size	FUP (months)	EDV changes (mL)	ESV changes (mL)	EF changes (%)	if MRI, changes in EF (%)
Hristov3	2006	RCT-Pb	5	482	4–6	nr	nr	4.21*	nr
Lipinski4	2007	RCT-Pb	10	698	6	−4.6	−7.4*	3.0*	nr
Martin-Rendon5	2008	RCT-Pb	13	811	3–6	−2.47	−4.74*	2.99*	nr
Zhang6	2009	RCT-Pb	6	525	5	−0.15	n.a.	4.77*	nr
Zhang7	2009	RCT-Pb	7	660	6	−0.15	−0.25*	4.04*	nr
Bai8	2010	RCT-Pb	10	814	6	nr	nr	3.79*	nr
Kuswardhani9	2011	RCT-Pb	10	906	4–60	−3.08*	−5.52*	2.07*	nr
Takagi10	2011	RCT-Pb	15	877	nr	−0.18*	−0.35*	2.87*	nr
Clifford11	2012	RCT-Pb	33	1765	<12	−3.52*	−4.47*	2.87*	1.78*
Zimmet12	2012	RCT-Pb	29	1830	3–6	−3.39*	−3.51*	2.7*	nr
Delewi13	2012	RCT-Pb	16	1641	3–6	nr	nr	2.55*	0.16%*
Chen14	2013	RCT-Pb	5	510	nr	−2.29	−4.47	4.18*	nr
Jeong15	2013	RCT-Pb	17	1072	3–6	−3.46	−4.98*	2.51*	nr
de Jong2	2014	RCT-Pb	22	1513	6	−2.8	−4.05*	2.1*	0.13
Liu16	2014	RCT-Pb	8	262	6–24	0.69	−0.99	3.17*	nr
Gyöngyösi1	2015	RCT-IPD	12	1275	12	1.2	0.4	0.96	nr
Cong17	2015	RCT-Pb	17	1318	12	−1.69	−3.92*	2.74*	nr

^*p<0.05;

FUP indicates follow-up; EDV, end-diastolic volume; ESV, end-systolic volume; EF, ejection fraction; MRI, magnetic resonance imaging; RCT, randomized controlled trial; Pb, publication-based; IPD, individual patient data-based; nr, not reported.

Meta-analyses focusing on specific outcome, e.g., imaging or long-term outcome or including patients with chronic ischemic heart diseases or cohort studies are excluded.^18–21

Comments to Table 1, Bai et al.⁸ divided the Meluzin study into 2 parts, the BOOST study included twice (with the 6 and 18 months results); Kuswardhani et al.⁹, same studies with different FUPs or subgroups included as separate studies; Zimmet et al.¹² intracoronary cell therapy with/without granulocyte-colony stimulating factor pooled; Delewi et al.¹³, summary data of subgroups included.

Heterogeneous outcomes regarding left ventricular function and remodeling in meta-analyses

The cardiac efficacy endpoint of all meta-analyses was the change in ejection fraction (EF), and most of the meta-analyses also reported changes in end-diastolic (EDV) and end-systolic volumes (ESV). There have been heterogeneous outcomes regarding left ventricular function and remodeling, expressed as changes in EF, EDV, and ESV in meta-analyses (Table 1). Aggregate data meta-analyses suggested there was a benefit from cell-based therapy, based on significant increases in EF compared with controls; whereas IPD analysis did not support this. There was significant improvement in EF in 16 of the 17 meta-analyses of STEMI patients with cell therapy, and 10 of these analyses reported a significant decrease in ESV with a parallel significant decrease in EDV in 4 studies (Table 1). Decreases in ESV and EDV have particular importance in the assessment of left ventricular systolic function and remodeling. However, as the EF is derivated from the EDV and ESV, parallel changes in both these measures might lead to an unchanged or similar calculated EF value, which should be considered in left ventricular function outcome studies.

One of the three meta-analyses assessing EF changes by cMRI showed no significant difference between cell treated and control patients (Table 1).² The ACCRUE did not include a subgroup analysis of EF measurement by cMRI, as significantly (p<0.001) more patients in the cell-treated group were evaluated using cMRI than the controls, due to 2:1 randomization of some included studies (eg. REGENT, LateTIME, TIME, SWISS-AMI).^{1, 22–25} Cardiac MRI was used to quantify left ventricular function in 7 of 12 studies included in the ACCRUE analysis, and 6 of the 7 reported no difference between the cell-treatment and control groups regarding EF, EDV, and ESV (REGENT, LateTIME, TIME, SWISS-AMI, CADUCEUS, SCAMI).^22–27 Nevertheless, the methods for evaluation of left ventricular function and remodeling parameters were inconsistent between the studies included in the ACCRUE, as mentioned in the original paper.¹

In contrast to the IPD approach, clinical and statistical heterogeneity was greater (up to 92.2%)³ with aggregate data pooling in the standard meta-analytical approach, based on the calculation of mean difference in EF between the groups. According to the large variability between the studies included in the publication-based meta-analyses, a larger change in EDV (−4.6 mL) or ESV (−4.47 mL) was not significant, which was in contrast to the much smaller but significant change (EDV of −0.18 mL or ESV of −0.25 mL) reported in other publications.^4,7,10,14 Though intra-trial variability may provide an explanation, the observed effect sizes raises the question of clinical relevance for the change in EDV and ESV in these reported ranges (−4.16 to +1.2 mL and −7.4 to −0.25 mL, respectively).

An evaluation of the relationship between the sample size of the meta-analyses and the time sequence of publications demonstrates a narrowing difference between cell-based treatment and control patients regarding changes in EF (Fig. 1). The larger studies are more likely to be recent due to increasing numbers of individual studies being published. Larger meta-analyses not only included more patients, but also more studies. There are profoundly different methodological and interpretative challenges between a meta-analysis consisting of 2000 patients from two studies and one with 2000 subjects from 50 studies. This perspective is helpful when considering the relationship between effect size and sample size. Larger meta-analyses tend to show smaller but significant changes in effect sizes. Although the larger meta-analyses may be associated with a smaller change in EF on average, increasing the number of studies (with growing variability between the separate non-synchronized studies) in the larger meta-analyses may enhance the background noise, making it harder to discern the signal (effect) identified by smaller homogeneous studies.²⁸ Thus, both the number of studies and the sample size included in meta-analyses determine their importance.

Figure 1. Association between the number of patients (sample size) and weighted mean difference in the left ventricular ejection fraction (EF) between cell-based treatment and controls in meta-analyses of intracoronary cell therapy in patients with recent acute myocardial infarction.

A, Sample size (blue columns) and changes in EF from baseline to follow-up (differences between cell-treated and controls) (red columns). B, Correlation between sample size and mean difference in EF between cell-based treatment and control groups.

Mortality and adverse events paradox in meta-analyses

The clinical endpoint results of the meta-analyses are listed in Table 2. Apart from mortality, evaluation of clinical adverse events in publication-based meta-analyses is difficult, because the endpoints of the separate studies have heterogeneous definitions. Approximately one-third of trials included hospitalization as an adverse event as a part of the composite outcome measure, or some additional adverse outcome, such as implantation of an automatic implantable cardioverter defibrillator, which did not count as an “event” in other studies. Notwithstanding, publication-based meta-analyses pool studies with diverse primary endpoint definitions that report one clinical outcome as an “adverse event”, resulting in a high level of heterogeneity and inconsistency between trials. This approach also leads to contradictory conclusions of the effect of cell therapy on mortality or combined or separate adverse events, with significant effects in one meta-analysis but not in another (Table 2).

Table 2.

Different results of meta-analyses are reporting mortality or combined or separate adverse events.

Cell therapy for cardiac repair	Number of studies	Sample size	FUP (months)	Mortality	Combined Adverse events	Separate adverse events significant in cell- treated group
Hristov3	5	482	6	nr	nr
Lipinski4	10	698	6	non-signif.	non-signif.	re-AMI lower
Martin-Rendon5	13	811	3–6	non-signif.	non-signif.
Zhang6	6	525	5	nr	non-signif.
Zhang7	7	660	6	non-signif.	non-signif.
Bai8	10	814	6	nr	nr
Kuswardhani9	10	906	4–60	non-signif.	nr
Takagi10	15	877	nr	nr	nr
Clifford11	33	1765	< or > 12	signif.	non-signif.
Zimmet12	29	1830	3–6	non-signif.	non-signif.	TVR lower
Delewi13	16	1641	3–6	nr	nr
Chen14	5	510	nr	nr	non-signif.
Jeong15	17	1072	3–6	nr	nr
de Jong2	22	1513	6	non-signif.	non-signif.
Liu16	8	262	6–24	signif.	non-signif	TVR higher
Gyöngyösi1	12	1275	12	non-signif.	non-signif.
Cong17	17	1318	12	non-signif.	nr

FUP indicates follow-up; nr, not reported; non-signif, no significant benefit of cell therapy; signif., significant benefit of cell therapy; re-AMI, re-infarction; TVR, target vessel revascularization.

Gyöngyösi et al.¹ ACCRUE individual patient data-based meta-analysis in contrast with all other publication-based meta-analyses

It is not uncommon for different meta-analyses to have conflicting results. As with clinical trials that use different methodologies (e.g., continuous endpoints versus dichotomous endpoints, different follow-up periods or imaging modalities), meta-analyses commonly have differences in their endpoints, outcome assessments, and conclusions. As long as the collection of studies or the methodologies of analyses are different, we can anticipate that results from the meta-analyses will vary. A difference in results is more likely when the overall effect size, the primun movens of the meta-analyses is small with large confidence intervals. An exception is the only IPD meta-analysis of the ACCRUE, which included a large number of patients using pre-defined clinical events for all studies. This translated into highly consistent estimates with low or no heterogeneity for clinical outcomes analyses. In addition, IPD collection allows time-to-event data to be generated for estimating time-dependent event-free survival, which is completely impossible in a publication-based meta-analysis.

Another exception is the Cochrane database analyses, which pooled studies with the same definitions for adverse outcome, or measurement of EF at the same time point using the same imaging tool.^5,11 Although this method appears to be the statistically most correct with the highest quality among publication-based meta-analyses, such an approach leads to reduction in the number of patients, in particular in the subgroups. It also might be responsible for some surprising results, such as a lower long-term (12–60 months) restenosis rate (2.7% in the bone marrow mononuclear cell [BM-MNC] group in four studies) than short-term (<12 months) restenosis rate (11.3% in the BM-MNC group) or a higher incidence of target vessel revascularization (11.9% in the BM-MNC group in nine studies with <12 months follow-up and 14.3% in the BM-MNC group with long-term follow-up) than restenosis rate after cell therapy or control treatment.¹¹ Furthermore, the continuous updating of the Cochrane database to include more studies with more patients (similar to other meta-analyses) leads to some contrasting results compared with the previous version of the meta-analysis.^5,11 However, even if ACCRUE used pre-specified dichotomous parameters, the imaging modalities for measuring the continuous parameter are different, and may vary in any of the IPD-based databases, which is one major drawback of such data collection.

Heterogeneous statements on the results of subgroup analyses

One important but surprising finding of the ACCRUE IPD meta-analysis was the similar EF improvement in both cell-based treatment and controls in the subgroup of patients with low baseline EF. Some cell therapy trials^22,29,30 and meta-analyses^12–14 are in agreement that patients with lower EF benefit more from cell therapy than patients with higher baseline EF, in terms of a greater increase in the EF at follow-up. This is in contrast with the results of another study³¹ and meta-analyses.^1,4 Conflicting statements describing the effect of time (time interval between STEMI and cell therapy/randomization) and the number of injected cells increases the uncertainty about the effect of cell therapy (Table 3). Nevertheless, an effect of a parameter on outcome may not be strongly convincing if significant in one meta-analysis but not in another.

Table 3.

Results of subgroup analyses in different meta-analyses.

Cell therapy for cardiac repair	Number of studies	Sample size	Effect of baseline EF on increase in EF in cell-treated group	Effect of time from AMI to cell delivery on increase in EF in cell-treated group	Effect of number of delivered cells on increase in EF in cell-treated group
Hristov3	5	482	nr	nr	nr
Lipinski4	10	698	non-signif.	non-signif.	non-signif.
Martin-Rendon5	13	811	nr	signif.	signif.
Zhang6	6	525	nr	nr	nr
Zhang7	7	660	nr	signif.	nr
Bai8	10	814	nr	nr	nr
Kuswardhani9	10	906	nr	nr	nr
Takagi10	15	877	nr	nr	nr
Clifford11	33	1765	nr	signif.	signif.
Zimmet12	29	1830	signif.	signif.	signif.
Delewi13	16	1641	signif.	non-signif.	non-signif.
Chen14	5	510	signif.	nr	nr
Jeong15	17	1072	nr	nr	nr
de Jong2	22	1513	nr	nr	non-signif.
Liu16	8	262	nr	nr	nr
Gyöngyösi1	12	1275	non-signif.	non-signif.	non-signif.
Cong17	17	1318	nr	nr	nr

EF indicates ejection fraction; AMI, acute myocardial infarction; nr, not reported; non-signif, non-significant; signif; significant. Gyöngyösi et al.¹ ACCRUE individual patient data-based meta-analysis in contrast with all other publication-based meta-analyses

Furthermore, as the EF increases continuously during the first year after STEMI under standardized treatment, a lower EF measured very early after STEMI attack is a priori associated with a greater increase in EF (Fig. 2).³² Resolution of the myocardial stunning in the immediate post-infarct period can dramatically improve the left ventricular EF. Thus, the timing of when the baseline EF is measured both in cell-treated and control patients is critical for the correct interpretation of the EF improvement. Accordingly, patients in the placebo group with lower EF demonstrated a greater increase in the EF during the follow-up in the ACCRUE. The changes in EF in the subgroups of cell-treated patients with baseline EF <50%, <45% or <40% were 4.1±9.9%, 4.5±9.8%, or 5.0±9.7%, respectively; these values were similar to the controls classified into the same subgroup categories (3.5±9.0%, 3.8±9.0%, or 4.1±9.6%, respectively) in the ACCRUE.¹ Though patients with cell-based treatment with lower EF at baseline have a greater increase in the EF (Fig. 2), this is also true for the control group. Both randomization arms received the standard medical therapy, with additional cell-based treatment in one group. However, the magnitude of the difference between the groups, which may be attributed to the cell therapy itself, was not significant.

Figure 2. Increased ejection fraction (EF) under standard medical treatment for acute myocardial infarction (AMI).

Note that the lower the baseline EF at randomization/cell treatment, the higher the increase in EF at the 1-year follow-up. (Adapted from Engblom et al.)³²

The meta-analyses pooled studies that used different methods for measuring EF, except for the Cochrane database.^5,11 Measurement of left ventricular function using various methods results in various normal values and measurement errors. The methodological error of measuring left ventricular EF is 2.97–3.56% for MRI, and up to 7.4% for echocardiography, and the absolute value of EF is lower if measured by radionuclide imaging.^33–35 In addition, the difference in EF between cell-based treatment and controls is always lower when MRI is used versus echocardiography or contrast ventriculography.³⁶ However, it is unclear whether a mean difference of 0.9–4% in the EF between cell therapy and controls is greater than the methodological error and represents a real, clinically relevant improvement.

Bias of meta-analyses

Data collection bias of aggregate data vs. pre-specified outcome collection in IPD

One of the major pitfalls of meta-analyses is collection bias related to the inclusion of clinically heterogeneous studies. Publication-based meta-analyses can include all studies but due to non-unique definitions of the parameters, they have high, nearly inacceptable heterogeneity between trials. In contrast, the IPD-meta-analysis (i.e. ACCRUE) used pre-defined endpoints; therefore the heterogeneity and inconsistency is low, near 0% for adverse events. The IPD approach allowed analysis of the data according to the intention-to-treat principle. However, the IPD meta-analysis results depend strongly on the choice of included studies (which is based on the availability of IPDs) and international cooperation between investigators. Thus, the inclusion of a majority of negative trials without the possibility of balanced inclusion of positive trials results in a negative outcome. The ACCRUE included 12 studies with 104 patients/group, in contrast with the non-participating 19 studies with a mean of 46 patients/group.¹ Participating in ACCRUE required an agreement between the principal investigators of the study and the ACCRUE data committee regarding the aim and objectives of the ACCRUE, and involvement should be approved by the institutional scientific and publication policy and local ethical committee. Furthermore, due to different definitions used in the original study and the ACCRUE database, unavoidable differences between the published summary and IPD may develop in regards to terms and the interpretation of the result. This is a vulnerable target for international critics and a major reason for not participating in an IPD-based meta-analysis.

The IPD-based meta-analysis focuses on the most important parameters and keeping the database as simple as possible; as a result, drawbacks of this approach include the omission of some surrogate outcome parameters and a lack of data on subjective measures. Publication-based meta-analyses can evaluate all published parameters, such as different follow-up times,^2,3,6,12 injected cell volume,^5,11 infarct size,^5,11 bone marrow aspiration in the control group,¹⁵ different cell types,^5,11,16 details on cell preparation,¹⁷ quality of life scores,⁵ or any subjective or semi-objective parameter. Some of these data are evaluated even if they are only reported in a fraction of the collected trials. For example, myocardial infarct size decreased by 3.51% (p=0.004 n=240 patients) in the study of Martin-Rendon⁵ and by 1.9% (not significant n=670 patients) in the Clifford study if follow-up was <12 months¹¹, but the improvement was 3.36% when studies with follow-up ≥12 months were selected (p=0.0021, n=353 patients in the bone-marrow cell therapy groups compared with controls).¹¹ Evaluating changes in the infarct size is particularly difficult because most of the studies did not assess baseline infarct size. Similarly, heterogeneous results have been published regarding changes in the wall motion score index: not significant if follow-up was <12 months (−0.06, p=0.17, n=747 patients) but significant in another meta-analysis (−0.06, p=0.002, n=793 patients), and with longer follow-up (≥12 months; −0.12, p=0.0042 n=279 patients; thus the statistics may be contradictory when including different studies.).^11,17 The incidences of cardiac arrhythmias were similar in subgroups of cell-treated and control patients.^5,6 As mentioned in the ACCRUE paper, even if such data were collected in IPD-based meta-analyses, data available for <50% of the patients may not represent the whole collective; therefore these data were not assessed.

Using IPDs avoids data conflicts. Additionally, the IPD-based meta-analysis using original data is protected from publication errors or from duplicate publications due to diverse subgroup analyses. Yet, a major problem of IPD analysis remains the long and overwhelming effort needed for data collection and analysis if no financial support is available.

Analysis bias

Meta-analyses that rely on published literature may report some inaccuracies in summarizing the results, as different studies provide data in various formats. An example of this is the statistical calculation of the benefit of cardiac cell therapy required for forest plotting for each of the control and cell treatment groups that includes 1) sample size, 2) mean changes from baseline to follow-up, and 3) the standard deviation (SD) of these changes (Fig. 3). However, if a study does not publish the changes in EF, but instead only the baseline and follow-up EF, the recalculation of the mean change is flawed because the mean change should include individuals who have evaluation at both baseline and follow-up. Thus, although this re-calculation is based on widely accepted mathematical-statistical formulas,³⁷ it may result in a different mean±SD than the unpublished data available to the trial’s investigators. Approximately half of the cell-based cardiac regeneration studies do not report arithmetic means and SDs for the changes in continuous outcome parameters, such as EF. Consequently, “virtual” re-calculated means±SDs of some studies are used in different meta-analyses (Table 4),^{2,20,S10–S19} (References S10–S19 in the Online Data Supplement) which enter different numerical data for the same study into the forest plot statistics and decrease precision in their overall estimate. The SDs are pivotal for computing the statistical weights of the studies; therefore such an approach can lead to biased conclusions, not assigning a real weight to the study, and ultimately masking the true statistical effect (Fig. 3). Moreover, summary data are often presented for the entire study population, even if paired baseline-follow-up data are not available for all patients. In contrast to these approaches, the IPD-based meta-analysis (such as the ACCRUE) uses the original data, with the ultimate benefit of not needing virtual data calculation. Accordingly, no difference was found between the IPD-calculated changes in the means and SDs and the values of the original studies, if the study published these parameters.

Figure 3. Schematic of a forest plot statistic for calculation of the mean difference in a continuous parameter between the treated and control groups.

Note that a 10-fold larger study has less weight in the final results due to a larger standard deviation (SD).

Table 4.

Examples of reported changes (mean±standard deviations, SD) of left ventricular ejection fraction (EF) from baseline to follow-up (FUP) in the original publications (Roman style), compared with the re-calculated values in the publications of Jeevananthan et al.²⁰(Italic) and de Jong et al.² (underlined).

Intra-coronary cell injection study	Reported Changes in EF (%)	Reported Changes in EF (%)	Calculated Changes in EF (%)	Calculated Changes in EF (%)	Calculated Changes in EF (%)	Calculated Changes in EF (%)	Calculated Changes in EF (%)	Calculated Changes in EF (%)
	Original publication	Original publication	Jeevananthan et al. 201220	Jeevananthan et al. 201220	Jeevananthan et al. 201220	de Jong et al. 20142	de Jong et al 20142	de Jong et al 20142
	mean±SD	mean±SD	mean±SD	mean±SD		mean±SD	mean±SD
	Cell-treated	Controls	Cell-treated	Controls	Weight of study in forest plot	Cell- treated	Controls	Weight of study in forest plot
GeS10	4.8+	−1.9+	4.8±9.6	−1.9±5.9	1.3%	4.8±5.2	3.5±1.9	4.8%
PenickaS11	15.4+	20.5+	15.4±5.5	20.5±4.6	2.1%	6±5	8±4.8	4.4%
MeluzinS12	3±1 and 5±1++	2±1	4.0±4.7	2.0±4.7	2.6%	5±6.6	0±8.9	4.1%
NogueiraS13	nr	nr	6.9±6.2	2±11	0.9%	6.7±5.5	2±11.5	1.7%
PlewkaS14	10±9	5±8	9±7	5±3.6	2.5%	9±5.8	5±4.9	5.2%
CaoS15	nr	nr	11.5±3.2	7.9±3.4	2.9%	9.4±1.8	7.1±2.6	6.5%
YaoS16	7.2±1.6 and 11.7±2.6++	2.9±2	9.8±3.5	3.0±2.3	2.8%	6.2±2.4	2.2±1.8	6.3%
GrajekS17	nr	nr	−3.4±5.9	−6.4±7.9	1.9%	−2.5±5.6	0±7.8	4.0%
PiepoliS18	13.1±1.9	5.3±2	9.5±2.6	3.5±2.9	2.8%	8.4±9.2	2.2±12.6	2.5%
HirschS19	3.8±7.4 and 4.2±6.2++	4.0±5.8				3.8±7.4	5.2±5.8	5.7%

Note the disparities between the original publications and re-calculated values of both meta-analyses.

⁺SD of mean not reported,

⁺⁺two different cell-treatment arms;

nr, mean changes ±SD not reported (only baseline and FUP absolute values are reported).

Weight of the here selected (not all) studies from total 36 studies in the meta-analysis of Jeevanantham et al.²⁰and 22 studies in the meta-analysis of de Jong et al.²

Some differences between the meta-analysis results can also be attributed to errors in the individual publications and meta-analyses. However, the current review does not address the discrepancies found in the individual studies or meta-analyses. The aim of presentation of data in Table 4 is to show some examples of how different the numbers entered into the decisive forest plot statistics can be if they are recalculated.

Summary of the advantages and disadvantages of different types of meta-analyses of cell-based cardiac studies

Currently, several types of meta-analyses have been conducted to analyze the effect of cell therapy in ischemic heart disease (Table 5). There are different high-quality meta-analyses with specific aims and outcome measures, delivering different messages, and all have their advantages and disadvantages regarding the differences in analysis methodology, bias in data collection and study inclusion. All individual studies included ≤204 patients, so respective meta-analyses must pool small and medium-sized clinical trials with non-uniform designs, cell types, delivery modes, and follow-up times. Consequently, there is a lack of homogeneity. In addition, the trials used cell therapy at different times after the ischemic event and had diverse inclusion or exclusion criteria. Even if the collection of IPD for meta-analyses is regarded as the gold standard,³⁷ no meta- analysis approach can replace randomized multicenter blinded studies.

Table 5.

Types of meta-analyses in cell-based cardiac regeneration

Meta-analysis	Example	Type	Analysis type	Advantage	Disadvantage
Pooling all studies	Jeevananthan20	RCT-Pb	All studies analyzed as one collective	Large sample size	Large heterogeneity, Missing data recalculated
Pooling certain studies with predefined inclusion criteria	Clifford11	RCT-Pb	Different studies in subgroups	Large sample size, but lower number of patients in subgroups	Less heterogeneity, Missing data recalculated
Pooling certain studies with predefined inclusion criteria	Delewi13	RCT-Pb, summary of mean of subgroups	Different studies in subgroups	Large sample size, but lower number of patients in subgroups	Less heterogeneity, Missing data recalculated
Pooling certain studies with predefined inclusion criteria	Gyöngyösi1	RCT-IPD	All patients and studies analyzed as one collective	Low heterogeneity, Original data in database	Lower number of patients and studies

RCT indicates randomized controlled studies; Pb, publication-based; IPD, individual patient data-based.

Pitfalls of evidence-based medicine: negative outcome of a randomized clinical study based on positive meta-analysis results

It has already been acknowledged that the results of meta-analyses can differ from subsequent large randomized clinical trials.^38,39 The quality of publication-based meta-analyses depends on the quality, outcome assessment and statistical report of the included clinical studies. Due to the variable precision of meta-analyses, the observed effect could be overestimated. Positive meta-analysis results can pave the way to initiating a large randomized clinical study with a neutral or negative outcome, as has been observed several times in medical literature and practice.^38,39

Large randomized trials are considered the gold standard with the highest quality Level I evidence for application of the study results in clinical practice based on the evidence-based medicine grading system. Importantly, the pre-specified data collected in IPD-based meta-analyses (e.g. ACCRUE) allows the results to truly reflect the original data, as well as pool them in a database in similar form as clinical trial case reports. Thus, IPD collection may be considered a novel prospective multicenter large randomized clinical trial and the IPD meta-analyses as evidence-based medicine.

Conclusion

The IPD meta-analysis is currently considered the gold standard for meta-analyses assessing the impact of a treatment on clinical outcomes, especially in the case of small and medium-sized clinical cardiac regeneration studies. An IPD approach, such as the ACCRUE, permits data verification and allows adjustment for the same variables across studies. When dealing with cardiovascular outcomes, this approach generates time-to-event data for estimating survival, can explore heterogeneity at the patient level, and allows subgroup analyses. Using pre-specified terms and conditions, the database is similar to that of a prospective multicenter randomized clinical trial with similar statistical assessment modalities combined with standardized approaches to evaluating meta-analyses.

Although data based on individual patients rather than summary measures across patients are preferable, these data are commonly unavailable. The IPD analyses rely on the data being available from studies. The IPD database is kept simple; therefore, a meta-analysis cannot evaluate some surrogate parameters if data are not gathered or factors are not available, such as different quality of life assessment scores.

The difficulties of the standard meta-analysis approaches have been reviewed here. Each has a place in the analysis of data when pivotal clinical trials are not available and each sheds light on the magnitude of the treatment effect in a complex health care field.

Non-standard Abbreviations and Acronyms

ACCRUE

Meta-Analysis of Cell-based CaRdiac stUdiEs

BM-MNCs

Bone marrow mononuclear cells

EDV

End-diastolic volume

EF

Ejection fraction

ESV

End-systolic volume

IPD

Individual patient data

MRI

Magnetic resonance imaging

SD

Standard deviation

STEMI

ST-segment elevation myocardial infarction

Appendix

List of the ACCRUE investigators: Mariann Gyöngyösi, MD; Wojciech Wojakowski, MD; Patricia Lemarchand, MD; Ketil Lunde, MD; Michal Tendera, MD; Jozef Bartunek, MD; Eduardo Marban, MD; Birgit Assmus, MD; Timothy D. Henry, MD; Jay H. Traverse, MD; Lemuel A. Moyé, MD, PhD; Daniel Sürder, MD; Roberto Corti, MD; Heikki Huikuri, MD; Johanna Miettinen, PhD; Jochen Wöhrle, MD; Slobodan Obradovic, MD; Jérome Roncalli, MD; Konstantinos Malliaras, MD; Evgeny Pokushalov, MD; Alexander Romanov, MD; Jens Kastrup, MD; Martin W. Bergmann, MD; Douwe E. Atsma, MD; Axel Diederichsen, MD, PhD; Istvan Edes, MD; Imre Benedek, MD; Teodora Benedek, MD; Hristo Pejkov, MD; Noemi Nyolczas, MD; Noemi Pavo, MD, MSc; Jutta Bergler-Klein, MD; Imre J Pavo, MD; Christer Sylven, MD; Sergio Berti, MD; Eliano P. Navarese; Gerald Maurer, MD.

Subinvestigators: Rayyan Hemetsberger, MD; Dietmar Glogar, MD; Sasko Kedev, MD; Erik Jørgensen, MD; Yongzhong Wang, MD; and Rasmus S. Ripa, MD. CCTRN acknowledgements: Carl J. Pepine, MD; James T. Willerson, MD; David X.M. Zhao, MD; Stephen G. Ellis, MD; John R. Forder, PhD; R. David Anderson, MD, MS; Antonis K. Hatzopoulos, PhD; Marc S. Penn, MD, PhD; Emerson C. Perin, MD, PhD; Jeffrey Chambers, MD; Kenneth W. Baran, MD; Ganesh Raveendran, MD; Charles Lambert, MD, PhD; James D. Thomas, MD; Ray F. Ebert, PhD; and Robert D. Simari, PhD.