Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Aug 15.
Published in final edited form as: Circ Res. 2014 Jul 8;115(5):484–487. doi: 10.1161/CIRCRESAHA.114.304767

DAMASCENE and Meta-Ecological Research: A Bridge Too Far

Lem Moyé 1
PMCID: PMC4288029  NIHMSID: NIHMS648115  PMID: 25006188

Abstract

Population level studies commonly induce the ecological fallacy i.e., the belief that a relationship identified at the population level implies a cause and effect relationship at the level of the individual. A recent research report (DAMASCENE) that related clinical trial effect sizes in cardiovascular cell therapy trials to reporting discrepancies in the trials themselves has compounded meta-analysis difficulties with ecological fallacy issues. This dangerous combination has produced misleading conclusions and speculation that the research community should set aside.

Keywords: ecological fallacy, meta-analysis, reporting errors, cell therapy

Is the increasing use of smartphones causing strokes in young Americans?

The data tell us that the incident stroke rate have increased in the 20–54 age group from 26 up to 48 cases per 100,000 over a ten year period.1 During this time cell phone use, particularly smartphone use, has increased from near zero to over 50%.2 While there have been concerns about the relationship between electromagnetic emissions of portable phones and brain cancer for years, this is the first time that a temporal correlation has been identified between the newer smart phones and chronic disease. The implications have immense health and financial consequences.

The implications are also false. This is an example of a spurious relationship residing within a population based study, known as an ecological fallacy.

The ecological fallacy is the belief that one can take population level correlations and transfer them to cause and effect relationships within the individual.3 In this example, what may be driving stroke incidence in these younger populations is the rising rate of obesity and diabetes mellitus. On the other hand, smartphone purchases can be motivated by declining prices, the attractiveness of a sleek device, and the desire for social connectivity. The population level smartphone-stroke relationship does not represent what is occurring at the level of the individual where motivations for smartphone purchases and the drivers of cerebrovascular disease are quite different. To believe the population level relationship can be successfully transferred to the individual is the nature of the ecological fallacy (other examples of temporal ecological fallacies may be found at http://www.tylervigen.com).

In this artificial case, we can see through the diaphanous population level relationship to get to the heart of the matter because we have a solid a priori sense of the reasons that motivate smartphone purchases and the causes of strokes. This understanding permits us to hold the conclusory statement that “smartphone use causes stroke” at bay as we think through the greater complexity of the problem. We buffer and protect ourselves from the incorrect conclusion, while analyzing the complexities of the important underlying relationships.

However, in other situations, the findings may be either too sensational to resist drawing conclusions or we do not possess the intimate knowledge required to think through the background logic of the “relationship” with all of its subtleties. This shortcoming is cured by manuscript authors who warn us of alternative explanations for their ecological findings and by the peer reviewers and editors who insist that the readers be cautioned about the weak logic underlying the author’s putative results. The absence of these checks and balances permits misperceptions to reverberate throughout the research community. Moreover, the lack of research discipline generates the sense that physician-scientists are out of control and that our conclusions are not the product of a systematic approach, but rather the result of a wild and random search for significance (Figure; published by King Features Publishing in 1997).

Figure.

Figure

Cartoon by Jim Borgman © 1997 Cincinnati Enquirer. Reprinted with permission of UNIVERSAL UCLICK. All rights reserved.

The Return of the Ecological Study

Once belittled as noncontributory and fallacious, ecological studies have rebounded in recent years.4 Part of the reason is the recognition that there are in fact societal trends at the population level that can inform causal relationships e.g., the attraction of armed urban gangs to a teenager, or the impact of high unemployment rates on the job hunter’s psyche. However, another reason for the surge in ecological research is the ease with which the data is collected and analyzed. Studying clinical data at the individual level takes substantial time and effort as one writes a protocol, navigates complex IRB rules, ensures HIPAA compliance, collects data, and performs sophisticated analyses, to name but a few of the tasks. This is in contrast to the smartphone-stroke relationship which required only five minutes of internet research. The ease with which these population level analyses are generated can make it seem like we are wallowing in a swamp of loudly proclaimed but methodologically unsound investigations.

An interesting new adaptation of population level research is to combine it with meta- analyses. The classic meta-analysis collects an effect size (e.g., mean effect, odds ratio, or relative risk), reflecting the strength of association of a risk factor and a disease within each study, and then combines each of these study specific measures into one composite assessment. The result is a summary of the within-study findings. The new adaptation attempts to build a relationship across studies. In this situation, measures of the risk factor and the event are obtained from each study (whether they are related to each other within that study or not) and then a relationship is built across studies. Thus, correlations are generated from studies within which the relationship may not exist— the heart of the ecological fallacy. This combined meta-ecological analysis is problematic because the weaknesses of meta-analyses (e.g., publication bias, differing endpoint definitions/follow-up times across studies, and analysis complexities) and those of ecological designs (i.e., inferring that population level effects impact individuals) do not cancel but rather amplify each other. These combined meta-ecological designs have among the weakest of causal foundations.

An example of this more complex population level evaluation is the finding by Mostofsky et. al. in 2012 that related caffeine ingestion and heart failure (HF). The standard approach would have been to compute the relative risk reflecting the strength of association between caffeine ingestion and the occurrence of HF in each study and then to combine them into one summary measure. However, these investigators went further. Taking the caffeine level from each study, and the HF incidence from each study, they build a mathematical (regression model) relationship between HF and caffeine ingestion across studies, concluding that four daily servings of coffee offered protection against HF.

The public health and economic implications of these meta-ecological research results are profound. Yet, in a response by Palatini5 it was pointed out that it was the CYP1A2 polymorphism, an important regulator of caffeine metabolism, that was the likely true driver of this relationship. The author’s letter concluded that studying the average risk of coffee drinking without taking into account population genetics was misleading.

Ecological relationships all have the same motif. Powered by a provocative description, its hint of a new causal tie with powerful implications titillates. Also, by perhaps igniting the tinder of our internal biases and subjectivities, we permit ourselves to be pulled along the easy path to their conclusion’s acceptance. This is the gravity well of the ecological fallacy.

However we can resist its pull by keeping in mind that 1) embedding a population level relationship within a meta-analysis is a red flag, because it injects the strong assumption (commonly fallacious) that there is cause-effect relationship at the level of the individual, and 2) the validity of ecological research is determined not by the results’ sensationalism, but the conclusion of a thoughtful and deliberate review of the findings in the field.

Ecological Investigation in Research Discrepancies: The DAMASCENE Study

A prime example of the ecological fallacy is the DAMASCENE study that appeared in the April 29, 2014 edition of the British Medical Journal.6 Beginning with a standard meta-analysis approach, the investigators identified 49 randomized trials that reported treatment effects measured by the change in left ventricular ejection fraction (LVEF). However, they also added an ecological evaluation, reporting on a population level relationship between LVEF effect size and the number of reporting discrepancies or mistakes per trial. They observed that trials with larger than average effect sizes also had larger than average discrepancy reporting rates. The extension of the meta-analysis to include this ecologically derived LVEF-discrepancy rate relationship makes DAMASCENE a meta-ecological analysis.

Publications of related analyses, focusing on research manuscript retractions have already appeared, and a review of that work provides a useful perspective on DAMASCENE. For example, Nath et.al., reported that the number of manuscript retractions is related to the number of authors on the masthead7, an equally unusual finding. While one could conclude that journals should support leaner mastheads with fewer authors, more careful consideration suggests that studies with numerous authors represent more complex research efforts with many interwoven parts; these might be expected to be more prone to error and retraction.

Nath et. al. also pointed out that the greatest number of retractions occur in the journals Science, Proceedings of the National Academy of Sciences, and Nature. Are we to conclude that these premier journals are particularly susceptible to shoddy research representations? Nath and her coauthors fortunately drew the more helpful conclusion that detection bias was in operation. In this case, highly publicized articles receive greater scrutiny by more readers and thereby have more errors discovered. This phenomenon, the “bandwagon effect,” has been observed in new drug development. There, the early and sensational announcement of new and dangerous adverse effects excites healthcare providers, pharmacists, and patients to report more problems with the drug, leading to the appearance that the pharmacologic agent in question generates more adverse effects than equivalent drugs.8 Is this not a possible explanation, already available in the literature, for the DAMASCENE effect seen in the controversial field of cardiovascular cell therapy? Its authors are mute.

Another well-known factor that could help our understanding of DAMASCENE is the Proteus effect, or the observation that early results in a new field commonly overstate the true effect size which is itself moderated in subsequent research efforts.9 The Proteus effect impacted early work in hypertension10 and hyperlipidemia treatment11,12 ultimately having a negligible impact in these two fields that moved on to maturity. If in DAMASCENE, the large effects with their discrepancies were seen in early work, and were followed by research efforts with smaller effects, then the authors’ findings may simply be a manifestation of the Proteus phenomenon. DAMASCENE authors say nothing about this well accepted possibility.

At the other end of the spectrum, there are phenomena, already developed in the clinical trial literature that may be more likely explanations of small effect sizes than a miniscule number of reporting discrepancies. Pseudo-placebo dose, and a greater than anticipated number of patients lost to follow-up, are each well-established causes that reduce effect sizes in clinical trials. These explanations must certainly be considered in any attempt to explain small effect sizes, yet DAMASCENE remains mute on this.

Perhaps the principal explanation of the findings resides within the methodological difficulty embedded in DAMASCENE’s approach to develop the relationship between discrepancy rates and LVEF effect size. Specifically, the discrepancy rate and effect size was extracted from each study (without any evaluation of whether discrepancies were related to effect size within the study), and then assembled across the studies. To believe the relationship across studies reflects the relationship within these studies without examining the within study relationship is the heart of DAMASCENE’s ecological fallacy.

Fortunately, the DAMASCENE appendices do offer an opportunity to examine some of the intricacies of these reporting discrepancy determinations. The authors’ decision to include in their error count very small mistakes (e.g., that a cohort size once stated as 16 was in reality 15, or that 8 women and 2 men became 9 women and 1 male) is worthy of comment. Certainly, all researchers and research writers should work to minimize the occurrences of these flubs.

However, what are the implications of their occurrences? To believe that reporting the rates of small errors is worthwhile is to assume that these errors undermine the research’s results. However, although perfectly positioned to examine this critical point, DAMASCENE does not provide the necessary data dissection to support this thesis. Would the DAMASCENE authors have the community believe that the presence of miniscule discrepancies in a research effort provides prima facie invalidation of its conclusions?

In complicated research enterprises, it is not uncommon to have different investigators assigned to different tasks; those who describe the research may not themselves conduct it. Poor manuscript editing implies only that the editing requires improvement, not that the research result is false. Epidemiology teaches that flawed methodology (e.g., that of DAMASCENE), not small errors or typos, invalidates study conclusions. An approach that gives substance to small mistakes of this magnitude is not a lever to overthrow the large body of evidence developed from properly conducted meta-analyses.13,14,15

The DAMASCENE weaknesses aside, we cannot dismiss all ecological or population level modeling. Incorporating cultural connections and social influences can be illuminating. For example, it may be difficult to understand why adults, surrounded by all of the monitories about the dangers of smoking, continue to engage in this behavior without understanding the culture of stress and the influence of peer groups. Individual determinants of disease are not the sole determinants of disease16 and to believe so is to embrace the “individualistic fallacy.”4 Since societal factors also play a role in the manifestation of individual risk factors, both population level and individual level effects must be considered to fully understand the results of ecological studies.

Solutions to the Ecological Dilemma

We must carefully choose our path through the imbroglio of ecological research. Epidemiologists and their biostatistical colleagues are working to build research efforts that do not just incorporate population level inference or individual level effects but include both—what is aptly titled a multilevel model.17 However, these models are quite complex and for the time being, remain in the realm of the quantitative experts. It is helpful, however, to recognize that combining ecological relationship construction with meta-analyses leads to results that are perplexing at best and delusory at worst.

Additionally, the difference must be clear in our own minds between 1) a factor’s association with an effect (i.e., they merely occur together) and 2) and a causal relationship, in which the factor actually excites the production of the effect. Having this association-causation polarity in mind enables us to mark the location of the ecological finding on this grid, and its placement challenges our own thought processes about the relationship’s plausibility. It is an open question whether society, continually buffeted by heavy media tidal flows, can stop to think these relationships through. However, as scientists and professionals, we can think and are relied upon to do so.

Therefore, we must remember the phrase caveat emptor (buyer beware). It is essential we be both cognizant and disciplined, distancing ourselves from ecological research incitements that appear to call for action before thoughtful consideration of explanations and reproducibility. We must not let the new, loudly proclaimed relationship impact our understanding and core point of view about an issue until we have taken the time to think about the underlying ties between the variables that drive the relationship. And – like a car purchase – the sweeter the deal, the more wary the purchaser should be. Buffering ourselves from these provocative conclusions e.g., DAMASCENE enables us to respect the complexity of inflammatory issues, be informed about the multidimensional web of causality, and avoid overreacting to a mere one dimensional (and misleading) representation of the finding. Building an ecologically derived relationship on top of a meta-analytic approach in order to span the complicated problem of research error is going a bridge too far. We need more clinical trial data, not beguiling and misleading attempts to understand and undermine the data already collected.

Acknowledgments

DISCLOSURES

Dr. Moye’ receives funds from NHLBI for his involvement in the Cardiovascular Cell Therapy Research Network (CCTRN).

Footnotes

The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.

References

  • 1.Sultan S, Elkind MS. Stroke in young adults: on the rise? Neurology. 2012;79:1752–1753. doi: 10.1212/WNL.0b013e31827040d6. [DOI] [PubMed] [Google Scholar]
  • 2.Smith A. Smartphone Ownership 2013. Pew Research. http://www.pewinternet.org/2013/06/05/smartphone-ownership-2013/. Updated June 5, 2013. Accessed May 27, 2014.
  • 3.Greenland S, Robins J. Invited commentary: ecologic studies–biases, misconceptions, and counterexamples. Am J Epidemiol. 1994;139:747–760. doi: 10.1093/oxfordjournals.aje.a117069. [DOI] [PubMed] [Google Scholar]
  • 4.Pearce N. The ecological fallacy strikes back. J Epidemiol Community Health. 2000;54:326–327. doi: 10.1136/jech.54.5.326. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Palatini P. Letter by Palatini regarding article, “Habitual coffee consumption and risk of heart failure: a dose-response meta-analysis”. Circ Heart Fail. 2012;5:e98. doi: 10.1161/CIRCHEARTFAILURE.112.970111. author reply e99. [DOI] [PubMed] [Google Scholar]
  • 6.Nowbar AN, Mielewczik M, Karavassilis M, Dehbi HM, Shun-Shin MJ, Jones S, Howard JP, Cole GD, Francis DP, DAMASCENE writing group Discrepancies in autologous bone marrow stem cell trials and enhancement of ejection fraction (DAMASCENE): weighted regression and meta-analysis. BMJ. 2014;348:g2688. doi: 10.1136/bmj.g2688. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Nath SB, Marcus SC, Druss BG. Retractions in the research literature: misconduct or mistakes? Med J Aust. 2006;185:152–154. doi: 10.5694/j.1326-5377.2006.tb00504.x. [DOI] [PubMed] [Google Scholar]
  • 8.Edwards IR, Aronson JK. Adverse drug reactions: definitions, diagnosis, and management. Lancet. 2000;356:1255–1259. doi: 10.1016/S0140-6736(00)02799-9. [DOI] [PubMed] [Google Scholar]
  • 9.Ioannidis JP, Trikalinos TA. Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. J Clin Epidemiol. 2005;58:543–549. doi: 10.1016/j.jclinepi.2004.10.019. [DOI] [PubMed] [Google Scholar]
  • 10.Multiple risk factor intervention trial. Risk factor changes and mortality results. Multiple Risk Factor Intervention Trial Research Group. JAMA. 1982;248:1465–1477. [PubMed] [Google Scholar]
  • 11.The coronary primary prevention trial: design and implementation: the Lipid Research Clinics Program. J Chronic Dis. 1979;32:609–631. doi: 10.1016/0021-9681(79)90092-4. [DOI] [PubMed] [Google Scholar]
  • 12.The Lipid Research Clinics Coronary Primary Prevention Trial results. I. Reduction in incidence of coronary heart disease. JAMA. 1984;251:351–364. doi: 10.1001/jama.1984.03340270029025. [DOI] [PubMed] [Google Scholar]
  • 13.Jeevanantham V, Butler M, Saad A, Abdel-Latif A, Zuba-Surma EK, Dawn B. Adult Bone Marrow Cell Therapy Improves Survival and Induces Long-Term Improvement in Cardiac Parameters: A Systematic Review and Meta-Analysis. Circulation. 2012;126(5):551–68. doi: 10.1161/CIRCULATIONAHA.111.086074. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Fisher SA, Brunskill SJ, Doree C, Mathur A, Taggart DP, Martin-Rendon E. Stem cell therapy for chronic ischaemic heart disease and congestive heart failure. Cochrane Database Syst Rev. 2014;(4):CD007888. doi: 10.1002/14651858.CD007888.pub2. [DOI] [PubMed] [Google Scholar]
  • 15.Tian T, Chen B, Xiao Y, Yang K, Zhou X. Intramyocardial autologous bone marrow cell transplantation for ischemic heart disease: a systematic review and meta-analysis of randomized controlled trials. Atherosclerosis. 2014;233:485–492. doi: 10.1016/j.atherosclerosis.2014.01.027. [DOI] [PubMed] [Google Scholar]
  • 16.Bobak M, Marmot M. East-West mortality divide and its potential explanations: proposed research agenda. BMJ. 1996;312:421–425. doi: 10.1136/bmj.312.7028.421. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Blakely TA, Woodward AJ. Ecological effects in multi-level studies. J Epidemiol Community Health. 2000;54:367–374. doi: 10.1136/jech.54.5.367. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES