Meta-analysis in applied ecology

Abstract

This overview examines research synthesis in applied ecology and conservation. Vote counting and pooling unweighted averages are widespread despite the superiority of syntheses based on weighted combination of effects. Such analyses allow exploration of methodological uncertainty in addition to consistency of effects across species, space and time, but exploring heterogeneity remains controversial. Meta-analyses are required to generalize in ecology, and to inform evidence-based decision-making, but the more sophisticated statistical techniques and registers of research used in other disciplines must be employed in ecology to fully realize their benefits.

1. What is meta-analysis?

Applied ecologists first used statistical methods to combine and summarize results from multiple studies 16 years ago (Fernandez-Duque & Vallegia 1994). Since then, approximately 220 analyses combining data from multiple studies have appeared (figure 1). Typically, applied ecologists are interested in the size of a difference or the strength of a relationship, relative to its variability, and how consistent this is across species sites or scales. Generation of such ‘effect sizes’ weighted in relation to their size (or inverse variance) provides effect magnitudes and intervals that convey more information than significance tests (Berger & Selke 1987). Although statisticians recommend that meta-analyses use weighted combination of effect sizes, approximately half the syntheses in applied ecology employ weaker designs such as vote counting, combining probabilities and pooling data across studies or species.

Figure 1.

Number of articles published in ISI journals in Ecology and Evolution judged as applied (without formal inclusion criteria) and labelled as meta-analyses (search to December 2008, updated 30 August 2009). An initial library of more than 500 analyses in ecology and evolution was compiled by Michael Jennions. Additional searches were undertaken and analyses without an applied component were discarded based on the judgement of the author.

The simplest method for synthesizing multiple independent studies is vote counting where the numbers of (statistically significant) positive and negative studies are summed (Light & Smith 1971). Leading meta-analysts deplore its use (Gurevitch & Hedges 1999) as it has low power (Hedges & Olkin 1980) and ignores sample size (Light & Smith 1971) and effect magnitude (Glass et al. 1981). Nevertheless, vote counting continues to be used in applied ecology (e.g. Davies & Pullin 2007). Combining probability values (or significance levels) from independent tests is a variant of vote counting with a long history (Fisher 1932) that also lingers beyond its sell-by date (e.g. Desouhant et al. 2003). Although these techniques may seem reasonable when study estimates are unavailable, biased results have been demonstrated from previous studies (Cooper & Hedges 1994) suggesting danger in this approach. When study estimates, but not variances, are available, investigators often synthesize unweighted averages of effect sizes (e.g. Parmesan & Yohe 2003; Georges & Fossette 2006). This approach is also suboptimal because it ignores the different amounts of information that studies of different sizes and different quality present.

2. The use of meta-analysis

Meta-analysis has multiple applications in applied ecology, but is particularly valuable for increasing power, exploring heterogeneity, identifying large-scale patterns and facilitating evidence-based decision-making. Such applications are not possible with methods such as vote counting.

Many ecological studies fail to detect changes because of their small sizes relative to the effects studied (Moller & Jennions 2002; Jennions & Moller 2003). Pooling effects across similar studies uses all available information and usefully increases the precision of estimation (e.g. Tonhasca & Byrne 1994), although more usually the value of meta-analysis lies in exploring variation across multiple studies. The existence of controversy is an often cited rationale for undertaking meta-analysis (Cooper & Hedges 1994) because relating substantial outcome heterogeneity to explanatory covariates may resolve controversy. Worm et al. (2002) used meta-analysis to explore the relative impact of productivity and disturbance on species diversity. Exploration of heterogeneity showed that positive effects of nutrient enrichment on diversity were only realized when consumers were present. The effects of productivity and disturbance on diversity depended on each other, and the direction of their effects and peak diversity shifted between sites of low and high productivity.

Exploring heterogeneity can also identify large-scale processes even when these are obscured by local factors, addressing the often intractable problem of scaling-up in ecology. The use of meta-analysis to identify systematic trends across diverse species and geographical regions is exemplified by analyses of coral decline (Gardner et al. 2003), elevated CO₂ effects on trees (Curtis & Wang 1998) and fish stock recruitment (Myers et al. 1999). Gardner et al. (2003) showed how declines in coral abundance varied with time, not space, suggesting that local causes operated in synchrony on a region-wide scale. Drivers of this decline are probably regional, hampering localized conservation effort. Curtis & Wang (1998) synthesized more than 500 effects of elevated CO₂ on woody plants and identified consistent responses across species for total biomass, regardless of growth conditions. This improved prediction of terrestrial feedbacks in the global carbon cycle and increased understanding of forest ecosystem functioning. In another application, fisheries models assumed that fish stocks could recover rapidly until a meta-analysis of more than 700 spawner–recruit relationships illustrated that most commercial marine fishes produce less than five viable young a year at low population sizes (Myers et al. 1999). This controversial finding challenged the beliefs of fisheries scientists that overexploited populations could rebound from depletion induced by fishing.

Meta-analysis also quantifies cumulative knowledge acquisition at the heart of evidence-based decision-making in medicine and the social sciences (Lipsey & Wilson 2001; Sutton & Higgins 2008). The ever-expanding volume of literature and the realization that the conclusions of many primary research studies may be misleading (Ioannidis 2005) imply that evidence-based decision-making requires critical evaluation and synthesis of multiple studies. Many scientific disciplines increasingly recognize that research and practice should be based on the totality of relevant and sound evidence and that meta-analysis is the appropriate tool to quantify the effects (Sutton & Higgins 2008). Meta-analysts in ecology have been aware of synergies with other disciplines for some time (Gurevitch & Hedges 1993, 1999; Osenberg et al. 1999), but calls for evidence-based approaches to conservation are more recent (Pullin & Knight 2001), with formal systematic reviews first published in 2005 (Stewart et al. 2005), and reviews accompanied by protocols later still (e.g. Stewart et al. 2007).

These needs-led syntheses are often controversial and challenge conventionally held notions about management effectiveness. For example, meta-analysis and systematic review show that predator control increases harvestable post-breeding populations but does not increase breeding bird population size, contrary to the prevalent management dogma (Côté & Sutherland 1997; Smith et al. in press). Likewise, river restoration techniques may not effectively increase fish population size (Stewart et al. 2009) and heathland management techniques may not achieve their objectives (Newton et al. 2009). Such meta-analyses also illustrate the danger of over-reliance on information from a limited range of studies or sites. Even when the same scientists apply the same treatments to the same species, different responses often ensue because of unique interactions between species and environmental factors (e.g. Stewart et al. 2008). Thus, estimates of associations from different situations in ecology deviate from each other in ways that systematically differ from chance. Meta-analysis offers an appealing mechanism for the exploration of this variation, simultaneously avoiding the danger of over-extrapolating from single context-dependent studies and allowing the consistency of (controversial) results from specific studies to be assessed in comparison to other studies.

3. Criticisms of meta-analysis

The criticisms levelled at ecological meta-analyses are common in other disciplines. I consider (and refute) each of the common criticisms in ecology.

It is a legitimate concern that ecological meta-analysis may overestimate effect size (Stewart et al. 2009) because of publication bias induced by failure to publish negative studies (Kotiaho & Tomkins 2002) or may be biased in favour of the prevalent paradigm (Koricheva 2003). However, the broad repeatable searches advocated by systematic review methodology minimize potential publication bias, albeit at the expense of introducing quality biases. Furthermore, by making such searches transparent, meta-analysis avoids the hidden publication bias probably present in other non-systematic syntheses.

The inclusion of heterogenic data has prompted much criticism of meta-analysis (e.g. Eysenck 1978; Markow & Clarke 1997). Ecological studies always differ and judgement is required about how similar they must be for pooled effects to be meaningful. Analysts make poor judgements and critics demand excessive homogeneity. Even where narrow sampling universes are defined, it is common for different comparators to be combined, particularly when effect sizes are based on correlations (e.g. Jones et al. 2008). Careful interpretation is required to avoid spuriously precise estimates of effect. Ecological meta-analysis inevitably involves synthesis of studies measured on different spatio-temporal scales, requiring a focus on exploration of heterogeneity in almost all cases. Only by exploring heterogeneity can consistency, and hence generalizability, be empirically assessed. Variation in sampling strategy and other stratified between-study variation is often a barrier to synthesis, but this can be resolved using hierarchical models, which explicitly model the non-independence in such data.

Another criticism focuses on the inclusion of ‘poor quality studies in syntheses'. However, meta-analysis of all available studies can identify the sensitivity of syntheses to methodological quality or specific characteristics through inclusion of covariates in a meta-regression. Curtis & Wang (1998) used this approach to quantify the difference between field and laboratory research. Other common quality criticisms emphasize the potential that effects may be confounded by unmeasured changes in the environment or that endpoints have been inappropriately measured (e.g. Hampton et al. 2005). Although often untestable, these criticisms may be determined to be unfounded by further analysis (e.g. Myers & Worm 2003). Nevertheless, it remains true that all syntheses are constrained by the quality of available data and the standards of reporting in primary research.

4. The future

Ecology will not advance by mere accumulation of data. New data must be placed in the context of existing knowledge and empirical studies. This context will vary across species, sites, space and time. Applied ecologists therefore have compelling and fundamental reasons for increasing their use of meta-analysis, but current ecological applications fail to take advantage of meta-analytic progress of other scientific disciplines and of cutting-edge statistical models to describe the unique structure in ecological data.

Establishment of a global register of environmental monitoring and primary research that requires submission of objectives and methods prior to the commencement of data collection would minimize publication bias. It would also encourage collaboration, standardization, and reduce unnecessary duplication, facilitating optimal use of research funds by funding agencies. Journal editors, research councils, government agencies and non-government organizations should encourage registration of projects and subsequent provision of data, as the medical community has done (Horton & Smith 1999; http://www.controlled-trials.com). Less radical (and less effective) approaches to addressing publication bias include improved information retrieval resources, improved reporting of primary research, increased dissemination of negative or confirmatory results and less reliance on p-values.

As well as improving the accessibility and quality of data on which synthesis relies, ecologists should increase the sophistication of their analytical techniques by employing modern methodologies developed in other disciplines. Ecological meta-analysts have recognized the benefit of using hierarchical Bayesian models to explore complex data (e.g. Myers 2001), but their full potential remains untapped. Both spatio-temporal modelling and missing data or misclassification tools can improve the analysis of heterogenic data (Ashby 2006). More flexible and innovative approaches are required to develop analytic strategies that integrate the large quantities of raw data available from surveys with summaries available in the literature or other sources (e.g. Sutton et al. in press). Use of multi-parameter evidence synthesis (Ades 2006) that compares effects from multiple interventions could prioritize conservation strategies, but ecologists have yet to bridge the gap between statistical analysis and decision-making to fully realize the benefit of an evidence-based paradigm. Hopefully ecologists will heed the lessons learned in other disciplines and by early pioneers of meta-analysis in ecology. The natural world is wonderfully varied. We will only appreciate the extent and importance of this variation if we explore it using the best tools available.