Drowning in Evidence: The Meta‐Analysis Boom and the Rigor It Demands

1. Introduction

In 2016, a provocative article titled: “The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses” was published [1]. Is this a problem for sports science and medicine too? A very recent meta‐research published in the Journal of Orthopaedic & Sports Physical Therapy found that out of 38 reviews on dry needling, only one was at low risk of bias, with 82% at high risk [2]. Sport‐related meta‐analyses now represent ~16% of all meta‐analyses in 2025 (Figure 1), an alarming figure for such a niche field. Although systematic reviews/meta‐analyses of randomized trials are considered at the top of the evidence pyramid [3], and frequently viewed as more conclusive than individual studies, they are research studies themselves and require rigorous methods to avoid bias, misleading advice, potential harm, and inefficient use of resources [4]. Similarly, meta‐analyses of physiological or biomechanical outcomes claiming causal mechanisms without adequate evidence may wrongly support hypotheses or build misleading theories.

FIGURE 1.

Growth of meta‐analyses in sport and exercise science relative to all fields, 2007–June 2026. Search queries (#1 vs. #1 AND #2): (#1) “meta analys*” [All Fields] OR “metaanalys*” [All Fields] OR “meta‐analysis” [All Fields], (#2) “sports” [All Fields] OR “exercise” [All Fields] OR “sports medicine” [All Fields] OR “athletic performance” [All Fields] OR “physical training” [All Fields] OR “resistance training” [All Fields] OR “rehabilitation” [All Fields] OR “sport*” [All Fields] OR “athlet*” [All Fields] OR “physical activity” [All Fields] OR “strength and conditioning” [All Fields] OR “endurance training” [All Fields] OR “aerobic training” [All Fields] OR “team sport*” [All Fields] OR “player*” [All Fields].

A large body of peer‐reviewed literature and online resources already covers how to conduct and report systematic reviews and meta‐analyses [5, 6, 7], including guidelines for complete reporting (e.g., PRISMA: https://www.prisma‐statement.org/) and tools showing the main sources of bias that threaten the validity of the review itself, regardless of the quality of the included studies (e.g., ROBIS) [7]. In this editorial, drawing on our experience as editors and in evidence synthesis, our aim is to flag the most common errors in meta‐analyses that are relatively simple to correct once authors know where to focus their attention.

To achieve our aim, we follow the main phases of the review process: identify, appraise, synthesize and interpretat. For each of these, we stress that appropriate expertise is necessary. Although some meta‐researchers suggest restricting systematic review authors to methodologists [8], we believe the author team should include experts in both methodology and the substantive content [9]. Artificial intelligence may improve the efficiency of some phases, but it cannot replace the critical judgment and oversight provided by human experts. Authors remain responsible for the rigorous application of appropriate methods and ultimately accountable for the final work, as recently emphasized by the major evidence synthesis organizations [10].

2. Identifying the Studies

A good study always starts from a clear and unambiguous research question. The research question is crucial to identify the:

1. Aim: Defines the aim of the review clearly.

2. Search Strategy: Each word in the question represents a concept. Each concept is mapped to synonyms or phrases, combined with Boolean logic. Using only PICO or PEO frameworks will be sub‐optimal because they relate more to inclusion and exclusion criteria, and may miss relevant concepts. Developing a proper search strategy is harder than it appears; working with an information specialist would help and reveal its complexity [11].

3. Proper risk of bias (RoB) tool: A clear question defines the estimand (target quantity to be estimated) of interest, which along with inclusion/exclusion criteria, determines the study designs that are eligible. Different RoB tools are used for different study designs (see below).

4. Screening and data extraction: two reviewers are usually recommended, and agreement between reviewers should be provided and discussed. When resources or time are limited (e.g., rapid review), one reviewer and a second person validating the response may be considered acceptable but needs to be justified [12].

3. Appraising the Studies

The RoB assessment is sometimes treated as a tick‐the‐box exercise. However, it is fundamental to interpreting the results correctly because it informs on how much we should trust the results. Some important aspects to consider:

• Quality of reporting in original studies: When studies do not include sufficient detail (e.g., due to word limits), systematic reviewers should spend the effort to obtain the information through study pre‐registration documentation (always recommended) when available, or request clarification/ethics submissions (which should contain detailed study protocol information) from the authors.

• Assessing the RoB. Review authors should not confuse quality assessment with RoB assessment. The former only indicates how many “safeguards” a study implemented, and does not appraise whether bias threatens the validity of the estimate [13]. Only after selecting the estimand of interest can an appropriate RoB tool be selected. Biases are directly related to whether the objective is descriptive, predictive, or causal (Table 1). Using a tool developed for a different objective because it is easier or shorter will likely produce inappropriate conclusions. RoB tools should address all potential sources of bias relevant to the estimand and the design. They require strong knowledge of the instruments, sufficient methodological expertise, and sufficient content knowledge to understand nuances for each domain to be rated appropriately. For example, a reviewer should be able to understand the difference between the effect of assignment versus effect of treatment [14], or that confounding is a causal concept that should not be evaluated in descriptive and prognostic studies unless the study addresses counterfactual prediction [15].

TABLE 1.

Common errors at each phase of a systematic review with meta‐analysis (illustrative but not exhaustive).

Phase of the review	Critical aspect	Common errors
Identification	Research question	–Vague or ambiguous research question
	Search strategy	–No expert support (information specialist or research librarian) –Insufficient piloting of the search –Searches that are overly specific (e.g., use NOT operator) but lack sensitivity
	Data extraction	–Not using structured extraction sheets (e.g., CHARMS) when available
Appraisal	Risk of bias	–Using reporting‐quality tools to judge study quality or risk of bias –RoB tool not aligned with the estimand or research question (e.g., RoB 2.0 for causal effects from RCTs, Robins‐I for causal effects in observational studies, Robins‐E for causal effects in environmental studies, PROBAST for diagnostic/prognostic studies) –Insufficient training to understand nuances within the RoB tool
Synthesis	Meta‐analysis	–Vote counting based on statistical significance –Calculating effect sizes using the standard error instead of the standard deviation –Combining studies at high risk and low risk of bias without sensitivity analyses –Statistically ignoring within‐study correlations, for example, including multiple outcomes and effect sizes from the same study while ignoring data dependence. –Not reporting prediction intervals when the results are meant to be applied to a new setting, similar to the included studies. –No assessment of whether conceptual and statistical assumptions of the method (e.g., homogeneity, transitivity, consistency in a network meta‐analysis) are plausible. –No sensitivity analysis when appropriate
Interpretation	Interpretation of results	–Focusing only on point estimates, ignoring uncertainty –Claims and conclusions inconsistent with the risk of bias, level of certainty, possible violations of essential underlying assumptions –Inappropriately interpreting random‐effects summary as a single common effect –Inappropriately interpreting meta‐regressions without acknowledging possible aggregation bias (ecological fallacy).

4. Synthesizing the Results Through Meta‐Analysis

A meta‐analysis within a systematic review is essentially a weighted average of the individual study results. Theoretically, the weights should be based on the quantity and validity of the information from the individual studies. However, choosing the appropriate weights (e.g., fixed effect versus common effects versus random effects models; bias‐adjusted estimates) requires expertise. Given the limited space, some common errors in meta‐analyses identified in previous sport science and medicine studies are presented in Table 1 [16, 17, 18, 19].

Within the sport literature, vote counting is commonly used to observe “trends” in the data instead of calculated summary estimates. With this method, authors draw conclusions from how many studies did or did not find a significant effect. However, leading evidence synthesis organizations have warned that “synthesis methods that involve vote counting based on statistical significance have serious limitations and are unacceptable” [20]. Vote counting may occasionally be justified (e.g., when only the direction of effect is reported), but should generally be avoided because (1) it conveys nothing about the magnitude of effects, (2) is often based solely on p‐values which themselves are generally misinterpreted [21], and (3) gives every study equal weight regardless of the uncertainty in the calculated study effect size.

5. Interpretation

Interpreting findings is challenging [22], and transparency and rigor remain the best safeguards. Meta‐analyses estimate the magnitude of effects or associations, which decision‐makers (athletes, clinicians, coaches) and other researchers then use to make recommendations or support theories. However, the move from estimate to decision (or theory) requires care. Some frameworks (including GRADE, https://www.gradeworkinggroup.org/) have been specifically developed for this purpose and should be used. For example, a recent injury‐prevention meta‐analysis rated the evidence low‐certainty (with studies at high risk of bias or with some concerns), yet still concluded that “the best evidence suggests [intervention] can reduce injury risk.” This claim is self‐contradictory: If the studies are so biased that the true effect could differ substantially from our estimate in either direction, how can one support a claim that the intervention works? Decision‐makers must then make a risk–benefit judgment: Is it acceptable, in our context, to implement something that has very little evidentiary support? Nonfinancial competing interest (the wish to see an intervention succeed, or to defend a position) should not shape how results are interpreted [23].

Every study has limitations. For example, sport science studies are often cross‐sectional and without justifiable and explicit causal assumptions supported by the literature, alternatives such as reverse causation must be considered. If possible, authors should conduct sensitivity analyses to explore whether alternative explanations could also explain the data.

6. Conclusion

A meta‐analysis does not become trustworthy simply because it carries the label, sits high on the evidence pyramid, or produces an estimate; it becomes trustworthy only through the quality of how its studies are selected, assessed, combined, and interpreted. When this process lacks rigor and there are (avoidable) errors, the result may be misleading evidence, presented with the authority of systematic reviews and with the potential to recommend inappropriate practice for years before being corrected.

Funding

The authors have nothing to report.

Conflicts of Interest

The authors have no potential financial conflicts of interest with the content of this article. Franco M. Impellizzeri is employed by a university and a primary academic responsibility is to obtain grants and publish papers. Franco M. Impellizzeri is the Editor‐in‐Chief of Science and Medicine in Football and is currently involved in large meta‐research, including research synthesis. Ian Shrier is retired but continues to hold grants and conduct research. Ian Shrier was the past president of the Society for Research Synthesis Methodology and former Editor‐in‐Chief of Research Synthesis Methods. Franco M. Impellizzeri and Ian Shrier are both members of the editorial board of the Scandinavian Journal of Medicine & Science in Sports.

Data Availability Statement

Data sharing not applicable to this article.