Reverse spin bias: preliminary observations of reporting bias in medical systematic reviews

Abstract

Background

While conducting an umbrella review of e-cigarettes for smoking cessation, we observed that in many instances, systematic review authors reported findings favorable to the treatment, yet they declined to recommend it or recommended against it despite the evidence of its effectiveness in their own systematic reviews.

Existing literature

We searched the literature for a term or category to describe this form of reporting bias where the authors’ recommendations dismiss their findings of treatment benefit. Ideally the term spin bias should apply to any conclusion or recommendation not supported by the findings of the study, but in practice spin bias is almost exclusively applied to the narrative attribution of significance or causation to statistically non-significant data or findings.

Issue under discussion

After observing that many systematic review authors dismissed their findings of effectiveness for e-cigarettes for cessation, we wondered if this form of reporting bias also occurs in the systematic reviews on other controversial treatments. We made a rapid search for recent systematic reviews on medical cannabis for pain, another controversial treatment. Here also we observed that many authors did not recommend cannabis for pain management even though their findings clearly showed treatment benefit. We tentatively offer the term reverse spin bias for the narrative discounting or dismissal of statistically significant findings. We catalogued the narrative turns that enabled reverse spin bias in 20 systematic reviews of e-cigarettes for cessation and medical cannabis for pain. We identified five mechanisms: discount the evidence base, discredit the primary studies, appeal to fear, dismiss the treatment modality a priori, and omit findings. We speculate that authors introduce reverse spin bias to improve their chances for publication or to support their position about a treatment.

Conclusion

A standard task for editors and peer reviewers is confirming that treatment recommendations are supported by the review’s data, yet our examples strongly suggest that this examination for reporting bias is frequently skipped. By proposing a new term, reverse spin bias, we hope to bring stronger scrutiny to bear on these instances of reporting bias that are detrimental to evidence-informed clinical practice.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41073-025-00185-9.

Background

The impetus for this Comment was our observations of reporting bias during our umbrella review on e-cigarettes for smoking cessation [1]. In many instances, systematic review authors reported findings favorable to the effectiveness of the treatment, yet they declined to recommend it or recommended against e-cigarettes despite the evidence of its effectiveness in their own systematic reviews.

We searched for a term on the narrative dismissal of significant findings but did not find a match for what we were observing. Ideally, the term spin bias would encompass all types of “recommendations for clinical practice not supported by findings” [2], but in the literature, spin bias is defined as the misrepresentation of nonsignificant results [3, 4]. Yavchitz et al. in their seminal paper on spin bias explicitly excluded “spin that understated the beneficial effect of the experimental intervention.” Why? Because they could not find any research on it [2].

We wondered if this type of reporting bias was limited to systematic reviews in our controversial field, or was it occurring in other systematic reviews on socially or politically contested medical treatments? To satisfy our curiosity we made a quick scoping search of systematic reviews on cannabis for pain. Here too we observed that findings of benefit were often discounted or discredited.

We are tentatively proposing the term reverse spin bias for the narrative discounting or dismissal of statistically significant findings of benefit. For this Comment, we flagged reverse spin bias in the systematic reviews of two controversial treatments: e-cigarettes for smoking cessation and medical cannabis for pain management. Through comparing the systematic review data with quotes of the authors’ recommendations, we sought to identify the narrative mechanisms employed to produce reverse spin bias. We also speculated on the potential motivations for authors to disavow their own findings. Our objective in this Comment was to illustrate possible examples of reverse spin, not to quantify its prevalence or to assess “conventional” spin bias, the reporting of null findings as significant or causative.

Observations of reverse spin bias

Example 1 – E-cigarettes for smoking cessation

We used the 16 systematic reviews published from 2021 to 2025 from the exhaustive search in our umbrella review. Thirteen systematic reviews reported findings that e-cigarettes were significantly more effective than other cessation methods but only three review author teams recommended their use as a cessation treatment. In ten systematic reviews with findings of benefit, five authors recommended against their use [5–9] and the other five declined to give a recommendation [10–14] even though there was evidence of benefit in their own reviews. See Table 1.

Table 1.

E-cigarette systematic review findings and reverse spin bias conclusions

Review	Findings	Conclusion quote
Hanewinkel 2022 [5]	ENDS more effective than NRT RR 1.58 [1.20–2.08]	“Clinicians should not recommend [ENDS] for smoking cessation due to the risk of permanent nicotine dependence”
Hedman 2021 [10]	ENDS more effective than control (NRT, placebo) OR 2.04 [1.51, 2.27]	“could not find quality evidence for an association between [ENDS] use and smoking cessation”
Li 2022 [6]	ENDS more effective than NRT RR 1.67 [1.21, 2.28]	“the harms between [ENDS] and NRT are still uncertain”
Oloyede 2021 [7]	5 RCTs pooled estimate NR	“clinicians should direct patients who use tobacco to other evidence-based tobacco cessation interventions”
Patnode 2021 [8]	5 RCTs: pooled estimate NR 3 RCTs (n = 2308) ENDs more effective than NRT [data NR] 2 RCTs (n = 807) no effect	“Data on the effectiveness and safety of electronic cigarettes for smoking cessation among adults are also limited and results are inconsistent.”
Thomas 2022 [11]	7 trials, ENDS high nicotine more effective than placebo OR 3.22 [1.63, 6.36]	“Although [ENDS] showed promise as cessation tools, more research is needed”
Vanderkam 2022 [12]	7 RCTs ENDS more effective than NRT RR 1.49 [1.14, 1.95] ENDS more effective than placebo RR 1.66 [1.01, 2.73]	“few studies carried out, which does not allow for an affirmation and recommendation of practice”
Wang 2021 [9]	Daily ENDS use more effective than no ENDS use OR 1.529 [1.158, 2.19]	“E-cigarettes should not be approved as consumer products but may warrant consideration as a prescription therapy”
Yazidjoglou 2021 [13]	ENDS more effective than no intervention or usual care RR 2.30 [1.19, 4.42]	“The balance of safety and efficacy of e-cigarettes needs to be considered in clinical decision making about their use for smoking cessation.”
Zakiyah 2021 [14]	Pooled estimate NR. One of 20 RCTs described ENDS as no more effective than other treatments, 8 RCTs described ENDS as effective, 7 RCTs cessation findings NR	“a very modest portion of subjects were abstinent”

[] – 95% confidence interval

ENDS electronic nicotine delivery system (e-cigarette, vapes)

NR not reported

NRT nicotine replacement therapy

OR odds ratio

RCT randomized controlled trials

RR relative risk

vs. versus, i.e. compared with

Example 2 – Medical cannabis for pain management

Current medical cannabis systematic reviews were retrieved with a rapid scoping search conducted December 2, 2024 in PubMed with the syntax (cannabis[Title/Abstract]) AND (pain[Title/Abstract]) Filters: Systematic Review, DATES 2021–2025. This was not an exhaustive search, but a reasonably comprehensive sample of current systematic reviews available in PubMed, a popular open access biomedical article database. The search retrieved 29 systematic reviews, of which 10 exhibited reverse spin bias [15–24], see Table 2. The full list of systematic reviews examined are in Supplementary Material, Supplementary Table 1.

Table 2.

Cannabis for pain systematic review findings and reverse spin bias conclusions

Review	Findings	Conclusion quote
Bialas 2022 [15]	6 long-term observational studies, N = 2686, non-cancer pain relief.20.8% [10.2, 34.0] experienced pain relief 50% or greater	“These findings are, however, based on very low quality evidence.” GRADE assessment not reported
Bywood 2021 [16]	6 of 8 systematic reviews found at least one statistically significant beneficial effect of cannabis on chronic pain (Table 4) Health Evidence Checklist assessment of reviews by authors: 4 strong quality, 3 moderate quality, 1 weak quality	“small or very small significant improvements were reported in low quality studies” “low quality studies…not reliable”
Giossi 2022 [17]	8 RCTs More than 4 weeks cannabinoid treatment significantly reduced pain compared to placebo (Visual Analog Scale MD =—1.28; CI [2.33,- 0.22] “All studies at low risk of bias for measurement of the outcome”	“limited benefit on pain relief” “generally low quality of evidence”
Häuser 2023 [18]	Analysis 1.1, Patient impression of improvement, favours THC (561 events) versus placebo (435 events) 0.06 [0.01, 0.12] Analysis 2.1, Mean pain intensity favours THC (n = 152) versus placebo (n = 149) −0.98 [−1.36, −0.60]	Cancer pain: “lack of good evidence of efficacy or harm”
International Association for the Study of Pain 2021 [19]	57 systematic reviews, 49 with AMSTAR2 ratings low or critically low, and 8 rated as moderate or high quality by the authors	“Did not find any moderate- or high-quality evidence” Harms data “lacking.”
Lambe 2024 [20]	6 systematic reviews, pooled estimate NR Compared THC and placebo for pain intensity: 4 “reported significant improvement” (low to very low certainty), 1 “mixed findings” (very low certainty) and 1 “no significant difference” (very low certainty) (Table 45) “evidence for neuropathic pain was promising”	“findings for [pain] outcomes were inconsistent at best”
Liang 2022 [21]	Gynecological pain: 7 cross-sectional studies N =,1,528 all low RoB”all studies reported that most patients who used cannabis experienced pain relief.” Pain relief rate of 61% to 95.5% (Table 2)	“varying cannabis formulations, delivery methods, and dosages preclude a definitive statement about cannabis for gynecologic pain relief” (Abstract)
Longo 2021 [22]	In the 8 RCTs described as having no treatment effect, 2 RCTs found a statistically significant decrease in pain in the oral mucosa cannabis spray sub-groups as compared to placebo	“Of the 13 studies, 8 found no treatment effect.” (Table 1)
Price 2022 [23]	“follow up ranges [4 weeks – 6 months], however, are very much within the typical range of pharmaceutical analgesics studies. No serious adverse events directly attributed to cannabis use were reported in the four studies analyzed.”	“Longer term treatment benefits or adverse side effects beyond the 6 month paradigm can therefore not be inferred”
Seffah 2023 [24]	Systematic review 32 RCT N = 5174 adults “medical cannabis improves chronic pain”	“Despite this seeming abundance of evidence leaning in favor of medical cannabis…we continue to recommend only …[for] chronic pain unresponsive to other modalities.” “Much of the famed benefits of the drug may only be anecdotal, at best”

[]– 95% confidence interval

MD median difference (10 point scale)

RCT randomized controlled trial

RoB risk of bias

Narrative mechanisms for reverse spin bias

How do systematic review authors engage in reverse spin bias of their own statistically significant evidence of benefit? The observations reported in Tables 1 and 2 suggest five mechanisms.

Discount the evidence base

The most common mechanism for reverse spin bias was systematic review authors discounting their evidence base. The reverse spin bias was introduced by describing their own evidence base as inconsistent, consisting of only low-quality studies, or having an insufficient number of primary studies, regardless of the number.

One narrative turn was to discount the evidence base as “inconsistent.” In systematic reviews with a meta-analysis, this statement should be validated by a test for heterogeneity, such as the I². In narrative systematic reviews, a tally of study data should demonstrate inconsistency across studies. Reverse spin bias occurs when the weight of the evidence favors a beneficial effect, but the authors negate it by referencing a smaller number of studies with non-significant findings.

Another narrative tactic was to state that the evidence base consisted of only low or very low quality primary studies without presenting a GRADE [25] or other formal evaluation of the evidence base. Many systematic reviews exclude studies at high risk of bias, but when they are included, it is standard practice to discuss the impact of the individual low-quality studies on the overall findings [26]. A blanket dismissal of the evidence base negates the potential findings of treatment benefit in the studies at low risk of bias.

A similar narrative device to negate the review’s evidence base was stating that there were few “high quality” studies. This is often true in systematic reviews, but focusing on the lack of “high quality” studies serves to exclude evidence from moderate quality studies and observational studies which achieved an increase in their GRADE rating to moderate.

Although systematic review authors do not always find sufficient studies to reach a conclusion or offer a recommendation, some authors dismissed their evidence as “limited” or “insufficient” but did not provide a test of statistical power or reference a methodological guideline for the number of studies necessary for a valid and reliable recommendation. A similar argument was that the experimental treatment had a smaller evidence base than established treatments. When the evidence base is very small, a conclusion that more research is needed is justified. For the identification of reverse spin with this statement, the number of studies and the number of participants requires further evaluation.

Discredit the primary studies

Systematic review authors introduce reverse spin bias by discrediting the findings in primary studies. Individual primary studies were described as “not high quality” or “low or very low quality” without reference to their individual risk of bias assessment, a standard evaluation in systematic reviews. Seffah et al. [24] were notably dismissive of their findings, calling the evidence “anecdotal” even though it was based on a systematic review of randomized controlled trials.

Another way that systematic review authors discounted beneficial outcomes in individual primary studies was to describe the statistically significant treatment benefit as small or limited. This statistical term can downplay treatment effectiveness for conditions where even a small improvement, such as for pain, is of clinical interest.

In one instance we observed, the systematic review authors identified multiple studies showing treatment benefit but declined to offer a conclusion by claiming excessive heterogeneity between the studies. Authors are expected to discuss the effects of heterogeneity on their findings instead of dismissing the findings outright.

Appeal to fear

Claims of unknown, unnamed future harms from the treatment may be invoked to dismiss findings of treatment benefit. “Unknown future harms” provoke fear, even where the authors stated that the durations of the studies were routine for pharmaceutical studies. A language of “risks” is employed in conclusions, even where adverse effects data were not included the review. This is not an evidence-based discussion of treatment safety but the rhetorical device of appeal to pathos, to the emotion of fear.

Dismiss the treatment modality a priori

A controversial treatment may be dismissed for reasons unrelated to its clinical benefits. One review team rejected e-cigarettes because of the risks of nicotine use, although the comparator was nicotine replacement therapy. Another review team cited regulatory concerns. In both instances, the authors’ position opposing the treatment was already established.

Omit findings

None of the systematic reviews in our e-cigarette umbrella review omitted findings, but we observed this mechanism in some of the cannabis systematic reviews. Per Wheeldon and Heidt [27] potential benefits of cannabis use have been ignored in the research. In some systematic reviews, findings of benefit in subgroups were omitted. The International Association for the Study of Pain [19] misreported that there were no moderate- or high-quality systematic reviews in their report but they assessed 8 of 57 of the reviews at moderate or high quality, omitting these findings from their evidence statement.

Motivations for reverse spin

We suspect that reverse spin bias has a similar function to spin bias, as a strategy to improve the chances of publication. In the case of e-cigarettes, since at least 2000, anti-smoking organizations have pressured journals not to publish any research supporting tobacco industry products “regardless of the scientific merits of the paper” [28]. For medical cannabis, the assumption that cannabis use has only harms is a position held by a number of journals, limiting the dissemination of contradicting evidence [27]. Consequently, systematic review authors may seek to improve their chances for publication by concluding that a controversial treatment is not recommended or by declining to recommend it.

Another potential motivation for authors for reverse spin bias is to frame the conclusions of the review to conform to their established positions on the treatment. Although confirmation bias causes authors to focus on findings that confirm their position, with reverse spin bias the authors apply narrative strategies to discredit unwanted findings of benefit for a treatment that they do not desire to recommend.

Limitations

To the best of our knowledge, no studies have identified reverse spin bias. Our observations offer evidence on how 20 systematic review teams have dismissed their own findings of treatment benefit, but this is a very small sample of the thousands of biomedical systematic reviews published every year. Our classifications of reverse spin bias mechanisms are tentative and certainly incomplete. Our assessment was restricted to identifying potential reverse spin, and other forms of spin bias were outside the scope of this Comment. The identification of reverse spin bias is limited in that it indicates potential reporting bias; it does not by itself provide an independent evaluation of the true balance of benefits and harms of the interventions under review.

Conclusion

Ideally, systematic review authors should resist any pressure to “play up (or play down)” their findings [3]. The consequences of reverse spin bias, a reporting bias, are that potentially effective treatments are neglected, and the data generated by clinical studies are wasted. We speculate that reverse spin bias could be occurring in the systematic reviews on other socially contested medical treatments, for example safe opioid consumption sites or managed alcohol programs. Wider investigations of reverse spin are necessary because reverse spin could occur in any context where review authors favour null or negative conclusions.

Editors and peer reviewers must be alert to discrepancies between the findings of biomedical systematic reviews and the treatment recommendations that their authors endorse. While it may seem like a standard task, looking at our examples strongly suggests that this critical examination for reporting bias has been all too frequently omitted. By proposing a new term, reverse spin bias, we hope to bring stronger scrutiny to bear on these instances of reporting bias that are detrimental to evidence-informed clinical practice.

Authors' contributions

RO conceptualized the study, performed the search for spin bias references, conducted the search for systematic reviews of cannabis, performed the data extractions of study data and recommendation quotes, analyzed the reverse spin mechanisms, and was the major contributor in writing the manuscript. GRMLR conceptualized the study, contributed to the search for spin bias references, conducted the search for systematic reviews of e-cigarettes, contributed to data extractions for e-cigarette reviews, validated the reverse spin mechanisms, and edited the final manuscript. RP conceptualized the study, contributed to the analysis of reverse spin mechanisms, and edited the initial and final drafts of the manuscript. All authors read and approved the final manuscript.

Funding

No specific funding was received for this study.

Data availability

All data generated and/or analyzed during this study are included in this published article and its Supplementary Materials.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

RO declares no competing interests.

GRMLR declares no competing interests.

RP is a full tenured professor of Internal Medicine at the University of Catania (Italy) and Medical Director of the Institute for Internal Medicine and Clinical Immunology at the same University. He has received grants from U-BIOPRED and AIR-PROM, Integral Rheumatology & Immunology Specialists Network (IRIS), Foundation for a Smoke Free World, Pfizer, GlaxoSmithKline, CV Therapeutics, NeuroSearch A/S, Sandoz, Merk Sharp & Dohme, Boehringer Ingelheim, Novartis, Arbi Group Srl., Duska Therapeutics, Forest Laboratories, Ministero dell Universita’ e della Ricerca (MUR) Bando PNRR 3277/2021 (CUP E63C22000900006) and 341/2022 (CUP E63C22002080006), funded by NextGenerationEU of the European Union (EU), and the ministerial grant PON REACT-EU 2021 GREENBando 3411/2021 by Ministero dell Universita’ e (MUR)—PNRR EU Community. He is the founder of the Centre for Tobacco Prevention and Treatment (CPCT) at the University of Catania and of the Centre of Excellence for the Acceleration of Harm Reduction at the same university. He receives consultancy fees from Pfizer, Boehringer Ingel-heim, Duska Therapeutics, Forest Laboratories, CV Therapeutics, Sermo Inc., GRG Health, Clarivate Analytics, Guidepoint Expert Network, and GLG Group. He receives textbooks royalties from Elsevier. He is also involved in a patent application for ECLAT Srl. He is a pro bono scientific advisor for Lega Italiana Anti Fumo (LIAF) and the International Network of Nicotine Consumers Organizations (INNCO); and he is Chair of the European Technical Committee for Standardization on “Requirements and test methods for emissions of electronic cigarettes” (CEN/TC 437; WG4). He declares no conflict of interest in regards to the published work.