A necessary requirement for development of trustworthy guidelines is to respect the relation between the quality (certainty) of evidence and strength of recommendations. Strong recommendations are justified when they are based on high quality evidence, because such recommendations are considered more accurate.1 On the other hand, uncertainty in benefits and harms (that is, low quality evidence) generally leads to weaker recommendations.
The failure to recognise this important principle results in a tendency to issue strong recommendations based on low quality evidence (which we call discordant recommendations), often leading to harm. For instance, based on advice from low quality evidence, women have experienced avoidable adverse effects from hormone replacement therapy prescribed for the prevention of cardiovascular disease; and women with breast cancer have undergone highly toxic stem cell transplantation without benefit. This practice of decoupling the quality of evidence from the strength of recommendations is usually justified by separating guidelines into consensus guidelines versus evidence based guidelines—a practice that does not appear to have abated over time.
Basing treatment decisions or clinical guidelines on low quality evidence means that the true effects of a treatment or clinical decision might differ considerably from best estimates. This discrepancy could result in launching campaigns (such as those designed to persuade women to use hormone replacement therapy) that are based on an unjustified faith in net benefit instead of transparently sharing the uncertainties in the quality of evidence on which the recommendations were based. Inappropriately strong recommendations have other problematic consequences, such as discouraging future randomised controlled trials that would generate higher quality evidence.
However, not all discordant recommendations are equally problematic.2 For instance, patients with a high likelihood of bad outcomes might all be willing to try an unproven, but potentially beneficial intervention.
Issuing discordant recommendations without a compelling rationale is not unusual. The problem has previously been highlighted by the World Health Organisation3 and Endocrine Society,4 and most recently in our study in The BMJ.5 We found that when the American College of Cardiology (ACC)/American Heart Association (AHA) and the American Society of Clinical Oncology (ASCO) (the two largest worldwide organisations that develop guidelines for heart disease and cancer, the two leading causes of death globally) faced low quality evidence, 41% and 20% of their recommendations proved to be inappropriate or discordant, respectively.5 Inappropriate discordant recommendations are those that do not meet the GRADE (grading of recommendations assessment, development, and evaluation) criteria of appropriateness. Although these leading organisations claim to use evidence based methods for their guidelines, the fact that up to 41% of their recommendations are inappropriate and discordant should raise concerns in both health professionals and patients.
Some organisations—including the ACC/AHA and ASCO—explicitly classify their guidelines as evidence based when much of the supporting evidence is deemed to be moderate or high quality, and classify their guidelines as consensus based when it is not. In their consensus versus evidence based guidelines, the odds of issuing inappropriate discordant recommendations proved 2.6 times higher in ACC/AHA guidelines and 5.1 times higher in ASCO guidelines.5 Classifying guidelines as consensus based might allow panels to be less rigorous in ensuring that the strength of recommendations are consistent with the underlying quality of evidence. All guidelines require judicious consideration of the relevant evidence—in other words, all guidelines should be evidence based—and organisations should focus on avoiding inappropriate discordant recommendations.
When facing low or very low quality evidence, guidelines should avoid issuing inappropriate discordant recommendations. Abandoning consensus based guidelines is likely to facilitate this goal.
Acknowledgments
This project was supported in part by grant number R01HS024917 from the Agency for Healthcare Research and Quality (Drs. Djulbegovic & Guyatt). The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Footnotes
Competing interests: none declared.
References
- 1.Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet 2017;390:415–23. doi: 10.1016/S0140-6736(16)31592-6 [DOI] [PubMed] [Google Scholar]
- 2.Andrews JC, Schünemann HJ, Oxman AD, et al. GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation’s direction and strength. J Clin Epidemiol 2013;66:726–35. doi: 10.1016/j.jclinepi.2013.02.003 [DOI] [PubMed] [Google Scholar]
- 3.Alexander PE, Bero L, Montori VM, et al. World Health Organization recommendations are often strong based on low confidence in effect estimates. J Clin Epidemiol 2014;67:629–34. doi: 10.1016/j.jclinepi.2013.09.020 [DOI] [PubMed] [Google Scholar]
- 4.Brito JP, Domecq JP, Murad MH, Guyatt GH, Montori VM. The Endocrine Society guidelines: when the confidence cart goes before the evidence horse. J Clin Endocrinol Metab 2013;98:3246–52. doi: 10.1210/jc.2013-1814 [DOI] [PubMed] [Google Scholar]
- 5.Yao L, Ahmed MM, Guyatt GH, et al. Discordant and inappropriate discordant recommendations in consensus and evidence based guidelines: empirical analysis. BMJ 2021;375:e066045. [DOI] [PMC free article] [PubMed] [Google Scholar]