To the Editor
Dr Hughes and colleagues1 found that blood pressure lowering with antihypertensive agents was associated with a reduced risk of dementia or cognitive impairment compared with a control. This conclusion was primarily driven by a statistically significant odds ratio of 0.93 (95% CI, 0.88–0.98) resulting from performing a meta-analysis on 13 effect estimates from 12 randomized clinical trials. We would like to point to one particular consideration regarding the conclusions drawn from this meta-analysis.
Of the 13 estimates in the meta-analysis, only 1 was statistically significant. In other words, when considered individually, 12 comparisons found no evidence of an effect, although the effect estimate was less than 1 in 9 of them. This phenomenon is not surprising because it is known that meta-analysis of multiple studies increases the statistical power to detect an effect. However, it does not rule out the possibility that the overall significance of the meta-analysis is driven entirely by a single study, with no evidence that the effect is replicated in at least 2 or more studies. The importance of replicability has been well recognized for clinical studies; for example, the US Food and Drug Administration requires at least 2 studies showing an effect to consider the finding as a scientific discovery.2
We suggest enhancing the meta-analysis with a replicability analysis. Such analyses provide an r value that quantifies the evidence of replicability (ie, whether the effect is present in at least 2 studies).3,4 If the r value is less than the type I error (eg, 0.05), it can be concluded that there was a treatment effect in at least 2 studies. Such a conclusion can be established even if none of the individual studies is statistically significant.3,4 If the r value is greater than 0.05, it does not preclude the effect from being true but emphasizes that the current evidence may rely critically on a single study.
Further research and more evidence on the replicability of the protective effect of blood pressure lowering on dementia are needed before a conclusion pertinent for health care practitioners can be made.
Footnotes
Conflict of Interest Disclosures: Dr Panagiotou reported receiving a grant from the Agency for Healthcare Research and Quality. Ms Jaljuli reported receiving a grant from the US-Israel Binational Science Foundation and the National Science Foundation. Dr Heller reported receiving a grant from the Israeli Science Foundation.
Contributor Information
Orestis A. Panagiotou, Department of Health Services, Policy and Practice, Brown University School of Public Health, Providence, Rhode Island;.
Iman Jaljuli, Department of Statistics and Operations Research, Tel-Aviv University, Tel-Aviv, Israel..
Ruth Heller, Department of Statistics and Operations Research, Tel-Aviv University, Tel-Aviv, Israel..
References
- 1.Hughes D, Judge C, Murphy R, et al. Association of blood pressure lowering with incident dementia or cognitive impairment: a systematic review and meta-analysis. JAMA. 2020;323(19):1934–1944. doi: 10.1001/jama.2020.4249 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.US Food and Drug Administration. Guidance for Industry: Providing Clinical Evidence of Effectiveness for Human Drugs and Biological Products. US Food and Drug Administration; 1998. [Google Scholar]
- 3.Wang J, Owen AB. Admissibility in partial conjunction testing. J Am Stat Assoc. 2019;114(525):158–168. doi: 10.1080/01621459.2017.1385465 [DOI] [Google Scholar]
- 4.Benjamini Y, Heller R. Screening for partial conjunction hypotheses. Biometrics. 2008;64(4):1215–1222. doi: 10.1111/j.1541-0420.2007.00984.x [DOI] [PubMed] [Google Scholar]