Skip to main content
PMC home page
Journal of Epidemiology and Community Health logoLink to Journal of Epidemiology and Community Health
. 2005 Oct;59(10):886–892. doi: 10.1136/jech.2005.034199

Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go?

D Ogilvie 1, M Egan 1, V Hamilton 1, M Petticrew 1
PMCID: PMC1732915  PMID: 16166365

Abstract

Study objective: There is little guidance on how to select the best available evidence of health effects of social interventions. The aim of this paper was to assess the implications of setting particular inclusion criteria for evidence synthesis.

Design: Analysis of all relevant studies for one systematic review, followed by sensitivity analysis of the effects of selecting studies based on a two dimensional hierarchy of study design and study population.

Setting: Case study of a systematic review of the effectiveness of interventions in promoting a population shift from using cars towards walking and cycling.

Main results: The distribution of available evidence was skewed. Population level interventions were less likely than individual level interventions to have been studied using the most rigorous study designs; nearly all of the population level evidence would have been missed if only randomised controlled trials had been included. Examining the studies that were excluded did not change the overall conclusions about effectiveness, but did identify additional categories of intervention such as health walks and parking charges that merit further research, and provided evidence to challenge assumptions about the actual effects of progressive urban transport policies.

Conclusions: Unthinking adherence to a hierarchy of study design as a means of selecting studies may reduce the value of evidence synthesis and reinforce an "inverse evidence law" whereby the least is known about the effects of interventions most likely to influence whole populations. Producing generalisable estimates of effect sizes is only one possible objective of evidence synthesis. Mapping the available evidence and uncertainty about effects may also be important.

Full Text

The Full Text of this article is available as a PDF (87.5 KB).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. Alderson P., Roberts I. Should journals publish systematic reviews that find no evidence to guide practice? Examples from injury research. BMJ. 2000 Feb 5;320(7231):376–377. doi: 10.1136/bmj.320.7231.376. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Barreto Mauricio L. Efficacy, effectiveness, and the evaluation of public health interventions. J Epidemiol Community Health. 2005 May;59(5):345–346. doi: 10.1136/jech.2004.020784. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Dickersin Kay. Systematic reviews in epidemiology: why are we so far behind? Int J Epidemiol. 2002 Feb;31(1):6–12. doi: 10.1093/ije/31.1.6. [DOI] [PubMed] [Google Scholar]
  4. Glasziou Paul, Vandenbroucke Jan P., Chalmers Iain. Assessing the quality of research. BMJ. 2004 Jan 3;328(7430):39–41. doi: 10.1136/bmj.328.7430.39. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Green J., Tones K. Towards a secure evidence base for health promotion. J Public Health Med. 1999 Jun;21(2):133–139. doi: 10.1093/pubmed/21.2.133. [DOI] [PubMed] [Google Scholar]
  6. Hawker Sheila, Payne Sheila, Kerr Christine, Hardey Michael, Powell Jackie. Appraising the evidence: reviewing disparate data systematically. Qual Health Res. 2002 Nov;12(9):1284–1299. doi: 10.1177/1049732302238251. [DOI] [PubMed] [Google Scholar]
  7. Kaptchuk Ted J. Effect of interpretive bias on research evidence. BMJ. 2003 Jun 28;326(7404):1453–1455. doi: 10.1136/bmj.326.7404.1453. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. McNally Rosalind, Alborz Alison. Developing methods for systematic reviewing in health services delivery and organization: an example from a review of access to health care for people with learning disabilities. Part 1. Identifying the literature. Health Info Libr J. 2004 Sep;21(3):182–192. doi: 10.1111/j.1471-1842.2004.00512.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Mindell J., Boaz A., Joffe M., Curtis S., Birley M. Enhancing the evidence base for health impact assessment. J Epidemiol Community Health. 2004 Jul;58(7):546–551. doi: 10.1136/jech.2003.012401. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Moller J. Reconsidering community based interventions. Inj Prev. 2004 Feb;10(1):2–3. doi: 10.1136/ip.2003.004606. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Ogilvie David, Egan Matt, Hamilton Val, Petticrew Mark. Promoting walking and cycling as an alternative to using cars: systematic review. BMJ. 2004 Sep 22;329(7469):763–763. doi: 10.1136/bmj.38216.714560.55. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Ogilvie David, Hamilton Val, Egan Matt, Petticrew Mark. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005 Sep;59(9):804–808. doi: 10.1136/jech.2005.034181. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Petticrew M., Roberts H. Evidence, hierarchies, and typologies: horses for courses. J Epidemiol Community Health. 2003 Jul;57(7):527–529. doi: 10.1136/jech.57.7.527. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Petticrew Mark. Presumed innocent. Why we need systematic reviews of social policies. Am J Prev Med. 2003 Apr;24(3 Suppl):2–3. doi: 10.1016/s0749-3797(02)00650-5. [DOI] [PubMed] [Google Scholar]
  15. Petticrew Mark, Whitehead Margaret, Macintyre Sally J., Graham Hilary, Egan Matt. Evidence for public health policy on inequalities: 1: the reality according to policymakers. J Epidemiol Community Health. 2004 Oct;58(10):811–816. doi: 10.1136/jech.2003.015289. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Rychetnik L., Frommer M., Hawe P., Shiell A. Criteria for evaluating evidence on public health interventions. J Epidemiol Community Health. 2002 Feb;56(2):119–127. doi: 10.1136/jech.56.2.119. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Slavin R. E. Best evidence synthesis: an intelligent alternative to meta-analysis. J Clin Epidemiol. 1995 Jan;48(1):9–18. doi: 10.1016/0895-4356(94)00097-a. [DOI] [PubMed] [Google Scholar]
  18. Thomson Hilary, Hoskins Robert, Petticrew Mark, Ogilvie David, Craig Neil, Quinn Tony, Lindsay Grace. Evaluating the health effects of social interventions. BMJ. 2004 Jan 31;328(7434):282–285. doi: 10.1136/bmj.328.7434.282. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Journal of Epidemiology and Community Health are provided here courtesy of BMJ Publishing Group

RESOURCES