Skip to main content
PMC home page
Journal of Epidemiology and Community Health logoLink to Journal of Epidemiology and Community Health
. 2002 Feb;56(2):119–127. doi: 10.1136/jech.56.2.119

Criteria for evaluating evidence on public health interventions

L Rychetnik 1, M Frommer 1, P Hawe 1, A Shiell 1
PMCID: PMC1732065  PMID: 11812811

Abstract

Public health interventions tend to be complex, programmatic, and context dependent. The evidence for their effectiveness must be sufficiently comprehensive to encompass that complexity. This paper asks whether and to what extent evaluative research on public health interventions can be adequately appraised by applying well established criteria for judging the quality of evidence in clinical practice. It is adduced that these criteria are useful in evaluating some aspects of evidence. However, there are other important aspects of evidence on public health interventions that are not covered by the established criteria. The evaluation of evidence must distinguish between the fidelity of the evaluation process in detecting the success or failure of an intervention, and the success or failure of the intervention itself. Moreover, if an intervention is unsuccessful, the evidence should help to determine whether the intervention was inherently faulty (that is, failure of intervention concept or theory), or just badly delivered (failure of implementation). Furthermore, proper interpretation of the evidence depends upon the availability of descriptive information on the intervention and its context, so that the transferability of the evidence can be determined. Study design alone is an inadequate marker of evidence quality in public health intervention evaluation.

Full Text

The Full Text of this article is available as a PDF (147.2 KB).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. Birch S. As a matter of fact: evidence-based decision-making unplugged. Health Econ. 1997 Nov-Dec;6(6):547–559. doi: 10.1002/(sici)1099-1050(199711)6:6<547::aid-hec307>3.0.co;2-p. [DOI] [PubMed] [Google Scholar]
  2. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ. 1996 May 11;312(7040):1215–1218. doi: 10.1136/bmj.312.7040.1215. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Blakely T., Thornley C. N. Screening for hepatitis B carriers: evidence and policy development in New Zealand. N Z Med J. 1999 Nov 12;112(1099):431–433. [PubMed] [Google Scholar]
  4. Briss P. A., Zaza S., Pappaioanou M., Fielding J., Wright-De Agüero L., Truman B. I., Hopkins D. P., Mullen P. D., Thompson R. S., Woolf S. H. Developing an evidence-based Guide to Community Preventive Services--methods. The Task Force on Community Preventive Services. Am J Prev Med. 2000 Jan;18(1 Suppl):35–43. doi: 10.1016/s0749-3797(99)00119-1. [DOI] [PubMed] [Google Scholar]
  5. Campbell M., Fitzpatrick R., Haines A., Kinmonth A. L., Sandercock P., Spiegelhalter D., Tyrer P. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000 Sep 16;321(7262):694–696. doi: 10.1136/bmj.321.7262.694. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Carande-Kulis V. G., Maciosek M. V., Briss P. A., Teutsch S. M., Zaza S., Truman B. I., Messonnier M. L., Pappaioanou M., Harris J. R., Fielding J. Methods for systematic reviews of economic evaluations for the Guide to Community Preventive Services. Task Force on Community Preventive Services. Am J Prev Med. 2000 Jan;18(1 Suppl):75–91. doi: 10.1016/s0749-3797(99)00120-8. [DOI] [PubMed] [Google Scholar]
  7. Diez-Roux A. V. Bringing context back into epidemiology: variables and fallacies in multilevel analysis. Am J Public Health. 1998 Feb;88(2):216–222. doi: 10.2105/ajph.88.2.216. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Drummond M. F., Jefferson T. O. Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ. 1996 Aug 3;313(7052):275–283. doi: 10.1136/bmj.313.7052.275. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Drummond M. F., Richardson W. S., O'Brien B. J., Levine M., Heyland D. Users' guides to the medical literature. XIII. How to use an article on economic analysis of clinical practice. A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA. 1997 May 21;277(19):1552–1557. doi: 10.1001/jama.277.19.1552. [DOI] [PubMed] [Google Scholar]
  10. Ebrahim S., Smith G. D. Effects of the Heartbeat Wales programme. Effects of government policies on health behaviour must be studied. BMJ. 1998 Sep 26;317(7162):886–886. [PMC free article] [PubMed] [Google Scholar]
  11. Feinstein A. R., Horwitz R. I. Problems in the "evidence" of "evidence-based medicine". Am J Med. 1997 Dec;103(6):529–535. doi: 10.1016/s0002-9343(97)00244-1. [DOI] [PubMed] [Google Scholar]
  12. Flay B. R. Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Prev Med. 1986 Sep;15(5):451–474. doi: 10.1016/0091-7435(86)90024-1. [DOI] [PubMed] [Google Scholar]
  13. Furberg C. D. Challenges to the funding of prevention research. Prev Med. 1994 Sep;23(5):599–601. doi: 10.1006/pmed.1994.1096. [DOI] [PubMed] [Google Scholar]
  14. Giacomini M. K., Cook D. J. Users' guides to the medical literature: XXIII. Qualitative research in health care A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA. 2000 Jul 19;284(3):357–362. doi: 10.1001/jama.284.3.357. [DOI] [PubMed] [Google Scholar]
  15. Glasziou P., Longbottom H. Evidence-based public health practice. Aust N Z J Public Health. 1999 Aug;23(4):436–440. doi: 10.1111/j.1467-842x.1999.tb01291.x. [DOI] [PubMed] [Google Scholar]
  16. Greenhalgh T., Taylor R. Papers that go beyond numbers (qualitative research) BMJ. 1997 Sep 20;315(7110):740–743. doi: 10.1136/bmj.315.7110.740. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Guyatt G. H., Sackett D. L., Cook D. J. Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA. 1993 Dec 1;270(21):2598–2601. doi: 10.1001/jama.270.21.2598. [DOI] [PubMed] [Google Scholar]
  18. Guyatt G. H., Sackett D. L., Cook D. J. Users' guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. JAMA. 1994 Jan 5;271(1):59–63. doi: 10.1001/jama.271.1.59. [DOI] [PubMed] [Google Scholar]
  19. Gyorkos T. W., Tannenbaum T. N., Abrahamowicz M., Oxman A. D., Scott E. A., Millson M. E., Rasooly I., Frank J. W., Riben P. D., Mathias R. G. An approach to the development of practice guidelines for community health interventions. Can J Public Health. 1994 Jul-Aug;85 (Suppl 1):S8–13. [PubMed] [Google Scholar]
  20. Harvey K. Effects of the Heartbeat Wales programme. Health promotion is a waste of time and money. BMJ. 1998 Sep 26;317(7162):887–887. [PubMed] [Google Scholar]
  21. Ioannidis J. P., Haidich A. B., Lau J. Any casualties in the clash of randomised and observational evidence? BMJ. 2001 Apr 14;322(7291):879–880. doi: 10.1136/bmj.322.7291.879. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Irwig L., Zwarenstein M., Zwi A., Chalmers I. A flow diagram to facilitate selection of interventions and research for health care. Bull World Health Organ. 1998;76(1):17–24. [PMC free article] [PubMed] [Google Scholar]
  23. Kerridge I., Lowe M., Henry D. Ethics and evidence based medicine. BMJ. 1998 Apr 11;316(7138):1151–1153. doi: 10.1136/bmj.316.7138.1151. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Krieger N., Zierler S. The need for epidemiologic theory. Epidemiology. 1997 Mar;8(2):212–214. [PubMed] [Google Scholar]
  25. Krieger N., Zierler S. What explains the public's health?--A call for epidemiologic theory. Epidemiology. 1996 Jan;7(1):107–109. doi: 10.1097/00001648-199601000-00020. [DOI] [PubMed] [Google Scholar]
  26. Little M. Assignments of meaning in epidemiology. Soc Sci Med. 1998 Nov;47(9):1135–1145. doi: 10.1016/s0277-9536(98)00185-3. [DOI] [PubMed] [Google Scholar]
  27. Lomas J. Research and evidence-based decision making. Aust N Z J Public Health. 1997 Aug;21(5):439–441. doi: 10.1111/j.1467-842x.1997.tb01730.x. [DOI] [PubMed] [Google Scholar]
  28. Mackenbach J. P. Public health epidemiology. J Epidemiol Community Health. 1995 Aug;49(4):333–334. doi: 10.1136/jech.49.4.333. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Mason J., Drummond M. Reporting guidelines for economic studies. Health Econ. 1995 Mar-Apr;4(2):85–94. doi: 10.1002/hec.4730040202. [DOI] [PubMed] [Google Scholar]
  30. Maynard A. Evidence-based medicine: an incomplete method for informing treatment choices. Lancet. 1997 Jan 11;349(9045):126–128. doi: 10.1016/S0140-6736(96)05153-7. [DOI] [PubMed] [Google Scholar]
  31. Mays N., Pope C. Qualitative research in health care. Assessing quality in qualitative research. BMJ. 2000 Jan 1;320(7226):50–52. doi: 10.1136/bmj.320.7226.50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Mays N., Pope C. Rigour and qualitative research. BMJ. 1995 Jul 8;311(6997):109–112. doi: 10.1136/bmj.311.6997.109. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. McLeroy K. R., Bibeau D., Steckler A., Glanz K. An ecological perspective on health promotion programs. Health Educ Q. 1988 Winter;15(4):351–377. doi: 10.1177/109019818801500401. [DOI] [PubMed] [Google Scholar]
  34. Mosteller F., Colditz G. A. Understanding research synthesis (meta-analysis). Annu Rev Public Health. 1996;17:1–23. doi: 10.1146/annurev.pu.17.050196.000245. [DOI] [PubMed] [Google Scholar]
  35. Murray C. J., Lopez A. D. Evidence-based health policy--lessons from the Global Burden of Disease Study. Science. 1996 Nov 1;274(5288):740–743. doi: 10.1126/science.274.5288.740. [DOI] [PubMed] [Google Scholar]
  36. O'Brien B. J., Heyland D., Richardson W. S., Levine M., Drummond M. F. Users' guides to the medical literature. XIII. How to use an article on economic analysis of clinical practice. B. What are the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. JAMA. 1997 Jun 11;277(22):1802–1806. doi: 10.1001/jama.277.22.1802. [DOI] [PubMed] [Google Scholar]
  37. Oakley A. Experimentation and social interventions: a forgotten but important history. BMJ. 1998 Oct 31;317(7167):1239–1242. doi: 10.1136/bmj.317.7167.1239. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Oxman A. D., Sackett D. L., Guyatt G. H. Users' guides to the medical literature. I. How to get started. The Evidence-Based Medicine Working Group. JAMA. 1993 Nov 3;270(17):2093–2095. [PubMed] [Google Scholar]
  39. Popay J., Rogers A., Williams G. Rationale and standards for the systematic review of qualitative literature in health services research. Qual Health Res. 1998 May;8(3):341–351. doi: 10.1177/104973239800800305. [DOI] [PubMed] [Google Scholar]
  40. Popay J., Williams G. Qualitative research and evidence-based healthcare. J R Soc Med. 1998;91 (Suppl 35):32–37. doi: 10.1177/014107689809135s08. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Russell L. B., Gold M. R., Siegel J. E., Daniels N., Weinstein M. C. The role of cost-effectiveness analysis in health and medicine. Panel on Cost-Effectiveness in Health and Medicine. JAMA. 1996 Oct 9;276(14):1172–1177. [PubMed] [Google Scholar]
  42. Sauerborn R., Nitayarumphong S., Gerhardus A. Strategies to enhance the use of health systems research for health sector reform. Trop Med Int Health. 1999 Dec;4(12):827–835. doi: 10.1046/j.1365-3156.1999.00497.x. [DOI] [PubMed] [Google Scholar]
  43. Shediac-Rizkallah M. C., Bone L. R. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998 Mar;13(1):87–108. doi: 10.1093/her/13.1.87. [DOI] [PubMed] [Google Scholar]
  44. Shefer A., Briss P., Rodewald L., Bernier R., Strikas R., Yusuf H., Ndiaye S., Wiliams S., Pappaioanou M., Hinman A. R. Improving immunization coverage rates: an evidence-based review of the literature. Epidemiol Rev. 1999;21(1):96–142. doi: 10.1093/oxfordjournals.epirev.a017992. [DOI] [PubMed] [Google Scholar]
  45. Smith P. J., Moffatt M. E., Gelskey S. C., Hudson S., Kaita K. Are community health interventions evaluated appropriately? A review of six journals. J Clin Epidemiol. 1997 Feb;50(2):137–146. doi: 10.1016/s0895-4356(96)00338-1. [DOI] [PubMed] [Google Scholar]
  46. Sorensen G., Emmons K., Hunt M. K., Johnston D. Implications of the results of community intervention trials. Annu Rev Public Health. 1998;19:379–416. doi: 10.1146/annurev.publhealth.19.1.379. [DOI] [PubMed] [Google Scholar]
  47. Susser M., Susser E. Choosing a future for epidemiology: I. Eras and paradigms. Am J Public Health. 1996 May;86(5):668–673. doi: 10.2105/ajph.86.5.668. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Susser M., Susser E. Choosing a future for epidemiology: II. From black box to Chinese boxes and eco-epidemiology. Am J Public Health. 1996 May;86(5):674–677. doi: 10.2105/ajph.86.5.674. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Susser M. The tribulations of trials--intervention in communities. Am J Public Health. 1995 Feb;85(2):156–158. doi: 10.2105/ajph.85.2.156. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Tudor-Smith C., Nutbeam D., Moore L., Catford J. Effects of the Heartbeat Wales programme over five years on behavioural risks for cardiovascular disease: quasi-experimental comparison of results from Wales and a matched reference area. BMJ. 1998 Mar 14;316(7134):818–822. doi: 10.1136/bmj.316.7134.818. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Watt D., Verma S., Flynn L. Wellness programs: a review of the evidence. CMAJ. 1998 Jan 27;158(2):224–230. [PMC free article] [PubMed] [Google Scholar]
  52. Weinstein M. C., Siegel J. E., Gold M. R., Kamlet M. S., Russell L. B. Recommendations of the Panel on Cost-effectiveness in Health and Medicine. JAMA. 1996 Oct 16;276(15):1253–1258. [PubMed] [Google Scholar]
  53. Winkleby M. A. The future of community-based cardiovascular disease intervention studies. Am J Public Health. 1994 Sep;84(9):1369–1372. doi: 10.2105/ajph.84.9.1369. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Woolf S. H., Battista R. N., Anderson G. M., Logan A. G., Wang E. Assessing the clinical effectiveness of preventive maneuvers: analytic principles and systematic methods in reviewing evidence and developing clinical practice recommendations. A report by the Canadian Task Force on the Periodic Health Examination. J Clin Epidemiol. 1990;43(9):891–905. doi: 10.1016/0895-4356(90)90073-x. [DOI] [PubMed] [Google Scholar]
  55. Ziglio E. How to move towards evidence-based health promotion interventions. Promot Educ. 1997 Jun;4(2):29–33. doi: 10.1177/102538239700400215. [DOI] [PubMed] [Google Scholar]

Articles from Journal of Epidemiology and Community Health are provided here courtesy of BMJ Publishing Group

RESOURCES