Skip to main content
NCBI home page
CMAJ : Canadian Medical Association Journal logoLink to CMAJ : Canadian Medical Association Journal
. 1996 Jun 15;154(12):1837–1843.

Prevention. How much harm? How much benefit? 2. Ten potential pitfalls in determining the clinical significance of benefits.

K G Marshall 1
PMCID: PMC1487736  PMID: 8653643

Abstract

A preventive program is only of value if it has proven benefits that outweigh any adverse consequences; unfortunately, determination of the clinical significance of reported benefits is not always easy. The first article of this series discussed the confusion caused by reporting results in terms of relative rates. In this article, 10 other pitfalls that may lead to misunderstanding of the degree of benefits are reviewed. These pitfalls are: the type of outcome chosen (surrogate v. clinically significant), the risk level in the population screened, the interval between the intervention and the benefit, the duration of intervention required to achieve the benefit, the overshadowing of one benefit by another, the application of a benefit for one variant of a disease to another variant, lower benefits in community settings than in clinical trials, publication bias, preferential citation of studies showing beneficial effects and "false-negative" results of studies. These pitfalls are illustrated through examples from the current medical literature.

Full text

PDF
1837

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. Adolfsson J., Carstensen J., Löwhagen T. Deferred treatment in clinically localised prostatic carcinoma. Br J Urol. 1992 Feb;69(2):183–187. doi: 10.1111/j.1464-410x.1992.tb15493.x. [DOI] [PubMed] [Google Scholar]
  2. Andrade S. E., Walker A. M., Gottlieb L. K., Hollenberg N. K., Testa M. A., Saperia G. M., Platt R. Discontinuation of antihyperlipidemic drugs--do rates reported in clinical trials reflect rates in primary care settings? N Engl J Med. 1995 Apr 27;332(17):1125–1131. doi: 10.1056/NEJM199504273321703. [DOI] [PubMed] [Google Scholar]
  3. Angell M. Patients' preferences in randomized clinical trials. N Engl J Med. 1984 May 24;310(21):1385–1387. doi: 10.1056/NEJM198405243102111. [DOI] [PubMed] [Google Scholar]
  4. Berlin J. A. Will publication bias vanish in the age of online journals? Online J Curr Clin Trials. 1992 Jul 8;DOC:[1337 words–paragraphs]. [PubMed] [Google Scholar]
  5. Black W. C., Welch H. G. Advances in diagnostic imaging and overestimations of disease prevalence and the benefits of therapy. N Engl J Med. 1993 Apr 29;328(17):1237–1243. doi: 10.1056/NEJM199304293281706. [DOI] [PubMed] [Google Scholar]
  6. Catalona W. J., Smith D. S., Ratliff T. L., Basler J. W. Detection of organ-confined prostate cancer is increased through prostate-specific antigen-based screening. JAMA. 1993 Aug 25;270(8):948–954. [PubMed] [Google Scholar]
  7. Cauley J. A., Seeley D. G., Ensrud K., Ettinger B., Black D., Cummings S. R. Estrogen replacement therapy and fractures in older women. Study of Osteoporotic Fractures Research Group. Ann Intern Med. 1995 Jan 1;122(1):9–16. doi: 10.7326/0003-4819-122-1-199501010-00002. [DOI] [PubMed] [Google Scholar]
  8. Chodak G. W. The role of watchful waiting in the management of localized prostate cancer. J Urol. 1994 Nov;152(5 Pt 2):1766–1768. doi: 10.1016/s0022-5347(17)32381-9. [DOI] [PubMed] [Google Scholar]
  9. Chodak G. W., Thisted R. A., Gerber G. S., Johansson J. E., Adolfsson J., Jones G. W., Chisholm G. D., Moskovitz B., Livne P. M., Warner J. Results of conservative management of clinically localized prostate cancer. N Engl J Med. 1994 Jan 27;330(4):242–248. doi: 10.1056/NEJM199401273300403. [DOI] [PubMed] [Google Scholar]
  10. Curb J. D., Borhani N. O., Blaszkowski T. P., Zimbaldi N., Fotiu S., Williams W. Long-term surveillance for adverse effects of antihypertensive drugs. JAMA. 1985 Jun 14;253(22):3263–3268. [PubMed] [Google Scholar]
  11. Dickersin K., Min Y. I. NIH clinical trials and publication bias. Online J Curr Clin Trials. 1993 Apr 28;DOC:[4967 words–paragraphs]. [PubMed] [Google Scholar]
  12. Easterbrook P. J., Berlin J. A., Gopalan R., Matthews D. R. Publication bias in clinical research. Lancet. 1991 Apr 13;337(8746):867–872. doi: 10.1016/0140-6736(91)90201-y. [DOI] [PubMed] [Google Scholar]
  13. Engelland A. L., Alderman M. H., Powell H. B. Blood pressure control in private practice: a case report. Am J Public Health. 1979 Jan;69(1):25–29. doi: 10.2105/ajph.69.1.25. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Ettinger B., Grady D. The waning effect of postmenopausal estrogen therapy on osteoporosis. N Engl J Med. 1993 Oct 14;329(16):1192–1193. doi: 10.1056/NEJM199310143291610. [DOI] [PubMed] [Google Scholar]
  15. Gillum R. F., Neutra R. R., Stason W. B., Solomon H. S. Determinants of dropout rate among hypertensive patients in an urban clinic. J Community Health. 1979 Winter;5(2):94–100. doi: 10.1007/BF01324011. [DOI] [PubMed] [Google Scholar]
  16. Goodman C. M., Busuttil A., Chisholm G. D. Age, and size and grade of tumour predict prognosis in incidentally diagnosed carcinoma of the prostate. Br J Urol. 1988 Dec;62(6):576–580. doi: 10.1111/j.1464-410x.1988.tb04429.x. [DOI] [PubMed] [Google Scholar]
  17. Guyatt G., Jaeschke R., Heddle N., Cook D., Shannon H., Walter S. Basic statistics for clinicians: 1. Hypothesis testing. CMAJ. 1995 Jan 1;152(1):27–32. [PMC free article] [PubMed] [Google Scholar]
  18. Handley M. R., Stuart M. E. The use of prostate specific antigen for prostate cancer screening: a managed care perspective. J Urol. 1994 Nov;152(5 Pt 2):1689–1692. doi: 10.1016/s0022-5347(17)32362-5. [DOI] [PubMed] [Google Scholar]
  19. Henderson B. E., Paganini-Hill A., Ross R. K. Estrogen replacement therapy and protection from acute myocardial infarction. Am J Obstet Gynecol. 1988 Aug;159(2):312–317. doi: 10.1016/s0002-9378(88)80074-7. [DOI] [PubMed] [Google Scholar]
  20. Hlatky M. A. Patient preferences and clinical guidelines. JAMA. 1995 Apr 19;273(15):1219–1220. [PubMed] [Google Scholar]
  21. ISIS-4: a randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58,050 patients with suspected acute myocardial infarction. ISIS-4 (Fourth International Study of Infarct Survival) Collaborative Group. Lancet. 1995 Mar 18;345(8951):669–685. [PubMed] [Google Scholar]
  22. Jaglal S. B., Kreiger N., Darlington G. Past and recent physical activity and risk of hip fracture. Am J Epidemiol. 1993 Jul 15;138(2):107–118. doi: 10.1093/oxfordjournals.aje.a116833. [DOI] [PubMed] [Google Scholar]
  23. Johansson J. E., Adami H. O., Andersson S. O., Bergström R., Holmberg L., Krusemo U. B. High 10-year survival rate in patients with early, untreated prostatic cancer. JAMA. 1992 Apr 22;267(16):2191–2196. [PubMed] [Google Scholar]
  24. Johansson J. E., Adami H. O., Andersson S. O., Bergström R., Krusemo U. B., Kraaz W. Natural history of localised prostatic cancer. A population-based study in 223 untreated patients. Lancet. 1989 Apr 15;1(8642):799–803. doi: 10.1016/s0140-6736(89)92269-1. [DOI] [PubMed] [Google Scholar]
  25. Johansson J. E. Expectant management of early stage prostatic cancer: Swedish experience. J Urol. 1994 Nov;152(5 Pt 2):1753–1756. doi: 10.1016/s0022-5347(17)32378-9. [DOI] [PubMed] [Google Scholar]
  26. Jones G. W. Prospective, conservative management of localized prostate cancer. Cancer. 1992 Jul 1;70(1 Suppl):307–310. doi: 10.1002/1097-0142(19920701)70:1+<307::aid-cncr2820701319>3.0.co;2-n. [DOI] [PubMed] [Google Scholar]
  27. Kassirer J. P. Adding insult to injury. Usurping patients' prerogatives. N Engl J Med. 1983 Apr 14;308(15):898–901. doi: 10.1056/NEJM198304143081511. [DOI] [PubMed] [Google Scholar]
  28. Kassirer J. P. Incorporating patients' preferences into medical decisions. N Engl J Med. 1994 Jun 30;330(26):1895–1896. doi: 10.1056/NEJM199406303302611. [DOI] [PubMed] [Google Scholar]
  29. Lakka T. A., Venäläinen J. M., Rauramaa R., Salonen R., Tuomilehto J., Salonen J. T. Relation of leisure-time physical activity and cardiorespiratory fitness to the risk of acute myocardial infarction. N Engl J Med. 1994 Jun 2;330(22):1549–1554. doi: 10.1056/NEJM199406023302201. [DOI] [PubMed] [Google Scholar]
  30. Lasker R. D. The diabetes control and complications trial. Implications for policy and practice. N Engl J Med. 1993 Sep 30;329(14):1035–1036. doi: 10.1056/NEJM199309303291410. [DOI] [PubMed] [Google Scholar]
  31. Moher D. Clinical-trial registration: a call for its implementation in Canada. CMAJ. 1993 Dec 1;149(11):1657–1658. [PMC free article] [PubMed] [Google Scholar]
  32. Morris J. N., Clayton D. G., Everitt M. G., Semmence A. M., Burgess E. H. Exercise in leisure time: coronary attack and death rates. Br Heart J. 1990 Jun;63(6):325–334. doi: 10.1136/hrt.63.6.325. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Moskovitz B., Nitecki S., Richter Levin D. Cancer of the prostate: is there a need for aggressive treatment? Urol Int. 1987;42(1):49–52. doi: 10.1159/000281849. [DOI] [PubMed] [Google Scholar]
  34. Nease R. F., Jr, Kneeland T., O'Connor G. T., Sumner W., Lumpkins C., Shaw L., Pryor D., Sox H. C. Variation in patient utilities for outcomes of the management of chronic stable angina. Implications for clinical practice guidelines. Ischemic Heart Disease Patient Outcomes Research Team. JAMA. 1995 Apr 19;273(15):1185–1190. [PubMed] [Google Scholar]
  35. Nelson M. E., Fiatarone M. A., Morganti C. M., Trice I., Greenberg R. A., Evans W. J. Effects of high-intensity strength training on multiple risk factors for osteoporotic fractures. A randomized controlled trial. JAMA. 1994 Dec 28;272(24):1909–1914. doi: 10.1001/jama.1994.03520240037038. [DOI] [PubMed] [Google Scholar]
  36. Nyström L., Rutqvist L. E., Wall S., Lindgren A., Lindqvist M., Rydén S., Andersson I., Bjurstam N., Fagerberg G., Frisell J. Breast cancer screening with mammography: overview of Swedish randomised trials. Lancet. 1993 Apr 17;341(8851):973–978. doi: 10.1016/0140-6736(93)91067-v. [DOI] [PubMed] [Google Scholar]
  37. O'Meara J. J., 3rd, McNutt R. A., Evans A. T., Moore S. W., Downs S. M. A decision analysis of streptokinase plus heparin as compared with heparin alone for deep-vein thrombosis. N Engl J Med. 1994 Jun 30;330(26):1864–1869. doi: 10.1056/NEJM199406303302605. [DOI] [PubMed] [Google Scholar]
  38. Paffenbarger R. S., Jr, Hyde R. T., Wing A. L., Lee I. M., Jung D. L., Kampert J. B. The association of changes in physical-activity level and other lifestyle characteristics with mortality among men. N Engl J Med. 1993 Feb 25;328(8):538–545. doi: 10.1056/NEJM199302253280804. [DOI] [PubMed] [Google Scholar]
  39. Paganini-Hill A., Chao A., Ross R. K., Henderson B. E. Exercise and other factors in the prevention of hip fracture: the Leisure World study. Epidemiology. 1991 Jan;2(1):16–25. doi: 10.1097/00001648-199101000-00004. [DOI] [PubMed] [Google Scholar]
  40. Perez Cano R., Galan Galan F., Dilsen G. Risk factors for hip fracture in Spanish and Turkish women. Bone. 1993;14 (Suppl 1):S69–S72. doi: 10.1016/8756-3282(93)90353-c. [DOI] [PubMed] [Google Scholar]
  41. Peto R., Collins R., Gray R. Large-scale randomized evidence: large, simple trials and overviews of trials. J Clin Epidemiol. 1995 Jan;48(1):23–40. doi: 10.1016/0895-4356(94)00150-o. [DOI] [PubMed] [Google Scholar]
  42. Peto R., Collins R., Gray R. Large-scale randomized evidence: large, simple trials and overviews of trials. Ann N Y Acad Sci. 1993 Dec 31;703:314–340. doi: 10.1111/j.1749-6632.1993.tb26369.x. [DOI] [PubMed] [Google Scholar]
  43. Powell K. E., Thompson P. D., Caspersen C. J., Kendrick J. S. Physical activity and the incidence of coronary heart disease. Annu Rev Public Health. 1987;8:253–287. doi: 10.1146/annurev.pu.08.050187.001345. [DOI] [PubMed] [Google Scholar]
  44. Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ. 1992 Jul 4;305(6844):15–19. doi: 10.1136/bmj.305.6844.15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Reichard P., Nilsson B. Y., Rosenqvist U. The effect of long-term intensified insulin treatment on the development of microvascular complications of diabetes mellitus. N Engl J Med. 1993 Jul 29;329(5):304–309. doi: 10.1056/NEJM199307293290502. [DOI] [PubMed] [Google Scholar]
  46. Robinson A. Research, practice and the Cochrane Collaboration. CMAJ. 1995 Mar 15;152(6):883–889. [PMC free article] [PubMed] [Google Scholar]
  47. Shaper A. G., Wannamethee G., Weatherall R. Physical activity and ischaemic heart disease in middle-aged British men. Br Heart J. 1991 Nov;66(5):384–394. doi: 10.1136/hrt.66.5.384. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Smith D. S., Catalona W. J. The nature of prostate cancer detected through prostate specific antigen based screening. J Urol. 1994 Nov;152(5 Pt 2):1732–1736. doi: 10.1016/s0022-5347(17)32372-8. [DOI] [PubMed] [Google Scholar]
  49. Smith G. D., Song F., Sheldon T. A. Cholesterol lowering and mortality: the importance of considering initial level of risk. BMJ. 1993 May 22;306(6889):1367–1373. doi: 10.1136/bmj.306.6889.1367. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Stampfer M. J., Colditz G. A., Willett W. C., Manson J. E., Rosner B., Speizer F. E., Hennekens C. H. Postmenopausal estrogen therapy and cardiovascular disease. Ten-year follow-up from the nurses' health study. N Engl J Med. 1991 Sep 12;325(11):756–762. doi: 10.1056/NEJM199109123251102. [DOI] [PubMed] [Google Scholar]
  51. Whitmore W. F., Jr, Warner J. A., Thompson I. M., Jr Expectant management of localized prostatic cancer. Cancer. 1991 Feb 15;67(4):1091–1096. doi: 10.1002/1097-0142(19910215)67:4<1091::aid-cncr2820670437>3.0.co;2-j. [DOI] [PubMed] [Google Scholar]

Articles from CMAJ: Canadian Medical Association Journal are provided here courtesy of Canadian Medical Association

RESOURCES