Skip to main content
NCBI home page
Quality & Safety in Health Care logoLink to Quality & Safety in Health Care
. 2003 Apr;12(2):122–128. doi: 10.1136/qhc.12.2.122

Using routine comparative data to assess the quality of health care: understanding and avoiding common pitfalls

A Powell 1, H Davies 1, R Thomson 1
PMCID: PMC1743685  PMID: 12679509

Abstract



 Measuring the quality of health care has become a major concern for funders and providers of health services in recent decades. One of the ways in which quality of care is currently assessed is by taking routinely collected data and analysing them quantitatively. The use of routine data has many advantages but there are also some important pitfalls. Collating numerical data in this way means that comparisons can be made—whether over time, with benchmarks, or with other healthcare providers (at individual or institutional levels of aggregation). Inevitably, such comparisons reveal variations. The natural inclination is then to assume that such variations imply rankings: that the measures reflect quality and that variations in the measures reflect variations in quality. This paper identifies reasons why these assumptions need to be applied with care, and illustrates the pitfalls with examples from recent empirical work. It is intended to guide not only those who wish to interpret comparative quality data, but also those who wish to develop systems for such analyses themselves.

Full Text

The Full Text of this article is available as a PDF (110.1 KB).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. Adab Peymané, Rouse Andrew M., Mohammed Mohammed A., Marshall Tom. Performance league tables: the NHS deserves better. BMJ. 2002 Jan 12;324(7329):95–98. doi: 10.1136/bmj.324.7329.95. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bickell N. A., Chassin M. R. Determining the quality of breast cancer care: do tumor registries measure up? Ann Intern Med. 2000 May 2;132(9):705–710. doi: 10.7326/0003-4819-132-9-200005020-00004. [DOI] [PubMed] [Google Scholar]
  3. Black N. High-quality clinical databases: breaking down barriers. Lancet. 1999 Apr 10;353(9160):1205–1206. doi: 10.1016/S0140-6736(99)00108-7. [DOI] [PubMed] [Google Scholar]
  4. Bland J. M., Altman D. G. Some examples of regression towards the mean. BMJ. 1994 Sep 24;309(6957):780–780. doi: 10.1136/bmj.309.6957.780. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Campbell S. M., Braspenning J., Hutchinson A., Marshall M. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care. 2002 Dec;11(4):358–364. doi: 10.1136/qhc.11.4.358. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Crombie I. K., Davies H. T. Beyond health outcomes: the advantages of measuring process. J Eval Clin Pract. 1998 Feb;4(1):31–38. doi: 10.1046/j.1365-2753.1998.t01-1-00003.x. [DOI] [PubMed] [Google Scholar]
  7. Davies H. T., Crombie I. K. Assessing the quality of care. BMJ. 1995 Sep 23;311(7008):766–766. doi: 10.1136/bmj.311.7008.766. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Davies H. T., Crombie I. K. Interpreting health outcomes. J Eval Clin Pract. 1997 Aug;3(3):187–199. doi: 10.1046/j.1365-2753.1997.00003.x. [DOI] [PubMed] [Google Scholar]
  9. Davies H. T., Marshall M. N. Public disclosure of performance data: does the public get what the public wants? Lancet. 1999 May 15;353(9165):1639–1640. doi: 10.1016/s0140-6736(99)90047-8. [DOI] [PubMed] [Google Scholar]
  10. Davies H. T. Public release of performance data and quality improvement: internal responses to external data by US health care providers. Qual Health Care. 2001 Jun;10(2):104–110. doi: 10.1136/qhc.10.2.104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Davies Huw T. O., Washington A. Eugene, Bindman Andrew B. Health care report cards: implications for vulnerable patient groups and the organizations providing them care. J Health Polit Policy Law. 2002 Jun;27(3):379–399. doi: 10.1215/03616878-27-3-379. [DOI] [PubMed] [Google Scholar]
  12. Druss B., Rosenheck R. Evaluation of the HEDIS measure of behavioral health care quality. Health Plan Employer Data and Information Set. Psychiatr Serv. 1997 Jan;48(1):71–75. doi: 10.1176/ps.48.1.71. [DOI] [PubMed] [Google Scholar]
  13. Fielding J. E., Sutherland C. E., Halfon N. Community health report cards. Results of a national survey. Am J Prev Med. 1999 Jul;17(1):79–86. doi: 10.1016/s0749-3797(99)00033-1. [DOI] [PubMed] [Google Scholar]
  14. Florey C. D. Sample size for beginners. BMJ. 1993 May 1;306(6886):1181–1184. doi: 10.1136/bmj.306.6886.1181. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Giuffrida A., Gravelle H., Roland M. Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomes. BMJ. 1999 Jul 10;319(7202):94–98. doi: 10.1136/bmj.319.7202.94. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Goldacre Michael J., Griffith Myfanwy, Gill Leicester, Mackintosh Anne. In-hospital deaths as fraction of all deaths within 30 days of hospital admission for surgery: analysis of routine statistics. BMJ. 2002 May 4;324(7345):1069–1070. doi: 10.1136/bmj.324.7345.1069. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Green J., Wintfeld N. Report cards on cardiac surgeons. Assessing New York State's approach. N Engl J Med. 1995 May 4;332(18):1229–1232. doi: 10.1056/NEJM199505043321812. [DOI] [PubMed] [Google Scholar]
  18. Hartz A. J., Kuhn E. M. Comparing hospitals that perform coronary artery bypass surgery: the effect of outcome measures and data sources. Am J Public Health. 1994 Oct;84(10):1609–1614. doi: 10.2105/ajph.84.10.1609. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Hayes R. J. Methods for assessing whether change depends on initial value. Stat Med. 1988 Sep;7(9):915–927. doi: 10.1002/sim.4780070903. [DOI] [PubMed] [Google Scholar]
  20. Hofer T. P., Hayward R. A. Can early re-admission rates accurately detect poor-quality hospitals? Med Care. 1995 Mar;33(3):234–245. doi: 10.1097/00005650-199503000-00003. [DOI] [PubMed] [Google Scholar]
  21. Hofer T. P., Hayward R. A., Greenfield S., Wagner E. H., Kaplan S. H., Manning W. G. The unreliability of individual physician "report cards" for assessing the costs and quality of care of a chronic disease. JAMA. 1999 Jun 9;281(22):2098–2105. doi: 10.1001/jama.281.22.2098. [DOI] [PubMed] [Google Scholar]
  22. Hofer T. P., Hayward R. A. Identifying poor-quality hospitals. Can hospital mortality rates detect quality problems for medical diagnoses? Med Care. 1996 Aug;34(8):737–753. doi: 10.1097/00005650-199608000-00002. [DOI] [PubMed] [Google Scholar]
  23. Iezzoni L. I., Davis R. B., Palmer R. H., Cahalane M., Hamel M. B., Mukamal K., Phillips R. S., Banks N. J., Davis D. T., Jr Does the Complications Screening Program flag cases with process of care problems? Using explicit criteria to judge processes. Int J Qual Health Care. 1999 Apr;11(2):107–118. doi: 10.1093/intqhc/11.2.107. [DOI] [PubMed] [Google Scholar]
  24. Iezzoni L. I., Shwartz M., Ash A. S., Mackiernan Y. D. Does severity explain differences in hospital length of stay for pneumonia patients? J Health Serv Res Policy. 1996 Apr;1(2):65–76. doi: 10.1177/135581969600100204. [DOI] [PubMed] [Google Scholar]
  25. Iezzoni L. I. The risks of risk adjustment. JAMA. 1997 Nov 19;278(19):1600–1607. doi: 10.1001/jama.278.19.1600. [DOI] [PubMed] [Google Scholar]
  26. Iezzoni L. I. Using risk-adjusted outcomes to assess clinical practice: an overview of issues pertaining to risk adjustment. Ann Thorac Surg. 1994 Dec;58(6):1822–1826. doi: 10.1016/0003-4975(94)91721-3. [DOI] [PubMed] [Google Scholar]
  27. Jenkins K. J., Newburger J. W., Lock J. E., Davis R. B., Coffman G. A., Iezzoni L. I. In-hospital mortality for surgical repair of congenital heart defects: preliminary observations of variation by hospital caseload. Pediatrics. 1995 Mar;95(3):323–330. [PubMed] [Google Scholar]
  28. Komaromy M., Lurie N., Osmond D., Vranizan K., Keane D., Bindman A. B. Physician practice style and rates of hospitalization for chronic medical conditions. Med Care. 1996 Jun;34(6):594–609. doi: 10.1097/00005650-199606000-00009. [DOI] [PubMed] [Google Scholar]
  29. Lawrance R. A., Dorsch M. F., Sapsford R. J., Mackintosh A. F., Greenwood D. C., Jackson B. M., Morrell C., Robinson M. B., Hall A. S. Use of cumulative mortality data in patients with acute myocardial infarction for early detection of variation in clinical practice: observational study. BMJ. 2001 Aug 11;323(7308):324–327. doi: 10.1136/bmj.323.7308.324. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Leyland A. H., Boddy F. A. League tables and acute myocardial infarction. Lancet. 1998 Feb 21;351(9102):555–558. doi: 10.1016/S0140-6736(97)09362-8. [DOI] [PubMed] [Google Scholar]
  31. Majeed A., Bardsley M., Morgan D., O'Sullivan C., Bindman A. B. Cross sectional study of primary care groups in London: association of measures of socioeconomic and health status with hospital admission rates. BMJ. 2000 Oct 28;321(7268):1057–1060. doi: 10.1136/bmj.321.7268.1057. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Majeed F. A., Cook D. G., Anderson H. R., Hilton S., Bunn S., Stones C. Using patient and general practice characteristics to explain variations in cervical smear uptake rates. BMJ. 1994 May 14;308(6939):1272–1276. doi: 10.1136/bmj.308.6939.1272. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Mannion Russell, Davies Huw T. O. Reporting health care performance: learning from the past, prospects for the future. J Eval Clin Pract. 2002 May;8(2):215–228. doi: 10.1046/j.1365-2753.2002.00331.x. [DOI] [PubMed] [Google Scholar]
  34. Mant J., Hicks N. Detecting differences in quality of care: the sensitivity of measures of process and outcome in treating acute myocardial infarction. BMJ. 1995 Sep 23;311(7008):793–796. doi: 10.1136/bmj.311.7008.793. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Marshall E. C., Spiegelhalter D. J. Reliability of league tables of in vitro fertilisation clinics: retrospective analysis of live birth rates. BMJ. 1998 Jun 6;316(7146):1701–1705. doi: 10.1136/bmj.316.7146.1701. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Marshall M., Davies H. Public release of information on quality of care: how are health services and the public expected to respond? J Health Serv Res Policy. 2001 Jul;6(3):158–162. doi: 10.1258/1355819011927413. [DOI] [PubMed] [Google Scholar]
  37. McCarthy E. P., Iezzoni L. I., Davis R. B., Palmer R. H., Cahalane M., Hamel M. B., Mukamal K., Phillips R. S., Davies D. T., Jr Does clinical evidence support ICD-9-CM diagnosis coding of complications? Med Care. 2000 Aug;38(8):868–876. doi: 10.1097/00005650-200008000-00010. [DOI] [PubMed] [Google Scholar]
  38. Mohammed M. A., Cheng K. K., Rouse A., Marshall T. Bristol, Shipman, and clinical governance: Shewhart's forgotten lessons. Lancet. 2001 Feb 10;357(9254):463–467. doi: 10.1016/s0140-6736(00)04019-8. [DOI] [PubMed] [Google Scholar]
  39. Morgan K., Prothero D., Frankel S. The rise in emergency admissions--crisis or artefact? Temporal analysis of health services data. BMJ. 1999 Jul 17;319(7203):158–159. doi: 10.1136/bmj.319.7203.158. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Noseworthy J. H., Ebers G. C., Vandervoort M. K., Farquhar R. E., Yetisir E., Roberts R. The impact of blinding on the results of a randomized, placebo-controlled multiple sclerosis clinical trial. Neurology. 1994 Jan;44(1):16–20. doi: 10.1212/wnl.44.1.16. [DOI] [PubMed] [Google Scholar]
  41. Nutley S., Smith P. C. League tables for performance improvement in health care. J Health Serv Res Policy. 1998 Jan;3(1):50–57. doi: 10.1177/135581969800300111. [DOI] [PubMed] [Google Scholar]
  42. Park R. E., Brook R. H., Kosecoff J., Keesey J., Rubenstein L., Keeler E., Kahn K. L., Rogers W. H., Chassin M. R. Explaining variations in hospital death rates. Randomness, severity of illness, quality of care. JAMA. 1990 Jul 25;264(4):484–490. [PubMed] [Google Scholar]
  43. Posnett J. Is bigger better? Concentration in the provision of secondary care. BMJ. 1999 Oct 16;319(7216):1063–1065. doi: 10.1136/bmj.319.7216.1063. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Rainwater J. A., Romano P. S., Antonius D. M. The California Hospital Outcomes Project: how useful is California's report card for quality improvement? Jt Comm J Qual Improv. 1998 Jan;24(1):31–39. doi: 10.1016/s1070-3241(16)30357-1. [DOI] [PubMed] [Google Scholar]
  45. Reid F. D., Cook D. G., Majeed A. Explaining variation in hospital admission rates between general practices: cross sectional study. BMJ. 1999 Jul 10;319(7202):98–103. doi: 10.1136/bmj.319.7202.98. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Romano P. S., Rainwater J. A., Antonius D. Grading the graders: how hospitals in California and New York perceive and interpret their report cards. Med Care. 1999 Mar;37(3):295–305. doi: 10.1097/00005650-199903000-00009. [DOI] [PubMed] [Google Scholar]
  47. Shwartz M., Iezzoni L. I., Ash A. S., Mackiernan Y. D. Do severity measures explain differences in length of hospital stay? The case of hip fracture. Health Serv Res. 1996 Oct;31(4):365–385. [PMC free article] [PubMed] [Google Scholar]
  48. Sowden A. J., Deeks J. J., Sheldon T. A. Volume and outcome in coronary artery bypass graft surgery: true association or artefact? BMJ. 1995 Jul 15;311(6998):151–155. doi: 10.1136/bmj.311.6998.151. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Spiegelhalter David J. Mortality and volume of cases in paediatric cardiac surgery: retrospective study based on routinely collected data. BMJ. 2002 Feb 2;324(7332):261–263. doi: 10.1136/bmj.324.7332.261. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Stange K. C., Zyzanski S. J., Smith T. F., Kelly R., Langa D. M., Flocke S. A., Jaén C. R. How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patients visits. Med Care. 1998 Jun;36(6):851–867. doi: 10.1097/00005650-199806000-00009. [DOI] [PubMed] [Google Scholar]
  51. Thomas J. W., Hofer T. P. Accuracy of risk-adjusted mortality rate as a measure of hospital quality of care. Med Care. 1999 Jan;37(1):83–92. doi: 10.1097/00005650-199901000-00012. [DOI] [PubMed] [Google Scholar]
  52. Wakefield D. S., Hendryx M. S., Uden-Holman T., Couch R., Helms C. M. Comparing providers' performance: problems in making the "report card" analogy fit. J Healthc Qual. 1996 Nov-Dec;18(6):4–10. doi: 10.1111/j.1945-1474.1996.tb00863.x. [DOI] [PubMed] [Google Scholar]

Articles from Quality & safety in health care are provided here courtesy of BMJ Publishing Group

RESOURCES