Abstract
Objectives: A problem can arise when a performance indicator shows substantially more variability than would be expected by chance alone, since ignoring such "over-dispersion" could lead to a large number of institutions being inappropriately classified as "abnormal". A number of options for handling this phenomenon are investigated, ranging from improved risk stratification to fitting a statistical model that robustly estimates the degree of over-dispersion.
Design: Retrospective analysis of publicly available data on survival following coronary artery bypass grafts, emergency readmission rates, and teenage pregnancies.
Setting: NHS trusts in England.
Results: Funnel plots clearly show the influence of the method chosen for dealing with over-dispersion on the "banding" a trust receives. Both multiplicative and additive approaches are feasible and give intuitively reasonable results, but the additive random effects formulation appears to have a stronger conceptual foundation.
Conclusion: A random effects model may offer a reasonable solution. This method has now been adopted by the UK Healthcare Commission in their derivation of star ratings.
Full Text
The Full Text of this article is available as a PDF (199.1 KB).
Selected References
These references are in PubMed. This may not be the complete list of references from this article.
- Aylin Paul, Best Nicky, Bottle Alex, Marshall Clare. Following Shipman: a pilot system for monitoring mortality rates in primary care. Lancet. 2003 Aug 9;362(9382):485–491. doi: 10.1016/S0140-6736(03)14077-9. [DOI] [PubMed] [Google Scholar]
- Christiansen C. L., Morris C. N. Improving the statistical approach to health care provider profiling. Ann Intern Med. 1997 Oct 15;127(8 Pt 2):764–768. doi: 10.7326/0003-4819-127-8_part_2-199710151-00065. [DOI] [PubMed] [Google Scholar]
- DerSimonian R., Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986 Sep;7(3):177–188. doi: 10.1016/0197-2456(86)90046-2. [DOI] [PubMed] [Google Scholar]
- Mohammed M. A., Cheng K. K., Rouse A., Marshall T. Bristol, Shipman, and clinical governance: Shewhart's forgotten lessons. Lancet. 2001 Feb 10;357(9254):463–467. doi: 10.1016/s0140-6736(00)04019-8. [DOI] [PubMed] [Google Scholar]
- Spiegelhalter D. Funnel plots for institutional comparison. Qual Saf Health Care. 2002 Dec;11(4):390–391. doi: 10.1136/qhc.11.4.390-a. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Spiegelhalter David J. Funnel plots for comparing institutional performance. Stat Med. 2005 Apr 30;24(8):1185–1202. doi: 10.1002/sim.1970. [DOI] [PubMed] [Google Scholar]
- Spiegelhalter David J. Mortality and volume of cases in paediatric cardiac surgery: retrospective study based on routinely collected data. BMJ. 2002 Feb 2;324(7332):261–263. doi: 10.1136/bmj.324.7332.261. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stark J. F., Gallivan S., Davis K., Hamilton J. R., Monro J. L., Pollock J. C., Watterson K. G. Assessment of mortality rates for congenital heart defects and surgeons' performance. Ann Thorac Surg. 2001 Jul;72(1):169–175. doi: 10.1016/s0003-4975(01)02689-3. [DOI] [PubMed] [Google Scholar]
- Stark J., Gallivan S., Lovegrove J., Hamilton J. R., Monro J. L., Pollock J. C., Watterson K. G. Mortality rates after surgery for congenital heart defects in children and surgeons' performance. Lancet. 2000 Mar 18;355(9208):1004–1007. doi: 10.1016/s0140-6736(00)90001-1. [DOI] [PubMed] [Google Scholar]
- Tekkis Paris P., McCulloch Peter, Steger Adrian C., Benjamin Irving S., Poloniecki Jan D. Mortality control charts for comparing performance of surgical units: validation study using hospital mortality data. BMJ. 2003 Apr 12;326(7393):786–788. doi: 10.1136/bmj.326.7393.786. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.