Skip to main content
PMC home page
CMAJ : Canadian Medical Association Journal logoLink to CMAJ : Canadian Medical Association Journal
. 2000 Jun 27;162(13):1809–1813.

How valid are utilization review tools in assessing appropriate use of acute care beds?

N Kalant 1, M Berlinguet 1, J G Diodati 1, L Dragatakis 1, F Marcotte 1
PMCID: PMC1231368  PMID: 10906913

Abstract

BACKGROUND: Despite their widespread acceptance, utilization review tools, which were designed to assess the appropriateness of care in acute care hospitals, have not been well validated in Canada. The aim of this study was to assess the validity of 3 such tools--ISD (Intensity of service, Severity of illness, Discharge screens), AEP (Appropriateness Evaluation Protocol) and MCAP (Managed Care Appropriateness Protocol)--as determined by their agreement with the clinical judgement of a panel of experts. METHODS: The cases of 75 patients admitted to an acute cardiology service were reviewed retrospectively. The criteria of each utilization review tool were applied by trained reviewers to each day the patients spent in hospital. An abstract of each case prepared in a day-by-day format was evaluated independently by 3 cardiologists, using clinical judgement to decide the appropriateness of each day spent in hospital. RESULTS: The panel considered 92% of the admissions and 67% of the subsequent hospital days to be appropriate. The ISD underestimated the appropriateness rates of admission and subsequent days; the AEP and MCAP overestimated the appropriateness rate of subsequent days in hospital. The kappa statistic of overall agreement between tool and panel was 0.45 for ISD, 0.24 for MCAP and 0.25 for AEP, indicating poor to fair validity of the tools. INTERPRETATION: Published validation studies had average kappa values of 0.32-0.44 (i.e., poor to fair) for admission days and for subsequent days in hospital for the 3 tools. The tools have only a low level of validity when compared with a panel of experts, which raises serious doubts about their usefulness for utilization review.

Full Text

The Full Text of this article is available as a PDF (471.7 KB).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. Apolone G., Alfieri V., Braga A., Caimi V., Cestari C., Crespi V., Crosti P. F., De Filippi F., Gelosa M., Lanzi E. A survey of the necessity of the hospitalization day in an Italian teaching hospital. Qual Assur Health Care. 1991;3(1):1–9. doi: 10.1093/intqhc/3.1.1. [DOI] [PubMed] [Google Scholar]
  2. Ash A., Schwartz M., Payne S. M., Restuccia J. D. The Self-Adapting Focused Review System. Probability sampling of medical records to monitor utilization and quality of care. Med Care. 1990 Nov;28(11):1025–1039. [PubMed] [Google Scholar]
  3. Booth B. M., Ludke R. L., Wakefield D. S., Kern D. C., du Mond C. E. Relationship between inappropriate admissions and days of care: implications for utilization management. Hosp Health Serv Adm. 1991 Fall;36(3):421–437. [PubMed] [Google Scholar]
  4. Butler J. S., Barrett B. J., Kent G., Haire R., Parfrey P. S. Detection and classification of inappropriate hospital stay. Clin Invest Med. 1996 Aug;19(4):251–258. [PubMed] [Google Scholar]
  5. Coast J., Inglis A., Morgan K., Gray S., Kammerling M., Frankel S. The hospital admissions study in England: are there alternatives to emergency hospital admission? J Epidemiol Community Health. 1995 Apr;49(2):194–199. doi: 10.1136/jech.49.2.194. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. DeCoster C., Roos N. P., Carrière K. C., Peterson S. Inappropriate hospital use by patients receiving care for medical conditions: targeting utilization review. CMAJ. 1997 Oct 1;157(7):889–896. [PMC free article] [PubMed] [Google Scholar]
  7. Doyle M., Barrett B. J., McDonald J., McGrath J., Parfrey P. S. The efficiency of acute care bed utilization in Newfoundland and Labrador. Healthc Manage Forum. 1998 Fall;11(3):15–25. doi: 10.1016/S0840-4704(10)60665-5. [DOI] [PubMed] [Google Scholar]
  8. Gertman P. M., Restuccia J. D. The appropriateness evaluation protocol: a technique for assessing unnecessary days of hospital care. Med Care. 1981 Aug;19(8):855–871. [PubMed] [Google Scholar]
  9. Goldman R. L., Weir C. R., Turner C. W., Smith C. B. Validity of utilization management criteria for psychiatry. Am J Psychiatry. 1997 Mar;154(3):349–354. doi: 10.1176/ajp.154.3.349. [DOI] [PubMed] [Google Scholar]
  10. Goldstein J. C., Waterhouse N. J., Juin P., Evan G. I., Green D. R. The coordinate release of cytochrome c during apoptosis is rapid, complete and kinetically invariant. Nat Cell Biol. 2000 Mar;2(3):156–162. doi: 10.1038/35004029. [DOI] [PubMed] [Google Scholar]
  11. Houghton A., Bowling A., Jones I., Clarke K. Appropriateness of admission and the last 24 hours of hospital care in medical wards in an east London teaching group hospital. Int J Qual Health Care. 1996 Dec;8(6):543–553. doi: 10.1093/intqhc/8.6.543. [DOI] [PubMed] [Google Scholar]
  12. Inglis A. L., Coast J., Gray S. F., Peters T. J., Frankel S. J. Appropriateness of hospital utilization. The validity and reliability of the Intensity-Severity-Discharge Review System in a United Kingdom acute hospital setting. Med Care. 1995 Sep;33(9):952–957. [PubMed] [Google Scholar]
  13. Kaplow M., Charest S., Mayo N., Benaroya S. Managing patient length of stay better using an appropriateness tool. Healthc Manage Forum. 1998 Summer;11(2):13–20. doi: 10.1016/S0840-4704(10)60640-0. [DOI] [PubMed] [Google Scholar]
  14. Ludke R. L., MacDowell N. M., Booth B. M., Hunter S. A. Appropriateness of admissions and discharges among readmitted patients. Health Serv Res. 1990 Aug;25(3):501–525. [PMC free article] [PubMed] [Google Scholar]
  15. McDonnell J., Meijler A., Kahan J. P., Bernstein S. J., Rigter H. Panellist consistency in the assessment of medical appropriateness. Health Policy. 1996 Sep;37(3):139–152. doi: 10.1016/s0168-8510(96)90021-4. [DOI] [PubMed] [Google Scholar]
  16. Merom D., Shohat T., Harari G., Oren M., Green M. S. Factors associated with inappropriate hospitalization days in internal medicine wards in Israel: a cross-national survey. Int J Qual Health Care. 1998 Apr;10(2):155–162. doi: 10.1093/intqhc/10.2.155. [DOI] [PubMed] [Google Scholar]
  17. Nelson M. F., Christenson R. H. The focused review process: a utilization management firm's experience with length of stay guidelines. Jt Comm J Qual Improv. 1995 Sep;21(9):477–487. doi: 10.1016/s1070-3241(16)30174-2. [DOI] [PubMed] [Google Scholar]
  18. Pearson S. D., Margolis C. Z., Davis S., Schreier L. K., Sokol H. N., Gottlieb L. K. Is consensus reproducible? A study of an algorithmic guidelines development process. Med Care. 1995 Jun;33(6):643–660. [PubMed] [Google Scholar]
  19. Peiró S., Meneu R., Roselló M. L., Portella E., Carbonell-Sanchís R., Fernández C., Lázaro G., Llorens M. A., Martínez-Mas E., Moreno E. Validez del protocolo de evaluación del uso inapropiado de la hospitalización. Med Clin (Barc) 1996 Jun 22;107(4):124–129. [PubMed] [Google Scholar]
  20. Restuccia J. D., Payne S. M., Lenhart G., Constantine H. P., Fulton J. P. Assessing the appropriateness of hospital utilization to improve efficiency and competitive position. Health Care Manage Rev. 1987 Summer;12(3):17–27. doi: 10.1097/00004010-198701230-00006. [DOI] [PubMed] [Google Scholar]
  21. Restuccia J. D. The evolution of hospital utilization review methods in the United States. Int J Qual Health Care. 1995 Sep;7(3):253–260. doi: 10.1093/intqhc/7.3.253. [DOI] [PubMed] [Google Scholar]
  22. Shekelle P. G., Kahan J. P., Bernstein S. J., Leape L. L., Kamberg C. J., Park R. E. The reproducibility of a method to identify the overuse and underuse of medical procedures. N Engl J Med. 1998 Jun 25;338(26):1888–1895. doi: 10.1056/NEJM199806253382607. [DOI] [PubMed] [Google Scholar]
  23. Smith C. B., Goldman R. L., Martin D. C., Williamson J., Weir C., Beauchamp C., Ashcraft M. Overutilization of acute-care beds in Veterans Affairs hospitals. Med Care. 1996 Jan;34(1):85–96. doi: 10.1097/00005650-199601000-00007. [DOI] [PubMed] [Google Scholar]
  24. Smith H. E., Pryce A., Carlisle L., Jones J. M., Scarpello J., Pantin C. Appropriateness of acute medical admissions and length of stay. J R Coll Physicians Lond. 1997 Sep-Oct;31(5):527–532. [PMC free article] [PubMed] [Google Scholar]
  25. Strumwasser I., Paranjpe N. V., Ronis D. L., Share D., Sell L. J. Reliability and validity of utilization review criteria. Appropriateness Evaluation Protocol, Standardized Medreview Instrument, and Intensity-Severity-Discharge criteria. Med Care. 1990 Feb;28(2):95–111. doi: 10.1097/00005650-199002000-00001. [DOI] [PubMed] [Google Scholar]
  26. Tsang P., Severs M. P. A study of appropriateness of acute geriatric admissions and an assessment of the Appropriateness Evaluation Protocol. J R Coll Physicians Lond. 1995 Jul-Aug;29(4):311–314. [PMC free article] [PubMed] [Google Scholar]
  27. Wassertheil-Smoller S., Tobin J., Steingart R. Assessing the appropriateness of medical care. N Engl J Med. 1998 Nov 12;339(20):1478–1480. doi: 10.1056/NEJM199811123392014. [DOI] [PubMed] [Google Scholar]

Articles from CMAJ: Canadian Medical Association Journal are provided here courtesy of Canadian Medical Association

RESOURCES