Abstract
Little has been done to examine the relative merit of measures used to assess the impact of diagnostic decision support systems (DDSS) on physician performance. In this study, 10 different single-measures of diagnostic performance were compared empirically. The measures were of three types: rank-order, all-or-none, and appropriateness. The responsiveness (RESP) of each measure was estimated under two repeated-measures experimental conditions. RESP is the degree to which a measure could detect differences between conditions of low and high performance. The diagnostic performance of 108 physicians was compared on medical cases of varying diagnostic difficulty and with or without a high level of assistance from a DDSS. The results showed that the RESP among the measures varied nearly tenfold. The rank-order measures tended to provide the highest RESP values (maximum = 2.14) while appropriateness measures provided the lowest RESP values (maximum = 1.41). The most responsive measures were rank-orders of the correct diagnosis within the top 5 to 10 listed diagnoses.
Full text
PDF
Selected References
These references are in PubMed. This may not be the complete list of references from this article.
- Beaton D. E., Hogg-Johnson S., Bombardier C. Evaluating changes in health status: reliability and responsiveness of five generic health status measures in workers with musculoskeletal disorders. J Clin Epidemiol. 1997 Jan;50(1):79–93. doi: 10.1016/s0895-4356(96)00296-x. [DOI] [PubMed] [Google Scholar]
- Berner E. S., Jackson J. R., Algina J. Relationships among performance scores of four diagnostic decision support systems. J Am Med Inform Assoc. 1996 May-Jun;3(3):208–215. doi: 10.1136/jamia.1996.96310634. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Berner E. S., Maisiak R. S., Cobbs C. G., Taunton O. D. Effects of a decision support system on physicians' diagnostic performance. J Am Med Inform Assoc. 1999 Sep-Oct;6(5):420–427. doi: 10.1136/jamia.1999.0060420. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Berner E. S., Maisiak R. S. Influence of case and physician characteristics on perceptions of decision support systems. J Am Med Inform Assoc. 1999 Sep-Oct;6(5):428–434. doi: 10.1136/jamia.1999.0060428. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Berner E. S., Webster G. D., Shugerman A. A., Jackson J. R., Algina J., Baker A. L., Ball E. V., Cobbs C. G., Dennis V. W., Frenkel E. P. Performance of four computer-based diagnostic systems. N Engl J Med. 1994 Jun 23;330(25):1792–1796. doi: 10.1056/NEJM199406233302506. [DOI] [PubMed] [Google Scholar]
- Friedman C. P., Elstein A. S., Wolf F. M., Murphy G. C., Franz T. M., Heckerling P. S., Fine P. L., Miller T. M., Abraham V. Enhancement of clinicians' diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. JAMA. 1999 Nov 17;282(19):1851–1856. doi: 10.1001/jama.282.19.1851. [DOI] [PubMed] [Google Scholar]
- Friedman C., Elstein A., Wolf F., Murphy G., Franz T., Fine P., Heckerling P., Miller T. Measuring the quality of diagnostic hypothesis sets for studies of decision support. Stud Health Technol Inform. 1998;52(Pt 2):864–868. [PubMed] [Google Scholar]
- Guyatt G., Walter S., Norman G. Measuring change over time: assessing the usefulness of evaluative instruments. J Chronic Dis. 1987;40(2):171–178. doi: 10.1016/0021-9681(87)90069-5. [DOI] [PubMed] [Google Scholar]
- Liang M. H., Larson M. G., Cullen K. E., Schwartz J. A. Comparative measurement efficiency and sensitivity of five health status instruments for arthritis research. Arthritis Rheum. 1985 May;28(5):542–547. doi: 10.1002/art.1780280513. [DOI] [PubMed] [Google Scholar]
- Roth S. H. Role of the rheumatologist in 1995: leadership. J Rheumatol. 1995 Jan;22(1):1–2. [PubMed] [Google Scholar]
- Wright J. G., Young N. L. A comparison of different indices of responsiveness. J Clin Epidemiol. 1997 Mar;50(3):239–246. doi: 10.1016/s0895-4356(96)00373-3. [DOI] [PubMed] [Google Scholar]