Skip to main content
NCBI home page
Journal of the American Medical Informatics Association : JAMIA logoLink to Journal of the American Medical Informatics Association : JAMIA
. 1996 May-Jun;3(3):208–215. doi: 10.1136/jamia.1996.96310634

Relationships among performance scores of four diagnostic decision support systems.

E S Berner 1, J R Jackson 1, J Algina 1
PMCID: PMC116302  PMID: 8723611

Abstract

OBJECTIVE: To examine the relationships among different performance scores for each of four diagnostic decision support systems (DDSSs). Design: Intercorrelations among seven performance scores on a set of 105 cases for each of four DDSSs (DXplain, Iliad, Meditel, QMR) were computed. METHODS: The performance scores for each case reflected: 1) presence or absence of the case diagnosis in the DDSS knowledge base; 2) presence or absence of the correct diagnosis anywhere on the DDSS diagnosis list; 3) presence or absence of the correct diagnosis in the top ten diagnoses; 4) relevance of the DDSS diagnosis list; 5) comprehensiveness of the DDSS diagnosis list; 6) whether the DDSS suggested additional diagnoses to the experts' list; and 7) the length of the DDSS diagnosis list. Results: For all DDSSs, the two Correct Diagnosis scores (top ten and total list) were significantly related: 1) to the presence of the correct diagnosis in the knowledge base; 2) to the Comprehensiveness score; and 3) to each other. There were significant differences among the four DDSSs on the magnitude and/or direction of the relationships between: 1) the two Correct Diagnosis scores; 2) the Relevance and Length scores; and 3) the Relevance and Additional Diagnoses scores. CONCLUSION: The production of a correct diagnosis for a given case is not related to the number of diagnoses suggested by the DDSS and, across different DDSSs, is not consistently related to other measures of performance. These data indicate that multiple measures are needed to fully describe the performance of a DDSS.

Full Text

The Full Text of this article is available as a PDF (929.5 KB).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. Aliferis C. F., Miller R. A. On the heuristic nature of medical decision-support systems. Methods Inf Med. 1995 Mar;34(1-2):5–14. [PubMed] [Google Scholar]
  2. Bankowitz R. A., Lave J. R., McNeil M. A. A method for assessing the impact of a computer-based decision support system on health care outcomes. Methods Inf Med. 1992 Feb;31(1):3–10. [PubMed] [Google Scholar]
  3. Bankowitz R. A., McNeil M. A., Challinor S. M., Miller R. A. Effect of a computer-assisted general medicine diagnostic consultation service on housestaff diagnostic strategy. Methods Inf Med. 1989 Nov;28(4):352–356. [PubMed] [Google Scholar]
  4. Bankowitz R. A., McNeil M. A., Challinor S. M., Parker R. C., Kapoor W. N., Miller R. A. A computer-assisted medical diagnostic consultation service. Implementation and prospective evaluation of a prototype. Ann Intern Med. 1989 May 15;110(10):824–832. doi: 10.7326/0003-4819-110-10-824. [DOI] [PubMed] [Google Scholar]
  5. Barness L. A., Tunnessen W. W., Jr, Worley W. E., Simmons T. L., Ringe T. B., Jr Computer-assisted diagnosis in pediatrics. Am J Dis Child. 1974 Jun;127(6):852–858. doi: 10.1001/archpedi.1974.02110250078011. [DOI] [PubMed] [Google Scholar]
  6. Berner E. S. Paradigms and problem-solving: a literature review. J Med Educ. 1984 Aug;59(8):625–633. doi: 10.1097/00001888-198408000-00002. [DOI] [PubMed] [Google Scholar]
  7. Berner E. S., Webster G. D., Shugerman A. A., Jackson J. R., Algina J., Baker A. L., Ball E. V., Cobbs C. G., Dennis V. W., Frenkel E. P. Performance of four computer-based diagnostic systems. N Engl J Med. 1994 Jun 23;330(25):1792–1796. doi: 10.1056/NEJM199406233302506. [DOI] [PubMed] [Google Scholar]
  8. Feldman M. J., Barnett G. O. An approach to evaluating the accuracy of DXplain. Comput Methods Programs Biomed. 1991 Aug;35(4):261–266. doi: 10.1016/0169-2607(91)90004-d. [DOI] [PubMed] [Google Scholar]
  9. Heckerling P. S., Elstein A. S., Terzian C. G., Kushner M. S. The effect of incomplete knowledge on the diagnoses of a computer consultant system. Med Inform (Lond) 1991 Oct-Dec;16(4):363–370. doi: 10.3109/14639239109067658. [DOI] [PubMed] [Google Scholar]
  10. Lau L. M., Warner H. R. Performance of a diagnostic system (Iliad) as a tool for quality assurance. Comput Biomed Res. 1992 Aug;25(4):314–323. doi: 10.1016/0010-4809(92)90022-3. [DOI] [PubMed] [Google Scholar]
  11. Middleton B., Shwe M. A., Heckerman D. E., Henrion M., Horvitz E. J., Lehmann H. P., Cooper G. F. Probabilistic diagnosis using a reformulation of the INTERNIST-1/QMR knowledge base. II. Evaluation of diagnostic performance. Methods Inf Med. 1991 Oct;30(4):256–267. [PubMed] [Google Scholar]
  12. Miller R. A. Medical diagnostic decision support systems--past, present, and future: a threaded bibliography and brief commentary. J Am Med Inform Assoc. 1994 Jan-Feb;1(1):8–27. doi: 10.1136/jamia.1994.95236141. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Miller R. A., Pople H. E., Jr, Myers J. D. Internist-1, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med. 1982 Aug 19;307(8):468–476. doi: 10.1056/NEJM198208193070803. [DOI] [PubMed] [Google Scholar]
  14. Nelson S. J., Blois M. S., Tuttle M. S., Erlbaum M., Harrison P., Kim H., Winkelmann B., Yamashita D. Evaluating RECONSIDER. A computer program for diagnostic prompting. J Med Syst. 1985 Dec;9(5-6):379–388. doi: 10.1007/BF00992575. [DOI] [PubMed] [Google Scholar]
  15. O'Shea J. S. Computer-assisted pediatric diagnosis. Am J Dis Child. 1975 Feb;129(2):199–202. doi: 10.1001/archpedi.1975.02120390033007. [DOI] [PubMed] [Google Scholar]
  16. Schoolman H. M. Obligations of the expert system builder: meeting the needs of the user. MD Comput. 1991 Sep-Oct;8(5):316–321. [PubMed] [Google Scholar]
  17. Swender P. T., Tunnessen W. W., Jr, Oski F. A. Computer-assisted diagnosis. Am J Dis Child. 1974 Jun;127(6):859–861. doi: 10.1001/archpedi.1974.02110250085012. [DOI] [PubMed] [Google Scholar]
  18. Waxman H. S., Worley W. E. Computer-assisted adult medical diagnosis: subject review and evaluation of a new microcomputer-based system. Medicine (Baltimore) 1990 May;69(3):125–136. [PubMed] [Google Scholar]
  19. Wexler J. R., Swender P. T., Tunnessen W. W., Jr, Oski F. A. Impact of a system of computer-assisted diagnosis. Initial evaluation of the hospitalized patient. Am J Dis Child. 1975 Feb;129(2):203–205. doi: 10.1001/archpedi.1975.02120390037008. [DOI] [PubMed] [Google Scholar]

Articles from Journal of the American Medical Informatics Association are provided here courtesy of Oxford University Press

RESOURCES