Abstract
OBJECTIVE: Evaluate the accuracy of the detailed diagnostic reasoning of the Heart Failure Program incorporating a new mechanism to handle temporal relationships and severity constraints. DESIGN: Tools were developed to summarize diagnoses and automatically generate evaluation forms. Five expert cardiologists were asked to review the reasoning of the program, with two analyzing each case. Cases were gathered retrospectively for diversity and difficulty and 26 randomly selected cases were evaluated. The underlying issues were identified and classified. RESULTS: Both reviewers rated the first diagnosis correct in 25% of the cases and at least one rated it wrong in 10%. Analyzing the detailed reasoning, 137 issues were raised, about 5.3 per case. Of these, 53% were possible concerns raised by one reviewer. Of the 5.3 issues per case, 2.5 were attributable to controversies, misunderstandings, or mistakes; 1 was due to the overly simplistic representation of the summaries; and 1.8 were issues related to the program. CONCLUSION: Overall, the program is capable of providing high-quality detailed diagnostic hypotheses for complex cardiovascular cases. The results highlight several issues: 1) the difficulty of effectively summarizing hypotheses, 2) the nature of a physician's causal explanation, and 3) some problems in evaluating detailed diagnostic reasoning. The mistakes the program made imply that some additional refinement is needed but that the reasoning mechanisms developed can support the appropriate reasoning. The appropriate next step is a prospective evaluation addressing the program's usefulness.
Full Text
The Full Text of this article is available as a PDF (1.6 MB).
Selected References
These references are in PubMed. This may not be the complete list of references from this article.
- Hickam D. H., Shortliffe E. H., Bischoff M. B., Scott A. C., Jacobs C. D. The treatment advice of a computer-based cancer chemotherapy protocol advisor. Ann Intern Med. 1985 Dec;103(6 ):928–936. doi: 10.7326/0003-4819-103-6-928. [DOI] [PubMed] [Google Scholar]
- Kahn C. E., Jr Validation, clinical trial, and evaluation of a radiology expert system. Methods Inf Med. 1991 Oct;30(4):268–274. [PubMed] [Google Scholar]
- Long W. J., Naimi S., Criscitiello M. G. Development of a knowledge base for diagnostic reasoning in cardiology. Comput Biomed Res. 1992 Jun;25(3):292–311. doi: 10.1016/0010-4809(92)90044-b. [DOI] [PubMed] [Google Scholar]
- Long W. J. The probability of disease. Proc Annu Symp Comput Appl Med Care. 1991:619–623. [PMC free article] [PubMed] [Google Scholar]
- Miller R. A., Pople H. E., Jr, Myers J. D. Internist-1, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med. 1982 Aug 19;307(8):468–476. doi: 10.1056/NEJM198208193070803. [DOI] [PubMed] [Google Scholar]
- Yu V. L., Fagan L. M., Wraith S. M., Clancey W. J., Scott A. C., Hannigan J., Blum R. L., Buchanan B. G., Cohen S. N. Antimicrobial selection by a computer. A blinded evaluation by infectious diseases experts. JAMA. 1979 Sep 21;242(12):1279–1282. [PubMed] [Google Scholar]