Abstract
OBJECTIVE: To evaluate the applicability of metrics collected during routine use to monitor the performance of a deployed expert system. METHODS: Two extensive formal evaluations of the GermWatcher (Washington University School of Medicine) expert system were performed approximately six months apart. Deficiencies noted during the first evaluation were corrected via a series of interim changes to the expert system rules, even though the expert system was in routine use. As part of their daily work routine, infection control nurses reviewed expert system output and changed the output results with which they disagreed. The rate of nurse disagreement with expert system output was used as an indirect or surrogate metric of expert system performance between formal evaluations. The results of the second evaluation were used to validate the disagreement rate as an indirect performance measure. Based on continued monitoring of user feedback, expert system changes incorporated after the second formal evaluation have resulted in additional improvements in performance. RESULTS: The rate of nurse disagreement with GermWatcher output decreased consistently after each change to the program. The second formal evaluation confirmed a marked improvement in the program's performance, justifying the use of the nurses' disagreement rate as an indirect performance metric. CONCLUSIONS: Metrics collected during the routine use of the GermWatcher expert system can be used to monitor the performance of the expert system. The impact of improvements to the program can be followed using continuous user feedback without requiring extensive formal evaluations after each modification. When possible, the design of an expert system should incorporate measures of system performance that can be collected and monitored during the routine use of the system.
Full Text
The Full Text of this article is available as a PDF (795.4 KB).
Selected References
These references are in PubMed. This may not be the complete list of references from this article.
- Broderick A., Mori M., Nettleman M. D., Streed S. A., Wenzel R. P. Nosocomial infections: validation of surveillance and computer modeling to identify patients at risk. Am J Epidemiol. 1990 Apr;131(4):734–742. doi: 10.1093/oxfordjournals.aje.a115558. [DOI] [PubMed] [Google Scholar]
- Emori T. G., Culver D. H., Horan T. C., Jarvis W. R., White J. W., Olson D. R., Banerjee S., Edwards J. R., Martone W. J., Gaynes R. P. National nosocomial infections surveillance system (NNIS): description of surveillance methods. Am J Infect Control. 1991 Feb;19(1):19–35. doi: 10.1016/0196-6553(91)90157-8. [DOI] [PubMed] [Google Scholar]
- Garner J. S., Jarvis W. R., Emori T. G., Horan T. C., Hughes J. M. CDC definitions for nosocomial infections, 1988. Am J Infect Control. 1988 Jun;16(3):128–140. doi: 10.1016/0196-6553(88)90053-3. [DOI] [PubMed] [Google Scholar]
- Hripcsak G., Clayton P. D. User comments on a clinical event monitor. Proc Annu Symp Comput Appl Med Care. 1994:636–640. [PMC free article] [PubMed] [Google Scholar]
- Hripcsak G. Monitoring the monitor: automated statistical tracking of a clinical event monitor. Comput Biomed Res. 1993 Oct;26(5):449–466. doi: 10.1006/cbmr.1993.1032. [DOI] [PubMed] [Google Scholar]
- Kahn M. G., Steib S. A., Fraser V. J., Dunagan W. C. An expert system for culture-based infection control surveillance. Proc Annu Symp Comput Appl Med Care. 1993:171–175. [PMC free article] [PubMed] [Google Scholar]
- Wyatt J., Spiegelhalter D. Evaluating medical expert systems: what to test and how? Med Inform (Lond) 1990 Jul-Sep;15(3):205–217. doi: 10.3109/14639239009025268. [DOI] [PubMed] [Google Scholar]