Abstract
The literature on the performance evaluation of medical expert system is extensive, yet most of the techniques used in the early stages of system development are inappropriate for deployed expert systems. Because extensive clinical and informatics expertise and resources are required to perform evaluations, efficient yet effective methods of monitoring performance during the long-term maintenance phase of the expert system life cycle must be devised. Statistical process control techniques provide a well-established methodology that can be used to define policies and procedures for continuous, concurrent performance evaluation. Although the field of statistical process control has been developed for monitoring industrial processes, its tools, techniques, and theory are easily transferred to the evaluation of expert systems. Statistical process tools provide convenient visual methods and heuristic guidelines for detecting meaningful changes in expert system performance. The underlying statistical theory provides estimates of the detection capabilities of alternative evaluation strategies. This paper describes a set of statistical process control tools that can be used to monitor the performance of a number of deployed medical expert systems. It describes how p-charts are used in practice to monitor the GermWatcher expert system. The case volume and error rate of GermWatcher are then used to demonstrate how different inspection strategies would perform.
Full Text
The Full Text of this article is available as a PDF (1.2 MB).
Selected References
These references are in PubMed. This may not be the complete list of references from this article.
- Chamberlin W. H., Lane K. A., Kennedy J. N., Bradley S. D., Rice C. L. Monitoring intensive care unit performance using statistical quality control charts. Int J Clin Monit Comput. 1993 Oct;10(3):155–161. doi: 10.1007/BF01246449. [DOI] [PubMed] [Google Scholar]
- Giuse D. A., Giuse N. B., Miller R. A. Evaluation of long-term maintenance of a large medical knowledge base. J Am Med Inform Assoc. 1995 Sep-Oct;2(5):297–306. doi: 10.1136/jamia.1995.96073832. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hand R., Piontek F., Klemka-Walden L., Inczauskis D. Use of statistical control charts to assess outcomes of medical care: pneumonia in Medicare patients. Am J Med Sci. 1994 May;307(5):329–334. doi: 10.1097/00000441-199405000-00003. [DOI] [PubMed] [Google Scholar]
- Kahn M. G., Steib S. A., Dunagan W. C., Fraser V. J. Monitoring expert system performance using continuous user feedback. J Am Med Inform Assoc. 1996 May-Jun;3(3):216–223. doi: 10.1136/jamia.1996.96310635. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kahn M. G., Steib S. A., Fraser V. J., Dunagan W. C. An expert system for culture-based infection control surveillance. Proc Annu Symp Comput Appl Med Care. 1993:171–175. [PMC free article] [PubMed] [Google Scholar]
- Lee J. T. Statistical process control charts. Infect Control Hosp Epidemiol. 1994 Apr;15(4 Pt 1):223–224. [PubMed] [Google Scholar]
- Sellick J. A., Jr The use of statistical process control charts in hospital epidemiology. Infect Control Hosp Epidemiol. 1993 Nov;14(11):649–656. doi: 10.1086/646659. [DOI] [PubMed] [Google Scholar]
- Stead W. W., Haynes R. B., Fuller S., Friedman C. P., Travis L. E., Beck J. R., Fenichel C. H., Chandrasekaran B., Buchanan B. G., Abola E. E. Designing medical informatics research and library--resource projects to increase what is learned. J Am Med Inform Assoc. 1994 Jan-Feb;1(1):28–33. doi: 10.1136/jamia.1994.95236134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stead W. W. Matching the level of evaluation to a project's stage of development. J Am Med Inform Assoc. 1996 Jan-Feb;3(1):92–94. doi: 10.1136/jamia.1996.96342652. [DOI] [PMC free article] [PubMed] [Google Scholar]