Abstract
OBJECTIVE: Evaluate the performance of a continuous-speech interface to a decision support system. DESIGN: The authors performed a prospective evaluation of a speech interface that matches unconstrained utterances of physicians with controlled-vocabulary terms from Quick Medical Reference (QMR). The performance of the speech interface was assessed in two stages: in the real-time experiment, physician subjects viewed audiovisual stimuli intended to evoke clinical findings, spoke a description of each finding into the speech interface, and then chose from a list generated by the interface the QMR term that most closely matched the finding. Subjects believed that the speech recognizer decoded their utterances; in reality, a hidden experimenter typed utterances into the interface (Wizard-of-Oz experimental design). Later, the authors replayed the same utterances through the speech recognizer and measured how accurately utterances matched with appropriate QMR terms using the results of the real-time experiment as the "gold standard." MEASUREMENTS: The authors measured how accurately the speech-recognition system converted input utterances to text strings (recognition accuracy) and how accurately the speech interface matched input utterances to appropriate QMR terms (semantic accuracy). RESULTS: Overall recognition accuracy was less than 50%. However, using language-processing techniques that match keywords in recognized utterances to keywords in QMR terms, the semantic accuracy of the system was 81%. CONCLUSIONS: Reasonable semantic accuracy was attained when language-processing techniques were used to accommodate for speech misrecognition. In addition, the Wizard-of-Oz experimental design offered many advantages for this evaluation. The authors believe that this technique may be useful to future evaluators of speech-input systems.
Full Text
The Full Text of this article is available as a PDF (1.4 MB).
Selected References
These references are in PubMed. This may not be the complete list of references from this article.
- Forsythe D. E., Buchanan B. G. Broadening our approach to evaluating medical information systems. Proc Annu Symp Comput Appl Med Care. 1991:8–12. [PMC free article] [PubMed] [Google Scholar]
- Isaacs E., Wulfman C. E., Rohn J. A., Lane C. D., Fagan L. M. Graphical access to medical expert systems: IV. Experiments to determine the role of spoken input. Methods Inf Med. 1993 Feb;32(1):18–32. [PubMed] [Google Scholar]
- Johnson K., Poon A., Shiffman S., Lin R., Fagan L. A history-taking system that uses continuous speech recognition. Proc Annu Symp Comput Appl Med Care. 1992:757–761. [PMC free article] [PubMed] [Google Scholar]
- Kuhn K., Gaus W., Wechsler J. G., Janowitz P., Tudyka J., Kratzer W., Swobodnik W., Ditschuneit H. Structured reporting of medical findings: evaluation of a system in gastroenterology. Methods Inf Med. 1992 Nov;31(4):268–274. [PubMed] [Google Scholar]
- Massey B. T., Geenen J. E., Hogan W. J. Evaluation of a voice recognition system for generation of therapeutic ERCP reports. Gastrointest Endosc. 1991 Nov-Dec;37(6):617–620. doi: 10.1016/s0016-5107(91)70866-3. [DOI] [PubMed] [Google Scholar]
- Miller R. A., McNeil M. A., Challinor S. M., Masarie F. E., Jr, Myers J. D. The INTERNIST-1/QUICK MEDICAL REFERENCE project--status report. West J Med. 1986 Dec;145(6):816–822. [PMC free article] [PubMed] [Google Scholar]
- Shiffman S., Detmer W. M., Lane C. D., Fagan L. M. A continuous-speech interface to a decision support system: I. Techniques to accommodate for misrecognized input. J Am Med Inform Assoc. 1995 Jan-Feb;2(1):36–45. doi: 10.1136/jamia.1995.95202546. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shiffman S., Lane C. D., Johnson K. B., Fagan L. M. The integration of a continuous-speech-recognition system with the QMR diagnostic program. Proc Annu Symp Comput Appl Med Care. 1992:767–771. [PMC free article] [PubMed] [Google Scholar]
- Wyatt J. C., Detmer W. M., Fagan L. M. Design and evaluation of multimedia stimuli to evoke clinical concepts. Proc Annu Symp Comput Appl Med Care. 1993:834–838. [PMC free article] [PubMed] [Google Scholar]
- Wyatt J., Spiegelhalter D. Evaluating medical expert systems: what to test and how? Med Inform (Lond) 1990 Jul-Sep;15(3):205–217. doi: 10.3109/14639239009025268. [DOI] [PubMed] [Google Scholar]