Get Both the Medicine and the Informatics Right

The best medical informatics research is grounded in an understanding of both the medical content area and the informatics techniques. For example, it is not attention to the medical literature alone that makes medical informatics different from information science. The difference comes from the integration of medical knowledge and the strategies or structures needed to formalize or represent it.

The paper by Piniewski-Bond et al.¹ provides an example of the need for collaboration of informatics experts with application domain experts. It compares two methods for identifying the set of ideas contained in a test set of papers from the published literature. The researchers use the set of ideas contained in three published review papers as a gold standard, and they use the papers that are referenced in those three reviews as the test set. The reader will recognize two potential sources of error in this analysis. First, Piniewski-Bond et al. may not have correctly identified all the ideas contained in the three review papers. Second, the authors of those review papers may have included ideas based on their prior experience that did not come directly from any of the papers that they reference. Such limits are not unusual, and they do not constitute a reason to reject a manuscript for publication in a medical informatics journal as long as they are correctly stated in the manuscript.

The review of the manuscript pointed out a limitation that may not be as obvious. The subject of the three published review papers is the specific association between human papillomavirus and cervical cancer. The expert in that clinical area among the reviewers of this paper stated clearly that the three published reviews were not the best available on this clinical topic. Accordingly, neither the ideas contained in them nor the ideas identified by examining their reference constitute a valid review of this clinical problem. In other words, the paper provides an apt demonstration of an information science technique; however, when viewed from the perspective of medicine, the clinical content is sub-par. We accepted the paper for publication because the author states clearly that she is not trying to provide a review of the clinical topic, the reviewers indicate that the paper adds to the base of knowledge in informatics; and we judge the risk of a clinical reader mistaking this paper for a clinical review as remote.

Nonetheless, this limitation might have been easily avoided if a clinical subject expert had participated at the start of the project to identify a clinically valid test set. If the test set had been clinically valid, the quantitative data could be placed in clinical context. A reader could judge whether differences in results were significant clinically.

One reviewer explained the problem with a fishing example. “The crappie is a round, slow fish not known for much of anything, pretty easy to catch. The musky, on the other hand, is a sleek and wily fish that hides in weeds, very difficult to catch. Would you have confidence in your new fishing rod if it had been tested on crappie and you wanted to catch musky?”