Commentary: Has pathology gone to the “birds” because we have just been “winging” it?

The recently published article about training the homing pigeon (Columba livia) to recognize benign from malignant pathology breast cancer images has received much attention.[1,2] Therefore, it is worthy of pointing out what the hype is all about and what can be inferred from the results in this era of digital pathology. Although the study also involved radiology images, this commentary will focus primarily on the pathology aspect.

The authors of this paper set out to answer the following questions, namely (1) Can pigeons be trained to discriminate between benign and malignant pathology images? (2) Once trained, can these pigeons accurately perform this task when confronted with a novel set of similar images? (3) How well would they perform when given extremely difficult images? and (4) If successful, are such skills of any practical use?

The investigators trained their pigeons using a food-reward system, to distinguish benign from malignant hematoxylin and eosin (H & E) stained human breast histopathology cases depicted in static medical images. The images (308 × 308 pixels) were presented to the birds on a 15-inch liquid crystal display touchscreen monitor [Figure 1]. The dataset was intermixed with some images that had been rotated. When a bird correctly categorized an image as benign or malignant, they were rewarded with pigeon pellets fed into a nearby cup. Their responses were recorded by having them peck on a report button (blue or yellow) that appeared to the left and right of the breast image [Figure 2]. After training trials, the animals underwent a test regimen where the session was comprised of both the original training images (with some images rotated) and a set of novel exemplars that had never been seen before. The study was conducted using images at low (×4), medium (×10), and high (×20) levels of magnification. This experiment was captured in an abridged video available on YouTube (http://www.youtu.be/flzGjnJLyS0); experimental procedures were approved by an institutional animal care and use committee.

Figure 1.

A pigeon being trained on an image dataset of cases with breast histopathology, displayed on a touchscreen monitor

Figure 2.

A pigeon correctly selecting the blue (“malignant”) response button for an image depicting invasive breast carcinoma

Remarkably, these promising “avian pathologists” were able to generalize what they had learned. They accurately discriminated malignant from benign breast images at multiple levels of magnification and in different spatial orientations. The pigeons’ accuracy increased from being no better than a coin flip (50% correct) on day 1 of the experiment to near 85% by days 13–15 (P = 0.001). Being trained at one magnification primed them to perform at the next magnification. When confronted with unseen images, the birds were able to accurately classify these novel cases. This signifies that they had not merely memorized images but rather that they were able to detect feature-based cues to classify the images. The birds did misidentify a few challenging images where malignant cases displayed similarities to benign images, were hypocellular, or also contained benign breast lobular structures in the image. Like pathologists who perform better by getting together with colleagues and arriving at a consensus, the birds were more accurate in a group (i.e., flock score proved to be better than individual scores). In fact, when a group of four birds was shown images, their “flock” accuracy level reached 99%. In addition, the birds only had a choice of benign and malignant. A third category—atypical—was not available to them, but this is a common diagnosis used by their human counterparts.

The study further demonstrated that image manipulation affected a pigeons’ discrimination performance. Accuracy to correctly recognize a breast image was modestly affected when the birds were shown monochrome (grayscale) but hue-normalized images, where color and luminance cues were now eliminated. Image brightness and contrast levels were adjusted because, in general, cancer samples (with increased cells that tend to be hyperchromatic) were darker and had higher contrast than benign images. Nevertheless, even when presented with manipulated images, the birds were still able to learn and generalize, suggesting that morphology and/or texture differences alone were probably sufficient for them to accurately classify. Thus, perhaps color fidelity may not be as important in digital pathology as we currently believe it to be. Image compression, using medium (15:1) or high (27:1) JPEG compression, similarly resulted in a slight reduction in the pigeon's accuracy. However, when appropriately trained, these birds were able to adapt to lossy compression in the images.

It is well known that the homing pigeon has an innate ability to find its way home over long distances. This skill is due to several of its unique global positioning system equivalent features (e.g., using magnetic fields, visual cues, odors, etc.).[3] Homing pigeons also have larger brains in comparison to other nonhoming pigeon breeds.[2] Hopefully, this is true for pathologists as well, as we too are quite good at diagnosing breast cancer based on images! Levenson et al.[1] elected to work with pigeons because they are phenomenal discriminators of visual stimuli. In fact, pigeons have been trained to visually discriminate many other types of images, including photographs of cat and dog faces[4] to paintings by Monet and Picasso.[5] Of interest, pigeons were able to only discriminate upside-down images of Picasso's paintings.

The authors suggest that their study has several potential benefits.[1] For example, pigeons as fitting surrogates to human observers could be used in image perception studies or help validate innovations in medical imaging for quality and reliability, instead of relying on computer-aided substitutes. Of great interest is the potential to elucidate key elements involved in visual discrimination tasks that may help guide the development of computer-assisted medical image recognition tools. The computational mechanisms of pattern recognition in pigeons have been well studied.[6,7] Several processes appear to be involved including visual, associative, and cognitive mechanisms. Pigeons are able to extract a wide variety of visual clues from images including pattern, color, size, shape, and texture. The aforementioned pigeon-based experiment parallels current machine-learning approaches that employ labeling of features (e.g., texture) in training data (so-called supervised learning) to then be generalizable for subsequent image analysis. Computer vision including machine-learning is an active area of research and growth in pathology informatics. Further work like this, with pigeons, will hopefully help us “home in” on better solutions for clinical pathology practice.