Hitting the Mark: Reducing Bias in AI Systems

Bias can creep in at many stages of the deep-learning process, and the standard practices in computer science aren’t designed to detect it (1).

Imagine that you’ve taken up the sport of archery. With practice, you achieve consistency, but when you aim at the center of the target, the arrows land 5 cm to the right of center. What would you do?

Naturally, you’d adjust your aim. By aiming 5 cm to the left, you overcome your systematic bias, and your arrows now find their mark. Skilled archers know to adjust their aim to account for factors such as their own bias, the distance to the target, and the presence of crosswind.

The machine learning (ML) models that have begun to show success in medical imaging require even more careful attention to address the effects of bias. It’s well known that ML models are supremely adept at recognizing patterns. Often, though, the patterns that they learn can incorporate features that the systems’ authors never intended. We’ve laughed—and despaired—over the systems that diagnosed pneumonia based on a radiographic marker (2) or those that detected pneumothorax based on confounding image features, such as an inserted chest tube (3).

Such failures are examples of “shortcut learning,” where an ML system learns a pattern that fits its training data, without learning to generalize properly to the complexities of the real world (4). Because ML systems are trained on historical data, they also can learn shortcuts that are not only undesirable, but that incorporate historical prejudice. A résumé-screening system to select a technology company’s top job candidates was found to discriminate against women (5). A widely used algorithm to assign health care resources was found to be racially biased (6). Given that artificial intelligence (AI) systems can discern a patient’s race and sex from a chest radiograph (7), we must be attentive to the possibility that unintended biases in radiology ML systems could interact adversely with other health care AI models.

Several proposed approaches can help make ML models—and the data from which they’re derived—more transparent. “Model cards” can describe how a model performs across a variety of demographic or phenotypic groups; they help disclose the context in which the model is intended to be used (8). “Datasheets for datasets” can document the motivation, composition, collection process, and recommended uses of a given dataset; these datasheets have potential to increase transparency and accountability (9).

To address the particular challenges of bias in medical imaging AI systems, Dr Bradley Erickson and colleagues have produced a new collection of articles on “Mitigating Bias in Radiology Machine Learning” (https://pubs.rsna.org/page/ai/mitigating_bias). These articles describe approaches to help identify and reduce bias in radiology ML systems. Their articles address three principal areas where bias can impact an ML model: data handling (10), model development (11), and performance evaluation (12).

These articles will help us identify and mitigate potential biases in radiology ML systems, and thus better assure trustworthy results that are truly “on target” for the care of our patients.