Interpretability of AI race detection model in medical imaging with saliency methods

Abstract

Deep neural networks (DNNs) are powerful tools for classifying images. Using these convolutional models for medical images is challenging due to their complexity and large number of parameters, making it hard to find clinically meaningful explanations for their decisions. To overcome the opaqueness inherent to such models, saliency techniques suggest generating maps that highlight the regions of an image important for the DNN's prediction. DNN models have shown the capability of race detection from medical images of different modalities, which is concerning as they under-diagnose patients from historically under-served races.

The objective of this paper is to use explainability methods to detect subtle bias that DNNs use to detect a patient's race from chest X-rays. Toward this end, we apply eight state-of-the-art methods and propose to evaluate their effectiveness. We demonstrate that the salient region's size is crucial to understanding network behavior. When the salient region covers 30% of the image, we find that only the Rise method is effective at locating salient areas, as it can both accurately predict a patient's race on chest X-ray images on its own and mislead the network on race detection when removed. We, therefore, note that saliency maps in the medical field should be used with caution, as there is no available ground truth, and the network may occasionally employ low-level image features to compute predictions.

Graphical abstract

1. Introduction

Artificial intelligence (AI), particularly Deep Neural Network (DNN) models, holds great promise for medical diagnosis. Numerous studies have demonstrated AI applications in a variety of tasks, including regression analysis [1], cancer segmentation [2], and medical diagnostics [3], [4], [5]. These algorithms have shown fast processing and strong performance, and it is speculated that they might support physicians in enhancing medical care through accelerated diagnoses at reduced costs and lower clinical workloads.

Unfortunately, DNNs are often perceived as a “black box” by medical experts because they lack simple mechanisms to substantiate diagnostic or prognostic decisions [6], [7]. If a model indicates that a patient is not likely to suffer from a disorder, this may result in less follow-up care for the patient and a progression of the disease to the point where it cannot be managed [8]. This is particularly important in scenarios where clinicians are unsure of the decision they should make and rely on the DNN algorithm as a second opinion.

As a result, these models should be utilized carefully, ensuring that they provide fair and reliable diagnoses [9], [10], [11], [12], [13]. Here, we define unfairness as differences in performance against or in favor of a subpopulation for a predictive task, e.g., lower performance on disease diagnosis in one racial, age, or sex group compared to another.

DNNs' performance can indeed be biased. Certain features may be underrepresented due to sampling bias, while certain classes may be misclassified due to framing bias [14]. Framing bias arises from human decisions influenced by how information is presented rather than the factual information itself. There are various types of bias. For example, bias in medical applications can be characterized by a strong likelihood of diagnosing a condition based on race or sex. It can also result from biases in the images, such as the presence of medical devices or labeling [8], [15], [16]. Additionally, scanner information can be used as features, particularly if a subpopulation is more likely to use a specific type of scanner [12], [17].

A recent study published by Gichoya et al. [15] has shown that AI models can predict patients' race with very high accuracy across multiple imaging modalities, diverse tasks, and datasets, surpassing the capabilities of human radiologists. This is especially concerning given that DNN models can be unfair to certain groups in disease diagnosis [9]. For example, they may disproportionately misreport Black patients as healthy compared to White patients, leading to critical care denial for underserved populations [8]. This kind of problem can occur if the model uses race information to predict disease detection, as clinicians may not appropriately assess the relevance of the patient's race in the final diagnosis.

Interpretability techniques can provide insight into how a black box model's prediction is produced. Attribution methods create saliency maps to identify the most important area of an image that contributed to a prediction. These maps are used to ‘explain’ the output of the model, i.e., by highlighting the most salient part(s) of the image.

Studies have shown that saliency methods yield different maps for the same image [18]. This variability may result from the numerous biases inherent in an image [18]. In this study, we define the optimal saliency method as one that can identify an area of an image which, when removed, misleads the network's prediction, while still accurately predicting the image when used independently.

Following the paper “AI recognition of patient race in medical imaging: a modeling study” by Gichoya et al. [15], which showed that deep neural networks (DNNs) could predict race from chest X-rays without being explicitly trained to do so, we aimed to investigate the areas in the images used by AI systems to predict race using saliency methods. Gichoya et al. [15] employed the Grad-CAM saliency method to identify regions of interest that contributed to the network's discrimination but found no salient regions. We decided to further investigate the network's behavior using eight state-of-the-art saliency methods and corroborated the findings with an atlas-based method.

2. Related work

2.1. From bias to unfairness in AI

When attributes and features are statistically connected to a class in the classification process but are not causally connected, the dataset is deemed biased. The model could rely on these spurious features to perform the classification task, focusing on them instead of the class-specific features it is supposed to classify. Different forms of bias exist. For example, image background is one of the variables that create bias [19]. There is a location bias in widely used datasets like ImageNet and Microsoft COCO [20], as the object of interest is typically centered in the image [21], [22], which may lead to suboptimal performance when detecting objects that are not centered. Another issue is the texture of the images [23]. In rare instances where images originate from another source, these circumstances may result in poor performance. Biased datasets can cause fairness issues in medical applications; recent studies have reported that factors such as age, gender, and ethnicity may lead machine learning algorithms to be biased toward certain groups of people [8], [9], [15], [24], [25], [26], [27]. This behavior contradicts principles of human justice, fairness, and ethics [28]. Addressing this concern is challenging due to the numerous potential sources of bias, including the task's design and data selection (for example, face recognition can be racially discriminatory [24]). The model may learn these biases because it is trained to use any features present in an image to optimize a cost function, regardless of fairness.

In this paper, we investigate the lack of fairness exposed in the study of Gichoya et al. [15].

2.2. AI recognition of patient race in medical imaging

In the study by Gichoya et al. [15], the authors demonstrate how DNNs can categorize race using several medical imaging modalities. They explored potential causes for the network's predictions but were unable to identify any proxies, anatomical causes, or cofactors that could explain the algorithm's success in race identification. Since we cannot guarantee that none of the racial traits will be used by DNNs for illness diagnosis, race detection by DNNs in medical imaging may be problematic if relied upon by clinicians. In a setting where all patients should receive the same level of care, justice and equality of treatment could be called into question.

To explore how DNNs detect a patient's race, the authors experimented with the Grad-CAM saliency method [29] to investigate potential anatomical locations associated with racial disparities. The results were inconclusive, as they could not identify any consistent areas on the images that reliably classified race.

In this work, we build on their study by investigating various saliency methods and anatomical atlases to better understand the neural network's behavior in race classification, and we suggest anatomical areas where race information might be present.

2.3. Saliency methods

Saliency methods are designed to visually interpret neural networks for a given input image. They generate an importance map that highlights the most salient areas influencing the network's prediction. Modern saliency methods can be broadly classified into two main categories: attribution methods and gradient-based methods.

Gradient based methods are computed by performing forward or backward propagation of the network's output or layer gradient to the input image. The initial approach used gradients [30], and later advancements included integrating gradients [31], which combine gradients with inputs to provide a linear interpolation of the images. Artifacts can be reduced using SmoothGrad [32], which introduces random noise to the input. While performing gradient calculations, methods such as DeconvNet [33] and Guided Backpropagation [34] restrict the output to positive values only.

Despite the variations in these methods, they all produce maps with salient areas scattered across the entire image (as opposed to a single connected region). Therefore, one would have to assume that the information used by the model is dispersed throughout the image rather than coming from a specific area or organ. The Grad-CAM approach [29] combines the input characteristics with the gradient of a particular layer to produce a more condensed output. However, since this approach upsamples the deepest layer of the network, the resulting map may include pixels that are not informative for predicting the image class.

Perturbation based methods perturb the input image to identify the region that most influences the output prediction. Occlusion and Rise [35] methods occlude the image using patches and measure the change in the output prediction. Meaningful perturbation [36] and extremal perturbation [37] methods optimize a mask to identify the smaller region that most affects the network's prediction.

In comparison, the RankPix approach [38] finds the smallest region that alters the prediction for each layer and averages all maps to generate the final salient mask. To create a more reliable and comprehensive map, Saliency Enhancing with Scaling and Sliding (SESS) [39] suggests fusing saliency maps of prominent portions of the image.

3. Methods and experiments

3.1. Data and models

The investigation utilized the MIMIC-CXR open-source chest X-ray dataset [40], which included information about the patient's race. For the study, a random selection of 200 Asian, 1,000 Black, and 1,000 White images was made from the test set. Any images containing visible medical devices or other clinically significant findings were excluded after visual inspection. A ResNet34 model was employed to classify the Asian, Black, and White racial groups into three classes using logistic regression [15]. The validation results for the AUC of the Asian, Black, and White groups on 10,000 scans were 0.92, 0.93, and 0.95, respectively. The test set produced results of 0.94, 0.97, and 0.97 for the same groups.

3.2. Atlas creation

To perform a group analysis of saliency masks, we constructed an overall atlas for all subjects, as well as separate atlases for each racial group. All the images were registered and then averaged. Below, we describe the process for constructing each of the atlases.

The Advanced Normalization Tools (ANTs) package [41] was used to perform rigid, affine, and non-linear registration in sequence. First, a random image was selected from each Group as the starting atlas, and all other images in the group were registered to that atlas. The resulting registered images were averaged to create a new atlas. This process was repeated five times for each of the rigid, affine, and non-linear deformations. A visual quality check was performed to remove any images where registration failed (e.g., obvious misregistration or unrealistic anatomical deformation). The final result was an atlas for each Group (right image of Fig. 1).

Fig. 1.

Atlas: average of all the scans after the Rigid, the Affine and the SyNRA deformation (from left to right). The right image represents the final atlas.

3.3. Saliency masks

In our study, we applied saliency methods to interpret the ResNet34 network, which achieved a successful AUC of 96% in detecting race from chest X-rays. We computed and compared eight state-of-the-art methods: DeconvNet, GradCAM, Gradient, Guided Backpropagation, Linear Approximation, Extremal Perturbation, and Rise to generate saliency maps for each image. The implementation of these methods was carried out using the TorchRay package [37]. Additionally, we utilized the RankPix maps technique as described in a separate publication [38].

For each method, a threshold was applied to convert the saliency map of a subject into a binary mask. Given the limited size of the Asian cohort, we focus our analysis on the White and Black racial groups.

Research by Fong et al. [37] highlights that using the 15% most informative pixels of an image can be as effective as using the entire image. Additionally, they found that perturbing 20% of an image's pixels produces a similar effect to perturbing the entire image, while retaining 25% of the most informative regions leads to higher model predictions. Based on these findings and considering that our study employs visualization methods based on image perturbation, we used two thresholds—15% and 30% of the image pixels—to identify the minimal mask that most accurately predicts the image.

3.4. Analysis of the saliency masks

We used the diffeomorphic deformation fields, computed to produce the atlas (Section 3.2), to register each subject's saliency mask to the corresponding atlas. The saliency masks for each group were then averaged to generate group-specific average saliency maps (see Fig. 4).

Fig. 4.

Average saliency maps for each race and each method.

Next, we inversely transformed the group-specific average binary mask back into each subject's original image space, creating a binary mask specific to each patient. This mask was then used to reclassify the image in two ways: (1) by retaining only the pixels within the mask and setting all pixels outside it to 0, and (2) by excluding the mask region, graying the pixels within the mask to a value of 0.5.

To compare the effectiveness of different saliency methods in identifying predictive regions, we used the AUC metric with the “Black vs. White” classification task. An ideal method would achieve an AUC of 1 when using its mask alone, and drop to 0.5 when classifying based solely on pixels outside the mask, indicating random prediction.

3.5. Anatomical difference between the races

The atlas procedure resulted in an atlas for the entire population, as well as separate atlases for each of the race groups. We subtracted atlases between pairs of groups to visualize the differences between White and Black, White and Asian, and Black and Asian. The resulting difference maps were normalized and are shown in Fig. 2.

Fig. 2.

Difference between each race, White and Black (left), the White and Asian (middle) and the Asian and Black (right) atlas.

To quantitatively compare the differences between averages, we first computed the mean and standard deviation for each pixel across randomly formed groups with an equal number of images in each group. For example, to compare the White and Black groups, we randomly formed 1,000 pseudo pairs, each consisting of 50 White and 50 Black images, and computed their differences. The hypothesis is that the difference between the averages of two groups, each containing 50 White and 50 Black images, should be zero. The mean and standard deviation over 1,000 such permutations were then computed, resulting in a map of mean and standard deviation for each pixel. The actual difference between 100 White and 100 Black images was then offset by the mean and divided by the standard deviation, producing a difference in Z-score units, as shown in Fig. 3.

Fig. 3.

Zscore of each race atlas (White and Black (left), the White and Asian (middle) and the Asian and Black (right) atlas.

4. Results

4.1. Anatomical difference between the races

Fig. 2 illustrates the difference between the atlases of each pair of Groups. In the figure, the areas highlighted in red indicate brighter regions, while blue represents darker areas. The differences were highly significant, with large areas above and below 5 standard deviations, as shown in Fig. 3.

When comparing the White and Black races, it is noticeable that the trachea, shoulder, and neck appear brighter for the White race, suggesting that there is more X-ray absorption or denser tissues present in those areas for the White and Asian scans compared to the Black scans. Similarly, the area below the diaphragm was found to be significantly brighter for the Black race compared to both the Asian and, especially, the White races.

Fig. 3 illustrates the Z-score difference between the atlases of each pair of Groups and highlights their significant distinctions.

4.2. Saliency maps

Fig. 4 shows the averaged saliency of the eight state-of-the-art saliency methods for each of the three Groups. We chose to construct the map for different areas using the extremal perturbation method at 5%, 10%, 20%, 30%, and 40%, as the method is area-specific. When the area exceeds 20%, the salient region is around the lungs. However, for smaller values, the approach highlights the trachea for both the Asian and Black scans, and the lung for the White scan. DeconvNet, Gradient, Guided Backpropagation, and RankPix all provide comparable maps for the three races. All approaches appear to agree that the trachea, portions of the shoulders, and parts of the lungs are the most salient locations for predicting the classes.

4.3. Saliency mask threshold at 15%

We thresholded all average saliency methods at 15% of their most salient pixels (see Fig. 5) and computed the registered mask for each subject. For the majority of the saliency methods, the trachea is identified as containing the most prominent regions in both the Black and White races.

Fig. 5.

Average saliency maps threshold at 15% for each race and each method.

Thereafter, the AUC was calculated as “Black vs. White.” Fig. 6 shows the original AUC of 0.98 using the whole image, as well as the results when using only the pixels within the mask or outside the mask. With the exception of Gradcam, Linear, and perturbation at 5% and 20%, all the methods had a value greater than 0.7, implying that they were sufficient for the network to predict the image correctly on their own. However, the orange bars demonstrate that when the pixels within the masks were removed, the network could still predict the race, albeit with reduced performance. Only the GuidedBackpropagation and RankPix methods exhibited the expected decrease in performance between the pixels within and outside the mask. The other methods showed the surprising opposite behavior: performance was much better when using the pixels outside the saliency mask compared to when using the pixels within it.

Fig. 6.

Performance of the DNN for race prediction when the saliencies were thresholded to 15%. The AUC metric (Black vs. White) is presented. The green plot represents the AUC of the original images, the blue bars show the AUC of the saliency mask thresholded at 15% of their most salient pixels, and the orange bars represent the AUC of the images when the most salient region of each method was removed. The maps were generated using the TorchRay package, except for the RankPix method.

We then investigated if bigger masks would make a difference.

4.3.1. Saliency mask threshold at 30%

The top 30% of pixels for each saliency approach are shown in Fig. 7. With this threshold, the methods have identified a greater number of prominent pixels, now covering a larger portion of the lungs.

Fig. 7.

Average saliency maps threshold at 30% for each race and each method.

4.4. Quantitative (AUC) results for saliencies threshold at 30%

Fig. 8 displays the AUC of the original scans in green, along with the AUC of the mask created from each method's 30% most prominent pixels in blue bars. The orange bars represent the AUC when these pixels are removed from the images.

Fig. 8.

Performance on the DNN for race prediction when the saliencies are thresholded at 30%. The AUC metric (Black vs White) is shown. The green plot represents the AUC of the original images, the blue bars show the AUC of the saliency mask thresholded at 30% of the most salient pixels, and the orange bars represent the AUC when removing the most salient region for each method. The maps were generated using the TorchRay package, except for the RankPix method.

When computing the prediction using only the masks, all methods except for Perturbation have an AUC above 0.75, suggesting that these regions are sufficient to predict the image on their own. However, when removing the mask, the Rise, Gradcam, and Linear methods achieved AUC scores of 0.5, 0.53, and 0.57, respectively, indicating performance close to random guessing.

Surprisingly, the perturbation techniques exhibited the opposite pattern, with the exception of case 10. For instance, Perturbation 40 showed that the network was misled by the mask, resulting in a random prediction. However, when the mask was removed, the network's prediction was just as accurate as when using the original image.

When the threshold was set at 30%, other methods, such as the gradient-based techniques (Guided Backpropagation, Gradient, DeconvNet), and RankPix, did not produce reliable salient regions for race prediction.

Since only the Rise, GradCAM, and Linear approaches provided accurate maps at this threshold, we focus on discussing their qualitative explanations in this section.

Fig. 7 shows the similarities between the maps generated by GradCAM and Linear, with both methods highlighting the trachea and lungs. However, there were differences: Black scans displayed more pronounced neck areas and fully visible lungs, while White scans highlighted only the left shoulder and the inner part of the right lung.

In Section 4.3, we demonstrated that the AUC for the perturbation methods at 5% and 20% was comparable to the original prediction when the salient region was removed. Additionally, we observed that the GradCAM and Linear maps were subregions of the Perturbation methods when their respective masks were removed, supporting the idea that the background of the scans does not hold saliency when considering the 30% most important pixels.

Among the saliency maps, the Rise map stood out at the 30% threshold. It revealed distinct differences between the White and Black races. The White race showed accentuated features on both shoulders, the right lung, a portion of the trachea, and the upper lung, while the Black race highlighted the right lung, a portion of the trachea, the upper lung, and a small patch at the bottom of the chest. Notably, the Rise saliency map also emphasized scan discontinuities that were relevant to racial discrimination.

5. Low-frequency features investigation

Since only the Rise method reliably identified a region of interest for race detection, covering 30% of the image area, we hypothesize that the network may rely on low-frequency features, causing race information to be dispersed across the entire image. This aligns with findings in “Reading Race: AI Recognizes Patient's Racial Identity in Medical Images,” which suggest that racial information in medical images may be encoded in subtle, low-level patterns.

To explore this hypothesis, we analyzed the network's reliance on low-frequency features by computing the DC gain across its CNN filters. The DC gain quantifies a filter's response to constant input values, serving as an indicator of its sensitivity to low-frequency or uniform image features. Our analysis revealed that all filters in the network exhibited non-zero DC gain, suggesting that the network effectively incorporates low-frequency information into its predictions.

While this finding suggests that subtle, distributed features may contribute to the network's race prediction, it represents only a preliminary avenue of investigation. Further studies are necessary to fully understand the role of low-frequency features in race prediction. Nonetheless, the primary focus of this paper remains on the use of visualization techniques, such as saliency maps, to interpret AI models in medical imaging.

6. Discussion

We examined the reliability of state-of-the-art saliency approaches in detecting bias using a DNN race classifier for chest X-ray images. We proposed calculating and comparing the AUC for various saliency map thresholds across several methods.

Importance of the Size of the Salient Region. In this study, we demonstrate that the threshold for the salient mask is a crucial parameter for identifying a region in the image that both predicts the class and misleads the network when removed. We observe a noticeable morphological difference between the Black and White scans. Additionally, we show that a meaningful explanation for race identification in scans requires a sufficiently large region of the image. This finding suggests that the network may be relying on low-level characteristics for its predictions. Since saliency methods assume that the network focuses on a specific region for its decision-making, it remains unclear whether the network relies on a distinct discriminative area or not. One might question whether the prominent region identified by gradient-based methods is the sole area relevant to the network's prediction.

Which Method Interprets the Race Problem the Best? In this study, we analyze several saliency algorithms and show that when the mask is set to 30

For the methods with lower performance, different regions of the image are emphasized. All methods highlight the trachea as a salient location, except for Guided Backpropagation, which focuses on the lung. Deconvnet, on the other hand, primarily highlights the right side of the scan. While we focused on the most salient pixels, Deconvnet and other gradient-based methods highlight all pixels without distinguishing between their importance. A possible explanation for this is that the gradient methods assign nearly equal importance to their salient pixels, resulting in minimal variation across all highlighted areas. Thresholding these methods to their most salient pixels can introduce bias into their AUC scores.

The Rise method best explains the network's behavior by generating a fragmented saliency map with several prominent regions. According to the study by Konate et al. [38], its saliency map is dispersed across the entire image, with its barycenter localized in the middle. This distribution makes it difficult to detect obvious and localized biases in images. Techniques like Perturbation or Rankpix, which produce localized and compact maps, may not accurately explain the network's behavior if the network relies on low-frequency features to determine the subject's race.

Indeed, Perturbation 40 cannot predict the images correctly using its mask alone, as its AUC value is 0.5, as shown in Fig. 8. However, when the mask is removed, the scan's race is still correctly identified. This result aligns with the findings of Rise and Gradcam methods, which largely do not overlap with Perturbation 40. The Rise method's ability to generate a useful map strengthens the hypothesis that, with enough pixels, it is capable of detecting low-frequency features.

On the other hand, GradCam and Linear suggest that the network used both the lung and trachea to predict Black and White races, with the distinction that the shoulder is emphasized for White scans, while the upper section of the trachea is more prominent for Black scans.

From the Atlas Difference to the Saliency Maps The difference between the atlases revealed a significant distinction between the Black and White scans, with the most notable variance observed in the upper part of the scans and the trachea. These findings are consistent with the saliency maps, which highlight that the network relied on these regions to predict the race of the scans. White scans exhibit higher tissue density than Black scans, potentially leading to a difference in intensity between the two. This intensity difference may serve as a simple proxy that the network utilized to predict race from X-ray scans. However, it is important to note that the underlying causes of these intensity differences were not explored in this study.

Should Saliency Methods Always Be Used for Explaining DNNs in the Medical Imaging Field? This study highlights the challenge of pinpointing the region responsible for race predictions in chest X-ray scans using saliency methods. While methods like Rise, Gradcam, and Linear succeed and achieve high performance when the salient region is sufficiently large, the identified region varies across methods, complicating the task of providing a reliable anatomical explanation. The study underscores the importance of caution when applying saliency techniques to deep neural network interpretation in medical image predictions. Each method emphasizes different regions of interest and highlights external attributes, such as the size of the area of interest, making it difficult to draw consistent and trustworthy conclusions.

We utilize the AUC measure to assess the accuracy of the saliency approaches. The purpose of all saliency approaches is to identify the most salient region in an image that contributes to the network's prediction. However, these methods vary in their application, raising the question of what the salient region truly represents. Does it reflect the most important area of the image for prediction, network activity, layer behavior, or perhaps a combination of these factors? Since saliency methods are grounded in computer vision and can be context-, network-, and task-dependent, they should be used with caution in real-world applications where ground truth is unavailable. Some methods may lack robustness, particularly when applied to specific domains such as healthcare.

Saliency techniques operate under the assumption that, similar to humans, the network bases its prediction on a specific feature. However, this is not always the case. Saliency maps may instead highlight a biomarker that does not represent the true discriminatory features for the network. The network might rely on highly predictable features that humans cannot easily perceive, such as low-frequency patterns. This is exemplified in the study by Ribeiro et al. [19], where a model identifying huskies and wolves relied on snow as a feature rather than the more relevant traits of the animals. In our study, the network may leverage unique low-frequency features of each race, even though they may be imperceptible to human vision. The assumption that different races have distinct anatomical chest structures is difficult to substantiate in our case study of a race classifier for chest X-rays, especially since there is no clear anatomical difference observed in the literature or by clinicians. Furthermore, the ground truth labels are based on self-identification, which may not always align with biological traits. For example, a person identifying as White may biologically be bi-racial. In this context, the learned features could be very low-frequency and highly dependent on the absolute gray levels in image regions—something that humans are unlikely to use for classification, as human vision is more sensitive to contrast than intensity.

Impact of Low-Frequency Features in a Network. The impact of low-frequency features in neural networks is significant, as they often cover larger portions of the image. Our findings suggest that the extent of saliency, which inherently includes low-frequency components, correlates with improved predictions in the network. Interestingly, humans are more sensitive to contrast than to intensity, which makes low-frequency details less noticeable to our visual system. Given this, it becomes crucial to incorporate data augmentation techniques to prevent the network from learning irrelevant gray-level patterns that might emerge from these low-frequency features. By understanding the role of low-frequency components and the limitations of human perception, we can enhance the network's robustness, especially in tasks like medical imaging that rely heavily on image analysis.

Real-World Applications of Interpretability Methods. These findings have significant implications for real-world clinical applications, particularly in ensuring fairness and accountability in AI-assisted medical decision-making. By systematically evaluating saliency methods, our study provides critical insights into the reliability of interpretability tools commonly used in medical imaging. These tools can be integrated into clinical workflows to identify and mitigate biases in AI models, ensuring robust performance across diverse patient populations. For instance, interpretability techniques could be employed during model validation to detect demographic-specific biases, or in real-time to provide clinicians with transparency regarding the regions of medical images driving AI predictions. Furthermore, our findings emphasize the need for rigorous evaluation of interpretability methods before deployment, encouraging the development of clinical AI systems that not only improve diagnostic accuracy but also uphold equity and trustworthiness in healthcare settings.

Future work. Saliency approaches, as well as other interpretability methods built using medical images, should be further explored in future studies to better understand and detect biases underlying race classification networks and to address fairness issues when using AI for analyzing chest X-ray images. Further investigation of network behavior using additional interpretability techniques is necessary to validate our findings and gain deeper insights into how deep neural networks (DNNs) successfully identify race from chest X-ray scans. Additionally, this research could help develop strategies to prevent such biases from influencing AI predictions.

7. Conclusion

Saliency methods are valuable tools for interpreting machine learning models in the medical imaging domain, but they should be applied with caution. In this study, the Rise method performed the best, as it was able to identify a region salient enough to mislead the network when removed. The study also highlights the lack of robustness and faithfulness in some saliency methods, showing that the most recent approaches do not necessarily yield the best results in this context. This issue may arise because attribution methods are often designed for computer vision applications, which typically involve more distinct semantic features compared to medical images. Developing task- and model-specific interpretability methods would be the most effective way to analyze these models. Additionally, this study underscores that neural networks can learn to discriminate between groups based on low-frequency features, such as intensities. As a result, data augmentation becomes essential to enhance the robustness of deep neural networks (DNNs) when applied in the medical imaging field.

CRediT authorship contribution statement

Salamata Konate: Writing – original draft, Visualization, Validation, Methodology, Investigation, Conceptualization. Léo Lebrat: Writing – review & editing, Validation, Methodology, Investigation. Rodrigo Santa Cruz: Writing – review & editing, Validation, Methodology, Investigation. Judy Wawira Gichoya: Validation, Resources. Brandon Price: Writing – review & editing, Resources. Laleh Seyyed-Kalantari: Writing – review & editing, Visualization, Validation, Resources. Clinton Fookes: Writing – review & editing, Visualization, Validation, Supervision. Andrew Bradley: Writing – review & editing, Visualization, Validation, Supervision, Methodology, Investigation, Conceptualization. Olivier Salvado: Writing – review & editing, Visualization, Validation, Supervision, Methodology, Conceptualization.

Declaration of Competing Interest

The authors declare that there are no conflicts of interest related to the submission of this paper.