Fig. 5. The visualization based on Gradient-weighted Class Activation Mapping (Grad-CAM) method of the proposed model in predicting 4-category molecular subtypes of breast cancer.

Case a–d indicate the category of Luminal A, Luminal B, HER2-enriched and Triple-negative, respectively. MG mammography, CC craniocaudal, MLO mediolateral oblique, US ultrasound, MulR Multi-ResNet, SA self-attention, iiSA intra- and inter-self-attention, CSA channel and spatial attention, interSA inter self-attention, interCSA inter channel and spatial attention, MDL-IIA multi-modal deep learning with intra- and inter-modality attention modules.