Figure 3.

The t-SNE visualization of the progress of the latent space embeddings during the training process for the single CTPA and EHR data modalities (top and middle rows respectively) and multimodality (bottom row). Features were extracted from the start (after only one epoch), middle and end of the training process (columns (a–c) respectively.). The multimodality features were extracted from the multimodal classifier and formed 128-d vectors, which were then embedded in the 2D space by t-SNE for visualization. The extracted features from each single modality classifier were of size 64-d. (c) The final trained embeddings are presented. For both single and multimodal classifiers the two groups of samples are well-separated in the 2D space. It is evident that the multimodal classifier performs better sample separation and earlier on in the training process.