Fig. 1.

Overview of the proposed framework to combine 2D CMR and echocardiography data for CRT response prediction. First, the ‘nnU-Net’ architecture is used to extract segmentations of the heart over the full cardiac cycle from the two modalities. Next, a multimodal deep learning classifier is used for CRT response prediction, which combines the latent spaces of the ‘nnU-Net’ models from the two modalities. At test test time, this framework can be used with 2D echocardiography data only, whilst taking advantage of the implicit relationship between CMR and echocardiography features learnt from the model.