Fig. 1:

The architecture of our joint model, along with an example chest radiograph x^I and its associated radiology report x^R. At training time, the model predicts the edema severity level from images and text through their respective encoders and classifiers, and compares the predictions with the labels. The joint embedding loss ${\mathcal{J}}_{\text{E}}$ associates image embeddings I with text embeddings R in the joint embedding space. At inference time, the image stream and the text stream are decoupled and only the image stream is used. Given a new chest radiograph (image), the image encoder and classifier compute its edema severity level.