Figure 2.

PDMI framework for material mass density inference including (a) model training and validation and (b) patient application. At training phase, model inputs are (a11) MRI and (a12) DECT images. (a31) DL models used in the framework. ResNet was implemented where ConvA, ConB, and Residual Blocks represent different convolutional components. (a32) Physics-constrained loss and (a33) conventional mass density loss for training. Material mass densities generated by (a41) conventional RN and (a42) PRN. (a43) Validation experiment to obtain measured RSP for tissue substitute phantoms using a 150 MeV proton spot and Zebra (IBA Dosimetry, Germany). For patient application, (b1–b4) inference of mass density map for patient applications using MRI and DECT images as inputs for PRN and RN. DECT, dual energy CT; DL, deep learning; PDMI, physics-constrained deep learning-based multimodal imaging; PRN, physics-constrained ResNet.