Figure 1.

An overview of the three prediction models. All the models take the clinical and the dermoscopic images of the skin lesion and the patient metadata as input. Note that we also perform an input ablation study (A multi-task prediction model section; Table 4). (a) The first model predicts the lesion diagnosis probabilities, ${\mathrm{DIAG}}_{\mathrm{pred}}$. (b) The second model predicts the management decision probabilities, ${\mathrm{MGMT}}_{\mathrm{pred}}$. (c) The third is a multi-task model and predicts the seven-point criteria ($\mathrm{Criterion}{\{1,2,\dots ,7\}}_{\mathrm{pred},\mathrm{multi}}$) in addition to ${\mathrm{DIAG}}_{\mathrm{pred},\mathrm{multi}}$ and ${\mathrm{MGMT}}_{\mathrm{pred},\mathrm{multi}}$. The argmax operation assigns 1 to the most likely label and 0 to all others. For (a), ${\mathrm{DIAG}}_{\mathrm{pred}}$ diagnosis is used to arrive at a management decision either using (a1) binary labeling, ${\mathrm{MGMT}}_{\mathrm{infr},\mathrm{binary}}$, or (a2) prior based inference, ${\mathrm{MGMT}}_{\mathrm{infr},\mathrm{all}}$. Similarly, the outputs of (b) can be used to directly predict a management decision using either (b1) binary labeling, ${\mathrm{MGMT}}_{\mathrm{pred},\mathrm{binary}}$, or (b2) all the labels, ${\mathrm{MGMT}}_{\mathrm{pred},\mathrm{all}}$. As explained in the text, the diagnosis labels are basal cell carcinoma (BCC), nevus (NEV), melanoma (MEL), seborrheic keratosis (SK), and others (MISC), and the management decision labels are ‘clinical follow up’ (CLNC), ‘excision’ (EXC), and ‘no further examination’ (NONE). In the case of binary management decisions, we predict whether a lesion should be excised (EXC) or not (NOEXC).