Algorithm 5 Token-Level Multimodal Fusion for Skin Cancer Classification

1:procedure Train/Infer($I,m,{\mathbf{e}}_d$)   2:    ${\mathbf{f}}_{\mathrm{img}}\leftarrow \mathrm{EfficientNet}\left(I\right)$   3:    ${\mathbf{h}}_{\mathrm{img}}^{tokens}\leftarrow \mathrm{reshape}({\mathbf{f}}_{\mathrm{img}},7\times 7\mathrm{patches})$       ▹ 49 tokens, 128-dim each   4:    ${\mathbf{h}}_{\mathrm{meta}}\leftarrow {\mathrm{MLP}}_{\mathrm{meta}}\left(\mathrm{encodeMetadata}\left(m\right)\right)$          ▹ 1 token, 32-dim   5:    ${\mathbf{h}}_{\mathrm{kg}}^{tokens}\leftarrow {\mathrm{MLP}}_{\mathrm{kg}}\left({\mathbf{e}}_d\right)$                ▹ 5 tokens, 32-dim each   6:    ${\mathbf{h}}_{fused}\leftarrow \mathrm{CrossAttn}({\mathbf{Q}}_{img},{\mathbf{K}}_{[meta,kg]},{\mathbf{V}}_{[meta,kg]})$         ▹ Token-level fusion   7:    $\mathbf{z}\leftarrow \mathrm{ReLU}({\mathbf{W}}_{h1}{\mathbf{h}}_{\mathrm{fused}}+{\mathbf{b}}_{h1})$   8:    $\widehat{\mathbf{y}}\leftarrow \mathrm{softmax}({\mathbf{W}}_c\mathbf{z}+{\mathbf{b}}_c)$   9:    return $\widehat{\mathbf{y}}$ 10:end procedure