Table 2.
Ablation study of ConTEXTual Net
| Model type | AVG Dice | SD |
|---|---|---|
| No augmentations | ||
| Baseline U-Net | 0.649 | 0.014 |
| ConTEXTual Net | 0.668 | 0.010 |
| Vision augmentations | ||
| Baseline U-Net | 0.680 | 0.014 |
| ConTEXTual Net | 0.716 | 0.016 |
| ConTEXTual Net with flipping | 0.675 | 0.016 |
| ConTEXTual Net w/o reports | 0.671 | 0.019 |
| Text augmentations | ||
| No text augmentations | 0.716 | 0.016 |
| Synonym Replacement | 0.705 | 0.008 |
| Sentence Shuffle | 0.713 | 0.023 |
| Synonym + Sentence Shuffle | 0.714 | 0.014 |
| Language models | ||
| ConTEXTual Net (T5) | 0.716 | 0.016 |
| ConTEXTual Net (RoBERTa-Large) | 0.713 | 0.010 |
| ConTEXTual Net (RadBERT) | 0.716 | 0.022 |
| ConTEXTual Net (BERT) | 0.713 | 0.020 |
| Activation functions | ||
| ConTEXTual Net (Tanh) | 0.716 | 0.016 |
| ConTEXTual Net (ReLU) | 0.698 | 0.027 |
| ConTEXTual Net (Sigmoid) | 0.710 | 0.010 |
| ConTEXTual Net (No Activation) | 0.704 | 0.011 |
| Cross-attention integration | ||
| Attention Module L4 | 0.712 | 0.019 |
| Attention Module L3 | 0.709 | 0.013 |
| Attention Module L2 | 0.685 | 0.021 |
| Attention Module L1 | 0.679 | 0.011 |
| Unfreezing language model | ||
| Unfreeze at start | 0.704 | 0.011 |
| Unfreeze at 25th epoch | 0.712 | 0.014 |
| Unfreeze at 50th epoch | 0.716 | 0.020 |
| Unfreeze at 75th epoch | 0.716 | 0.010 |
| Frozen | 0.716 | 0.022 |
Bold values denote the highest-performing configuration of ConTEXTual Net