Table 2.
Ablation study results on the heart dataset.
| Model variant | Description | Test accuracy (%) | Macro F1-score (%) | Peak VRAM (GB) | Fine-tuning time (h) |
|---|---|---|---|---|---|
| scReformer-BERT (full) | Reformer + pre-training + Gene2vec | 95.5 | 94.8 | 35 | 7 |
| w/o Reformer (Standard BERT) | Standard transformer on full gene set | N/A (OOM) | N/A (OOM) | >40 (OOM)* | N/A |
| w/o Reformer + gene filtering | Standard transformer on top 3,000 HVGs | 92.6 | 91.3 | 28 | 6.5 |
| w/o Pre-training | scReformer-BERT trained from scratch | 89.1 | 88.9 | 35 | 6.8 |
| w/o Gene2vec | Full model with random embedding init. | 93.1 | 92.2 | 35 | 7 |
*
OOM, out of memory on a 40GB NVIDIA A100 GPU.