Table 2.
Comparison of BLURB test performance with various pretraining settings: standard BERT pretraining; BERT pretraining without NSP (i.e., MLM only); BERT pretraining with MLM only and single-sequence (single-segment ID); ELECTRA
| BERT | BERT (no NSP) | BERT (no NSP, single seq) | ELECTRA | |
|---|---|---|---|---|
| BC5-chem | 93.33∗ | 93.21 | 93.20 | 93.00 |
| BC5-disease | 85.62∗ | 85.29 | 85.44 | 84.84 |
| NCBI-disease | 87.82 | 88.29 | 88.68∗ | 87.17 |
| BC2GM | 84.52 | 84.41 | 84.63∗ | 84.03 |
| JNLPBA | 79.10∗ | 79.01 | 79.10∗ | 78.57 |
| EBM PICO | 73.38 | 73.87∗ | 73.64 | 73.57 |
| ChemProt | 77.24∗ | 76.82 | 76.88 | 76.34 |
| DDI | 82.36 | 82.64∗ | 82.45 | 80.58 |
| GAD | 83.96∗ | 82.30 | 83.24 | 83.40 |
| BIOSSES | 93.46∗ | 93.12 | 75.50 | 80.24 |
| HoC | 82.32 | 82.37∗ | 81.91 | 81.28 |
| PubMedQA | 55.84 | 56.40 | 66.66∗ | 64.96 |
| BioASQ | 87.56 | 83.57 | 85.64 | 88.93∗ |
| BLURB score | 81.35∗ | 81.00 | 79.04 | 79.61 |
∗
Highest performance for task (row).