Table 4.

Performance comparison of various R-BERT (Bidirectional Encoder Representations from Transformers) models built by (1) applying different language models (LMs) as embedding layers and (2) fine-tuning different data sets.

Data set	Vanilla BERT [45]	BioBERT [57]	PubMedBERT [58]	ClinicalBERT [59]	SciBERT [60]	Average
1	0.9389	0.9583	0.9503	0.9519	0.9496	0.9498
2	0.9435	0.9528	0.9487	0.9489	0.9459	0.9480
3	0.9360	0.9522	0.9447	0.9432	0.9497	0.9451
4	0.9406	0.9486	0.9372	0.9448	0.9468	0.9436
5	0.9093	0.9223	0.9039	0.9262	0.9179	0.9159
6	0.8847	0.9268	0.9143	0.9102	0.9113	0.9095
7	0.8965	0.9194	0.9059	0.9132	0.8961	0.9062
8	0.8781	0.9089	0.8856	0.9162	0.9100	0.8998
Average	0.9159	0.9362	0.9238	0.9318	0.9284	N/Aa

^aN/A: not applicable.