Table 1.

Results of the pre-training evaluation

Models	Highest scored token			5 highest scored tokens
	Average	Variance	Standard dev.	Average	Variance	Standard dev.
G-BERT	14.4%	0.00%	0.6%	23.3%	0.01%	0.98%
G-BERT+Rad	37.33%	0.01%	0.81%	42.94%	0.01%	1.07%
G-BERT+Rad+Flex	35.11%	0.01%	0.01%	40.56%	0.01%	0.83%
GM-BERT	14.11%	0.01%	0.86%	23.74%	0.00%	0.7%
GM-BERT+Rad	37.02%	0.02%	1.29%	42.72%	0.01%	1.17%
GM-BERT+Rad+Flex	34.61%	0.01%	1.13%	40.53%	0.00%	0.63%
RadBERT	48.05%	0.00%	0.73%	66.46%	0.00%	0.94%