Table 8.

Ablation study on the impact of model pruning by comparing test performance on BLURB tasks after removing top layers of PubMedBERT

	Layers removed				Performance drop
	0	2	4	6
BC5-chem	93.33	93.22	92.96	92.40	−0.93
BC5-disease	85.62	85.35	85.17	84.29	−1.33
NCBI-disease	87.82	88.38	87.99	87.38	−0.44
BC2GM	84.52	84.32	83.46	82.05	−2.47
JNLPBA	79.10	78.94	78.88	78.07	−1.03
EBM PICO	73.38	73.38	73.33	73.35	−0.05
ChemProt	77.24	76.11	73.40	72.74	−4.50
DDI	82.36	82.16	79.71	79.30	−3.06
GAD	83.96	82.33	80.23	79.21	−4.75
BIOSSES	92.30	92.66	92.80	92.12	−0.18
HoC	82.32	82.46	82.43	82.01	−0.31
PubMedQA	55.84	51.22	49.76	50.08	−6.08
BioASQ	87.56	83.73	77.50	74.00	−13.56