TABLE 5. Ablation Analysis on the Effectiveness of the Label Module and Label Pair modules.

	Label-based measures								Instance-based measures
	Macro-F1		Macro-AP		Micro-F1		Micro-AP		F1		Accuracy
	Mean	Max	Mean	Max	Mean	Max	Mean	Max	Mean	Max	Mean	Max
LitCovid BioCreative
LitMC-BERT (ours)	0.8776	0.8921	0.8048	0.8223	0.9129	0.9212	0.8553	0.8663	0.9314	0.9384	0.8022	0.8188
-No Label Module	0.8635	0.8741	0.7868	0.7984	0.9071	0.9117	0.8474	0.8528	0.9258	0.9288	0.7942	0.8068
-No Label Pair	0.8752	0.8838	0.8018	0.8121	0.9116	0.9166	0.8536	0.8606	0.9298	0.9326	0.8011	0.8104
-No both modules	0.8569	0.8791	0.7796	0.8066	0.9067	0.9163	0.8461	0.8607	0.9254	0.9341	0.7915	0.8072
Hoc
LitMC-BERT (ours)	0.8733	0.8882	0.7894	0.8118	0.8648	0.8787	0.7697	0.7905	0.9036	0.9169	0.6854	0.7270
-No Label Module	0.8691	0.8796	0.7860	0.8027	0.8580	0.8638	0.7602	0.7717	0.8984	0.9046	0.6803	0.7016
-No Label Pair	0.8644	0.8803	0.7762	0.8005	0.8590	0.8700	0.7618	0.7807	0.8998	0.9071	0.6756	0.7206
-No both modules	0.8599	0.8711	0.7690	0.7875	0.8554	0.8637	0.7547	0.7670	0.8941	0.9018	0.6695	0.6857

Note That ‘no both modules’ is linear bert.