Table 5.

Ablation study of language models utilized in the ensemble module. The statistical mean of the language model outputs was used as the ensembling method.

Experiment	Language model ensemble				Pearson correlation coefficient on internal test
	IIT-MTL-ClinicalBERTa	BioBERTb	MT-DNNc	RoBERTad
1	✓e	—f	—	—	0.8711
2	—	✓	—	—	0.8707
3	—	—	✓	—	0.8685
4	—	—	—	✓	0.8578
5	✓	✓	—	—	0.8754
6	—	✓	✓	—	0.8780
7	—	—	✓	✓	0.8722
8	✓	—	—	✓	0.8741
9	✓	—	✓	—	0.8796
10	—	✓	—	✓	0.8720
11	✓	✓	✓	—	0.8809 g
12	—	✓	✓	✓	0.8769
13	✓	—	✓	✓	0.8787
14	✓	✓	—	✓	0.8764
15	✓	✓	✓	✓	0.8795

^aIIT-MTL-ClinicalBERT: iterative intermediate training using multi-task learning on ClinicalBERT.

^bBioBERT: bidirectional encoder representations from transformers for biomedical text mining.

^cMT-DNN: multi-task deep neural networks.

^dRoBERTa: robustly optimized bidirectional encoder representations from transformers approach.

^eIndicates which language models are included in the ensemble.

^fIndicates language model was not used for this experiment.

^gItalics signify the highest Pearson correlation coefficient obtained on internal test data set.