Table 1. Test Set Performance under Three Benchmark Settings Are Evaluated in ROC-AUC and PR-AUC^a.

ROC-AUC
		benchmark
goal of comparison	model	dissimilar GPCR	kinase inhibitor	random
DISAE	ALBERT frozen transformer (distilled triplets)	0.725	0.690	0.889
the effect of pretraining	ALBERT frozen transformer (pretrained on GPCR)	0.441		0.849
the effect of distilled sequence	ALBERT frozen transformer (distilled singlets)	0.583		0.656
the effect of fine-tuning	ALBERT frozen embedding	0.585		0.889
	ALBERT all unfrozen	0.680		0.891
baseline against ALBERT	TransformerCPI (full sequence)	0.570	0.680	0.896
	TransformerCPI (distilled singlets)	0.645	0.682	0.897
	TAPE (full sequence)	0.610	0.640	0.825
	TAPE (distilled singlets)	0.680	0.619	0.829
	LSTM (full sequence)	0.524	0.662	0.911
	LSTM (distilled singlets)	0.652	0.642	0.907
	LSTM (distilled triplets)	0.476	0.667	0.908

PR-AUC
		benchmark
goal of comparison	model	dissimilar GPCR	kinase inhibitor	random
DISAE	ALBERT frozen transformer (distilled triplets)	0.589	0.673	0.783
The effect of pretraining	ALBERT frozen transformer (pretrained on GPCR)	0.215		0.728
the effect of distilled sequence	ALBERT frozen transformer (distilled singlets)	0.370		0.477
the effect of fine-tuning	ALBERT frozen embedding	0.278		0.783
	ALBERT all unfrozen	0.418		0.785
baseline against ALBERT	TransformerCPI (full sequence)	0.300	0.620	0.782
	TransformerCPI (distilled singlets)	0.350	0.624	0.778
	TAPE (full sequence)	0.300	0.610	0.684
	TAPE (distilled singlets)	0.387	0.584	0.698
	LSTM (full sequence)	0.262	0.628	0.803
	LSTM (distilled singlets)	0.372	0.614	0.798
	LSTM (distilled triplets)	0.261	0.590	0.804

^aALBERT pretrained transformer-frozen model outperforms other models, and its performance is stable across all settings. Hence, it is recommended as the optimal configuration for the pretrained ALBERT model. Four variants of DISAE models are compared to the frozen transformer one. Unless specified in the parentheses, ALBERT is pretrained on whole Pfam proteins in the form of distilled triplets. The four DISAE variants are organized into three groups based on the goal of comparison. Three state-of-the-art models TAPE, TransformerCPI and LSTM are compared with the ALBERT pretrained models as baselines. Protein similarity based splitting uses a threshold of similarity score of 0.035 (Figure 2).