Table 2.

Hyper-parameters considered for pre-trained models

Model	Parameter	Value
GraphBERT	Hidden Layer Number	2
	Subgraph Size	7
	Learning Rate	5e-4
	Hidden Size	32
	Hidden Dropout Rate	0.5
	Attention Head Number	3
	Attention Dropout Rate	0.4
	Weight Decay	0.01
BERT	Number of Transformer Blocks	12
	Training Batch Size	32
	Learning Rate	5e-4
	Weight Decay	0.01