Table 1.

Model hyperparameters.

Hyperparameter	Value
Number of layers	24
Number of heads	16
Hidden size	1024
Optimizer	Adam with ${\beta }_1$ = 0.99, ${\beta }_2$ = 0.999, $ϵ=$ $1\times {10}^{-8}$
Learning rate	$1.0\times {10}^{-5}$
Learning rate scheduler	linear with a warm-up ratio of 0.1
Batch size	16
Gradient accumulation steps	4