Skip to main content
. 2022 Apr 8;13:1914. doi: 10.1038/s41467-022-29443-w

Table 1.

The impact of fine-tuning and initialization on downstream model performance.

Remote Homology Fluorescence Stability
Resnet LSTM Trans Resnet LSTM Trans Resnet LSTM Trans
Pre+Fix 0.27 0.37 0.27 0.23 0.74 0.48 0.65 0.70 0.62
Pre+Fin 0.17 0.26 0.21 0.21 0.67 0.68 0.73 0.69 0.73
Rng+Fix 0.03 0.10 0.04 0.25 0.63 0.14 0.21 0.61
Rng+Fin 0.10 0.12 0.09 − 0.28 0.21 0.22 0.61 0.28 − 0.06
Baseline 0.09 (Accuracy) 0.14 (Correlation) 0.19 (Correlation)

The embedding models were either randomly initialized (Rng) or pre-trained (Pre), and subsequently either fixed (Fix) or fine-tuned to the task (Fin). The baseline is a simple one-hot encoding of the sequence. Although fine-tuning is beneficial on some task/model combinations, we see clear signs of overfitting in the majority of cases (best results in bold).