Algorithm 1 Knowledge Distillation

1:Input: Teacher model $M_t$, Student model $M_s$, temperature T, balancing factor $\alpha$ 2:for each batch of input data $(x,y)$ do 3:    Forward pass $M_t$ and $M_s$ to obtain logits $z_t$ and $z_s$ 4:    Compute softened probabilities: $P_t=\mathrm{softmax}(z_t/T)$, $P_s=\mathrm{softmax}(z_s/T)$ 5:    Calculate ${\mathcal{L}}_{\mathrm{task}}=\mathrm{CE}(y,P_s)$ 6:    Calculate ${\mathcal{L}}_{\mathrm{distill}}=T^2·\mathrm{KL}(P_t\Vert P_s)$ 7:    Compute total loss: $\mathcal{L}=(1-\alpha )·{\mathcal{L}}_{\mathrm{task}}+\alpha ·{\mathcal{L}}_{\mathrm{distill}}$ 8:    Update $M_s$ by minimizing $\mathcal{L}$ 9:end for