Algorithm 3 Transformer-based IoT attack

Require: Dataset $D=\{X,y\}$, learning rate ${\eta }_0$, batch size B, epochs E, attention heads h, decay factor $\lambda$, folds K

Ensure: Trained model $\mathcal{M}$, evaluation metrics $\mathcal{R}$

1: Normalize features: $x^{\prime }={\displaystyle \frac{x-x_{min}}{x_{max}-x_{min}}},\forall x\in X$

2: Initialize weights $W_Q,W_K,W_V,W_O\sim \mathcal{N}(0,{\sigma }^2)$

3: Initialize $\mathcal{R}\leftarrow \varnothing$

4: $F\leftarrow$ Stratified K-fold split on $(X,y)$

5: for all $(X_{\mathrm{train}},y_{\mathrm{train}},X_{\mathrm{test}},y_{\mathrm{test}})\in F$ do

6:     Apply SMOTE: $(X_{\mathrm{train}}^{\mathrm{res}},y_{\mathrm{train}}^{\mathrm{res}})\leftarrow \mathrm{SMOTE}(X_{\mathrm{train}},y_{\mathrm{train}})$

7:     $\eta \leftarrow {\eta }_0$

8:     for $e=1$ to E do

9:         Shuffle$(X_{\mathrm{train}}^{\mathrm{res}},y_{\mathrm{train}}^{\mathrm{res}})$

10:         for all $B_i=(X_B,y_B)\subset (X_{\mathrm{train}}^{\mathrm{res}},y_{\mathrm{train}}^{\mathrm{res}})$ do

11:              $Q=W_Q{X}_B, K=W_K{X}_B, V=W_V{X}_B$

12:             $\alpha ={\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}, A=\mathrm{softmax}\left(\alpha \right)V$

13:             $H=\mathrm{Concat}(A_1,\dots ,A_h)W_O$

14:             $\widehat{H}=\mathrm{LayerNorm}(H+X_B)$

15:             $Z=W_1\widehat{H}+b_1, \tilde{Z}=\mathrm{ReLU}\left(Z\right)$

16:             $F=W_2\tilde{Z}+b_2, \widehat{F}=\mathrm{LayerNorm}(F+\widehat{H})$

17:             $\widehat{y}=\mathrm{Softmax}\left(\widehat{F}\right)$

18:             $L=-{\sum }_{i=1}^N{y}_{i}log\left({\widehat{y}}_i\right)$

19:             Update weights: $W\leftarrow W-\eta {\displaystyle \frac{\partial L}{\partial W}}, b\leftarrow b-\eta {\displaystyle \frac{\partial L}{\partial b}}$

20:         end for

21:         $\eta \leftarrow {\eta }_0\times {\lambda }^{{\displaystyle \frac{e}{E}}}$

22:         ${\mathcal{R}}_e\leftarrow \mathrm{Evaluate}(\mathcal{M},X_{\mathrm{test}},y_{\mathrm{test}})$

23:         if EarlyStopping(${\mathcal{R}}_e$) then

24:              Break

25:         end if

26:     end for

27:     Append ${\mathcal{R}}_e$ to $\mathcal{R}$

28: end for

29: return $\mathcal{M},\mathcal{R}$