Algorithm 2.

Adaptive training scheme.

1. Train DNN for θ* by minimizing the loss function in equation (1);

for i = 1, 2, …, Nadapt do

2. Run Algorithm 1 for $\left\{p_i^{*}(k)\mid i=1,2,\dots ,N_{train},k=1,2,\dots ,K\right\}$;

3. Find a set of perturbed samples $P\subset \{x_i+p_i^{*}(k)\mid i=1,2,\dots ,N_{train},k=1,2,\dots ,K\}$ that were found vulnerable;

4. If P = ∅, set i = Nadapt;

Otherwise, Set S = S ∪ P, update θ* over enlarged S by minimizing the loss function in equation (1);

end for

Output θ*