Algorithm 1 BIOS optimization algorithm based on DQN (Part 1).

Input: N Initial capacity of the playback pool,

M Number of final exploration frames for optimization,

T Number of optimization steps per iteration process,

$s_{initial}$ State corresponding to the initial configuration

1 Initialize replay memory (D) to capacity N;

2 Initialize Q-network with random weights $\theta$;

3 Initialize environment;

4 For episode = 1, M do

5   Initialize environment state $s=s_{initial}$;

6   For t = 1, T do

7     Calculate $Q(s,a;\theta )$ in state $s$;

8     With probability $\epsilon$ select a random action $a$,

Otherwise, select $a=\arg {\max }_a(Q(s,a;\theta ))$;

9     Execute action $a$ in the environment,

obtain a reward $r$, next state $s^{\prime }$, and whether server downtime;

10      If server downtime: store the same transition in D $k$ times;

11      else: store one transition in D;

12      Sample random minibatch of transitions from D

13      Update parameters $\theta$ in Q-network with Formula (5)

14      Every C steps reset ${\theta }^{-}=\theta$

15   End For

16 End For