Table 1.

Frequently used activation functions.

Activation Function	Equation
Sigmoid	$\sigma \left(\mathit{x}\right)=\frac{1}{1+{\mathit{e}}^{-\mathit{x}}}$	(7)
ReLU	$\mathrm{ReLU}\left(\mathit{x}\right)=\max (\mathit{x},0)=\left\{\begin{array}{c}\mathit{x},\mathit{x}\ge 0\\ 0,\mathit{x}<0\end{array}\right.$	(8)
ELU	$\mathrm{ELU}\left(\mathit{x}\right)=\left\{\begin{array}{c}\mathit{x},\mathit{x}\ge 0\\ \alpha *({\mathit{e}}^x-1),\mathit{x}<0\end{array}\right.$	(9)

ReLU: Rectified Linear Unit; ELU: Exponential Linear Unit; $\alpha$ defaults to 1.0.