Table 1.

Mathematical expressions of activation functions.

Activation Function	Equation	Range
Linear Function	$f\left(x\right)=x$	$(-\infty ,\infty )$
Step Function	$f\left(x\right)=\left\{\begin{array}{cc}0,\hfill & \mathrm{for}x<0\hfill \\ 1,\hfill & \mathrm{for}x\ge 0\hfill \end{array}\right.$	$\{0,1\}$
Sigmoid Function	$f\left(x\right)=\sigma \left(x\right)=\frac{1}{1+e^{-x}}$	$(0,1)$
Hyperbolic Tangent Function	$f\left(x\right)=tanh\left(x\right)=\frac{{\mathrm{e}}^{\mathrm{x}}-{\mathrm{e}}^{-\mathrm{x}}}{{\mathrm{e}}^{\mathrm{x}}+e^{-x}}$	$(0,1)$
ReLU	$f\left(x\right)=\left\{\begin{array}{cc}0,\hfill & \mathrm{for}x<0\hfill \\ x,\hfill & \mathrm{for}x\ge 0\hfill \end{array}\right.$	$(0,\infty )$
Leaky ReLU	$f\left(x\right)=\left\{\begin{array}{cc}0.01,\hfill & \mathrm{for}x<0\hfill \\ x,\hfill & \mathrm{for}x\ge 0\hfill \end{array}\right.$	$(-\infty ,\infty )$
Swish Function	$f\left(x\right)=x\sigma \left(\beta \left(x\right)\right)=\left\{\begin{array}{cc}\frac{x}{2},\hfill & \beta =0\hfill \\ max(0,x),\hfill & \beta \to \infty \hfill \end{array}\right.$	$(-\infty ,\infty )$