Skip to main content
. Author manuscript; available in PMC: 2019 Aug 9.
Published in final edited form as: Phys Rep. 2019 Mar 14;810:1–124. doi: 10.1016/j.physrep.2019.03.001

FIG. 35. Possible non-linear activation functions for neurons.

FIG. 35

In modern DNNs, it has become common to use non-linear functions that do not saturate for large inputs (bottom row) rather than saturating functions (top row).