Table 5.
Activation performance (accuracy) on VGG16.
| ACTIVATION | CH | HE | LO | TR | RN | TB | LY | MA | LG | LA | CO | BG | LAR | RT | HP | AVG | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VGG16 MAXINPUT = 1 | MeLU (k = 8) | 99.69 | 92.09 | 98.00 | 92.91 | 59.00 | 60.93 | 78.67 | 87.92 | 86.67 | 93.14 | 95.20 | 89.67 | 90.53 | 73.73 | 42.61 | 82.71 |
| Leaky ReLU | 99.08 | 91.98 | 98.00 | 93.45 | 66.50 | 61.13 | 80.00 | 92.08 | 86.67 | 91.81 | 95.62 | 91.33 | 88.94 | 74.86 | 38.07 | 83.30 | |
| ELU | 98.77 | 93.95 | 97.00 | 92.36 | 56.00 | 59.69 | 81.60 | 90.83 | 78.33 | 85.90 | 95.78 | 93.00 | 90.45 | 71.55 | 40.91 | 81.74 | |
| MeLU (k = 4) | 99.38 | 91.16 | 97.60 | 92.73 | 64.50 | 62.37 | 81.07 | 89.58 | 86.00 | 89.71 | 95.82 | 89.67 | 93.18 | 75.20 | 42.61 | 83.37 | |
| PReLU | 99.08 | 90.47 | 97.80 | 94.55 | 64.00 | 60.00 | 81.33 | 92.92 | 78.33 | 91.05 | 95.80 | 92.67 | 90.38 | 73.74 | 35.23 | 82.49 | |
| SReLU | 99.08 | 91.16 | 97.00 | 93.64 | 65.50 | 60.62 | 82.67 | 90.00 | 79.33 | 93.33 | 96.10 | 94.00 | 92.58 | 76.80 | 45.45 | 83.81 | |
| APLU | 99.08 | 92.33 | 97.60 | 91.82 | 63.50 | 62.27 | 77.33 | 90.00 | 82.00 | 92.38 | 96.00 | 91.33 | 90.98 | 76.58 | 34.66 | 82.52 | |
| ReLU | 99.69 | 93.60 | 98.20 | 93.27 | 69.50 | 61.44 | 80.80 | 85.00 | 85.33 | 88.57 | 95.50 | 93.00 | 91.44 | 73.68 | 40.34 | 83.29 | |
| Small GaLU | 98.46 | 91.63 | 97.80 | 91.35 | 64.50 | 59.79 | 80.53 | 89.58 | 77.33 | 92.76 | 95.70 | 91.67 | 91.97 | 72.63 | 44.32 | 82.66 | |
| GaLU | 98.46 | 94.07 | 97.40 | 92.36 | 65.00 | 59.07 | 81.07 | 92.08 | 75.67 | 93.71 | 95.68 | 88.67 | 91.74 | 75.81 | 39.20 | 82.66 | |
| Flexible MeLU | 97.54 | 94.19 | 96.60 | 94.91 | 59.00 | 62.68 | 77.07 | 90.00 | 89.00 | 91.81 | 95.94 | 92.67 | 89.92 | 72.15 | 38.64 | 82.80 | |
| TanELU | 97.85 | 93.14 | 97.00 | 92.36 | 61.00 | 61.44 | 72.80 | 89.17 | 77.33 | 91.62 | 95.28 | 89.67 | 90.23 | 72.84 | 43.75 | 81.69 | |
| 2D MeLU | 97.85 | 93.72 | 97.20 | 92.73 | 61.00 | 61.34 | 81.60 | 91.25 | 92.33 | 94.48 | 95.86 | 89.67 | 92.35 | 71.91 | 38.64 | 83.46 | |
| MeLU + GaLU | 98.15 | 93.72 | 98.20 | 93.64 | 60.00 | 60.82 | 77.60 | 92.08 | 81.00 | 93.14 | 95.54 | 92.33 | 89.47 | 75.60 | 47.16 | 83.23 | |
| Splash | 97.85 | 92.79 | 97.80 | 92.18 | 58.50 | 62.06 | 75.73 | 88.33 | 83.67 | 85.90 | 95.02 | 91.67 | 90.15 | 74.29 | 42.05 | 81.86 | |
| Symmetric GaLU | 99.08 | 92.79 | 97.20 | 92.91 | 60.50 | 60.00 | 78.93 | 88.33 | 79.33 | 91.62 | 95.52 | 92.67 | 91.67 | 73.91 | 40.34 | 82.32 | |
| Symmetric MeLU | 98.46 | 92.91 | 96.60 | 92.18 | 56.50 | 59.69 | 74.93 | 90.00 | 85.00 | 87.05 | 94.76 | 90.33 | 90.68 | 72.87 | 41.48 | 81.56 | |
| Soft Learnable v2 | 95.69 | 87.91 | 94.60 | 93.45 | 34.50 | 55.57 | 50.67 | 77.50 | 64.67 | 29.71 | 94.08 | 67.67 | 92.35 | 68.96 | 35.80 | 69.54 | |
| Soft Learnable | 98.15 | 92.91 | 97.00 | 91.82 | 47.50 | 54.33 | 62.13 | 86.67 | 95.67 | 65.90 | 95.04 | 84.33 | 90.38 | 71.08 | 40.34 | 78.21 | |
| PDELU | 98.77 | 93.60 | 96.40 | 92.18 | 59.00 | 58.25 | 76.80 | 87.92 | 87.67 | 89.33 | 95.36 | 90.33 | 91.74 | 75.24 | 42.05 | 82.30 | |
| Mish | 96.31 | 90.70 | 94.60 | 93.64 | 18.50 | 46.80 | 54.13 | 66.67 | 73.67 | 56.38 | 93.88 | 80.00 | 82.73 | 73.89 | 44.32 | 71.08 | |
| SRS | 71.08 | 59.19 | 45.00 | 51.64 | 29.50 | 31.44 | 57.60 | 61.25 | 61.00 | 45.33 | 86.88 | 57.00 | 67.50 | 39.74 | 19.32 | 52.23 | |
| Swish Learnable | 97.54 | 91.86 | 97.00 | 93.64 | 43.50 | 54.64 | 66.67 | 87.08 | 81.00 | 79.43 | 94.46 | 81.00 | 85.23 | 70.02 | 35.23 | 77.22 | |
| Swish | 98.77 | 92.56 | 96.80 | 93.64 | 63.50 | 58.97 | 80.80 | 90.00 | 89.00 | 93.14 | 94.68 | 93.33 | 91.74 | 75.24 | 39.77 | 83.46 | |
| ENS | 99.38 | 93.84 | 98.40 | 95.64 | 68.00 | 65.67 | 85.07 | 92.08 | 85.00 | 96.38 | 96.74 | 94.33 | 92.65 | 75.55 | 44.89 | 85.57 | |
| ENS_G | 99.69 | 94.65 | 99.00 | 95.45 | 72.00 | 64.95 | 86.93 | 92.50 | 83.33 | 97.14 | 96.72 | 94.67 | 92.65 | 75.56 | 45.45 | 86.07 | |
| ALL | 99.69 | 95.35 | 98.80 | 95.45 | 72.00 | 66.80 | 84.00 | 94.17 | 85.67 | 97.14 | 96.66 | 95.00 | 93.18 | 75.85 | 48.30 | 86.53 | |
| VGG16 MAXINPUT = 255 |
MeLU (k = 8) | 99.69 | 92.09 | 97.40 | 93.09 | 59.50 | 60.82 | 80.53 | 88.75 | 80.33 | 88.57 | 95.94 | 90.33 | 88.33 | 73.01 | 47.73 | 82.40 |
| MeLU (k = 4) | 99.38 | 91.98 | 98.60 | 92.55 | 66.50 | 59.59 | 84.53 | 91.67 | 88.00 | 94.86 | 95.46 | 93.00 | 93.03 | 72.21 | 38.64 | 84.00 | |
| SReLU | 98.77 | 93.14 | 97.00 | 92.18 | 65.00 | 62.47 | 77.60 | 89.58 | 76.00 | 96.00 | 95.84 | 94.33 | 89.85 | 74.04 | 42.61 | 82.96 | |
| APLU | 98.77 | 92.91 | 97.40 | 93.09 | 63.00 | 57.32 | 82.67 | 90.42 | 77.00 | 90.67 | 94.90 | 93.00 | 91.21 | 75.65 | 36.36 | 82.29 | |
| Small GaLU | 99.38 | 92.91 | 97.00 | 92.73 | 50.50 | 62.16 | 78.40 | 90.42 | 73.00 | 94.48 | 95.32 | 92.00 | 90.98 | 73.61 | 42.61 | 81.70 | |
| GaLU | 98.77 | 92.91 | 97.60 | 93.09 | 66.50 | 59.48 | 83.47 | 90.83 | 95.00 | 85.52 | 95.96 | 91.67 | 93.41 | 75.45 | 38.64 | 83.88 | |
| Flexible MeLU | 99.08 | 95.00 | 97.20 | 93.45 | 62.00 | 55.98 | 76.80 | 89.17 | 83.00 | 88.57 | 95.64 | 91.33 | 91.29 | 73.00 | 37.50 | 81.93 | |
| MeLU + GaLU | 98.46 | 94.42 | 96.80 | 92.00 | 54.50 | 60.82 | 79.73 | 90.83 | 78.67 | 93.33 | 96.26 | 89.67 | 91.14 | 74.79 | 40.34 | 82.11 | |
| Symmetric GaLU | 97.85 | 92.21 | 97.40 | 93.64 | 58.00 | 58.14 | 73.87 | 91.67 | 79.33 | 91.43 | 95.18 | 90.33 | 89.55 | 74.47 | 34.09 | 81.14 | |
| Symmetric MeLU | 98.46 | 92.33 | 96.80 | 92.18 | 56.50 | 61.24 | 75.47 | 89.17 | 82.00 | 88.00 | 95.32 | 92.67 | 88.86 | 74.27 | 38.07 | 81.42 | |
| ENS | 99.38 | 93.84 | 98.80 | 95.27 | 68.50 | 64.23 | 84.53 | 92.50 | 81.33 | 96.57 | 96.66 | 95.00 | 92.20 | 75.27 | 43.75 | 85.18 | |
| ENS_G | 99.38 | 94.88 | 98.80 | 95.64 | 70.50 | 65.88 | 85.87 | 93.75 | 81.67 | 96.38 | 96.70 | 95.67 | 92.80 | 75.26 | 44.32 | 85.83 | |
| ALL | 99.69 | 95.47 | 98.40 | 95.45 | 70.00 | 63.92 | 83.73 | 94.17 | 82.67 | 96.38 | 96.60 | 95.00 | 92.73 | 75.78 | 45.45 | 85.69 | |
| EENS | 99.38 | 94.07 | 98.80 | 95.64 | 69.00 | 65.88 | 85.87 | 93.33 | 82.67 | 96.57 | 96.88 | 95.33 | 92.50 | 74.99 | 43.18 | 85.60 | |
| EENS_G | 99.69 | 94.65 | 99.00 | 95.27 | 70.50 | 65.57 | 86.93 | 92.92 | 83.33 | 97.71 | 96.82 | 95.00 | 92.42 | 76.09 | 44.32 | 86.01 | |
| EALL | 99.69 | 95.70 | 98.80 | 95.45 | 71.50 | 65.98 | 83.73 | 94.58 | 85.67 | 96.38 | 96.70 | 95.00 | 92.50 | 75.42 | 47.16 | 86.28 | |
| 15RELU | 99.08 | 95.35 | 98.60 | 94.91 | 64.50 | 64.64 | 79.20 | 95.00 | 83.00 | 92.76 | 96.38 | 94.00 | 92.42 | 74.34 | 50.57 | 84.98 | |
| SELECTION | 99.69 | 95.26 | 98.60 | 94.91 | 71.00 | 64.85 | 86.67 | 94.58 | 84.67 | 95.24 | 96.72 | 94.33 | 93.56 | 75.48 | 47.16 | 86.18 | |
| STOC_4 | 99.69 | 96.05 | 98.60 | 95.27 | 74.50 | 67.53 | 83.47 | 95.00 | 84.00 | 95.62 | 96.78 | 92.67 | 93.48 | 74.87 | 51.70 | 86.61 |