Table 4.
Performance (accuracy) of activation function obtained using ResNet50.
| Activation | CH | HE | LO | TR | RN | TB | LY | MA | LG | LA | CO | BG | LAR | RT | HP | Avg | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ResNet50 MaxInput = 1 |
MeLU (k = 8) | 92.92 | 86.40 | 91.80 | 82.91 | 25.50 | 56.29 | 67.47 | 76.25 | 91.00 | 82.48 | 94.82 | 89.67 | 88.79 | 68.36 | 48.86 | 76.23 |
| Leaky ReLU | 89.23 | 87.09 | 92.80 | 84.18 | 34.00 | 57.11 | 70.93 | 79.17 | 93.67 | 82.48 | 95.66 | 90.33 | 87.27 | 69.72 | 45.45 | 77.27 | |
| ELU | 90.15 | 86.74 | 94.00 | 85.82 | 48.00 | 60.82 | 65.33 | 85.00 | 96.00 | 90.10 | 95.14 | 89.33 | 89.92 | 73.50 | 40.91 | 79.38 | |
| MeLU (k = 4) | 91.08 | 85.35 | 92.80 | 84.91 | 27.50 | 55.36 | 68.53 | 77.08 | 90.00 | 79.43 | 95.34 | 89.33 | 87.20 | 72.24 | 51.14 | 76.48 | |
| PReLU | 92.00 | 85.35 | 91.40 | 81.64 | 33.50 | 57.11 | 68.80 | 76.25 | 88.33 | 82.10 | 95.68 | 88.67 | 89.55 | 71.20 | 44.89 | 76.43 | |
| SReLU | 91.38 | 85.58 | 92.60 | 83.27 | 30.00 | 55.88 | 69.33 | 75.00 | 88.00 | 82.10 | 95.66 | 89.00 | 89.47 | 69.98 | 42.61 | 75.99 | |
| APLU | 92.31 | 87.09 | 93.20 | 80.91 | 25.00 | 54.12 | 67.20 | 76.67 | 93.00 | 82.67 | 95.46 | 90.33 | 88.86 | 71.65 | 48.30 | 76.45 | |
| ReLU | 93.54 | 89.88 | 95.60 | 90.00 | 55.00 | 58.45 | 77.87 | 90.00 | 93.00 | 85.14 | 94.92 | 88.67 | 87.05 | 69.77 | 48.86 | 81.18 | |
| Small GaLU | 92.31 | 87.91 | 93.20 | 91.09 | 52.00 | 60.00 | 72.53 | 90.00 | 95.33 | 87.43 | 95.38 | 87.67 | 88.79 | 67.57 | 44.32 | 80.36 | |
| GaLU | 92.92 | 88.37 | 92.20 | 90.36 | 41.50 | 57.84 | 73.60 | 89.17 | 92.67 | 88.76 | 94.90 | 90.33 | 90.00 | 72.98 | 48.86 | 80.29 | |
| Flexible MeLU | 91.69 | 88.49 | 93.00 | 91.64 | 38.50 | 60.31 | 73.33 | 88.33 | 95.67 | 87.62 | 94.72 | 89.67 | 86.67 | 67.35 | 44.32 | 79.42 | |
| TanELU | 93.54 | 86.16 | 90.60 | 90.91 | 40.00 | 58.56 | 69.60 | 86.25 | 95.33 | 83.05 | 94.80 | 87.67 | 86.89 | 73.95 | 43.18 | 78.69 | |
| 2D MeLU | 91.69 | 87.67 | 93.00 | 91.64 | 48.00 | 60.41 | 72.00 | 91.67 | 96.00 | 88.38 | 95.42 | 89.00 | 87.58 | 70.53 | 42.61 | 80.37 | |
| MeLU + GaLU | 93.23 | 88.02 | 93.40 | 92.91 | 54.50 | 59.18 | 72.53 | 89.58 | 95.33 | 86.29 | 95.34 | 88.64 | 88.64 | 69.29 | 43.18 | 80.67 | |
| Splash | 93.54 | 87.56 | 93.80 | 90.00 | 47.50 | 55.98 | 72.00 | 82.92 | 94.33 | 84.19 | 95.02 | 86.00 | 87.12 | 75.70 | 42.61 | 79.21 | |
| Symmetric GaLU | 93.85 | 84.19 | 92.80 | 89.45 | 47.50 | 58.66 | 72.80 | 87.08 | 95.33 | 82.67 | 94.44 | 87.33 | 87.80 | 71.52 | 52.84 | 79.88 | |
| Symmetric MeLU | 92.62 | 86.63 | 92.40 | 89.27 | 50.00 | 60.62 | 72.27 | 85.42 | 95.00 | 85.14 | 94.72 | 90.00 | 87.58 | 66.71 | 50.57 | 79.93 | |
| Soft Learnable v2 | 93.93 | 87.33 | 93.60 | 92.55 | 46.00 | 60.31 | 69.07 | 89.58 | 94.67 | 86.10 | 95.00 | 89.67 | 87.05 | 73.72 | 54.55 | 80.87 | |
| Soft Learnable | 94.15 | 87.44 | 93.40 | 90.36 | 47.00 | 59.18 | 67.73 | 88.33 | 95.00 | 85.52 | 95.52 | 89.33 | 88.26 | 72.04 | 46.59 | 79.99 | |
| PDELU | 94.15 | 87.21 | 92.00 | 91.64 | 51.50 | 56.70 | 70.93 | 89.58 | 96.33 | 86.67 | 95.08 | 89.67 | 88.18 | 72.76 | 46.59 | 80.59 | |
| Mish | 95.08 | 87.56 | 93.20 | 91.82 | 45.00 | 58.45 | 69.07 | 86.67 | 95.33 | 86.67 | 95.48 | 90.00 | 88.41 | 53.41 | 34.09 | 78.01 | |
| SRS | 93.23 | 88.84 | 93.40 | 91.09 | 51.50 | 60.10 | 69.87 | 88.75 | 95.00 | 86.48 | 95.72 | 88.33 | 89.47 | 54.06 | 48.86 | 79.64 | |
| Swish Learnable | 93.54 | 87.91 | 94.40 | 91.64 | 48.00 | 59.28 | 69.33 | 88.75 | 95.33 | 83.24 | 96.10 | 90.00 | 89.32 | 41.15 | 39.77 | 77.85 | |
| Swish | 94.15 | 88.02 | 94.20 | 90.73 | 48.50 | 59.90 | 70.13 | 89.17 | 92.67 | 86.10 | 95.66 | 87.67 | 87.65 | 65.05 | 32.39 | 78.79 | |
| ENS | 95.38 | 89.53 | 97.00 | 89.82 | 59.00 | 62.78 | 76.53 | 86.67 | 96.00 | 91.43 | 96.60 | 91.00 | 89.92 | 74.00 | 50.00 | 83.04 | |
| ENS_G | 93.54 | 90.70 | 97.20 | 92.73 | 56.00 | 63.92 | 77.60 | 90.83 | 96.33 | 91.43 | 96.42 | 90.00 | 90.00 | 73.76 | 50.00 | 83.36 | |
| ALL | 97.23 | 91.16 | 97.20 | 95.27 | 58.00 | 65.15 | 76.80 | 92.92 | 98.00 | 90.10 | 96.58 | 90.00 | 90.38 | 74.67 | 53.98 | 84.49 | |
| ResNet50 MaxInput = 255 |
MeLU (k = 8) | 94.46 | 89.30 | 94.20 | 92.18 | 54.00 | 61.86 | 75.73 | 89.17 | 97.00 | 88.57 | 95.60 | 87.67 | 88.71 | 72.09 | 52.27 | 82.18 |
| MeLU (k = 4) | 92.92 | 90.23 | 95.00 | 91.82 | 57.00 | 59.79 | 78.40 | 87.50 | 97.33 | 85.14 | 95.72 | 89.33 | 88.26 | 66.20 | 48.30 | 81.52 | |
| SReLU | 92.31 | 89.42 | 93.00 | 90.73 | 56.50 | 59.69 | 73.33 | 91.67 | 98.33 | 88.95 | 95.52 | 89.67 | 87.88 | 68.94 | 48.30 | 81.61 | |
| APLU | 95.08 | 89.19 | 93.60 | 90.73 | 47.50 | 56.91 | 75.20 | 89.17 | 97.33 | 87.05 | 95.68 | 89.67 | 89.47 | 71.44 | 51.14 | 81.27 | |
| Small GaLU | 93.54 | 87.79 | 95.60 | 89.82 | 55.00 | 63.09 | 76.00 | 90.42 | 95.00 | 85.33 | 95.08 | 89.67 | 89.77 | 72.14 | 45.45 | 81.58 | |
| GaLU | 92.92 | 87.21 | 92.00 | 91.27 | 47.50 | 60.10 | 74.13 | 87.92 | 96.00 | 86.86 | 95.56 | 89.33 | 87.73 | 70.26 | 44.32 | 80.20 | |
| Flexible MeLU | 92.62 | 87.09 | 91.60 | 91.09 | 48.50 | 57.01 | 69.60 | 86.67 | 95.00 | 87.81 | 95.26 | 89.00 | 88.11 | 70.83 | 46.59 | 79.78 | |
| 2D MeLU | 95.08 | 90.23 | 93.00 | 91.45 | 54.00 | 57.42 | 69.60 | 90.42 | 96.00 | 87.43 | 91.84 | 87.67 | 90.76 | 73.44 | 54.55 | 81.52 | |
| MeLU + GaLU | 93.23 | 87.33 | 92.20 | 90.91 | 54.00 | 58.66 | 73.87 | 89.58 | 95.33 | 88.76 | 95.42 | 86.33 | 86.74 | 70.91 | 48.86 | 80.92 | |
| Splash | 96.00 | 87.67 | 92.80 | 93.82 | 50.50 | 60.62 | 78.13 | 89.58 | 96.67 | 87.81 | 95.18 | 90.33 | 91.36 | 68.81 | 51.70 | 82.06 | |
| Symmetric GaLU | 92.00 | 85.58 | 91.20 | 89.64 | 43.50 | 57.94 | 70.93 | 79.58 | 91.33 | 85.14 | 95.34 | 87.33 | 85.98 | 69.37 | 47.16 | 78.13 | |
| Symmetric MeLU | 92.92 | 88.37 | 93.40 | 92.00 | 44.00 | 58.56 | 69.60 | 91.67 | 93.33 | 84.00 | 94.94 | 87.33 | 88.79 | 70.30 | 44.89 | 79.60 | |
| ENS | 93.85 | 91.28 | 96.20 | 93.27 | 59.00 | 63.30 | 77.60 | 91.67 | 98.00 | 87.43 | 96.30 | 89.00 | 89.17 | 71.11 | 50.00 | 83.14 | |
| ENS_G | 95.08 | 91.28 | 96.20 | 94.18 | 63.00 | 64.85 | 78.67 | 92.50 | 97.67 | 87.62 | 96.54 | 89.67 | 89.77 | 71.36 | 51.14 | 83.96 | |
| ALL | 96.00 | 91.16 | 96.60 | 94.55 | 60.50 | 64.74 | 77.60 | 92.92 | 97.67 | 89.52 | 96.62 | 89.33 | 90.68 | 74.37 | 52.27 | 84.30 | |
| eENS | 94.77 | 91.40 | 97.00 | 92.91 | 60.00 | 64.74 | 77.87 | 88.75 | 98.00 | 90.10 | 96.50 | 90.00 | 89.77 | 73.23 | 50.57 | 83.70 | |
| eENS_G | 95.08 | 91.28 | 96.80 | 93.45 | 62.50 | 65.26 | 78.93 | 91.67 | 96.67 | 90.48 | 96.60 | 89.33 | 89.85 | 73.60 | 50.00 | 84.10 | |
| eALL | 96.92 | 91.28 | 97.20 | 95.45 | 60.50 | 64.64 | 77.87 | 93.75 | 97.67 | 90.10 | 96.58 | 89.67 | 90.68 | 74.37 | 52.27 | 84.59 | |
| 15ReLU | 95.40 | 91.10 | 96.20 | 95.01 | 58.50 | 64.80 | 76.00 | 92.90 | 97.30 | 89.30 | 96.30 | 90.00 | 90.04 | 73.00 | 50.57 | 83.76 | |
| Selection | 96.62 | 91.40 | 97.00 | 95.09 | 60.00 | 64.85 | 77.87 | 93.75 | 98.00 | 90.29 | 96.78 | 90.00 | 90.98 | 74.04 | 54.55 | 84.74 | |
| Stoc_1 | 97.81 | 91.51 | 96.66 | 95.87 | 60.04 | 65.83 | 80.02 | 92.96 | 99.09 | 91.24 | 96.61 | 90.77 | 91.03 | 74.20 | 50.57 | 84.95 | |
| Stoc_2 | 98.82 | 93.42 | 97.87 | 96.48 | 65.58 | 66.92 | 85.65 | 92.94 | 99.77 | 94.33 | 96.63 | 91.36 | 92.34 | 76.83 | 54.55 | 86.89 | |
| Stoc_3 | 99.43 | 93.93 | 98.04 | 96.06 | 64.55 | 66.41 | 83.24 | 90.04 | 96.04 | 93.93 | 96.72 | 92.05 | 91.34 | 75.89 | 51.70 | 85.95 | |
| Stoc_4 | 98.77 | 92.09 | 97.40 | 96.55 | 63.00 | 67.01 | 81.87 | 93.33 | 100 | 93.52 | 96.72 | 93.00 | 92.27 | 76.38 | 51.70 | 86.24 |