Table 14.
The most common optimization algorithms task
| Year | Optimizer Name | Common Task used |
|---|---|---|
| 1951 | Stochastic Gradient Descent |
Federated Learning Image Classification |
| 1999 | SGD with Momentum |
Image Classification Object Detection |
| 2000 | Rung Kutta optimization | Object Detection |
| 2011 | AdaGrad | Language Modelling |
| 2013 | RMSProp | Image Classification |
| 2014 | Adam | Language Modelling |
| 2014 | Feedback Alignment |
Object Detection Knowledge Distillation |
| 2016 | Deep Ensembles | Image Classification |
| 2016 | Direct Feedback Alignment |
Image Classification Out-of-Distribution Detection |
| 2017 | LARS |
Image Classification Self-Supervised Learning |
| 2018 | Adafactor | Question Answering |
| 2019 | AMSGrad | Time Series |
| 2021 | Gravity | Numerical Integration |