Figure 4.

Impact of deep learning design on learning: effect of learning rate. (A) Efficient learning. Cost function (network error) gradually descends (‘Gradient descent’) to achieve the optimal point (called local minimum) as a function of weight. (B) Learning rate is too high, so that the cost function overshoots the minimum and oscillates. This network design may not be trained effectively for this problem. (C) Gradient descent examining two variables on cost function simultaneously.