Gradient descent, four optimizers side by side.

The four rules

Each optimizer takes the same gradient g at the current point and turns it into an update. What differs is how much memory of past gradients it keeps and what it does with it.

SGD       x -= lr * g

Momentum  v  = 0.9 * v + g
          x -= lr * v

RMSProp   s  = 0.9 * s + 0.1 * g^2
          x -= lr * g / (sqrt(s) + 1e-8)

Adam      m  = 0.9  * m + 0.1  * g
          v  = 0.999 * v + 0.001 * g^2
          m^ = m / (1 - 0.9^t)
          v^ = v / (1 - 0.999^t)
          x -= lr * m^ / (sqrt(v^) + 1e-8)

Momentum carries velocity forward, so a chain of gradients pointing the same way builds speed. RMSProp divides each coordinate by the size of its recent gradients, which shrinks the step in fast-changing directions and enlarges it in slow ones. Adam does momentum on the numerator and RMSProp on the denominator, then bias-corrects both because early on the running averages are pulled toward zero.

The four surfaces

Bowl is x² + y². Everyone gets to the minimum. On Valley, 10x² + y², the x-axis is ten times steeper than the y-axis, and plain SGD zigzags across the trough because its step is tuned by whichever axis needs the smallest one. Rosenbrock, (1 − x)² + 100 (y − x²)², is the standard hard case: a narrow curved valley whose minimum sits at (1, 1) and whose floor is nearly flat. Himmelblau has four separate minima, so where you start decides which one you find.

What to try

Start on Valley with the learning rate at 0.05 and watch SGD bounce off the walls while momentum smooths the ride and Adam walks a nearly straight diagonal. Switch to Rosenbrock, drop the learning rate to 0.005, and see who reaches (1, 1) inside a thousand steps. On Himmelblau, click near the origin and reset a few times; small differences in the seed send the four optimizers to different minima.

Sources

• Polyak, B. T. (1964). Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5), 1–17. The momentum term.
• Tieleman, T. & Hinton, G. (2012). Lecture 6.5 — RMSProp. COURSERA: Neural Networks for Machine Learning. Never published as a paper.
• Kingma, D. P. & Ba, J. (2015). Adam: A method for stochastic optimization. ICLR 2015. Introduces the bias correction that lets the running averages start at zero.
• Rosenbrock, H. H. (1960). An automatic method for finding the greatest or least value of a function. Computer Journal, 3(3), 175–184. The banana valley.
• Himmelblau, D. M. (1972). Applied nonlinear programming. McGraw-Hill. The four-minima test function.