Mathematics · Simulator 046
Gradient descent finds a minimum by repeatedly stepping downhill. Pick a landscape and a starting point, set the learning rate, and watch the path. Too small and it crawls; too large and it zig-zags or flies off. Momentum can help it through long narrow valleys.
Each step moves against the gradient (the direction of steepest increase):
θ ← θ − η ∇f(θ)
η is the learning rate. With momentum β, the update remembers a running velocity: v ← βv − η∇f, θ ← θ + v.
For a quadratic bowl f = ½(a x² + b y²), plain gradient descent converges only if η < 2 / max(a, b). Above that it diverges. When a and b differ a lot (an ill-conditioned problem), the safe η is set by the steep direction, so progress along the shallow direction is painfully slow. That is the classic zig-zag.
The Rosenbrock “banana” valley and a two-minimum landscape show curved valleys and how the starting point decides which minimum you reach.
Deterministic, full-gradient descent on 2-D toy functions; machine-learning training uses noisy mini-batch gradients in millions of dimensions.
For f(x) = 2x², what is the largest learning rate that still converges? What happens at exactly η = 0.25?
f′(x) = 4x, so the update is x ← x(1 − 4η). It converges when |1 − 4η| < 1, so η < 0.5. At η = 0.25, 1 − 4η = 0, so it jumps to the minimum in a single step: the ideal rate is 1/f″.