Gradient Descent

The goal of machine learning is to use data to minimize some particular objective (loss) function. The most widespread way to do this is through gradient descent, which essentially takes small steps in the direction of steepest descent for a particular objective function to find the “minimum”.

$$x_{t+1}=x_t-{\alpha }_t\nabla f(x_t)$$

• $\alpha$ = learning rate

In practice, we use stochastic gradient descent (SGD), which uses one data point at a time for a single step and uses a much smaller subset of data points at any given step.

Method	Data Used per Update	Update Rule	Speed	Stability	Noise	Best Used When
Vanilla GD / Batch GD	All training data $\{(x_i,y_i){\}}_{i=1}^N$	$\theta =\theta -\alpha \cdot \frac{1}{N}{\sum }_{i=1}^N{\nabla }_{\theta }{\mathcal{L}}_i(\theta )$	Slow	Very stable	No noise	Small datasets
Stochastic GD (SGD)	One sample $(x_i,y_i)$	$\theta =\theta -\alpha \cdot {\nabla }_{\theta }{\mathcal{L}}_i(\theta )$	Fast	High variance	High noise	Large datasets or online learning
Mini-batch GD	A small batch of $m\ll N$ samples	$\theta =\theta -\alpha \cdot \frac{1}{m}{\sum }_{j=1}^m{\nabla }_{\theta }{\mathcal{L}}_j(\theta )$	Medium	Good tradeoff	Medium noise	Deep learning (standard default)

Learning Rate

The step size $\alpha$ is the single most important hyperparameter. Too small and convergence crawls; too large and the iterates oscillate or diverge. For a smooth loss with $L$-Lipschitz gradient, stability requires $\alpha <2/L$. In practice a schedule (warmup then decay, cosine, or step decay) beats a fixed rate.

Momentum

Plain SGD zig-zags across narrow valleys. Momentum accumulates an exponentially weighted average of past gradients to damp oscillation and accelerate along consistent directions:

$$v_{t+1}=\beta v_t+\nabla f(x_t),x_{t+1}=x_t-\alpha v_{t+1},$$

with $\beta$ typically 0.9. Nesterov momentum evaluates the gradient at the look-ahead point $x_t-\alpha \beta v_t$ for a sharper correction.

Convergence Intuition

• Convex, smooth: batch GD converges at rate $O(1/t)$, and $O(1/t^2)$ with acceleration.
• Strongly convex: linear (geometric) convergence.
• Non-convex (deep nets): no global guarantee, but SGD reliably reaches good local minima; the gradient noise itself acts as a regularizer and helps escape saddle points, which are far more common than bad local minima in high dimensions.

Code

import numpy as np
 
def sgd_momentum(grad_fn, x0, lr=0.1, beta=0.9, steps=100):
    x = x0.copy()
    v = np.zeros_like(x)
    for _ in range(steps):
        g = grad_fn(x)                 # gradient on current mini-batch
        v = beta * v + g
        x -= lr * v
    return x

import torch
opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
for xb, yb in loader:
    opt.zero_grad()
    loss = loss_fn(model(xb), yb)
    loss.backward()                    # backprop fills .grad
    opt.step()                         # apply the update rule

Pitfalls

• Unscaled features stretch the loss surface into ellipses, slowing descent; standardize inputs or use normalization layers.
• A diverging loss almost always means the learning rate is too high.
• Momentum can overshoot near the minimum; anneal $\alpha$ toward the end.

Related

• Adam combines momentum with per-parameter adaptive step sizes.
• Part of the broader family of First-Order Methods.