Logistic Regression

Summary

Logistic Regression is an algorithm to predict probabilities for binary classification problems, using a sigmoid function to squash predictions between 0 and 1.

$$\hat{y}=\sigma (w^{T}x+b)$$

Sigmoid

The logistic (sigmoid) function maps any real score to a probability in $(0,1)$:

$$\sigma (z)=\frac{1}{1+e^{-z}},{\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z)).$$

The linear score $z=w^{\top }x+b$ is the log-odds (logit) of the positive class: $z=\log \frac{p}{1-p}$.

Cross-Entropy Loss

We fit $w,b$ by maximum likelihood, equivalently by minimizing the binary cross-entropy (log loss):

$$L=-\frac{1}{N}\sum _{i=1}^N[y_i\log {\hat{y}}_i+(1-y_i)\log (1-{\hat{y}}_i)].$$

This loss is convex in $(w,b)$, so gradient descent reaches the global optimum.

Gradient

The sigmoid derivative makes the gradient remarkably clean. For a single example:

$$\frac{\partial L}{\partial w}=(\hat{y}-y)x,\frac{\partial L}{\partial b}=(\hat{y}-y).$$

The residual $(\hat{y}-y)$ scales the update, exactly as in linear regression. Optimize with Gradient Descent.

Decision Boundary

Predicting class 1 when $\hat{y}\ge 0.5$ is equivalent to $w^{\top }x+b\ge 0$, which is a linear hyperplane in feature space. Logistic regression is a linear classifier; curved boundaries require adding nonlinear features or a kernel.

Multiclass Softmax

For $K$ classes, replace the sigmoid with softmax over per-class logits $z_k=w_k^{\top }x+b_k$:

$${\hat{y}}_k=\frac{e^{z_k}}{{\sum }_{j=1}^K{e}^{z_j}},L=-\frac{1}{N}\sum _i\sum _k{y}_{ik}\log {\hat{y}}_{ik}.$$

The gradient keeps the same form: $\partial L/\partial z_k={\hat{y}}_k-y_k$.

torch

import torch
import torch.nn as nn
 
model = nn.Linear(n_features, n_classes)   # produces logits
loss_fn = nn.CrossEntropyLoss()            # softmax + NLL, numerically stable
opt = torch.optim.SGD(model.parameters(), lr=0.1)
 
for xb, yb in loader:                      # yb: integer class labels
    opt.zero_grad()
    logits = model(xb)
    loss = loss_fn(logits, yb)
    loss.backward()
    opt.step()
 
# binary case: probability via sigmoid on a single logit
prob = torch.sigmoid(model(x)[:, 1] - model(x)[:, 0])

Pitfalls

• Do not apply softmax before CrossEntropyLoss; it expects raw logits and applies log-softmax internally for stability.
• Perfectly separable data drives weights to infinity; use L2 regularization to keep them finite.
• Predicted probabilities are not automatically calibrated across all settings; check with a reliability curve if you rely on them.
• Standardize features so gradient descent converges quickly.