Generative Adversarial Networks (GANs) learn a data distribution through a two-player game between a generator that fabricates samples and a discriminator that tells real from fake. Training is implicit: there is no explicit likelihood, only the adversarial signal.
Generator vs Discriminator
- Generator maps noise to samples, trying to fool the discriminator.
- Discriminator outputs the probability that is real, trying to catch fakes.
- They are trained simultaneously with opposing objectives.
Minimax Objective
- At the optimum , and the game minimizes the Jensen-Shannon divergence between and .
- In practice maximizes (non-saturating loss) for stronger early gradients.
Training Dynamics and Instability
- The game is a saddle-point problem, not a single loss minimization, so convergence is delicate.
- If becomes too strong, gradients to vanish; if too weak, gets no useful signal.
- Requires careful balancing of learning rates, updates per step, and architecture.
Mode Collapse
can map many inputs to a few outputs that reliably fool , capturing only part of the data distribution. Symptoms are low sample diversity despite low generator loss.
Common Variants
| Variant | Key idea |
|---|---|
| DCGAN | Convolutional generator/discriminator, batch norm, stable image generation |
| WGAN | Wasserstein distance with a critic; smoother gradients, less mode collapse |
| WGAN-GP | Gradient penalty replaces weight clipping for the Lipschitz constraint |
| Conditional GAN | Condition and on labels for controllable generation |
Contrast with Other Generative Models
- Variational Autoencoders optimize an explicit likelihood lower bound; they train stably but tend to produce blurrier samples.
- Diffusion Models learn to denoise across many steps; slower sampling but very stable training and high fidelity, largely displacing GANs for image synthesis.
- GANs give sharp samples and fast single-pass generation but lack a likelihood and are hardest to train.