Transfer learning reuses representations learned on a large source task to improve learning on a smaller target task, cutting data and compute requirements. It is the default recipe whenever labeled target data is limited.
Pretraining and Fine-Tuning
- Pretraining: train a large model on an abundant source dataset (e.g. ImageNet, web text) to learn general features.
- Fine-tuning: continue training some or all weights on the target task, usually with a smaller learning rate.
- Early layers learn generic features (edges, syntax); later layers are more task-specific.
Feature Extraction vs Fine-Tuning
| Approach | What is trained | When to use |
|---|---|---|
| Feature extraction | Freeze backbone, train only a new head | Small target data, similar domain |
| Fine-tuning | Unfreeze some/all layers, keep pretrained init | More target data or domain shift |
- A middle ground unfreezes only the top layers, keeping low-level features fixed.
- Fine-tuning uses a low learning rate to avoid destroying pretrained weights (catastrophic forgetting).
Using Pre-Trained Networks
- Replace the final classifier layer to match target classes.
- Common backbones: ResNet, ViT for vision; BERT, GPT-style models for language.
- Match input preprocessing (normalization, resolution) to the pretraining setup.
Data Augmentation
- Expands effective dataset size and improves generalization, especially with little target data.
- Vision: crops, flips, rotations, color jitter, cutout/mixup.
- Text/audio: token masking, back-translation, time/frequency masking.
- Acts as a form of Regularization by encouraging invariance.
When It Helps
- Source and target share low-level structure (same modality or domain).
- Target dataset is small relative to model capacity.
- It can hurt (negative transfer) when domains differ sharply; then train more layers or from scratch.