Some notes on architecture and design choices when training large neural nets. The “recipe” is the collection of decisions (data, tokenizer, objective, and optimization schedule) that turns a pile of text and GPUs into a capable base model. See Scaling Laws for how compute, data, and parameters trade off, and LLM Architecture and Training Design Choices for the model itself.
Data curation and mixtures
Data quality dominates. A typical pipeline: crawl, deduplicate (exact and near-duplicate via MinHash), filter for quality (classifiers, heuristics, language ID), remove PII and toxic content, and decontaminate against evaluation sets.
Different sources are mixed with deliberate weights: high-quality web (filtered CommonCrawl), code, books, Wikipedia, math, and academic text. Upsampling high-quality domains and code tends to help reasoning. The mixture is a hyperparameter, not an afterthought.
Benchmark contamination
If test-set text leaks into training data, evaluation numbers become meaningless. Always decontaminate before trusting a score.
Tokenizer
The tokenizer is trained on a representative sample of the final data mixture (see Tokenization). Vocabulary size (commonly 32k to 256k) trades sequence length against embedding size. Once chosen it is effectively frozen: changing it invalidates all learned embeddings.
Objective: next-token prediction
Standard causal language modeling minimizes the negative log-likelihood of each token given its prefix:
Loss is reported in nats or bits per token; perplexity is . Documents are usually packed end to end and split into fixed-length sequences to avoid wasted padding.
Context length
Longer context lets the model condition on more, but attention cost scales quadratically with sequence length. A common strategy is to train most of the run at a shorter length (say 4k) for efficiency, then do a short long-context extension phase at a longer length, often adjusting the RoPE base frequency (see Positional Encoding).
Batch size
Large batches (measured in millions of tokens) stabilize the gradient estimate and improve hardware utilization. Batch size is often ramped up during training. It interacts with learning rate: bigger batches tolerate (and often need) a larger LR.
Learning rate schedule
The near-universal recipe is linear warmup then cosine decay:
def lr(step, base_lr, warmup, total, min_ratio=0.1):
if step < warmup:
return base_lr * step / warmup # linear warmup
progress = (step - warmup) / (total - warmup)
cosine = 0.5 * (1 + math.cos(math.pi * progress))
return base_lr * (min_ratio + (1 - min_ratio) * cosine)Warmup (typically a few thousand steps) avoids blowing up early when gradients are large; cosine decay to roughly 10 percent of peak LR gives a smooth landing. AdamW with weight decay and is standard.
Initialization
Weights are drawn from a small normal, commonly with standard deviation scaled by depth so signal neither explodes nor vanishes. A frequent choice scales output projections by to keep the residual stream variance controlled.
Stability tricks
- Gradient clipping (global norm, e.g. 1.0) to survive loss spikes.
- Mixed precision: bf16 for most math, fp32 master weights; bf16 avoids the overflow issues of fp16.
- z-loss on the softmax logits to keep them from drifting large.
- Pre-norm residual blocks and RMSNorm for smoother optimization.
- Checkpointing frequently so a divergence can be rewound and the offending batch skipped or the LR lowered.
Loss spikes
Large runs periodically spike. Standard mitigations: skip the batch, roll back to the last good checkpoint, lower the LR, or clip harder. Persistent spikes often trace back to a data-mixture or numerical issue.