Brayden Zhang

LLM Reasoning

3 min read

LLM reasoning is the set of methods that get a model to work through a problem in steps rather than guessing an answer in one forward pass. The unifying idea is to spend more computation, whether at prompt time, inference time, or training time, on problems that need it.

Chain of thought (CoT)

Prompting the model to “think step by step” produces intermediate reasoning tokens before the final answer. This helps because the extra tokens act as a scratchpad: the model can decompose the problem and condition each step on the previous ones. Zero-shot CoT (just appending “Let’s think step by step”) and few-shot CoT (showing worked examples) are the two standard forms.

Self-consistency

Instead of taking one greedy chain, sample many independent chains at temperature and take a majority vote over the final answers.

answers = [extract_answer(model.sample(prompt, temp=0.7)) for _ in range(k)]
final = Counter(answers).most_common(1)[0][0]   # majority vote

Correct reasoning paths tend to converge on the same answer while errors scatter, so voting improves accuracy at the cost of times the inference compute.

Tool use

Some reasoning is better offloaded than done in-weights. Letting the model call a calculator, run code, or query a search engine turns unreliable mental arithmetic into a deterministic computation. This is the reasoning side of Agents, and the interleaving of thought and action is exactly the ReAct pattern.

Test-time compute

The broad principle: accuracy can be bought at inference time. Levers include longer chains, sampling more candidates (self-consistency, best-of-n), and search over reasoning steps (tree/graph of thought, guided by a verifier). A verifier or reward model scores candidate solutions so the system can select the best rather than the most probable.

RL for reasoning

Reasoning ability can be trained, not just prompted. Given problems with checkable answers (math, code), one can define a reward and optimize the policy to produce chains that lead to correct answers.

  • Verifier / reward: outcome reward (final answer correct) or process reward (each step scored). Verifiable domains give cheap, reliable signal.
  • Optimization: policy-gradient methods (PPO, GRPO) push the model toward high-reward reasoning traces. This is the same machinery as Reinforcement Learning from Human Feedback (RLHF), but the reward comes from correctness rather than a learned human-preference model.
  • Long-CoT RL can teach models to backtrack, self-check, and explore, producing much longer and more deliberate reasoning traces.

Distillation

Once a strong (often expensive) model produces good reasoning traces, a smaller model can be fine-tuned on those traces to imitate the behavior at a fraction of the cost. This transfers much of the reasoning ability without running RL on the small model directly.

Common pitfalls

  • Unfaithful reasoning: the stated chain may not reflect the actual computation; a plausible-looking rationale can accompany a wrong (or right) answer for the wrong reasons.
  • Reward hacking: RL against a weak verifier teaches the model to game the metric.
  • Cost: test-time methods multiply inference cost; use them where the problem warrants it.

Graph View