Actor-Critic Methods

Actor-critic methods combine the policy-based and value-based worlds: an actor parameterizes the policy ${\pi }_{\theta }(a|s)$ and is trained by policy gradient, while a critic learns a value function and supplies a low-variance estimate of the advantage in place of the noisy Monte-Carlo return used by REINFORCE. This is the workhorse structure underneath Proximal Policy Optimization (PPO) and the off-policy methods in Advanced Policy Optimization.

The Actor-Critic Decomposition

Recall the advantage form of the policy gradient: ${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{{\pi }_{\theta }}[{\nabla }_{\theta }\log {\pi }_{\theta }(a|s)A^{\pi }(s,a)],A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }(s).$ REINFORCE plugs in the sampled return $G_t$ for $Q^{\pi }$, which is unbiased but high-variance. Actor-critic instead learns the value function (the critic) and uses it to form the advantage:

• Actor ($\theta$): the policy, updated by the policy gradient with the critic’s advantage as the weight.
• Critic ($w$): a value estimate $V_w(s)$ (or $Q_w(s,a)$), updated by temporal-difference (TD) learning toward a bootstrapped target.

The critic acts as a learned, state-dependent baseline (see baselines in Policy Gradient), which is exactly what drives the variance down.

The Critic as a Learned Baseline

Using $V_w(s)$ as the baseline, the one-step advantage estimate is the TD error: ${\delta }_t=r_t+\gamma V_w(s_{t+1})-V_w(s_t)\approx A^{\pi }(s_t,a_t).$ $\mathbb{E}[{\delta }_t\mid s_t,a_t]=A^{\pi }(s_t,a_t)$ when $V_w=V^{\pi }$, so ${\delta }_t$ is a valid (unbiased-in-the-limit) advantage signal. The critic itself is trained by regression onto its bootstrap target: $\mathcal{L}(w)=\frac{1}{2}(r_t+\gamma V_w(s_{t+1})-V_w(s_t){)}^2,w\leftarrow w+\beta {\delta }_t{\nabla }_w{V}_w(s_t).$

TD vs. Monte-Carlo: The Bias-Variance Tradeoff

The advantage can be estimated with different horizons of bootstrapping, and this is the central design axis:

• Monte-Carlo ($A_t=G_t-V_w(s_t)$): uses the full sampled return. Unbiased but high-variance (accumulates noise over the whole episode), and needs complete episodes.
• One-step TD ($A_t={\delta }_t$): bootstraps immediately. Low-variance but biased whenever $V_w\ne V^{\pi }$; works online.
• $n$-step ($A_t={\sum }_{k=0}^{n-1}{\gamma }^k{r}_{t+k}+{\gamma }^n{V}_w(s_{t+n})-V_w(s_t)$): interpolates. Larger $n$ means more variance, less bias.

$\underset{\text{high variance, no bias}}{\underbrace{\text{MC (n}\to \infty )}}⟷\underset{\text{low variance, bias}}{\underbrace{\text{TD(0)}}}$

Generalized Advantage Estimation (GAE)

GAE (Schulman et al., 2016) does not pick a single $n$; it takes an exponentially weighted average of all $n$-step advantage estimators, controlled by $\lambda \in [0,1]$. Define the TD residual ${\delta }_t=r_t+\gamma V(s_{t+1})-V(s_t)$. Then $\boxed{{\hat{A}}_t^{GAE(\gamma ,\lambda )}=\sum _{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l}}$ Derivation. Let ${\hat{A}}_t^{(n)}$ be the $n$-step advantage. One can show it telescopes into TD residuals: ${\hat{A}}_t^{(n)}={\sum }_{l=0}^{n-1}{\gamma }^l{\delta }_{t+l}$. GAE is the geometric ($\lambda$-weighted) average of these: ${\hat{A}}_t^{GAE}=(1-\lambda ){\sum }_{n=1}^{\infty }{\lambda }^{n-1}{\hat{A}}_t^{(n)}=(1-\lambda ){\sum }_{n=1}^{\infty }{\lambda }^{n-1}{\sum }_{l=0}^{n-1}{\gamma }^l{\delta }_{t+l}={\sum }_{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l},$ where the last step swaps the order of summation and sums the geometric series in $\lambda$. The two limits recover the endpoints of the tradeoff: $\lambda =0\Rightarrow {\hat{A}}_t={\delta }_t(\text{TD, low variance, biased}),\lambda =1\Rightarrow {\hat{A}}_t={\sum }_l{\gamma }^l{\delta }_{t+l}=G_t-V(s_t)(\text{MC, unbiased, high variance}).$ Typical values are $\lambda \approx 0.95$, $\gamma \approx 0.99$. GAE is computed by a cheap backward recursion: ${\hat{A}}_t={\delta }_t+\gamma \lambda {\hat{A}}_{t+1}$.

A2C (Advantage Actor-Critic)

A2C is the synchronous advantage actor-critic. The actor maximizes $\mathbb{E}[\log {\pi }_{\theta }(a|s)\hat{A}]$, the critic regresses toward the return, and an entropy bonus keeps the policy from collapsing.

initialize actor theta, critic w
for each iteration:
    collect a batch of transitions using pi_theta (n parallel envs)
    compute returns / advantages (e.g. GAE) with critic V_w
    normalize advantages
    actor loss   = -mean( log pi_theta(a|s) * A_hat ) - c_ent * entropy
    critic loss  = mean( (V_w(s) - return)^2 )
    take a gradient step on (actor loss + c_v * critic loss)

Key benefits: the advantage function reduces variance versus REINFORCE, and bootstrapping the critic gives a usable signal without waiting for full episodes.

A3C (Asynchronous Advantage Actor-Critic)

A3C (Mnih et al., 2016) predates A2C and uses multiple asynchronous workers: each worker holds its own environment and a copy of the parameters, computes gradients on its own rollout, and pushes them to a shared global network asynchronously (Hogwild-style updates without locks).

• Decorrelated data: many workers in different parts of the state space break the temporal correlation of samples, playing the role that experience replay plays in DQN, so no replay buffer is needed.
• Speed and exploration: parallel actors explore simultaneously and reduce overfitting to any single environment instance.

Synchronous vs. Asynchronous

Why A2C replaced A3C in practice

A3C’s asynchronous updates apply stale gradients: a worker computes a gradient against an old parameter copy, so by the time it lands the global params have moved. A2C makes the same idea synchronous: a coordinator waits for all workers, averages their gradients, and applies one clean update. This removes the staleness/noise, is more GPU-friendly (one big batched forward pass), and empirically matches or beats A3C. The asynchrony in A3C was a benefit mainly on CPU clusters, not a fundamental algorithmic advantage.

IMPALA

IMPALA scales actor-critic to many distributed actors that only send trajectories (not gradients) to a central learner. Because actors lag behind the learner’s policy, the data is slightly off-policy, so IMPALA corrects it with V-trace, a truncated importance-sampling correction on the value targets. This decoupling of acting from learning is what makes it throughput-efficient at scale.

Critic bias feeds the actor

The critic is only an approximation, so its errors bias the advantage and therefore the policy gradient. Keep the critic well-fit (multiple value epochs, target normalization) and lean toward smaller $\lambda$ only when variance, not bias, is the bottleneck.