Inverse RL

Main Idea

Learn the Reward function of the environment based on expert’s demonstrations, then find the optimal policy using reinforcement learning

Algorithm

• start with a set of expert’s demonstrations
• then estimate the parameterized reward function that would cause the expert’s behavior/policy
• Repeat the following until good-enough policy is achieved
  • Update the reward function parameters
  • Then we solve the reinforced learning problem (given the reward function, we try to find the optimal policy).
  • Finally, we compare the newly learned policy with the expert’s policy.

Two main approaches: model-given and model-free

Why recover a reward at all?

Behavior cloning (see Imitation Learning) copies the expert’s actions directly. Inverse RL (IRL) instead asks a deeper question: what were they trying to achieve? The reward function is often the most compact, transferable description of a task. A good reward generalizes to new dynamics, new start states, and longer horizons in a way that a cloned action mapping usually does not. Recovering intent rather than surface behavior is the core promise of IRL.

Formally, we assume the expert acts (near) optimally with respect to some unknown reward $R^{\ast }(s,a)$, typically linear in features:

$$R(s,a)=w^{\top }\varphi (s,a)$$

The value of a policy is then linear in expected feature counts (the discounted average of features visited):

$$\mu (\pi )=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t\varphi (s_t,a_t)|\pi \right],\mathbb{E}[R\mid \pi ]=w^{\top }\mu (\pi )$$

Matching the expert’s expected return therefore reduces to matching feature expectations $\mu (\pi )\approx \mu ({\pi }_E)$.

The ambiguity problem

IRL is fundamentally ill-posed. Many reward functions explain the same demonstrations equally well, and some are degenerate.

Reward ambiguity

• The all-zero reward $R\equiv 0$ makes every policy optimal, so it trivially “explains” any demonstration.
• Scaling a reward by a positive constant, or adding a potential-based shaping term, leaves the optimal policy unchanged.
• With finite demonstrations, feature counts are estimated with noise, so many $w$ fit within sampling error.

Because of this, every practical IRL method needs an extra principle to pick a unique, non-degenerate reward: a margin (apprenticeship learning), an entropy criterion (MaxEnt), or an adversarial discriminator (GAIL).

Apprenticeship learning (max-margin)

Abbeel and Ng’s approach avoids committing to a single reward and instead finds a policy whose feature expectations match the expert’s. It iterates a max-margin step: find the weight vector $w$ (with $\Vert w{\Vert }_2\le 1$) that most separates the expert from all policies found so far,

$$\underset{t,\Vert w{\Vert }_2\le 1}{\max }t\text{s.t.}{w}^{\top }\mu ({\pi }_E)\ge w^{\top }\mu ({\pi }_i)+t\forall i,$$

then solve the RL problem for that $w$, add the resulting policy, and repeat until the margin $t$ falls below a tolerance. At convergence no reward in the class distinguishes the learner from the expert, so their expected returns are provably close.

Maximum-entropy IRL

Maximum-entropy IRL (Ziebart et al.) resolves ambiguity with a clean probabilistic principle: among all trajectory distributions that match the expert’s feature counts, choose the one with maximum entropy (the least committed, most uniform one). This yields an exponential-family distribution over trajectories $\tau$:

$$p(\tau )=\frac{1}{Z}\exp \left(\sum _t{R}_w(s_t,a_t)\right)=\frac{1}{Z}{e}^{w^{\top }\mu (\tau )},Z=\sum _{\tau }{e}^{w^{\top }\mu (\tau )}$$

Higher-reward trajectories are exponentially more likely, but the distribution never collapses onto a single path, which naturally models suboptimal, noisy human demonstrations. We fit $w$ by maximizing the log-likelihood of the observed demonstrations; its gradient is beautifully interpretable:

$${\nabla }_w\mathcal{L}=\mu ({\pi }_E)-{\mathbb{E}}_{p(\tau )}[\mu (\tau )]$$

That is, adjust the reward so the model’s expected feature counts match the expert’s. The bottleneck is the partition function $Z$: computing it exactly requires the MDP’s dynamics and dynamic programming. Deep MaxEnt variants (e.g. Guided Cost Learning) estimate the expectation with sampled rollouts, which removes the need for known dynamics.

MaxEnt gradient step

1. Given current $w$, compute the soft-optimal policy under $R_w$ (soft value iteration).
2. Roll it out to estimate the learner’s expected feature counts ${\mathbb{E}}_{p(\tau )}[\mu (\tau )]$.
3. Update $w\leftarrow w+\eta (\mu ({\pi }_E)-{\mathbb{E}}_{p(\tau )}[\mu (\tau )])$.
4. Repeat. Convergence means the learner visits features just like the expert.

Relation to adversarial imitation (GAIL)

The inner RL loop of classic IRL is expensive: every reward update requires (near) fully solving an RL problem. Generative Adversarial Imitation Learning (GAIL) sidesteps recovering an explicit reward and instead matches the occupancy measure (state-action visitation distribution) of expert and learner directly, in a GAN-style game:

$$\underset{\pi }{\min }\underset{D}{\max }{\mathbb{E}}_{\pi }[\log D(s,a)]+{\mathbb{E}}_{{\pi }_E}[\log (1-D(s,a))]-\lambda H(\pi )$$

The discriminator $D$ tries to tell learner transitions from expert ones; its output acts as a surrogate reward $r(s,a)=-\log D(s,a)$ that the policy maximizes via Policy Gradient methods. This is exactly the max-entropy IRL objective with the reward folded into a learned discriminator, which is why GAIL is understood as adversarial IRL without an explicit reward readout (AIRL does recover a reward this way). It scales to high-dimensional, continuous-control robot tasks far better than classic IRL.

Contrast with behavior cloning

	Behavior Cloning	Inverse RL
Learns	A direct state to action map	A reward function (then a policy)
Objective	Supervised action matching	Match intent / feature expectations
Compounding error	Severe (covariate shift)	Mitigated: policy is RL-optimized, revisits states
Transfer to new dynamics	Poor	Good: reward is dynamics-agnostic
Cost	Cheap, one pass	Expensive: RL in the inner loop
Data assumption	Needs many labeled actions	Can work from trajectories/states

Practical pitfalls

• Inner-loop cost: solving RL to optimality for every reward update is the main scalability bottleneck; sampled-based and adversarial variants trade exactness for speed.
• Feature design: linear-reward IRL is only as good as $\varphi (s,a)$; a missing feature means the true intent is unrepresentable.
• Suboptimal experts: if demonstrators are inconsistent, deterministic IRL breaks; MaxEnt’s stochastic model is more forgiving.
• Reward non-identifiability: even a “recovered” reward is unique only up to shaping and scaling, so read learned rewards with care.

Applications

Applications

• Simulated highway driving
• imagery-based navigation