Offline RL

Offline (batch) RL learns a policy purely from a fixed, previously collected dataset, with no further interaction with the environment during training. The central difficulty is that the agent cannot try out new ideas, so it must avoid trusting value estimates for actions the data never covered.

Tldr

Offline RL is a type of RL where the agent does NOT interact with the environment during training. Instead it learns from a fixed dataset of logged experience: (state, action, reward, next state) tuples.

Setup

The dataset is a fixed collection of transitions gathered by some behavior policy ${\pi }_{\beta }$ (a prior agent, a mix of policies, or human demonstrations): $D=\{(s_i,a_i,r_i,s_i^{\prime }){\}}_{i=1}^n,a_i\sim {\pi }_{\beta }(\cdot \mid s_i).$

• $s_i$: state (observation) at step $i$.
• $a_i$: action taken by the behavior policy.
• $r_i$: reward received.
• $s_i^{\prime }$: resulting next state.

$D$ is collected before training and cannot be augmented. The goal is still to find a policy $\pi$ maximizing expected return, but now evaluated on a distribution we can only sample indirectly through $D$.

Distribution Shift and Extrapolation Error

The learned policy $\pi$ induces a state-action distribution that differs from ${\pi }_{\beta }$‘s. When the Bellman target evaluates ${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })$, the maximizing $a^{\prime }$ is often an out-of-distribution (OOD) action the dataset never contains. The function approximator extrapolates arbitrarily there, and typically overestimates, since the max seeks out errors. $y=r+\gamma {\max }_{a^{\prime }}{Q}_{\theta }(s^{\prime },a^{\prime }),$ if the argmax lands on an unsupported $a^{\prime }$, $y$ is anchored to a fantasy value. There is no environment feedback to correct it, so the error compounds through bootstrapping.

Why off-policy methods fail offline

Online, off-policy methods like DQN correct overestimation because the agent eventually tries the overvalued action, sees a low reward, and pushes the value back down. Offline, that corrective loop is severed: overestimated OOD actions are never sampled, so errors accumulate without bound and the policy chases them. More data does not fix this; it is a query problem, not a sample-size problem.

Formally the issue is a query distribution mismatch: the Bellman backup queries $Q$ at $(s^{\prime },\pi (s^{\prime }))$, but $Q$ is only reliable on the support of ${\pi }_{\beta }$.

Policy Constraint Methods (BCQ, BEAR)

Keep the learned policy close to the behavior policy so that value queries stay in-distribution.

• BCQ (Batch-Constrained Q-learning): trains a generative model (VAE) of ${\pi }_{\beta }$ and only considers actions near the behavior actions, then perturbs them slightly. The max is taken over a constrained action set rather than all actions.
• BEAR: relaxes the hard constraint to a distributional one, keeping $\pi$ within a support set of ${\pi }_{\beta }$ using a Maximum Mean Discrepancy (MMD) penalty, so unlikely-but-supported actions remain available while OOD ones are excluded.

Tradeoff: constraining to ${\pi }_{\beta }$ limits improvement when the data is poor, and estimating ${\pi }_{\beta }$ well can be hard for multimodal or narrow datasets.

Conservative Value Methods (CQL)

Instead of constraining the policy, penalize the values themselves. Conservative Q-Learning (CQL) adds a regularizer that pushes down Q-values on OOD actions and pulls up values on dataset actions, learning a lower bound on the true $Q^{\pi }$:

$$\underset{\theta }{\min }\alpha ({\mathbb{E}}_{s\sim D,a\sim \mu }[Q_{\theta }(s,a)]-{\mathbb{E}}_{s,a\sim D}[Q_{\theta }(s,a)])+\frac{1}{2}{\mathbb{E}}_D[(Q_{\theta }-\hat{\mathcal{B}}Q{)}^2].$$

Because the value is conservative, the greedy policy will not be lured toward unsupported actions. CQL avoids explicitly modeling ${\pi }_{\beta }$, which makes it robust and popular in practice.

Implicit Methods (IQL)

Implicit Q-Learning (IQL) never queries $Q$ at OOD actions at all. It fits $V(s)$ to an expectile of $Q(s,a)$ over dataset actions, approximating the max using only in-sample actions: $L_V={\mathbb{E}}_{(s,a)\sim D}\left[L_2^{\tau }(Q_{{\theta }^{-}}(s,a)-V_{\psi }(s))\right],L_2^{\tau }(u)=|\tau -1(u<0)|u^2,$ where the expectile parameter $\tau \to 1$ approaches the in-support max. The critic then bootstraps with $r+\gamma V_{\psi }(s^{\prime })$ (no max over actions), and the policy is extracted afterward by advantage-weighted regression: ${\max }_{\pi }{\mathbb{E}}_{(s,a)\sim D}\left[\exp (\beta (Q(s,a)-V(s)))\log \pi (a\mid s)\right].$ IQL is simple, stable, and strong because it decouples value learning (never leaves the data) from policy extraction.

Sequence-Model Approaches (Decision Transformer)

Decision Transformer reframes offline RL as conditional sequence modeling rather than value learning. It feeds a transformer a sequence of (return-to-go, state, action) tokens and trains it to predict the next action autoregressively: ${\hat{a}}_t=\text{Transformer}(\dots ,{\hat{R}}_t,s_t),{\hat{R}}_t={\sum }_{k\ge t}{r}_k.$ At test time you condition on a desired high return-to-go and let the model generate actions. It sidesteps bootstrapping and the deadly triad entirely (it is supervised learning), which avoids OOD value explosions, but it cannot “stitch” together sub-trajectories as well as value-based methods and relies on the dataset containing high-return behavior.

Model-Based Offline RL

An orthogonal family learns a dynamics model $\hat{P},\hat{R}$ from $D$ and plans or trains inside it, which lets the agent generate synthetic rollouts and stitch behavior more aggressively. The catch is the same distribution shift: the model is only accurate near the data.

• MOPO penalizes rewards by an uncertainty estimate of the model, $\tilde{r}=\hat{r}-\lambda u(s,a)$, so the policy is discouraged from exploiting regions where the model is unreliable.
• MOReL builds a pessimistic MDP that routes unknown state-actions to a low-reward absorbing state.

The unifying theme across all offline methods is pessimism under uncertainty: constrain the policy, lower-bound the values, or penalize unknown dynamics.

When Offline RL Helps vs Behavior Cloning

Both learn from a fixed dataset, but they answer different questions.

• Behavior cloning (Imitation Learning) fits $\pi (a\mid s)\approx {\pi }_{\beta }$: it copies the data. It works only if the data is near-expert and cannot exceed the demonstrator.
• Offline RL uses rewards and Bellman backups to improve over ${\pi }_{\beta }$ by stitching good decisions from different trajectories.

When each wins

If the dataset is expert and dense, behavior cloning is simple and hard to beat. If the dataset is a mixture of mediocre trajectories that each contain useful segments, offline RL can stitch them into a policy far better than any single demonstrator, which cloning cannot do. When the dataset is narrow and the reward signal weak, cloning is the safer choice because offline RL’s value estimates have little support to stand on.