Sim2Real

Train in simulation, deploy in real world (with real-time adaptation)

Why simulators for robot learning?

• Most RL-based algos are very sample inefficient
• They are cheap/fast/scalable/safe/labeled

Problems of Sim2Real

• Non-parametric mismatches (simulator doesn’t consider some effects at all)
  • complex aerodynamics, fluid dynamics, tire dynamics, etc
• Parametric mismatches (simulator uses different parameters than real)
  • robot mass/friction,etc

The reality gap

The “reality gap” is the distribution shift between the simulator and the physical world. A policy trained in sim implicitly overfits to the simulator’s quirks: its contact model, its rendering, its exact friction and mass values. Deployed on hardware, those assumptions break and performance collapses. We can frame it as a mismatch between the training dynamics $f_{\text{sim}}$ and the real dynamics $f_{\text{real}}$:

$$x_{t+1}=f_{\text{sim}}(x_t,u_t)\text{vs.}{x}_{t+1}=f_{\text{real}}(x_t,u_t)$$

The gap has two flavors, matching the note above: parametric (right equations, wrong numbers, e.g. friction coefficient off by 30%) and non-parametric / structural (the simulator omits an entire phenomenon, e.g. cable dynamics, backlash, aerodynamic drag). Parametric gaps are addressable by randomizing or identifying parameters; structural gaps are harder and usually need real-world adaptation.

Domain randomization

Rather than trying to build one perfectly accurate simulator, domain randomization deliberately trains across a distribution of simulators. If the real world looks like just another sample from that distribution, a policy robust to the whole family transfers zero-shot.

Domain Randomization

• Randomize $e$ in $x_{t+1}=f(x_t,u_t,e)$
• Train a single RL policy $\pi (x)$ that works for the whole distribution of $e$
  • Approximation of robust control
• What is randomized?
  • Physics parameters (mass, gravity, friction, etc)
  • Sensor noise (camera blur, pixel noise, quantization, etc)
  • Rendering (lighting, textures, backgrounds)

Formally, we optimize expected return marginalized over the randomization distribution $p(e)$:

$${\pi }^{\ast }=\arg \underset{\pi }{\max }{\mathbb{E}}_{e\sim p(e)}{\mathbb{E}}_{\pi }\left[\sum _t{\gamma }^{t}r(x_t,u_t)\right]$$

This is essentially an approximation of robust control: by forcing one policy to succeed across many worlds, we hope it also covers the real one.

Cube reorientation (OpenAI Dactyl)

A dexterous hand learned to reorient a cube trained only in simulation. Randomizing object mass, size, friction, actuator delays, gravity, and visual textures produced a policy that transferred to the physical Shadow Hand without any real-world training. The policy effectively learned to identify and adapt to conditions on the fly.

Tuning the randomization range

• Too narrow: the real world falls outside the training distribution and transfer fails.
• Too wide: the task becomes so hard that the policy converges to an overly cautious, low-performance behavior (or fails to learn at all).
• Automatic curricula (e.g. Automatic Domain Randomization) grow the range only as fast as the policy can handle, easing this tradeoff.

Domain adaptation

Domain adaptation reduces the gap by aligning representations rather than physics. The idea: learn features that are invariant between sim and real, so a policy trained on sim features works on real features. In vision this often means adversarially training a feature extractor so a domain classifier cannot tell whether a feature came from a rendered or a real image, or using image-to-image translation (e.g. CycleGAN, RCAN) to map real images into the simulator’s visual style before feeding the policy.

Learning to Adapt (via Privileged Information)

• Randomize $e$ in $x_{t+1}=f(x_t,u_t,e)$
• Train an adaptive RL policy $\pi (x,e)$ that works for many $e$
  • approximation of adaptive control
• Issue! $e$ is often unknown in real world
  • Solution! Learning from a privileged teacher
    • Sim: First Train a teacher policy with privileged information $\pi (x,e)$
    • Sim: Student policy ${\pi }_s(x,\text{available info in the real})$ learns from $\pi (x,e)$
    • Real: Deploy student policy ${\pi }_s(x,\text{available info in the real})$
  • Basically becomes an Imitation Learning problem

System identification and real-to-sim

System identification (sysID) attacks the parametric gap head-on: measure real trajectories and fit the simulator’s parameters $e$ so it matches. This is the classic least-squares fit:

$$e^{\ast }=\arg \underset{e}{\min }\sum _t{\Vert x_{t+1}^{\text{real}}-f_{\text{sim}}(x_t^{\text{real}},u_t;e)\Vert }^2$$

The modern “real-to-sim” or digital-twin view generalizes this: use real data to build or correct the simulator (calibrating physics, reconstructing scene geometry with NeRF/Gaussian splatting, learning residual dynamics), then train in that improved sim. A residual model is a common compromise, keeping the analytic simulator and learning only the mismatch:

$$f_{\text{real}}(x,u)\approx f_{\text{sim}}(x,u)+g_{\theta }(x,u)$$

where $g_{\theta }$ is fit from real interaction. This connects tightly to World Models, where the “simulator” is itself learned from data, and closes the loop: sim to real to sim.

Why it matters for robot learning

Data is the bottleneck in robot learning. Real interaction is slow (real time), costly (hardware wear, human supervision), and unsafe (a bad exploratory action can break the robot or its surroundings). Simulation offers cheap, fast, parallel, perfectly labeled, and safe data. Sim2Real is the bridge that lets us cash in those advantages: it is what makes large-scale RL, aggressive exploration, and rare-event training feasible for physical robots. Without a strategy to cross the reality gap, a simulator-trained policy is just an expensive demo.

Broader pitfalls

• Silent overfitting to sim artifacts: great sim numbers tell you little; always measure on hardware.
• The observation gap, not just dynamics: cameras, latency, and actuator noise differ too, so randomize sensing and control, not only physics.
• Compounding at deployment: small per-step model errors accumulate; closed-loop adaptation (adaptive policies, online sysID) matters more than one-shot accuracy.

Connections

Sim2Real sits between World Models (learned, data-driven simulators and residual dynamics) and Imitation Learning (the teacher-student privileged-information trick reduces to imitation). Domain randomization is also the workhorse that lets Foundation Models for Robotics scale synthetic data across environments.