Motivation
Instead training robots to learn how to do a specific tasks by collecting tons of specific data (much like basic ML tasks from 10 years ago), we can follow the trend of foundation models and collect tons of different data with different skills, robots, etc and try to get these robots to generalize!
The bet is the same one that reshaped NLP and vision: a single large model pretrained on broad, diverse data acquires reusable, transferable competence, and then adapts to specific tasks with little extra data. For robotics this means moving away from one hand-tuned policy per task toward generalist policies that carry semantic and physical priors into new tasks, objects, and even new robot bodies.
Challenges in Robotics
- Poor Generalization: Current robot algorithms are pretty bad at generalizing to new robotic hardware, environments; hyperparameters and algorithms are usually hand-crafted for a specific task or environment
- Data Scarcity: Data is essential for deep learning-based methods, but the huge range of tasks and environments makes it hard to create large-scale datasets. Simulation is one of the leading ways to help solve this, where synthetic data and techniques like domain randomization, help improve generalization and faster training.
- Task specification: It is hard for users to exactly specify the task, we want robots to understand things semantically.
- Uncertainty and safety: Inherent uncertainty in the environments and task specifications
Foundation Models used in Robotics
The most apparent application is for object recognition and scene understanding, extracting semantic information from the VLM and spatial information from objects/scenes that the robots interact with. Another application is for state estimation and localization, using CLIP and GPT to detect objects and generate labels, helping with SLAM.
Task planning and action generation is another obvious direction. For example, SayCan, uses LLMs… Reward Generation: Language to Rewards for Robotic Skill Synthesis
RT-2: A vision-language-action (VLA) models
Vision-Language-Action (VLA) models
A VLA takes the same backbone that powers a Vision Language Models and adds a third modality: actions. It maps an image (or several) plus a natural-language instruction directly to robot actions:
The key trick (introduced by RT-2) is to treat robot actions as just another language to be predicted. Continuous action dimensions are discretized into tokens, so end-effector deltas and gripper commands become entries in the vocabulary, and the model is co-trained on both web vision-language data and robot trajectories:
Because the same weights ingested internet-scale image-text data, the model inherits open-vocabulary recognition and commonsense reasoning: it can act on objects and instructions it never saw in the robot dataset (for example “pick up the extinct animal” resolving to a toy dinosaur). This overlaps heavily with Imitation Learning, since the robot-trajectory portion of training is behavior cloning on tokenized actions.
generalist policy
- Takes internet-scale pre-training data + Open X-embodiment dataset + collected task data, and combines that with a pre-trained VLM (PaliGemma), with another flow-matching model (action expert), to produce continuous actions.
- Model can then be fine-tuned on downstream tasks
The flow-matching (a continuous cousin of diffusion) action expert is what lets output smooth, high-frequency, multimodal continuous actions instead of coarse discretized tokens, which is exactly the strength of a Diffusion Policy. Conditioning the generative action head on VLM features marries broad semantic understanding with precise low-level control.
Sequential Reasoning for VLAs
- VLA with Embodied Chain-Of-Thought
Embodied chain-of-thought has the model reason in intermediate steps (identify the object, infer the sub-goal, plan a motion) before emitting an action, trading a little inference latency for markedly better performance on long-horizon, compositional tasks. It is the robotics analogue of letting an LLM “think step by step.”
Cross-embodiment learning
Different robots have different bodies: arm counts, joint limits, action spaces, and camera placements all vary. Cross-embodiment learning pools data across many robot types so a single policy benefits from all of it.
Why pool across robots
The Open X-Embodiment dataset aggregates data from many labs and robot platforms. Models trained on the union (RT-X) outperformed those trained on any single platform’s data, evidence that skills learned on one embodiment transfer and boost others.
The practical challenge is a shared interface: normalizing heterogeneous action/observation spaces into a common representation, or letting the model condition on the embodiment, so one set of weights controls many bodies.
Robotic Foundation Models
Offline RL most methods trained a visual policy in the simulation then transferred the model to real world
Current limitations
Where foundation models for robotics still fall short
- Data scale mismatch: robot datasets are tiny (millions of trajectories) next to the trillions of web tokens behind LLMs; embodied data remains the binding constraint.
- Inference latency: large VLA backbones are slow, which is at odds with high-frequency closed-loop control; distillation and action-chunking are common workarounds.
- Brittle generalization: performance still drops sharply under novel viewpoints, lighting, and truly unseen objects, well short of the robustness LLMs show in language.
- Weak safety guarantees: these policies do not reliably know when they are uncertain or unsafe, and can confidently take wrong physical actions.
- Evaluation is hard: real-robot evaluation is slow, expensive, and noisy, making progress difficult to measure reproducibly.
Other Interesting Directions
- Improving Simulations and Sim2Real
- LLM for Reward Design with Eureka
- Doing imitation learning
- No simulator. Collect data from real → learn a model → design a policy → deploy
- Meta-learned dynamics model + online adaptive control