LeRobot v0.6.0 adds world‑model policies, reward models, new benchmarks and faster dataset tooling

LeRobot v0.6.0 aims to close the robot‑learning loop by adding policies that learn to imagine future observations, a unified reward‑models API, a deployment CLI that turns run‑time failures into training data, and six new simulation benchmarks accessible through a single CLI. The release also introduces depth sensing, VLM‑powered dataset annotation, customizable video encoding, HF Jobs cloud training, and a leaner installation.

Key highlights

• Three world‑model policies: VLA‑JEPA, LingBot‑VA, FastWAM — each trains by imagining future frames or actions.
• New and updated VLAs: GR00T N1.7, MolmoAct2, EO‑1, EVO1, Multitask DiT, and others.
• New reward models API (lerobot.rewards) including Robometer and TOPReward.
• Six new simulation benchmarks under lerobot‑eval.
• lerobot‑rollout CLI with DAgger‑style human‑in‑the‑loop correction collection.
• FSDP training, cloud training via HF Jobs, end‑to‑end depth video, and automated language annotations.
• Configurable video codecs and up to 2x faster data loading out of the box.

World models: policies that imagine

v0.6.0 includes three policies that incorporate future imagination during training:

• VLA‑JEPA: trains a compact VLA (built on Qwen3‑VL‑2B) to predict future latent frames via a JEPA world model during training; the world model is not used at inference, providing world‑model supervision without extra inference cost. Three ready checkpoints are available on the Hub.

• LingBot‑VA: an autoregressive video‑action model that jointly predicts future video and actions chunk by chunk, feeding real observations back in to keep predictions grounded. You can optionally save predicted videos to compare imagined vs. actual outcomes. Inference fits on a single 24–32 GB GPU.

• FastWAM: pairs a ~5B video‑generation expert with a compact action expert in a single network so the model learns to generate its own rollouts during training. At inference it skips generation and directly denoises action chunks.

See the LeRobot docs and the referenced papers for technical details.

VLAs: the model zoo keeps growing

New and refreshed VLA integrations:

• GR00T N1.7: NVIDIA GR00T integration upgraded to N1.7, now using Cosmos‑Reason2‑2B (Qwen3‑VL) with a flow‑matching action head. The integration is parity‑tested against NVIDIA’s Isaac‑GR00T; flash‑attention is optional and NVIDIA checkpoints can be loaded directly. GR00T N1.7 replaces N1.5 in LeRobot (pin lerobot==0.5.1 if you need N1.5).

• MolmoAct2: the Allen Institute for AI’s vision‑language‑action model is ported into LeRobot with lifecycle support (full or LoRA fine‑tuning, evaluation, and real‑robot deployment). Prepackaged checkpoints include calibration corrections; inference fits in ~12 GB at bf16, and LoRA fine‑tuning fits on a single 24 GB GPU.

• EO‑1: a VLA pretrained on interleaved vision‑text‑action data with a Qwen2.5‑VL‑3B backbone and flow‑matching action head, contributed by an author of the paper.

• Multitask DiT: a ~450M parameter diffusion transformer conditioned on CLIP vision and language embeddings, supporting diffusion and flow‑matching objectives for multitask learning by natural language.

• EVO1: a compact 0.77B parameter policy (InternVL3‑1B backbone, flow‑matching action head) designed for real‑time use and small GPU footprints.

Documentation pages include usage and training recipes.

Reward models: knowing when your robot succeeds

LeRobot introduces a unified reward models API (lerobot.rewards) that mirrors the policies API and exposes multiple reward models, including two new additions:

• Robometer: a pretrained, general‑purpose reward model (lerobot/Robometer‑4B). Given raw video and a language instruction, it scores task progress and success without task‑specific training. It is based on Qwen3‑VL‑4B and trained using trajectory comparisons over a dataset of more than one million robot trajectories.

• TOPReward: a zero‑shot approach that wraps an off‑the‑shelf VLM (Qwen3‑VL) and reads the log‑probability of the token "True" given the trajectory video and task instruction; any capable VLM becomes a reward function.

Both models include labeling scripts that write per‑frame progress curves into datasets for reward‑aware behavior cloning, dataset inspection, and progress overlay videos.

Datasets: faster loading, richer data

• Custom video codec options: the new --dataset.rgb_encoder.* settings expose codec, quality, pixel format, GOP and presets. vcodec=auto probes for hardware encoders (NVENC, VideoToolbox, VAAPI, QSV) before falling back to software AV1. Existing datasets can be reencoded via a lerobot‑edit‑dataset command.

• Depth support end‑to‑end: with an Intel RealSense and use_depth: true, LeRobot records depth maps in millimeters as compact 12‑bit depth video streams alongside RGB, and decodes them back to physical units at training time. Live depth rendering is available during recording and in lerobot‑dataset‑viz; supported platforms include SO‑100/101, Koch, OpenArm, reBot, Unitree G1 and more.

• Language annotations at scale: datasets can now store timestamped subtasks, plans, memory, corrections, speech, and per‑camera VQA pairs. The lerobot‑annotate CLI uses a VLM (e.g., Qwen/Qwen2.5‑VL‑7B‑Instruct) to auto‑fill these annotations and render them into chat‑style training messages via a YAML recipe layer.

• Up to 2x faster data loading: parallel multi‑camera decode, worker processes passing compact uint8 frames (4x less inter‑process memory), persistent worker decoder caches, and deterministic resumable sampling. Example benchmark: loading a subset went from 275 s to 0.06 s.

Benchmarks: one CLI to evaluate them all

Six new simulation benchmarks are available under lerobot‑eval, each with docs, a Docker image, and a SmolVLA baseline smoke‑tested in CI:

• LIBERO‑plus: ~10,000 perturbed LIBERO variants testing robustness across lighting, viewpoints and rewritten instructions.
• RoboTwin 2.0: 50 bimanual manipulation tasks on SAPIEN with heavy domain randomization and more than 100k ready‑to‑train trajectories on the Hub.
• RoboCasa365: 365 kitchen tasks in 2,500 procedurally generated kitchens on a mobile manipulator.
• RoboCerebra: long‑horizon evaluation chaining 3–6 subgoals with a 6,660‑episode dataset.
• RoboMME: memory exam across 16 tasks in 4 memory suites (counting repetitions, tracking hidden objects, imitating procedures).
• VLABench: reasoning and knowledge tests for manipulation tasks from physics questions to composite workflows.

Simulator backends have system dependencies; each benchmark doc lists the install recipe and ready Docker images are provided to skip setup. Together with existing LIBERO, Meta‑World and NVIDIA IsaacLab‑Arena families, LeRobot now hosts nine benchmark families under one roof.

Training & inference

• lerobot‑rollout: a dedicated deployment CLI offering strategies such as base, sentry, highlight, episodic and dagger. The dagger strategy records human interventions: when the operator corrects a failing policy, correction frames are tagged with an intervention flag and collected into a dataset ready for fine‑tuning.

• FSDP: support for Fully Sharded Data Parallel training via Accelerate so models larger than a single GPU can be trained; checkpoints are reassembled into a single model.safetensors file and runs may be resumed on a different number of GPUs.

• HF Jobs cloud training: the same lerobot‑train command can be run in the cloud by adding --job.target, with LeRobot uploading datasets to a private Hub repo if needed, submitting the job, streaming logs and pushing the trained policy to the Hub. Compute options range from T4 to 8x H200 billed pay‑as‑you‑go.

Codebase: leaner and cleaner

• pip install lerobot is significantly lighter (about 40% fewer base dependencies). Feature‑scoped extras cover optional capabilities.
• Supported PyTorch versions 2.7–2.11, with CUDA 12.8 wheels pinned for Linux uv installs.
• --policy.dtype=bfloat16 enables real mixed‑precision training via Accelerate.
• uv.lock is the authoritative dependency spec for CI/Docker/development and install routes are documented.
• Foxglove support for streaming teleoperation and recordings (--display_mode=foxglove) and scrubbable dataset playback via lerobot‑dataset‑viz.
• lerobot_env_* pip packages self‑register; plugin system covers robots, cameras, teleoperators, policies and envs.
• Keyboard controls during recording now work on Wayland, over SSH, on headless rigs, and on macOS without Accessibility permissions.

See the release notes for the full list of changes and migration pointers for breaking changes.

Community & ecosystem

• LeLab: a browser UI that provides the full LeRobot workflow (calibrate, teleoperate, record, train locally or on HF Jobs, deploy); currently supports SO‑ARM101.
• Isaac Teleop: SO‑101 teleoperation via VR controller using NVIDIA’s Isaac Teleop stack over CloudXR/OpenXR, developed with NVIDIA.
• A compute hardware guide documents GPU needs and reference training times (from RTX 4090 to 4x H100).
• An updated Adding a Policy guide explains how to ship your own policy in tree or as a plugin package without a PR.

Final thoughts

Beyond the headline features, v0.6.0 includes hundreds of bug fixes, documentation improvements and quality‑of‑life upgrades across the codebase. The LeRobot team thanks the community contributors from academia, industry and hobbyist projects whose PRs and reports helped shape this release.

Get started via the project documentation — The LeRobot team ❤️