A world model gives a robot an internal simulator. Show it the current view and the action you want to take, and it generates a realistic video of what would happen next — letting the robot "imagine" outcomes before moving a single motor in the real world.

Our model, IRASim, makes every generated frame line up precisely with the action behind it. The result is video that captures the subtle stuff robots actually struggle with — a bowl slipping from the gripper, a block being nudged, a drawer sliding shut — in high resolution and over long horizons.

Why it matters:

• Test policies without a real robot. Evaluations in IRASim closely track what happens on the real hardware.
• Smarter robots that "think before they act." Letting a policy imagine several options and pick the best one substantially improves task success.
  
• Drive the robot yourself. Control the virtual arm with a keyboard or VR controller — IRASim follows along in real time, even for moves it has never seen.

More details can be found at: https://gen-irasim.github.io/

See the demos below 👇

If a world model can faithfully imagine what happens next, then a robot can do more than just watch — it can practice inside its own head.

That's the idea behind our follow-up work, WMPO (World-Model-based Policy Optimization). Instead of sending the robot back into the lab to collect thousands of new trials, we let it train entirely inside the imagined world. The robot tries, fails, tries again — all in pixels generated by our world model — and gradually becomes better at the real task.

What this unlocks:

• Learning from failure, not just demonstrations. Robots trained by copying humans never see what going wrong looks like. Inside the world model, failures are cheap and abundant — so the policy learns how to recover from them.
• Emergent self-correction. After training in imagination, the robot starts doing things it was never shown: nudging a misaligned part back into place, lifting and retrying instead of getting stuck.
• Far less real-robot data. A small handful of real rollouts is enough to ground the imagination; the rest of the practice happens for free.
• It transfers. A policy refined entirely in imagination still works on the real robot — including a delicate insertion task with only 5 mm of clearance.

More details can be found at: https://wm-po.github.io/

What if the robot didn't need a separate world model running alongside it — what if imagining the future became part of how it thinks?

That's the leap behind HALO. Given a goal like "arrange the blocks in red-green-blue order," it works the way people do:

• 💭 Think it through in words — break the task into clear steps.
• 🖼️ Picture the next moment — generate what the scene should look like after the next step.
• 🤖 Then act, guided by both.

Why it matters:

• Robust to messy, unfamiliar scenes. Changed lighting, new background, unseen objects — HALO keeps working, because it's reasoning about the task, not pattern-matching pixels.
• Long, multi-step tasks become tractable — like sweeping buttons into a dustpan, or opening a drawer to place something inside.
• You can see what it's thinking. The reasoning and the imagined goal are right there — no black box.

Our code is released at: https://github.com/qshou-coder/HALO

See HALO in action below 👇

Robots fail a lot. Most methods treat a failed attempt as one big "bad example" — but inside a failed try, most of the actions were fine. The mistake is usually one specific moment that sent everything off course.

RedFlow zooms in on those moments. For each failure, it pinpoints the exact step that went wrong, finds a similar moment from a successful attempt, and uses it as a concrete example of what should have happened. Instead of just "don't do that," it shows the robot "do this instead."

Why it matters:

• Real recovery skills emerge. When a T-shirt slipped out of one arm's reach, the robot learned — from past failures alone — to use the other arm to pull it back and retry. No one demonstrated this.
• Far less practice needed — roughly 10× less data than methods requiring fresh real-world trials, making real-robot training actually practical.

See RedFlow recover from its own mistakes below 👇

Research is one thing. Getting it to actually run on a robot in front of you — synchronized cameras, smooth motion, safe execution — is another.

RTC-Anything is the deployment framework we built to close that gap. It takes any of the policies above and runs them on real Agilex robots, with real-time action chunking that keeps motion fluid instead of jerky. The framework is model-agnostic (swap in your own policy backend) and task-agnostic (clothes folding, sweeping, cleaning — all share the same runtime).

What's included:

• Out-of-the-box support for Agilex Cobot Magic (dual-arm) and Piper (single-arm) robots.
• An open dataset on Hugging Face for training your own policies.
• Step-by-step guides for three real-world tasks we've already validated.

You can find RTC-Anything at: https://github.com/PEILAB-PhysAI/RTC-Anything

Everything is open-source — grab the code, plug in your robot, and start deploying 👇