2020MuZero: reward/value-only world model masters Go, chess, Atari — no pixel generation
2025DreamerV3 in Nature (150+ benchmarks); V4 replaces RNN with transformers, 25× faster
Thread B · 2016–2025
Learning from Watching (Video)
2016Finn et al.: action-conditional video prediction for real robot pushing — too blurry for complex tasks
2022R3M, VPT: internet-scale video pretraining transfers physical knowledge to robots
2024Sora: video diffusion at scale — objects fall, light scatters. But bidirectional, not interactive
2024Genie 1: latent action models learn interaction from unlabeled video
2025AR-DiT + Self Forcing: video diffusion becomes causal and real-time — the convergence moment
2026DreamDojo: r=0.995 policy evaluation; DreamZero: joint video + action prediction
📐 Five Properties of a True World Model
Xun Huang proposed five properties that separate a world model from a video generator. The first two are binary prerequisites.
Hard Constraint
🕐 Causal
Time flows forward only. Bidirectional attention (like Sora) violates this. Without causality, you cannot inject an action mid-stream.
Hard Constraint
🕹️ Interactive
Responds to actions in real time. Without this, it's a movie, not a simulation. The closed loop — act, see consequence, react — is what robotics requires.
Spectrum
🔁 Persistent
Maintains coherence over duration. Current SOTA: Genie 3 sustains minutes at 720p. Hours remain unsolved for any system.
Spectrum
⚡ Real-time
Fast enough for the application. SOTA: 10–30 FPS. DreamZero runs at 7Hz — sufficient for some robotics, not for fast manipulation.
Spectrum
🌍 Physically Accurate
Respects real-world physics. The hardest property and the most contested. Generating convincing video ≠ accurate physical simulation.
🎯 What Are World Models Good For?
From most proven to most speculative — the honest maturity map of world model applications.
The 800-pound gorilla. Every layer open-sourced under Apache 2.0. Strategy: become the CUDA layer for physical AI — give away the software, sell the hardware (Blackwell GB200).
14B params, 200M video clips. Apache 2.0 open-source.
For startups building pure world models, this is existential: DreamDojo is free. The moat must come from domain-specific data NVIDIA doesn't have, faster inference, or vertical integration.
🧠 The JEPA Contrarian Bet
Not everyone is building video world models. Yann LeCun and AMI Labs ($1.03B) are betting that predicting pixels is fundamentally wasteful.
🧮
The JEPA Argument
Most pixel-level detail is irrelevant to understanding dynamics. JEPA encodes observations into abstract representations and predicts future representations directly — never generating video. V-JEPA 2: pretrained on 1M+ hours, fine-tuned on 62 hours of robot data, 80% zero-shot pick-and-place.
🎥
The Counter-Argument
Pixel-level prediction may capture physical details that abstract representations miss. More importantly: you can watch what a video model thinks will happen. JEPA's predictions are abstract vectors that humans cannot inspect. Physical contact forces, deformation, and texture may require pixel-level fidelity.
🔀 The Great Parallel
Jim Fan (NVIDIA GEAR Lab) names the pattern: robotics is copying the LLM playbook step by step.
📚
Step 1 · Pretraining
LLMs: predict the next token on internet text. World models: predict the next physical state from internet video. Cosmos and DreamDojo are doing this now.
🎯
Step 2 · Action Fine-tuning
LLMs: instruction tuning / RLHF collapses the distribution to useful outputs. Robots: action fine-tuning collapses simulation onto the thin slice that matters for real-world tasks.
🏋️
Step 3 · RL Last Mile
LLMs: RL carries alignment and reasoning. Robots: RL carries the last mile of dexterous manipulation where pattern matching from demonstrations isn't enough.
"Our generation was born too late to explore the Earth and too early to explore the stars. We may have arrived just in time to teach machines how to dream the physical world into existence." — Henry Yin & Naomi Xia
🔵 Knowledge Graph Explorer
30 entities · 33 relationships · click nodes/edges to resolve IRIs via URIBurner
Click node to inspect · drag to pin · dbl-click unpin
👤 People🏢 Orgs📦 Systems💭 Concepts
Charge strength
Link distance
Edge labels
Resolver
Arrow style
Person
Organization
System/Model
Concept
❓ FAQ
Key questions about world models, the JEPA bet, and the investment landscape.
Xun Huang proposed five properties: Causal (time flows forward only), Interactive (responds to actions in real time), Persistent (maintains coherence over duration), Real-time (10–30 FPS), and Physically accurate (respects real-world physics). Causality and interactivity are binary — without them you have a movie, not a world model. Sora fails on both; DreamDojo satisfies all five.
DreamDojo — pretrained on 44,711 hours of egocentric video, achieving r=0.995 policy evaluation accuracy — is free under Apache 2.0. "We built a world model" is no longer a moat. Defensibility must come from domain-specific data NVIDIA doesn't have, faster inference, or vertical integration into a product beyond the model itself.
Yann LeCun and AMI Labs ($1.03B) argue that predicting pixels is fundamentally wasteful. JEPA predicts in abstract representation space, never generating video. V-JEPA 2 achieved 80% zero-shot pick-and-place from just 62 hours of robot data. The counter: pixel prediction may capture physical details abstract representations miss, and you can watch what a video model thinks will happen.
No — they are converging. Pi-0.7 (the leading VLA, April 2026) integrates a lightweight world model for subgoal planning. The question isn't which approach is winning; it's whether either is close enough to general manipulation that scaling will finish the job.
Jim Fan's observation that robotics is copying the LLM playbook: world model pretraining mirrors token prediction; action fine-tuning mirrors instruction tuning; RL carries the last mile. The same three-step recipe that took LLMs from GPT-3 to o1 in six years. If correct, current systems are the GPT-2 of Physical AI.
Policy evaluation is the clearest near-term value: DreamDojo achieves r=0.995 with real-world outcomes. AV simulation (Wayve GAIA) is the most mature production deployment. Entertainment (Oasis, GameNGen) has the most tangible demos. Direct robot control remains the most speculative.
OpenAI shut down the Sora app in March 2026, framing it as a pivot to "world simulation for robotics." Sora uses bidirectional attention — making it a movie generator, not a true world model. The team lead left OpenAI three weeks after the shutdown announcement.
Mechanism (Genie) that discovers an action space from unlabeled video by compressing "what changed between two frames" into a vector — without labeled actions.
Recurrent State Space Model (Hafner, PlaNet 2019). Combines deterministic memory with stochastic uncertainty — solved a core representation problem enabling the Dreamer series.
Autoregressive Diffusion Transformer (Xun Huang, CVPR 2025). Makes video diffusion causal — generates frames sequentially rather than all at once. Prerequisite for real-time interactivity.
NeurIPS 2025 technique distilling 35 denoising steps into 4. Enables real-time interactive generation from general-purpose video foundation models for the first time.
NVIDIA's scaling law (R²=0.9983) between hours of human egocentric video and robot task performance — the empirical foundation for using human internet video to train robots.
2022 system (Hafner et al.) that deployed Dreamer on real robots. Quadruped learned to walk in 1 hour by imagining thousands of practice runs between each real attempt.
Visual encoder (CoRL 2022) pretrained on Ego4D egocentric video. A Franka arm using R3M features learned manipulation from 20 demos — early proof human video transfers physical knowledge.