Embodied AI 101

NVIDIA Cosmos: World Foundation Models for Physical AI

Shaoqing Tan — Wed, 20 May 2026 05:19:16 -0700

World foundation models for video and physics prediction with SynthID watermarking for responsible AI practices. Developed in collaboration with Google DeepMind.

LATENT: Teaching a Humanoid to Play Tennis from Imperfect Data

Shaoqing Tan — Tue, 19 May 2026 14:11:09 -0700

Introduces a three-stage pipeline that extracts a latent action space from noisy, low-quality human motion capture, then trains a high-level RL policy in simulation to compose and execute dynamic whole-body tennis skills. Achieves volleys at human-level performance on a humanoid robot.

CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models

Shaoqing Tan — Tue, 19 May 2026 05:26:22 -0700

Closed-loop framework coupling Vision-Language Models with Video Generation Models at step-level granularity. Mitigates long-horizon drift and mid-clip errors in goal-directed video reasoning for robotic planning.

World Action Models: The Next Frontier in Embodied AI

Shaoqing Tan — Tue, 19 May 2026 05:10:48 -0700

First systematic survey defining World Action Models (WAMs) as embodied foundation models that jointly predict future states and generate actions. Covers architectures, data ecosystems, and evaluation protocols.

Training a Whole-Body Control Foundation Model

Shaoqing Tan — Mon, 18 May 2026 14:26:59 -0700

Describes end-to-end learning of a foundation model for adaptive whole-body humanoid control via massive simulation variation. Combines proprioceptive perception and policy adaptation across embodiments.

DexJoCo: A Unified Benchmark for Task-Oriented Dexterous Manipulation

Shaoqing Tan — Mon, 18 May 2026 14:11:24 -0700

Releases an open-source MuJoCo-based benchmark with 11 dexterous tasks, low-cost teleoperation hardware, and 1.1K human demonstrations. Designed to evaluate and train modern VLA/robotic policies.

MMSkills: Building Multimodal Skill Libraries for Visual Agents

Shaoqing Tan — Mon, 18 May 2026 05:29:29 -0700

Skill library, demonstrations, and dataset for multi-modal robotic skill learning and manipulation tasks.

PhysBrain 1.0 VLA (TwinBrainVLA): Dual-Brain Vision-Language-Action with Physics-Grounded Learning

Shaoqing Tan — Mon, 18 May 2026 05:16:17 -0700

Introduces dual-brain fusion Vision-Language-Action model with LangForce physics-grounded training methodology.

MolmoAct2-LIBERO: An Open Vision-Language-Action Model for Robotics

Shaoqing Tan — Sun, 17 May 2026 14:24:27 -0700

Vision-Language-Action (VLA) model fine-tuned on the merged LIBERO robotics dataset (1,693 episodes, 273k+ frames) achieving 98.25% success rate on manipulation tasks. Released with both checkpoint and dataset for VLA finetuning.

SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Diffusion Transformers

Shaoqing Tan — Sun, 17 May 2026 14:12:10 -0700

A 2.6B-parameter open-source world model that generates coherent 720p, minute-long videos with precise 6-DoF camera control on a single GPU using a Hybrid Linear Diffusion Transformer + Gated DeltaNet for long-context efficiency. Targets controllable physics simulation.

WildClawBench: A Real-World, Long-Horizon Benchmark for AI Agents

Shaoqing Tan — Sun, 17 May 2026 05:24:48 -0700

New benchmark and dataset for robotic manipulation in unconstrained 'wild' environments. Includes standardized containers, leaderboards, and evaluation protocols for cross-embodiment policies.

MCP-Cosmos: Bring Your Own World Model

Shaoqing Tan — Sun, 17 May 2026 05:15:11 -0700

Introduces a latent-space world model framework that lets agents simulate state transitions and iteratively refine plans before real-world execution. Evaluated on 20+ MCP-Bench tasks with measurable gains in tool-use success.

OpenAI o1: Teaching LLMs to Think Slow and Deep

Shaoqing Tan — Sat, 16 May 2026 18:41:07 -0700

Details OpenAI's reasoning-focused o1 model and its 'long thought' approach using test-time compute scaling. Explores how extended reasoning during inference can improve model performance on complex tasks.

The Llama 3 Herd of Models

Shaoqing Tan — Sat, 16 May 2026 18:32:22 -0700

Comprehensive technical report on the Llama 3 family, covering architecture, training at scale, multimodal extensions, and real-world impact. Details the development of Meta's flagship open-source language model series.

LATENT: Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data

Shaoqing Tan — Sat, 16 May 2026 18:03:21 -0700

Introduces a three-stage pipeline that extracts a latent action space from low-quality human tennis demonstrations, then trains a high-level policy in simulation via reinforcement learning. Enables dynamic whole-body humanoid tennis play with back-and-forth volleys at human level.

AnyFlow: Any-Step Video Diffusion for Predictive World Modeling

Shaoqing Tan — Thu, 14 May 2026 16:13:25 -0700

First any-step video diffusion framework using flow maps, allowing a single model to adapt to arbitrary inference budgets for scalable high-quality video generation relevant to predictive world modeling.

# Robotics: The Endgame

Shaoqing Tan — Thu, 14 May 2026 16:02:02 -0700

Technical roadmap mirroring LLM scaling: critiques VLAs, advocates video world models as second pretraining phase, introduces World Action Models (WAM), manipulation data flywheels, EgoScale with new Dexterity Scaling Law, and DreamDojo end-to-end neural physics engine for sim RL.

Claw-Eval: Toward Trustworthy and Transparent Evaluation of Autonomous Agents

Shaoqing Tan — Wed, 08 Apr 2026 07:19:18 -0700

Benchmark with 2,159 rubric items across 300 tasks using trajectory-aware grading and 3-trial Pass^3 scoring to mitigate luck. Evaluates agent reliability in real-world robotics settings.

LIBERO-Para: Paraphrase Robustness in Robotic Manipulation

Shaoqing Tan — Wed, 08 Apr 2026 07:18:01 -0700

Reveals paraphrase fragility in VLAs causing 22-52% success drops due to task misidentification. Introduces PRIDE metric weighting success by paraphrase difficulty on LIBERO benchmark manipulation tasks.

YOR: Your Own Mobile Manipulator for Generalizable Robotics

Shaoqing Tan — Tue, 07 Apr 2026 07:41:37 -0700

Low-cost mobile manipulator design and training strategies for broad generalization in real-world tasks.

EgoSim: Egocentric World Simulator for Embodied Interaction Generation

Shaoqing Tan — Tue, 07 Apr 2026 07:29:11 -0700

Closed-loop egocentric video simulator maintaining persistent 3D scene state for consistent interactions, enabling cross-embodiment transfer from human videos to robotic manipulation.

Accelerating Video World Models: From Generative Videos to Real-Time Simulators

Shaoqing Tan — Mon, 06 Apr 2026 22:17:58 -0700

Comprehensive survey taxonomizing efficient architectures/algorithms for video world models as simulators, targeting compute bottlenecks in embodied AI, autonomous driving, and games with techniques like short-window attention for real-time long-horizon prediction.

From Tokens to Thoughts: Continuous Latent Reasoning in Large Models and Robot Control

Shaoqing Tan — Mon, 06 Apr 2026 22:14:05 -0700

Curated collection of 100+ works surveying shift to continuous latent spaces in LLMs/VLMs/VLAs for improved reasoning over discrete tokens, with relevance to robotics action modeling.

CaP-X: Coding Agents for Physical eXecution

Shaoqing Tan — Mon, 06 Apr 2026 07:11:45 -0700

CaP-X is an open-source agentic robotics framework where LLMs/VLMs generate code to call perception and control APIs for execution across diverse simulated and real robots in CaP-Gym's 187 manipulation tasks. The framework includes CaP-Bench for evaluating frontier models and CaP-RL, which boosts a 7B model's success from 20% to 72% with minimal sim-to-real gap.

DoRA: Weight-Decomposed Low-Rank Adaptation

Shaoqing Tan — Sun, 05 Apr 2026 22:30:22 -0700

An upgrade over LoRA for parameter-efficient fine-tuning, enabling better performance in LLMs by decomposing weights into magnitude and direction components.

AI Model Collapse: What Happens When AI Trains on Its Own Outputs

Shaoqing Tan — Sun, 05 Apr 2026 22:15:47 -0700

Seminal work showing how training on AI-generated data leads to 'model collapse' in neural networks, with urgent implications for future scaling.

PhAIL: Benchmarking Vision-Language-Action Models on Real-World Bin-Picking

Shaoqing Tan — Sun, 05 Apr 2026 07:19:40 -0700

Real-world hardware evaluation of VLAs on blind bin-to-bin picking, achieving max 64 picks/hour across hundreds of runs, with full videos/data exposing gaps in production-scale robotic manipulation reliability.

Co-training Large Behavior Models: Data Modalities and Training Strategies for Robot Manipulation

Shaoqing Tan — Sat, 04 Apr 2026 22:42:13 -0700

Comprehensive evaluation of 89 policies showing optimal co-training practices mixing real robot data with sim/egocentric human videos to boost diversity and performance in large robotics foundation models.

HyDRA: Hybrid Memory for Dynamic Video World Models

Shaoqing Tan — Sat, 04 Apr 2026 22:31:30 -0700

Novel memory system preserving dynamic object identity and motion continuity across occlusions in video world models, addressing frozen/vanishing issues for improved predictive physics in embodied AI.

# WildWorld: Dynamic World Modeling with Actions and Explicit State

Shaoqing Tan — Sat, 04 Apr 2026 07:29:53 -0700

Massive dataset enabling dynamic world models with explicit states and actions, supporting predictive modeling for cross-embodiment robotic control.

Omni-WorldBench: Evaluating Interactive 4D World Models

Shaoqing Tan — Sat, 04 Apr 2026 07:18:25 -0700

New benchmark assessing world models on interaction tasks, pushing predictive physics and video modeling towards robotics applications with action-conditioned evaluation.

SIMART: From Static Meshes to Sim-Ready Articulated Models

Shaoqing Tan — Fri, 03 Apr 2026 22:37:59 -0700

Unified MLLM framework with Sparse 3D VQ-VAE (70% token reduction) for part-level mesh decomposition and kinematic chain prediction, enabling physics-based robotic simulation from monolithic assets.

EgoSim: An Egocentric World Simulator for Embodied Interaction

Shaoqing Tan — Fri, 03 Apr 2026 22:23:00 -0700

Closed-loop egocentric simulator persistently updating 3D scene state to generate spatially consistent interaction videos for continuous simulation, enabling cross-embodiment transfer from human videos to robotic manipulation tasks.

Digit's New Motor Cortex: Sim-to-Real RL for Whole-Body Control

Shaoqing Tan — Fri, 03 Apr 2026 07:13:57 -0700

AI-trained capabilities for new whole-body motions using mocap/teleop data and sim-to-real reinforcement learning, deployable overnight on hardware.

EgoNav: Diffusion-Based Humanoid Navigation from Human Egocentric Video

Shaoqing Tan — Thu, 02 Apr 2026 22:32:11 -0700

Diffusion-based humanoid navigation trained solely on 5 hours of human egocentric video data, enabling zero-shot deployment on Unitree G1 for complex behaviors like handling glass walls, crowds, and dynamic obstacles via 360° visual memory and hybrid trajectory sampling; upcoming release of dataset, models, and code.

CaP-X: A Code-as-Policy Framework for Robot Manipulation

Shaoqing Tan — Thu, 02 Apr 2026 22:19:27 -0700

Comprehensive open-source agentic robotics framework treating VLMs/LLMs as code-generating APIs for perception (SAM3, Molmo) and control (IK, grasping), with CaP-Gym benchmark of 187 diverse manipulation tasks (tabletop, bimanual, mobile; sim/real) and CaP-Bench evaluating 12 frontier models; demonstrates rapid RL gains (7B model from 20% to 72% success) with strong sim-to-real transfer.

Embodied Intelligence Breakthrough: Generalist AI’s GEN-1 Robots

Shaoqing Tan — Thu, 02 Apr 2026 12:58:30 -0700

We've created GEN-1, our latest milestone in scaling robot learning. We believe it to be the first general-purpose AI model that crosses a new performance threshold: mastery of simple physical tasks. It improves average success rates to 99% on tasks where previous models achieve 64%, completes tasks roughly 3x faster than state of the art, and requires only 1 hour of robot data for each of these results. GEN-1 unlocks commercial viability across a broad range of applications—and while it cannot solve all tasks today, it is a significant step towards our mission of creating generalist intelligence for the physical world.

CaP-X: LMs' First Physical Exam

Shaoqing Tan — Thu, 02 Apr 2026 12:43:57 -0700

A novel benchmark that evaluates language models on physical examination tasks, testing their ability to understand and perform clinical physical exam procedures in simulated environments. This work introduces a comprehensive evaluation framework for AI systems in medical/clinical settings.

AI Model Collapse: The Danger of Training on AI-Generated Data

Shaoqing Tan — Tue, 31 Mar 2026 07:36:21 -0700

Demonstrated that LLMs trained recursively on AI-generated data suffer model collapse, a degenerative process where they lose grasp of true data distributions. Sparked critical debates on data provenance and the importance of preserving human-generated training data.

High-Level Automated Reasoning with Qwen2.5-7B

Shaoqing Tan — Tue, 31 Mar 2026 07:35:15 -0700

Qwen2.5-7B achieved 79.6% on MATH benchmark, surpassing GPT-4o, by employing atomic reasoning actions combined with Monte Carlo Tree Search. Demonstrated that strategic reasoning architectures can enable smaller models to outperform much larger ones.

Co-Training Large Behavior Models: Multimodal Data for Robot Manipulation

Shaoqing Tan — Mon, 30 Mar 2026 22:19:22 -0700

Explores data modalities and co-training strategies to enhance large behavior models (foundation models) for improved performance in robot manipulation tasks, supporting end-to-end learning and cross-embodiment generalization.

HyDRA: Hybrid Memory for Dynamic Video World Models

Shaoqing Tan — Sun, 29 Mar 2026 22:20:50 -0700

Memory architecture preserving identity and motion continuity for out-of-view dynamic subjects, addressing frozen/vanishing issues in video world models.

DexWM: Leveraging Human Videos for Dexterous Robot World Models

Shaoqing Tan — Sun, 29 Mar 2026 22:18:43 -0700

Dataset of robot trajectories designed for training world models to learn dexterous hand-object interactions directly from human videos.

World Models in Robotics

Shaoqing Tan — Sun, 29 Mar 2026 07:14:29 -0700

Technical survey categorizing world models into action-conditioned, video-inverse dynamics, and joint world-action models (WAMs), discussing their generalization, video data leverage, and trends for closing the robotics data gap.

SIMART: Decomposing Monolithic Meshes into Sim-Ready Articulated Assets

Shaoqing Tan — Sat, 28 Mar 2026 07:21:21 -0700

Unified MLLM framework with Sparse 3D VQ-VAE that reduces tokens by 70% for efficient part-level decomposition and kinematic prediction in physics-based robotic simulations.

LeWorldModel: A Stable JEPA World Model from Pixels

Shaoqing Tan — Fri, 27 Mar 2026 22:16:24 -0700

Stable end-to-end JEPA world model trained directly from pixels using simple MSE prediction loss and SIGReg anti-collapse regularization, enabling efficient latent planning under 1 second on 15M params with emergent spatial structure outperforming prior methods.

World Models for Robots: The Next Big Leap?

Shaoqing Tan — Fri, 27 Mar 2026 07:39:49 -0700

Technical overview defining world models in robotics, their potential to solve diverse problems via video prediction, and key enablers like scale.

Harnessing Long-Running AI in Embodied Systems

Shaoqing Tan — Fri, 27 Mar 2026 00:11:43 -0700

As AI moves from quick Q&A to marathon tasks, designers grapple with continuity. This episode explores how Anthropics harness design principles translate to embodied AI - robots that need to maintain context across long-running missions.

HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations

Shaoqing Tan — Wed, 25 Mar 2026 22:29:11 -0700

Whole-Body Mobile Manipulation Interface (HoMMI) that learns bimanual and whole-body manipulation, long-horizon navigation, and active perception directly from egocentric human demonstrations without teleoperation.

TurboQuant: Redefining AI Efficiency with Extreme Compression

Shaoqing Tan — Wed, 25 Mar 2026 17:52:48 -0700

This episode explores TurboQuant, a revolutionary set of quantization algorithms from Google Research that redefines AI efficiency through extreme compression.

We dive deep into how TurboQuant addresses one of AI's most pressing challenges: the memory bottleneck created by high-dimensional vectors in key-value caches. The research introduces theoretically grounded quantization methods that enable massive compression for large language models and vector search engines without sacrificing performance.

Key topics covered:

The theoretical foundations of TurboQuant's quantization algorithms
How extreme compression works for LLMs and vector search engines
Impact on high-dimensional vectors and key-value cache memory bottlenecks
Performance metrics and comparisons with existing methods
Practical implications for AI deployment and efficiency

Links:
Paper: https://arxiv.org/pdf/2504.19874
Blog: https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/

DexWM: Learning Dexterous Object Manipulation from Human Videos

Shaoqing Tan — Wed, 25 Mar 2026 07:19:46 -0700

Dataset of robot trajectories designed for training world models that learn dexterous hand-object interactions from human videos, released on Hugging Face.

FlashAttention-3: Fast & Accurate Attention with Asynchrony & Low-Precision

Shaoqing Tan — Tue, 24 Mar 2026 22:54:40 -0700

Major efficiency leap for Transformer attention mechanisms, enabling faster training/inference on long sequences with low-precision compute.

When AI Trains on Its Own Output: The Model Collapse Problem

Shaoqing Tan — Tue, 24 Mar 2026 22:39:06 -0700

Warns of "model collapse" in LLMs trained on synthetic data from prior models, urging preservation of human-generated data. One of 2024's most influential papers.

MolmoBot: A Vision-Language Model for Zero-Shot Robot Manipulation

Shaoqing Tan — Tue, 24 Mar 2026 07:22:35 -0700

Vision-language model (VLM) for zero-shot robot manipulation, trained entirely in simulation without real-world data; achieves 79.2% success rate on real-world tabletop tasks, outperforming π₀.₅ baseline at 39.2%.

LeWorldModel: Stable End-to-End JEPA from Pixels

Shaoqing Tan — Tue, 24 Mar 2026 01:12:52 -0700

A stable end-to-end Joint Embedding Predictive Architecture (JEPA) trained directly from pixels that enables robust world modeling for embodied AI systems.

EgoVerse: An Egocentric Data Ecosystem for Scaling Robot Learning

Shaoqing Tan — Mon, 23 Mar 2026 22:18:26 -0700

Ecosystem with over 1300 hours of egocentric human video data spanning 240 scenes and 2000+ tasks, designed for scalable robot policy training via behavior cloning; includes cloud infrastructure, data viewer, and human-to-robot transfer algorithms to enable cross-embodiment learning without teleoperation.

HSImul3R: Physics-Driven Reconstruction of Human–Scene Interactions

Shaoqing Tan — Mon, 23 Mar 2026 22:15:11 -0700

Physics-in-the-loop bi-directional optimization pipeline reconstructing stable, simulation-ready 3D human-scene interactions from casual videos, deployable directly to humanoid robots for world modeling and manipulation.

MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation

Shaoqing Tan — Mon, 23 Mar 2026 11:48:08 -0700

Open-source suite of large-scale simulation environments and benchmarks designed for advancing end-to-end learning in robot navigation and manipulation across multiple embodiments.

DreamZero: World Action Models Are Zero-Shot Policies

Shaoqing Tan — Mon, 23 Mar 2026 11:36:32 -0700

Introduces World Action Models (WAMs), a family of 14B-parameter autoregressive diffusion models that jointly predict video and robotic actions to enable zero-shot generalization across manipulation tasks, outperforming fine-tuned Vision-Language-Action models on benchmarks like MolmoSpaces and RoboArena.

Kinema4D: A 4D Generative Simulator for Embodied AI

Shaoqing Tan — Sun, 22 Mar 2026 19:16:00 -0700

An action-conditioned 4D generative robotic simulator that disentangles precise kinematic control from environmental dynamics, facilitating physically-plausible simulations of complex robot-world interactions for training and world modeling.

VEGA-3D: Teaching multimodal LLMs spatial reasoning through video generation

Shaoqing Tan — Sun, 22 Mar 2026 19:02:22 -0700

A plug-and-play framework extracts implicit 3D priors from video diffusion models to enhance multimodal LLMs with spatial reasoning capabilities, enabling improved geometric scene understanding and embodied decision-making without explicit 3D supervision.