In the previous article, World Models 101, we answered three questions: what are world models, what types exist, and what can they do? At that time, we borrowed Li Feifei's functional tripartite approach—renderer, simulator, and planner—but also pointed out its limitations: it only specifies outputs without addressing how world models internally represent and operate on the world.

This article fills in that gap by adopting a more fundamental perspective—representation-centric—classifying existing world models based on how they represent and propagate the 'world state.' Along this axis, there are three main technical architectures:

Observation-Level Models: Directly predict on pixels, voxels, or video tokens;

Latent Space Models: Learn compact predictive states;

3D-Augmented and Object-Centric Models: Introduce explicit spatial, geometric, or entity-level structures.

Each route has its trade-offs. Above them, a broader trend is emerging: generative, understanding, interaction, and multimodal foundational capabilities are being integrated into a shared unified architecture. Let's explore each in detail.

Observation-Level Models: Direct Pixel Prediction, Where Scale Is Power

Observation-level generative world models interpret 'world modeling' as the synthesis of high-dimensional observations—directly predicting future pixels, voxels, or video tokens. This primarily refers to video generation models in the current digital world: OpenAI's Sora, ByteDance's Seedance, Google's Genie, and Wayve's GAIA for autonomous driving. Strictly speaking, they are not decision-centric world models but represent an 'observation-level' path for learning physical priors from large-scale videos.

Their core strength lies in scale. Internet videos provide a vast weakly supervised corpus for learning appearance, motion, object interactions, and scene dynamics—directly fulfilling the 'data determines the upper limit' logic from 101. These models typically achieve high visual fidelity and broad coverage of real-world phenomena.

However, they have two critical flaws:

Visual plausibility ≠ physical correctness. Pixel-space models may produce 'locally credible but globally inconsistent' inferences—especially over longer timescales, where errors in object permanence, contact dynamics, causality, and scene consistency accumulate.

They are computationally expensive, as predictions occur in high-dimensional observation space, requiring every frame to be 'drawn' at high cost.

These flaws motivate the second route.

Latent Space Models: No Pixels, Just 'Decision-Relevant' States

The core shift in latent space world models is from modeling visual appearance to modeling the evolution of task-relevant states—no longer performing reductive modeling of all pixels like pixel-space approaches.

Early representatives include the PlaNet and Dreamer series: they encode high-dimensional observations into compact state representations using latent dynamics models (e.g., Recurrent State-Space Models, RSSM) and perform 'imagination-based planning.' Since inference operates on compact latent states rather than full-resolution pixels, these methods naturally scale to long-term predictions, better handle partial observability, and allow agents to evaluate more hypothetical futures within fixed computational budgets.

Beyond efficiency, latent modeling alters the optimization objective itself. Instead of expending compute on reconstructing every RGB detail, the model focuses on factors that change slowly and are closely tied to decision-making—the very elements that determine 'future controllability': geometry, object relationships, affordances, contact dynamics, and temporal evolution. A series of progressive evidence supports this path:

TD-MPC/TD-MPC2 demonstrate that planning can occur directly in task-oriented latent spaces without a full pixel decoder;

Contrastive and reconstruction-free world models (e.g., C-SWM) show that bypassing pixel reconstruction reduces overfitting to high-frequency appearance distractions like texture and lighting, improving robustness;

JEPA-style predictive representation learning (I-JEPA, V-JEPA) further question the necessity of 'complete visual reconstruction'—a technical realization of the LeCun school's stance from 101;

Embodied learning systems like LDA-1B and Motus prove that effective policies can be learned from compact task-oriented latent representations without photorealistic reconstruction of the entire visual scene.

In short, this route's advantage: it retains low-frequency, causally relevant structures while suppressing superficial visual details, making it faster and more aligned with control logic.

But it comes at a cost—the risk of over-abstraction: the model may ignore visually subtle yet decision-critical information, such as robot-object contact, small obstacles, minute changes in object pose, or deformation. Thus, the core challenge is not 'how aggressively to compress' but learning a latent model that filters out appearance noise while preserving controllability factors. This also shifts evaluation: latent world models should be judged not by reconstruction or prediction error alone but by whether their inferences retain information necessary for subsequent planning and control.

3D-Augmented and Object-Centric Models: Equipping the World with Geometry and Entities

The third route addresses a blind spot shared by the first two: internal states are hard to interpret and may not align with the scene's true 3D layout. For embodied AI, this is a clear shortcoming —models must support reasoning about free space, object positions, occlusion, viewpoint changes, object permanence, and physical plausibility. Recent research introduces three types of structured state spaces:

First, occupancy and bird's-eye view (BEV) representations. Compared to pixels, they more directly capture spatial structure for reasoning about free space, obstacles, semantic regions, and future scene evolution. Examples include OccWorld, OccSora, Drive-OccWorld, and BEVWorld in autonomous driving, as well as RoboOccWorld, which extends this to indoor embodied prediction. Their key advantage: predictions are not just visually plausible but support spatial queries.

Second, continuous 3D scene representations. NeRF and 3D Gaussian Splatting provide geometry-aware state representations, supporting novel view rendering and stronger viewpoint consistency. Systems like Marble, RenderWorld, GaussianWorld, and GWM extend this to persistent 3D world generation, driving modeling, and robotic manipulation. Compared to pure video generation, they better maintain spatial consistency across viewpoints and time—though dynamic scenes and long-term interactions remain challenging.

Third, object-centric representations. They address a complementary issue: even with dense 3D states, key entities may not be separated. Methods like OP3, C-SWM, Slot Attention, SAVi, and SlotFormer learn object-level/slot-level representations and dynamics, supporting compositional prediction and reasoning. Robotic systems like FOCUS further connect object-level abstraction with manipulation and instruction-conditioned prediction. The value of this abstraction: many embodied tasks are defined around persistent entities and their relationships, not global image patterns.

The most promising direction is combination, not competition: occupancy for free space and collision reasoning, NeRF/3D Gaussian for viewpoint-consistent rendering, and object slots for compositional reasoning. Hybrid embodied systems like EnerVerse and EnerVerse-AC already move in this direction—linking multi-view prediction, memory, 4D scene representation, action conditioning, and embodied future generation.

A word of caution: introducing 3D structures does not automatically solve world modeling. Models may accurately reconstruct space yet fail to capture affordances, causality, and task semantics; object-centric models are interpretable but struggle with reliable object discovery in cluttered and occluded scenes; NeRF/3D Gaussian can render fine views, but dynamic scenes, long-term prediction, and closed-loop interaction remain full of challenges. The key question is never 'does it look more 3D?' but whether the state effectively and coherently represents the physical world—geometrically consistent, temporally stable, object-aware, physically plausible, and accessible to downstream reasoning, planning, and control modules.

Trend Toward Unification I: Integration of Generation and Understanding

After examining the three routes, let's turn to the fusion happening above them. A growing consensus in research: future world models should not just be generators but models that integrate generation and understanding—capable of synthesizing future observations while also comprehending objects, spatial relationships, temporal dynamics, causality, and physical plausibility.

This direction is embodied in systems like LWM, WorldGPT, the HERMES series, GaussianDWM, and UniDrive-WM: all attempt to link scene/video understanding with future state generation within a unified framework. Adjacent multimodal foundation models—Chameleon, Janus/Janus-Pro, and the Emu series—echo this trend: understanding and generation can be unified within shared token or autoregressive architectures. World modeling is moving beyond 'visually compelling synthesis' toward architectures that can co-represent, reason about, and generate dynamic worlds.

Trend Toward Unification II: World-Action Models, Linking 'Imagination' to 'Action'

But the physical world is not just observed—it is acted upon. From the perspective of partially observable decision-making, an embodied agent only observes partial aspects of the underlying world state, and its actions intervene in state transitions, shaping future observations. Thus, integrating generation and understanding is insufficient—world models must also connect imagined futures to executable actions.

This has given rise to World-Action Models (WAMs): they integrate world modeling and action/policy modeling into a single generative framework, treating dynamic prediction and policy learning not as independent modules but as coupled relationships learned directly from 'video-action' data. For example:

DreamZero shows that joint modeling of video and actions can directly convert world models into zero-shot policies;

LingBot-VA unifies visual dynamics prediction and action inference within an autoregressive video-action framework;

τ0-WM integrates action generation, video prediction, and future state evaluation for robotic manipulation;

UWM, Cosmos-Policy, GigaWorld-Policy, Fast-WAM, and Flash-WAM explore various combinations, such as joint diffusion, action-centric prediction, and fast inference without explicit video imagination at test time.

These works collectively mark a shift: from passive world simulation to models that can both imagine physical futures and generate actions based on them. This aligns with the judgment in 101—WAMs are simultaneously simulators and planners because future state prediction is inherently part of action generation.

Trend Toward Unification III: Full Modality, Toward a Universal Backend for Physical AI

The third layer of unification occurs along the modality dimension: world models are expanding from single or few modalities to full-modal physical modeling—integrating language, images, videos, audio, actions, 3D structures, and sensor data into a shared representation of the physical world. Recall the three characteristics of world models from 101: 'full modality' ranks first, and here is its realization.

NVIDIA Cosmos 3 and Seedance 2.0 are typical examples of this ambitious attempt: they explicitly aim to unify language, images, videos, audio, and actions into a hybrid Transformer architecture, transcending vision-centric generation to move toward a universal physical AI backend. Another group of systems, while not constituting complete full-modal world models, provide important building blocks: Seedance 2.0 explores generating unified multimodal audio-video content from text, image, audio, and video inputs; Wan/Wan2.1 offers a scalable open video generation backbone; Tencent's HunyuanVideo and HunyuanWorld extend video generation to explorable, interactive 3D world construction; Kling-Omni integrates multimodal instruction following, generation, editing, and reasoning; Qwen-Omni demonstrates how language, vision, audio, and video can be collaboration processed within a unified full-modal foundation model.

Conclusion: Three Routes, One Destination

To compress this article into three sentences:

The three technical routes essentially represent three representation choices—observation-level models bet on 'scale for physics,' latent space models bet on 'compression for control,' and 3D/object-centric models bet on 'structure for understanding.' Their respective weaknesses are each other's strengths; the future is more likely to be a jigsaw combination than a single choice.

Evaluation criteria are shifting—not about how visually similar or low the reconstruction error is, but whether the model's states and inferences retain information necessary for downstream reasoning, planning, and control.

Unification is the common endpoint—the integration of generation and understanding, world-action coupling (WAM), and full modality integration. The fundamental driving force behind these three trends is the same: physical AI requires a model capable of understanding the present, imagining possible futures, interacting with the world, and integrating multimodal physical signals.

*Reproduction and excerpting are strictly prohibited without permission-