On the final day of GTC 2026, Sanja Fidler, NVIDIA's VP of AI Research and a professor at the University of Toronto, delivered a 40-minute autonomous driving session. The first 25 minutes featured a presentation, followed by a 15-minute live demo by two Chinese researchers: NVIDIA's driving policy model, Alpamayo, first drove closed-loop in a city entirely generated in real-time by the world model. Then, the steering wheel was handed over to a human, and two audience members were invited onstage to experience the product.

The session primarily promoted AlpaDreams, a world model built on the Cosmos foundational model. Trained with 20 million hours of video and 90 quadrillion tokens, then fine-tuned with 20,000 hours of autonomous driving data, it features just 2 billion parameters. Denoising steps were reduced from 50–100 to just 2, enabling 12 frames at 54 FPS (220ms per frame) on a single GPU.

Key numbers to note: NVIDIA's autonomous driving product team runs 2 million tests daily in reconstructed scenarios and trains thousands of policy versions annually.

Beyond the numbers, NVIDIA's approach to "simulation" stands out. Fidler's words: "The model deployed in the vehicle is the star, but every star has a massive crew behind it." She referred not to the in-vehicle model but to the "crew": the methods, speed, and scale used to select which policy version gets deployed. While China's industry has framed "world models" as part of in-vehicle capabilities over the past two years, NVIDIA positioned it as development infrastructure.

This analysis follows the presentation's structure, clarifying information from the slides and speeches before offering insights and critiques from the perspective of China's autonomous driving industry. We examine NVIDIA's simulation world model and the methodology behind its autonomous driving algorithm training and simulation.

S82446 *Advancing Autonomous Driving with World Models*, Speaker: Sanja Fidler, NVIDIA & University of Toronto I. Opening: AI's Four Waves and the "First Robots to Arrive" From AlexNet to Physical AI

2012 AlexNet → Perceptual AI → Generative AI → Agent AI → Physical AI; NVIDIA places autonomous driving and general-purpose robots at the peak of the curve.

Fidler opened with an AI evolution curve. AI has existed since the 1980s or earlier, but 2012 marked the moment when "a new era was upon us." She recounted her experience as a postdoc at the University of Toronto, witnessing Hinton and students develop AlexNet: "Alex joined group meetings every three weeks to show which records he'd broken. You could feel something was about to change." Then came ChatGPT less than five years ago, making AI accessible to the public for the first time. At this GTC, agent AI took center stage, already creating value for enterprises.

Next is physical AI—robots that will eventually surround us. She defined autonomous vehicles as "likely the first robots to arrive, extremely difficult to build but probably the first to deliver massive value."

From AV 1.0 to AV 3.0

AV 1.0 Perceptual AI → AV 1.5 Perception + Model-Based Planning → AV 2.0 Generative AI / End-to-End → AV 3.0 Agent / Physical AI (Reasoning-Based VLA)

She divided autonomous driving's technological evolution into four stages: Early AI handled perception only, understanding the surroundings. Machine learning then entered control and planning. Next came generative AI's end-to-end systems, directly mapping sensor data to actions—"entirely data-trained, becoming more data-centric." She called the upcoming generation AV 3.0: Foundational models, large language models, and world models jointly drive the robot's brain to solve edge cases.

Vehicle Review

First, NVIDIA's phases align with China's industry narrative but differ in focus. Domestic automakers and suppliers have widely adopted "VLA," "world models," and "end-to-end 2.0" as names for next-gen solutions since 2025—essentially the same as NVIDIA's AV 3.0. The difference lies in domestic emphasis on in-vehicle model reasoning, while Fidler barely discussed the in-vehicle policy Alpamayo, focusing entirely on "how to evaluate and train it." This focus difference persisted throughout.

Second, positioning autonomous driving as the "first robots to arrive" benefits and warns China's intelligent driving industry. NVIDIA treats autonomous driving as the lead market for physical AI, meaning general-purpose robotics tools like Cosmos, NuRec, and simulation frameworks will mature in automotive before expanding to humanoid and industrial robots. Domestic intelligent driving teams can transfer their capabilities to robotics, and vice versa—robotics companies will enter the intelligent driving supply chain.

II. Stars and Crew: Why Simulation is Core to AV Development

The VLA policy model receives sensor and text tokens, outputting reasoning trajectories and action tokens; the world simulator provides a closed loop for training and evaluation.

This is the familiar diagram: The driving policy acts as the robot's brain, receiving sensor data and commands, outputting reasoning, then actions—typically trajectories executed by control modules. "This is the model everyone gets excited about. It's the star."

She then explained a development workflow challenge. Training many policy versions is necessary—"I think NVIDIA trains thousands annually." Testing each version by deploying it in a vehicle, observing performance, and iterating is impractical: "There's no time for that." The solution is a simulator akin to a game engine, providing instant feedback. But this requires the simulator to be hyper-realistic and diverse—"basically indistinguishable from the real world."

The result: NVIDIA's policy evolved from its first end-to-end model to "driving exceptionally well" by year-end, as she confirmed after a test drive with NVIDIA's automotive head, Xinzhou Wu, last Saturday.

Vehicle Review

First, "thousands of policy versions annually" defines simulation demand. Domestic discussions often start from the supply side—reconstruction accuracy, rendering speed, scenario library scale. NVIDIA reasons from the demand side: Thousands of candidate models annually require evaluation, necessitating 2 million daily closed-loop tests. Domestic automakers can ask themselves: How many versions do we iterate annually? What signals determine deployment? How quickly are those signals obtained?

Second, "reliable simulation training" remains a challenge. Leading domestic automakers have tens to hundreds of thousands of vehicles relaying data, with "shadow mode" making real-world signals relatively inexpensive. However, shadow mode provides open-loop signals—differences between model outputs and human drivers, not how the world reacts to the model's actions. NVIDIA's focus is closed-loop: The policy acts, the world responds, and feedback loops back. Our Tesla analysis (*Full Interpretation of Tesla's World Model Patents: From "Seeing" to "Imagining," the Evolutionary Singularity of Physical AI*) also highlighted Tesla's closed-loop training. Fleet scale can replace data collection but not closed-loop evaluation.

III. Simulation 1.0 to 2.0: From Game Engines to Neural Reconstruction The Limits of Manual Modeling

Simulation 1.0 (Graphics): Artists manually create all assets, ray tracing enables physically accurate simulation; Simulation 2.0 (Neural Reconstruction): Assets rebuilt from real driving data, neural primitives enable ray tracing, visually faithful replay of real-world conditions.

When Fidler joined NVIDIA seven years ago, evaluators relied on graphics-based game engines. The issue: All content required manual creation: "If I wanted to test a new intersection in San Francisco, it took a month or two for artists to build the environment." Such a pipeline couldn't scale globally.

In 2020, Neural Radiance Fields (NeRF) emerged, followed by Gaussian Splatting. "The day that paper came out, we said this technology's killer app would be autonomous driving simulation." The reason was straightforward: Autonomous driving generates hundreds of thousands of hours of real recorded data, now reconstructable into 3D by AI—creating replayable simulation environments for policies to experience new scenarios.

NuRec: Already in Production for a Year

Explicit Gaussian particle representation decouples static and dynamic elements; dynamic objects are controlled by off-the-shelf behavior models. Every NVIDIA mass-produced policy model undergoes closed-loop evaluation—"most versions fail this test." Hundreds of thousands of reconstructed scenarios, 2 million daily tests.

She briefly explained the reconstruction process: A mesh is generated as collision geometry for roads, guiding vehicles on where to drive and locating speed bumps. All dynamic assets are then reconstructed and repositioned using behavior models. The entire scene is decoupled into "highly manipulable" components.

This tech stack, Omniverse NuRec, represents "our largest technical deployment at NVIDIA": It has run in production for a year, continuously delivering value to autonomous driving teams. It reconstructed hundreds of thousands of scenarios and runs 2 million daily tests—"the tool for selecting which policy gets deployed." An unspoken slide note: Every trained NVIDIA mass-production policy model must pass closed-loop evaluation, with most versions failing.

What's Open This Week

Tools & Libraries: NCore temporal video data format (GitHub); Generative Models: Fixer (artifact removal), Asset Harvester (image-to-3D asset conversion) (Hugging Face); Data: 1,000 pre-reconstructed scenarios, 300,000 scenarios from the Physical AI dataset; Workflows: NuRec container (NGC), AlpaSim open-source simulation framework (GitHub).

NVIDIA has open-sourced parts of this stack and released new code/models at GTC, with "more to come in the next few months." She highlighted three categories: Data tools for efficient video processing and rapid reconstruction; generative models to enhance simulation—reconstruction typically works within meters of original trajectories, while generative models enable lane changes; and asset harvesting (released this week): "Every dynamic object, even seen only from the side, can be completed in all directions," including pedestrians, with code now available.

Alpamayo's original driving scene in San Francisco (left); four right panels: original trajectory reconstruction, scooter insertion, new trajectory with motorcycle, new trajectory with cone. The NuRec pipeline provides 10x scenario diversity for Alpamayo development.

She showed a video: The original recording (left) was fully reconstructed, with scooters, motorcycles, and cones "harvested" from other scenes inserted. "This lets you create many variants of the same scene." The slide title emphasized the pipeline's 10x scenario diversity for Alpamayo development.

A promotional video revealed additional details: NuRec supports robots in kitchens, offices, and warehouses, not just roads; 3D Gaussian scenes now enable physical interactions; when combined with Cosmos, NuRec can generate 3D simulation environments from text prompts—a new "text-to-3D environment" version releases this weekend.

Vehicle Review

First, "2 million daily closed-loop tests" serves as a benchmarkable operational metric. Domestic automakers often cite "X billion simulated kilometers" or "Y ten-thousand scenario libraries" when discussing simulation. NVIDIA uses daily closed-loop test counts and policy rejection rates—metrics closer to R&D assembly line (R&D pipeline) throughput. While not directly convertible, the latter better indicates whether simulation is embedded in the release process rather than a parallel demo project.

Second, the statement 'most versions fail closed-loop evaluation' warrants serious reflection by domestic teams. Its implication is that within NVIDIA, simulation does not serve as a bonus for models but as a death sentence. Many domestic teams still treat simulation as a form of 'regression testing'—confirming that new versions do not degrade performance in known scenarios, rather than proactively using it to eliminate candidate versions. The upper limit of simulation's value depends on whether an organization is willing to grant it veto power.

Third, asset collection and open-source strategies are a double-edged sword for domestic simulation vendors. NVIDIA has open-sourced the NCore data format, Asset Harvester, and AlpaSim, releasing 1,000 pre-reconstructed scenes. On one hand, this devalues 'reconstruction-based simulation' as a standalone product—domestic startups in reconstruction-based simulation now face a free baseline. On the other hand, NVIDIA's reconstruction toolchain defaults to the Omniverse and its own GPU ecosystem, meaning domestic automakers adopting it become tied to NVIDIA for simulation as well. While discussions on domestic chip substitutes for vehicle-end applications have persisted for years, dependency on simulation is rising instead.

IV. Simulation 3.0: What Generative World Models Cannot Achieve

Simulation 3.0: Rendering and simulation are entirely generative, freeing content creation from data lake limitations and enabling simulation of complex visual and behavioral effects (joint movements, adverse weather, etc.).

NuRec is already delivering value, 'but as researchers, this is not where we stop.' There are many things reconstruction cannot achieve: 'A mattress strapped to a car roof is already difficult; any weather, like adding snow or slush on the road, is extremely challenging.' She then displayed a snowy scene: 'This is AI-generated.'

Generative simulation is built on models that 'learn how to simulate.' These models do not learn fragment-by-fragment but instead learn how the world operates from massive datasets—'given past frames, it can generate subsequent frames based on data priors, regardless of what the policy intends to do next.' Her assessment of the ceiling: 'Given the success of large language models, it's hard to imagine where this could fail.'

Cosmos-Drive-2's text-conditioned video generation, Gen3C's large-offset novel view synthesis, and ChronoEdit's scene editing (removing road objects, adding a group of toddlers to the sidewalk).

She showcased several results and Specially explained two sentences (deliberately explained two points): No one was harmed in these scenes—'all generated'—and their unrealistic appearance stems from 'we lack that type of data.' In driving simulation, there is no need to simulate collisions: 'Once a collision occurs, the scenario stops—the game ends. Robots should not behave that way.' The value of generative models lies in three areas: achieving novel views beyond reconstruction's capabilities while deviating significantly from the original vehicle trajectory; enabling effortless editing ('write a prompt, add a pedestrian, swap the vehicle, erase everything, add children to the road'); and simulating weather and long-tail scenarios that reconstruction cannot handle.

Vehicle Review

First, NVIDIA's current stance of 'reconstruction-first, generation-augmented' differs from domestic narratives. Many domestic launches position 'world model-generated training data' as the main storyline, but NVIDIA clarified: what runs 2 million times daily on production lines is reconstruction. Generative models, now real-time, primarily serve to 'enhance' reconstruction—adding distant perspectives, weather, and long-tail objects. Generation has a high ceiling; reconstruction offers stability. If domestic teams must choose one path, NVIDIA's answer is to prioritize reconstruction as infrastructure.

Second, 'not simulating collisions' reflects an evaluation philosophy worthy of domestic discussion. Domestic discussions on world models often emphasize 'generating extreme corner cases to teach models accident handling.' Fidler argues that collisions signify termination—no need to generate post-collision frames since the policy has already failed. This treats simulation as a referee, not a textbook. Both approaches are valid, but mixing them leads to vastly different requirements for 'how real the world model needs to be.'

V. Cosmos: The Foundation of Everything

Cosmos Predict 2.5 (world generation, fully customizable), Cosmos Transfer 2.5 (photorealistic data augmentation), Cosmos Reason 2 (physical AI reasoning visual-language model); over 6 million downloads on Hugging Face, available on Azure/AWS/GCP.

Everything showcased next is built on Cosmos. First unveiled at CES a year ago, Cosmos has iterated to version 2.5 with three models: Predict, which generates seconds of video from a frame and a prompt; Transfer, which accepts detailed conditions like segmentation maps or HD maps with lane lines and object bounding boxes ('at this point, Cosmos functions more like an advanced renderer'); and Reason, which ingests video for reasoning and analysis, then generates descriptions. Today's focus is on the first two.

Pre-training data distribution: 11% driving, 16% hand-object manipulation, 16% human motion, 16% spatial perception and navigation, 7% first-person views, 20% natural dynamics, 8% camera dynamics, 4% synthetic; 20 million hours of video, 9 quadrillion input tokens, 2,000+ hours of training; post-training: 20,000 hours of multi-sensor NVIDIA autonomous driving data plus purchased datasets, with 1,700 hours of multi-camera data publicly released.

Cosmos was trained on 20 million hours of video. 'NVIDIA covered this cost—an unbelievable cost—not just for data but also for the infrastructure to train these models, all open-sourced.' Post-training follows application-specific adjustments: for autonomous driving, 'training on tens of thousands of hours from Xinzhou Wu's team suddenly enabled simulation of driving videos.' Slides stated 20,000 hours of NVIDIA's proprietary multi-sensor data plus purchased datasets, with ~1,700 hours of multi-camera data released (the speech mentioned 'nearly 2,000 hours').

Vehicle Review

First, the data distribution chart is the most critical slide for domestic automakers. In Cosmos pre-training, driving data accounts for only 11%, with the rest comprising general physical videos like hand manipulations, human motion, navigation, and natural dynamics. NVIDIA's logic: a world model's 'physical common sense' derives from general videos, with driving as a downstream task for post-training. Domestic automakers possess massive driving data from hundreds of thousands of vehicles but lack general physical video accumulation. This means self-developed world models in China would lack physical priors if trained from scratch; using open-source foundations means accepting dependencies on others.

Second, '20,000 hours of post-training suffices' is good news for China. The required scale of driving data for post-training is tens of thousands of hours, not millions of kilometers—a volume any leading domestic automaker can provide. The true barriers lie in foundations and compute: pre-training with 20 million video hours and 9 quadrillion tokens is a scale few in China will replicate. Thus, the practical path for China is likely 'open-source foundation + proprietary driving data post-training'—exactly what NVIDIA aims to achieve by open-sourcing Cosmos.

Third, compliance boundaries matter. While Cosmos is open-source and downloadable via Hugging Face, the 1,700 hours of multi-camera data NVIDIA released were collected in the U.S. Domestic automakers using overseas data for post-training on Chinese vehicles—or vice versa—risk violating data export and geolocation regulations. Technically 'plug-and-play,' but procedurally requires navigating multiple regulatory hurdles.

VI. AlpaDreams: From Minutes to Real-Time Interactivity—Jensen Huang as the First Driver

A causal autoregressive video model that interacts with policies to generate minutes-long driving sequences, achieving up to 54 FPS real-time inference on B300 (previously requiring minutes per generation), now integrated into closed-loop systems; the right image shows Jensen Huang test-driving in his office.

GTC's announced AlpaDreams marks the model's first real-time capability. Last year, generating a five-second video took minutes; now it's real-time, interactive, and user-in-the-loop. 'Our first driver was Jensen'—at a research meeting a month ago, the team brought equipment to his office, and he drove first. 'He complained it wasn't like his Ferrari but gave the green light overall, letting us present at GTC.'

March 2026 AlpaDreams: causal autoregressive, policy-in-the-loop generating minutes-long sequences, 2B model quality, up to 54 FPS on B300, closed-loop integration; March 2025 Cosmos-Drive-Dreams: bidirectional video diffusion, generating 5-second clips, 7B model, minutes of latency, open-loop only.

Compared to a year ago: the previous model generated five-second videos from text prompts, requiring stitching for longer sequences, with quality degradation after steps—max 20 seconds of high quality. Each segment took minutes: 'you could see the future, but it was entirely impractical'—and only open-loop, without policy interaction. AlpaDreams became autoregressive, rapidly responding to each policy or user action in real-time.

How the Closed Loop Connects

Policy model (Alpamayo 1) or human user outputs actions → simulation runtime AlpaSim updates abstract state (including traffic participants) → world model AlpaDreams renders synthetic camera frames → back to policy.

The architecture: the policy on the left outputs actions; these enter simulation runtime AlpaSim, which orchestrates APIs like a traffic model to move dynamic object bounding boxes; it then calls world model AlpaDreams for rendering; rendered frames return to the policy, looping endlessly.

Inputs: next abstract world state (HD map and object bounding boxes), text prompts, historical frame cache; autoregressive causal video generation; outputs: next sensor frame.

The model accepts three types of conditions: text prompts specifying scene events; the first frame and subsequent historical frame cache ('meaning you can directly use a real-world frame as the starting point'); and maps—where lane lines and dynamic objects are. Map conditions serve two purposes: truly controlling scene appearance and improving quality—'driving is too complex; without conditions, it can only generate boring scenes.'

Vehicle Review

First, using 'HD map + object bounding boxes' as conditions marks a key difference between AlpaDreams and most domestic world models. Domestic discussions often emphasize 'end-to-end generation without structured inputs,' but NVIDIA prioritizes structured abstract states as primary conditions for simplicity: controllability and quality. This creates a chain reaction: AlpaSim's traffic model determines other vehicles' movements; AlpaDreams only renders them—behavior and appearance are split into two layers, each replaceable. Domestic teams combining behavior and rendering in one model save on intermediate representations but sacrifice scene controllability and evaluation reproducibility.

Second, 'user-in-the-loop' is not a gimmick but a new evaluation tool. Letting engineers drive in generated worlds reveals physical consistency within minutes—whether the car body follows turns, whether wipers activate in rain, whether driving onto sidewalks causes clipping. Automated metrics struggle to capture these. Most domestic teams treat world model outputs as training data or offline metrics, rarely making them 'drivable.'

VII. Training Methodology: Four-Stage Post-Training

Cosmos foundation model → Cosmos-Drive post-training → causal training → self-consistency and distillation → AlpaDreams model.

Fidler stated, 'We wanted extreme speed,' so they started with a 2 billion-parameter Cosmos model and applied staged post-training to achieve both real-time performance and interactivity. She offered technical details in sequence below.

Pre-train the visual foundation model Cosmos-Predict2.5-2B with the Real Driving Scenarios (RDS) dataset; incorporate HD map conditions for precise layout control; extend to multi-view generation.

Post-train Cosmos using 20,000 hours of autonomous driving data. In the final stages of training, new conditions—lane lines (HD maps) and dynamic bounding boxes—are introduced. "Now the model has become controllable." Then, multi-view capability is added: robots use more than one camera. Here, the focus is on cameras, demonstrating a four-view model, with the capability to support up to seven cameras.

Causal attention, adapted for autoregressive generation, paired with KV caching for efficient inference; Causal DiT generates the next frame from historical clean frames and the current noisy frame.

The original model processed multiple frames for denoising, generating five seconds of output at once. This is now changed to a recurrent approach: generating a clean image from noise, then repeating the process. "This is diffusion-based, for those who are familiar." This is another round of post-training. "Training such a model takes a few days."

Bidirectional multi-step teacher models are distilled into causal two-step student models; autoregressive unfolding from noise; NVIDIA’s open-source library FastGen implements these algorithms.

Distilling diffusion models from noise to clean images typically requires 50 or even 100 denoising steps. "If we want steering wheel control over this model, that’s clearly unmanageable." So, another round of post-training is done to reduce denoising to two steps. "There’s a slight loss of quality at each step, but it’s very close to the original model." She provided a link: NVIDIA’s open-source FastGen library implements these algorithms, "so you can build it yourself."

Vehicle Review

First, "2 billion parameters, two-step denoising" represents an engineering trade-off, and it’s reproducible. Compared to last year’s 7 billion parameters and minutes-per-segment models, AlpaDreams follows a path of "small model + distillation + causality." Each of the four stages has open-source algorithms and code (Cosmos, FastGen), with training cycles lasting "a few days." For domestic teams, this means the barrier to creating a real-time interactive driving world model has dropped from "needing a foundational model team" to "needing a team that understands diffusion distillation." The gap isn’t in methodology but in data and iteration speed.

Second, "seven cameras" is NVIDIA’s default assumption for mass-production sensor configurations. The four-view demo and seven-camera limit correspond to NVIDIA’s own vehicle sensor layouts. Domestic automakers have varying camera counts and layouts (7 to 11 cameras are common, some with LiDAR). When fine-tuning open-source models, multi-view consistency will likely require retraining.

VIII. Performance and Results: 220 ms, 54 FPS, One B300 GPU

Single-view model: 1 GPU – 220 ms / 54 FPS, 16 GPUs – 146 ms / 82 FPS; four-view model: 1 GPU – 1290 ms / 12 FPS, 16 GPUs – 152 ms / 105 FPS; single-view generates 12 frames per batch, four-view generates 16 frames per batch, two-step diffusion, 704×1280 resolution.

The horizontal axis represents the number of GPUs, with inference parallelized across multiple cards; the vertical axis represents latency—how long it takes to receive a batch of frames. The model doesn’t generate frames one by one but in batches of 12, at 30 FPS, so each batch covers roughly one-third of a second of content. The single-view model takes 220 ms on one GPU and nearly halves that with 16 GPUs; the four-view model takes over a second on one GPU but matches single-view speed with multiple GPUs. "Around 100 ms, combined with some steering wheel implementation tricks, crosses the real-time threshold—people feel like they’re driving a game engine." In terms of throughput, each batch of 12 frames translates to 54 FPS per card.

The top row shows conditions (lane lines and object bounding boxes); the bottom row shows generated results from four cameras: front-left, front, front-telephoto, and front-right.

She highlighted several points about the results: the model generates vulnerable road users (VRUs)—pedestrians and cyclists—well; rain automatically triggers windshield wipers. "We didn’t tell it to wipe; it learned from the data that it should wipe when it rains, and it’s very consistent—physics is somewhat learned." Multi-view consistency is entirely learned: "No special tricks, just a Transformer attending to multiple views, learning cross-view 3D consistency."

For the same original scene, a single prompt—"change to night" or "make it snow"—simultaneously alters all four cameras, with winter trees changing accordingly.

Editing is her favorite part: "Neural reconstruction can’t do this at all, or at least it’s much harder. Here, with a single prompt, the same scene can become nighttime or snowy—it even changes the trees because winter trees look different."

For the same world state and planned trajectory, the top-right shows AlpaDreams’ pure generative rendering, the bottom-right shows NuRec’s reconstruction rendering; after deviating from the original trajectory, reconstruction deteriorates, while the generative model "continues drawing."

Finally, a full-pipeline comparison: the top-left shows conditions, identical for both rendering engines; the top-right shows AlpaDreams, the bottom-right shows reconstruction; the bottom-left shows the actual autonomous driving software outputting the trajectory. Both see the same first frame, but reconstruction relies on the entire original clip and deteriorates once the trajectory deviates. "The generative model keeps drawing—maybe fabricating, but it keeps going." She specifically highlighted people: "Humans are hard to reconstruct, especially legs, because they move fast—reconstructing from a moving car is especially hard. The generative model does quite well; it learned what people look like from the data." (This video segment wasn’t shown at the event.)

Vehicle Review

First, performance figures must be read alongside resolution and batch size. 54 FPS is for single-view, 704×1280, 12 frames per batch, two-step denoising; four-view achieves only 12 FPS on a single card and requires 8+ cards for real-time performance. If domestic teams aim for these numbers, they must align with the same metrics—otherwise, "we achieved real-time too" is meaningless. Another detail: the demo ran on one B300—NVIDIA’s latest data center GPU. The available compute in China lags behind, so latency on domestically available cards will differ.

Second, the comparison design—"reconstruction watches the entire clip, generation watches the first frame"—is fair and revealing. It clearly delineates the boundaries of both methods: near the original trajectory, reconstruction is more realistic; away from it, generation is more stable. Domestic teams often evaluate world models using offline metrics like FID or FVD, but NVIDIA uses "which rendering collapses first under the same planned trajectory"—a more application-relevant evaluation.

Third, the automatic appearance of windshield wipers is a small detail but illustrates the value of post-training data. The model learned "wipe when it rains" not from annotations but because NVIDIA’s fleet data always showed rain and wipers together. Domestic automakers’ data is far more diverse in weather, road conditions, and vehicle models than NVIDIA’s 20,000 hours—this is the true advantage of domestic post-training: not more data, but data distributions closer to real Chinese road conditions.

IX. Live Demo: Let the Model Drive First, Then a Human, Finally the Audience

The demo was conducted by two NVIDIA Spatial Intelligence Lab researchers, Jia Liang and Rui Long. Fidler noted, "They haven’t slept in months." The setup included a desktop DGX with force-feedback steering wheel and pedals—"what Jensen’s been playing with."

Segment 1: Alpamayo Closed-Loop "First-ever stage demo": The planning model Alpamayo generates trajectories as input for AlpaDreams; AlpaDreams generates the next frame, and Alpamayo navigates within it, all in a closed loop. The scene is a generated urban area, with the steering wheel turning autonomously "because the planner decided to turn here," featuring shadows, sunlight, and multi-view consistency. Rui Long emphasized real-time generation: "It looks like video because everything’s in the loop." Then, the weather was changed—"just replaced the first frame and text prompt"—turning the same scene rainy. A box was placed in the road, and Alpamayo’s thought chain scrolled at the top: "Slightly left to avoid the box, leaving space."

Segment 2: Human Driving Rui Long took the wheel in single-view mode: "This little DGX can run it locally, everything runs on-device." His steering inputs were rendered into an HD map as conditions for the video model, which autoregressively generated the scene in the DGX. He drove into a bike lane, against traffic, onto the sidewalk, and made a U-turn: "I didn’t know how to U-turn, but I did it anyway." Switching to snowy weather triggered the wipers automatically: "This is new." Switching to rain blurred the lens. Jia Liang explained consistency comes from KV Cache storing historical generations and noted: "Long-horizon autoregressive video generation has long been a challenging research problem in academia," yet AlpaDreams can drive for minutes while remaining reasonable and consistent. At 60 km/h, Jia Liang exclaimed, "Too fast!"

Segment 3: Audience Driving "To prove it’s not scripted," two audience members took turns. One drove into a building; another drove off the map boundary—"it still worked normally." Fidler’s offline comment: "Nice force feedback." The audience members later said, "It’s harder than it looks."

Vehicle Review

First, this was a "dare-to-fail" demo, rare in domestic launches. Letting random audience members drive, leave the map, or crash into buildings could easily cause glitches or collapses. NVIDIA’s willingness to do this shows confidence in the model’s out-of-distribution behavior and treats "collapsing is fine" as a normal research demo state. Domestic launches typically use pre-recorded or highly controlled demos—not a technical gap but a cultural one that affects perceptions of technical maturity.

Second, the detail "everything runs locally" matters practically in China. Single-view mode runs on a desktop DGX, meaning such world models don’t require data centers and can fit on every algorithm engineer’s desk. Domestic automakers’ simulation resources are mostly cloud-based, with queue times for compute. If world models can run locally, iteration cycles will change entirely.

Regrettably, the speech did not detail Alpamayo’s architecture, parameter count, or training specifics.

X. Final Thoughts: What Can China’s Industry Take from This?

Summarizing the nine sections, NVIDIA’s answer is one sentence: First, use neural reconstruction to build closed-loop evaluation as a daily 2-million-iteration infrastructure, then use generative world models to fill what reconstruction can’t do, sharing the same runtime and abstract state, with the planning model iteratively culled. From China’s industry perspective, I see five direct takeaways.

1. "World models" at NVIDIA are R&D tools, not vehicle selling points. The entire speech avoided vehicle-side compute or mass-production timelines, focusing entirely on training and evaluation. Domestically, world models and VLA have been marketed as consumer differentiators in the past two years, but iteration speed depends on simulation-side world models. Both are called "world models," but their teams, compute, and metrics should be separate.

2. Reconstruction precedes generation; veto power precedes scenario count. NVIDIA runs reconstruction on production lines, with every mass-production candidate undergoing closed-loop evaluation—"most versions fail." If domestic teams’ simulation is still at regression testing and demos, the first step isn’t adopting generative models but giving existing simulation veto power over releases.

3. Open-source foundations will level the "world model" playing field, shifting gaps to data distribution and iteration speed. Cosmos, NuRec toolchain, AlpaSim, and FastGen are all open-source; training a real-time interactive driving world model requires only tens of thousands of hours of data and a few days. Domestic automakers’ advantage lies in data distributions closer to Chinese road conditions; their shortcoming is foundations, compute, and next-gen GPU availability. Using open-source foundations with proprietary data for post-training is a realistic path but requires accepting rising dependence on NVIDIA’s ecosystem for simulation.

4. Separate behavior and rendering. AlpaSim controls how traffic participants move; AlpaDreams controls how the scene looks; HD maps and object bounding boxes serve as the interface. This layering makes scenarios controllable, evaluations reproducible, and modules replaceable. When designing simulation architectures, domestic teams should define this interface, even if using others’ generative models.

V. Let Engineers Take the Wheel. User-in-the-loop, local operation of desktop DGX, and audience members randomly taking the stage—the common thread in these actions is transforming world models from 'offline metrics' into 'something you can drive.' Many issues of physical consistency—windshield wipers, U-turns, driving onto sidewalks—are things humans can perceive within minutes of driving, yet are difficult for automated metrics to capture. This is a low-cost endeavor that is rarely done domestically.

Finally, a detail. When Fidler discussed the previous generation of models a year ago, she said, 'You can see the future, but it's not practical.' A year later, she handed the wheel to randomly selected audience members. In the Chinese industry, when it comes to world models, we don't lack product launches that 'show the future.' What we lack is turning it into the kind of production crew that runs 2 million times a day and can be driven from every workstation—that is, implementation.

References and Images

【PPT】Advancing Autonomous Vehicles With World Models / Sanja Fidler*Reproduction and excerpting are strictly prohibited without permission-