A real driving video can tell us what the vehicle saw and did at the time, but it cannot answer another question: what would happen next if a different decision had been made at that moment?

This is precisely the challenge facing autonomous driving testing.

For end-to-end models that learn driving decisions directly from sensor inputs, evaluating their behavior in a real driving closed-loop system solely based on how closely their predictions match real trajectories in existing data is insufficient for a comprehensive assessment.

01 Why Can't Open-Loop Testing See Another Future?

Open-loop testing refers to a model making predictions on fixed real-world road data, where its outputs do not alter the subsequent input data.

The development and evaluation of autonomous driving models have long relied heavily on real-world road data.

Vehicles record sensor data from cameras, LiDAR, and other devices, which the model then re-reads to determine if it can understand the road environment or make reasonable driving decisions.

While this approach holds significant value because the data comes directly from real roads, it has an inherent limitation: the future is already recorded.

For example, when a vehicle approaches an intersection, the real vehicle ultimately turns right. If the model predicts a leftward trajectory in an open-loop test, the subsequent observational data does not change based on this prediction. The next frame still comes from the original road recorded after the real vehicle turned right.

In other words, the model's actions do not genuinely alter the environment.

This is the core distinction between open-loop and closed-loop systems.

Open-loop evaluation can efficiently compare a model's predictive capabilities on fixed data, but the model does not receive a different observation at the next moment based on its own actions.

The CVPR 2025 paper 'DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation' explicitly points out that many autonomous driving evaluation methods predict waypoints on fixed routes in public datasets. Such open-loop methods lack dynamic feedback, making it difficult to assess a model's decision-making capabilities in response to environmental changes.

'Pseudo-Simulation for Autonomous Driving' also notes from another perspective that while open-loop evaluation is efficient and uses real-world data, its evaluation metrics often fail to fully reflect the cumulative errors from continuous decision-making.

This further illustrates that if a model makes actions different from the original driving trajectory, corresponding subsequent observations do not exist in the real data.

Yet, this is precisely what autonomous driving most needs to verify.

02 Closed-Loop Testing Truly Changes the World After an Action

Closed-loop testing refers to a model's output actions altering the environmental state, which then generates new observations fed back to the model, forming a cycle of action-environment-new observation-new action.

Closed-loop testing is not simply about making a video more realistic; it involves the model's actions genuinely participating in environmental evolution.

This process can be understood as follows: when the model sees the current road and decides to turn left, the simulation environment receives this action, and the vehicle's state changes. The environment then evolves based on the new vehicle state and generates sensor observations for the next moment. The model then makes its next decision based on these new observations.

Thus, the testing question shifts from whether the model can reproduce the original answer to what will happen in the world after the model makes a decision.

In a closed-loop environment with scene control capabilities, most conditions can be fixed while only one variable is changed.

For example, a neighboring vehicle suddenly moves closer to the lane where the ego vehicle is located, and the model's response is observed. If the model brakes, the environment continues to evolve; if the model does not brake, the environment also continues to evolve.

Image Source: Internet

In this way, the focus is no longer solely on whether a specific action is close to the standard answer in the data but on whether new risks arise after that action enters the driving closed-loop system.

The CVPR 2025 paper 'DrivingSphere' constructs a 4D world containing static environments and dynamic traffic participants, generating multi-view videos through visual scene synthesis to enable closed-loop interaction of driving strategies under environmental feedback.

Similarly, 'DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving,' presented at ICCV 2025, combines a traffic simulator with a generative world model, allowing driving agents capable of processing real images to drive in a closed-loop manner within the generated environment.

These studies demonstrate that closed-loop simulation is not about generating more driving videos but about enabling driving strategies to genuinely enter the environmental feedback process.

Of course, closed-loop testing is not meant to replace open-loop testing. Open-loop testing remains suitable for large-scale, repeatable model comparisons and problem identification; closed-loop testing further evaluates the continuous behavior of strategies after interacting with the environment. They address different issues.

03 Why Does Closed-Loop Simulation Still Need a World Model?

Given the importance of closed-loop testing, can traditional simulators accomplish this?

The answer is yes. Simulation environments based on rules, physics engines, or traffic models can also enable closed-loop interactions between vehicles and virtual traffic participants.

However, such simulators often feed the model abstract state information (e.g., position, speed, lane markings) rather than data consistent with real onboard cameras.

For end-to-end models that directly use raw sensor inputs like cameras, traditional simulators are insufficient.

These models do not perceive simple 3D coordinates but rather images captured by specific cameras at specific positions, with specific fields of view and imaging conditions.

If the simulation environment only provides abstract road and vehicle states, discrepancies may still exist between the model's actual inputs and those of a real vehicle.

DrivingSphere identifies this issue as a significant challenge in closed-loop simulation in its paper.

Some existing closed-loop methods can provide feedback but cannot directly process real visual sensor inputs. Meanwhile, simulators based on game engines, while capable of building interactive environments, may generate results that differ from real sensor data.

Image Source: Internet

This explains why world models have begun to enter autonomous driving simulation in recent years.

World models offer a different approach. Instead of merely creating a virtual scene where vehicles can move, they continuously generate observations corresponding to real sensor inputs for the next moment based on the current world state and driving actions.

Thus, world models address two issues simultaneously.

One is visual authenticity: whether the generated images closely resemble what a real vehicle would see.

The other is dynamic plausibility: after the vehicle changes direction, speed, or trajectory, whether subsequent images change reasonably according to the new state.

The latter is where closed-loop world models face their greatest challenge.

The CVPR 2025 paper 'DriveDreamer4D' points out that traditional sensor simulation methods like NeRF and 3D Gaussian Splatting have limitations in handling complex driving actions, whereas world models can be used to construct 4D driving scene representations.

The paper 'ZYT-World: A Real-Time Controllable World Model for Closed-Loop Autonomous-Driving Simulation,' publicly released by ZhiYu Technology in September 2026, further advances the issue to real-time closed-loop simulation.

The paper proposes a real-time, controllable world model for end-to-end and VLA driving strategies, modeling the hybrid camera configurations of mass-produced vehicles. Its solution natively generates four fisheye and three pinhole perspectives, with fisheye fields of view exceeding 180°.

For real vehicles, this is not merely a difference in visual appearance. Different cameras have distinct projection methods, intrinsic and extrinsic parameters, and fields of view. If the world model ultimately serves as input to the driving model, these sensor characteristics must be preserved as much as possible.

04 From Replaying the Past to Verifying Another Future

Another significant value of closed-loop simulation is enabling autonomous driving testing to possess counterfactual capabilities.

Real-world road recordings capture what has already happened, but autonomous driving faces situations with multiple possible outcomes.

The vehicle ahead can brake or continue; a neighboring vehicle can stay in its lane or suddenly merge; the ego vehicle can decelerate early or proceed.

Real roads cannot repeat the same high-risk scenario hundreds of thousands of times to verify a model. However, in a simulation environment with control capabilities, specific variables can be manipulated while keeping other conditions constant.

Thus, testing can evolve from 'What happened in this scenario?' to 'What would happen next if the vehicle had made a different choice at that moment?'

This is the value of counterfactual scenarios for autonomous driving testing.

However, a crucial prerequisite exists: the alternative future generated by the simulation world must be sufficiently credible.

If the model generates a scene that appears realistic but violates traffic laws or spatial relationships in response to vehicle actions, the closed-loop system cannot guarantee valid testing results.

Once a driving strategy enters an erroneous environment, all subsequent evaluations may lose meaning.

Therefore, the true challenge for closed-loop world models lies not merely in generating a realistic image but in maintaining consistency in time, space, and causality across continuously generated worlds.

ZYT-World addresses this issue by designing mechanisms such as causal consistency distillation, self-rollback distribution matching distillation, RigCritic, and memory modules to handle real-time generation, multi-camera consistency, and scene revisitation.

ZYT-World Distillation Flowchart, Image Source: ZhiYu

The paper evaluated its solution using a self-constructed multi-view test set, showing that the one-step generation model retained over 90% of the teacher model's PSNR and SSIM. Under the paper's specified generator-only timing conditions, the generator was 107.7 times faster than the 40-step bidirectional teacher model.

It should be noted that this figure corresponds to specific generator stages and testing conditions and should not be interpreted as the entire autonomous driving simulation process achieving a 107.7-fold speedup.

This also indicates that closed-loop world models are better understood as a rapidly evolving simulation technology route rather than as having completely solved autonomous driving testing challenges.

05 Concluding Remarks

In the past, we primarily asked whether a model could make judgments similar to real driving after viewing certain data.

Closed-loop testing now asks: if driving decisions are truly entrusted to this model, where will each of its decisions lead the world next?

This is why autonomous driving testing is gradually transitioning from open-loop to closed-loop systems.

Open-loop data will not disappear, nor will real-world road testing vanish due to simulation advancements.

What is truly changing is the boundary of testing capabilities. Scenarios that are difficult to repeat, impossible to exhaustively enumerate, yet require verification of the consequences of continuous decision-making can now be repeatedly attempted in controllable closed-loop environments.

Closed-loop testing does not replicate the real world inside a computer but provides autonomous driving models with a testing space outside the real world where they can repeatedly make mistakes, alter the future, and undergo re-verification.

#AutonomousDriving #OpenLoopTesting #ClosedLoopTesting