An end-to-end model may exhibit excellent performance metrics on offline test sets, only to reveal unforeseen issues when deployed in real vehicles. Why do end-to-end models perform well during training but encounter problems on the road? Let's delve into this topic today.

01 Why Does Normal Training Lead to Issues in Real-World Deployment?

During the training phase, the model learns the relationship between inputs and driving behaviors from extensive historical driving data. By feeding the model numerous samples containing camera feeds, vehicle status, and navigation data, it can learn to map inputs to driving behaviors.

In the validation phase, a separate batch of data, not used during training, is employed to verify whether the model can maintain consistent performance.

Image source: Internet

While this type of testing is crucial, it remains a relatively typical offline, open-loop evaluation. The model's predictions do not retroactively alter the pre-recorded data for subsequent moments. Once the model participates in the vehicle's closed-loop decision-making, its outputs—whether driving intentions, trajectories, or other intermediate results—will ultimately influence the vehicle's subsequent speed, position, and state through execution.

Consequently, the next input the model receives may differ from the next frame in the original data. There is a significant distinction between these two testing methods.

Offline testing asks the model how to handle a given scenario; real-world driving observes the final state of the vehicle after the model makes continuous decisions. For instance, if the model slightly deviates from the ideal trajectory at an intersection, this deviation may not be severe on its own. However, after the vehicle's position changes, the lane markings, surrounding vehicles, and road spatial relationships observed in the next moment will also change. If the model introduces another slight deviation in this new state, errors may continue to accumulate.

Therefore, strong predictive performance at a single moment or on a single data segment does not automatically equate to stability during prolonged continuous driving.

This is why autonomous driving development increasingly emphasizes closed-loop testing, where the model's decisions influence the subsequent evolution of the vehicle's state and environment, and the results of continuous decision-making are observed, rather than merely comparing the model's outputs to predefined reference answers in the data.

02 The Model May Encounter Unfamiliar States

End-to-end models frequently confront the issue of distribution shifts. Training data records past driving behaviors, but the behaviors generated by the model after deployment may not align perfectly with those in the data.

Suppose most vehicles in the training data are positioned relatively commonly within lanes, and the model learns how to drive in these states. When the model drives independently, if a judgment causes the vehicle to deviate slightly from the normal trajectory, the subsequent inputs it receives may differ from the typical states in the training data.

In this case, the model is not simply encountering an unseen vehicle but rather an unseen combination of vehicle position, road relationships, and traffic conditions. This is a unique aspect of end-to-end systems.

Image source: Internet

Traditional modular systems can break down problems into multiple stages, such as perception, prediction, and planning, making it easier to trace potential issue sources across different modules when problems arise.

End-to-end models, however, incorporate more capabilities into a unified learning system, causing many issues to manifest directly as abnormal model output behaviors.

Moreover, distribution shifts in real-world driving do not originate solely from the road itself. Weather, lighting, road construction, traffic participant behaviors, vehicle status, and regional driving habits can all create differences between real-world and training data.

Thus, a training set that covers sufficient ground does not guarantee the disappearance of all issues. Because real roads are not a finite set of scenarios but rather a complex system formed by the continuous combination of varying environments, participants, and vehicle behaviors.

03 Why Are Rare Scenarios More Prone to Exposing Issues?

Scenarios such as normal car-following, routine intersection navigation, and ordinary lane changes are more likely to yield abundant samples in actual data collection. However, some long-tail scenarios that truly determine the system's upper limits are exceedingly rare.

For example, scenarios like sudden changes in road boundaries due to construction zones, abnormal vehicle maneuvers, inconsistencies between road markings and actual traffic flow, or the simultaneous occurrence of multiple low-probability factors are difficult to collect. These scenarios may not be entirely absent from the training data but might lack sufficient variability.

This creates a problem: the model may learn how to handle common scenarios in the training data but fail to adequately learn how to respond stably when states shift.

Image source: Internet

Therefore, the data loop for end-to-end models is not simply about continuously increasing total mileage but rather about continually identifying scenarios where the model is prone to errors and reintroducing these scenarios into the training, validation, and testing frameworks.

This is why autonomous driving development increasingly emphasizes long-tail scenarios and difficult sample mining. Consequently, the value of simulation technologies in training large autonomous driving models continues to rise.

Simulation does not merely allow large models to accumulate virtual mileage; it enables proactive modification of variables such as weather, traffic participant behaviors, and road conditions to construct scenarios that are difficult to collect in large numbers on real roads and further conduct closed-loop testing.

04 Why Does Training Success Not Guarantee On-Road Success?

Training and offline validation focus on whether the model can produce reasonable results given existing data. In contrast, real-world driving emphasizes whether the model can consistently maintain the vehicle in a safe and controllable state after making a series of continuous decisions.

Between these two lies a critical chain: model output → vehicle state change → new environmental input → subsequent decision-making → potential error accumulation.

If a deviation occurs at any step, causing the vehicle to enter a state less covered by training data, subsequent issues may amplify further.

Therefore, the true challenge for end-to-end models is not simply achieving higher accuracy on training sets but gradually narrowing the gap between offline data distributions and real-world closed-loop driving distributions.

Image source: Internet

This is why modern end-to-end autonomous driving development does not conclude after model training. A continuous data loop must form among real-world road data, difficult scenario mining, simulation, closed-loop testing, and subsequent training.

Thus, the phenomenon of training success but on-road failure is not a simple flaw unique to end-to-end models; it is an inherent challenge learning-based driving systems face when transitioning from offline data to real-world closed-loop operations.

A truly mature end-to-end system must continually identify states where the model is prone to failure and incorporate these states into the system's training, validation, and testing coverage through data, training, and closed-loop verification.

The true difficulty of end-to-end systems has never been merely training the model well but enabling it to consistently make correct decisions in the real world. #AutonomousDriving #End-to-EndLargeModel