In 2026, Waymo's research team introduced the WOD-E2E (also referred to as Rare-E2E) end-to-end driving dataset at CVPR 2026. This dataset comprises 4,021 driving clips, totaling approximately 12 hours, extracted from around 6.4 million miles of real-world road testing.

It specifically categorizes long-tail scenarios that occur with a frequency of less than 0.03% in daily driving and introduces the RFS (Rater Feedback Score) evaluation metric to assess whether driving behavior is reasonable and stable.

In fact, once an end-to-end autonomous driving system can reliably navigate a multitude of common driving scenarios, it is not the frequently encountered normal roads that truly test the model's capabilities, but rather the rare, complex, and error-prone scenarios.

01 Why Is It Essential to Actively Seek Errors as the System Matures?

End-to-end autonomous driving necessitates learning the correlation between environmental changes and vehicle behavior from a vast amount of driving data.

High-frequency scenarios, such as following, lane-changing, and navigating through ordinary intersections, can generate a substantial volume of training data through continuous collection. However, there exists another category of data on real roads that is less frequent but more likely to reveal system vulnerabilities.

Examples include sudden cut-ins by the vehicle ahead, pedestrians emerging from obscured areas, construction altering the original road layout, or multiple traffic participants making unexpected moves simultaneously.

These scenarios constitute a very small proportion of the massive amount of normal driving data but may directly influence the model's performance in complex environments.

This presents an urgent challenge: the more common a scenario is, the easier it is to amass a large amount of data, whereas the rarer a scenario is, the more data-deficient it becomes.

Therefore, once a sufficient quantity of data for common scenarios has been accumulated, continuing to add a large number of highly similar normal driving data may gradually diminish the marginal value in uncovering the model's long-tail capability deficiencies.

At this juncture, data that can expose model issues becomes more noteworthy.

02 Truly Valuable Data Does Not Necessarily Involve Already Committed Errors

Training solely with data where collisions occur or the vehicle makes blatantly incorrect decisions is far from adequate.

What autonomous driving truly needs to focus on is identifying the model's weaknesses as much as possible before errors lead to severe consequences.

For instance, a vehicle safely navigates through a complex intersection, but when confronted with simultaneous occurrences of occlusions, crossing pedestrians, and oncoming vehicles, the model's output driving behavior becomes noticeably unstable.

Although this data does not result in an accident, it may be highly valuable as it exposes the model's instability when facing specific scenario combinations.

Image source: Internet

There is also data where the vehicle ultimately completes the driving task but exhibits abrupt braking, frequent trajectory adjustments, or significantly different driving behaviors in similar scenarios.

These situations are equally deserving of exploration and analysis. From the perspective of a data loop, this type of data holds more value than merely recording what transpired.

At this point, it is necessary to scrutinize such data to ascertain whether it is an isolated incident or a recurring issue in certain scenarios. Is it due to insufficient perceptual information, inadequate model understanding of scenario relationships, insufficient training data coverage, or generalization issues under specific conditions?

In other words, what truly warrants exploration is not just data where errors have already occurred but data that can assist the engineering team in understanding why errors occur.

Edge scenarios represent a crucial category among them.

Research related to Safety Of The Intended Functionality (SOTIF) also highlights that although these scenarios occur infrequently, they may expose the functional limitations of autonomous driving systems and pose greater challenges to system safety.

The RFS metric proposed by WOD-E2E is precisely designed to address such issues. In long-tail scenarios where traditional distance error metrics fail to capture the multimodality of driving behavior, RFS introduces human preference ratings for driving behavior, shifting the evaluation from whether the trajectory appears similar to whether the driving is reasonable and stable.

03 How Should Data Be Utilized After Errors Are Identified?

For end-to-end autonomous driving, simply discovering an edge scenario and incorporating it into the training set is insufficient. Data generated from real roads is extremely intricate, and the same abnormal performance may correspond to entirely different issues.

After data mining, it is necessary to analyze information such as vehicle behavior, model output, and scenario characteristics to determine whether the sample is an isolated incident or has a recurring pattern. If the same issue arises in a large number of similar scenarios, its value significantly increases.

The engineering team can supplement data, refine annotations, and integrate them into the training and testing processes for such scenarios.

Image source: Internet

Following model updates, it is also necessary to re-verify whether the original issues have been rectified while checking for the introduction of new problems.

This forms a complete data loop of real-world operation → discovering anomalies → mining difficult data → analyzing issues → training and testing → re-verification.

The most challenging aspect of this loop is not collecting data but identifying the truly valuable components to learn from within the massive amount of data.

This is why, as end-to-end intelligent driving progresses to a more mature stage, simply pursuing the scale of the training set is no longer the primary objective.

It is not to suggest that data scale is unimportant, but rather that while continuously accumulating data, there is an increasing need to actively seek data that can expose model weaknesses.

Because for a system that can already handle a large number of conventional scenarios, what truly determines the next step for improvement is not how much it has already mastered but in which situations it is still prone to making mistakes and whether it can transform these mistakes into data for the next round of model enhancement.

#AutonomousDriving #EndToEndLargeModel #EdgeScenarios