The stability and safety of an autonomous driving system hinge on its ability to continuously learn and adapt. Such a system cannot rely solely on pre-programmed instructions indefinitely. During operation, it frequently encounters situations that it cannot "comprehend" or "accurately assess." If these issues and novel scenarios from real-world driving are not relayed to the R&D team, it becomes challenging for the team to rectify defects and enhance system capabilities.

The data closed loop is designed precisely to tackle this challenge. It involves the continuous transmission of data collected by vehicles during real-world driving or testing back to the development team. After processing, learning, validation, and redeployment, the updated system is then applied to the vehicles. As long as this cycle functions effectively, the autonomous driving system can continually evolve.

Image Source: Internet

The primary objective of the data closed loop is to swiftly identify, label, analyze, and utilize new issues encountered in real-world traffic scenarios to update models, thereby preventing the recurrence of similar problems. This process bears resemblance to the version iteration in software development, where issues are identified, feedback is gathered, fixes are implemented in the background, new versions are rolled out, and the cycle repeats. However, in autonomous driving, due to the involvement of extensive sensor data, machine learning, and simulation testing, the data closed loop system it relies on is significantly more intricate.

Data Collection: The First Step in the Data Closed Loop

To establish an effective data closed loop, data collection is paramount. Autonomous vehicles are equipped with an array of sensors, including cameras, millimeter-wave radars, and LiDAR, which capture real-time information about the vehicle's surroundings. This sensor data is the most original and comprehensive, reflecting road conditions, obstacles, traffic signals, and the behavior of other road users. This real-time data forms the bedrock of the entire closed-loop system.

The sources of this raw data can be categorized into two types: data collected by test vehicles operating in closed test sites or on open roads, and data gathered by mass-produced vehicles during actual road operations. The former allows for controlled test scenarios, covering various predetermined conditions; the latter captures real problems and numerous edge cases in real-world traffic environments. The collected data is then transmitted to the cloud or data center for subsequent processing.

It is crucial to note that this data is not as easily organized as conventional system logs. It encompasses diverse information types such as images, LiDAR point clouds, and radar signals, which are complex and varied. Most of this content cannot be directly utilized for model training. Therefore, the collected data must first undergo a screening process to extract the most valuable road condition segments, specific error scenarios, etc. This ensures that subsequent processing steps are not hindered by massive amounts of invalid data, enabling more focused optimization and learning of key issues.

Data Preprocessing and Cleaning: Crucial Steps

The raw data just collected cannot be directly used for model training and must undergo preprocessing and cleaning. The goal of this step is to eliminate interfering information from the data and extract the truly useful parts.

Preprocessing involves operations such as data format conversion, time alignment, and coordinate unification. This is because different sensors on autonomous vehicles have their own clocks and coordinate reference systems. If their data is not aligned in time and space, subsequent analysis will be chaotic. For instance, if the position of an obstacle detected by LiDAR is not synchronized in time with the image captured by the camera, it will be difficult to ascertain whether the obstacle truly exists.

Cleaning entails screening out data with obvious errors, missing data, or incomplete parts. For example, sensors may be obstructed or interfered with during high-speed driving, resulting in unreliable data. If such data is used for training, it may cause the model to learn incorrect patterns. Therefore, data cleaning is essential to ensure the effectiveness of model training.

At this stage, automatic labeling technology is also employed. Through automatic labeling tools, the positions and types of objects such as pedestrians, vehicles, and traffic signs in images can be initially identified and labeled. Subsequently, experienced engineers review and correct the automatic labeling results to ensure accuracy. The combination of "automatic labeling + manual review" significantly enhances the efficiency of the labeling process.

Training and Optimizing Models with Data

The cleaned and labeled data is then utilized for model training. In autonomous driving systems, most perception, prediction, and planning functions rely on machine learning models, which require vast amounts of accurately labeled data to "learn" how to recognize scenarios and make correct judgments.

Training work is typically conducted on high-performance computing clusters in the cloud. Before that, the prepared data is categorized based on its purpose, such as data for perception model training, data for prediction model training, and data for simulation testing, and then combined into training sets and validation sets. Machine learning algorithms repeatedly adjust the internal parameters of the model to enable it to make accurate judgments when encountering new data.

This training is not a one-time process but an ongoing iteration. Whenever new data is labeled, it can be added to the training set to provide the model with more diverse training. This allows the model to continually learn new situations and enhance its accuracy.

Some technical solutions also incorporate large models to expedite this process. With stronger understanding capabilities, large models can automatically identify complex scenarios and extract features, thereby reducing manual involvement and improving training efficiency.

Simulation Testing: Validating Updates in a Virtual Environment

After the model is trained, it cannot be directly deployed to vehicles for operation; it must undergo rigorous testing. Although real-world road testing is necessary, it is costly and risky. Therefore, simulation testing is an indispensable component of the data closed loop.

Simulation environments can replicate various road scenarios, traffic conditions, and weather conditions. The newly trained model can be repeatedly tested in the simulation environment to verify whether it can maintain safety and stability under diverse circumstances. Scenarios such as peak-hour congestion, suddenly crossing pedestrians, and complex intersections can all be repeatedly tested in simulations.

One crucial role of simulation testing is to uncover edge scenarios that the model may encounter on real roads but has not yet encountered. Due to their low probability of occurrence, these scenarios are difficult to capture through actual road testing. However, if encountered, they may cause the system to fail. Therefore, simulation testing can compensate for the lack of coverage of these scenarios.

Simulation systems can also generate new test scenarios based on existing data to supplement the deficiencies of real-world data, which is also an important way to enhance training coverage and model robustness.

Vehicle-End Validation and Deployment

Models that have passed training and simulation testing can be deployed to vehicles for validation. At this stage, the vehicle operates under a wider range of real-world road conditions to observe whether the performance of the autonomous driving system aligns with simulation testing.

Vehicle-end validation still generates a substantial amount of data, which can be fed back to the cloud again for the next cycle of collection and analysis. Through this process, the operational validation of the new model becomes the input for the next closed-loop iteration.

At this stage, the most critical task is to ensure effective monitoring and anomaly detection. The system needs to record in real-time the differences between each decision, each prediction, and the actual situation. Once it detects a trend of judgment bias in specific scenarios, the relevant data must be promptly extracted and used as important material for the next round of training.

Through such continuous validation and feedback, the entire autonomous driving system can gradually improve, evolving from initially being able to operate only in simple road conditions to eventually becoming a mature system capable of handling real-world challenges such as complex traffic environments and adverse weather conditions.

Challenges in Deploying a Closed-Loop System

Building an efficient data closed loop is not as simple as merely transmitting data from vehicles back to the backend. It is more akin to constructing an automated "learning assembly line," requiring close coordination among multiple steps and the provision of corresponding tools and platforms.

Due to the enormous volume and diverse types of data generated in the data closed loop, high-performance storage and large-scale data processing capabilities are essential for efficiently storing, accessing, and organizing this vast amount of information.

Automatic labeling and data processing tools are also crucial. They determine whether raw data can be quickly and accurately converted into training samples for model learning, which directly affects the progress and quality of subsequent steps.

Simultaneously, powerful training and simulation computing platforms are indispensable. Model iterative learning relies on sufficient computing power support, while simulation environments can safely and efficiently verify algorithm performance in numerous scenarios.

Additionally, a model deployment and real-time monitoring system must be established. This ensures that the updated model can be smoothly applied to vehicles and continuously monitors its performance during actual operation, promptly identifying issues and triggering a new round of optimization.

It is important to note that throughout the closed-loop process, data collection and processing must adhere to compliance and privacy protection principles. The data collected by autonomous vehicles sometimes involves personal image information or other sensitive content. This data must be desensitized during transmission and storage to ensure that personal privacy is not compromised. Furthermore, various countries and regions have stringent regulations on the use and cross-border transmission of autonomous driving data, and development teams must comply with these legal and regulatory requirements.

In summary, the data closed loop requires systematic construction across the entire chain, from collection, storage, processing, training, testing to deployment and validation, forming a highly automated and rapidly responsive operational mechanism. Only in this way can the closed loop truly function, driving the continuous evolution of the autonomous driving system.

Final Thoughts

The advancement of autonomous driving technology is inextricably linked to the data closed loop. A well-established data closed loop system enables various new situations encountered by vehicles in real-world traffic scenarios to be promptly captured, organized, learned, and utilized for system updates. This not only enhances the safety and stability of the system but also accelerates the overall R&D progress.

-- END --