In the modular era of autonomous driving, data annotation primarily involved drawing boxes around objects—such as framing pedestrians and vehicles in images and enclosing point clouds with cubes.

However, merely drawing boxes only informs the AI of what is present, not what will happen next. Whether a vehicle is accelerating or a pedestrian is about to cross the road, such dynamic information is often overlooked during the annotation process.

With the advent of end-to-end systems, the requirements for annotation have undergone a fundamental shift.

Models no longer rely on simple boxes; instead, they require the driving trajectories of human drivers, such as the degree of steering wheel turn or accelerator depression. Simultaneously, end-to-end models demand massive amounts of data covering various scenarios, yet real-world dangerous scenarios are inherently rare.

The industry has recognized the need for a new data infrastructure to address this challenge, leading to the emergence of the cloud data engine concept.

01 Why a Data Engine is Needed

The cloud data engine is not a single algorithm but an automated data pipeline deployed in the cloud. Its primary task is to automatically convert massive amounts of raw driving data transmitted from vehicles into high-quality datasets suitable for training models.

Tencent's Data Closed-Loop Platform, Huawei's World Engine within the Qiankun WEWA architecture, and Tesla's data engine are all different implementations of this concept.

Image Source: Internet

This engine liberates human labor from repetitive and strenuous tasks.

Traditional manual annotation of a single frame of 3D point cloud data can cost up to 17 yuan, with an individual able to complete only three frames per day. Annotating 500,000 frames would require 8.5 million yuan and 167,000 person-days, equivalent to 100 people annotating continuously for 4.5 years.

In contrast, the engine can fully automate the annotation process, achieving efficiency improvements by several orders of magnitude.

Tesla's 4D automatic annotation system can mark 10,000 driving trajectories in just 12 hours of cluster operation, a task that would take 5 million person-hours manually.

More importantly, it transforms the mindset behind annotation.

Traditional annotation is passive, annotating whatever is provided, resulting in low value density. The engine, however, can proactively mine high-value scenarios.

From the massive amounts of data transmitted daily, it automatically filters difficult segments such as narrow road meetings or suddenly appearing pedestrians, prioritizing them for the annotation pipeline. Ideal Auto has already deployed over 200 data trigger conditions to produce high-value video clips lasting 15 to 45 seconds from 1.5 billion kilometers of driving data.

This ensures that limited computational and storage resources are used effectively. For end-to-end models, the engine provides a capability that traditional annotation cannot achieve.

It no longer annotates only static objects in single frames but reconstructs the 4D spatiotemporal information of entire video segments offline.

Each dynamic target is assigned a unique ID across hundreds of consecutive frames, and the system calculates its precise trajectory and speed. This provides the model not with photographs but with complete video footage containing all temporal changes.

Tesla employs multi-pass scene reconstruction technology to achieve 4D automatic annotation, obtaining pseudo-ground truth that largely replaces manual annotation. Based on annotated real-world scenarios, the engine can also automatically adjust lighting, weather, and backgrounds to generate numerous variant datasets.

A single scenario of a rainy night can be derived into thousands of versions with varying rainfall, brightness, and traffic density. Huawei's World Engine, based on a diffusion generation model, can increase the generation density of key difficult scenarios by 1,000 times.

This significantly expands the coverage of the training set, allowing models to encounter more diverse situations during training. According to a report released by SenseTime, using world models to automatically generate annotated image and video data for training can save companies nearly half of their data costs while improving data preparation efficiency by about 70%.

02 How Does the Engine Work?

The cloud data engine consists of three layers, each serving a different function. The first layer addresses data filtering. Each vehicle in the fleet transmits large amounts of data daily, but most are repetitive everyday scenarios.

The engine runs a scene understanding model at this layer, automatically assigning semantic labels to each video segment (e.g., rainy, nighttime, congested, no lane markings). It then calculates an uncertainty score for each segment using active learning algorithms. The more the model struggles to understand a scenario, the higher its priority.

Research has shown that active learning methods based on inconsistency can achieve the same detection accuracy as random sampling using only 50% of the annotated data.

Only high-value segments are sent to the next stage. This step essentially increases the value density of the data, avoiding the waste of computational resources on mundane material.

Image Source: Internet

The second layer is the core of automatic annotation. It calls upon large models in the cloud to jointly process the filtered video segments. 2D images from cameras, 3D point clouds from LiDAR, and vehicle IMU data are first aligned within a unified spatiotemporal coordinate system.

This step requires high precision, with microsecond-level timestamps and millimeter-level spatial coordinates needing to coincide perfectly. Alignment errors exceeding a few centimeters can lead to deviations in subsequent annotation positions. Fusion4DAL proposes an offline multimodal 3D object detection method that overcomes point cloud sparsity issues through a virtual point mechanism, enhancing detection accuracy.

After alignment, the algorithm can process all frames of the entire video segment at once, using information from known futures to retroactively correct the positions of occluded objects in the current frame, a technique known as backward annotation. Each dynamic target is continuously tracked across hundreds of frames, and the system outputs 4D ground truth data with complete motion trajectories, all without human intervention.

Tencent's 4D Dynamic Annotation Large Model supports feature fusion from multiple cameras and LiDAR as well as dynamic learning of model weights. The third layer handles verification and expansion. After automatic annotation, the system quickly verifies the quality. Data with sufficient confidence is directly packaged and sent to the training cluster, while doubtful data is forwarded to human experts for local fine-tuning.

This model of large-scale automatic annotation combined with manual quality inspection and correction is replacing traditional purely manual annotation.

Additionally, the engine can generate numerous variant datasets based on annotated real-world scenarios, exponentially expanding the training set's scale at a relatively low cost.

03 Concluding Remarks

The cloud data engine is transforming the cost structure of autonomous driving research and development. The initial investment in building such an engine is substantial, requiring thousands of GPUs and complex engineering systems. However, once the fleet size is large enough, the marginal cost drops sharply.

This is why leading players are willing to invest heavily—it represents not just technical capability but also economies of scale.

From a market perspective, progress in this direction varies among companies. The true determinant of the gap likely lies not in the choice of model architecture but in the depth of the data infrastructure.

Whether for world models or end-to-end models, without high-quality 4D training data, predictive capabilities cannot be developed.

The quality of the data engine itself, in turn, depends on the accuracy and coverage of the automatic annotation algorithms.

This is an interlocking system where shortcomings in any link can become bottlenecks.

Image Source: Internet

Another noteworthy issue is that generative models used to create long-tail scenarios are most prone to defects precisely in those long-tail and out-of-distribution regions.

The rarer the scenario, the more likely the generated content is to exhibit physical inconsistencies and target drift.

The industry generally believes that relying solely on data loops cannot solve all problems, with the main bottleneck being the convergence of long-tail scenarios.

World models, while tools for addressing data scarcity, also face the issue of data scarcity themselves.

This contradiction has no perfect solution in the short term and is likely to become a key factor limiting the effectiveness of data engines.

#AutonomousDriving #DataAnnotation #CloudDataEngine