Recently, BYD's Auto New Technology Research Institute unveiled an academic paper titled HyWorldVLA, marking BYD's inaugural public venture into the realm of foundational models for autonomous driving.

Image Source: BYD's Paper

The paper introduces the concept of a hybrid world model, which integrates both pixel-level and latent space world models.

What exactly are these two types of world models? What roles do they serve? Why did BYD opt to develop a hybrid world model?

01 What is a Pixel-Level World Model?

A pixel-level world model, as its name implies, empowers AI to predict future road conditions frame by frame, akin to playing a movie.

For instance, if a car is changing lanes ahead, the model will precisely forecast the car's position, shape, and trajectory in each subsequent frame.

During a right turn, the model reliably forecasts future lane markings even when they are obscured from the front-view camera's perspective.

Image Source: BYD's Paper

Its role is to furnish autonomous driving with dense spatial and dynamic training constraints.

What does that entail? It means the model must genuinely comprehend the geometric layout of the scene and the interrelationships between object motions to accurately predict the next frame.

This high-precision prediction enables the AI to retain a wealth of detailed information, such as lane line directions, vehicle trajectories, and occlusion relationships, which is vital for safe driving.

However, precisely because it predicts frame by frame, the pixel-level world model harbors significant drawbacks.

The computational cost of a pixel-level world model is exorbitant, as each frame necessitates detailed calculation, consuming substantial computational resources.

Moreover, external disturbances in the real world, such as rain, fog, sensor dirt, or abrupt changes in lighting, can be exacerbated during the prediction process, ultimately leading to grossly inaccurate predictions.

It's akin to wearing foggy glasses to observe the road—the view becomes increasingly indistinct.

02 What is a Latent Space World Model?

To grasp the latent space world model, we must first elucidate what latent space entails.

Simply put, latent space compresses complete video footage, discarding pixel-level details and only abstractly recording information such as what is on the road and how it is moving.

It's akin to condensing a dashcam video into a written note.

The latent space world model makes predictions in this compressed abstract space. It disregards the appearance of each frame; it solely focuses on changes in abstract-level states.

For instance, if a car is approaching from the opposite lane, the latent space model won't precisely depict the car's color and model but will capture information like a car is approaching at a certain speed and direction.

The advantages of the latent space model are twofold.

One is computational efficiency. Since predictions are made in a compact feature space, it demands less effort than in pixel space.

The other is robust anti-interference capability. Because it bypasses specific pixel-level images, visual noise like rain, fog, or dirt has minimal impact.

Visual Interference Diagram. Image Source: BYD's Paper

However, the problems with the latent space model are also apparent.

If trained solely on abstract descriptions without sufficient visual supervision signals, the model may only learn hollow statistical patterns and forfeit critical details for driving, such as the location of lane lines or whether the car ahead is accelerating or decelerating.

After all, purely imagining scenes abstractly is never as vivid as witnessing them firsthand.

03 What Has BYD's Hybrid World Model Achieved?

The pixel-level model offers clarity but is vulnerable to interference and computationally sluggish; the latent space model is swift and resistant to interference but lacks detail.

Each approach harbors its strengths and problems.

Comparison of World Model Paradigms. (a) Pixel-level world model; (b) Latent space world model; (c) Hybrid world model.

Image Source: BYD's Paper

BYD's HyWorldVLA proposes a solution: amalgamate both approaches during pre-training but utilize only one for actual inference.

During pre-training, the pixel-level and latent space world models operate concurrently.

The latent space's abstract analysis aids in stabilizing the pixel-level model, preventing external disturbances like rain or lighting changes from skewing the frame predictions.

Conversely, the pixel-level model's high-precision frames assist in calibrating the latent space model, ensuring that abstract representations do not forfeit critical details.

These two signals—one stringent and one lax, one dense and one sparse—collaborate to train a model that is both informative and not overly sensitive.

During real-world inference, the model employs only the latent space model for predictions. This ensures prediction accuracy (as it has been fine-tuned by the pixel-level model during training) while maintaining operational efficiency (as it relies solely on the lightweight latent space model).

Overall Architecture of HyWorldVLA. Image Source: BYD's Paper

This approach has yielded remarkable results on the autonomous driving public test platform NAVSIM, scoring 90.59 on NAVSIM v1, setting a new public benchmark; it also scored 89.71 on the more comprehensive NAVSIM v2.

The comprehensive metric employed in testing, EPDMS, does not merely evaluate accuracy but assesses whether the vehicle will collide, remain within drivable areas, maintain safe distances, achieve driving goals, and drive smoothly and comfortably.

04 What Is the Significance of BYD's Efforts?

For an extended period, BYD has been stereotyped as prioritizing manufacturing over R&D.

This paper, independently completed by BYD's Auto New Technology Research Institute, substantiates that BYD is not only engaged in AI R&D but also possesses foundational model development capabilities from scratch.

The paper also signifies BYD's key technological preparations for Level 3 autonomous driving.

Shortly before the paper's release, BYD introduced its self-developed 4nm advanced intelligent driving chip, Xuanji A3, boasting a computing power of approximately 700TOPS per chip. From self-developed algorithms to self-developed chips, BYD is constructing a complete closed-loop for intelligent driving technology.

Moreover, world models represent a cutting-edge direction in autonomous driving, enabling AI to transition from merely recognizing roads to comprehending the world.

This is also the direction being explored by top players like Tesla FSD, Waymo, and NVIDIA. BYD's entry at this juncture underscores its determination not to lag in the next phase of the intelligent driving competition.

05 Final Thoughts

Someone mentioned this paper in the comments section of a leading intelligent driving platform, prompting me to invest some time in studying it.

Frankly, this paper has genuinely impressed me about BYD.

In the past, when discussing BYD's intelligence, people predominantly thought of supply chain integration and large-scale deployment.

Image Source: Internet

However, this HyWorldVLA paper, at least at the technical R&D level, demonstrates that BYD can stand at the forefront of autonomous driving technology.

But this is merely a paper; we must maintain composure. There remains a protracted journey between paper achievements and actual mass production.

Scoring high in simulation tests does not guarantee stellar performance on real roads. The complexity and extreme scenarios of real roads cannot be fully replicated by simulation platforms.

That said, BYD possesses a unique advantage that other automakers lack: the vast amount of real-world driving data generated by its enormous sales volume.

For autonomous driving large models, data is the paramount fuel for training world models. Once BYD effectively harnesses its data advantage, it will forge a formidable moat in the intelligent driving arena.

This HyWorldVLA paper is likely just a small fish BYD has released into its technological pond. Whether this fish can metamorphose into a whale remains to be seen.

#AutonomousDriving #HyWorldVLA #BYD