When Hello is mentioned, shared bicycles that dot the urban landscape spring to mind for many. However, the company has already made significant strides into the Robotaxi arena.

Recently, Hello Robotaxi unveiled its groundbreaking Hello World generative driving world model.

One of the standout features of this innovation is that researchers now have the capability to actively modify vehicle trajectories, traffic participant configurations, and weather conditions. This allows the model to produce corresponding multi-view driving observations and endeavors to link the generated environment with driving strategies.

For autonomous driving research and development (R&D), this offers a technical pathway distinct from fixed data replay. Technicians can not only review past driving processes but also explore how the same scenario evolves under varying trajectories and traffic conditions.

Of course, the ability to generate scenes is not entirely novel.

What sets Hello World apart is its ambitious attempt to consolidate scene control, seven-camera synchronization generation, LiDAR point cloud generation, and driving strategy interaction into a unified system.

To what extent have these functionalities been realized, and what challenges remain to be addressed? Let's delve deeper today.

01 World Models: Beyond Video Generation, Towards Scene Control

Real-world road data captures driving processes that have already transpired.

Whether utilized for model training or testing, researchers can sift through this data to locate specific scenarios. However, if modifications to vehicle trajectories or traffic participant layouts in the original recordings are required, additional data construction or simulation techniques become necessary.

Hello World endeavors to provide a generative solution.

According to its public introduction, researchers can describe target scenes using natural language or directly edit vehicles, pedestrians, and their motion trajectories.

For instance, they can instruct a normally driving vehicle ahead to decelerate and come to a halt within the lane or adjust the position and movement of pedestrians to simulate a road-crossing scenario. The model then generates corresponding driving observations based on these inputs.

Weather and lighting conditions can also serve as control variables.

Day-night transitions, environmental factors such as rain, snow, and fog, as well as scenarios like construction zones and dense traffic, can all guide the generation process.

Image Source: Paper

Such capabilities ensure that data construction is more closely aligned with specific R&D objectives.

If a team is investigating driving strategies for sudden stops by the vehicle ahead, they can construct diverse cases centered around the leading vehicle's trajectory, rather than solely relying on the volume of similar data collected from real roads.

However, it is crucial to distinguish between two concepts.

The ability to generate specified scenes does not equate to the creation of true and reliable test samples.

The presence of pedestrians in the scene does not guarantee that their movements will adhere to real traffic patterns; a vehicle stopping along a specified trajectory does not imply that the entire scene satisfies complete vehicle dynamics and traffic constraints.

Therefore, in terms of scene generation, what Hello World currently showcases is a controllable data generation capability.

Whether the generated data can effectively supplement long-tail scenarios and enhance downstream driving models remains to be validated through subsequent training and evaluation.

02 Seven-Camera Synchronization Generation: Beyond Realistic Visuals

For driving world models, generating visuals is merely the foundation.

Following a vehicle's alteration in motion trajectory, the system must also produce road and traffic observations that align with the new perspective. The same vehicle, as seen by different cameras, should maintain reasonable positional, appearance, and motion relationships.

Hello World employs progressive training to gradually transform visual and motion knowledge from general videos into controllable driving generation capabilities.

The model, boasting approximately 2 billion parameters, gradually incorporates conditions such as ego vehicle pose, camera calibration, high-definition maps, and 3D bounding boxes during training.

Overview of the Hello World Algorithm. Image Source: Paper

It is essential to differentiate between two types of control information here.

Ego vehicle pose is utilized to control the vehicle's spatial position and orientation changes, based on which the model generates corresponding driving observations. High-definition maps and 3D bounding boxes constrain the spatial layout of road structures and traffic participants.

These two types of conditions collaborate to enable the model to adjust the ego vehicle's route and the surrounding scene independently.

For instance, if the vehicle is required to transition from driving straight to turning left, the model must generate observations corresponding to the new trajectory, rather than merely shifting the road scene in the original video to the left.

For a seven-camera system, each perspective must also maintain geometric correspondence as closely as possible.

In multi-view evaluations conducted by Hello, the model was compared with Cosmos Transfer 2.5.

Under settings where both models were provided with the first frame, the trajectory rotation error in generated videos decreased from 5.534° to 2.027°, representing a 63.4% reduction. The cross-camera geometric error decreased from 6.549 pixels to 4.180 pixels, marking a 36.2% reduction.

This indicates that, under the evaluation conditions, Hello World demonstrated improvement in responding to specified trajectories and maintaining cross-camera geometric correspondence.

However, trajectory rotation error measures the estimated motion results from generated observations, not the actual vehicle's control error. Cross-camera geometric metrics do not represent complete 3D geometric accuracy.

In addition to visual generation, Hello World also supports the generation of corresponding LiDAR point clouds based on camera RGB observations and processes them for valid echoes, distance information, and point cloud structure.

This endows it with the additional capability of conditional LiDAR point cloud generation, beyond multi-view video generation, offering new avenues for multi-sensor data construction.

However, whether the visuals and point clouds accurately correspond, whether the generated point clouds conform to the distance information of real sensors, and whether the echo characteristics and point cloud structure are precise still require separate verification.

Simply because the system outputs both images and point clouds simultaneously does not mean it has fully simulated the sensor system of a real vehicle.

03 From Scene Generation to Strategy Interaction: The Path to Reliable Simulation

Hello World also endeavors to link the generated environment with driving strategies.

In public demonstrations, the driving strategy Alpamayo 1.5 outputs the ego vehicle's trajectory based on current observations. Hello World then generates subsequent scenes based on the trajectory and previously generated observations for the strategy to continue making judgments.

Compared to fixed trajectory replay, this approach can generate subsequent observations corresponding to new trajectories provided by the strategy.

Researchers thus have the opportunity to investigate how the same scenario evolves under different driving choices, without being entirely confined to the single path in the original recordings.

However, the closed loop here refers to the interaction process between the driving strategy and generated observations, not that the system has established a fully validated vehicle dynamics simulation environment.

During continuous generation, the world model continually reads its own previously generated history.

Early generation errors may influence subsequent observations, and the new trajectories chosen by the strategy may also cause the scene to gradually deviate from the scope covered by real recordings.

To mitigate this issue, Hello World employs block causal generation, enabling the model to generate subsequent observations segment by segment. Through training adaptation with perturbed history and its own generated history, it enhances the model's ability to handle imperfect contexts during continuous operation.

The model also utilizes few-step distillation to compress a 20-step teacher model into a 4-step student model, reducing the computational cost of repeated generation.

Overview of the Generation Framework. Image Source: Paper

In efficiency tests conducted, the model utilized 8 GPUs to generate content at 720p resolution, with seven cameras, 61 frames, and the first frame provided.

The sampling time decreased from 357.7 seconds to 77.8 seconds, representing a 4.60x speedup. The end-to-end time, including video writing and other steps, decreased from 426.7 seconds to 143.2 seconds, marking a 2.98x speedup.

These results demonstrate that reducing generation steps can lower inference overhead under specific configurations, but minute-level end-to-end time does not represent real-time generation.

It also does not, in isolation, prove that the model can stably simulate long-duration driving processes.

More critically, strategy performance in the generated environment cannot be directly equated with driving ability on real roads.

Vehicle dynamics, the responses of traffic participants to the ego vehicle's actions, and error accumulation during long-term operation all necessitate further investigation.

Whether the generated environment can be utilized for reliable safety evaluation also requires systematic verification.

04 Closing Remarks

Hello World represents an ambitious attempt to integrate controllable scene generation, seven-camera synchronization generation, LiDAR point cloud generation, and driving strategy interaction into a single system.

This propels world models from merely generating driving visuals to constructing data and facilitating interactive simulation for autonomous driving R&D.

However, generative capabilities do not equate to reliable testing capabilities.

Image Source: Internet

Whether generated data can enhance downstream models, whether strategy interaction can remain stable during long-term operation, and whether simulation results can effectively reflect real driving performance still require more comprehensive evaluation.

For the autonomous driving industry, the value of world models lies not merely in generating more scenarios but in whether these scenarios can be transformed into repeatable, evaluable, and genuinely useful test resources for R&D.