Recently, the results of the CVPR 2024 Autonomous Grand Challenge were announced, with NVIDIA emerging as the winner in the 'End-to-End Autonomous Driving' category, beating over 400 participants globally with its Hydra-MDP model.

This international autonomous driving challenge, organized by the Shanghai AI Laboratory and top universities, research institutions, and enterprises at home and abroad, is the largest and highest-level autonomous driving competition in the world, relying on the International Conference on Computer Vision and Pattern Recognition (CVPR). The winning teams represent the highest level of research and development in autonomous driving in today's world.

It is understood that NVIDIA's Hydra-MDP model also won the Innovation Award at this year's CVPR, reflecting the recognition of international research institutions for NVIDIA's leading R&D capabilities in the development of autonomous vehicles.

NVIDIA's steady growth in the automotive business is attributed to the promotion of its autonomous driving solutions. NVIDIA DRIVE Orin has a very high market share in the mid-to-high-end smart car segment.

According to incomplete statistics, major automotive companies adopting DRIVE Orin include BYD, NIO, XPeng, SAIC, Feifan, Geely, Lotus, Chery, Great Wall, and over 30 other mainstream manufacturers. Even Xiaomi's popular SU7 series is fully equipped with the DRIVE Orin series.

Moreover, automakers such as BYD, GAC, XPeng, Chery, Geely, and Changan have chosen NVIDIA's latest generation of NVIDIA DRIVE Thor platform based on the Blackwell GPU architecture to provide computing power support for their next-generation autonomous driving systems.

Beyond autonomous driving computing chips, NVIDIA is also assisting car manufacturers and partners in achieving digital upgrades across the entire value chain through AI and accelerated computing solutions for the automotive industry, bringing innovation to areas such as automotive research and development, design, manufacturing, and marketing.

This article explores new opportunities and values of AI in enabling the digitization of the automotive industry, taking cooperation cases between NVIDIA and the four major manufacturers - BYD Electronics, XPeng, NIO, and Lixiang - as examples.

NVIDIA Isaac + Omniverse, Creating BYD's Digital Factory

In the current automotive industry context, the concept of a virtual factory has emerged.

As a physically accurate digital twin of a real factory, a virtual factory allows manufacturers to model, simulate, analyze, and optimize production processes, resources, and operations without physical prototypes.

This powerful simulation capability brings manufacturers advantages in areas such as simplified communication, tailored planning, optimized facility design, and intelligent operation.

Let's specifically examine BYD's cooperation with NVIDIA - extending from the vehicle to the cloud. From the next-generation DRIVE Thor on-board chip to cloud-based AI development and training for autonomous vehicles, and collaborating with BYD Electronics on smart factory initiatives.

With NVIDIA Isaac and NVIDIA Omniverse, BYD Electronics is developing autonomous mobile robots (AMR) and virtual factory planning, including using Omniverse for simulation of autonomous machines and digitally constructing, simulating, and operating factory digital twins, thus enhancing production intelligence.

NVIDIA Isaac is NVIDIA's robotics platform for robotics and visual AI, including a complete set of acceleration systems, libraries, application frameworks, and generative AI models.

Isaac Sim benefits from Omniverse's OpenUSD interoperability across 3D and simulation tools, enabling developers to easily design, import, build, and share robot models and virtual training environments.

The adoption of NVIDIA Isaac and NVIDIA Omniverse technology platforms can help BYD Electronics gain more wings in the process of building intelligent factories.

For a large factory like BYD with an annual sales of up to 3 million vehicles, improving production efficiency and logistics deployment is crucial. Omniverse virtual reality technology optimizes the workflow of automotive manufacturing factories, using NVIDIA Isaac Sim on Omniverse and AMR's high-precision simulation to achieve fleet unified scheduling, management, and solution testing, saving factory planning costs and improving AMR implementation efficiency.

NVIDIA's technology has armored BYD's intelligent factory efficiency, saved factory planning costs, and accelerated the achievement of its digital transformation goals.

NVIDIA Omniverse， Initiate Xiaopeng Automotive Collaborative Design Innovation

In the traditional automotive design process, there are common challenges such as team collaboration difficulties, poor communication, low review efficiency, and high physical prototype costs. These issues not only slow down the speed of product launch, but also affect design quality.

NVIDIA's technology provides an answer in improving design process efficiency and collaboration. Xiaopeng Motors used NVIDIA Omniverse to streamline the design process of its ultra intelligent seven seater flagship model, the Xiaopeng X9, achieving optimization and innovation from product design to team collaboration.

NVIDIA Omniverse is a platform that includes APIs, services, and software development kits (SDKs). Based on OpenUSD, applications running on Omniverse fundamentally change complex 3D workflows, enabling individuals and teams to build unified tools and data workflows, and simulate physically accurate virtual worlds.

By using NVIDIA Omniverse, Xiaopeng Motors can create realistic, physically accurate, and real-time rendering visual assets of vehicle exterior and interior, and enable design and engineering teams to test and adjust products in a virtual environment, significantly accelerating the automotive design workflow and improving work efficiency, thereby saving a lot of time and money costs.

NVIDIA AI Enterprise + Instant NeRF， Accelerate the Implementation of Ideal Automobile Large Model Applications

The main battlefield of new energy vehicles now is intelligence.

With the opportunity to get on the big model, Ideal quietly turned intelligence into its own strength.

With the development of intelligent driving technology, Ideal Automobile has begun to explore end-to-end models, which achieve modeling throughout the entire process from perception to tracking, prediction, decision-making, and planning. Tesla's V12 version represents the end-to-end model, achieving complete modeling and end-to-end virtuality.

However, Ideal Automobile believes that even with data-driven and end-to-end models, achieving L4 level autonomous driving still faces challenges, as L4 level requires no manual supervision in any unknown scenario.

Therefore, Ideal Automotive has proposed a new paradigm driven by knowledge, utilizing large language models and multimodal visual language models to enhance vehicles' understanding of the world and their ability to respond to unknown scenarios.

The development framework for autonomous driving in Ideal Cars is based on the concept of fast and slow systems, similar to human thinking patterns. The fast system (System1) is responsible for intuitive reactions, while the slow system (System2) is responsible for logical thinking and handling complex scenarios. Both the L3 level autonomous driving of Ideal Automobile and the future L4 level autonomous driving will be developed based on this framework.

At present, Ideal Automobile has made progress in the development of end-to-end models, especially in the 3D perception, dynamic perception, static perception, and tracking modeling of BEV (camera based perception). The ADMax3.0 system of Ideal Automobile has achieved industry-leading full scenario NOA, including high-speed and urban driving, as well as handling complex scenarios such as left and right turns, braking, and start stop at traffic light intersections.

During this process, Ideal Automotive and NVIDIA collaborated extensively and prominently on accelerating the deployment and inference of the entire model through NVIDIA AI Enterprise, elevating the visual workflow from processing 9 frames per second to processing 21 frames per second.

NVIDIA AI Enterprise is an end-to-end cloud native software platform that includes SDKs and platforms such as DALI, CV-CUDA, TensorRT, and Triton. It provides enterprise level software and support for NVIDIA AI platforms, helping automotive industry partners accelerate development through generative AI. The TensorRT-LLM framework provides efficient inference capabilities for large models and supports running complex neural network models on the DRIVE Orin platform.

In addition to reasoning acceleration, for automotive companies, the reuse and adaptation of data between vehicle models is also a major challenge. Therefore, it is particularly important to link the entire development process with a closed loop of data and data reuse.

NVIDIA Instant NeRF is a brand new 3D rendering technology, named the Best Invention of 2022 by Time magazine in the United States. This technology can be used to train robots and autonomous vehicle. Developers can transform groups of static images into 3D digital scenes in just a few seconds, and modify and build on this basis.

With the help of NVIDIA Instant NeRF, Ideal Automobile is able to optimize the use and generation of data on the system, such as reconstructing and dynamically editing the data of the Ideal L9 model, effectively utilizing historical data, thereby improving the efficiency of data processing and the generalization ability of model training.

In the field of intelligent cockpit, the Ideal Automotive AI Assistant "Ideal Classmate" constructed by the multimodal cognitive model Mind GPT integrates multiple perceptual information such as voice, vision, and touch, which can fully understand language and user needs, and provide valuable responses.

During this process, the powerful computing power of NVIDIA GPU supports the training and inference of large models, enabling "ideal classmates" to quickly and accurately process and respond to user instructions.

At present, NVIDIA is actively embracing big models and promoting technological innovation in intelligent driving and intelligent cabins through a comprehensive AI acceleration solution from cloud to vehicle.

Deep learning+autonomous driving, solving the problem of training NIO perception models

With the continuous development of deep learning models, the scale and complexity of models are also increasing, which puts higher demands on computing resources and memory. To solve this problem, tensor parallel technology has emerged.

The advantages of Tensor Parallel technology are improving computational efficiency, reducing memory consumption, and improving model performance. This precisely corresponds to NIO's challenge - with the strong computing power of the central computing platform ADAM, NIO's autonomous driving technology is rapidly developing. Due to the use of multiple camera inputs and deep convolutional backbone networks, the GPU memory used to train autonomous driving perception models is very large.

picture

Nvidia and Weilai have jointly designed and implemented a tensor parallel CNN training scheme, which allocates computing tasks of deep learning models among multiple GPUs, thus significantly reducing the memory occupation and bandwidth pressure on a single GPU, achieving faster computing speed and higher computing efficiency, and ultimately helping to improve the training efficiency and GPU utilization of the autonomous vehicle perception model.

The benchmark test shows that this method performs well in the NIO autonomous driving scenario and effectively solves the training problem of visual large models.

Prior to this, NIO also chose NVIDIA Triton inference server to build high-performance inference services for NIO's Autonomous Driving Development Platform (NADP). Triton makes model orchestration and expansion easier, increases inference speed by 6 times, and saves 24% of resources.

From Nvidia's cooperation with the four major manufacturers, the automotive industry is showing several major trends

Firstly, fully autonomous driving is moving from AI 1.0 to AI 2.0. In the past, software defined cars, known as the AI 1.0 era, required a large number of engineers to intervene.

In the era of AI 2.0, which is what we call AI defined cars, most models can be trained in the cloud, that is, through data-driven methods. Generative AI and accelerated computing will play an increasingly important role in the next generation of autonomous vehicle.

Secondly, BYD Electronics' virtual factory is a powerful practice of industrial Internet from single point empowerment to system empowerment. AI robot technology empowering factory is accelerating the era of manufacturing automation and industrial efficiency.

In the production workshop of BYD Electronics, autonomous mobile robots have been helping to improve the efficiency of automobile manufacturing. By using NVIDIA digital twin technology to replicate physical factories, precise physical world scale simulations can be created for existing robot factories, making production, warehouse logistics, and distribution centers more efficient, guiding us to elevate automation and industrial efficiency to new levels.

Thirdly, the intelligence of automobiles is entering a stage close to the advanced level of human thinking.

Based on the NVIDIA AI Enterprise software platform and GPU computing power support, Ideal Automotive is exploring end-to-end models, which achieve modeling throughout the entire process from perception to tracking, prediction, decision-making, and planning. The development framework for autonomous driving is based on the concept of fast and slow systems, similar to human thinking patterns.

Intelligent driving is not as intelligent as others, and it is a consensus that complex driving scenarios require human intervention. From an ideal case study, through end-to-end and comprehensive AI acceleration solutions, intelligent cars can already handle complex scenarios such as high-speed and urban driving, as well as turning left and right, braking, and starting and stopping at traffic lights. The intelligentization of automobiles is entering a new high-level stage.

Fourthly, the intelligentization of automobile manufacturers from production to use. Based on the above four cases, it can be seen that AI is integrating into multiple aspects such as design, engineering, production, and sales, and the intelligentization of automobiles has entered the era of AI Industry 2.0.

The cooperation between BYD Electronics, NIO, Xiaopeng, Ideal, and NVIDIA can be seen as a signal of the digital transformation of the automotive industry, as automotive manufacturing is achieving a full workflow digital upgrade.

In the era of AI, technology serves people and drives the transformation of the times, allowing us to see the arrival of the next era.