Long-time readers who have seen our previous article 'Full Text and Detailed Explanation of Tesla's CVPR 2026 Presentation: Turning Autonomous Driving into a 'Foundational Model for All Robots'' can grasp the overall logic of Tesla's FSD.

However, what is the exact structure of Tesla's FSD algorithm? How does the FSD end-to-end system escape the black box? How is it explainable and reasoned? What is Tesla's stance on VLA? How much computing power does Tesla use and require to train its models? Ashok Elluswamy did not reveal these details in that presentation.

However, Vehicle found a presentation by their Head of Engineering, Phil Duan (Tesla), titled 'Self-Driving at Scale with Foundation Models' from the CVPR 2026 DriveX Workshop. In this presentation, we found answers to the above questions. After all, Phil Duan is the Head of Engineering responsible for development and implementation. Ashok Elluswamy, perhaps having been a leader for too long, tends to focus more on macro-level information.

Therefore, this article shares and explains the answers to the above questions based on Phil Duan's (Tesla) presentation titled 'Self-Driving at Scale with Foundation Models,' combined with his PPT.

End-to-End: A Single Function

Phil Duan began his presentation with a lot of background on FSD safety and why rule-based modular algorithms don't work, which long-time Vehicle readers are already familiar with and won't be shared here. New readers can refer to our previous articles.

The core formula of the entire presentation, and the algorithmic narrative shared by Tesla from Elon Musk to Ashok:

Tesla has reformulated autonomous driving from 'a patchwork of engineering modules' into 'a next-action prediction problem within a video-based foundational model'—using the same methodology as training GPT, but replacing 'the next word' with 'the next steering wheel, accelerator, and brake actions.'

Its correspondence with LLMs is very straightforward: input to output, end-to-end.

However, unlike LLMs, the physical AI closed loop in the real world is crucial.

However, the physical AI model doesn't just 'predict once and done'; it continuously 'sees, acts, observes world changes, and sees again' at extremely high frame rates. Moreover, the real world is far more brutal and complex than the text world, requiring consideration of latency.

Two Major Issues: Algorithmic Black Boxes and Deception

1. The Curse of Dimensionality (Severe Underconstraint)

Input Side: Enormously large, with 8 cameras × 36 Hz × high resolution. If following standard practices (slicing images into patches and then tokenizing), the number of tokens would be 'billions of billions.'

Phil Duan provided a more understandable reference in his presentation: even for a very short video clip, the number of tokens is more than 2,000 times that of 'CLIP (Contrastive Language–Image Pretraining) processing a single 224×224×3 image.'""Science Popularization Insert: When CLIP processes a 224×224 image, it slices it into 16×16 patches, resulting in 196 tokens. Multiplying by 2,000 gives approximately 400,000 tokens. And this is just for a 'very short clip.' The complete driving context is far more extensive. This is why the presentation states that 'extremely complex methods must be invented to compress inputs into a manageable context length'—video tokenization/spatiotemporal compression is one of Tesla's core technological assets.

Output Side: Extremely small, with only 2 degrees of freedom: steering angle + acceleration/deceleration.

With abundant data, you're tasked with deriving motion in just two dimensions while preventing collisions—this is Massively underconstrained.

Phil Duan placed four images at the bottom of his presentation PPT as examples: nighttime highway, snowy terrain, dense fog, and daytime multi-lane roads. These four visually distinct worlds yield identical AI labels: .

This is catastrophic for the algorithm. In supervised learning terms: your input entropy is extremely high, while your output entropy is extremely low.

The model's gradients can only backpropagate through 'the error of two numbers' to a massive encoder tasked with understanding the entire world. At this point, the model may discover a shortcut: 'No matter what I see, I'll just go straight; it's usually correct.' This is classic shortcut learning, where the model achieves low loss but learns nothing.

2. Causal Relevance (Eliminating Spurious Correlations)

Phil Duan displayed an image of an intersection in his presentation: humans focus only on the 'left-turn green arrow,' but neural networks see a plethora of signs, a background full of green lights, and even a 'No Entry' sign in the distance.

With only two output numbers as supervisory signals, how does the model know to 'look at the left-turn arrow, not the background green light'? It could easily learn the spurious shortcut that 'if there's green in the background, I can go.' Such spurious correlations work 99% of the time but become fatal in the remaining 1%.

How can we determine the causality between environmental inputs and motion control?

How does Tesla's algorithm address the complexity of physical AI?

Tesla's Core Algorithmic Solution: Multi-task Co-training with Proxy Tasks

This might be the least understood part of Tesla's algorithm. Having read too many PR articles, people often perceive FSD end-to-end as an unexplainable large model.

In reality, this is the most 'counterintuitive yet crucial' step in Tesla's approach. They didn't discard the legacy of the modular era but downgraded it into auxiliary signals for training.

Phil Duan listed 16 categories of proxy tasks in his presentation PPT: 3D geometry, 2D/3D object detection, object tracking, semantic segmentation, instance segmentation, dense depth, scene flow and optical flow, OCR and sign understanding, image classification, scene understanding, visual question answering, localization and mapping, future frame prediction, image reconstruction (+ panoptic segmentation, 3D occupancy, human mesh, keypoint tracking, and text recognition in the architecture diagram).

Remember these algorithmic terms and stay tuned to Vehicle for future explanations. These auxiliary signals constitute Tesla's three-tiered understanding of the algorithm:

1. They help gradients flow through the network. The gradient from two numbers is too sparse to support a massive visual encoder. However, tasks like 'which object does this pixel belong to,' 'how far is this point from you,' and 'what does this sign say' provide dense supervisory signals per pixel and per frame, with gradients several orders of magnitude higher.

2. They force the model to form correct representations. If the encoder's features must simultaneously support 'predicting 3D occupancy' and 'recognizing sign text,' it cannot learn shortcuts like 'go straight if the image is bright'—because such features would fail at segmentation and OCR tasks. Proxy tasks act as regularizers preventing shortcut learning.

3. This is how 'causality' is injected into the model. Only by enabling the model to read 'left-turn green arrow' (OCR + sign understanding) can it establish the correct causality that 'green arrow means left turn,' rather than 'if there's green in the background, go.'""Even more ingeniously, Phil Duan stated that the metrics for these auxiliary proxy tasks are not the goal. Tesla doesn't aim to maximize each proxy task's metrics, as that might not improve the final model. They only need these metrics to be 'healthy enough.' The only metric Tesla truly measures is for the deployed task: driving.

This logic directly overturns the evaluation logic of academic autonomous driving benchmarks (e.g., maximizing nuScenes detection mAP or occupancy mIoU). In Tesla's framework, these metrics are training tools, not product goals.

However, virtually all domestic end-to-end algorithms currently use this methodology. The difference lies in how well they apply it, whether their algorithmic modules are well-combined, and whether their end-to-end training is aligned.

Having dissected the algorithm, what is the next moat?

The Moat: Data Engine

Phil Duan showcased Tesla's data metrics in his presentation. Currently, Tesla's fleet of approximately 7 million vehicles generates about 500 years of driving duration daily. Typical datasets used daily exceed 120 years of driving duration and hundreds of petabytes, with an internal dashboard peak of 105 million video clips/148 years of duration. FSD's cumulative mileage has just surpassed 10 billion miles.

""In Tesla's datasets, the concept of 'edge cases' is dissolved by scale. Everyone in the industry talks about edge cases and builds simulators to synthesize rare scenarios. But when you have 7 million vehicles, any edge case becomes a daily occurrence. Scenarios a single human driver might never see in a lifetime are collected by Tesla daily.

""In his presentation, Phil Duan noted that if video data is randomly sampled, 90% of it is entirely useless, such as a highway driving video.

Thus, more data isn't always better; diversity is key. How does Tesla extract the desired data from its vast ocean?

Tesla follows two criteria for data extraction: genuinely interesting clips (i.e., typical and rare scenarios) and clips where the model's output disagrees with human driver actions.

Clips with model-human disagreement essentially represent disagreement mining/hard example mining.

The logic is: training on areas where the model and humans drive similarly yields no information gain; only areas where 'the model wants to drive this way, but the human actually drove that way' carry knowledge the model hasn't yet mastered.

Thus, Tesla effectively uses its entire fleet in an unprecedented active learning loop. Every customer vehicle running FSD on the road automatically annotates 'where I still fall short' for the next model iteration.

Many ask: since these vehicles only have cameras (no LiDAR), where does the Ground Truth for What is needed year-round (perennially needed) 3D occupancy and 3D detection come from?

Phil Duan revealed in his presentation that the answer lies in offline auto-labeling. Tesla uses a cloud-based large model without computing power constraints to run inference on collected videos, generating 3D ground truth to supervise onboard models. This is Tesla's cloud-based 'teacher model teaching the camera-end model'—knowledge distillation + offline reconstruction, the key engineering solution enabling pure vision approaches to obtain 3D supervision.

How can this theory be generalized and applied to robots?

How to Generalize: Internet Data and Optimus Data

During his speech, Phil Duan reaffirmed that Tesla will leverage knowledge data from the internet, such as videos, along with its proprietary data and reasoning capabilities, to support generalization.

In his speech, Phil Duan used the example of a "chair on the highway" to perfectly illustrate the issue: If a chair appears on the highway, the vehicle must avoid it, but you can't wait for the fleet to collect enough samples of "chairs on the highway" before the model learns to recognize them.

However, in any internet video or indoor data from Optimus labs, chairs are abundant. As long as the model knows what a chair is, it can recognize it on the highway.

During the speech, someone asked about Tesla's opinion on the current VLA (if you don't know what VLA is, click on our previous article "A Comprehensive Guide to Visual Language Action Models (VLA) and Their Applications").

Phil Duan replied, "Language is a means to an end." Language is one (but not the only) carrier for acquiring knowledge of the world and reasoning capabilities. The core idea: "What truly matters is not language, but knowledge of the physical world + reasoning capabilities."

How to Ensure Safety: Reinforcement Learning with Verifiable Rewards

During the Q&A session of the speech, Phil Duan shared that Tesla's end-to-end model relies on imitation learning and reinforcement learning (RL) to learn "how to drive normally" from human driving data and "what absolutely must not be done" using verifiable rewards, thereby preventing accidents.

"Verifiable rewards" are currently the hottest concept in post-training large models (RLVR, Reinforcement Learning from Verifiable Rewards). The reason RL works well in mathematics and coding is that the correctness of answers can be automatically determined by programs without human scoring. "Whether there is a collision" happens to be such a perfect verifiable signal: In simulations or replays, collisions are geometrically determinable, unambiguous, and automatically computable. Therefore, it can be used on a large scale as an RL reward function at no cost. This explains why Phil said "bake in is very straightforward."

Nowadays, many assisted driving/autonomous driving companies add a rule-based safety monitoring layer (trajectory collision verification, multi-plan switching, fallback to conservative strategies) after the neural network output, using this "safety fallback layer" post-processing to ensure the safety of the model's output.

Phil Duan's response during the speech was: Tesla does not implement such a fallback layer. "During testing/deployment, we basically rely on the model itself to handle this."

This is consistent with Tesla's architecture. If you add a rule-based fallback layer at the output, you reintroduce "human-written rules," reverting to the wrong side of the bitter lesson and undermining the advantages of "homogeneous computing power + deterministic latency."

Computing Power: 300,000 Cards

With algorithms and data, the next question is computing power. In this speech, Phil Duan officially announced the development and current status of Tesla's computing power. Measured in H100 equivalents for Tesla AI training computing power:

2021 Mid – Early 2023: Nearly zero

Late 2023: Stepwise increase begins

Mid-2024: Approximately 90,000

Full-year 2025: Approximately 100,000 – 120,000

March 2026: Approximately 140,000

June 2026: Approximately 280,000 (vertical leap)

It seems that Tesla's computing power cards will double in Q2 2026, indicating a large-scale cluster delivery and joining the wave of computing power investment in the AI era.

Summary: Tesla's Algorithm

Ashok talked about where Tesla wants to go, while Phil Duan explained how Tesla will get there. The former is about vision, and the latter is about engineering. For those in the industry, the latter is more useful.

The most counterintuitive part of Phil Duan's speech is that it brings back what the industry has abandoned over the past decade.

Perception, segmentation, detection, depth, optical flow, OCR—these old tasks from the modular era are not dead. They have simply transformed from "the skeleton of the system" to "the scaffolding for training." Once the building is complete, the scaffolding is removed, but the shape of the building is determined by the scaffolding.

This is also why the most widespread belief that "end-to-end = black box" is actually wrong. FSD has numerous intermediate representations that humans can fully understand, but they are no longer exported in the vehicle. Not exporting them does not mean they do not exist.

Finally, a question for everyone: If surrogate tasks are the key to preventing the collapse of end-to-end systems, then is the gap between domestic players and Tesla primarily due to differences in model scale and computing power, or is it due to whether these dozen or so heads are selected and trained correctly and uniformly? Feel free to discuss in the comments section.

References and Images

Speech content from CVPR 2026 Self-Driving at Scale with Foundation Models - Phil Duan (Tesla),

*Reproduction and excerpting are strictly prohibited without permission.