When a vehicle is in operation on roads, a multitude of factors can come into play. Vehicle vibrations, prolonged shaking, rough road surfaces, load changes, temperature variations, and even maintenance procedures, collisions, or the reassembly of sensor brackets can all lead to minor physical deviations or alterations in the attitude/height of perception sensors (such as cameras, LiDAR, and millimeter-wave radar). If these deviations are not addressed promptly, they can disrupt the previously established spatial relationships (i.e., the "calibration") of the sensors. This, in turn, can result in errors in perception, localization, data fusion, and decision-making processes.

Why Do Sensors Deviate?

Autonomous vehicles, when operating on roads, especially over extended periods, are prone to various vibrations, impacts, and shocks. These can occur when the vehicle passes over speed bumps, rough roads, or uneven surfaces. Additionally, driving over potholes, gravel, or through potholes, as well as braking, carrying loads, or taking turns, can induce slight deformations or vibrations in the vehicle body. Over time, perception sensors like cameras, LiDAR, and millimeter-wave radar, which are mounted on the vehicle, may experience "positional deviations" or "attitude changes" (including changes in orientation, tilt, and height). These changes can be attributed to loose brackets, adhesive fatigue, slight vehicle deformations, or even loose screws.

Furthermore, activities such as replacing components, changing tires, repairing wheel rims, conducting vehicle inspections, or performing collision repairs can also impact the sensor's mounting rigidity. This, in turn, can alter its original mounting attitude.

These physical deviations, although seemingly minor, measuring only a few millimeters or degrees, can have profound effects on autonomous driving systems. Autonomous driving systems impose stringent requirements on the installation position, orientation, and attitude of sensors. This is because all sensor data must ultimately be transformed into a common vehicle coordinate system or world coordinate system. Only by placing this data within the same "spatial reference frame" can the system effectively fuse information from different sources, such as cameras, LiDAR, and millimeter-wave radar. This fusion enables the system to accurately perceive the surrounding environment and make informed decisions regarding path planning and vehicle control. The process of precisely aligning each sensor's relationship with the vehicle coordinate system is commonly referred to as "calibration."

Once a sensor deviates from its calibrated position, the original calibration relationship becomes inaccurate. This is akin to a camera or radar being slightly "twisted." When the system utilizes this inaccurate data to interpret the external environment, deviations occur. These deviations can lead to inaccurate judgments regarding object positions, distances, directions, and even speeds. In turn, inaccurate perception affects decision-making and control, thereby compromising safety.

Therefore, in the practical deployment of autonomous driving, such deviations cannot be overlooked, and appropriate countermeasures must be devised.

How to Prevent and Detect "Deviations"?

1) Strict Initial Calibration (Offline Calibration) Is Crucial

When perception sensors are initially installed on a vehicle, a rigorous calibration process is essential. This calibration process determines both the "intrinsic parameters" of each sensor (such as camera focal length, distortion, lens model, as well as LiDAR and radar's internal reference parameters) and the "extrinsic parameters" (i.e., the sensor's position and attitude relative to the vehicle coordinate system, encompassing translation, rotation, orientation, tilt, and height).

This step establishes the baseline for the system's "worldview." If the calibration is sufficiently precise, with errors controlled within the millimeter and angular ranges, the autonomous driving system can function normally during the perception, fusion, localization, and control stages.

However, initial calibration only guarantees accuracy at the moment of installation or for a short period thereafter. Long-term use, vibrations, maintenance activities, or environmental changes can induce minor sensor deviations. Thus, relying solely on initial calibration is insufficient to ensure long-term system stability.

2) Regular/Executive Maintenance Is Imperative

For mass-produced vehicles, manufacturers or maintenance service points may recommend (or even require) recalibration of autonomous driving sensors (cameras, radar, LiDAR) after significant maintenance, collisions, tire replacements, suspension repairs, deformation inspections, or body checks. This recalibration ensures that the vehicle's structure or brackets are restored to symmetry, free of looseness, and that the sensor's mounting attitude is corrected. It is akin to the alignment checks performed after wheel alignment or four-wheel alignment. For vehicles equipped with ADAS (Advanced Driver Assistance Systems), such calibration is highly necessary.

However, manual recalibration is time-consuming, labor-intensive, and requires a dedicated facility. It cannot be performed frequently due to these constraints. For autonomous vehicles, especially those in commercial operation, such as Robotaxis or fleets, this method is inflexible, costly, and affects operational efficiency.

Due to these limitations, current autonomous driving systems tend to adopt "online calibration" or "automatic calibration" mechanisms. These mechanisms enable vehicles to automatically detect and correct minor deviations during daily operation without relying on manual intervention each time.

Online/Automatic Calibration & Real-Time Monitoring Offer Self-Correction Capabilities

In recent years, both academia and industry have been exploring ways to enable autonomous driving systems to automatically detect and correct deviations without manual intervention each time. The 2024 paper "Automatic Miscalibration Detection and Correction of LiDAR and Camera Using Motion Cues" proposes a framework for automatically detecting and correcting calibration deviations between LiDAR and cameras.

The fundamental concept is to leverage the vehicle's/sensor's own motion information. During vehicle operation, the system continuously monitors each frame of data. If it detects that the projection constraints between LiDAR point clouds and camera images are no longer satisfied (i.e., their spatial relationship no longer aligns with the original calibration), it infers that a "deviation/drift" (miscalibration/drift) may have occurred. Then, by calculating the alignment transformation between camera motion and LiDAR scans, the LiDAR point clouds are aligned back to the image space, achieving automatic deviation correction.

In simpler terms, as the system operates continuously, navigating through various road conditions, multiple turns, accelerations/decelerations, and vibrations, it can recognize potential misalignments of perception sensors. Using mathematical/geometric methods, it can automatically restore perception to the correct state. This represents a closed-loop self-correction mechanism.

Additionally, open-source toolkits (such as OpenCalib) are available for calibrating/recalibrating multi-sensors (including cameras, LiDAR, IMUs, radar, etc.). These toolkits support automatic, semi-automatic, and manual calibration processes across different usage scenarios.

Therefore, a truly effective autonomous driving system does not end with installation; instead, it must incorporate automatic or semi-automatic calibration mechanisms to regularly/continuously monitor sensor deviations and correct them promptly.

Multi-Sensor Fusion + Fault-Tolerant Redundancy Design: Not Relying on a Single Sensor

Even with automatic calibration, absolute accuracy cannot be guaranteed. Therefore, autonomous driving systems should not rely solely on a single sensor but should adopt a multi-sensor fusion + redundancy design + fault-tolerant mechanisms to enhance robustness against single-sensor deviations/failures.

An autonomous vehicle typically carries multiple perception sensors, including cameras, LiDAR, millimeter-wave radar, inertial measurement units (IMUs), GPS/GNSS/INS, etc. For instance, even if a camera deviates due to a loose bracket or collision, LiDAR or radar + IMU + GPS can still provide environmental perception or localization references. The system can detect inconsistencies between the camera's output and other sensors/inertial/localization systems, temporarily reduce or discard its data, and issue warnings to the driver/operator.

Furthermore, some systems implement time synchronization and clock synchronization (time-synchronization) to ensure that all sensors sample data under a unified clock source. This prevents fusion errors caused by time differences, data delays, or inconsistent sampling across sensors. For multi-sensor fusion, spatiotemporal synchronization is fundamental.

Through multi-sensor fusion + redundancy + fault tolerance + spatiotemporal synchronization + automatic/online calibration/detection mechanisms, autonomous driving systems can maintain overall perception capabilities and ensure safety even when a single sensor temporarily loses accuracy or deviates.

Final Thoughts

Every sensor on a vehicle does not operate in isolation; they must function cohesively as a team to ensure the stable and reliable operation of the autonomous driving system. While physical sensor deviations may seem minor, measuring only a few millimeters or degrees, they can introduce significant errors in the system's comprehension of the environment. Therefore, the true challenge for the autonomous driving industry is not merely installing sensors accurately but maintaining their accuracy over the long term.

Hence, the design philosophy for autonomous driving should shift from "calibrating once during installation" to "sustaining calibration throughout the vehicle's lifecycle." Only systems capable of achieving this can truly enable autonomous driving to progress from being operational to remaining operational over the long term.

-- END --