In the realm of pure vision solutions for autonomous driving, monocular cameras are inherently constrained in their environmental perception capabilities due to their inability to directly gauge depth. In response to this limitation, binocular vision technology has emerged. By mimicking the function of human eyes, it leverages the disparity observed between two cameras to estimate distances, transforming two-dimensional images into three-dimensional data and thereby endowing vehicles with crucial depth perception abilities for decision-making.

What Constitutes Binocular Depth Estimation?

Perceiving depth with our eyes is, in essence, a natural form of depth estimation. Our eyes are spaced apart, and the brain integrates the slightly differing images captured by each eye to judge distances. In the field of computer vision, "binocular depth estimation" adopts this principle by positioning two cameras side by side to capture the same scene, then analyzing the discrepancies between the two images to estimate distances.

A two-dimensional image captured by a monocular camera solely contains color and brightness information, lacking the direct provision of distance data for objects within the scene. To acquire the vital depth information of "distance," the crux lies in exploiting disparity—placing a second camera at a distinct location to simultaneously image the same scene. Consequently, objects will exhibit positional offsets in the images from the two perspectives. By computing this offset, the three-dimensional distance of the object can be precisely estimated.

If we ascertain the distance between the two cameras (baseline) and the focal length of the cameras, we can employ a straightforward formula to calculate the true depth of a point once we identify the positional difference (disparity) of corresponding points of the same object in the two images:

Depth = Focal Length × Baseline / Disparity.

From this formula, it is evident that a larger disparity indicates a closer object, whereas a smaller disparity signifies a farther object.

Principal Steps in Binocular Depth Estimation

Now that disparity has been mentioned, a pivotal question arises: How do we locate these corresponding points from a pair of left and right images? This process actually encompasses multiple steps.

When two cameras are manufactured and assembled, there may be inherent positional and angular inaccuracies. Hence, we initially need to conduct geometric calibration to ascertain the internal parameters of each camera (such as focal length and principal point position) and their external relationship (position and orientation). Only then can the pixel positions be accurately aligned when comparing the two images subsequently.

Following the calibration of the binocular system, the next step is stereo rectification. This procedure adjusts both images so that they align on the same horizontal plane. Consequently, the correspondence of the same scene point in the left and right images solely varies horizontally, significantly simplifying the subsequent matching process.

The core task of stereo matching is to identify corresponding pixel points for the same object in the left and right images. Given that stereo rectification has already been executed, this search is considerably streamlined. We merely need to scan along the same horizontal line in the right image corresponding to a pixel point in the left image to locate the most analogous region. Nevertheless, pinpointing corresponding points for every pixel in the image still entails a substantial computational burden. Efficient matching can be achieved through methods such as classic Block Matching or the more efficacious Semi-Global Matching (SGM).

Upon establishing the corresponding relationship for each pixel, we can compute the disparity value. Disparity represents the horizontal coordinate difference of the same point in the left and right images. A larger disparity implies that the point is nearer to the camera. Ultimately, by substituting the disparity value into the aforementioned formula, we can derive the depth value corresponding to each pixel. This method enables us to generate a "depth map" where each pixel encompasses not only color information but also a distance value.

What Is the Role of Deep Learning in Binocular Depth Estimation?

Achieving binocular depth estimation through traditional computer vision methods, as outlined above, is feasible. However, traditional approaches predominantly rely on manually devised features and matching algorithms, such as comparing the similarity of pixel blocks in the left and right images to ascertain if they are corresponding points. This methodology is susceptible to errors in regions with sparse textures or pronounced lighting variations, and the computational load is also substantial.

In recent years, deep learning has been introduced into the domain of binocular depth estimation. Its core objective remains aligned with traditional methods—identifying corresponding relationships between left and right images and computing disparity—but the implementation has undergone a radical transformation. Deep learning no longer hinges on manually crafted matching costs and rules; instead, it employs convolutional neural networks to autonomously learn matching features from data.

The network ingests left and right views as input and directly outputs a disparity map or depth map. Trained on a vast quantity of stereo image data, the network can independently determine which image features are advantageous for matching and which scenes are prone to ambiguity, substantially enhancing matching robustness. Consequently, in scenarios where traditional methods are prone to failure, such as occluded regions, repetitive textures, or textureless environments, deep learning-based methods exhibit superior accuracy and stability.

The processing flow of deep learning methods entails utilizing a neural network to extract features from the left and right images, then constructing a "cost volume" that represents the matching costs of left and right features at various disparity values. The network subsequently learns to regress the final disparity values from the cost volume. The entire process can be trained end-to-end without the need for manual parameter tuning.

Of course, end-to-end deep learning systems necessitate copious amounts of data with ground-truth depth annotations to train the model, and performance may deteriorate when the training data does not align with real-world application scenarios. This underscores the need for strategies such as self-supervision and data augmentation to enhance robustness.

What Challenges Can Binocular Depth Estimation Encounter?

A prevalent issue in binocular depth estimation is inaccurate pixel matching. If an object's surface lacks texture, the images from the two perspectives will appear identical, rendering it arduous for the system to ascertain which point corresponds to which. Some algorithms endeavor to mitigate this issue by employing more intricate features or contextual information for matching, but it is not infallible.

The matching process delineated earlier presumes that the two images are captured simultaneously. If there are moving objects in the scene, such as pedestrians or vehicles, and the two cameras capture the scene at slightly disparate times, matching becomes more arduous. Deep learning methods can leverage temporal information to alleviate this issue, but it remains fundamentally complex.

In the design of binocular stereo systems, the selection of baseline length essentially entails a trade-off between measurement accuracy and engineering feasibility. A longer baseline yields a larger disparity for the same object in the left and right images, simplifying matching and effectively enhancing depth estimation accuracy. However, an excessively long baseline introduces issues such as increased installation space requirements, diminished mechanical stability, and reduced overlapping field of view. Conversely, a baseline that is too short will result in exceedingly small disparities for distant objects, which can be readily overshadowed by image noise, quantization errors, and other factors at the pixel level, leading to depth estimation failure.

Finally, real-world scenarios involving lighting fluctuations, occlusions, and reflective surfaces can render matching unstable. This is why considerable effort is devoted to image preprocessing, matching optimization, and post-processing filtering in the design of binocular systems.

Concluding Remarks

Binocular depth estimation boasts a broad spectrum of applications. Beyond autonomous driving, it plays a pivotal role in industrial inspection, drone mapping, real-time 3D modeling, and other domains. In scenarios necessitating rapid perception and reconstruction of 3D spaces, binocular vision, when combined with point cloud generation and other technologies, enables efficient real-time environmental modeling. While active sensors like LiDAR offer superior accuracy, binocular solutions provide a cost-effective alternative, rendering them ideal for numerous cost-sensitive applications.

-- END --