(SD2022-255) A robust approach to camera radar fusion

Abstract

Researchers from UC San Diego have developed RadSenNet, a new approach of sequential fusing of information from radars and cameras. The key idea of sequential fusion is to fundamentally shift the center of focus in radar-camera fusion systems from cameras to radars. This shift enables their invention (RadSegNet) to achieve all-weather perception benefits of radar sensing. Keeping radars as the primary modality ensures reliability in all situations including occlusions, longrange and bad weather.

Technology Description

Perception systems for autonomous driving have seen significant advancements in their performance over last few years. However, these systems struggle to show robustness in extreme weather conditions because sensors like lidars and cameras, which are the primary sensors in a sensor suite, see a decline in performance under these conditions. In order to solve this problem, camera-radar fusion systems provide a unique opportunity for all weather reliable high quality perception. Cameras provides rich semantic information while radars can work through occlusions and in all weather conditions. In this work we show that the state-of-the-art fusion methods perform poorly when camera input is de-graded, which essentially results in losing the all-weather reliability they set out to achieve.

 Researchers from UC San Diego have developed RadSenNet, a new approach of sequential fusing of information from radars and cameras. The key idea of sequential fusion is to fundamentally shift the center of focus in radar-camera fusion systems from cameras to radars. This shift enables their invention (RadSegNet) to achieve all-weather perception benefits of radar sensing. Keeping radars as the primary modality ensures reliability in all situations including occlusions, longrange and bad weather.

RadSegNet architects the sequential fusion by decoupling the simultaneous feature extraction from both the cameras and radars, used in state-of-art architectures. Instead, features are sequentially extracted first from the camera and then propagated to radar point clouds, thus retaining reliable
radar inputs at all times along with a hint of semantic information from cameras. This is the first technology that performs all-weather reliable sensor fusion of radar point clouds and camera images, at long ranges, while keeping radars as the primary sensing modality.

A fusion between the camera and radar modalities can be performed in multiple ways. Some approaches do late fusion, wherein individual detections from multiple modalities are combined, but, here, we can not learn any joint representation or perform joint feature extraction. Alternatively, there exist early fusion approaches that work in two ways. Some works project radar points to camera perspective view RGB image, but operating in perspective view makes it harder to distinguish between small objects close to the sensor and large objects at a longer range. Others perform an inverse mapping of the camera's RGB image to radar's Bird's eye view (BEV) but an inverse mapping of the camera to BEV plane is ill-defined due to the lack of depth information in-camera images.

To solve the problems in late and early fusion techniques, the state-of-the-art approaches perform middle fusion or feature-level fusion. These approaches use multi-view feature aggregation from radar and image data. For a particular object region proposal, features are fused from radar point clouds and camera images simultaneously. Unfortunately, it turns out that this simultaneous feature extraction approach for camera and radar fusion has a heavy reliance on cameras. The performance severely degrades in cases where the camera input is degraded, like in bad weather conditions. We propose a new fusion method, that treats Radars as the primary sensor which truly achieves all-weather reliability, essentially solving all the problems mentioned above.

Applications

Advantages

State Of Development

Intellectual Property Info

This patent-pending technology is available for commercialization.

https://patentimages.storage.googleapis.com/a3/d6/44/730944908274af/WO2024015891A1.pdf

Related Materials

• Kshitiz Bansal, Keshav Rungta, Dinesh Bharadia RadSegNet: A Reliable Approach to Radar Camera Fusion - 07/22/2022