Cost-Effective Radar-Based Depth Perception and Scene Interpretation

One of the current challenges for autonomous vehicles (AV) is providing provable safety in urban driving and highway driving. LIDAR provides accurate depth and shape information, but remains quite expensive. In this research the research team explores the viability of fusing 4D radar images with camera images as an alternative to using LIDAR. The research aims to establish the data fusion of radar and camera images in autonomous driving as a replacement for data fusion using traditional LIDAR, radar, and camera images. The current approach demonstrates that a trained radar detector can achieve approximately 3.5 meters of Unidirectional Chamfer Distance (UCD) against ground truth LiDAR data on a dataset recently released by Delft University of Technology (DUT) in the Netherlands, which is 28% better than the state of the art (SOTA). The approach integrates the images obtained from deep neural network (DNN) 4D radar detectors with conventional camera RGB images. It introduces a novel pixel positional encoding algorithm inspired by Bartlett’s spectrum estimation technique. This algorithm transforms radar depth maps and RGB images into a unified pixel image subspace facilitating the learning of their potential correspondence. The method effectively leverages high-resolution camera images to train radar depth map generative models, addressing the limitations of conventional radar detectors in complex vehicular environments while sharpening the radar output. The research team develops spectrum estimation algorithms tailored for radar depth maps and RGB images, a comprehensive training framework for data-driven generative models, and a camera-radar deployment scheme for AV operation. The research plan is as follows: (1) Develop a deeper analysis of the training framework to develop more advanced deep learning-based radar detectors. (2) Scale current results on about 100 seconds of representative AV driving from a DUT dataset to a significantly larger number of frames, including a wider range of training and testing scenarios. (3) Quantify additional performance benefit from real-time (instead of current offline) fusion schemes and deployment frameworks for 4D radars and cameras. (4) Use the existing data to measure the underlying latency in the proposed data fusion of 4D radar and camera images. This will help quantify the capabilities and limitations of the scheme. (5) Conduct a detailed error analysis to understand the distribution of error between the point cloud the team generates with the data fusion approach against the ground truth. This will help establish any existing bias in the error the team obtains in the UCD. (6) DUT’s 4D radar training data set was generated in the Netherlands. The team will investigate the augmentation of these data by data collected in collaboration with the deployment partner Blue Fusion, who will allow us the use of one or more of their high-performance radar. This effort will address one of the important pitfalls of machine learning systems, distribution shift or domain adaptation. The results will be instrumental in providing a cost-effective and weather-resilient solution to the perception part of the provable safety challenge in L4 and L5 Autonomous Vehicles.

• Record URL:
  • https://ppms.cit.cmu.edu/projects/detail/592

Language

• English

Project

• Status: Completed
• Funding: $73,244.00
• Contract Numbers:

  69A3552344811

  69A3552348316

• Sponsor Organizations:

  Office of the Assistant Secretary for Research and Technology

  University Transportation Centers Program
  Department of Transportation
  Washington, DC  United States  20590
• Managing Organizations:

  Safety21 University Transportation Center

  Carnegie Mellon University
  Pittsburgh, PA  United States  15213

  Carnegie Mellon University

  Pittsburgh, PA  United States 
• Performing Organizations:

  Carnegie Mellon University

  Pittsburgh, PA  United States 
• Principal Investigators:

  Dolan, John

• Start Date: 20250701
• Expected Completion Date: 20260831
• Actual Completion Date: 20260831
• USDOT Program: University Transportation Centers

Subject/Index Terms

• TRT Terms: Alternatives analysis; Autonomous vehicles; Data fusion; Data quality; Deep learning; Image analysis; Laser radar; Radar; Vehicle safety
• Subject Areas: Data and Information Technology; Highways; Safety and Human Factors; Vehicles and Equipment;

Filing Info

• Accession Number: 02002854
• Record Type: Research project
• Source Agency: Safety21 University Transportation Center
• Contract Numbers: 69A3552344811, 69A3552348316
• Files: UTC, RIP
• Created Date: Sep 17 2026 4:33PM