5G-enabled Safe and Robust Deep Multi-agent Reinforcement Learning Framework for CAV Coordination

We address the problem of coordination and control of Connected and Automated Vehicles (CAVs) in the presence of imperfect observations in mixed traffic environment. A commonly used approach is learning-based decision-making, such as reinforcement learning (RL). However, most existing safe RL methods suffer from two limitations: (i) they assume accurate state information, and (ii) safety is generally defined over the expectation of the trajectories. It remains challenging to design optimal coordination between multi-agents while ensuring hard safety constraints under system state uncertainties (e.g., those that arise from noisy sensor measurements, communication, or state estimation methods) at every time step. We propose a safety guaranteed hierarchical coordination and control scheme called Safe-RMM to address the challenge. Specifically, the high-level coordination policy of CAVs in mixed traffic environment is trained by the Robust Multi-Agent Proximal Policy Optimization (RMAPPO) method. Though trained without uncertainty, our method leverages a worst-case Q network to ensure the model's robust performances when state uncertainties are present during testing. The low-level controller is implemented using model predictive control (MPC) with robust Control Barrier Functions (CBFs) to guarantee safety through their forward invariance property. We compare our method with baselines in different road networks in the CARLA simulator. Results show that our method provides the best evaluated safety and efficiency in challenging mixed traffic environments with uncertainties.

Language

• English

Project

• Status: Completed
• Funding: $230000
• Contract Numbers:

  69A3552348301

• Sponsor Organizations:

  Office of the Assistant Secretary for Research and Technology

  University Transportation Centers Program
  Department of Transportation
  Washington, DC  United States  20590
• Managing Organizations:

  University of Massachusetts, Amherst

  Department of Civil and Environmental Engineering
  130 Natural Resources Road
  Amherst, MA  United States  01003
• Performing Organizations:

  University of Connecticut, Storrs

  Connecticut Transportation Institute
  270 Middle Turnpike, Unit 5202
  Storrs, CT  United States  06269-5202
• Principal Investigators:

  Miao, Fei

  Han, Song

• Start Date: 20240101
• Expected Completion Date: 20250831
• Actual Completion Date: 0
• USDOT Program: University Transportation Centers Program
• Subprogram: University Transportation Centers

Subject/Index Terms

• TRT Terms: Autonomous vehicles; Connected vehicles; Machine learning; Multi-agent systems; Vehicle to infrastructure communications; Vehicle to vehicle communications
• Subject Areas: Data and Information Technology; Operations and Traffic Management; Safety and Human Factors; Vehicles and Equipment;

Filing Info

• Accession Number: 01904668
• Record Type: Research project
• Source Agency: New England University Transportation Center
• Contract Numbers: 69A3552348301
• Files: UTC, RIP
• Created Date: Jan 16 2024 12:08PM