Hierarchical Decision Making and Control in RL-based Autonomous Driving for Improved Safety in Complex Traffic Scenarios

In this work, the research team proposes a framework for vehicle planning and decision-making and vehicle control. The team formulates the vehicle control problem as an MDP (Markov Decision Process) and leverage reinforcement learning to learn high-quality autonomous driving strategies. The motivation behind this research stems from the complexity of highway driving, which requires the seamless coordination of multiple tasks to ensure safe and efficient navigation. In the realm of vehicle control research, various methodologies have been explored, including model-based and rule-based approaches, as well as data-driven methods such as supervised learning and reinforcement learning. Model-based and rule-based methods provide transparency and interpretability in decision-making processes, but they necessitate tailored controllers for diverse driving scenarios [1-3]. On the other hand, data-driven methods like supervised learning require extensive dataset collection [4], while reinforcement learning generates its own data through simulations and learns driving strategies through interactions with the environment. A common limitation across prior works in reinforcement learning lies in their reliance on single-layer controllers, lacking higher-level decision-making capabilities [5][6]. Furthermore, existing studies often focus on relatively simplistic traffic scenarios, limiting the generalizability of their findings [7][8]. To address these limitations and to address the challenges posed by complex highway driving scenarios, the team adopts a Hierarchical Deep Reinforcement Learning (HDRL) framework. Specifically, the upper-level controller will set the vehicle’s target speed and desired lane or lane change behavior, while the lower-level controller undertakes the fine-grained task of managing the driving dynamics of the vehicle’s longitudinal acceleration and lateral steering control. This hierarchical framework endows the controller with enhanced interpretability and empowers it to navigate complex and intricate traffic environments more effectively than single-layer counterparts. By incorporating high-level decision-making abilities, the proposed approach presents a significant advancement over traditional reinforcement learning-based controllers. The team contemplates two pivotal advantages from this approach. Firstly, HDRL facilitates the hierarchical decomposition of decision-making tasks, enhancing the efficiency of task execution. Secondly, within the HDRL framework, the dual layers of controllers exhibit distinct exploration capabilities in unfamiliar environments. This empowers the team to harness the upper-level controller's proficiency in mapping potential driving trajectories within intricate autonomous driving settings, while concurrently engaging the lower-level controller for precise real-time adjustments and control of vehicle behavior. By leveraging the advantages of hierarchical learning, the team aims to achieve better coordination and decision-making capabilities, leading to improved safety and exploration in high-stakes environments. Through comprehensive simulations and experiments, the team aims to demonstrate the superiority of the hierarchical controller in handling complex driving situations and contribute to the advancement of safe and efficient autonomous driving technology. The team will design driving scenarios, including challenging “trap” scenarios, to test the reinforcement learning framework and compare its performance with traditional single-layer reinforcement learning controllers. These traps involve other traffic vehicles obstructing the autonomous vehicle’s desired path, testing the systems’ ability to identify and navigate around such obstructions. By evaluating the performance of the controller and refining the HDRL framework, the team endeavors to strike an optimal equilibrium between safety imperatives and decision making and operational efficiency.

• Record URL:
  https://ppms.cit.cmu.edu/projects/detail/413

Language

• English

Project

• Status: Active
• Funding: $160749
• Contract Numbers:

  69A3552344811

• Sponsor Organizations:

  Carnegie Mellon University

  Safety21 National UTC for Promoting Safety
  Pittsburgh, PA  United States  15213

  Office of the Assistant Secretary for Research and Technology

  University Transportation Center Program
  ,    
• Managing Organizations:

  Carnegie Mellon University

  Safety21 National UTC for Promoting Safety
  Pittsburgh, PA  United States  15213
• Project Managers:

  Stearns, Amy

• Performing Organizations:

  The Ohio State University

  ,    
• Principal Investigators:

  Redmill, Keith

• Start Date: 20230701
• Expected Completion Date: 20240630
• Actual Completion Date: 0
• USDOT Program: University Transportation Centers

Subject/Index Terms

• TRT Terms: Automatic controllers; Autonomous vehicles; Decision making; Machine learning; Vehicle trajectories
• Subject Areas: Highways; Operations and Traffic Management; Vehicles and Equipment;

Filing Info

• Accession Number: 01900231
• Record Type: Research project
• Source Agency: Safety21
• Contract Numbers: 69A3552344811
• Files: UTC, RIP
• Created Date: Nov 20 2023 7:28PM