Deep Reinforcement Learning Based Driving Strategies Bootstrapped by Suboptimal Policies

Designing autonomous driving systems requires a modular approach. This includes vehicle sensing, perception, localization, scene representation, path planning, decision-making, and vehicle control. The study focuses on decision-making in complex driving scenarios. Traditional methods often use model-based, heuristic, or rule-based controllers. Heuristic and rule-based controllers rely on human expertise to handle different driving situations. They include safety measures and rules to avoid deadlocks. However, these methods face limitations as designing rules to cover every possible driving scenario is highly challenging and resource intensive. Reinforcement learning (RL), on the other hand, offers a self-supervised approach, learning control policies through interaction with the environment. With advancements in neural networks, deep reinforcement learning (DRL) has shown the ability to efficiently map state-action transitions in high-dimensional environments. However, a persistent challenge in RL is achieving sufficient exploration of the environment. Research has shown that RL agents require extensive exploration to accurately map state-action values and discover high-reward states. Even with mechanisms like stochastic DRL algorithms that aim to embed exploration throughout the training process, agents often fail to discover optimal policies in complex scenarios. As a result, RL agents struggle with delayed rewards, where limited and inefficient exploration can result in sub-optimal performance. Driving environments encompass a range of challenging scenarios, such as roundabouts, overtaking, merging, intersections, and highway exits. These scenarios involve delayed rewards, which discourage exploration and complicate the RL learning process. Examples of delayed rewards include the cumulative benefit of using a free lane to achieve optimal cruising speed, waiting for the right moment to enter a roundabout safely, or merging into traffic while avoiding collisions—even if it requires temporarily stopping in the lane. Failure to explore and identify these states during training can lead to sub-optimal policies or even unsafe behaviors, such as remaining stuck behind a slow vehicle, rushing into a roundabout prematurely, or misjudging merges in traffic. In this study, the research team proposes to address these limitations by combining the stability of rule-based sub-optimal controllers with the adaptive learning capabilities of DRL. The team aims to utilize sub-optimal policies to bootstrap efficient training during the RL learning process and to overcome exploration barriers. Specifically, the team leverages a rule-based controller to prevent deadlocks and guides RL agents toward promising states, improving their ability to learn optimal driving strategies. This approach not only improves collision avoidance but also enhances training efficiency by guiding agents through challenging scenarios where rewards are delayed. By integrating suboptimal strategies, the method strikes a balance between simplicity and adaptability, enabling agents to learn more effectively. The experiments are conducted in a complex driving scenario where the driving policy must optimize long-term rewards. The key contributions of this study are summarized as follows: (1) The sub-optimal policies: The team designs sub-optimal controllers capable of making driving decisions under reasonable motivations. (2) Guidance for RL agents: The sub-optimal controllers guide RL agent toward exploring states associated with delayed rewards, improving exploration and learning efficiency. (3) Validation in complex driving scenarios: The team validates the effectiveness of the method in complex driving scenarios, demonstrating its ability to optimize long-term rewards and handle challenging conditions effectively.

• Record URL:
  • https://ppms.cit.cmu.edu/projects/detail/597

Language

• English

Project

• Status: Completed
• Funding: $155,789.00
• Contract Numbers:

  69A3552344811

  69A3552348316

• Sponsor Organizations:

  Office of the Assistant Secretary for Research and Technology

  University Transportation Centers Program
  Department of Transportation
  Washington, DC  United States  20590
• Managing Organizations:

  Safety21 University Transportation Center

  Carnegie Mellon University
  Pittsburgh, PA  United States  15213

  Carnegie Mellon University

  Pittsburgh, PA  United States 
• Performing Organizations:

  Ohio State University

  Columbus, OH  United States 
• Principal Investigators:

  Redmill, Keith

  Zhang, Zhihao

  Yurtsever, Ekim

• Start Date: 20250701
• Expected Completion Date: 20260831
• Actual Completion Date: 20260831
• USDOT Program: University Transportation Centers

Subject/Index Terms

• TRT Terms: Artificial intelligence; Automatic controllers; Autonomous vehicles; Decision making; Machine learning; Safety
• Subject Areas: Highways; Operations and Traffic Management; Planning and Forecasting; Safety and Human Factors; Vehicles and Equipment;

Filing Info

• Accession Number: 02002856
• Record Type: Research project
• Source Agency: Safety21 University Transportation Center
• Contract Numbers: 69A3552344811, 69A3552348316
• Files: UTC, RIP
• Created Date: Sep 17 2026 4:49PM