Towards Safe and Efficient Autonomous Driving: A Synergistic Approach with Human Expertise and Multimodal Large Language Models

Autonomous driving (AD) systems often face challenges with corner cases due to limited scene comprehension and insufficient learning of human knowledge in safety-critical situations. To address this, the research team proposes a dual-stage approach integrating multimodal large language models (MLLMs) and human expertise. The MLLM will employ Chain-of-Thought (CoT) reasoning for improved decision-making and be continuously fine-tuned through reinforcement learning (RL), with human expertise injected through human-artificial intelligence (AI) interaction supported by an accident warning system. Additionally, a unified platform will be developed to integrate scenario generation, algorithm development, and testing. Comprehensive closed-loop evaluations across benchmarks will demonstrate the model’s lightweight, fast, and reliable performance in end-to-end AD applications.

• Record URL:
  • https://ccat.umtri.umich.edu/research/wisconsin/towards-safe-and-efficient-autonomous-driving-a-synergistic-approach-with-human-expertise-and-multimodal-large-language-models/

Language

• English

Project

• Status: Active
• Funding: $330,000.00
• Contract Numbers:

  69A3552348305

• Sponsor Organizations:

  Office of the Assistant Secretary for Research and Technology

  University Transportation Centers Program
  Department of Transportation
  Washington, DC  United States  20590
• Managing Organizations:

  University of Michigan Transportation Research Institute

  2901 Baxter Road
  Ann Arbor, Michigan  United States  48109
• Project Managers:

  Bezzina, Debra

  Stearns, Amy

• Performing Organizations:

  University of Wisconsin, Madison

  Department of Civil and Environmental Engineering
  1415 Engineering Drive
  Madison, WI  United States  53706

  Purdue University, Lyles School of Civil Engineering

  550 Stadium Mall Drive
  West Lafayette, IN  United States  47907
• Principal Investigators:

  Chen, Sikai

  Feng, Yiheng

• Start Date: 20251015
• Expected Completion Date: 20261015
• Actual Completion Date: 0
• USDOT Program: University Transportation Centers Program

Subject/Index Terms

• TRT Terms: Artificial intelligence; Autonomous vehicles; Connected vehicles; Decision making; Human machine systems; Machine learning; Traffic safety
• Subject Areas: Highways; Planning and Forecasting; Safety and Human Factors; Vehicles and Equipment;

Filing Info

• Accession Number: 01970974
• Record Type: Research project
• Source Agency: Center for Connected and Automated Transportation
• Contract Numbers: 69A3552348305
• Files: UTC, RIP
• Created Date: Nov 13 2025 4:06PM