Safe and Personalized Control of Autonomous Vehicles with On-Board Vision Language Models: System Design and Real-World Validation

This project focuses on enhancing autonomous vehicle control systems by integrating on-board Vision-Language Models (VLMs) for safe and personalized driving experiences. Building on the previously awarded Center for Connected and Automated Transportation (CCAT) project on “CAV Pilot Development and Deployment in Midwest Winter,” this research addresses critical challenges in autonomous vehicle development regarding limited on-board computational resources by implementing lightweight VLM frameworks and Retrieval-Augmented Generation (RAG)-based memory modules. The project will validate the system’s ability to handle challenging urban scenarios, reduce human takeover rates, and adapt to diverse environmental conditions.

• Record URL:
  • https://ccat.umtri.umich.edu/research/purdue/safe-and-personalized-control-of-autonomous-vehicles-with-on-board-vision-language-models-system-design-and-real-world-validation/

Language

• English

Project

• Status: Active
• Funding: $120,000.00
• Contract Numbers:

  69A3552348305

• Sponsor Organizations:

  Office of the Assistant Secretary for Research and Technology

  University Transportation Centers Program
  Department of Transportation
  Washington, DC  United States  20590
• Managing Organizations:

  University of Michigan Transportation Research Institute

  2901 Baxter Road
  Ann Arbor, Michigan  United States  48109
• Project Managers:

  Bezzina, Debra

  Stearns, Amy

• Performing Organizations:

  Purdue University, Lyles School of Civil Engineering

  550 Stadium Mall Drive
  West Lafayette, IN  United States  47907
• Principal Investigators:

  Wang, Ziran

• Start Date: 20251015
• Expected Completion Date: 20261015
• Actual Completion Date: 0
• USDOT Program: University Transportation Centers Program

Subject/Index Terms

• TRT Terms: Automated vehicle control; Autonomous vehicles; Connected vehicles; Machine vision; System design; Validation
• Subject Areas: Data and Information Technology; Design; Highways; Safety and Human Factors; Vehicles and Equipment;

Filing Info

• Accession Number: 01970970
• Record Type: Research project
• Source Agency: Center for Connected and Automated Transportation
• Contract Numbers: 69A3552348305
• Files: UTC, RIP
• Created Date: Nov 13 2025 3:43PM