Explaining Visual Attention for Autonomous Vehicle Controllers

End to end deep learning controllers can produce strong driving performance, but their internal decision processes are difficult to interpret. This lack of clarity makes it harder for engineers to diagnose failures and can reduce public confidence in automated systems. This project will create a counterfactual explanation framework that identifies which elements in camera images, such as vehicles, pedestrians, or traffic control devices, guide actions like braking or steering. The research will apply generative video inpainting to remove or alter specific visual elements and then observe how the autonomous controller responds to these modified scenarios. The study will integrate this method with the ADAPT architecture and evaluate it using benchmark datasets and both real and simulated environments, including the QCar testbed. The goal is to provide clear, intuitive explanations for controller decisions that support transparency and improve safety analysis. The framework will help engineers understand system behavior, locate potential weaknesses, and develop autonomous vehicle (AV) technologies that behave in ways that can be evaluated and verified.

• Record URL:
  • https://www.umass.edu/neutc/projects/explaining-visual-attention-autonomous-vehicle-controllers

Language

• English

Project

• Status: Active
• Funding: $214,000.00
• Contract Numbers:

  69A3552348301

• Sponsor Organizations:

  Office of the Assistant Secretary for Research and Technology

  University Transportation Centers Program
  Department of Transportation
  Washington, DC  United States  20590
• Managing Organizations:

  University of Massachusetts, Amherst

  Department of Civil and Environmental Engineering
  130 Natural Resources Road
  Amherst, MA  United States  01003
• Performing Organizations:

  University of Connecticut, Storrs

  Connecticut Transportation Institute
  270 Middle Turnpike, Unit 5202
  Storrs, CT  United States  06269-5202
• Principal Investigators:

  Bi, Jinbo

• Start Date: 20260101
• Expected Completion Date: 20261231
• Actual Completion Date: 0
• USDOT Program: University Transportation Centers Program
• Subprogram: University Transportation Centers

Subject/Index Terms

• TRT Terms: Automatic controllers; Autonomous vehicles; Decision making; Deep learning; Image analysis; Video
• Subject Areas: Highways; Safety and Human Factors; Vehicles and Equipment;

Filing Info

• Accession Number: 01973941
• Record Type: Research project
• Source Agency: New England University Transportation Center
• Contract Numbers: 69A3552348301
• Files: UTC, RIP
• Created Date: Dec 11 2025 1:37PM