Generating Unsafe Road Activity from LLMs and Road Imagery to Learn Better Safety

The Federal Highway Administration (FHWA) zero death vision is a strategic goal to eliminate all traffic fatalities and severe injuries. One of the pillars of this strategy is to advance life-saving technology in vehicles and infrastructure. Many such in-vehicle technological advances have been developed and deployed in the past decade such as backup cameras, lane departure warning systems, adaptive cruise control, emergency brake systems, driver attention monitoring, etc. However, deployments of technology in the infrastructure have lagged. Most infrastructure advances have centered around redesigning roads to control traffic and improve visibility of vulnerable road users (VRUs), i.e., anyone in the road environment unprotected by a vehicle's frame. Ideally, the infrastructure should also monitor road activity for unsafe conditions and communicate those conditions to drivers and VRUs. To accomplish this goal, the infrastructure needs to be instrumented with sensors to capture data for real-time analysis and devices for relaying information to all road users. With prior UTC support, the research team has addressed some of the building blocks needed for a ‘smart’ infrastructure such developing and deploying methods for reliably detecting and tracking vehicles, classifying anomalous activity, and estimating 3D vehicle parameters (speed and direction). These methods were developed using images collected from cameras deployed at intersections. The main challenges that the team faced were difficulty identifying a variety of ‘unsafe’ events because they rarely occurred at the deployment locations and difficulty tracking pedestrians and bicyclist because they were rarely captured. In the proposed work, the team will address these challenges by developing a system to simulate unsafe vehicle, pedestrian, bicyclist, and road worker activity from previously acquired images and large language models (LLMs). The proposed work also enables safety technology development without exhausting resources for camera installation. Focus will be on generating visual data of normal and anomalous activities such as near misses, erratic driving, blocked bicycle lanes, etc. Methods will be developed to model realistic driver behavior based on observations from datasets acquired from prior UTC projects in collaboration with deployment partners City of Pittsburgh Department of Mobility & Infrastructure and Shaler Township. Realistic bicyclist behavior will be modeled using bicyclist-centric data provided by partner dashcam.bike. Methods will be developed to leverage these models with LLMs to generate unsafe activity involving vehicles and VRUs based on text entered by users. The system will be deployed to a website so that anyone can create their own dataset. The benefit of such a road activity system will enable safety research and permit data driven evaluation of road or intersection safety without installing cameras and storing/analyzing hours upon hours of images per camera.

• Record URL:
  • https://ppms.cit.cmu.edu/projects/detail/603

Language

• English

Project

• Status: Completed
• Funding: $180,000.00
• Contract Numbers:

  69A3552344811

  69A3552348316

• Sponsor Organizations:

  Office of the Assistant Secretary for Research and Technology

  University Transportation Centers Program
  Department of Transportation
  Washington, DC  United States  20590
• Managing Organizations:

  Safety21 University Transportation Center

  Carnegie Mellon University
  Pittsburgh, PA  United States  15213

  Carnegie Mellon University

  Pittsburgh, PA  United States 
• Performing Organizations:

  Carnegie Mellon University

  Pittsburgh, PA  United States 
• Principal Investigators:

  Tamburo, Robert

  Narasimhan, Srinivas

• Start Date: 20250701
• Expected Completion Date: 20260831
• Actual Completion Date: 20260831
• USDOT Program: University Transportation Centers

Subject/Index Terms

• TRT Terms: Artificial intelligence; Behavior; Data collection; Image analysis; Infrastructure; Simulation; Traffic safety; Vulnerable road users
• Subject Areas: Data and Information Technology; Highways; Pedestrians and Bicyclists; Safety and Human Factors;

Filing Info

• Accession Number: 02002858
• Record Type: Research project
• Source Agency: Safety21 University Transportation Center
• Contract Numbers: 69A3552344811, 69A3552348316
• Files: UTC, RIP
• Created Date: Sep 17 2026 5:05PM