Mixed Reality Assisted Infrastructure Inspections

This project will develop a functional wearable device based on Mixed Reality technology and Real-Time Machine Learning to assist inspectors in the field in performing enhanced visual assessment of civil infrastructures. Work in Stage 1 will focus mainly on field data collection, data preparation and annotation, training the machine learning models and development of the core functions of the Mixed Reality system. Real world image data for various concrete defect types will be collected from different types of structures and under different environmental conditions. Different deep learning architectures will be explored for the optimal performance in terms of speed and accuracy. Also, for each architecture type, optimal training parameters will be searched through an iterative process of deep learning training. Based on model evaluation results, the best performing model will be selected for use in the remaining project work. Image tracking libraries will be configured in the platform environment, a camera calibration procedure will be developed, and an accurate distance estimation will be accomplished. Work in the Stage 2 will focus on establishing a methodology for element condition assessment, wearable device implementation, and extensive testing of the system in both the laboratory and field environment. Procedures for quantified assessment in which the limit values are interpreted from major inspection guidelines will be investigated. Necessary system implementation will be completed and synchronization of assessment data to a central management system will be accomplished. The mixed reality system, machine learning models and condition assessment procedure will be implemented in a wearable headset device. First, the mixed reality environment along with the trained machine learning models will be deployed, followed by the development of the condition assessment procedure for AASHTO guidelines. Finally, the system optimizations will be performed. A robust user interface for the mixed reality system will be developed. Input from bridge inspectors will be used to optimize the interface for highway agencies’ inspection practices. The operational feasibility of the system will be extensively evaluated both in the laboratory and the field. First, a series of laboratory tests will be performed to measure the obtained level of accuracy in different conditions to define minimum requirements for reliable use of the system. Next, the feasibility of the system will be investigated through field tests by evaluating improvements in inspection speed and personnel safety. The final report will provide all the relevant data, findings, and conclusions along with recommendations on using the developed technique for a more efficient and effective civil infrastructure inspection and the plans for its implementation by state DOTs.


  • English


  • Status: Active
  • Contract Numbers:

    Project 20-30, IDEA 222

  • Sponsor Organizations:

    National Cooperative Highway Research Program

    Transportation Research Board
    500 Fifth Street, NW
    Washington, DC  United States  20001

    American Association of State Highway and Transportation Officials (AASHTO)

    444 North Capitol Street, NW
    Washington, DC  United States  20001

    Federal Highway Administration

    1200 New Jersey Avenue, SE
    Washington, DC  United States  20590
  • Project Managers:

    Jawed, Inam

  • Performing Organizations:

    University of Central Florida

  • Principal Investigators:

    Catbas, Necati

  • Start Date: 20200511
  • Expected Completion Date: 0
  • Actual Completion Date: 0

Subject/Index Terms

Filing Info

  • Accession Number: 01739005
  • Record Type: Research project
  • Source Agency: Transportation Research Board
  • Contract Numbers: Project 20-30, IDEA 222
  • Files: TRB, RIP
  • Created Date: May 11 2020 5:48PM