Digital Twin of Built Structures assisted by Computer Vision Techniques: Overview and Preliminary Results



Published Sep 4, 2023
Yasutaka Narazaki


Digital Twin is an effective platform for analyzing, visualizing, and interpreting the condition of as-built structures based on sensor measurement data. Based on the data inflow from the as-built structures, the associated models (typically finite element models) are updated, and/or the full behaviors of the structures are replicated in the virtual space. Despite its potential, field implementation of digital twin concepts often faces challenge, because the specific forms of digital twin, such as sensor types/locations, structural model development, and data fusion algorithms, depend strongly on the case-specific objectives. Focusing on the digital twin concepts assisted by computer vision techniques, this research aims at facilitating the implementation of those concepts by presenting the definition, setting, and preliminary results of digital twin in different application contexts, including post-earthquake building assessment and structural health monitoring based on multiple types of measurements. This research is expected to contribute to the broader impact of digital twin concepts in the structural engineering community.

Abstract 30 | PDF Downloads 73



Digital Twin, Computer Vision, Deep Learning, Structural Inspection, Structural Health Monitoring

Bruhn, A., Weickert, J., & Schnörr, C. (2005). Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods. International Journal of Computer Vision, 61(3), 1–21.

Dong, C.-Z., & Catbas, F. N. (2020). A review of computer vision–based structural health monitoring at local and global levels. Structural Health Monitoring, 147592172093558.

Feng, D., & Feng, M. Q. (2016). Vision-based multipoint displacement measurement for structural health monitoring. Structural Control and Health Monitoring, 23(5), 876–890.

Glaessgen, E. H., & Stargel, D. S. (2012). The digital twin paradigm for future NASA and U.S. Air force vehicles. Collection of Technical Papers - AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference.

Gomez, F., Narazaki, Y., Hoskere, V., Spencer, B. F., & Smith, M. D. (2022). Bayesian inference of dense structural response using vision-based measurements. Engineering Structures, 256, 113970.

Hoskere, V., Narazaki, Y., Hoang, T. A., & Spencer, B. F. (2020). MaDnet: Multi-task Semantic Segmentation of Multiple types of Structural Materials and Damage in Images of Civil Infrastructure. Submitted to the Journal of Civil Structural Health Monitoring.

Hoskere, V., Narazaki, Y., Hoang, T. A., & Spencer Jr., B. F. (2018, September). Towards Automated PostEarthquake Inspections with Deep Learning-based Condition-Aware Models. The 7th World Conference on Structural Control and Monitoring (7WCSCM).

Hoskere, V., Narazaki, Y., & Spencer, B. F. (2022). Physics-Based Graphics Models in 3D Synthetic Environments as Autonomous Vision-Based Inspection Testbeds. Sensors 2022, Vol. 22, Page 532, 22(2), 532.

Jiang, F., Ma, L., Broyd, T., & Chen, K. (2021). Digital twin and its implementations in the civil engineering sector. Automation in Construction, 130, 103838.

Lai, Y., Chen, J., Hong, Q., Li, Z., Liu, H., Lu, B., Ma, R., Yu, C., Sun, R., Demartino, C., & Narazaki, Y. (2022). Framework for long-term structural health monitoring by computer vision and vibration-based model updating. Case Studies in Construction Materials, 16, e01020.

Levine, N. M., Narazaki, Y., & Spencer, B. F. (2022). Performance-based post-earthquake building evaluations using computer vision-derived damage observations. Advances in Structural Engineering, 25(16), 3425–3449.

Levine, N. M., Narazaki, Y., & Spencer, B. F. (2023). Development of a building information model-guided post-earthquake building inspection framework using 3D synthetic environments. Earthquake Engineering and Engineering Vibration 2023, 1–29.

Levine, N. M., & Spencer, B. F. (2022). Post-Earthquake Building Evaluation Using UAVs: A BIM-Based Digital Twin Framework. Sensors 2022, Vol. 22, Page 873, 22(3), 873.

Liu, C. (2009). Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral Thesis. Massachusetts Institute of Technology.

Lucas, B. D., & Kanade, T. (1981). An iterative image registration technique with an application in stereo vision. Seventh International Joint Conference on Artificial Intelligence (IJCAI-81), 674–679.

Narazaki, Y., Gomez, F., Hoskere, V., Smith, M. D., & Spencer, B. F. (2020). Efficient development of vision-based dense three-dimensional displacement measurement algorithms using physics-based graphics models. Structural Health Monitoring, 147592172093952.

Narazaki, Y., Hoskere, V., Chowdhary, G., & Spencer, B. F. (2022). Vision-based navigation planning for autonomous post-earthquake inspection of reinforced concrete railway viaducts using unmanned aerial vehicles. Automation in Construction, 137, 104214.

Narazaki, Y., Hoskere, V., Eick, B. A., Smith, M. D., & Spencer, B. F. (2019). Vision-based dense displacement and strain estimation of miter gates with the performance evaluation using physics-based graphics models. Smart Structures and Systems, 24(6), 709–721.

Narazaki, Y., Hoskere, V., Hoang, T. A., Fujino, Y., Sakurai, A., & Spencer, B. F. (2020). Vision‐based automated bridge component recognition with high‐level scene consistency. Computer-Aided Civil and Infrastructure Engineering, 35(5), 465–482.

Narazaki, Y., Hoskere, V., Yoshida, K., Spencer, B. F., & Fujino, Y. (2021). Synthetic environments for visionbased structural condition assessment of Japanese high-speed railway viaducts. Mechanical Systems and Signal Processing. 07850

Pan, B., Tian, L., & Song, X. (2016). Real-time, non-contact and targetless measurement of vertical deflection of bridges using off-axis digital image correlation. NDT & E International, 79, 73–80.

Sacks, R., Brilakis, I., Pikas, E., Xie, H. S., & Girolami, M. (2020). Construction with digital twin information systems. Data-Centric Engineering, 1.

Spencer, B. F., Hoskere, V., & Narazaki, Y. (2019). Advances in Computer Vision-Based Civil Infrastructure Inspection and Monitoring. Engineering, 5(2), 199–222. Szeliski, R. (2011). Computer Vision. Springer London.

Tomasi, C. (1991). Detection and Tracking of Point Features. School of Computer Science, Carnegie Mellon Univ., 91(April), 1–22.

Tuegel, E. J., Ingraffea, A. R., Eason, T. G., & Spottswood, S. M. (2011). Reengineering aircraft structural life prediction using a digital twin. International Journal of Aerospace Engineering.

Wagg, D. J., Worden, K., Barthorpe, R. J., & Gardner, P. (2020). Digital Twins: State-of-the-Art and Future Directions for Modeling and Simulation in Engineering Dynamics Applications. ASCE-ASME J Risk and Uncert in Engrg Sys Part B Mech Engrg, 6(3).

Wang, S., Rodgers, C., Fillmore, T., Welsh, B., Golecki, T., V. Shajihan, S. A., Eick, B. A., & Spencer, B. F. (2023). Vision-based model updating and evaluation of miter gates on inland waterways. Engineering Structures, 280, 115674.

Wang, S., Rodgers, C., Zhai, G., Matiki, T. N., Welsh, B., Najafi, A., Wang, J., Narazaki, Y., Hoskere, V., & Spencer, B. F. (2022). A graphics-based digital twin framework for computer vision-based post-earthquake structural inspection and evaluation using unmanned aerial vehicles. Journal of Infrastructure Intelligence and Resilience, 1(1), 100003.

Wang, X., Demartino, C., Narazaki, Y., Monti, G., & Spencer, B. F. (2023). Rapid seismic risk assessment of bridges using UAV aerial photogrammetry. Engineering Structures, 279, 115589.

Yoon, H., Hoskere, V., Park, J.-W., & Spencer, B. (2017). Cross-Correlation-Based Structural System Identification Using Unmanned Aerial Vehicles. Sensors, 17(9), 2075.

Yuen, K. V., Beck, J. L., & Katafygiotis, L. S. (2006). Efficient model updating and health monitoring methodology using incomplete modal data without mode matching. Structural Control and Health Monitoring, 13(1), 91–107.

Zhai, G., Narazaki, Y., Wang, S., Shajihan, S. A. V., & Spencer, B. F. (2022). Synthetic data augmentation for pixel-wise steel fatigue crack identification using fully convolutional networks. Smart Structures and Systems, 29(1), 237–250.

Zhang, Y., & Lin, W. (2021). Computer-vision-based differential remeshing for updating the geometry of finite element model. Computer-Aided Civil and Infrastructure Engineering.
Special Session Papers