Digital Twin of Built Structures assisted by Computer Vision Techniques: Overview and Preliminary Results
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Sep 4, 2023
Yasutaka Narazaki
Abstract
Digital Twin is an effective platform for analyzing, visualizing, and interpreting the condition of as-built structures based on sensor measurement data. Based on the data inflow from the as-built structures, the associated models (typically finite element models) are updated, and/or the full behaviors of the structures are replicated in the virtual space. Despite its potential, field implementation of digital twin concepts often faces challenge, because the specific forms of digital twin, such as sensor types/locations, structural model development, and data fusion algorithms, depend strongly on the case-specific objectives. Focusing on the digital twin concepts assisted by computer vision techniques, this research aims at facilitating the implementation of those concepts by presenting the definition, setting, and preliminary results of digital twin in different application contexts, including post-earthquake building assessment and structural health monitoring based on multiple types of measurements. This research is expected to contribute to the broader impact of digital twin concepts in the structural engineering community.
##plugins.themes.bootstrap3.article.details##
Keywords
Digital Twin, Computer Vision, Deep Learning, Structural Inspection, Structural Health Monitoring
References
Bruhn, A., Weickert, J., & Schnörr, C. (2005). Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods. International Journal of Computer Vision, 61(3), 1–21. https://doi.org/10.1023/B:VISI.0000045324.43199.43
Dong, C.-Z., & Catbas, F. N. (2020). A review of computer vision–based structural health monitoring at local and global levels. Structural Health Monitoring, 147592172093558. https://doi.org/10.1177/1475921720935585
Feng, D., & Feng, M. Q. (2016). Vision-based multipoint displacement measurement for structural health monitoring. Structural Control and Health Monitoring, 23(5), 876–890. https://doi.org/10.1002/stc.1819
Glaessgen, E. H., & Stargel, D. S. (2012). The digital twin paradigm for future NASA and U.S. Air force vehicles. Collection of Technical Papers - AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference. https://doi.org/10.2514/6.2012-1818
Gomez, F., Narazaki, Y., Hoskere, V., Spencer, B. F., & Smith, M. D. (2022). Bayesian inference of dense structural response using vision-based measurements. Engineering Structures, 256, 113970. https://doi.org/10.1016/J.ENGSTRUCT.2022.113970
Hoskere, V., Narazaki, Y., Hoang, T. A., & Spencer, B. F. (2020). MaDnet: Multi-task Semantic Segmentation of Multiple types of Structural Materials and Damage in Images of Civil Infrastructure. Submitted to the Journal of Civil Structural Health Monitoring.
Hoskere, V., Narazaki, Y., Hoang, T. A., & Spencer Jr., B. F. (2018, September). Towards Automated PostEarthquake Inspections with Deep Learning-based Condition-Aware Models. The 7th World Conference on Structural Control and Monitoring (7WCSCM).
Hoskere, V., Narazaki, Y., & Spencer, B. F. (2022). Physics-Based Graphics Models in 3D Synthetic Environments as Autonomous Vision-Based Inspection Testbeds. Sensors 2022, Vol. 22, Page 532, 22(2), 532. https://doi.org/10.3390/S22020532
Jiang, F., Ma, L., Broyd, T., & Chen, K. (2021). Digital twin and its implementations in the civil engineering sector. Automation in Construction, 130, 103838. https://doi.org/10.1016/J.AUTCON.2021.103838
Lai, Y., Chen, J., Hong, Q., Li, Z., Liu, H., Lu, B., Ma, R., Yu, C., Sun, R., Demartino, C., & Narazaki, Y. (2022). Framework for long-term structural health monitoring by computer vision and vibration-based model updating. Case Studies in Construction Materials, 16, e01020. https://doi.org/10.1016/J.CSCM.2022.E01020
Levine, N. M., Narazaki, Y., & Spencer, B. F. (2022). Performance-based post-earthquake building evaluations using computer vision-derived damage observations. Advances in Structural Engineering, 25(16), 3425–3449. https://doi.org/10.1177/13694332221119883
Levine, N. M., Narazaki, Y., & Spencer, B. F. (2023). Development of a building information model-guided post-earthquake building inspection framework using 3D synthetic environments. Earthquake Engineering and Engineering Vibration 2023, 1–29. https://doi.org/10.1007/S11803-023-2167-Y
Levine, N. M., & Spencer, B. F. (2022). Post-Earthquake Building Evaluation Using UAVs: A BIM-Based Digital Twin Framework. Sensors 2022, Vol. 22, Page 873, 22(3), 873. https://doi.org/10.3390/S22030873
Liu, C. (2009). Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral Thesis. Massachusetts Institute of Technology.
Lucas, B. D., & Kanade, T. (1981). An iterative image registration technique with an application in stereo vision. Seventh International Joint Conference on Artificial Intelligence (IJCAI-81), 674–679.
Narazaki, Y., Gomez, F., Hoskere, V., Smith, M. D., & Spencer, B. F. (2020). Efficient development of vision-based dense three-dimensional displacement measurement algorithms using physics-based graphics models. Structural Health Monitoring, 147592172093952. https://doi.org/10.1177/1475921720939522
Narazaki, Y., Hoskere, V., Chowdhary, G., & Spencer, B. F. (2022). Vision-based navigation planning for autonomous post-earthquake inspection of reinforced concrete railway viaducts using unmanned aerial vehicles. Automation in Construction, 137, 104214. https://doi.org/10.1016/J.AUTCON.2022.104214
Narazaki, Y., Hoskere, V., Eick, B. A., Smith, M. D., & Spencer, B. F. (2019). Vision-based dense displacement and strain estimation of miter gates with the performance evaluation using physics-based graphics models. Smart Structures and Systems, 24(6), 709–721. https://doi.org/10.12989/SSS.2019.24.6.709
Narazaki, Y., Hoskere, V., Hoang, T. A., Fujino, Y., Sakurai, A., & Spencer, B. F. (2020). Vision‐based automated bridge component recognition with high‐level scene consistency. Computer-Aided Civil and Infrastructure Engineering, 35(5), 465–482. https://doi.org/10.1111/mice.12505
Narazaki, Y., Hoskere, V., Yoshida, K., Spencer, B. F., & Fujino, Y. (2021). Synthetic environments for visionbased structural condition assessment of Japanese high-speed railway viaducts. Mechanical Systems and Signal Processing. https://doi.org/https://doi.org/10.1016/j.ymssp.2021.1 07850
Pan, B., Tian, L., & Song, X. (2016). Real-time, non-contact and targetless measurement of vertical deflection of bridges using off-axis digital image correlation. NDT & E International, 79, 73–80. https://doi.org/10.1016/J.NDTEINT.2015.12.006
Sacks, R., Brilakis, I., Pikas, E., Xie, H. S., & Girolami, M. (2020). Construction with digital twin information systems. Data-Centric Engineering, 1. https://doi.org/10.1017/DCE.2020.16
Spencer, B. F., Hoskere, V., & Narazaki, Y. (2019). Advances in Computer Vision-Based Civil Infrastructure Inspection and Monitoring. Engineering, 5(2), 199–222. https://doi.org/10.1016/J.ENG.2018.11.030 Szeliski, R. (2011). Computer Vision. Springer London. https://doi.org/10.1007/978-1-84882-935-0
Tomasi, C. (1991). Detection and Tracking of Point Features. School of Computer Science, Carnegie Mellon Univ., 91(April), 1–22. https://doi.org/10.1016/S0031-3203(03)00234-6
Tuegel, E. J., Ingraffea, A. R., Eason, T. G., & Spottswood, S. M. (2011). Reengineering aircraft structural life prediction using a digital twin. International Journal of Aerospace Engineering. https://doi.org/10.1155/2011/154798
Wagg, D. J., Worden, K., Barthorpe, R. J., & Gardner, P. (2020). Digital Twins: State-of-the-Art and Future Directions for Modeling and Simulation in Engineering Dynamics Applications. ASCE-ASME J Risk and Uncert in Engrg Sys Part B Mech Engrg, 6(3). https://doi.org/10.1115/1.4046739
Wang, S., Rodgers, C., Fillmore, T., Welsh, B., Golecki, T., V. Shajihan, S. A., Eick, B. A., & Spencer, B. F. (2023). Vision-based model updating and evaluation of miter gates on inland waterways. Engineering Structures, 280, 115674. https://doi.org/10.1016/J.ENGSTRUCT.2023.115674
Wang, S., Rodgers, C., Zhai, G., Matiki, T. N., Welsh, B., Najafi, A., Wang, J., Narazaki, Y., Hoskere, V., & Spencer, B. F. (2022). A graphics-based digital twin framework for computer vision-based post-earthquake structural inspection and evaluation using unmanned aerial vehicles. Journal of Infrastructure Intelligence and Resilience, 1(1), 100003. https://doi.org/10.1016/J.IINTEL.2022.100003
Wang, X., Demartino, C., Narazaki, Y., Monti, G., & Spencer, B. F. (2023). Rapid seismic risk assessment of bridges using UAV aerial photogrammetry. Engineering Structures, 279, 115589. https://doi.org/10.1016/J.ENGSTRUCT.2023.115589
Yoon, H., Hoskere, V., Park, J.-W., & Spencer, B. (2017). Cross-Correlation-Based Structural System Identification Using Unmanned Aerial Vehicles. Sensors, 17(9), 2075. https://doi.org/10.3390/s17092075
Yuen, K. V., Beck, J. L., & Katafygiotis, L. S. (2006). Efficient model updating and health monitoring methodology using incomplete modal data without mode matching. Structural Control and Health Monitoring, 13(1), 91–107. https://doi.org/10.1002/stc.144
Zhai, G., Narazaki, Y., Wang, S., Shajihan, S. A. V., & Spencer, B. F. (2022). Synthetic data augmentation for pixel-wise steel fatigue crack identification using fully convolutional networks. Smart Structures and Systems, 29(1), 237–250. https://doi.org/10.12989/SSS.2022.29.1.237
Zhang, Y., & Lin, W. (2021). Computer-vision-based differential remeshing for updating the geometry of finite element model. Computer-Aided Civil and Infrastructure Engineering. https://doi.org/10.1111/MICE.12708
Dong, C.-Z., & Catbas, F. N. (2020). A review of computer vision–based structural health monitoring at local and global levels. Structural Health Monitoring, 147592172093558. https://doi.org/10.1177/1475921720935585
Feng, D., & Feng, M. Q. (2016). Vision-based multipoint displacement measurement for structural health monitoring. Structural Control and Health Monitoring, 23(5), 876–890. https://doi.org/10.1002/stc.1819
Glaessgen, E. H., & Stargel, D. S. (2012). The digital twin paradigm for future NASA and U.S. Air force vehicles. Collection of Technical Papers - AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference. https://doi.org/10.2514/6.2012-1818
Gomez, F., Narazaki, Y., Hoskere, V., Spencer, B. F., & Smith, M. D. (2022). Bayesian inference of dense structural response using vision-based measurements. Engineering Structures, 256, 113970. https://doi.org/10.1016/J.ENGSTRUCT.2022.113970
Hoskere, V., Narazaki, Y., Hoang, T. A., & Spencer, B. F. (2020). MaDnet: Multi-task Semantic Segmentation of Multiple types of Structural Materials and Damage in Images of Civil Infrastructure. Submitted to the Journal of Civil Structural Health Monitoring.
Hoskere, V., Narazaki, Y., Hoang, T. A., & Spencer Jr., B. F. (2018, September). Towards Automated PostEarthquake Inspections with Deep Learning-based Condition-Aware Models. The 7th World Conference on Structural Control and Monitoring (7WCSCM).
Hoskere, V., Narazaki, Y., & Spencer, B. F. (2022). Physics-Based Graphics Models in 3D Synthetic Environments as Autonomous Vision-Based Inspection Testbeds. Sensors 2022, Vol. 22, Page 532, 22(2), 532. https://doi.org/10.3390/S22020532
Jiang, F., Ma, L., Broyd, T., & Chen, K. (2021). Digital twin and its implementations in the civil engineering sector. Automation in Construction, 130, 103838. https://doi.org/10.1016/J.AUTCON.2021.103838
Lai, Y., Chen, J., Hong, Q., Li, Z., Liu, H., Lu, B., Ma, R., Yu, C., Sun, R., Demartino, C., & Narazaki, Y. (2022). Framework for long-term structural health monitoring by computer vision and vibration-based model updating. Case Studies in Construction Materials, 16, e01020. https://doi.org/10.1016/J.CSCM.2022.E01020
Levine, N. M., Narazaki, Y., & Spencer, B. F. (2022). Performance-based post-earthquake building evaluations using computer vision-derived damage observations. Advances in Structural Engineering, 25(16), 3425–3449. https://doi.org/10.1177/13694332221119883
Levine, N. M., Narazaki, Y., & Spencer, B. F. (2023). Development of a building information model-guided post-earthquake building inspection framework using 3D synthetic environments. Earthquake Engineering and Engineering Vibration 2023, 1–29. https://doi.org/10.1007/S11803-023-2167-Y
Levine, N. M., & Spencer, B. F. (2022). Post-Earthquake Building Evaluation Using UAVs: A BIM-Based Digital Twin Framework. Sensors 2022, Vol. 22, Page 873, 22(3), 873. https://doi.org/10.3390/S22030873
Liu, C. (2009). Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral Thesis. Massachusetts Institute of Technology.
Lucas, B. D., & Kanade, T. (1981). An iterative image registration technique with an application in stereo vision. Seventh International Joint Conference on Artificial Intelligence (IJCAI-81), 674–679.
Narazaki, Y., Gomez, F., Hoskere, V., Smith, M. D., & Spencer, B. F. (2020). Efficient development of vision-based dense three-dimensional displacement measurement algorithms using physics-based graphics models. Structural Health Monitoring, 147592172093952. https://doi.org/10.1177/1475921720939522
Narazaki, Y., Hoskere, V., Chowdhary, G., & Spencer, B. F. (2022). Vision-based navigation planning for autonomous post-earthquake inspection of reinforced concrete railway viaducts using unmanned aerial vehicles. Automation in Construction, 137, 104214. https://doi.org/10.1016/J.AUTCON.2022.104214
Narazaki, Y., Hoskere, V., Eick, B. A., Smith, M. D., & Spencer, B. F. (2019). Vision-based dense displacement and strain estimation of miter gates with the performance evaluation using physics-based graphics models. Smart Structures and Systems, 24(6), 709–721. https://doi.org/10.12989/SSS.2019.24.6.709
Narazaki, Y., Hoskere, V., Hoang, T. A., Fujino, Y., Sakurai, A., & Spencer, B. F. (2020). Vision‐based automated bridge component recognition with high‐level scene consistency. Computer-Aided Civil and Infrastructure Engineering, 35(5), 465–482. https://doi.org/10.1111/mice.12505
Narazaki, Y., Hoskere, V., Yoshida, K., Spencer, B. F., & Fujino, Y. (2021). Synthetic environments for visionbased structural condition assessment of Japanese high-speed railway viaducts. Mechanical Systems and Signal Processing. https://doi.org/https://doi.org/10.1016/j.ymssp.2021.1 07850
Pan, B., Tian, L., & Song, X. (2016). Real-time, non-contact and targetless measurement of vertical deflection of bridges using off-axis digital image correlation. NDT & E International, 79, 73–80. https://doi.org/10.1016/J.NDTEINT.2015.12.006
Sacks, R., Brilakis, I., Pikas, E., Xie, H. S., & Girolami, M. (2020). Construction with digital twin information systems. Data-Centric Engineering, 1. https://doi.org/10.1017/DCE.2020.16
Spencer, B. F., Hoskere, V., & Narazaki, Y. (2019). Advances in Computer Vision-Based Civil Infrastructure Inspection and Monitoring. Engineering, 5(2), 199–222. https://doi.org/10.1016/J.ENG.2018.11.030 Szeliski, R. (2011). Computer Vision. Springer London. https://doi.org/10.1007/978-1-84882-935-0
Tomasi, C. (1991). Detection and Tracking of Point Features. School of Computer Science, Carnegie Mellon Univ., 91(April), 1–22. https://doi.org/10.1016/S0031-3203(03)00234-6
Tuegel, E. J., Ingraffea, A. R., Eason, T. G., & Spottswood, S. M. (2011). Reengineering aircraft structural life prediction using a digital twin. International Journal of Aerospace Engineering. https://doi.org/10.1155/2011/154798
Wagg, D. J., Worden, K., Barthorpe, R. J., & Gardner, P. (2020). Digital Twins: State-of-the-Art and Future Directions for Modeling and Simulation in Engineering Dynamics Applications. ASCE-ASME J Risk and Uncert in Engrg Sys Part B Mech Engrg, 6(3). https://doi.org/10.1115/1.4046739
Wang, S., Rodgers, C., Fillmore, T., Welsh, B., Golecki, T., V. Shajihan, S. A., Eick, B. A., & Spencer, B. F. (2023). Vision-based model updating and evaluation of miter gates on inland waterways. Engineering Structures, 280, 115674. https://doi.org/10.1016/J.ENGSTRUCT.2023.115674
Wang, S., Rodgers, C., Zhai, G., Matiki, T. N., Welsh, B., Najafi, A., Wang, J., Narazaki, Y., Hoskere, V., & Spencer, B. F. (2022). A graphics-based digital twin framework for computer vision-based post-earthquake structural inspection and evaluation using unmanned aerial vehicles. Journal of Infrastructure Intelligence and Resilience, 1(1), 100003. https://doi.org/10.1016/J.IINTEL.2022.100003
Wang, X., Demartino, C., Narazaki, Y., Monti, G., & Spencer, B. F. (2023). Rapid seismic risk assessment of bridges using UAV aerial photogrammetry. Engineering Structures, 279, 115589. https://doi.org/10.1016/J.ENGSTRUCT.2023.115589
Yoon, H., Hoskere, V., Park, J.-W., & Spencer, B. (2017). Cross-Correlation-Based Structural System Identification Using Unmanned Aerial Vehicles. Sensors, 17(9), 2075. https://doi.org/10.3390/s17092075
Yuen, K. V., Beck, J. L., & Katafygiotis, L. S. (2006). Efficient model updating and health monitoring methodology using incomplete modal data without mode matching. Structural Control and Health Monitoring, 13(1), 91–107. https://doi.org/10.1002/stc.144
Zhai, G., Narazaki, Y., Wang, S., Shajihan, S. A. V., & Spencer, B. F. (2022). Synthetic data augmentation for pixel-wise steel fatigue crack identification using fully convolutional networks. Smart Structures and Systems, 29(1), 237–250. https://doi.org/10.12989/SSS.2022.29.1.237
Zhang, Y., & Lin, W. (2021). Computer-vision-based differential remeshing for updating the geometry of finite element model. Computer-Aided Civil and Infrastructure Engineering. https://doi.org/10.1111/MICE.12708
Section
Special Session Papers
This work is licensed under a Creative Commons Attribution 3.0 Unported License.