Maintenance Strategies for Sewer Pipes with Multi-State Degradation and Deep Reinforcement Learning
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
https://orcid.org/0000-0003-3062-8408
Thiago D. Simão
Zaharah Bukhsh
Tiedo Tinga
Hajo Molegraaf
Nils Jansen
Marielle Stoelinga
Abstract
Large-scale infrastructure systems are crucial for societal welfare, and their effective management requires strategic forecasting and intervention methods that account for various complexities. Our study addresses two challenges within the Prognostics and Health Management (PHM) framework applied to sewer assets: modeling pipe degradation across severity levels and developing effective maintenance policies. We employ Multi-State Degradation Models (MSDM) to represent the stochastic degradation process in sewer pipes and use Deep Reinforcement Learning (DRL) to devise maintenance strategies. A case study of a Dutch sewer network exemplifies our methodology. Our findings demonstrate the model's effectiveness in generating intelligent, cost-saving maintenance strategies that surpass heuristics. It adapts its management strategy based on the pipe's age, opting for a passive approach for newer pipes and transitioning to active strategies for older ones to prevent failures and reduce costs. This research highlights DRL's potential in optimizing maintenance policies. Future research will aim improve the model by incorporating partial observability, exploring various reinforcement learning algorithms, and extending this methodology to comprehensive infrastructure management.
How to Cite
##plugins.themes.bootstrap3.article.details##
Reinforcement Learning, Prognostics and Health Management, Predictive Maintenance, Sewer Asset Management, Maintenance Policy Optimization
network management. Journal of construction engineering and management, 124(5), 402–410.
Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining.
Ana, E., & Bauwens, W. (2010). Modeling the structural deterioration of urban drainage pipes: the state-of-the-art in statistical methods. Urban Water Journal, 7(1), 47–59.
Ansel, J., Yang, E., He, H., Gimelshein, N., Jain, A., Voznesensky, M., ... others (2024). Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the 29th acm international conference on architectural support for programming languages and operating systems, volume 2 (pp. 929–947).
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017). Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6), 26–38.
Assaf, G., & Assaad, R. H. (2023). Optimal preventive maintenance, repair, and replacement program for catch basins to reduce urban flooding: Integrating agent-based modeling and monte carlo simulation. Sustainability, 15(11), 8527.
Caradot, N., Riechel, M., Fesneau, M., Hernandez, N., Torres, A., Sonnenberg, H., ... Rouault, P. (2018). Practical benchmarking of statistical and machine learning models for predicting the condition of sewer pipes in berlin, germany. Journal of Hydroinformatics, 20(5), 1131–1147.
Cardoso, M., Almeida, M. d. C., & Santos Silva, M. (2016). Sewer asset management planning–implementation of a structured approach in wastewater utilities. Urban Water Journal, 13(1), 15–27.
De Jonge, B., & Scarf, P. A. (2020). A review on maintenance optimization. European journal of operational research, 285(3), 805–824. Elmasry, M., Zayed, T., & Hawari, A. (2019). Multi-objective optimization model for inspection scheduling of sewer pipelines. Journal of Construction Engineering and Management, 145(2), 04018129. Fenner, R. A. (2000). Approaches to sewer maintenance: a review. Urban water, 2(4), 343–356. Hawari, A., Alkadour, F., Elmasry, M., & Zayed, T. (2017). Simulation-based condition assessment model for sewer pipelines. Journal of Performance of Constructed Facilities, 31(1), 04016066. Hern´andez, N., Caradot, N., Sonnenberg, H., Rouault, P., & Torres, A. (2021). Optimizing svm models as predicting
tools for sewer pipes conditions in the two main cities in colombia for different sewer asset management purposes. Structure and Infrastructure Engineering, 17(2), 156–169. Investigation and assessment of drain and sewer systems outside buildings - Part 1: General Requirements (Standard). (2012, October). Avenue Marnix 17, B-1000 Brussels: European Committee for Standardization (CEN). Investigation and assessment of drain and sewer systems outside buildings - Part 2: Visual inspection coding system (Standard). (2011, May). Avenue Marnix 17, B-1000 Brussels: European Committee for Standardization (CEN).
Jimenez-Roa, L. A., Heskes, T., Tinga, T., Molegraaf, H. J., & Stoelinga, M. (2022). Deterioration modeling of sewer pipes via discrete-time markov chains: A large-scale case study in the netherlands. In 32nd european safety and reliability conference, esrel 2022: Understanding and managing risk and reliability for a sustainable future (pp. 12991306). Jimenez-Roa, L. A., Tinga, T., Heskes, T., & Stoelinga, M. (2024). Comparing homogeneous and inhomogeneous time markov chains for modelling degradation in sewer pipe networks. In Proceedings of the european safety and reliability conference (esrel 2024). (Under review) Jones, E., Oliphant, T., Peterson, P., et al. (2001–). SciPy: Open source scientific tools for Python. Retrieved from http://www.scipy.org/ Kerkkamp, D., Bukhsh, Z. A., Zhang, Y., & Jansen, N. (2022). Grouping of maintenance actions with deep reinforcement learning and graph convolutional networks. In Icaart (2) (pp. 574–585). Khurelbaatar, G., Al Marzuqi, B., Van Afferden, M., M¨uller,
R. A., & Friesen, J. (2021). Data reduced method for cost comparison of wastewater management scenarios–case study for two settlements in jordan and oman. Frontiers in Environmental Science, 9, 626634. Laakso, T., Kokkonen, T., Mellin, I., & Vahala, R. (2019). Sewer life span prediction: Comparison of methods and assessment of the sample impact on the results. Water, 11(12), 2657. Lee, J., Park, C. Y., Baek, S., Han, S. H., & Yun, S. (2021). Risk-based prioritization of sewer pipe inspection from infrastructure asset management perspective. Sustainability, 13(13), 7213. Malek Mohammadi, M., Najafi, M., Kaushal, V., Serajiantehrani, R., Salehabadi, N., & Ashoori, T. (2019). Sewer pipes condition prediction models: A state-of-the-art review. Infrastructures, 4(4), 64. Marug´an, A. P. (2023). Applications of reinforcement learning for maintenance of engineering systems: A review. Advances in Engineering Software, 183, 103487. Montserrat, A., Bosch, L., Kiser, M., Poch, M., & Corominas,
L. (2015). Using data from monitoring combined sewer overflows to assess, improve, and maintain combined sewer systems. Science of the Total Environment, 505, 10531061. Mullapudi, A., Lewis, M. J., Gruden, C. L., & Kerkez, B. (2020). Deep reinforcement learning for the real time control of stormwater systems. Advances in water resources, 140, 103600. Ogunfowora, O., & Najjaran, H. (2023). Reinforcement and deep reinforcement learning-based solutions for machine
maintenance planning, scheduling policies, and optimization. Journal of Manufacturing Systems, 70, 244–263. Puterman, M. L. (1990). Markov decision processes. Handbooks in operations research and management science, 2, 331–434. Qasem, A., & Jamil, R. (2021). Gis-based financial analysis model for integrated maintenance and rehabilitation of underground pipe networks. Journal of Performance of Constructed Facilities, 35(5), 04021046. Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., & Dormann, N. (2021). Stable-baselines3: Reliable reinforcement learning implementations. The Journal of Machine Learning Research, 22(1), 12348–12355. Ramos-Salgado, C., Mu˜nuzuri, J., Aparicio-Ruiz, P., & Onieva, L. (2022). A comprehensive framework to efficiently plan short and long-term investments in water supply and sewer networks. Reliability Engineering & System Safety, 219, 108248. Saddiqi, M. M., Zhao, W., Cotterill, S., & Dereli, R. K. (2023). Smart management of combined sewer overflows: From an ancient technology to artificial intelligence. Wiley Interdisciplinary Reviews: Water, 10(3), e1635. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Taillandier, F., Elachachi, S., & Bennabi, A. (2020). A decision-support framework to manage a sewer system considering uncertainties. Urban Water Journal, 17(4), 344355. Tian, W., Fu, G., Xin, K., Zhang, Z., & Liao, Z. (2024). Improving the interpretability of deep reinforcement learning in urban drainage system operation. Water Research, 249, 120912. Tian, W., Liao, Z., Zhi, G., Zhang, Z., & Wang, X. (2022). Combined sewer overflow and flooding mitigation through a reliable real-time control based on multi-reinforcement learning and model predictive control. Water Resources Research, 58(7), e2021WR030703. Tscheikner-Gratl, F., Caradot, N., Cherqui, F., Leit˜ao, J. P., Ahmadi, M., Langeveld, J. G., .. . others (2019). Sewer asset management–state of the art and research needs. Urban Water Journal, 16(9), 662–675. Turnbull, B. W. (1976). The empirical distribution function with arbitrarily grouped, censored and truncated data. Journal of the Royal Statistical Society: Series B (Methodological), 38(3), 290–295. Wirahadikusumah, R., & Abraham, D. M. (2003). Application of dynamic programming and simulation for sewer management. Engineering, Construction and Architectural Management, 10(3), 193–208.
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
The Prognostic and Health Management Society advocates open-access to scientific data and uses a Creative Commons license for publishing and distributing any papers. A Creative Commons license does not relinquish the author’s copyright; rather it allows them to share some of their rights with any member of the public under certain conditions whilst enjoying full legal protection. By submitting an article to the International Conference of the Prognostics and Health Management Society, the authors agree to be bound by the associated terms and conditions including the following:
As the author, you retain the copyright to your Work. By submitting your Work, you are granting anybody the right to copy, distribute and transmit your Work and to adapt your Work with proper attribution under the terms of the Creative Commons Attribution 3.0 United States license. You assign rights to the Prognostics and Health Management Society to publish and disseminate your Work through electronic and print media if it is accepted for publication. A license note citing the Creative Commons Attribution 3.0 United States License as shown below needs to be placed in the footnote on the first page of the article.
First Author et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.