Development of an Interactive Twinverse System : Metaverse Platform Integrating Digital Twin and AI Agent

Abstract

Digital twin (DT) platforms are increasingly used for PHM, yet most systems still lack real-time, bi-directional control between physical assets and virtual models, and a unified semantic layer that grounds natural-language commands in plant constraints. To address this gap, this research present Twinverse, an interactive metaverse environment that integrates a ROS/Kafka-based bi-directional DT, a knowledge-graph (KG) semantic backbone, and an LLM-powered agent. The KG encodes structural/operational constraints (e.g., kinematics, limits) and is serialized into a vector store to support RAG-based intent interpretation, while a constraint-aware execution pipeline verifies workspace, joint limits, and speed bounds prior to dispatch. Implemented on an industrial robot cell in Unity, the system provides real-time synchronization and multi-user operation within a single immersive interface. In evaluation, the platform maintained tight virtual–physical tracking and stable latency under increasing user load, and it enabled PHM-oriented functions such as anomaly interrogation and explainable, context-aware action generation. Our contribution is a cohesive DT–KG–LLM architecture that (1) grounds language-to-action in machine-readable plant constraints, (2) closes the loop from natural-language intent to verified execution, and (3) operationalizes PHM analytics inside an immersive DT environment. This work demonstrates a practical path toward interactive, explainable, and real-time PHM decision support.