Session: CIE-31-02 VARE User Experience (UX) and Human-Machine Interaction

Paper Number: 196422

196422 - Development of a Human–ai–robot Interaction System for Multimodal Spatial Command Execution Using a Digital Twin 

Translating workers' object-centered, context-dependent spatial expressions into executable commands for industrial robots remains a core challenge in human-centric smart manufacturing. A gap persists between human multimodal communication, and the coordinate-based forms robots need, as workers refer to objects through pointing and relative spatial language rather than absolute coordinates. To close this gap, systems must read not only which object a worker points to, but also the spatial relation intended around it. Prior studies on Digital Twin (DT), Mixed Reality, and natural language interfaces have improved human–robot interaction but still offer limited support for mapping object-relative spatial cues into executable targets. To this end, we propose a DT-based human–AI–robot interaction system that turns multimodal inputs into structured robot-ready forms. The worker's egocentric view, pointing, and voice commands are captured and grounded in a Unity-based digital twin to build targets via Model Context Protocol (MCP), Robot Operating System (ROS), and MoveIt. A case study with repeated trials reached a mean position error of 5.63 mm, showing that the DT works not only as a visualization space but as an interface linking object-relative human intent into robot-ready motion. These findings support the DT as an interface bridging worker input and robot execution in practice.

Presenting Author: Jaewon Seo Sungkyunkwan University

Presenting Author Biography: Jaewon Seo is a Master's student in the Department of Mechanical Engineering at Sungkyunkwan University, Republic of Korea. His research interests include human–robot interaction, digital twin, mixed reality, and AI-driven manufacturing systems. He received his B.S. degree in Mechanical Engineering from Sungkyunkwan University in 2025. His current work focuses on multimodal interfaces that bridge human intent and industrial robot execution.