Session: CIE-31-01 VARE User Experience (UX) and Human-Machine Interaction

Paper Number: 196438

196438 - A Digital Twin-Based Visual Inspection Framework Using Vision-Language Models in Mixed Reality 

Vision-language models (VLMs) offer a promising means of supporting visual decision making on the manufacturing shop-floor. However, their direct application to industrial scenes is often unreliable because they lack explicit guidance regarding what to inspect and from which viewpoint to observe. This study proposes an inspection framework in which a digital twin provides such guidance by specifying what to check, where to look, and how each view should be evaluated. Within this framework, the VLM performs reasoning only under the structure defined by the digital twin, while ambiguous cases are escalated to the operator for review. The procedure is delivered through a mixed reality interface and demonstrated in a milling fixture workcell across 13 scenarios using four VLMs. The proposed pipeline improves accuracy by 3 to 23 percentage points over a single-view baseline. Moreover, this improvement is not simply attributable to the use of additional views, as naive multi-view aggregation yields little benefit. These results confirm that the reliability gain stems from the structural guidance provided by the digital twin rather than from the VLM alone. This supports conservative, traceable, and practically deployable inspection decisions for workcells where uncertain observations must be handled safely and transparently.

Presenting Author: Seongbin Choi Sungkyunkwan University

Presenting Author Biography: Seongbin Choi is an M.S. candidate in the Department of Mechanical Engineering at Sungkyunkwan University (SKKU), Suwon, Republic of Korea, and a researcher in the Sustainable Design and Manufacturing Laboratory.

His research interests include prognostics and health management, digital-twin-based smart manufacturing, and human-robot-AI interaction.

His recent work has focused on synthetic anomaly dataset generation for industrial robot systems, health-index estimation for motor-driven power steering.

He received his B.S. in Mechanical Engineering from Sungkyunkwan University in 2024.