Session: CIE-06-01: AL/ML AI and ML for System Engineering

Paper Number: 192148

192148 - AffordRAG-Factory: A Hierarchical Edge VLM and Knowledge Graph Framework For Ambiguous Multi-Robot Task Execution in Manufacturing 

High-Mix Low-Volume (HMLV) production is the normal mode for many small and medium-sized enterprises (SMEs) and typically requires deep worker involvement. With the advancement of smart manufacturing, computing power has become a critical resource. Large manufacturers usually have sufficient resources, whereas SMEs are constrained by limited computing budgets and, due to data security and compliance concerns, are often reluctant to rely on cloud-based platforms. As a result, achieving intelligent upgrading under limited local computing resources has become a central challenge.

In addition, workers often use ambiguous expressions in daily communication without clearly specifying the required tools, actions, or constraint parameters. This makes end-to-end multimodal models prone to resource inconsistency, out-of-bound actions, and unstable multi-step planning in real deployment. To address this issue, this paper proposes \textit{AffordRAG-Factory}, a hierarchical embodied multi-robot task execution framework for human-robot collaboration under low-compute constraints. The framework combines the semantic understanding ability of edge-side small vision-language models (VLMs) with the structured knowledge representation, retrieval-augmented reasoning, and constraint filtering mechanisms of knowledge graphs (KGs), and supports cross-platform executable deployment through a unified Skill-Code protocol, making the decision process explicit and interpretable. Experimental results show that the framework achieves high accuracy and low latency in understanding ambiguous workshop instructions. Ablation studies further confirm the importance of KG constraints for stable and executable generation. The framework was also validated on three heterogeneous real-robot platforms across multiple scenarios, demonstrating strong generalization and deployment feasibility.

Presenting Author: Yang Meng University of Georgia

Presenting Author Biography: Yang Meng is a Ph.D. student in the Mechanical Engineering Department at the University of Georgia. He works as a researcher in the MODEL group lab under the guidance of Dr. Beshoy Morkos. He obtained his master’s degree from Johns Hopkins University in 2023, focusing mainly on AI and medical robotics. He completed his undergraduate studies at Florida Institute of Technology and Wuhan Institute of Technology, graduating with Magna Cum Laude honors. He previously worked at the Fraunhofer Intelligent Manufacturing Center at Shanghai Jiao Tong University, where he was mainly responsible for weld seam identification and Industry 4.0 research. He is very interested in the application of control theory and AI in various emerging fields.