Session: CIE-09-02 AMS/SEIKM: Artificial Intelligence and Machine Learning in Design and Manufacturing

Paper Number: 141805

141805 - Human Assembly Action Recognition Framework for Real-World Assembly Operation 

In recent years, significant progress has been proceeded in human action recognition (HAR) technology. When applying HAR to practical scenarios, especially in the task of recognizing Standard Operating Procedures (SOP) for factory operators, some challenges such as resource requirements, differentiation of diverse movements, and difficulties in manual annotation exists. In this research, an assembly action recognition framework (AARF) was proposed to recognize the assembly operation conducted at the station level using the SlowFast video understanding model. First, the pre-trained YOLOv7 model is employed as a non-assembly action filter to detect the main operator, and then the irrelevant background is masked for a refined region of interest. Then, the preprocessed data is fed into SlowFast for action recognition through both temporal and spatial features. Finally, the confidence calibration strategy is implemented to enhance the robustness of results. A series of ablative experiments were conducted to study for optimizing the selection of model parameters, incorporating attention mechanism, non-assembly action filtering, and recognition result filter. Multiple assembly motion videos were collected from two similar product lines at a real-world production line. According to experimental results, the proposed method can improve the accuracy of human motion recognition on real factory production lines by up to 86.3%.

Presenting Author: Chao-Lung Yang National Taiwan University of Science and Technology

Presenting Author Biography: Chao-Lung Yang received a B.S. degree in Mechanical Engineering and an M.S. degree in Automation Control from National Taiwan University of Science and Technology, Taiwan, in 1996 and 1998, respectively. He also received the M.S.I.E. and Ph.D. degree in Industrial Engineering from Purdue University, West Lafayette, in 2004 and 2009, respectively. He is currently with the Department of Industrial Management, National Taiwan University of Science and Technology, Taiwan, as a professor. His research area are data mining, machine learning, big data analytics, metaheuristic algorithm, and human action recognition. He has developed a data streaming analytics framework by applying deep learning models such as CNN and LSTM, and metaheuristic algorithms to quickly detect the process shift and classify the product detects. Recently, he works in the computer vision domain to develop machine learning models to recognize human action, particularly in manufacturing operations. He is a member of INFORMS, IEEE, and ASME.

Authors:

Chao-Lung Yang National Taiwan University of Science and Technology
Tzu Ching Kao National Taiwan University of Science and Technology
Po Ting Lin National Taiwan University of Science and Technology
Pincheng Fu AVer Information Inc.