Lightweight Skeleton–Based Hand Gesture Recognition Using Machine Learning for Human-Robot Interaction

Panca Mudjirahardjo (Department of Electrical Engineering, Faculty of Engineering, Universitas Brawijaya, Malang, Indonesia Design & Innovation on IC & Embedded System (DIICES) RG, Faculty of Engineering, Universitas Brawijaya, Malang, Indonesia Robot BRONE Research Gr)
Rahmadwati (Department of Electrical Engineering, Faculty of Engineering, Universitas Brawijaya, Malang, Indonesia Design & Innovation on IC & Embedded System (DIICES) RG, Faculty of Engineering, Universitas Brawijaya, Malang)
Angger Abdul Razak (Department of Electrical Engineering, Faculty of Engineering, Universitas Brawijaya, Malang, Indonesia Design & Innovation on IC & Embedded System (DIICES) RG, Faculty of Engineering, Universitas Brawijaya, Malang)
Raden Arief Setyawan (Department of Electrical Engineering, Faculty of Engineering, Universitas Brawijaya, Malang, Indonesia Design & Innovation on IC & Embedded System (DIICES) RG, Faculty of Engineering, Universitas Brawijaya, Malang)
Tanjo Yui (Department of Mechanical and Control Engineering, Graduate School of Engineering, Kyushu Institute of Technology, Kitakyushu)

Journal of Information Systems Engineering and Business Intelligence
Vol. 12 No. 2 (2026): June

Background: Gesture-based control has emerged as a natural and intuitive approach for human–robot interaction (HRI). Advances in computer vision, particularly skeletal pose estimation, provide robust solutions to hand gesture recognition that are independent of lighting conditions and other sensor dependencies. However, most previous studies have relied heavily on deep neural network approaches, such as graph convolutional networks (GCN) and transformer-based architectures, which require substantial computational resources, making them less suitable for real-time robotic system applications. Objective: This study aims to develop a lightweight skeleton pose-based gesture recognition framework for human – robot interaction, focusing on accuracy, robustness, and computational efficiency. Methods: In this experiment, hand movement reference points were extracted using skeleton pose estimation. From these reference points, referred to as joint coordinates, feature vectors are constructed. These feature vectors are then used as input to the ML model, including SVMs, RF, GBM, and LGBM. The machine learning models were comparatively evaluated in terms of F1-score recognition, latency, and robustness in dynamic environments. Results: The SVM classifier consistently outperformed RF, GBM, and LGBM, achieving an F1-score of 91.3%. Real-time processing with an average latency of 44 ms per frame (≈22.7 FPS) and demonstrated stable performance under dynamic operating conditions. These results indicate that the selected skeleton keypoints and feature representation are effective in capturing discriminative hand-gesture patterns while maintaining low computational overhead. Conclusion: When combined with conventional machine learning classifiers, this study demonstrates that efficient skeleton pose-based feature construction offers a viable alternative for lightweight computation in gesture-based human – robot interaction. This study’s findings indicate that reliable and responsive gesture-based control supports practical deployment on embedded humanoid robot platforms.   Keywords: Skeleton-based gesture recognition, human – robot interaction, machine learning, computer vision, embedded robotics