Article Per Year (5 Year)
p-Index From 2021 - 2026
0.23
P-Index
This Author published in this journals
All Journal International Journal of Electrical and Computer Engineering
Junhoo Byun
Seogang University
Computer Science & IT Electrical & Electronics Engineering

Published : 1 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 1 Documents
Search

 1 
Semantic-aligned multimodal human activity recognition using visual and audio data Yeeun Park; Junhoo Byun; Siwoo Byun
International Journal of Electrical and Computer Engineering (IJECE) Vol 16, No 4: August 2026
Publisher : Institute of Advanced Engineering and Science

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.11591/ijece.v16i4.pp2087-2095

Abstract

Human activity recognition (HAR) requires robust performance under heterogeneous sensing conditions for practical deployment. However, single-modality approaches are limited in capturing the rich contextual information inherent in complex human behaviors. This paper presents a semantic-aligned multimodal HAR framework that integrates visual and audio information without assuming instance-level synchronization. To address dataset heterogeneity, samples from the HMDB51 video dataset and the ESC-50 audio dataset are aligned by mapping fine-grained classes into a shared high-level activity label space. For each modality, ResNet-18-based models are trained independently using frame-based visual inputs and 64-bin Mel-spectrogram-based audio representations. During inference, the output logits of the two models are combined through score-level weighted linear fusion. Experimental results show that the proposed multimodal approach consistently outperforms unimodal baselines in terms of accuracy and Macro-F1 score, with particularly notable improvements in activity groups where environmental context plays a significant role. These findings indicate that semantic-aligned score-level fusion can improve recognition robustness even under mismatched dataset conditions.
Co-Authors Siwoo Byun Yeeun Park