Journal of Data Insights
Vol 4 No 1 (2026): Journal of Data Insights

Systematic Review of Multimodal Emotion Recognition in the Wild: Integrating Facial Expressions, Speech, and Physiological Signals for Enhanced Context-Aware Applications : Tinjauan Sistematis Pengenalan Emosi Multimodal di Lingkungan Alami: Mengintegrasikan Ekspresi Wajah, Ucapan, dan Sinyal Fisiologis untuk Aplikasi yang Lebih Sadar Konteks

Muhammad Munsarif (Universitas Muhammadiyah Semarang)
Norshuhani Zamin (Universiti Tenaga Nasional)
Richmond Ampah-Mensah (Universitas Muhammadiyah Semarang)



Article Info

Publish Date
30 Jun 2026

Abstract

Facial Emotion Recognition (FER) has become a critical component of affective computing, human–computer interaction, intelligent healthcare, adaptive education, and assistive technologies. This systematic literature review synthesizes recent developments in multimodal emotion recognition in the wild by examining how facial expressions, speech, physiological signals, deep learning architectures, and deployment technologies shape robust context-aware FER systems. Following the PRISMA protocol, literature was identified from Scopus using FER-related deep learning keywords, resulting in 202 initial records and 61 eligible studies for thematic synthesis, trend analysis, methodological classification, and qualitative interpretation. The findings show that FER research has shifted strongly from handcrafted features toward CNN-based deep learning, transfer learning, hybrid architectures, attention mechanisms, and bio-inspired optimization. Human–computer interaction emerged as the dominant research context, while healthcare, autism spectrum disorder screening, education, assistive technology, mining safety, and smart services represent increasingly important application domains. Transfer learning dominated robustness strategies, while multimodal fusion using facial images, speech, EEG, wearable sensors, and audio-visual signals gained stronger academic attention because it improves contextual understanding and reduces the limitations of unimodal FER. The synthesis also reveals persistent challenges, including poor generalization in uncontrolled environments, dataset imbalance, cultural variation, micro-expression recognition, computational complexity, real-time deployment, and limited explainability. The review contributes a multidimensional conceptual perspective that integrates multimodal sensing, deep learning optimization, edge/IoT deployment, and ethical-aware application design. Future research should prioritize lightweight multimodal FER, cross-cultural datasets, explainable AI, privacy-preserving learning, real-world clinical validation, and adaptive systems capable of operating reliably under noisy, dynamic, and socially sensitive conditions.

Copyrights © 2026






Journal Info

Abbrev

jodi

Publisher

Subject

Computer Science & IT Mathematics

Description

The Journal of Data Insights is an open access publication for peer-reviewed scholarly journals. The Journal of Data Insights focuses on the processing, analysis and interpretation of data for data-driven decisions and solutions in industry, hospitals, government and universities. All articles ...