Claim Missing Document
Check
Articles

Found 2 Documents
Search

Implementasi Multimodal Emotion Recognition Menggunakan CNN-BiLSTM-Attention pada Audio dan Mobilenetv2 pada Ekspresi Wajah Ai Solihah; Alun Sujjada; Ivana Lucia Kharisma
TIN: Terapan Informatika Nusantara Vol 7 No 2 (2026): July 2026
Publisher : Forum Kerjasama Pendidikan Tinggi (FKPT)

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.47065/tin.v7i2.10629

Abstract

The advancement of artificial intelligence technology has significantly contributed to the development of emotion recognition systems capable of understanding human emotional states more accurately. However, unimodal approaches that rely on a single data source, such as speech or facial expressions, still face limitations when dealing with dynamic environmental conditions. This study aims to implement a Multimodal Emotion Recognition system by integrating audio and visual modalities to improve emotion detection accuracy. The audio modality is processed using a CNN-BiLSTM-Attention architecture with Mel-Spectrogram representations as input, while the visual modality is analyzed using MobileNetV2 to recognize facial expressions. The integration of both modalities is performed using the Late Fusion method at the decision level through a weighted fusion approach, assigning weights of 0.4 and 0.6 to the audio and visual modalities, respectively. This research employs an experimental method with a quantitative approach. The research stages include literature review, data collection, audio and video preprocessing, model design, model training, multimodal integration, and system evaluation. Audio data are transformed into Mel-Spectrogram features, while video data are processed using MediaPipe Face Detection to extract facial regions prior to emotion classification. The proposed system is designed to recognize three emotional classes: neutral, happy, and angry. Model performance is evaluated using a Confusion Matrix, accuracy, precision, recall, and F1-score metrics. The results indicate that multimodal integration using the Late Fusion method effectively leverages the strengths of each modality, resulting in more accurate and robust emotion detection compared to unimodal approaches. Furthermore, the developed system is capable of performing real-time emotion classification through a camera and microphone, making it potentially applicable in various domains, including education, customer service, human-computer interaction systems, and user emotional state monitoring. The primary contribution of this research is the development of a multimodal emotion recognition system that integrates a CNN-BiLSTM-Attention model for the audio modality and MobileNetV2 for the visual modality using a late fusion method. The proposed system leverages emotional information from both modalities simultaneously to enhance the accuracy and stability of real-time emotion detection, thereby offering an alternative solution for the development of AI-based human-computer interaction systems.
Integrasi Analisis Butir Soal dan Klasifikasi Kognitif Untuk Personalisasi Rekomendasi Remedial Adaptif Menggunakan K-Nearest Neighbor Karimah Agustin; Alun Sujjada; Muhammad Ikhsan Thohir
Journal of Information System Research (JOSH) Vol 8 No 1 (2026): October 2026
Publisher : Forum Kerjasama Pendidikan Tinggi (FKPT)

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.47065/josh.v8i1.11066

Abstract

In many secondary schools, classroom assessments still use tests that are written by hand, which makes it hard for teachers to ensure each question matches the right level of thinking from Bloom's taxonomy or to check if the questions are statistically reliable. This study tackles that issue by creating a web-based testing system for a madrasah tsanawiyah, which automatically sorts multiple-choice questions, checks their quality using psychometric methods, and offers custom learning suggestions to help students improve. Cognitive-level tags (C1 to C6) are assigned by matching keywords that describe actions, and a kNearest Neighbor classifier uses TF-IDF features to sort items into groups that show more or less complex thinking. Item quality is checked using Classical Test Theory module automatically computes the difficulty index and discrimination index of each item from students answer patterns, and materials that help improve learning are chosen based on each student's weak areas through content-based filtering. The system was developed using the Waterfall model and was tested on 600 items taken from three different subjects. The classifier achieved 86.67% accuracy when using k=11, which was verified through 10- fold cross-validation showing 81.83% ± 5.19%, and all 23 black-box test cases were successfully passed. The results show that using automatic item analysis along with adaptive remediation is possible to implement in assessment practices at the madrasah level. The main contribution of this study is the integration of automatic cognitive classification, CTT-based psychometric analysis, and content-based filtering remedial recommendation into a single real-time examination platform, a combination that has not been found in previous similar studies, which generally address only one of these three aspects separately.