Claim Missing Document
Check
Articles

Systematic Review of Hyperparameter Adjustment and Evaluation Metrics in Bert-Based Sentiment Analysis Bahari, Aris Rifki Setiya; Utomo, Fandy Setyo; Berlilana, Berlilana
Journal La Multiapp Vol. 7 No. 2 (2026): Journal La Multiapp
Publisher : Newinera Publisher

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.37899/journallamultiapp.v7i2.3046

Abstract

The development of sentiment analysis towards Aspect-Based Sentiment Analysis (ABSA) has made significant progress thanks to deep learning technology, especially through the Bidirectional Encoder Representations from Transformers (BERT) architecture. Despite its increasing popularity, a comprehensive synthesis of global research patterns and optimal model configurations is still urgently needed. This study presents a Systematic Literature Review (SLR) combined with bibliometric analysis to examine BERT-based ABSA research indexed in Scopus. Using the PRISMA and VOSviewer frameworks for visualization, a total of 62 eligible articles up to mid-2025 were analyzed. The results of the study show a strong upward trend of publications with a peak in 2024, where China, India, and Indonesia emerged as the major contributors in this domain. Further, the review identified a critical technical standard for effective model training: the Adam optimizer was the most dominant choice, typically paired with a learning rate between 1e-5 to 2e-5 and a batch size of 16. Regarding performance evaluation, Accuracy and F1-Score are set as de facto standard metrics. These findings provide strategic guidance for researchers to optimize BERT implementation and identify future directions in more in-depth sentiment analysis tasks.
Adaptive Test Model Enhancement Based on Salmon Salar Optimization and Partially Observable Markov Decision Process Saputro, Rujianto Eko; Utomo, Fandy Setyo; Wanti, Linda Perdana
Journal of Applied Data Sciences Vol 7, No 1: January 2026
Publisher : Bright Publisher

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.47738/jads.v7i1.1065

Abstract

Cognitive Diagnosis Models (CDMs) in Computerized Adaptive Testing (CAT) are widely used to assess students’ cognitive abilities; however, existing approaches face significant limitations. The Latent Trait Model often suffers from specification errors due to its complexity, the Diagnostic Classification Model encounters difficulties in integrating hierarchical structures, and Deep Learning Models demand substantial computational resources. To address these challenges, this study introduces Salmon Salar Optimization (SSO) to enhance CDM performance and integrates the Partially Observable Markov Decision Process (POMDP) to improve dynamic question selection. The proposed adaptive testing framework comprises three components: preprocessing, CDM, and a selection algorithm. Experimental results on the ASSISTments 2009-2010 dataset demonstrate that SSO outperforms representative baselines from both deep learning: Neural CD and Latent Trait Model: MIRT approaches. Using 5-fold cross-validation, the proposed model achieved superior predictive performance with 75.51% accuracy and an AUC of 0.8191, highlighting its robustness compared to existing state-of-the-art methods. Furthermore, adaptive test simulations reveal that the SSO- and POMDP-based model delivers superior outcomes, attaining 80.3% accuracy with a reward of 8.03 for 10-question exams and 79.8% accuracy with a reward of 11.97 for 15-question exams. These findings confirm the effectiveness of the proposed model in enhancing cognitive diagnosis and adaptive testing performance.
WhatsApp Hybrid Chatbot Architecture Rasa-DeepSeek: Design and Performance Evaluation Had, Iqbaluddin Syam; Utomo, Fandy Setyo; Karyono, Giat; Kinding, Dwi Putriana Nuramanah
Sistemasi: Jurnal Sistem Informasi Vol 15, No 2 (2026): Sistemasi: Jurnal Sistem Informasi
Publisher : Program Studi Sistem Informasi Fakultas Teknik dan Ilmu Komputer

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.32520/stmsi.v15i2.5791

Abstract

This study designed and evaluated a hybrid chatbot for a domain-specific application by addressing two main issues: limited NLU coverage and the variability of latency and cost when all queries are routed directly to an LLM. The proposed solution integrates a deterministic Rasa-based pipeline with a DeepSeek fallback mechanism. In this architecture, Rasa handles NLU processing, rules, stories, and context storage for mk and jk, while the LLM is only invoked when the NLU confidence score falls below a defined threshold. The methodology includes end-to-end implementation through a Node.js bridge connected to Rasa, functional testing to validate the intent–entity–action flow, and performance testing using load (stress) testing across two access paths: the Rasa REST endpoint and the Node-to-Rasa bridge. Meanwhile, the LLM pipeline was profiled separately through instrumented action calls. The results indicate that domain-specific conversations were successfully answered using curated knowledge, and both deterministic access paths met the service level objective (SLO), achieving a median latency of approximately 32 milliseconds with no observed errors. This study contributes by demonstrating that a hybrid chatbot architecture separating deterministic and generative pipelines can maintain SLO compliance in domain-specific settings. In addition, it highlights limitations of LLMs in understanding domain ontologies, reinforcing the need for semantic guardrails.
Optimization of Phishing Detection Performance with Variable Correlation Analysis and Imbalance Learning Arifin, Samsul; Setyo Utomo, Fandy
Sistemasi: Jurnal Sistem Informasi Vol 15, No 2 (2026): Sistemasi: Jurnal Sistem Informasi
Publisher : Program Studi Sistem Informasi Fakultas Teknik dan Ilmu Komputer

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.32520/stmsi.v15i2.4671

Abstract

Phishing is a common cyber security threat in which attackers attempt to deceive users into disclosing personal information such as passwords, credit card numbers, and other sensitive data. With the rapid advancement of technology, phishing techniques have become increasingly sophisticated and harder to detect using traditional methods. Therefore, it is essential to develop detection techniques capable of identifying phishing websites with high accuracy. This study aims to optimize phishing detection performance by integrating variable correlation analysis for feature selection and applying imbalanced learning techniques to address data imbalance. The research stages include Data Collection, Data Preprocessing, and Data Exploration, which involve correlation analysis, removal of low-correlation features, and data visualization. In the Model Building and Training phase, the dataset is split into features and labels, followed by training and the application of data balancing techniques, ending with Model Evaluation. The evaluated algorithms include Logistic Regression, Naive Bayes, K-Nearest Neighbors (KNN), Support Vector Machine (SVM), Multi-Layer Perceptron, Decision Tree, Random Forest, Gradient Boosting, and CatBoost. The results show that the KNN algorithm delivers the best performance, achieving an accuracy of 91.25% and optimal scores in Precision (0.906943), Recall (0.927858), and F1-Score (0.922141), along with the lowest Hamming Loss at 0.0875. In contrast, the SVM algorithm recorded the lowest performance among the tested models. The implementation of this method is expected to contribute to the development of more reliable and accurate phishing detection systems in the future.
Usability Evaluation of a School Library OPAC Using Heuristic Evaluation and User Testing Faradina Faradina; Taqwa Hariguna; Fandy Setyo Utomo
Journal of Information System and Informatics Vol 8 No 1 (2026): February
Publisher : Asosiasi Doktor Sistem Informasi Indonesia

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.63158/journalisi.v8i1.1528

Abstract

This study evaluates the usability of the Online Public Access Catalog (OPAC) at SMK Negeri 1 Purwokerto to address the persistent gap between traditional library information architectures and the modern search behaviors of vocational students within the Kurikulum Merdeka ecosystem. The research aims to solve the problem of "mental model dissonance" that hinders independent information literacy among digital native learners. A hybrid evaluation approach was employed, integrating a Heuristic Evaluation by three experts with empirical User Testing involving students. The study utilized the Think-Aloud protocol and the System Usability Scale (SUS) to capture both performance and perception data. Result: The expert inspection identified 18 significant usability violations, primarily in library technical jargon (H2) and error prevention (H5). Empirical testing revealed a low average Task Success Rate (TSR) of 49.3% and a mean SUS score of 55.0, placing the system in the "Unacceptable" category. These figures confirm that current cataloging logic significantly obstructs retrieval efficiency. The originality of this research lies in the identification of specific dissonance points between vocational students' mental models and bibliographic metadata. It provides a strategic framework for interface restructuring through semantic simplification and department-based navigation, offering a practical model for developing user-centric "smart" library services in vocational education.
ENHANCING HANDWRITTEN DIGIT RECOGNITION ACCURACY ON THE MNIST DATASET USING A HYBRID CNN-BILSTM MODEL WITH DATA AUGMENTATION Yugi, Muhtyas; Latif, Ahmad; Utomo, Fandy Setyo; Barkah, Azhari Shouni
JIPI (Jurnal Ilmiah Penelitian dan Pembelajaran Informatika) Vol 11, No 1 (2026)
Publisher : STKIP PGRI Tulungagung

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.29100/jipi.v11i1.7758

Abstract

Handwritten digit recognition is a classic challenge in the field of computer vision and machine learning, and continues to be developed to achieve higher accuracy. This study proposes a hybrid method that combines Convolutional Neural Networks (CNN) and Bidirectional Long Short-Term Memory (BiLSTM) to enhance performance in handwritten digit classification using the MNIST dataset. CNNs are em-ployed to extract spatial features from digit images, while BiLSTMs are used to capture the temporal patterns and sequential context from the extracted features. To address limitations in data variation and improve the model’s generalization capabilities, the study also applies data augmentation techniques based on image transformations such as rota-tion, translation, scaling, and flipping. Experimental results demonstrate that the hybrid CNN-BiLSTM model with data augmentation signifi-cantly improves classification accuracy compared to baseline ap-proaches without augmentation or without BiLSTM. The model achieved the following accuracy on the MNIST test data: CNN Model Accuracy: Before Augmentation: 98.0%. After Augmentation: 98.5%; CNN-BiLSTM Model Accuracy: Before Augmentation: 98.0%. After Augmentation: 98.7%. These results highlight the effectiveness of the hybrid approach in enhancing handwritten digit recognition perfor-mance. This research contributes to the development of more accurate and robust deep learning models for handwritten image processing
COMPARISON OF THE PERFORMANCE OF SVM, RANDOM FOREST, AND NEURAL NETWORK ALGORITHMS IN SENTIMENT ANALYSIS OF OPENAI APPLICATION REVIEWS ON THE GOOGLE PLAY STORE Latif, Ahmad; Yugi, Muhtyas; Utomo, Fandy Setyo; Hariguna, Taqwa
JIPI (Jurnal Ilmiah Penelitian dan Pembelajaran Informatika) Vol 11, No 1 (2026)
Publisher : STKIP PGRI Tulungagung

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.29100/jipi.v11i1.7793

Abstract

This study compares the performance of three machine learning algo-rithms—Support Vector Machine (SVM), Random Forest (RF), and Neural Network (NN)—in sentiment analysis of user reviews for the OpenAI application on the Google Play Store. The primary objective of this study is to evaluate the effectiveness of each algorithm in clas-sifying user reviews into three sentiment categories: positive, negative, and neutral. The dataset used consists of user reviews of the OpenAI application, collected directly from the Google Play Store. Model per-formance was evaluated using accuracy, precision, recall, and F1-score metrics. The results indicate that the Neural Network algorithm achieved the best overall performance in terms of accuracy and F1-score. SVM demonstrated competitive performance, particularly in classifying positive and neutral sentiments, while Random Forest showed an advantage in terms of precision but performed lower over-all, especially in classifying negative sentiments. Therefore, the Neural Network is considered the most effective algorithm for sentiment analysis tasks in this study
Rectified Linear Units and Adaptive Moment Estimation Optimizer on ANN with Saved Model Prediction to Improve The Stock Price Prediction Framework Performance Sekhudin, Sekhudin; Purwati, Yuli; Utomo, Fandy Setyo; Azmi, Mohd Sanusi; Subarkah, Pungkas
ILKOM Jurnal Ilmiah Vol 15, No 2 (2023)
Publisher : Prodi Teknik Informatika FIK Universitas Muslim Indonesia

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.33096/ilkom.v15i2.1586.271-282

Abstract

A stock is a high-risk, high-return investment product. Prediction is one way to minimize risk by estimating future prices based on past data. There are limitations to solving the stock prediction problem from previous research: limited stock data, practical aspects of application, and less than optimal stock price prediction results. The main objective of this study is to improve the prediction performance by formulating and developing the stock price prediction framework. Furthermore, the research provides a stock price prediction framework that can produce better prediction results than the previous study with fast computation time. The proposed framework deals with data generation, pre-processing and model prediction. In further, the proposed framework includes two prediction methods for predicting stock closing prices: stored model prediction and current model prediction. This study uses an artificial neural network with Rectified Linear Units as an activation function and Adam Optimizer to predict stock prices. The model we have built for each forecasting method shows a better MAPE value than the model in previous studies. Previous research showed that the lowest MAPE was 1.38% for TLKM shares and 0.81% for BBRI. Our proposed framework based on the stored model prediction method shows a MAPE value of 0.67% for TLKM shares and 0.42% for BBRI. While the current model prediction method shows a MAPE value of 0.69% for TLKM shares and 0.89% for BBRI. Furthermore, the stored model prediction method takes 1.0 seconds to process a single prediction request, while the current model prediction takes 220 seconds.
A Comprehensive Evaluation of CatBoost and LightGBM Algorithms for Honorarium Prediction on Categorical Datasets with Class Imbalance Slamet Widodo; Fandy Setyo Utomo; Berlilana
JUITA: Jurnal Informatika JUITA Vol. 13 Issue 3, November 2025
Publisher : Department of Informatics Engineering, Universitas Muhammadiyah Purwokerto

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.30595/juita.v13i3.27363

Abstract

Determining income, including honoraria in the academic environment, is often done manually and subjectively, necessitating a predictive model to objectively determine the honorarium amount. However, the development of the prediction model faces challenges due to the dataset's characteristics, which include categorical data and an imbalanced class distribution. This research aims to evaluate the predictive performance and computational resource efficiency of the CatBoost and LightGBM algorithms in predicting honorariums. The dataset used includes 58,332 actual honorarium data of employees from higher education institution "A" in Purwokerto for the period from January 2024 to February 2025. The methods used include data preprocessing, dataset splitting using Stratified Splitting, modeling with CatBoost, LightGBM, Random Forest, Neural Network, and Linear Regression, as well as evaluation using MSE, RMSE, MAE, R² metrics, and computational resources (execution time, memory, CPU time). LightGBM achieved an RMSE of 665.960 and an R² of 0.54, while recording the lowest memory usage at only 2.67 MB. CatBoost produced an RMSE of 667.395 and an R² of 0.53, excelling in processing categorical features without one-hot encoding. Meanwhile, Linear Regression showed the lowest accuracy and high memory usage. These results confirm that LightGBM is the most optimal choice for fast, efficient, and accurate honorarium predictions. However, this research is limited to testing in a laboratory environment. Further research is recommended to implement direct integration with an active database and the integration of information retrieval methods to enhance the effectiveness and security of real-time honorarium predictions, as well as to integrate interpretability methods such as SHAP to improve decision-making transparency.
Studi Komparasi Kinerja Algoritma AdaBoost dan CatBoost dalam Prediksi Perilaku Pembelian Pelanggan Kafilla, Princess Iqlima; Utomo, Fandy Setyo; Karyono, Giat
Building of Informatics, Technology and Science (BITS) Vol 7 No 4 (2026): March 2026
Publisher : Forum Kerjasama Pendidikan Tinggi

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.47065/bits.v7i4.7947

Abstract

Customer purchase behavior is a crucial factor in the development of effective marketing strategies. By leveraging predictive analytics, businesses can personalize recommendations, optimize marketing campaigns and improve user experience, ultimately contributing to increased conversion rates and customer retention. This research compares the performance of AdaBoost and CatBoost algorithms in predicting customer purchase behavior. The dataset used includes demographic attributes and customer behavior history, allowing for comprehensive analysis. The results showed that CatBoost performed better overall with an accuracy of 94%, while AdaBoost showed higher recall and F1-score values in the positive class. This study concludes that both algorithms have reliability in predicting customer behavior, where CatBoost is superior in handling categorical features, while AdaBoost offers good adaptability on simpler datasets. As a next step, future research can explore the implementation of these models in real-time scenarios.
Co-Authors Adiya, Az Zahra Dwi Nur Afit Ajis Solihin Aisha Hukama Setyowati Aji Saeful Aji Septa, Adrian Ajis Solihin, Afit Amar Al Farizi Anas Nur Khafid Anggini, Melisa Anggraeni, Mutia Dwi Anggraini, Nova Anggriani, Epri Anies Indah Hariyanti Azhari Shouni Barkah Azmi, Mohd Sanusi Bagus Adhi Kusuma Bahari, Aris Ridky Setiya Bahari, Aris Rifki Setiya Baihaqi, Wiga Maulana Balit, Muhamad Naufal Burhanuddin Berlilana Berlilana Berlilana Burhanuddin Balit, Muhamad Naufal Churil Aeni, Agustina Chyntia Raras Ajeng Widiawati Chyntia Raras Ajeng Widiawati Darmono Dedi Purwanto, Dedi Didi Prasetyo Dwi Krisbiantoro, Dwi Dwi Putriana Nuramanah Kinding Dzaky Candy Fahrezy Fadhilah, Siti Nur Fajar Rohmattulloh Faradina Faradina Febriansyah Husni Adiatma Giat Karyono Giat Karyono Gilang Miftkahul Fahmi Fahmi Had, Iqbaluddin Syam Hanif Hidayatulloh Hendra Marcos, Hendra hidayatulloh, hanif Ilham, Rifqi Arifin Imam Tahyudin Imam Tahyudin Indriyani, Ria Jamie Mayliana Alyza Kafilla, Princess Iqlima Kusuma, Bagus Adhi Kusuma, Velizha Sandy Lasmedi Afuan Latif, Ahmad Lubna, Zuhriyatul Lukita, Dita Maulana Baihaqi, Wiga Mohd Fairuz Iskandar Othman Mohd Nazrin Muhammad Mohd Sanusi Azmi Muaziz, Imam Muhamad Naufal Burhanuddin Balit Muhtyas Yugi Muhtyas Yugi Murtiyoso Murtiyoso Nandang Hermanto Nanna Suryana Nikmah Trinarsih Nugroho, Khabib Adi Nur Cholis Romadhon Octavia, Annisa Suci Prayoga, Fandhi Dhuga Pungkas Subarkah Purbo, Yevi Septiray Purwidiantoro, Moch. Hari Pyawai, Hero Galuh R. Vitto Mahendra Putranto Ramadhan, Aziz Ramadhan, Rio Fadly Rifqi Arifin Ilham RR. Ella Evrita Hestiandari Rujianto Eko Saputro Sagita, Selvi Samsul Arifin Sarmini - Sarmini Sarmini Sarmini Sekhudin, Sekhudin Setiabudi, Rizki Setiawan, Ito Shafira, Lulu Shendy Filanzi Slamet Widodo Slamet Widodo Sofa, Nur Sri Hartini Sugianto, Dwi Suryana, Nanna Taqwa Hariguna Taqwa Hariguna Titi Safitri Maharani Trinarsih, Nikmah Turino, Turino Utomo, Dadang Wahyu Wahid, Arif Mu'amar Wanti, Linda Perdana Wibisono, Arif Cahyo Wiga Maulana Baihaqi Yugi, Muhtyas Yuli Purwat Yuli Purwati Yuli Purwati Yulianto, Koko Edy