Article Per Year (5 Year)
p-Index From 2021 - 2026
0.61
P-Index
This Author published in this journals
All Journal Building of Informatics, Technology and Science JOURNAL OF INFORMATION SYSTEM RESEARCH (JOSH) Branding: Jurnal Manajemen & Bisnis
Sajarwo Anggai
Universitas Pamulang, Tangerang Selatan
Computer Science & IT Decision Sciences, Operations Research & Management Economics, Econometrics & Finance

Published : 3 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 3 Documents
Search

 1 
Comparative Evaluation of Local Llama and Mistral Models in RAG for Helpdesk Documentation Septian Pratama; Sajarwo Anggai; Murni Handayani
Building of Informatics, Technology and Science (BITS) Vol 8 No 1 (2026): June 2026
Publisher : Forum Kerjasama Pendidikan Tinggi

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.47065/bits.v8i1.8895

Abstract

The increasing volume of technical documentation and repetitive support requests at PT XYZ has made it difficult for helpdesk personnel to retrieve accurate information efficiently and consistently. The 114 internal applications that PT XYZ oversees have different documentation, troubleshooting protocols, and frequently asked questions, which leads to dispersed knowledge sources and drawn-out problem solving. This study uses locally deployed Large Language Models (LLMs), specifically Llama and Mistral, to design and assess a Retrieval-Augmented Generation (RAG) based chatbot for internal helpdesk knowledge retrieval in order to solve this issue. The suggested solution combines generative language models with semantic document retrieval in PostgreSQL using PGVector. The study compares the effectiveness of local LLMs in RAG and non-RAG configurations, employing 1,068 internal question-answer pairs for knowledge retrieval and 214 evaluation questions for performance assessment. ROUGE, BLEU, and Cosine Similarity measures are used for evaluation. According to experimental findings, RAG greatly enhances both models' performance. While Mistral improved from 0.0984 to 0.8983, Llama ROUGE-1 score rose from 0.1710 to 0.7345. With a ROUGE-1 score of 0.8983, a BLEU-1 score of 0.8161, and a Cosine Similarity score of 0.8916, Mistral with RAG performed the best of all setups. These results show that integrating RAG with locally installed LLMs greatly improves contextual correctness and response relevance, enabling safe and scalable implementation for enterprise helpdesk knowledge management.
Optimasi Akurasi Jawaban Aplikasi Chatbot Layanan Pelanggan dengan Metode RAGRetrieval-Augmented Generation Dhaman Dhaman; Sajarwo Anggai; Arya Adhyaksa Waskita
Journal of Information System Research (JOSH) Vol 6 No 4 (2025): July 2025
Publisher : Forum Kerjasama Pendidikan Tinggi (FKPT)

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.47065/josh.v6i4.8048

Abstract

This research addresses the issue of low answer accuracy in chatbot systems based on Large Language Models (LLMs) when responding to questions derived from customer service documents. To overcome this problem, the Retrieval-Augmented Generation (RAG) method is applied to improve the quality of responses by adding relevant context from external documents. Three LLM models used in this study are LLaMA3.1 8B, LLaMA3.2 1B, and LLaMA3.2 3B from Meta AI. Evaluation is conducted using automatic ROUGE metrics (ROUGE-1, ROUGE-2, and ROUGE-L) and manual human evaluation assessing accuracy, relevance, and hallucination. This research contributes to the development of more reliable question-answering systems based on LLMs enhanced with external contextual documents related to customer service information. The results show a significant improvement across all models after applying the RAG method. ROUGE F1-scores increased consistently, with Llama3.1:8b showing the highest gain (from 0.12 to 0.58 on ROUGE-1). Human evaluation also confirmed improvements in accuracy (up to +2.73 points) and reductions in hallucination (up to −2.63 points). These improvements were evident not only in larger models but also in smaller ones, indicating that the benefits of RAG are not dependent on model size. In conclusion, RAG is highly effective in enhancing the accuracy and reliability of chatbot responses, especially in document-based question-answering scenarios. By leveraging contextual information from external documents, the system produces more factual, relevant, and hallucination-free responses. RAG has proven to be an effective approach for enhancing the response quality of LLM, including those with smaller parameter sizes.
TOPIC ANALYSIS OF STUDENT FEEDBACK ON LEARNING MANAGEMENT SYSTEMS USING BERTOPIC: A COMPARATIVE STUDY OF INDOBERT, DISTILBERT, AND SBERT Aldi Aditya Perdana; Sajarwo Anggai; Winarni
Branding: Jurnal Manajemen dan Bisnis Vol 4 No 2 (2025): BRANDING: Jurnal Manajemen dan Bisnis
Publisher : UIN Sunan Gunung Djati Bandung

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.15575/jb.v4i2.54192

Abstract

The widespread adoption of Learning Management Systems (LMS) in digital education has generated large volumes of student feedback in the form of unstructured free-text data, making manual analysis increasingly impractical. This study aims to identify the dominant themes emerging from student feedback on LMS platforms and to compare the performance of different Transformer-based embedding models in topic modeling tasks. The proposed approach employs BERTopic with three embedding models, namely IndoBERT, DistilBERT, and Sentence-BERT (SBERT). Student feedback data were collected from an institutional LMS and processed through text preprocessing, embedding generation, and topic modeling stages. Model performance was evaluated using multiple coherence metrics (c_v, c_npmi, u_mass, and c_uci), topic diversity, and the proportion of outlier documents. The results indicate that the IndoBERT-family model achieves the highest coherence scores, particularly in c_v and c_npmi, suggesting superior semantic consistency of the generated topics. DistilBERT produces the lowest proportion of outliers but yields a more limited number of topics, while SBERT demonstrates a balanced performance between topic quality and thematic diversity. These findings highlight that the choice of embedding model significantly influences the quality of topic modeling outcomes for Indonesian-language student feedback data.
Co-Authors Aldi Aditya Perdana Arya Adhyaksa Waskita Dhaman Dhaman Murni Handayani Septian Pratama Winarni