Article Per Year (5 Year)
p-Index From 2021 - 2026
0.23
P-Index
This Author published in this journals
All Journal Journal of Information Systems and Informatics
Daeng Ahmad Nurdin
Mercu Buana University
Computer Science & IT

Published : 1 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 1 Documents
Search

 1 
Benchmarking Embedding Models for POJK Regulatory Retrieval in Indonesian Insurance Documents Daeng Ahmad Nurdin; Bambang Jokonowo; Bagus Priambodo
Journal of Information System and Informatics Vol 8 No 5 (2026): October
Publisher : Asosiasi Doktor Sistem Informasi Indonesia

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.63158/journalisi.v8i5.1781

Abstract

Indonesian insurance regulations are clause-dense and terminologically specific, making manual retrieval of applicable provisions difficult. This study presents a retrieval-only benchmark comparing embedding and lexical methods on POJK 26/2025 and POJK 36/2025. The corpus contains 181 context chunks and 181 synthetic Indonesian queries, each assigned one primary relevant chunk. Queries were generated in JSON mode using Gemma 4 E4B via Ollama (ollama:gemma4:e4b), with malformed outputs retried and heuristic fallbacks prohibited. Evaluation comprised a four-model local dense comparison and a separate native BGE-M3/BM25 ablation using exact exhaustive ranking. The local comparison reports MRR and Hit rates, while the native ablation additionally reports NDCG@10, MAP@10, bootstrap confidence intervals, paired tests, and rank bins. Qwen3 Embedding achieved the highest local MRR (0.8075). Under a prespecified descriptive composite rule combining MRR, Hit rates, and measured latency, BGE-M3 ranked first. In the native ablation, BGE-M3 combined retrieval achieved NDCG@10 of 0.8810. A practitioner-validated subset audit and artifact-backed qualitative analysis supplemented the quantitative results. Findings are limited to this two-document benchmark and do not evaluate answer generation, citation correctness, or end-to-end RAG faithfulness.
Co-Authors Bagus Priambodo Bambang Jokonowo