Article Per Year (5 Year)
p-Index From 2021 - 2026
0.23
P-Index
This Author published in this journals
All Journal Jurnal Ilmu Ekonomi, Pendidikan dan Teknik (IDENTIK)
Nevitya Elmaira Nurjannah
Universitas Negeri Surabaya
Religion Civil Engineering, Building, Construction & Architecture Economics, Econometrics & Finance Education Engineering

Published : 1 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 1 Documents
Search

 1 
Analisis Perbandingan Kualitas Jawaban Pada Qa Berbasis Rag : Kombinasi Zero-Shot Instruction Prompting Dan Self-Verification Nevitya Elmaira Nurjannah; Cendra Devayana Putra
Jurnal Ilmu Ekonomi, Pendidikan dan Teknik Vol. 3 No. 5 (2026): IDENTIK - September
Publisher : CV. SINAR HOWUHOWU

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.70134/identik.v3i5.1962

Abstract

Retrieval-Augmented Generation (RAG) reduces hallucinations by grounding language model (LM) responses in the context of retrieved results, but does not automatically guarantee evidence-based (faithful) responses. This study examines the effect of combining zero-shot instruction prompting and self-verification on the faithfulness of responses to answerable questions, across four RAG pipeline configurations: (1) RAG, (2) RAG + prompting, (3) RAG + self-verification, and (4) RAG + prompting + self-verification, which were tested on three scales of the Qwen3 language model (0.6B, 4B, 8B) using the SQuAD v2.0 dataset. A total of 100 queries were selected via stratified random sampling from the validation split to avoid topic bias. Faithfulness was measured at the claim level using an NLI model (DeBERTa-v3-large). The results show that basic RAG (configuration 1) achieved the highest average faithfulness score, while configurations 2–4 (with prompting and/or self-verification) showed relatively similar and lower scores. In SLM, adding self-verification lowered faithfulness the most, while in LLM the score remained relatively stable across configurations. A retrieval-quality control analysis indicated that this decline was linked to the generator's capacity rather than retrieval quality. These findings suggest that the combination of prompting and self-verification does not automatically improve the quality of evidence-based answers, and its benefits depend on the adequacy of the language model's capacity.  
Co-Authors Cendra Devayana Putra