Sistemasi: Jurnal Sistem Informasi
Vol 15, No 5 (2026): Sistemasi: Jurnal Sistem Informasi

SAST Implementation for Evaluating LLM-Generated Code Quality using Prompt Engineering

Muhammad Luthfi Abdillah (Universitas '
Aisyiyah Yogyakarta)

Tikaridha Hardiani (Universitas '
Aisyiyah Yogyakarta)



Article Info

Publish Date
26 May 2026

Abstract

The use of Large Language Models (LLMs) for generating programming code has become increasingly widespread; however, the quality of the generated output heavily depends on the instructions or prompts provided. This study aims to evaluate the influence of prompt engineering techniques on the quality of non-functional code generated by LLMs. The research employed a quantitative experimental approach involving five Python game development tasks using four prompt variations: zero-shot, few-shot, chain-of-thought, and role-based prompting. A total of 200 code snippets were analyzed using Static Application Security Testing (SAST) with the DeepSource tool to detect issues across seven categories: secrets, bug risk, anti-pattern, security, performance, style, and documentation. The results indicate that few-shot prompting produced the lowest total number of issues overall (1,328 out of 6,932 issues), demonstrating particular advantages in the anti-pattern and performance categories. However, this technique also recorded a higher number of critical issues (3 issues) compared to zero-shot and role-based prompting (1 issue each), indicating a trade-off between the overall volume of issues and the severity of certain issues. Role-based prompting generated the highest number of issues (2,516 issues), particularly in the style and documentation categories. This study recommends few-shot prompting as a foundational approach for AI-assisted software development and highlights the importance of integrating SAST into CI/CD pipelines to ensure code security and quality.

Copyrights © 2026






Journal Info

Abbrev

stmsi

Publisher

Subject

Computer Science & IT Electrical & Electronics Engineering

Description

Sistemasi adalah nama terbitan jurnal ilmiah dalam bidang ilmu sains komputer program studi Sistem Informasi Universitas Islam Indragiri, Tembilahan Riau. Jurnal Sistemasi Terbit 3x setahun yaitu bulan Januari, Mei dan September,Focus dan Scope Umum dari Sistemasi yaitu Bidang Sistem Informasi, ...