Article Per Year (5 Year)
p-Index From 2020 - 2025
0.23
P-Index
This Author published in this journals
All Journal LISANIA: Journal of Arabic Education and Literature
Chafidloh Rizqiyah
Unknown Affiliation
Religion Education Languange, Linguistic, Communication & Media Social Sciences

Published : 1 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 1 Documents
Search

 1 
Assessing Copilot’s Semantic Depth in Classical Arabic: A Mixed-Methods Evaluation Using Alfiyah ibn Malik and Nadham Al-Imrithy Zaimah, Nely Rahmawati; Syamsul Hadi; Chafidloh Rizqiyah; Estu, Risty Kamila Wening; Akhmad Roja Badrus Zaman
LISANIA: Journal of Arabic Education and Literature Vol. 9 No. 1 (2025)
Publisher : Fakultas Tarbiyah dan Ilmu Keguruan IAIN Salatiga

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.18326/lisania.v9i1.41-59

Abstract

It was rather surprising that Windows users readily embraced Copilot, even trusting it with translation projects. Surely, not many users would trust its accuracy in providing cross-language explanations for prompts solely based on the developer's claims. Building on that, this research aimed to test it in a manner distinct from other assessments. Researchers evaluated how accurately Copilot interpreted and understood the advanced Arabic prose from the intricate works of Alfiyah ibn Malik and Nadham Al-Imrithy. The aim was to understand Copilot’s strengths and weaknesses in terms of literal accuracy, terminological-analogical mastery, and contextual depth. Using a mixed-method approach under the Collect-Measure-Repeat (CMR) framework of Responsible AI, the researchers conducted qualitative performance assessments with three experts and quantitative evaluations using METEOR (Metric for Evaluation of Translation with Explicit Ordering). The results showed that although Copilot had no issues comprehending and translating simple Arabic commands, especially word-for-word, it struggled with contextual understanding for many of the complex texts and displayed numerous inconsistencies when the instructions were vague. Copilot's performance issues in context saturation were evident during iterative phases. This led to the conclusion that, while Copilot is competent enough to attempt the challenging task of interpreting complex linguistic structures, it still needs human assistance and cross-references.
Co-Authors Akhmad Roja Badrus Zaman Risti Kamila Wening Estu Syamsul Ph.D M.A. Hadi Zaimah, Nely Rahmawati