Exploring Rater Perceptions in Argumentative Writing Assessment: A Qualitative Case Study of Experience, Nativeness, and Rhetorical Variability - Gül KARAAGAC ZAN (Journal of Language, Literature, and Educational Research, 2026)
Purpose of the study: This study aims to explore how linguistic background and teaching experience influence evaluation criteria, rhetorical interpretation, and scoring decisions during assessment of argumentative writing. It also examines how raters perceive rhetorical variation in essays written by students from different cultural backgrounds. Methodology: This study employed an exploratory qualitative case study design using Concurrent Think-Aloud Protocols (TAP). Participants consisted of four English as a Foreign Language instructors with different linguistic backgrounds and teaching experience. Data were collected through audio-recorded assessment sessions and analyzed using structural coding and pattern coding to identify recurring evaluation themes and scoring behaviors. Main Findings: Findings revealed that native English-speaking raters emphasized coherence, audience awareness, and rhetorical organization, whereas non-native English-speaking raters focused primarily on grammatical accuracy and content. Rhetorical variation influenced evaluation decisions, producing differences in assigned scores across participants. Teaching experience and linguistic background contributed to variation in assessment criteria, rhetorical interpretation, and scoring practices during evaluation of argumentative writing. Novelty/Originality of this study: This study integrates linguistic nativeness, teaching experience, and rhetorical variation within one qualitative investigation using Think-Aloud Protocols. Unlike previous studies emphasizing score comparison, it explores cognitive processes underlying writing assessment decisions. Findings provide deeper understanding of how rater characteristics shape evaluation practices and support development of more inclusive and reliable writing assessment in multilingual educational contexts.