JOURNAL OF INFORMATION SYSTEM RESEARCH (JOSH)
Vol 8 No 1 (2026): October 2026

Sistem Lost and Found Berbasis Vision-Language Retrieval dengan Explainable AI untuk Lingkungan Smart Campus

Enrico Samuel Djimesha (Universitas Klabat, Minahasa Utara)
Regi Fernando Najoan (Universitas Klabat, Minahasa Utara)
Argha Orion Silitonga (Universitas Klabat, Minahasa Utara)
Wilsen Grivin Mokodaser (Universitas Klabat, Minahasa Utara)
George Morris William Tangka (Universitas Klabat, Minahasa Utara)



Article Info

Publish Date
11 Oct 2026

Abstract

Evidence for combined image-text retrieval in campus lost-and-found systems remains limited, and zero-shot CLIP's ability to distinguish item attributes and identities is uncertain. This study evaluates CLIP ViT-B/32 on an item-image collection covering eight categories. The controlled multimodal evaluation uses 404 images, 85 metadata-defined relevance groups as proxy ground truth, and 237 leave-one-out queries. Separate image-only validation uses 387 images from 79 provenance-based Object IDs after duplicate removal and quality control; each Object ID contains multiple images of the same physical item. In Object-ID validation, CLIP achieved Recall@1 of 0.674, Recall@10 of 0.935, and mAP of 0.419 (95% object-cluster bootstrap CI: 0.369–0.469). In multimodal evaluation, image-only achieved mAP 0.624, text-only 0.409, and image+text reached the best mAP of 0.732 at α=0.5, selected in four of five group cross-validation folds. Ablation showed that the single fused-index outperformed the three-index configuration (mAP 0.715 versus 0.692); therefore, three-index is not claimed as an accuracy improvement. Direct Indonesian input to English CLIP achieved mAP 0.163, compared with 0.363 using multilingual CLIP and 0.409 through translation. Attention Rollout is positioned only as a supporting visualization because its deletion-curve AUC of 0.871 lacks a random-occlusion baseline. These results indicate the benefit of image-text fusion under controlled evaluation but do not establish performance for free-form user descriptions or production readiness.

Copyrights © 2026






Journal Info

Abbrev

josh

Publisher

Subject

Computer Science & IT Decision Sciences, Operations Research & Management

Description

Artikel yang dimuat melalui proses Blind Review oleh Jurnal JOSH, dengan mempertimbangkan antara lain: terpenuhinya persyaratan baku publikasi jurnal, metodologi riset yang digunakan, dan signifikansi kontribusi hasil riset terhadap pengembangan keilmuan bidang teknologi dan informasi. Fokus Journal ...