A Multimodal Deep Learning Framework for Herbal Plant Classification with Large Language Model Integration - Shalsabila Dwi Aprilia; Nunu Nurdiana; Harun Sujadi (Politeknik Negeri Cilacap, 2026)
Accurate identification of herbal plants remains challenging due to high species diversity and the need for specialized botanical knowledge. This study proposes a multimodal artificial intelligence framework that integrates deep learning-based image classification with a large language model to provide both plant identification and descriptive information. The objective is to develop a more informative and practical system that extends beyond conventional classification approaches. The method employs a ResNet50V2 architecture with transfer learning to classify herbal plant images, combined with a large language model to automatically generate descriptions of plant benefits. The model was trained on 19,213 images across 131 plant classes. Experimental results demonstrate an accuracy of 89.43%, with a macro F1-score of 0.85 and a weighted F1-score of 0.89, indicating strong and consistent performance across classes. The integration of visual recognition and language generation enables the system to deliver richer, user-oriented outputs compared to existing methods that only provide classification labels. In addition, the system achieves real-time inference performance, making it suitable for web-based applications. This study contributes a scalable and comprehensive multimodal framework for herbal plant identification, highlighting the effectiveness of combining computer vision and large language models to enhance both accuracy and usability.