A Multimodal Deep Learning Framework for Herbal Plant Classification with Large Language Model Integration - Shalsabila Dwi Aprilia; Nunu Nurdiana; Harun Sujadi (Journal of Innovation Information Technology and Application (JINITA), 2026)
Accurate identification of herbal plants remains challenging due to high species diversity and the need for specialized botanical knowledge. This study proposes a multimodal artificial intelligence framework that integrates deep learning-based image classification with a large language model to provide both plant identification and descriptive information. The objective is to develop a more informative and practical system that extends beyond conventional classification approaches. The method employs a ResNet50V2 architecture with transfer learning to classify herbal plant images, combined with a large language model to automatically generate descriptions of plant benefits. The model was trained on 19,213 images across 131 plant classes. Experimental results demonstrate an accuracy of 89.43%, with a macro F1-score of 0.85 and a weighted F1-score of 0.89, indicating strong and consistent performance across classes. The integration of visual recognition and language generation enables the system to deliver richer, user-oriented outputs compared to existing methods that only provide classification labels. In addition, the system achieves real-time inference performance, making it suitable for web-based applications. This study contributes a scalable and comprehensive multimodal framework for herbal plant identification, highlighting the effectiveness of combining computer vision and large language models to enhance both accuracy and usability.