Journal of Innovation Information Technology and Application (JINITA)
Vol 8 No 1 (2026): JINITA, June 2026

A Multimodal Deep Learning Framework for Herbal Plant Classification with Large Language Model Integration

Shalsabila Dwi Aprilia (Department of Informatics, Faculty of Engineering, Universitas Majalengka, Majalengka, West Java, Indonesia)
Nunu Nurdiana (Universitas Majalengka)
Harun Sujadi (Universitas Majalengka)



Article Info

Publish Date
30 Jun 2026

Abstract

Accurate identification of herbal plants remains challenging due to high species diversity and the need for specialized botanical knowledge. This study proposes a multimodal artificial intelligence framework that integrates deep learning-based image classification with a large language model to provide both plant identification and descriptive information. The objective is to develop a more informative and practical system that extends beyond conventional classification approaches. The method employs a ResNet50V2 architecture with transfer learning to classify herbal plant images, combined with a large language model to automatically generate descriptions of plant benefits. The model was trained on 19,213 images across 131 plant classes. Experimental results demonstrate an accuracy of 89.43%, with a macro F1-score of 0.85 and a weighted F1-score of 0.89, indicating strong and consistent performance across classes. The integration of visual recognition and language generation enables the system to deliver richer, user-oriented outputs compared to existing methods that only provide classification labels. In addition, the system achieves real-time inference performance, making it suitable for web-based applications. This study contributes a scalable and comprehensive multimodal framework for herbal plant identification, highlighting the effectiveness of combining computer vision and large language models to enhance both accuracy and usability.

Copyrights © 2026






Journal Info

Abbrev

jinita

Publisher

Subject

Computer Science & IT Decision Sciences, Operations Research & Management Engineering

Description

Software Engineering, Mobile Technology and Applications, Robotics, Database System, Information Engineering, Interactive Multimedia, Computer Networking, Information System, Computer Architecture, Embedded System, Computer Security, Digital Forensic Human-Computer Interaction, Virtual/Augmented ...