Kinetik: Game Technology, Information System, Computer Network, Computing, Electronics, and Control
Vol. 7, No. 1, February 2022

A Study on Visual Understanding Image Captioning using Different Word Embeddings and CNN-Based Feature Extractions

Dhomas Hatta Fudholi (Universitas Islam Indonesia)
Annisa Zahra (Universitas Islam Indonesia)
Royan Abida N. Nayoan (Universitas Islam Indonesia)



Article Info

Publish Date
10 Mar 2022

Abstract

Image captioning is a task that can provide a description of an image in natural language. Image captioning can be used for a variety of applications, such as image indexing and virtual assistants. In this research, we compared the performance of three different word embeddings, namely, GloVe, Word2Vec, FastText and six CNN-based feature extraction architectures such as, Inception V3, InceptionResNet V2, ResNet152 V2, EfficientNet B3 V1, EfficientNet B7 V1, and NASNetLarge which then will be combined with LSTM as the decoder to perform image captioning. We used ten different household objects (bed, cell phone, chair, couch, oven, potted plant, refrigerator, sink, table, and tv) that were obtained from MSCOCO dataset to develop the model. Then, we created five new captions in Bahasa Indonesia for the selected images. The captions might contain details about the name, the location, the color, the size, and the characteristics of an object and its surrounding area. In our 18 experimental models, we used different combination of the word embedding and CNN-based feature extraction architecture, along with LSTM to train the model. As the result, models that used the combination of Word2Vec + NASNetLarge performed better in generating Indonesian captions than the other models based on BLEU-4 metric.

Copyrights © 2022






Journal Info

Abbrev

kinetik

Publisher

Subject

Computer Science & IT Control & Systems Engineering Electrical & Electronics Engineering Energy Engineering

Description

Kinetik: Game Technology, Information System, Computer Network, Computing, Electronics, and Control was published by Universitas Muhammadiyah Malang. journal is open access journal in the field of Informatics and Electrical Engineering. This journal is available for researchers who want to improve ...