Image Captioning Approach for Household Environment Visual Understanding - Dhomas Hatta Fudholi (IJISTECH (International Journal of Information System and Technology), 2021)
Image captioning task generates a description from an image in a form of a sentence. It holds various critical usage in different applications and domains, such as indexing in retrieval system, by capturing semantic information within the images. In the area of live quality support system, an image captioning model may be used by visual impaired people to achieve visual understanding of their surroundings. In this paper, we present a novel image captioning model that aims to give visual understanding on household environment. To develop the model, we use five different household objects (sinks, chairs, tables, beds, couches) from MS COCO datasets. We create three new captions, in Bahasa Indonesia, for the selected data. The captions describe the name, the color, the position/location, the size, and the type/characteristic of the related object and its close surrounding. InceptionV3 and LSTM architecture is used to train the model with GloVe as the word embedding. In this study, our developed image captioning model can generate caption well and achieved BLEU-1 score of 0.502033, BLEU-2 score of 0.312539, BLEU-3 score of 0.193333, BLEU-4 score of 0.106111, METEOR score of 0.183193, ROUGE-L score of 0.358339, and CIDEr score of 0.348903.