Natural language description of images using hybrid recurrent neural network

Md. Asifuzzaman Jishan (University of Liberal Arts Bangladesh (ULAB))
Khan Raqib Mahmud (University of Liberal Arts Bangladesh (ULAB))
Abul Kalam Al Azad (University of Liberal Arts Bangladesh (ULAB))

International Journal of Electrical and Computer Engineering
Vol 9, No 4: August 2019

We presented a learning model that generated natural language description of images. The model utilized the connections between natural language and visual data by produced text line based contents from a given image. Our Hybrid Recurrent Neural Network model is based on the intricacies of Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Bi-directional Recurrent Neural Network (BRNN) models. We conducted experiments on three benchmark datasets, e.g., Flickr8K, Flickr30K, and MS COCO. Our hybrid model utilized LSTM model to encode text line or sentences independent of the object location and BRNN for word representation, this reduced the computational complexities without compromising the accuracy of the descriptor. The model produced better accuracy in retrieving natural language based description on the dataset.