Pemodelan Lip Reading Bahasa Indonesia Berbasis Visem Menggunakan VGG16 serta Jaro-Winkler Similarity dan Bigram

Henry Wicaksono (Program Studi Teknik Informatika, Universitas Kristen Petra Surabaya)
Liliana Liliana (Program Studi Teknik Informatika, Universitas Kristen Petra Surabaya)
Alvin Nathaniel Tjondrowiguno (Program Studi Teknik Informatika, Universitas Kristen Petra Surabaya)

Jurnal Infra
Vol 10, No 2 (2022)

Lip reading is a technique used to understand spoken words through visual representation of lip movements. Lip reading has many uses, such as aids for laryngectomy patients and aids for people with hearing disabilities. A research shows that 2.6% of Indonesia’s population has a hearing disability. Thus, lip reading can be a relevant solution in Indonesia. This study aims to model a viseme-based Indonesian lip reading system. The method used in this research is VGG16 which is used as a classifier and Jaro-Winkler similarity and bigram (JW-bigram) which is used as a decoder. The dataset used consists of 25 Indonesian sentences composed of 50 different words and spoken by 12 speakers. The results showed that the lip reading system made using VGG16 and JW-bigram was more effective in terms of accuracy and speed compared to other methods combinations.