Advanced UI test automation (AUTA) for BIOS validation using OpenCV and OCR

Eissa Abdullah Awadh Mohammed (Universiti Malaysia Perlis (UniMAP))
Muslim Mustapa (Universiti Malaysia Perlis (UniMAP))
Hasliza Rahim (Universiti Malaysia Perlis (UniMAP))
Mohd Natashah Norizan (Universiti Malaysia Perlis (UniMAP))

Indonesian Journal of Electrical Engineering and Computer Science
Vol 23, No 3: September 2021

Basic input output system (BIOS) validation is performed on both graphical user interface (GUI) and command-line interface (CLI) by a test engineer. Keyboard and mouse are used to insert test cases commands into system under test (SUT). Test engineer monitors test cases progress on a monitor for validation. This method is time-consuming and relatively more expensive than automation. In this project we designed an independent automation system that able to mimic human interaction in BIOS validation. The approach can be divided into two main parts. The first part is the input device to enter commands into SUT and the second part is the advanced image recognizer. The keyboard and mouse emulator is used as an input device to reproduce test commands and send them to an SUT. The image analyzer algorithm is developed using OpenCV and optical character recognizer (OCR) tools to help automate some test challenges. Our result shows that advanced user interface (UI) test automation (AUTA) can perform a 125 test cases within 5 hours compared to 48 hours for a human to complete the job.