Motion detection is a key element in modern surveillance systems to ensure optimal security. However, the accuracy of motion detection is often a challenge under various environmental conditions. This research investigates the use of the Viola-Jones method and labeling algorithm as a solution to improve object detection accuracy by sampling 15 videos that record various environmental conditions, including indoors, outdoors, and at night. The Viola-Jones method is implemented for face detection as a first step in human object identification, while the labeling algorithm is used to refine and validate the detection results in more detail. Experimental results show that combining the two methods succeeded in increasing object detection accuracy. Of the 15 videos analyzed, only 4 videos experienced inaccurate detection results, while the other 11 videos managed to get accurate results. Evaluation of system performance using Precision, Recall, and Accuracy metrics produces a Precision rate of 73.33%, a Recall rate of 100%, and an Accuracy rate of 73.33%. Apart from that, manual calculations were also carried out which resulted in an accuracy rate of 91.77%.