Malay buildings have distinctive architectural characteristics and require technology-based identification systems to support cultural documentation and preservation. This study aims to develop an identification system for Malay and non-Malay buildings using video data extracted into frame-by-frame images. The use of video data in this study is not intended to analyze the physical movement of buildings, but to utilize visual variations caused by changes in camera angle, recording distance, lighting, object composition, and visible building elements. The proposed method is CNN-LSTM, where CNN extracts visual features from each frame, while LSTM learns inter-frame feature relationships as a sequence of visual information. To reduce redundant information between adjacent frames and minimize the risk of excessive similarity between training and testing data, the number of frames was limited to a maximum of 15 frames per video folder, and data splitting was performed by considering video source groups. The dataset consists of Riau Malay buildings, Kalimantan Malay buildings, and non-Malay buildings. The research stages include frame extraction, image resizing to 224×224 pixels, normalization, data augmentation, class labeling, train-test splitting, modeling, evaluation, GroupKFold validation, and web-based system implementation. The best testing scenario was obtained using an 80:20 data split, 80 maximum epochs, and a batch size of 16, achieving an accuracy of 0.9916 and a test loss of 0.0852. GroupKFold validation produced an average accuracy of 99.1% with a standard deviation of 0.5%. These results indicate that the model can recognize architectural visual patterns, such as roofs, windows, doors, ornaments, and overall building appearance, while the performance should still be interpreted within the scope of the dataset and evaluation scenario used in this study.