Coffee is one of the most consumed beverages today. The coffee beans are first sorted by the farmers. This is because there are many types of coffee beans that differ in terms of shape and texture. After sorting, farmers must detect whether the coffee beans are damaged or not. The process is still done manually by coffee farmers so it takes a long time and results in errors due to lack of knowledge about coffee. In addition, efforts are also being made to improve the quality of the coffee beans which will affect the selling value of the coffee beans. Based on these problems, this study aims to design a deep learning model to detect coffee bean damage and evaluate the architecture of ResNet-34 and VGG-16. The classification model built using a Convolutional Neural Network (CNN) is expected to be able to know a better architecture and be able to detect damaged or normal coffee beans accurately and precisely