Rice is the most widely consumed food worldwide. The many types of rice cause various difficulties in the process of classifying rice varieties. The process of manually classifying rice varieties that rely on human power has drawbacks including the subjectivity of assessment between observers, limited physical capabilities, and longer observation times. In this research a rice variety classification system has been developed using the Convolutional Neural Network with the YOLOv7 and VGG16 algorithms. The rice varieties classified are basmati, IR64, and rojolele varieties. The model with the YOLOv7 algorithm is trained for object segmentation of rice grains and is used to create rice grain image datasets. The model with the VGG16 algorithm was trained by transfer learning and used for classifying rice grain varieties. The model with a learning rate hyperparameter of 0,000061, the ReLU activation function, the number of neurons 256 in the second classification layer, with the fine-tuning training method, has the best performance with an accuracy value of 100%. The best VGG16 model weight is used in application implementation. Identification of the type of rice with the application can be done on the image of a batch of homogeneous and heterogeneous rice grains with various arrangements.