Rice grain quality assessment is important for maintaining product consistency and quality standards in the food industry. Manual inspection is still widely used in practice, but it is often subjective and time-consuming especially in fine-grained classification problems where the visual differences between classes are relatively subtle. Another common challenge in rice grain datasets is class imbalance where the number of normal samples is much larger than the number of defective classes. In this paper, we investigate various imbalance handling strategies for deep learning-based rice grain classification using the GrainSet dataset, which contains 30,962 rice grain images divided into eight quality classes with a highly imbalanced class distribution. The proposed approach combines weighted sampling, data augmentation, and hyperparameter optimization to address class imbalance, while Focal Loss and Class-Balanced Loss are used as comparison methods for performance evaluation. Three deep learning architectures, namely MobileNetV3, ResNet50, and ViT-Small, were evaluated using Accuracy, Macro F1-score, and Weighted F1-score metrics. Experimental results show that the proposed approach achieved the most consistent overall performance. Among the tested models, ResNet50 achieved the best result with 98.87% accuracy and a Macro F1-Score 0.9799. The results also show that convolution-based architectures are more stable than transformer-based models for texture-oriented datasets with limited training data. Furthermore, the combination of sampling-based balancing, augmentation and suitable hyperparameter configuration contributed to better recognition performance on minority classes.