Manual post-harvest sorting of tomato fruit is prone to subjectivity and inconsistency, necessitating an automated quality assessment approach. Single Shot Detector (SSD) and EfficientNetV2 are both included in the Deep Learning architecture for efficient object detection and classification. This research develops a hybrid model that processes SSD data through a single direct detection, making it lighter than other methods, while EfficientNetV2 serves as the backbone model, capable of producing deep features efficiently. The design of the hybrid SSD-EfficientNetV2 model for the automatic detection and classification of tomato fruit quality (Solanum lycopersicum) into two classes, namely Grade A with fresh and marketable fruit conditions and Grade B with damaged or rotten conditions, is expected to replace the manual sorting process, which is prone to inconsistencies. The data was directly collected from the sales centers and local tomato farms in Nganjuk Regency. The obtained data underwent preprocessing, including resizing, normalization, and augmentation in the form of brightness adjustment, contrast, and hue saturation manipulation. The data is divided into 60% training data, 15% validation data, and 25% testing data. The model was trained for 32 epochs using the AdamW optimizer with a learning rate warm-up and cosine decay scheme. The final evaluation resulted in a classification accuracy of 95.12%, a macro F1 Score of 95.11%, and a Mean Average Precision (mAP) of 85.70% with a precision of Grade A at 94.87% and Grade B at 95.35%. The proposed model offers a reliable contribution as a foundation for an artificial intelligence-based sorting system in the post-harvest tomato industry.