Coffee bean quality classification is a critical process in the agricultural industry, yet it faces significant challenges due to imbalanced data distributions that degrade standard classifiers such as Support Vector Machines (SVM). This study proposes an end-to-end machine learning framework following the CRISP-DM methodology to optimize SVM hyperparameters—specifically the penalty parameter (C) and kernel coefficient (γ)—using a continuous Genetic Algorithm (GA). A 32-dimensional visual feature extraction pipeline combining HSV color moments, RGB histograms, and grayscale texture statistics was engineered from 1,411 coffee bean images across five distinct classes (peaberry, longberry, premium, defect, and miscellaneous). To prevent data leakage, a stratified 5-fold cross-validation was implemented exclusively on 1,128 training samples to compute the cost-sensitive Macro F1-Score fitness function. The evolutionary search successfully identified optimal continuous parameters (C = 838.9254, γ = 0.0016). Experimental results on an independent holdout test set of 283 samples demonstrated that the proposed GA-SVM significantly outperformed the default baseline, elevating classification accuracy from 77.39% to 85.16% and the Macro F1-Score from 0.7742 to 0.8496. Notably, true positive detection for the commercially critical "defect" class rose from 38 to 52 instances, yielding a recall of 0.87 compared to 0.63 for the baseline. The optimized pipeline was deployed into a real-time Gradio web application featuring automated inference and Canny edge verification with latency under 150 ms, providing an accessible, interpretable decision support tool for small-scale agricultural processing facilities.