Automated activity classification is essential for passive military perimeter surveillance, where reliable detection of personnel and vehicle intrusion must be achieved without revealing the presence of monitoring infrastructure. Existing approaches often experience performance degradation under complex vibration patterns and environmental noise, motivating the need for a robust classification method. This study proposes a vibration-based activity classification framework that combines discriminative time- and frequency-domain vibration features with a Support Vector Machine using a Radial Basis Function (SVM-RBF) kernel. The principal novelty of the proposed framework lies in its integration of handcrafted multidomain vibration features with maximum-margin nonlinear classification to enable accurate recognition of multiple intrusion activities from passive seismic signals. SVM-RBF was selected because of its strong generalization capability and effectiveness in handling nonlinear decision boundaries in moderate-sized datasets, making it well suited for vibration-based activity recognition. The proposed model was evaluated on the Perimeter Vibration Accelerometer Dataset containing 2,400 balanced signal segments representing four activity classes: Heavy Vehicle, Light Vehicle, Personnel, and Background. Experimental results demonstrate that the proposed approach achieved an overall accuracy of 0.9250 and a macro-F1 score of 0.9247, outperforming k-nearest neighbor, Random Forest, and Multilayer Perceptron classifiers. These findings indicate that the proposed framework provides a reliable and computationally efficient solution for passive military perimeter surveillance, supporting early intrusion detection while maintaining the covert operation of defense monitoring systems. Future work will investigate multi-sensor fusion, deep learning on raw vibration signals, and embedded deployment for real-time operation in diverse environmental conditions.